Private AI Infrastructure & LLM Gateway Setup

01 What gets built

Fig. 01 — The gateway is the seam: left of it nothing changes, right of it everything can.

02 What you receive

All of it open source. None of it ours.

There is no Guztia software in this arrangement and no licence to renew. If you stop working with us, everything keeps running.

Layer	What you get	Why it matters
Gateway	One endpoint in front of every model	Budgets, keys, fallbacks and routing in one place. Makes everything behind it swappable.
Record	Every request traced and costed	Cost per team, latency and quality over time. Self-hosted, so the record stays inside your company.
Serving	Private models on capacity in your name	A cost per unit of work you can forecast — where the economics justify it, and not before.
Compute	Your cloud, your region, your discounts	Placed where your data and users already are. Sized against measured traffic, not a guess.
Identity	Your existing corporate sign-in	Cost lands on a real person and team; revoking access revokes model access too.
Everything	Infrastructure as code, in your repository	Reviewed in your pull requests, with a runbook written for whoever operates it next. We do not hold the state file.

The specific open-source components and their configuration are chosen per workload and named in your proposal, under NDA, before you commit.

03 How it runs

Four stages, each one reversible.

We do not start by deploying anything. We start by finding out what you actually send to model APIs today, because that number is usually a surprise.

1 — Measure

Where AI is used today, by whom, at what volume and what cost — and what your data obligations actually are.

You get: a baseline you can argue with, and which workloads move first.

2 — Serve

Private model capacity stood up in your account and proven against your own traffic rather than a synthetic benchmark.

You get: measured throughput, latency and cost per unit of work.

3 — Control

The gateway and the usage record go live, tied to your identity provider, with budgets and a capped premium route. Reporting built for the questions finance will ask.

You get: one endpoint, every call traced and costed.

4 — Hand over

Engineers, services and pipelines cut over with a short internal guide. Then the documentation and a walkthrough with whoever will own it.

You get: the arrangement, the code, the runbook, and a team that can operate it without us.

04 The engineer-facing change

This is the whole migration.

Not a rewrite, not a new SDK, not a platform your team has to learn. Everything already speaks the same widely-adopted API format — the assistants, editors, frameworks, your own service code, your CI evals. The gateway answers in that format.

Which is also why the reverse is true: if you ever want to go back to a vendor API, that is the same two lines in the other direction. We think that matters. A migration you cannot reverse is not a migration, it is a different lock-in.

~/.zshrc — engineer workstation diff

- OPENAI_BASE_URL=https://api.openai.com/v1 - OPENAI_API_KEY=sk-proj-… # billed per token, untraced + OPENAI_BASE_URL=https://llm.yourco.internal + OPENAI_API_KEY=sk-… # your gateway, your budget, your records

That is the entire migration on the engineer's side. Everything behind the URL — which model answers, which GPU it runs on, what it costs, who asked — is now yours to change.

05 What it costs

SCOPE A

Single model, single team

€8k–€12k

One open-weight model, gateway, tracing, one team migrated. Sized to your current infrastructure. Two to three weeks.

SCOPE B

Multi-model, org-wide

€15k–€25k

Several models across tiers, SSO, per-team budgets, autoscaling, vendor fallback, whole engineering org migrated. Three to four weeks.

SCOPE C

Regulated or cross-border

Quoted

Data residency constraints, audit requirements, mainland China presence, air-gapped environments. Scoped after week one.

Fixed fee, not time and materials. GPU and cloud costs are yours and billed by your provider directly — we never resell compute, because a margin on your inference would give us an interest in you using more of it. We model the break-even from your actual traffic before recommending anything — that is what the assessment is for.

06 The questions everyone asks

Do we need our own GPUs?

No. Most clients rent capacity in their own cloud account. Buying hardware only makes sense at sustained high utilisation, and we will tell you plainly when you are nowhere near that. Sometimes the honest answer is "keep using the vendor API for now, but put the gateway in so you can see what you're spending."

Can we keep OpenAI or Anthropic?

Yes, and most clients do. The gateway keeps a route to a premium model for the work that needs one, with a budget cap on that path. You get the cost curve of self-hosting and the ceiling of the best available model, and you can finally see the split.

Which model should we run?

Whichever wins on your work, which is not the same as whichever wins on a leaderboard. We test candidates against your own traffic before anything moves, and we expect the answer to change within a year — that is what the gateway is for.

What if the model is worse?

For some tasks it will be, and you should know which ones before you migrate them. That is the point of measuring first and of keeping a fallback route. Nothing moves to a self-hosted model until it passes on your work.

Which components do you deploy?

Established, permissively licensed open-source software throughout — no Guztia code and no licence to renew. The exact selection depends on your workload and existing infrastructure, and is set out in your proposal under NDA before you commit to anything.

What happens if we stop working with you?

Everything keeps running. The infrastructure is in your cloud account, the code is in your repository, the components are open source, and the runbook is written for someone who is not us. There is no Guztia component to remove.

Have you actually done this?

We run this exact arrangement in our own production and have operated infrastructure for other organisations since 2015. Ask on the call and we will walk you through a live instance — the real reporting and controls, not a slide.

AI Infrastructure Setup