Why Open-Source Models Haven't Killed the Big Dogs
Spoiler alert: Inference is Hard
Over the last few weeks, I’ve spent a ton of time talking with investors.
Who knew trying to build a neocloud with weird hardware would get this much attention?!
Obviously, I’m flattered. Nothing validates my existence as a founder quite like a bunch of people with bags of cash who might actually give some of it to me.
I’m joking. Mostly. 😂
Putting aside all the questions about whether CueCloud can win in an increasingly crowded neocloud market, one much bigger question keeps coming up:
If open-source models are getting this good, why isn’t everyone serving them? And why haven’t they killed OpenAI and Anthropic?
Okay, technically that’s two questions. But they both get at the same thing: if model quality is becoming commoditized, why are OpenAI and Anthropic still making ungodly amounts of money while charging more for their best models?
The short answer is that open weights aren’t the same thing as usable inference. A model can be free to download and still be expensive, annoying and surprisingly difficult to serve reliably.
OpenAI and Anthropic aren’t just selling intelligence. They’re selling the convenience of accessing that intelligence through an API.
And as we’re about to see, that convenience covers up an absolutely disgusting amount of infrastructure work.
Part 1: “Just Self-Host It” Is Doing a Lot of Work
1.1 The Fantasy
The pitch for open-source AI sounds almost offensively simple.
Go to Hugging Face, find a model that performs nearly as well as the best closed models and download it for free. Rent some hardware, get the model running and put an API in front of it.
Congratulations. You’ve replaced OpenAI.
Somebody should probably tell Sam Altman that it’s over.
Honestly, I understand why people think about it this way. I had basically the same reaction when we first started looking seriously at open models. If the model is free and the hardware costs less than the API bill, what exactly are we still paying OpenAI and Anthropic for?
And if you’re only trying to get a model running for yourself, this argument isn’t completely wrong. The tooling has gotten good enough that a technical person can rent a machine, follow some documentation and start generating tokens without building everything from scratch.
That’s usually the moment where people stop doing the math.
And if all you look at is the sticker price, the math can look pretty compelling.
Runpod currently lists an H100 SXM for $2.99 an hour. Meanwhile, GPT-5.5 costs $30 per million output tokens.
Let’s do some extremely irresponsible napkin math.
Suppose your open model generates 50 tokens per second on that H100. That number is intentionally hypothetical. Actual speed will depend on the model, quantization, runtime, context length and how many requests you’re processing at once.
But 50 tokens per second gives us a nice, round example.
If you kept the GPU generating at that speed for a full hour, it would produce 180,000 output tokens. Buying the same number of output tokens from GPT-5.5 would cost $5.40.
The H100 costs $2.99.
Holy shit. Fire Sam Altman. We’ve solved inference.
Except that comparison quietly assumes your GPU is generating tokens every second of every hour. Real usage generally doesn’t arrive in one perfectly smooth line. People go to lunch. Developers stop working. Traffic spikes, disappears and then returns at exactly the moment you’ve decided to change something.
At 25% utilization, the same GPU would produce only 45,000 output tokens during that hour. Those output tokens would cost $1.35 from GPT-5.5, while your H100 still costs $2.99.
Using these admittedly crude assumptions, the GPU needs to average roughly 28 output tokens per second across the entire hour just to break even against GPT-5.5’s output pricing. If it tops out at 50 tokens per second, you need to keep more than half of its capacity busy all the time.
And this is the generous comparison.
We’re treating our hypothetical open model and GPT-5.5 as if they provide the same quality, consume the same number of reasoning tokens and complete the same amount of useful work. Obviously, that isn’t always going to be true. A cheaper model that needs more attempts, writes worse code or requires more human correction may not actually be cheaper at all.
The math gets even less exciting when you compare self-hosting with an API that already serves open models. Together AI currently charges $1.04 per million tokens for Llama 3.3 70B. At that point, you’re not choosing between an expensive closed model and a cheap open model. You’re choosing between operating the open model yourself and paying someone else a relatively small amount to make it their problem.
That doesn’t mean self-hosting is a bad idea. At high enough utilization, it can be dramatically cheaper. You also get more control over your models, your data and the hardware underneath everything.
But the simple comparison between an hourly GPU price and an API token price leaves out most of the actual problem.
The real question is whether you can keep the hardware busy enough, serve enough people at once and operate the whole thing cheaply enough for those savings to survive contact with reality.
1.2 The $2.99 H100 Doesn’t Come With an API
But even our extremely irresponsible napkin math skips an important step.
We’ve been talking as if renting an H100 means you now have a working model endpoint. It doesn’t. It means you have access to a very expensive chip in somebody else’s data center.
That chip might come attached to a machine with system memory, storage, networking, an operating system and working NVIDIA drivers. Depending on the provider, you may also get a convenient template with some common software already installed.
What you don’t necessarily get is a model that your products, coding agents and internal workflows can immediately talk to.
First, you have to get the model onto the machine. For a large model, that can mean downloading hundreds of gigabytes of weights, choosing the right precision and making sure the whole thing fits in the available GPU memory.
I’ve already written an obscenely long article about memory and inference deployment that gets into this particular headache. The short version is that fitting a model into memory is hard, especially once it has to be split across several pieces of hardware.
But fitting the model still isn’t enough.
At that point, you have a large collection of numbers sitting on a machine. You still need software that knows how to load those numbers, move requests through the model and turn the results back into text.
This is the job of an inference runtime such as vLLM, SGLang or Hugging Face’s Text Generation Inference. These runtimes do much more than call the model. They handle things like tokenization, memory allocation, batching, streaming and communication between GPUs.
They can also expose the model through an HTTP server. vLLM, for example, can create an OpenAI-compatible API. In theory, that means an application already written for OpenAI can point at your server instead by changing the URL and API key.
That’s a huge improvement over building the serving layer from scratch. It’s also where the phrase “we got the model running” starts becoming a little slippery.
If vLLM is listening on localhost:8000, the model is technically serving requests. But it’s only accessible from that machine. Your coding agent, production application and coworker sitting in another state can’t do much with it yet.
Now you have to make the endpoint reachable.
That means deciding how traffic gets into the machine, encrypting the connection, issuing API keys and making sure one leaked credential doesn’t let a stranger use your H100 as their personal chatbot. You’ll probably want rate limits too, unless you’re comfortable letting one enthusiastic engineer consume the entire machine with a single script.
Then more than one person starts using it.
Requests now need to wait somewhere when the model is busy. Some requests can be grouped together so the GPU processes them more efficiently. Others have enormous prompts, ask for long outputs or get cancelled halfway through.
The serving runtime has to keep all of that organized without letting one giant request ruin the experience for everyone else. This is why production engines talk so much about features like continuous batching and streaming. Hugging Face’s serving documentation lists both alongside tensor parallelism, quantization, tracing and metrics because they’re all part of turning a model into a service.
And you actually need those metrics.
It isn’t enough to know whether the server is technically alive. You need to know how long requests are waiting, how full the batches are, how quickly tokens are being generated and whether memory is about to run out. Hugging Face’s TGI runtime exposes an entire collection of Prometheus metrics just to monitor batch size, queueing, decoding time and inference performance.
None of this means you need to build every component yourself. Modern runtimes and neocloud platforms automate a lot of it.
But there’s an important sliding scale here:
A raw GPU rental gives you hardware and leaves most of the serving work to you.
A managed deployment handles more of the runtime and operational setup.
A serverless endpoint hides nearly everything and charges you for usage.
The further you move down that list, the less the product looks like a cheap GPU rental and the more it starts looking like the hosted API you were supposedly trying to avoid.
So the $2.99 hourly price is real. It just isn’t the full price of inference.
It’s the price of the hardware before you’ve paid for the software, setup and people required to turn it into something your company can actually use.
Part 2: What Does It Actually Cost to Run Inference Yourself?
At this point, I want to stop hand-waving and try to calculate the fully loaded cost of doing this yourself.
And by fully loaded, I don’t mean taking the hourly GPU price and multiplying it by 720 hours. That part is easy. I mean the hardware, the capacity you aren’t using, the backup plan, the people keeping everything alive and the random operational expenses that inevitably appear once other humans begin depending on your endpoint.
This obviously won’t produce one universal number. Every model, workload and company is different. But we can at least build a reasonable example, state our assumptions and see where the money actually goes.
Let’s pick something real
I’m going to use Llama 3.3 70B because NVIDIA publishes real performance numbers for the model, and Together AI publishes a hosted price for it. That gives us a reasonably clean comparison between running an open model ourselves and paying someone else to run the same model.
Here’s the setup NVIDIA tested:
Model: Llama 3.3 70B
Precision: FP8
Hardware: Two H100s
Concurrent requests: 50
Input per request: 5,000 tokens
Output per request: 500 tokens
Aggregate output speed: Roughly 814 tokens per second
Output speed per active request: Roughly 16 to 17 tokens per second
Time to first token: Roughly 835 milliseconds
Those numbers seem perfectly usable for a shared internal service. Fifty requests can remain active at once, each user sees tokens arriving at a reasonable speed and the first token appears in under a second.
There’s an important caveat, though. This is an optimized NVIDIA deployment using FP8 precision and tensor parallelism. Renting two H100s doesn’t guarantee that you’ll immediately reproduce these results, but they’re still much better than pulling an imaginary throughput number out of my ass.
Also, this isn’t a comparison between Llama and GPT-5.5. We’ll come back to model quality later. For now, I want to isolate the cost of serving the same open model in two different ways.
The number that looks amazing
Lambda currently charges $4.19 per hour for each H100 in a two-GPU instance. That gives us:
So the number that initially goes into the spreadsheet is $6,034 per month.
If the deployment ran at NVIDIA’s benchmarked throughput for every second of the month, it could generate around 2.1 billion output tokens. The benchmark uses ten input tokens for every output token, so the same traffic pattern would also process roughly 21.1 billion input tokens.
That gives us approximately 23.2 billion total tokens.
Together AI currently charges $1.04 per million input or output tokens for Llama 3.3 70B. Sending the same 23.2 billion tokens through Together would cost a little over $24,000.
This is where self-hosting looks awesome. You’re running the same model and, at least on paper, saving around 75%.
Unfortunately, we’ve assumed those GPUs are running near their limit every second of the month. That’s a fairly heroic assumption for an internal service used by actual people.
The utilization problem
Usage doesn’t arrive in one smooth line. Engineers sleep, weekends exist and half the company can disappear into meetings for several hours. A coding agent might hammer the endpoint for twenty minutes and then sit idle while tests run.
Together only charges you when tokens move through its API. Lambda continues charging you because the machine is still sitting there.
Once we account for that, the comparison starts changing:
At around 25% utilization, the API and the raw GPU rental cost almost exactly the same.
If you can keep the GPUs 50% or 75% busy, the rental starts looking really good. If the endpoint mostly sits around waiting for someone to use it, the API wins without doing anything particularly clever.
There’s another subtlety here too. High GPU utilization doesn’t always mean the machine is producing valuable work. You can keep a GPU busy processing wasteful prompts, generating outputs nobody uses or repeatedly running a model that isn’t good enough for the task.
What matters is how much useful work comes out of the hardware you’re paying for.
Now somebody has to keep it alive
The $6,034 figure also assumes we’re comfortable running one deployment with no real fallback. If that machine dies, the endpoint dies with it.
Maybe that’s fine when five engineers are testing a model. Everyone can go get coffee while the machine restarts. It’s different when fifty engineers use the endpoint all day or when the model sits inside a product that customers are paying for.
There are several ways to deal with failures:
Keep an identical backup deployment ready
Maintain a smaller fallback model
Share spare capacity across several models
Restart the deployment and accept some downtime
Send emergency traffic to an external API
For this example, we’ll use the simplest and most expensive option: another two-H100 deployment. That immediately doubles our monthly compute bill to roughly $12,068.
A real company could almost certainly design something smarter. But if the endpoint matters, the backup plan still needs to exist somewhere, and it probably won’t be free.
The engineer doesn’t materialize for free
This is the cost I see left out of self-hosting comparisons all the time. Somehow the GPU always has a precise hourly rate, but the engineer responsible for making it useful is treated like a woodland creature who appears whenever the endpoint needs help.
Someone has to own:
Deploying and upgrading the model
Testing quantization
Tuning batching and concurrency
Monitoring latency and memory
Managing API keys and authentication
Setting rate limits
Planning capacity
Applying security patches
Responding when the endpoint gets weird
For the sake of this example, let’s assume an experienced inference engineer costs the company $250,000 a year after salary, benefits, payroll costs and everything else attached to employing someone. That comes out to roughly $20,800 per month.
Maybe this deployment only consumes half of that person’s time once it’s stable. I think that’s a fairly generous assumption for a service used heavily by fifty people, but we’ll use it. Half of the engineer adds approximately $10,400 per month.
You can absolutely argue with this number. If one engineer operates a large fleet used by thousands of people, the cost per deployment drops substantially. If the model is constantly breaking, being upgraded or getting replaced, half an engineer may be wildly optimistic.
The point is simply that the person belongs in the spreadsheet.
The smaller stuff still exists
There are also a bunch of costs that probably won’t destroy the economics individually but still need to be paid:
Model and container storage
Logs and metrics
Network traffic
Monitoring software
Secrets management
Security tooling
Backups
Taxes and support plans
Some providers include parts of this in the instance price. Others will find creative new ways to send you a bill.
I’m going to use $1,500 per month for these expenses. That isn’t an industry benchmark. It’s a visible assumption that we can replace with a different number depending on the provider and deployment.
Now we can look at the whole bill
We started with a $6,034 GPU rental. After adding a backup deployment, half of one engineer and some basic operational costs, we’re sitting at roughly $24,000 per month.
That’s almost exactly what Together would charge if our primary deployment ran at full capacity for the entire month. If usage is lower, the hosted endpoint becomes substantially cheaper because its bill falls while most of our self-hosted costs stay fixed.
None of this proves that self-hosting is a bad idea.
Self-hosting starts looking much better when:
You can keep the hardware consistently busy
You already employ the necessary infrastructure talent
One team can operate several models and deployments
You can spread backup capacity across multiple workloads
You own the hardware and use it for several years
Privacy and control justify some additional cost
Those are real advantages. They’re a huge part of why companies eventually bring inference onto infrastructure they control.
But self-hosting doesn’t become cheaper simply because $8.38 per hour looks smaller than $1.04 per million tokens. The GPU rental is one line item.
Inference is the whole bill.
Part 3: The API Is the Actual Product
At this point, you can probably see why most companies don’t want to self-host anything.
It’s not that they couldn’t download an open model, rent a few GPUs and eventually get it running. They probably could. The problem is that they’d also inherit everything we just spent the last two sections talking about: utilization, batching, scaling, authentication, monitoring, failures and upgrades.
Most companies don’t actually want a model. They want an endpoint.
They want to send text to a URL and get an answer back. They want that URL to keep working when traffic spikes, when a GPU dies and when someone accidentally launches an agent that tries to read the entire company Google Drive before lunch.
That’s why OpenAI and Anthropic remain so hard to displace. They aren’t just selling intelligence. They’re selling intelligence with all the annoying shit already handled.
Open Models Already Have Their Own Convenience Layer
Self-hosting and calling OpenAI aren’t the only two options.
Companies like Fireworks AI, Baseten and Together AI have built a pretty compelling middle ground. You can access open models through a familiar API while somebody else handles the infrastructure underneath them.
Depending on the provider, that can include:
Loading and configuring the model
Optimizing the inference runtime
Batching requests
Autoscaling
Monitoring
Replacing failed hardware
Managing capacity
Providing an OpenAI-compatible endpoint
Fireworks offers both serverless inference and dedicated deployments. Baseten offers hosted model APIs and dedicated infrastructure. Together has a large catalog of serverless and dedicated open-model endpoints.
In other words, these companies have taken most of the mess we just described and wrapped it in something developers can actually use.
Naturally, that convenience costs money. Fireworks currently lists dedicated H100 and H200 capacity at $7 per GPU hour, while B200s run $10 per hour. That’s considerably more than renting a raw GPU, but it’s also not the same product. You’re paying someone to turn that GPU into something your company can actually depend on.
It’s basically the same problem Spider-Man ran into when The Odyssey hogged the IMAX screens. You can have the movie ready to go, but you still need somewhere to show the damn thing.
Open models work the same way. The weights might be free, but getting them in front of users quickly and reliably definitely isn’t.
So Why Not Use One of Them Instead of OpenAI?
Increasingly, people do.
The growth of these inference providers is proof that companies want open models without the joyless responsibility of running them. But giving an open model a great endpoint doesn’t erase every advantage held by OpenAI and Anthropic.
The most obvious issue is still model quality. Fireworks can make an open model as easy to call as GPT, but it can’t guarantee that the model will behave exactly like GPT inside your application.
Models handle tool calls differently. They format structured outputs differently. They follow system prompts differently. One model might crush a coding benchmark and then completely lose its mind when asked to call the same function four times in a row.
That matters because companies don’t evaluate models in a vacuum. They evaluate them inside workflows.
If a company has built its coding agent around Claude’s tool-calling behavior, switching models isn’t always as easy as changing the URL. Prompts may need to be rewritten, evaluations need to be rerun and old edge cases suddenly reappear.
OpenAI and Anthropic also control the entire stack. They build the model, tune the infrastructure and design the surrounding products together. A direct inference provider can optimize the hell out of an open model, but it doesn’t control how that model was trained or what weird behavioral quirks came along for the ride.
Then there’s the boring advantage nobody likes talking about: incumbency.
OpenAI and Anthropic already have enterprise contracts, security reviews, billing relationships and APIs buried inside thousands of applications. Developers understand their documentation. Procurement has already approved them. Finance recognizes the invoices.
A competing model can’t merely be cheaper. It has to be cheap enough, good enough or strategically important enough to justify reopening all of those decisions.
That’s a much higher bar than winning a benchmark.
Where This Gets Dangerous for the Big Dogs
The threat becomes real when three things happen at the same time: open models get good enough, inference providers make them easy to deploy and the final price is meaningfully lower.
Quality is only the first piece. An open model might come within a few benchmark points of Claude or GPT, but that doesn’t help much if serving it still requires a small army of inference engineers. Fireworks, Baseten and Together are important because they’re solving that second problem. They’re turning open weights into reliable APIs that companies can actually plug into their workflows.
Then there’s price. Fireworks currently lists DeepSeek V4 Pro at $1.74 per million input tokens and $3.48 per million output tokens. Baseten lists the same model at the same headline prices.
Because the weights are open, multiple providers can serve the same model and compete on price, latency, uptime and developer experience. If one provider gets too expensive, customers can move elsewhere without abandoning the model entirely.
That’s very different from Claude or GPT. If you want Claude, Anthropic ultimately controls the supply. If you want GPT, OpenAI controls the supply. Nobody can quietly download GPT-5.6, optimize it for cheaper hardware and start undercutting OpenAI.
That’s where this gets dangerous. Once the model quality is close enough, the inference stack has been handled and the price is substantially lower, the closed labs are no longer competing against a research project. They’re competing against a complete product.
A Tiny CueCloud Plug 😅
At CueCloud, we’re betting that memory-heavy, consumer-grade hardware can make open-model inference much cheaper. We handle the messy infrastructure. Developers get a simple endpoint and predictable pricing.
Okay, plug over.
That’s why open-source models haven’t killed the big dogs yet. The models may be open, but making them reliable and easy to use is still hard.
As inference providers remove more of that headache, OpenAI and Anthropic lose one of their biggest advantages. At that point, they’ll have to compete much more directly on model quality and price.