r/badeconomics • u/Uptons_BJs • Jun 22 '26
No, you're not really getting "$8000 worth"of AI usage with your $200 subscription - The weird economics of AI Inference
The badeconomics:
There exists tons of posts claiming that your monthly AI usage plan is insanely subsidized, for instance, this one:
What happens when they stop subsidizing LLM subscriptions? : r/LocalLLaMA
We are literally burning through VC money like crazy with our coding subscriptions. I read the $200 Anthropic sub gets you $8000 worth of API calls.
Now, the logic behind this often comes from studies like this:
A $200 ChatGPT Pro 20x subscription could cost as much as $14,000 in API pricing if fully utilized. Anthropic's Claude Max 20x plan, also priced at $200 per month, has a comparable ceiling, with potential usage totaling roughly $8,000 in token costs.
This is correct, it is true that, if you compare token costs with subscription costs, you might be able to consume enough tokens so that it would cost you $14,000 if you paid per token, but the following claim that the article makes is incorrect:
Anthropic breaks even on Claude Pro and Claude Max 5x at around 20% utilization. OpenAI's margin is thinner. It begins losing money on ChatGPT Plus and ChatGPT Pro 5x once usage climbs above 11.4%.
Let me try to explain the economics of AI inference, and how these plans actually work.
First some terminology and definitions if you're not familiar with LLMs
Ok, so the type of AI we are discussing here is the large language model, and there's a few concepts that you should be familiar with to make the following discussion make sense.
- AI inference - AI inference refers to using a model, running an already trained model to process data. Here's a good explainer from Google.
- Token - in the AI context, a token is the smallest piece of information that the model can process. The data you pass into the model gets broken down into individual tokens, which are then processed. Here's a primer from Nvidia.
- Input vs Output token - Input tokens refer to the information you enter into the model, broken down into tokens. Output tokens refer to the information generated by the model, broken down into tokens.
If you look at an AI provider's pricing, they would typically offer you a price for input tokens and output tokens. for instance, over at OpenAI, GPT 5.5 costs $5/1 million input tokens and $30/1 million output tokens.
The simple R1:
According to OpenAI's documents submitted to the SEC, they are turning a gross profit on AI inference. In 2025, their revenue was $13.07 billion, while their cost of revenue was $7.5 billion. From Arstechnica.
According to the Wall Street Journal, Anthropic is actually turned an operating profit last quarter.
Since I don't work at any of these firms, I do not have the breakdown of their financials exactly. But it is pretty safe to say that AI Providers have positive gross margins on AI inference services. But let me try my best to break down the numbers a bit and explain why "you can get $14000 worth of tokens for $200" could be true, yet AI inference as a whole can still be profitable.
Why AI Inference can be profitable:
The best analogy I have for AI inference is the gym. Think about how a gym operates:
- It costs money up front to build the gym and purchase the weights and machines
- The weights and machines depreciate over time
- It costs money to upkeep and operate the machines
- You pay a monthly price to access the gym
- When the gym gets really busy, you might not be able to get in, or they might kick you out for hogging the machines
Now the one difference between the gym and AI is:
- AI companies offer "per token" billing, the gym doesn't offer "per rep" billing.
Now imagine if the gym offers a per rep billing model - You pay 2 cents every time you lift a weight! Now imagine if some bodybuilder who goes to the gym as much as possible does the math and says "I only paid $100/month for my gym membership, but if I go to the gym as much as possible, this membership would have provided me with $4000 worth of reps! The gym loses up to $3900 per membership!"
Let me start by saying, I don't know how much hardware these closed models truly require. And how many tokens per second the setups at these AI companies can do, but we can do some speculation and back of napkin math here.
AI has a massive up front cost. You need to purchase servers and GPUs, which depreciates. (as an aside - Depreciation math for GPUs is really weird right now, this stuff effectively doesn't depreciate right now, but do understand that we're in a really weird time.) Then there is the fixed operating costs - Renting racks in the data center (or renting the whole thing), the staff to administer and maintain the hardware, the internet connections, the insurance, etc, etc.
Now there is an incremental cost in electricity - If the GPU or if the whole server isn't in use, it could be powered down.
The Nvidia H200 GPU (one of the current top data center GPUs) has a TDP of up to 700W. Which you can understand it as if you run it full tilt, the GPU consumes 700W, or 0.7 kWh if you run it for an hour.
Assuming electricity costs 10 cents per kWh, and ignoring the rest of the system (CPU, RAM, etc for now):
- If your model needs 1 H200, to run it for an hour costs you 7 cents
- If your model needs 4 H200s, to run it for an hour costs you 28 cents
Now considering that the current top models run at a speed of 50 - 70 output tokens per second*60, if you assume an average of 60, to run it for one hour you're looking at 216,000 tokens per hour if you run 1 concurrent instance.
Depending on how many GPUs this model needs, it costs the company somewhere between 7 - 28 cents per hour to generate these 216,000 tokens. At a pricing of $30/million output tokens, that's a bit over $6 in tokens generated.
Look, I understand that this is all handwavy and relies on a ton of assumptions, but my basic argument here is this: If you pay per token, AI companies use their GPUs and LLMs to convert pennies worth of electricity into dollars worth of tokens.
This is also why a lot of providers have some sort of "off peak discount". At OpenAI, if you use the Batch API, where you send in your requests and have them process it at their convenience (instead of synchronous usage), you get a 50% discount.
Look, I know exactly what you're going to say right now - But what about depreciation?
As far as I can tell, data center hardware does not go down in value based on usage, but primarily based on age. It's not like a car where each additional mile drives down value, with GPUs, each additional token doesn't decrease value further.
The key thing to understand is that the cost for each incremental token, assuming that there is sufficient hardware, is trivial. Token pricing is arbitrary and set at a level to maximize profit for the provider.
Now let's circle back to monthly subscriptions -
When you use the service a lot at a time, especially at peak times, you will get rate limited and throttled. Why? That's because they don't have enough capacity. It's like if you go to the gym at peak times and it is full, you have to wait to use the machines and weights. But when the provider is not at full capacity, when you use the service, it only costs the provider a very small amount, so of course you get a lot of usage off peak - just like how you can walk into the gym when it is empty in the middle of the night and use their machines as much as you want.
The studies that show you get "$8000 worth of tokens for a $200 subscription" are done with a script that use the service as much as possible, thus, most of the tokens that they are using is off peak, when the providers have excess capacity.
And you know what, honestly, at the end of the day, AI inference is a lot like say, an all you can eat buffet - Every once in a while some guy with a massive appetite can make the restaurant lose money, but that's ok, since the vast majority of customers don't eat that much. The majority of customers at your local all you can eat spot are not Homer Simpson, just like how the majority of AI users aren't running a script intent on using as many tokens as possible.
Hell, if you talk to anybody who uses an AI subscription heavily recently, and all you hear are complaints about throttling, usage limits, and how there isn't enough capacity when they use it at regular hours (when everyone else wants to use it too).
If AI inference is gross profitable, why are labs like OpenAI losing so much money then?
There are tons of businesses where serving an additional incremental customer is almost free, but could still lose a ton of money. Gyms, movie theatres, SAAS providers, etc. These businesses typically have very high fixed operating costs and upfront costs, despite having a very low cost for serving an incremental customer. AI labs fall into the same category.
Why is this the case? Well, consider this:
Labs like OpenAI have customers because they build leading models. People are willing to pay OpenAI because they want to use OpenAI's model, and OpenAI is the only place to get it.
In order to keep this business running, the labs end up spending huge amounts of money on R&D to build a better model. Look at that OpenAI breakdown from Arstechnica again - their biggest expense is R&D - $19.18 billion last year versus $7.5 billion for cost of revenue.
Even if you ignore R&D, inference requires a large upfront investment (buying the hardware), and large fixed operating costs (renting data center space, paying staff, etc).
FWIW, there are tons of companies out there that offer AI inference without developing their own models. Either they run other people's models and sell you tokens (See: Deepinfra), or just straight up rent you the hardware (See: Runpod)
Do these companies make money? I honestly don't know, but it's not like serving commodity software is an unprofitable business - Web hosting is a profitable business that has been around for decades after all. The demand for this stuff is obviously there - See all the gamers complaining that local AI people have bid up prices for the RTX5090 from ~$1999 to ~$5000 now.
85
u/Freyr90 Jun 22 '26 edited Jun 22 '26
Now considering that the current top models run at a speed of 50 - 70 output tokens per second*60
Oh that's extremely optimistic. Flagship models simply won't run on a single GPU.
Big models with 1T (or more) parameters without quantization would require 32-64 such GPUs (or more) (I think there is also no open data on how big GPT 5.5 or latest Anthropic models, but they are likely bigger). You can be easily close to $15+ /hour (even more if you account for training, marketing, subsidies, other infrastructure), so like what these models actually cost or maybe even more.
18
u/ShareACokeWithBoonen Jun 23 '26
I find this comment itself to be badecon. Badarithmetic even - to get to $15 per hour with OP's stated assumptions, you need over 200 H200s, not '32-64'.
10
u/Mitchford Jun 23 '26 edited Jun 24 '26
You also need to be using them for that long. I’m a lawyer, I use Ai frequently for tasks through the day, my queries take like a minute at most even for the most complex, and I’m only using like maybe 2 to 5 per hour depending on follow ups . Maybe in the motion picture or gaming industry you’d see people running it that way, but that is not at all how it’s currently used in the professional fields.
10
u/banshee10 Jun 24 '26
Everyone building software (and that's a vast number of companies - think every single Fortune 2000 company building stuff in-house) creates large numbers of long-running tasks. Current AI dev tooling pretty much says "what do you mean ONE bonfire for tokens? I want ALL the bonfires and I want them RIGHT NOW. And more bonfires to check the outputs of the first set of bonfires, please.
8
u/PythagorasWasntReal Jun 25 '26
I spent $1200 in Claude 4.8 tokens with about 18 hours of "thinking" last week. Software dev is vastly more costly than legal or philosophical questions.
5
u/banshee10 Jun 29 '26
I suspect it's because no one has built the right tooling for legal yet. It's very likely that we'll see things like Claude Code for every profession, and soon, so everyone will have the ability to spend vast quantities of money on tokens in parallel to do whatever legal tasks you're working on.
Philosophical questions is interesting though; it'd be really easy to spin up a bunch of tasks that basically consumed the entire world supply of tokens attempting to solve philosophical questions. In fact, I'd think that it'd actually be easier to spend tokens on that than it would be to spend tokens on programming. It's the paperclip maximizer, but trying for the best answer to What is the nature of the world we live in.
1
3
u/JoSquarebox Jun 25 '26
Theres a lot of cool tech around splitting models across GPUs and running multiple batches of requests over the same set of GPUs at once (expert parralelism & batch inference), so while yes you might be touching upwards of a hundred GPUs with a request for an individual token, you share that pod with hundreds of other requests.
(not to mention that due to speculative decoding most tokens dont have to be sampled from the big model, but I have no idea how that works)
2
u/RealSuperdau Jun 26 '26
But that is ignoring batch sizes.
A cluster of 32-64 GPUs will probably create thousands of tokens per second, it's just that it has to be split between dozens of requests.
-2
u/hprather1 Jun 22 '26
>easily close to $15+ /hour
Even if that's remotely accurate, it's very easy to see how AI investment will recoup its costs by displacing human workers. Everyone sees these multi-billion dollar announcements of AI investments and thinks it'll never get ROI but $15/hr is nothing for a company once AI is consistently better than the average worker.
20
27
u/Freyr90 Jun 22 '26
It's not 15/hour, it's 15/hour at token rates from OPs post. It's $15-30 per million tokens which can turn into thousands per month.
It's also extremely hard to estimate ROI of these tokens. Many companies start noticing that work is not being done much faster despite running out of token budgets. Some argue that things llms do either never were the bottleneck, or the bottleneck moved somewhere else (results validation).
We are yet to see if these costs justify productivity boost. My post was about OPs calculations of inference costs.
-2
u/hprather1 Jun 22 '26
Sure, I'm just assuming that AI will continue to improve and companies will eventually find a level of commensurate value to cover their costs. Even if we're in the right ballpark of $15/hr, that seems like an easy level of value to reach.
10
u/VineFynn spiritual undergrad Jun 23 '26 edited Jun 24 '26
Everyone sees these multi-billion dollar announcements of AI investments and thinks it'll never get ROI but $15/hr is nothing for a company once AI is consistently better than the average worker.
You're generalising unconstructively here. What you're saying supposes that these investments in "AI" are in a kind of AI that winds up being able to do that profitably. This is a distinction I see basically noone make, but it is a meaningful one in this context. There's no reason any given investment in "AI" should be considered to be necessarily pushing "AI" in general towards any particular level of future productivity, because investments in "AI" are actually in a particular AI technology, and that technology may have constraints as to its highest profitable level of productivity.
Consider: particular ML investments have been profitable for a long time, in manufacturing for example- if I'm not mistaken- well before the current wave, but the research into that kind of ML was not the sort that gave rise to ChatGPT (in fact it comes out of lossy text compression). LLMs may wind up hitting a dead end before they get over the line you mention in the industries making these investments; a different technology which requires a completely seperate investment may be what's necessary to get there for the jobs LLMs are currently getting thrown at.
Whether these or any other investments wind up actually contributing to that outcome is contingent on the composition of the investments, the cross-applicability of research in different technologies, and how far LLMs wind up being able to profitably go.
The same thing has happened many times in lighting, heating, and engines. It's not a new phenomenon.
11
u/LovecraftInDC Jun 22 '26
Your last line is doing an incredible amount of work.
1
u/hprather1 Jun 22 '26
Is it really? Do you seriously think AI is as useful as it will ever be?
20
u/LovecraftInDC Jun 22 '26
I think that bringing AI to a ‘better than the average worker’ level is a high bar, considering the quality level of a competent worker will continue to rise when supplemented with AI.
5
u/hprather1 Jun 22 '26
Well I was originally going to write "better than the average $15/hr worker." That's an easier bar to clear.
I'm also not quite sure what point you're making. A competent worker that is supplemented with AI is still almost certainly worth more than the additional $15/hr of AI costs. My previous comment was just stating that it's easy to see how the costs of AI can be recouped despite what so many people are saying about the gargantuan investments.
6
u/explohd Jun 24 '26
Productivity drops when employees have to go through the output and verify the results due to AI hallucinations. If the output requires accuracy, anything above 0% hallucination rate is unacceptable. At least the $15/hour worker can ask questions and try to problem solve.
3
u/hprather1 Jun 24 '26
The number of assumptions you are making here is high.
Today's AI is the worst it will ever be. Technological development is a story of constant improvement. We can expect the same of AI.
Humans don't operate with 100% accuracy so the idea that AI should is absurd. It need only be better than humans to be useful.
Even with current AI, it's not at all clear there's a 1:1 tradeoff of output vs output verification. Even if there was, per my point above, AI will only continue getting better.
Have you met humans? I work in IT support. Getting the human on the other end to read and comprehend a plain English error message is a tall order. The assumption that people will ask questions or attempt problem solving is optimistic.
3
u/EebstertheGreat Jun 27 '26
If the kind of investment we see now were made 20 years ago, it would all be for naught. People then could have said "our large language models are already impressive and only improving. Within a decade, we can only expect better results. Our AIs might not do every last thing a worker can do now, but they can do some things well, and soon they will replace the workers entirely."
But they would be mistaken. It turns out that in 2016, our AI models could not in fact do that, and even if investments were made, there is no way the EUV tools could have been available soon enough, let alone high-NA or the chips made by them.
How do we know we aren't in that position now? There is no difference. We have powerful but limited AI tools that we aren't sure will be all that useful in the long run, and which constantly run up against hardware limitations in memory and processing. As far as we could tell, this investment could end up not being anywhere close to worthwhile, our models could hit hard walls like they did in the 2000s, and the whole public will be left holding the bag.
Yet you accuse everyone who is skeptical of making a high number of assumptions. You have to consider the assumptions you yourself are making. It is in no way inevitable that we will see the gains you are predicting.
0
u/hprather1 Jun 27 '26
The people working on this technology are making the investments now, not ten years ago, because they think this is the appropriate time for it else they would have invested a decade ago. Now they think they will make valuable technology with these investments. Your hypothetical doesn't make sense. Clearly the tech industry didn't think 10 years ago was the right time and there's probably a good reason for that.
→ More replies (0)
21
u/RashmaDu Jun 22 '26
I really appreciate your detailed breakdown of the industry, but I think I do disagree with your premise. As you say:
And you know what, honestly, at the end of the day, AI inference is a lot like say, an all you can eat buffet - Every once in a while some guy with a massive appetite can make the restaurant lose money, but that's ok, since the vast majority of customers don't eat that much. The majority of customers at your local all you can eat spot are not Homer Simpson, just like how the majority of AI users aren't running a script intent on using as many tokens as possible.
Ultimately, isn't that the point those kinds of articles try to make? That if you use AI to a large extent (and based on my experience and your quick maths here, that seems broadly correct), if you are a heavy and just modestly strategic user of current tools, you are getting at least a couple times your money's worth in terms of willingness to pay?
I know for a fact if push came to show be I'd be willing to pay 2x my current basic Pro Claude subscription, so anthropic could make more money off me. I also know that I use it for very specific tasks that the average user probably doesn't, and probably gain more from the exercise than the average user does.
All in all, it seems eminently reasonable to me that a lot of "heavy" AI users are turning a profit comparing their WTP and the current cost. Easily 2x, with 10x being very realistic. Being 25% of a Homer Simpson at an all you can eat buffet still nets me significant value, to reuse your analogy. That is also entirely compatible with most users being net profitable to Anthropic (and indeed I'd guess I, who'd "only" be willing to pay 2x what I do now) probably am.
Whether the average person reading these articles and being persuaded to buy an expensive Claude subscription as a result of them will is an entirely different question...
4
u/JustDeetjies Jun 23 '26
I agree with what you’re saying here, I just wanted to note the saying is “when push comes to shove” not show :)
6
16
u/nospacebar14 Jun 23 '26
You're leaving out the costs of pre- and post- training, and you can't get an accurate picture of the costs without it.
AI companies have been categorizing training as a capital expense to make inference look profitable, implying that it's a process that can be done once and then finished. In reality, though, training has to continue forever and should be counted as an operating expense -- if you ever stop training, you can keep serving tokens but the output quickly becomes worthless.
3
u/Jlocke98 Jun 25 '26
Yeah at this point the western labs are what, 3-4 months ahead of zhipu? (glm).
1
u/Trojan_Horse_of_Fate RSS is a market failure Jun 24 '26
> In reality, though, training has to continue forever and should be counted as an operating expense -- if you ever stop training, you can keep serving tokens but the output quickly becomes worthless.
If AI models never got better I would happily still pay 200 USD for them. In fact I'd probably be willing to pay more.
11
u/nospacebar14 Jun 24 '26
"Who's the prime minister of the UK?"
"Keir Starmer"
That's the kind of problem you get if you stop training. You can see how this becomes an issue pretty quickly!
6
u/Trojan_Horse_of_Fate RSS is a market failure Jun 24 '26
That's an out-of-date understanding of how these models work. In case I wanted a model to search for information, I would have it use tool call, not use internal context. Internal context is often wrong and leads to errors
14
u/bneidk Jun 24 '26
Since LLMs can only learn new information from the training stage, the information produced by an LLM will become outdated. It is impossible to use an ever-increasing context window to compensate for this. For simple things like who is the prime minister, sure, you can have the search call, but for knowing about new scientific results or new methods, this would work very poorly without being trained in. There is simply too much context to add to cover all new developments, and these are things that are not easy to solve by searching the internet.
10
u/nospacebar14 Jun 25 '26
Also, then they've got to keep the search call up to date! Making sure the model returns results from reputable sources instead of conspiracy.ru is a project that will never be finished.
3
u/Stubbby Jun 26 '26
Thats incorrect. The LLM is not retrained on the news, the LLM pulls news into context.
1
u/ex_nihilo Jun 26 '26
Modern LLMs can just search the internet or use an MCP server though, is what this guy was trying to say. I haven’t seen many people in this thread who know what they’re talking about. Are you all bots?
6
u/EebstertheGreat Jun 27 '26
The issue is less "the LLM responded incorrectly to the query I obviously should have just pasted into Google instead." It's more "I didn't ask the LLM to Google anything when asking about UK politics, and its entire response assumed Kier Starmer was the current PM." You can't expect your model to just Google everything it isn't sure about, because (1) that's far too expensive, (2) Google won't allow it, and (3) the model doesn't even know what it doesn't know. It's confident about everything all the time, even when it's wrong.
2
u/nospacebar14 Jun 27 '26
Ok, so I ask the model who the prime minister of the UK is, and it goes to the Internet.
Wikipedia has been vandalized and says "Daffy Duck."
Conspiracy.ru says that the UK is governed by a cabal of lizard people from space.
An NBC news article from three months ago says "current prime minister Keir Starmer."
How does the model know that all three of these answers are wrong? That's what I mean when I say training is never finished.
1
10
u/Sparkworker Jun 22 '26
We went through this with cloud storage, where every good deal and free tier was heavily curtailed eventually because 0.01% of the user base decided to store petabytes of data if they could. The market is stable now and we've found the price users are willing to pay to some extent.
6
u/Fairleee Jun 24 '26
There is a significant question however about the extent to which OpenAI’s financial reporting corresponds to standard accounting practices. The big question is whether they are fully reporting all inference costs under their cost of revenue figure. The really big question mark here is their reported marketing costs - they’ve reported $5.7bn in marketing expenditure which is absolutely nuts to anyone with any understanding of this. For reference, Coca Cola - one of the most heavily marketed brands in the world - spends around $4-5bn annually on marketing. Coca Cola’s marketing is everywhere; they’re currently sponsoring the World Cup for example. Where is OpenAI’s marketing? The assumption I have (and I have seen other industry analysts making the same claim, including Ed Zitron - I know he gets a lot of flak but given his background is in PR and marketing I think he has a good understanding of the topic here) is that OpenAI are most likely counting free trial users, or their token credits for businesses, within those marketing costs. If that’s the case then all they are doing is taking is taking figures out of the cost of revenue sum and sticking it in a different sum, but given that token production is fundamentally an operating cost, those inference costs should be reported as cost of revenue. To put it another way - if Coca Cola decided to give a billion dollars of Coke out to businesses for free, they still had to pay their typical operating costs to produce that product.
It’s also telling that leaked reporting shows that in the first 3 quarters of 2025, OpenAI had spent $8.76bn in inference. But their figures now are only showing $7.5bn - a lower figure after 4 full quarters. Either the reporting last year is inaccurate (which seems unlikely given several outlets corroborated the figures), or they’ve done some financial fudgywudgyness to move figures around.
Ultimately my perspective is that, if these companies were genuinely profitable on inference, they would be telling us that directly - this would go a massive way to restore investor confidence as it indicates the fundamentals of the business are solid, even if the business model still needs time to develop. Especially ahead of all these planned IPOs. It isn’t just OpenAI, either. Google and Microsoft don’t want to tell us what their AI production costs are, as they aren’t providing detailed breakdowns in their financial reporting. This suggests either all these companies either don’t know what the actual costs are (unlikely), or they do know and don’t want to tell us because it looks bad.
1
Jul 29 '26
[removed] — view removed comment
2
u/Fairleee Jul 29 '26
What specifically is your issue with his analysis? I agree he is loud and abrasive, but I don’t think it’s easy to dismiss his core thesis that the fundamentals of the financing don’t look good. The amount of capex being sunk into AI is very worrying considering the lack of returns, and he’s hardly the only analyst pointing out the issues we are seeing with off-book debt management through SPVs; circular financing arrangements etc. Further, there’s plenty of analysis outside of purely financial/economic looking at the problems from a logistical and supply chain perspective, as well as from engineering perspectives. If there’s a five to seven year lead time on producing the generators for the gas turbines that are going to power the AI data centres that need to be built, then how are we going to get gigawatts of compute turned on by 2030?
5
u/Legitimate_Concern_5 Jun 25 '26 edited Jun 26 '26
You’re taking at face value that their cost of revenue covers all their costs. It excludes for instance training which is required for inference. You can’t separate them. If you stop training, your model is stuck in time. See model drift. So you have to amortize the cost of training into the inference costs based on volume of real usage.
So excluding training is like saying your uber has a 90% profit margin if you exclude gasoline. Like I guess but you still have to fill up or your business won’t last long.
Then you’re assuming they aren’t moving inference costs into other buckets. OpenAI had an $5.8B sales and marketing spend in 2025. More than the Coca Cola Corporation, but the only ad I’ve seen is a billboard of a blurry man sitting next to a desk with the word Codex in the corner. What they’re doing is putting inference costs for free users, and discounts to enterprise users in that bucket.
Once you put that into cost of revenue it’s strictly unprofitable by any measure.
When you say “inference is profitable…” Profitable at what price point? How much does each token have to be sold for before an incremental profit can be turned? If everyone had to pay that, would they? What’s elasticity look like in a world where open weight models exist with comparable performance for 1/10th the price?
Yes you can set a price for tokens that yields a profit by definition as a function of volume. What is it?
3
u/Stubbby Jun 26 '26
- If your model needs 4 H200s, to run it for an hour costs you 28 cents
Last gen models need 1250+ GB which maps towards 9 - 10 H200s. (Current models like Fable5 likely need significantly more)
Ten H200s cost around $30/hour from most cloud providers.
Assuming the AI labs can cut down all the cloud providers profit margins and optimize things better than any data center, we are still talking ballpark $5 to $10.
19
u/Stuffssss Jun 22 '26
Good explanation on the difference between fixed upfront costs and marginal costs.
I would reckon a 200$ subscription does offer more than 200$ worth of AI usage through capital spend. Case in point being none of these AI companies have made a profit, and a path to profitability remains unclear in the near future.
Not including hardware costs into per token pricing I think ignore that AI inference is more expensive than just the electric costs of running the GPUs. We dont really know yet how often these GPUs will need to be replaced because their value is fixed to them being the most cutting edge. If a more advanced GPU comes out that outperforms the current leading edge GPU the market value of older generation models will drop. But older GPUs can still run models so despite market depriciation they still have utility.
This is all still excluding the training costs associated with R&D which is necessary to capture any market share where users move to whatever model has the highest performance. There is a huge upfront cost and the margins are just not there yet to recoup those by selling inference.
3
u/milkcarton232 Jun 22 '26
Wait what do you mean by inference is more expensive than just running the gpu?
11
u/Stuffssss Jun 22 '26 edited Jun 22 '26
OP is saying that inference pricing shouldn't include the cost of the GPU since GPUs depreciate with time not use. But GPUs can only do so much inference per second essentially so that should be amortized relative to the inference output of a GPU. You also cant ignore all the other overhead associated with running a datacenter like technicians, realestate, cooling etc.
So by that point I think that basically unlimited access to compute is worth more than 200$ a month in terms of infrastructure costs.
7
u/Uptons_BJs Jun 22 '26
I think it's worth differentiating fixed vs incremental cost here.
Whether you're building AI inference, a gym, a ski resort, a golf club, etc, any membership-based business needs to have sufficient capacity that peak time quality of service isn't too bad. If the quality of service at the time when everyone wants to use the service is shit, than people won't want to subscribe anymore (FWIW - this is why a lot of membership-based businesses will sell heavily discounted off peak memberships that limit you to off peak times). For subscription plans, AI companies control peak time capacity by throttling you and rate limiting you at peak times for subscription customers. OpenAI gives you a 50% discount for async usage.
Now consider the fixed costs:
- Renting racks from the data center
- Staff to maintain the equipment
- Depreciation on equipment over time
- etc
This is dependent on the amount of users you have at peak. Since more users at peak times = more equipment needed. You have to buy more hardware, rent more racks, hire more staff, etc. Each additional user you wish you service will increase this fixed cost, since you should assume the user will use the service at peak times, and thus, requires you to expand capacity.
Whereas off peak, since the capacity is already there, the only additional cost additional usage will incur is electricity.
When the studies show that you get some huge number of tokens for just $200, they get the token number by running a script 24/7 trying to use as many tokens as possible as long as they aren't limited by a rate limit. But when you do this, most of the token usage is off peak, where there is capacity to spare.
9
u/boyyouguysaredumb Jun 22 '26
Anthropic is on pace for its first profitable quarter this year, earning $559M on $10.9B in revenue
19
u/das_war_ein_Befehl Jun 22 '26
They are having a profitable quarter because their deal with SpaceX has free inference for like the first 1-3 months. Not coincidence that they suddenly have a profitable quarter
17
1
u/Bwint Jun 22 '26
Big, if true. Source?
4
u/IntoTheNightSky Jun 22 '26
Anthropic’s revenue is set to more than double to $10.9 billion in the second quarter, an explosive rate of growth that will help it turn an operating profit for the first time.
The company disclosed the figures to investors as part of an ongoing funding round that is likely to push its valuation above OpenAI’s. The projections, which were reviewed by The Wall Street Journal, provide a window into the meteoric rise of a startup that was once a laggard in the artificial-intelligence race, and defy the conventional wisdom that AI companies’ huge spending needs hamper near-term profitability.
Anthropic generated $4.8 billion in sales in the first quarter. Its quarterly revenue is now growing faster than Zoom did during the pandemic, and Google and Facebook in the run-up to their initial public offerings. It is set to turn an operating profit of $559 million in the June quarter.
14
u/CouperinLaGrande2 Jun 23 '26
This projection which Anthropic has put about doesn't conform to any recognised accounting standard. You can't treat it like any ordinary financial statement.
1
3
Jun 25 '26
[removed] — view removed comment
4
u/PythagorasWasntReal Jun 25 '26
To be fair they are using less power if they aren't in use, so power down is wrong but only slightly.
1
u/benjaminovich Jun 26 '26
That's not how I took it. I think op means that if capacity is significantly higher than demand, you will turn off gpus (exactly because they run 24/7)
3
u/kaeptnphlop Jun 29 '26
That's all nice, but without considering that OpenAI has apparently shoved all the free and subsidized token expenditures under their "Sales & Marketing" categroy to the tune of a couple billion dollars, exceeding the S&M budget of even the largest worldwide spenders on marketing like Coca Cola, you won't be able to make a fair assessment of whether AI inference is profitable.
Don't let them lie to you with their financial trickery.
2
u/Holbrad Jun 24 '26
Brilliant write up, there probably not making that much on consumer plans.
But your spot on, that they aren't losing thousands of dollars to heavy users!
Margins on inference are high!
1
u/gizram84 Jun 22 '26
I'm getting more than $8000 of value and I just use the free versions
10
u/MacroDemarco Jun 22 '26
Would you actually pay $8000 for it if they charged that much?
1
u/gizram84 Jun 22 '26
It has literally cut my working hours by 75%.
So yes, if that was the absolutely only option, I would gladly pay $8000 vs having to work triple the amount of hours I do right now.
However, that's not how competitive markets work, and there are at least a half dozen solid AI competitors in the space right now, which nearly guarantees that prices will remain very low.
7
u/MacroDemarco Jun 22 '26
which nearly guarantees that prices will remain very low.
That does not necessarily follow, as investors will eventually start demanding returns on capital
4
u/gizram84 Jun 22 '26
They can demand all they want.. If a competitor can remain profitable offering a cheaper price, they will. That's how competition works
10
u/MacroDemarco Jun 22 '26
Right, but none of them are currently profitable or on track to be profitable, and unlike software they don't actually have negligible marginal costs, in fact because of how electricity markets work they may have increasing marginal costs. Eventually investors expect to turn a profit and present a return on investment, so expect prices to rise eventually.
3
u/davidbasil Jun 23 '26
AI doesn't need to be profitable. It's 3-4 companies investing into each other and main driving force is FOMO
5
u/MacroDemarco Jun 23 '26
What are they fearing they're missing out on? (Tye answer is future profits)
-3
2
u/theunseenmiddle Jun 22 '26
But this coupon book says it has over $5,000 in value and it only costs $20!
2
3
u/Altruistic-Body-4852 Jun 25 '26
you lost me at the gym analogy. if you can't see the difference between operating a data center/server rack/computer vs a piece of gym equipment then I would assume your entire thesis is riddled with inaccuracies, false equivalencies, and flawed data.
1
u/LaGigs Jun 23 '26
What about post-training? I've heard this term casually thrown out recently so I wonder
1
u/frankster Jun 24 '26
You wouldn't expect 1 GPU bank to serve exactly 1 worker. They would be serving several perhaps
1
1
1
u/I_Ski_Freely Jun 28 '26
Really good analysis overall. I will add that OpenAI doesn't own any of the infrastructure, they just have commitments to pay for renting it at set prices regardless of their actual demand.
That insulates them from depreciation, but if they can't grow revenue fast enough, they will be like SpaceX, just sitting on idle rented gpus instead of owning them... Not sure which direction it plays out, but they're certainly better at making models than xai 😂
Also, open source Chinese models (glm-5.2 for example is competitive with Opus-4.7 and gpt-5.4) will undercut this at 5x cheaper API billing, and companies will rent cloud gpus to run them which would mean even greater savings for those who understand this, and there's plenty of cloud gpus to rent, with more coming online all the time even if it's hard for normal people to buy the hardware right now.
Some larger companies are already doing this, and with the token maxing issues (companies finding out the API billing from things like copilot is really expensive if you just let software developers use as many tokens as they want) this will probably increase, meaning their biggest profit margin revenue stream will dry up pretty fast..
The value prop for a company is basically they can either pay to send their data to an AI company, or they can use open source models, train on their own data and processes, and own it end to end for a tiny fraction of the cost of the API billing, and more reliable than the increasingly limited subscription plans.
1
u/Equal_Heat5947 Aug 11 '26
Anthropics LEAKED report did not include operating or training expenses. In other words, it's a marketing stunt.
1
u/Jacket_Sweaty 4d ago
Yeah its marketing, thre is some inefficiency with the architecture i didn't realise, been using aiyellowjacket.com to process the harder stuff whilst holding off going to max on claude which is way too expensive
1
u/Drnk_watcher Jun 23 '26
At a high level I think you're right. There is this insinuation that all tokens and their pricing are created or billed exactly equally when that simply isn't the case. Token caps on individualized upfront subscription accounts aren't cleanly compared 1:1 against pay as you go API usage. They serve different customers types and the cost of a token billed to end users certainly isn't reflective of the actual compute cost to run that token. With all that being before you get into volume based discounts things like commercial customers have access too.
This is all pretty routine for large service based organizations. Pricing is fuzzy and malleable as they weight upfront cash in hand against potential upside of high but variable volume consumers.
The available usage comparisons of different plans can certainly be helpful napkin math when deciding what to sign up for but you're right that it's a bit far of field for people to act like customer facing pricing is in anyway representative of real cost.
That said I think it's silly any of us pretend to know the real cost of running a token. Sure GPUs are most talked about but all these systems are also burning massive amounts of CPU compute cost as they handle load balancing, task orchestration, user state, etc. Then there is the move towards making these systems more memory intensive by using more and more RAM and solid state storage to improve conversation recall speed: with high read/write storage always having been particularly expensive.
Plus then different models consume different amounts of hardware overhead for different amounts of time, with core orchestrators/agents now actually dynamically intermixing models needed to complete tasks on the fly.
And then you allude to the eventual depreciation cost of all this hardware and needing to take hardware offline to upgrade it. These companies also need to take hardware out of service or have reserve hardware to render/bake/pick your adjective new models.
You're right that the cost of a token plan to plan is a moving target and that those costs to use a token aren't in anyway actually fixed to real world compute time or cost. So it's decoupled three ways. Simultaneously we don't really know what a raw unit price of a token is.
1
u/nysoxfan01 Jun 25 '26
If your argument was true we would not see large companies panicking over their AI costs exploding.
1
u/PythagorasWasntReal Jun 25 '26
These subscriptions are absolutely subsidized. You get far more value out of a $200 Claude subscription than $200 in Claude tokens.
Yesterday it cost me $120 to use about 500k tokens. I could've used the same amount of tokens on a Max plan and it would only cost me $200/mo.
And gyms don't offer different tiers based on how swole you are, you either pay the max tier or you don't get to go to the gym.
Here, when I was using Claude a little I stuck to the $20 plan and constantly capped it out. Now, I stick to the $100 plan and only cap it out toward the end of the week. There is no logical sense to pay for the $100 plan if you're usage fits within the $20 tier. It's like if a gym offered a $5/mo sub that you could "only" use three times a week. The vast majority of people are going to be perfectly happy with that tier, and now all your gym space is full of low paying customers.
1
u/TerminalJammer Jun 25 '26
It's true, you're not making money. OpenAI etc are losing at least that much money.
1
0
-8
-3
Jun 22 '26
[deleted]
16
u/Uptons_BJs Jun 22 '26
I've been here at r/badeconomics for years mate, all my bad takes are my own! No assistance needed.
76
u/NewUserND Jun 22 '26
What I got from your detailed explanation is inference costs could be profitable if upfront costs and depreciation are manageable. Seems like we need a follow on post from an insider on these two aspects because the amount of money spent on these are soo high, the articles might be right on principle but wrong on cause.