Got a C$71K Google Cloud bill overnight. Any advice?

[u/sheep1165]

I’m a cofounder of an early-stage startup. We recently had a Vertex AI job run overnight that ended up costing us around C$75K in 14 hours.

Our startup credits ran out during the job. The first charge we saw was around C$8K on our credit card, and we immediately shut everything down. But due to delayed billing data, the charges kept showing up hours and even days later. By the time everything propagated, we were left with a C$71K bill that we simply can’t afford.

We requested a one-time adjustment, but Google Cloud Billing Support rejected it. Our account rep also tried helping but got pushback.

Has anyone been through something similar and managed to get a refund or adjustment? Would really appreciate any advice on who to reach out to or how to escalate this further.

Edit: I agree that this was a legitimate bill. But if anyone understands the startup economics and the situation of affordability and knows if google can help in that regard, please lmk!!!

[u/Slow_Chemistry_8523]

We had a situation where a large query that looped over quite a bit of data ended up costing 5 million because of an intern. Our company is pretty large and Google still didn’t remediate. Sorry to be on the negative side but reality is that money isn’t coming back no matter how much support to try to find…

[u/paul_h → u/Slow_Chemistry_8523]

Oof - it’s a shame cloud utilization can’t be wholly part of intra-as-code and have code review (incl ai bots like code rabbit) catch noob mistakes. And the per-developer billing roll up gets limits under a designation that marks it as pre-testing

[u/_splug]

It’s not deserving of a credit/forgiveness. It was spent AI/compute. Not a Google issue.

[u/Glittering_Sail3262]

Why were your quotas setup in a manner that allowed a query that expensive 😳 

[u/Magikstm]

Pay your bill and fix your pipeline.

Why do you want to escalate this? You used the compute.

[u/Jamiemufu]

Sllllooooooop

[u/ReturnOfNogginboink]

The lack of empathy in this thread is depressing. Even if OP did something wrong here can't we at least be kind?

OP, I'm sorry you're in this situation. It sucks. But it does seem that it's a legitimate bill. Hopefully your account manager can find another escalation path.

[u/how_me_a_frog → u/ReturnOfNogginboink]

Empathy for what? This is no different than someone running a bitcoin farm in his house and then trying to negotiate away the massive electricity bill. You have to pay for services you use, this applies to everyone from individuals to multinationals. Startups aren't special.

[u/ReturnOfNogginboink → u/how_me_a_frog]

It's a lot easier to make a simple mistake that runs up a cloud bill than it is too accidentally run a Bitcoin farm.

[u/r3m4k3333 → u/ReturnOfNogginboink]

What does the op want to get from people here? Just pay your bills, that’s the reality and there’s no need to be over kind

[u/ReturnOfNogginboink → u/r3m4k3333]

"No need to be kind to people" isn't exactly a personal trademark I'd want to live by.

[u/zambono_2 → u/ReturnOfNogginboink]

It might be deeper, more like many on here might be software devs who are sick and tired of people thinking they can do their jobs properly due to a couple YouTube videos. I for one am extremely happy the cost of using AI is being passed down

[u/crackdepirate]

first , read doc on how secure an api key properly and billing alert and hard cap. stop vibecoding until this is not completed.

[u/sheep1165 → u/crackdepirate]

The API key wasn’t exposed. It was an internal pipeline. Billing data was so delayed that nearly C$49K had accumulated while the dashboard still showed C$0, so traditional billing alerts wouldn’t have helped. We definitely needed better application-level limits though.

[u/Jamiemufu → u/sheep1165]

And proper engineers

[u/r3m4k3333]

Pay it. You’ve used the resources, now it’s time to pay for them. What if one of your flor ts came to you and asked for a refund even if they used the service already?

[u/lavenderviking]

$10 budget alert will save you next time!

[u/martin_omander]

Here is a friendly reminder to everyone that you can set spend caps on Vertex AI, Gemini API, Cloud Run, and Cloud Run Functions. If your spend exceeds your cap, the service stops working within a few minutes. Here is a video where I show how to create a cap and answer common questions about it:

Prevent surprise cloud bills: enforce hard spending caps on Gemini API & Vertex AI

[u/impressive_clearing]

ouch, that's a gut punch right to the startup bank account

maybe try tweeting at their cloud ceo publicly, companies tend to move faster when the mess is visible

[u/NightKnight529 → u/impressive_clearing]

Not sure how it’s a mess. Seems like a legitimate bill they want a break on?

[u/massix93 → u/NightKnight529]

Yeah but if it's a clear mistake they could do something

[u/r3m4k3333 → u/massix93]

Do what? And why?

[u/massix93 → u/r3m4k3333]

Investigate and if it’s a clear mistake waive the bill or find an agreement. What do they gain from make a startup fail?

[u/tankerkiller125real → u/massix93]

When it comes to the mega corporations? Often less competition.