r/analytics • u/datawazo • Apr 01 '26
Discussion Client pulling the plug, moving it all to Claude
I've run a small analytics agency since 2017. Primarily in the database layer (organizing, cleaning prepping data) and then shipping it to PBI and Tableau for dashboards.
Met with one of my favorite clients today for our weekly and he said he doesn't want to talk about PowerBI - he wanted to show me everything he's built himself in Claude.
What followed was an hour demo of - more or less - how he was planning on replacing us with this Claude Cowork pipeline.
Luckily they are good people, and they like us, the conversation was along the lines of
"How can you support us transitioning in this direction".
It just have easily could have been "bye felicia".
But man - what a wakeup call. I spent the next hour on the treadmill, crafting my advice.
Their plan was to have Claude sit directly on top of an ETL tool (won't name names, there are many options for this). They could ask it any question they wanted, AI would go to the tool, pull in the right data and answer the question. They'd even set it up to write to specific google sheets too. It was impressive.
But risky. Here were my bullets back.
Traceability - when (not if) something goes wrong, how can you find it, and how easy is it to fix. It's a black box you don't have access to. Troubleshooting it is near impossible.
Consistency - factoring just human nature aside, asking the exact same question on different days could lead to different results. Based on algorithm changes (infrequent but they happen) or based on existing/new context in a chat. It's really hard to guarantee consistency with AI. Try it yourself ask a question today, interact with the chat and ask the same question tomorrow, is the output identical?
KPI definitions - you ask it for conversions from google ads. Does it know what a conversion is? Does it know how to calculate net sales? And tying to above, will it be the same twice?
A few other things too like privacy and token usage. My suggestion was to do the ETL into BigQuery, then create a curated dbt layer with all the logic, proper naming, agreed kpi definitions, and condensed data in there. And then have Claude sit on top of that instead.
Idk, we'll see where it goes. Eye opening day where, basically what I knew as always coming, came.
118
u/captain_vee Apr 01 '26
In my experience Claude is great but only if you actually know how to do what you’re asking it to do.
It often makes coding mistakes or math mistakes that can drastically change results. The only reason I catch these mistakes is because I make Claude show its work and I always double check it.
I call Claude my junior analyst. It can do simpler stuff but it still needs review because it can make some pretty gnarly mistakes
9
u/analytix_guru Apr 02 '26 edited Apr 02 '26
LLMs were trained on a corpus of information. While there are numerous examples of simple projects and logic to be learned from, content becomes sparse quickly with intermediate and advanced topics. The you layer in industry differences, tech stack differences, version differences, and the stuff you don't even know about because it is behind a corporate walled garden of sorts. This is untouched area for these Models and so they won't have the reference to properly asses and action on specific prompts that cover this information
Was listening to one of Joe Reis podcasts and he had Claude or ChatGPT vibe code something for him, and at the moment I can't remember if it worked, yet the point was that it was tens of thousands of lines of code and it would have been burdensome for him to review the entire project generated by the LLM.
AI generated slop, even if it works.
2
u/Odd-String29 Apr 02 '26
I had Gemini trying to use GENERATE_UUID() as a key in a CTE....so every time it called the CTE it generated a different UUID, which messed up a join. No clue why it just didn't make a surrogate key based on some columns.
3
u/captain_vee Apr 02 '26
lol - yeah so many times it has suggested something so incredibly complex and I’ll ask it why can’t we just do it like this [insert much simpler way], and then of course it says “oh yeah that’s way better”
2
u/w0rdyeti Apr 23 '26
It is the same thing over in Design; the AI produces things that *look* like they work, because in the limited demos & prototypes they do.
And then you push to real-world, start putting stress on them, and they shatter.
And then you try to fix the broken stuff, and as the OP said above, it takes an unholy amount of time to trace where TF the problem is coming from ... and fixing that one thing breaks 4 more.
Tell clients who are junking humans in favor of AI for mission-critical applications that they are basically adopting & feeding a pet Hydra. Cut off one evil head ... two more grow back.
2
u/occasionallylo Apr 03 '26
Yes, I use copilot and I understand Claude might be better but like a junior analyst copilot does make mistakes and misinterpretations that you have to know to catch. I also use it as a SME and it can help me bridge the gap between business or clinical SMEs. We’re not close to the point where they’re replaceable. We just get way more done.
129
u/Yourdataisunclean Apr 01 '26
Reminds me of this: https://www.reddit.com/r/analytics/comments/1r4dsq2/we_just_found_out_our_ai_has_been_making_up/?depth=4
When they realize the lack of traceability and nondeterministic outputs they may realize this is a bad idea. There's also the fact that LLMs can be pushed into bullshitting easily if you just keep pressing.
There is still a lot of value for boring methods that do things the same each time. But that's not where the hype is at right now.
33
u/datawazo Apr 01 '26
Yeah and I worry that AI let's you get comfortable with it in early days letting you do easy things and validating and trusting it. Then you let it cook unmonitored before you notice too late it's gone on a journey of it's own.
Although that's all anecdotal and hard to prove
30
u/Yourdataisunclean Apr 01 '26
Everyone without a background in AI, seems to fall into the trap of assuming these things have actual reasoning abilities until they've used them awhile and see enough truly terrible outputs to start questioning things. If you can't convince them initially by giving a high level overview of how LLM's work and can't be reasoning with the current architecture. Then sometimes you have to let them cook awhile first.
4
u/hockey3331 Apr 02 '26
Im afraid because I see people in technical roles and people in position of authority in technical roles discussing feeding unstructured and inconsistent data to LLMs to replace deterministic transforms and (more importantly) a broken data collection process.
Like guys, its not ricket science. Fix the collection process so we get standard data and can feed it without a black box around it.
1
u/razealghoul Apr 01 '26
I am curious what experience you have with using coding agents with these use cases?
I might be someone falling into this trap but I have started using coding agents over the last couple of weeks and their outputs are very good. I have them connected via mcp connections to various databases and leverage .md files to store logic. Have you run Into the issues you described with these kinds of agents?
12
u/ElasticSpeakers Apr 01 '26
All that you really need to mention is the non-determinism. There's nothing there to prove - it's exactly how these models are designed in the first place!
The only way it works is having a human in the loop that can identify an iteration that's going down a bad path and add things to course correct after the fact.
5
u/analytix_guru Apr 02 '26
This.
Some of the best examples of beat in class AI projects are where an expert in the field partners with an AI to produce something, and is able to review all of the LLM Agent's work to ensure quality and accuracy for the project.
As in, the expert could do it by themselves, but they have the LLM to help with speed of delivery.
6
u/analytix_guru Apr 02 '26
People have shared anecdotal examples and there have been some news events (e.g. Deloitte had to reimburse almost $300k USD for AI generated reporting for a client that was wrong). Amazon AWS cost explorer went down in December 2025 because AI deleted and recreated a production environment, although the company blamed employees due to permission issues. However, I doubt a human employee with access would have done the same thing.
Also have been YouTubers and Influencers that have cut out staff at their companies because we are going all in GenAI and 6-12 months later people find job postings for those companies trying to hire for the positions they let go, because it wasn't working as expected.
So yes, anecdotal, but some of these stories are verifyable.
2
u/AdSpiritual3205 Apr 02 '26
This is the new trend. We are doing this now for a lot of major F50 companies.
But there's a difference. You cannot just stick a LLM on your data and expect it to work. It won't.
You generally need a multi-agent model with a lot of specific training and a really good semantic layer so that the LLM knows the right places to get the data from, but those places (agents and workers) know how to answer questions about the data correctly. So it's not the LLM giving the answers on its own.
1
Apr 01 '26
[removed] — view removed comment
7
u/datawazo Apr 01 '26
That's something else I brought up. One of their reasons is costs and it's like sure, but look at how much money anthropic is netting right now and tell me how sustainable that is.
14
u/IlliterateJedi Apr 01 '26
For what it's worth, I still think that post was made up considering OP declined to answer any follow up questions.
8
20
u/analytix_guru Apr 01 '26
I'm not saying your client can't eventually get there, but that is a huge ask for them to get to that state so that they can rely on Claude for answers as a BI agent. This is the foundational issue at almost every company currently as well as the semantic layer for metrics and definitions.
To think in a 15-year time frame it has accelerated through Analytics to machine learning and data science to now large language models trying to provide business intelligence, yet all of the foundational competencies required for any of these disciplines isn't 100% addressed at any company that I am aware of.
Worse off, you get the one person in a company (I have seen lately where it's been a hands on director or leader), and they get a few curated case studies built on a CSV extract with definitions they defined based on their department metrics, and it happens to work. Then the assumption is that it will work at scale, not realizing they have no idea what enterprise data is like behind the scenes, as they are given extracts off of dashboard workbooks or data dumps from the data team.
I am in consulting and have had a lot of trouble getting clients for similar reasons. Pivoting to a project based model where you advise on the path to LLM BI Agent might be the way to go.
8
u/datawazo Apr 01 '26
Internally that's my take away. We need to start getting ahead of these conversations instead of letting the clients get there on their own and being shocked by it. When we need to start doing that six months ago.
Like I said we're lucky it happened with this client and we're lucky we have good growth relationships with many of our clients. Because there's a real world situation where this is an invoice we don't sent next month because they're lifting and shifting without us.
4
u/Kindly_Ad995 Apr 01 '26
Your third paragraph just annoyed me so much because I know exactly who it is at my company.
5
u/analytix_guru Apr 01 '26
What's worse is I had to fight to convince business partners that my analysis was correct when the results were not favorable. Just imagine an LLM providing an analysis that had unfavorable results and then you don't even have a person to talk to and determine whether the analysis is correct or there was a mistake in the process.
Or even worse, the user tells the LLM that they aren't sure the results are correct because they are bad, and then the LLM just hallucinates or fakes results to make it look good :(
2
u/thedoctorisout25 Apr 02 '26
Not kidding you describe what I’m doing perfectly. I work for a F100 company and I’m working on a project directly from our C-suite of exactly this. My team working with our engineering team - collecting, cleaning and putting together perfect data that’s taking days and days from my team and the business / data teams who own it.. ‘standardizing’ data that doesn’t belong together.. all so they can ask an agent super high level questions. The expectation is we scale it out to our other enterprise dashboards and reports.
18
u/CaptCurmudgeon Apr 01 '26
Maybe my data stack isn't mature enough to compare, but I feel like I have pretty good traceability with Claude. The key is to have separate agents doing different auditable small tasks. Then when a pipeline breaks or a kpi or measure is questioned, you can see where the failure occurred and try to bandage against it for next time. My 2 cents.
12
u/GamingTitBit Apr 01 '26
Or what we do is have a deterministic gate and traditional ML with confidence in-between each agent. If you have agents feeding agents you get silent errors compounding as they propagate forward
2
u/razealghoul Apr 01 '26
I agree, I haven't had too many issues with Claude. I feel a lot of the concerns around traceability are from people who aren't using coding agents or who used the free models from a year ago. The latest gen of paid agents are leaps and bounds above from where they used to be.
5
u/seguleh25 Apr 02 '26
The models are great, but you still have to know what you are doing. For a non technical user, it's too easy to give ambiguous instructions that get misinterpreted by the bot.
1
u/razealghoul Apr 02 '26
Yeah but the same can be said about a human analyst too.
2
u/seguleh25 Apr 02 '26
It seems obvious to me that the 2 are not similar at all. A big part of my job as a human analyst is to make sense of vague requirements.
1
u/razealghoul Apr 02 '26
I think you haven't used the latest models. That gap you are describing is a lot smaller than it was even a couple months ago. I am not saying you can install an LLM on top of your data warehouse and you are good to go. You still need the proper semantic model in place to guide the LLMs but once they are in place you can have non technical users describe in plain English what they want and the llm should be smart enough to fetch the data. Again there are limitations but the performance I have seen this year has been phenomenal.
2
u/seguleh25 Apr 02 '26
As a matter of fact I do use the latest models. I'm not talking about a gap. I'm talking about a fundamental difference. Half the time I have to push back to my end users and explain to them the report they are asking for won't mean what they want. Then we figure out together what they are actually looking for. And in many cases putting it in plain English would be more work than writing code, because English is not as precise.
Not to say there is no place for LLMs, they can be incredibly valuable and I use them everyday at work and for personal projects.
1
u/razealghoul Apr 02 '26
Then you misunderstand the point I am making. This thread is talking about someone who had his consultation contract terminated and replaced by AI and I am saying I can see that future.
For example in my company I used to get a ton of ad hoc reporting requests asking for my details on what was displayed on my dashboards.i tend to do a lot of pipeline and marketing questions. Stuff like:
- I can see the forecast for q2 for XYZ products has gone down by 5%. What is the root cause?
- What is causing our mql conversions to slow down this quarter?
These are the types of nuance you can't get out of a dashboard. So what I did last month was I set up an AI agent and connected it with mcp connections to our data warehouse, as well a couple of others including slack. Now I have those stakeholders send those same questions to my bot via slack and it's able to answer them directly.
I have it tackle for complex tasks too. For example it was able to scrap the data in our jira ticketing system and build out full product roadmaps and full on presentations in the performance of our marketing last month. These outputs even give very detailed explanations on what is working what is not and even provides suggestions on what to dig deeper on. For example I had a stakeholder digging in to some buying patterns of some of our high value customer and the agent was smart enough to flag customer at risk based on their account activity.
2
u/seguleh25 Apr 02 '26
My point was that works because you know what you are doing to begin with. You could do the task without the LLM, so you can properly evaluate if it's working in the desired way.
1
u/razealghoul Apr 02 '26
Yeah but because I know what I am doing I am able to configure agents to talk to my stakeholders directly bypassing me.
→ More replies (0)
9
u/Jagsfan82 Apr 02 '26
If you cant build it yourself then you cant manage ir deploy an AI system reliably. Its not possible
You have to be able to have a clue what its doing and if the outputs are correct
8
u/freedumz Apr 01 '26
This is probably the reason why I'm thinking to accept an offer in the public sector as analytic engineer
7
u/Creative-Tea-9157 Apr 02 '26 edited Apr 02 '26
It’s incredible how business teams who can barely give developers requirements to build dashboards are now so confident in their own abilities to build a reliable AI agent. All you can do is let them. I recognize soft skills but aside from the tangible product / service a company produces, the next tangible thing is analytics. I have so many leadership teams around me leveraging AI and showing off their half baked work products just because it’s finally a visible way to show their value. And as long as they don’t have to pay you, even better. So don’t take it personally. Don’t explain the risk, they do not care. That’ll be the responsibility of the next leader to clean up. Support them through the migration and charge them 4x when they come back to you for help.
7
u/vincenzodelavegas Apr 01 '26
Could they be making mistakes understanding the data? I work in healthcare data analytics and even with the ability to generate a dashboard, not sure they’d dare analysing it themselves
5
u/StamGoat Apr 01 '26
I'm skeptical of this approach and also concerned: it's like switching to AI to manage bank accounts balances. This is what BI should provide: 100% trusted, auditable, data that measures a certain company or segment performance at a level of aggregation that represents the strategy. It's like a bank account: you want to trust the bank with the right balance. Every day you access the refreshed dashboard and you have your trusted data. Every day you access your bank account and your money is there, the right amount.
For sure AI is starting to be helpful in speeding up some parts of code generation, some artifacts development, brain storm on data exploration. But I would never move what has to be deterministic to what is, at the end of the day, a stochastic model.
Probably we are in the same phase as the "data democratization" one of few years ago. On a much bigger and fuzzy scale.
2
u/analytix_guru Apr 02 '26
Been saying data democratization for over a decade now, and it is still all the same fundamental problems over that decade that AI won't fix.
7
u/ithinkiboughtadingo Apr 02 '26
AI is only as good as the metadata it has access to, and most organizations' metadata is terrible to non-existent. I'd be talking to them about how to build up a deep repository of high-quality metadata and things like production readiness labels, column and row-wise access controls so the bots don't do a data breach, things of that nature.
4
u/heimmann Apr 02 '26
AND on top, owning and updating it when processes change which, naturally, changes the data flowing in. What we defined as a “lead” the last 5 years suddenly changes, when then new sales director comes in and start seeing existing customers as leads too (an example). If the metadata and process context does not update along with the process, AI, an everyone else, is gonna have a bad time.
6
u/Training_Advantage21 Apr 02 '26
Yes, if you think AI can replace ETL and data transformations in dbt, then you are going to create beautiful reports that contain wrong and meaningless data.
Using AI to generate SQL is one thing. Using AI as a replacement of the data team, good luck.
2
u/Creative-Tea-9157 Apr 02 '26
Indeed. I have clients with transactional data telling me they don’t need a database. 😂
4
u/Jeepsalesg Apr 02 '26
In my experience Claude can be extremely efficient and correct when given the right Kontext. We for example have created a knowledge layer which contains all information what we are trying to archive and what tools etc exits and Claude can navigate this easily.
One thing we realized tho is that you get way better results if the tool you are asking via AI already does the compute so that the AI just has to interprete and not calculate and to reduce tokens needed.
4
u/GamingTitBit Apr 02 '26
For everyone worried about Gen AI. Remember the ROI hasn't been great for Gen AI. Two things I've observed as a Data Scientist (I know it's different but it applies to analytics). Your value was never in the dashboards or visualisations you created, it's around your understanding of the Data and how to translate that into value. AI is getting pretty good at coding, but it's still pretty bad at creating value BECAUSE ITS DESIGNED TO PLEASE. For instance as an experiment I have GPT 5.4 newest version my skeleton code, I then told it to do some research and make it better, gave it an evaluation set, and how to measure success. What did it do? Hardcoded results to the input questions. I told it not to hard code results. So it just hard coded it a different way. It was desperate to show improvement regardless of how.
Your value as an Analyst is understanding the real world impact of your data. Translating that to the business, predicting impacts, and communicating that to your stakeholders in a way that is relevant to them. GenAI will try to do all those things but due to constantly wanting to please and succeed, and due to its inability to know anything outside the world of data you present to it, it will fail.
AI is coming for our jobs. But it's not quite there yet. Our value was never in the code or the dashboards. Just wanna spread some hope.
3
3
u/decrementsf Apr 02 '26
In a ridiculous billable hours consulting firm they had a filter where nobody interacts with the client without the analyst background. The idea being familiarity and comfort with the detailed tedious data below the analysis, in order to speak to the complexities accurately. To avoid misinterpretations of the analysis. Intuitively I think you're tapping into that risk. There needs to be a complete chain of somebody who has reviewed and signed off on the full chain that there's not a clear issue somewhere in the chain. That is lost with AI in the middle. Is like having a communications major without data analytics basics presenting results to the client without the background necessary to validate thus unable to be authoritative, sets the stage for costly strategic errors.
Second is recognition that AI tools are subsidized, currently. Claude and other tools are burning funds to create dependency. Create the audience that relies on their tools for analysis. And then once the use case and dependency has been creates they must raise costs to become profitable. A workflow created now may make sense at the subsidized rate. And then when the price of tokens goes up rug pulls the now dependent organization. I do not see budgeting for expected dramatic rise in cost of compute in the future. This is the cheap party stage where everyone gets drunk on the near-free teaser bump to get you hooked on the drug. Party. Have your fun. Don't burn all your vendor bridges.
2
u/Chaluliss Apr 01 '26
In my org the biggest challenge always has been getting clean, validated data together.
I imagine with well curated data and some ontological validation layers you could get analytics consumers to directly access the analytics they want via LLMs, but that is still a limited scope of automation compared to everything I do as an analyst.
I build new data sets based on new topics of interest somewhat regularly. Often this comes from the business using new tools and technologies across its marketing, sales, ops stacks. Data engineers are helpful for sure, but when it comes to getting a dataset that can serve diverse analytics needs, engineers dont have the business context/knowledge. This is where analysts like myself come into play with building data sets you can actually report on top of.
Also, some of the reporting I build is niche, and complex, using custom indexes which simplify monitoring some specific space of interest. I dont get to do this as much as I would like either because I am constantly on call for simpler ad hoc analytics. If an LLM layer serves some of the simpler stuff, neat, but I would be surprised if you could expect it to generate novel, niche solutions like a skilled analyst.
All this is just to point out gaps in the function of LLMs that I see compared to my current responsibility set. I still think analysts have a place, maybe just less places, who knows though.
2
u/GeeMarcos Apr 02 '26
Bummer, my buddy just went back to school for data analytics. Hopefully it wont be a waste for him.
3
u/indiankidhs Apr 01 '26
How receptive was the client to your concerns and did they actually bring up ways they planned on addressing them or just kinda shrug them off?
8
u/datawazo Apr 01 '26 edited Apr 02 '26
On the call they were open to listening. I said it's all new, it works now let's keep an eye on it and make sure it keeps delivering as expected. I mentioned bigquery on the phone and they were receptive, especially at the opportunity of a future lift and shift.
But it wasnt till the afternoon in long email that i spelled out the more concrete plan above and I havent heard back yet.
edit: they've accepted my proposal.
1
u/alurkerhere Apr 01 '26
Your suggestion is the correct one and it's scalable. I still wouldn't trust Claude Cowork to manipulate a BI tool unless the average use case can consistently prove me wrong.
Claude is pretty good at common definitions but if you have logic and definitions that are client-specific, those would need to be part of the context.
Gen AI needs the right scaffolding to really shine, and in the same way of ML, don't apply it to everything.
1
u/contrivedgiraffe Apr 01 '26
Don’t forget enshittification. What’s their plan for when Anthropic doubles their price twice in a year? Or when they put this functionality in a new tier at a higher price point?
0
u/Cold-Dark4148 Apr 01 '26
Huh it would still be cheaper than highering a data analyst sitting on 100k salary
1
u/timerfree Apr 01 '26
i think it’s more so that tableau and powerBI gets replaced by claude rather than you. analysis never ends and claude does that ok but not unless you have a data person managing it. data drifts due to reasons that aren’t necessarily documented.
dashboarding is usually just translating business metrics to some UI which claude can do pretty well given the right prompts and data integrity.
1
u/Think-Trouble623 Apr 01 '26
Why not show them how to connect Claude to PowerBI and enable them to query data from PowerBI in natural language with Claude? Then you manage the pipelines and semantic model. Turn the MCP server off for editing so it’s read only.
Some honest feedback though, your product offering should already be including AI. Even if it isn’t ready for the prime time without heavily curated datasets and instructions, you cannot ignore it. Any executive wants to use AI so you need to tailor your product and skillset to it.
1
u/EverythingDataDude Apr 01 '26
I have some more points to add here:
You are right. AI with the right data can do a lot of good things including creating "dashboards" and generating valuable insights.
What's the LLM API price the company willing to pay to use the solution and deploy? If they want to chat with the data that consumes extra tokens which means extra costs.
Are dashboards dead? As an enterprise, you still need to report on the right numbers that the entire org can get behind. This in my opinion will still be the need for a dashboard. There are also some drill down functionalities that Powerbi has that react and angular applications can't do that well. There is no cost for drill down in Powerbi. There is a cost to chat with your AI to drill down.
You can advise on an architecture of data and product (including AI solutions) + find the balance of Powerbi and AI where they will save costs instead of running it all through AI and using your business.
On a side note ... Microsoft needs to up its AI insight on semantic model. They literally have the competitive advantage on semantic models but haven't given two shits about coming into the new era. Just my two cents!
3
u/datawazo Apr 02 '26
These guys definitely aren't getting the full value of dashboards. I've had a hard time transitioning them to static reports done in PBI to true exploratory dashboards - which is definitely part of the driver.
They also take issue with getting licenses for all their people - so yeah maybe some bullets for me on getting better engagement with the data products to try and prevent this.
microsoft and tableau need to up their AI game. They are going to bleed out customers to this if they don't.
1
u/analytix_guru Apr 02 '26
I feel like this always comes back to the discussion of why are we building a dashboard in the first place and the good old who cares question with what gets put in it.
Unless it is a monitoring dashboard for audit or risk teams, if it doesn't result in a decision then don't make a dashboard.
1
1
u/Jeepsalesg Apr 02 '26
In my experience Claude can be extremely efficient and correct when given the right Kontext. We for example have created a knowledge layer which contains all information what we are trying to archive and what tools etc exits and Claude can navigate this easily.
One thing we realized tho is that you get way better results if the tool you are asking via AI already does the compute so that the AI just has to interprete and not calculate and to reduce tokens needed.
1
u/gunners_1886 Apr 02 '26
Claude is good at a lot of things. Creating and maintaining consistent, governed metrics from raw API source tables over time is not one of them. It's also expensive.
Your client will be back the first time they make a bad decision on raw data LLM output.
1
1
u/PrisonersDiploma Apr 02 '26
Even if you remove AI from the equation, the internet in general has provided enough firepower for anyone who is interested to analyze data or produce a dashboard. The hype around AI has simply accelerated this.
Anyone can take their datasets and dump them into an AI model and ask for some sort of analysis. The real value is understanding the context behind the data. Questions like: What dataset is the best source for what I am trying to evaluate? What does the business processes that produces this data look like? How are these values coded and where are they sourced from?
For smaller organizations the answers to these questions are simple, but the analysis will also be simple and AI will not add as much value. For larger organizations this becomes much more complex, and that is where things start to get messy due to a lack of organization and understanding of their data infrastructure. Without this understanding, any analysis is useless because it is difficult to reliably re-tell a story you don't already know.
1
u/Ok_Procedure199 Apr 02 '26
Can it be run side-by-side with your way for a time where they can experience if they are able to prompt claude in a way that shows how the consistency is over time?
1
u/SavageLittleArms Apr 02 '26
that's wild, moving everything to claude feels like a huge shift. honestly i feel the pain of having to switch up workflows because of budget or "efficiency" pivots. i usually try to keep my stack pretty lean anyway i use buffer for scheduling, mailchimp for the emails, and runable for all the visual stuff like carousels and videos. it makes those corporate pivots a bit easier when you have a set of core tools you actually like using. hope the claude transition is smoother than it sounds lol.
1
u/ToroldoBaggins Apr 02 '26
On the traceability part: would it be useful to work in an environment like Databricks delta tables where you have logs for everything? Then ask the AI to look directly at those tables.
2
1
u/Fit_Doubt_9826 Apr 02 '26
I use Claude for work everyday, it’s only great at getting you an answer quickly by generating you some code, but without checking the code logic and if the answer is reproducible across different parameters etc, it shouldn’t be trusted, definitely highlight that point to your client. tl;dr we should only get non deterministic AI to produce code that is deterministic that we can trust.
1
u/EnvironmentalYear898 Apr 02 '26
I agree you would need to build out the semantic layer and specify all related relationships within your data platform — and that in of itself is a major project. Also in addition, having to maintain definitions on metrics, joins, field definitions, etc to even support AI usage on top of your data.
If not they’re essentially going to have AI either make up stuff or think it’s pulling proper data just to realize it’s also wrong.
1
u/Electronic-Cat185 Apr 03 '26
this feels less like replacement and more like a shift in where the value sits, the hard part isnt querying data anymore its making sure the data model and definitions are actually trustworthy
1
u/nitrousconsumed Apr 03 '26
Judging by your points it sounds like you're not really up to date on what AI systems can currently do, because any good governance layer takes care of those points.
Source: I design and implement AI systems at a Big 4 for enterprise clients.
1
u/heimmann Apr 03 '26
The “issue” is getting that governance layer in place, not the tech side. What’s your approach for the governance setup and getting enough process and contextual kowledge into the ai?
1
u/nitrousconsumed Apr 03 '26
I dont know if I'm understanding your comment, but governance isn't something you bolt-on as an afterthought. It's a foundational layer that affects all downstream architecture. In OP's case, KPI Definitions are just code that can be version controlled. Traceability is a logging problem. Consistency comes from scoping. Process and contextual knowledge is business logic, which is also documentable.
1
u/heimmann Apr 03 '26
I know but tell that to a business function who has not been doing governance for +15 years already. It’s too late, for them it will be an afterthought like it or not.
1
u/nitrousconsumed Apr 04 '26
but tell that to a business function who has not been doing governance for +15 years already.
I do, regularly, that's a large part of my job. They all have to start somewhere though.
1
u/heimmann Apr 04 '26
I think AI can be used as the lever to get them into this.
Before: “please document your processes so we don’t lose corporate knowledge” - err no thanks don’t have time.
Today: “ please document your processes so we can use AI”. OMG yes!!
1
u/chicagokp8 Apr 03 '26
My question always is what’s the data security posture of this type of setup? Once you open the data to AI, what does AI keep? Is the data proprietary, sensitive or customer owned? Are privacy policies impacted?
Most company leaders say, hey, look what AI can do and don’t look beyond potential data security impacts.
1
u/Able-Art-3042 Apr 04 '26
i mean the workflow in power bi sucks and was already dated before llms. i recebtly built something with lightdash plus dbt and semantic layer woth the help of vibe coding. I focus on business logic and requirements. proper testing plus well defined semabtic layer abd I execute at lightspeed.
1
u/thisisnice96 Apr 04 '26
As an analyst, I’ve been burying my head in the sand with all of this.
God what a wake up call.
1
u/writeafilthysong Apr 05 '26
Giving all your analytics to Claude is like hiring an English professor to be your Accountant.
But tbh if you put in the guardrails and have good governance and observability it's possible.
From your client's perspective your work is probably just as much of a black box as what Claude provides.
1
1
u/numbworks Apr 05 '26
Are they aware that they are giving all their confidential analytical data to a third-party company that will use them for answering other customers prompts once they re-train their models?
1
u/IntelligentAsk6875 Apr 05 '26
Just do it. They will crawl back to you in a couple of months. You cannot trust AI 100%, it's an assistant, catalyst, not a stand alone human replacement
1
u/hardcherry- Apr 06 '26
Dude - pull everything into a db read only. Install a discrete multi-user chat interface for every user who wants the data. Claude Cowork training & deployment for reports/charts/analysis PowerPoint whatever.
Internal db. Claude cli agent will help you do all of this.
1
u/SVG_47 Apr 06 '26
Just based on the way you write, you come across as thoughtful, caring, insightful, and capable of figuring things out in unexpected ways. That’s ultimately what people pay you for even if it looks like it’s for the unglamorous data cleaning and dashboard building.
Yes AI will have an effect here, somewhat, but your real value is in your judgment and ability to connect with people.
1
1
u/Important_Flight2198 Apr 06 '26
Pure analytics is cooked guys. You still need to know what you’re doing, but a single analyst can gate 100x more done now.
1
Apr 06 '26
the llm model training has been done on the past data and the testing phase does not generally have 100% accuracy. So when you are deciding anything for your company or strategy to build or for that matter just the revenue growth you are not sure that if you have got the right solution that you the AI has given.(reliability concerns are very high)
But on the other hand if the same work is done by a human , he / She has done the research and will feed only the 98th or may be 100% correct solution by getting the same discussed and tested.
1
u/WearFabulous3042 Apr 06 '26
Your client isn’t trying to replace you, they’re chasing open AI budget and cutting “legacy” costs (like your ETL). The problem? They’re about to build their AI pipeline on quicksand: unpredictable token costs, no vendor control, and Anthropic will tell the cost, not them,
Your move:
Position yourself as the AI Ops partner who helps them:
- Build use-cases on top of your ETL (so their data is solid, auditable, and consistent).
- Create a predictable token-cost environment (so their AI budget isn’t a ticking time bomb).
- Keep the AI vendor risk-free (so they can switch AI vendors when they're getting too expensive, uptimes are getting bad, or their new models are broken, without rebuilding everything).
This isn’t just “data cleaning”, it’s owning the foundation that makes their AI investments safe and scalable.
1
u/Feisty-Donut-5546 Apr 07 '26
The semantic layer is the real game changer. Claude is only as consistent as the definitions you give it; which is why a properly modelled dbt layer beats letting it guess what conversion means every time.
1
u/adaptaiseo Apr 13 '26
AI isn’t replacing agencies, it’s replacing low leverage work inside agencies.
1
u/holisticsbi Apr 15 '26
One thing I don't see discussed, why did the client go rogue in the first place? Usually it's because the analyst/agency/BI interface is the bottleneck for every analytics question. Claude just made it easier to try with natural language questions (doesn't mean the answers are correct, but prolly giving a false impression of them being useful)
Good news though, a lot of people in this thread are pointing to semantic layers and for good reason. Data team defines metrics, relationships, and business logic once. Then it doesn't matter whether a human queries it through a dashboard or an AI queries it through a chat interface, the answers are consistent because the definitions are locked.
I work at Holistics where we've built this pattern, so I'm biased toward that approach. But the principle holds regardless of tool.
1
u/medmentall Apr 16 '26
Honestly, that’s not them replacing you, it’s them skipping straight to the part where your work matters most. What they built looks slick, but it’s sitting on top of all the messy stuff you’ve been quietly handling definitions, consistency, and trust and that doesn’t go away just because Claude is in the loop. If anything, they’re about to realize pretty fast they still need you, just less for dashboards and more for making sure the answers don’t fall apart.
1
u/satiated_maven Apr 19 '26 edited Apr 19 '26
You raised great points to client. Here’s my alternative take (hate me now): it depends somewhat on clients business and what they are trying to model in the real world. You can make a super complicated model to predict, using lots of interesting statistical theory and approaches but in the end if the model output results are about the same as a very simple model, the simpler model might be the best approach because (A) it’s easier to run, (B) easier to explain results to a nontechnical audience and (C) easier to defend to upper management and (D) more straight forward for pragmatic decision making. So imho a client could be better off running something “out of the box” vendor specified, but it really depends on what they are doing. I usually see the value of doing this in smaller (segmented) parts of a model, but not an entire model itself. In the end, if you’re in the space of modeling the real world or making a prediction, most models have some kind of overlay that tweaks results & that can’t really be done by a machine, because it requires “expert judgement” (a doctor, a banker, some kind of industry expert that can make a tweak along the lines of “an art form” and then everyone is left to debate if they’re a genius or dimmed egomaniac). The irony for me is whether “judgment” overlays actual make a model better or worse? It’s debatable, has been controversial in many industries, and of course what makes an “expert” is a separate conversation entirely…but I digress. If they’re not a data person or never worked on models, it’s all going to go over clients head imo, maybe AI can help you convince the client why this is a bad idea? Might work depending on what bias it built into the models about themselves, but in the end everyone is too polarized on “AI will replace…”/“AI can NEVER replace?..” the truth is somewhere in the middle, pretend client does what they say but still needs expert consultants, and now you charge higher price for that service, letting AI do the work that can be done and meeting client in the middle. It changes the business model, but don’t throw the baby out with the bathwater?
1
1
u/Obvious_Lawyer_4672 Apr 30 '26
That’s not a “replace you” moment, it’s a “shift your role” moment.
What they built looks good in a demo because AI is great at answering questions on clean data. It breaks when definitions matter, numbers get challenged, or something goes wrong. That’s where your value is.
Your advice was solid. Put structure underneath, let AI sit on top. Most teams learn this the hard way after inconsistent numbers start causing real decisions.
If anything, this is a signal to move up the stack. Less “build dashboards,” more “own data logic, definitions, and reliability.” That part doesn’t get replaced, it gets more important.
0
u/Altruistic_Look_7868 Apr 01 '26
This field is a dead end. I desperately want to pivot out, but have no way to in this shit economy.
1
0
u/Cold-Dark4148 Apr 01 '26
Ey I’m confused what’s up with the analysing the ads? Everything in marketing analytics is already automated?
1
u/datawazo Apr 01 '26
It's larger than marketing analytics
0
u/Cold-Dark4148 Apr 01 '26
Elaborate?
1
u/datawazo Apr 01 '26
I just mean they aren't only analyzing ads, marketing is a facet of this but they've got data across departments that we were/they are integrating
0
u/heimmann Apr 01 '26
I’ve seen a few demos now where there are deterministic models below the Claude/chat layer, which ensures consistency and no hallucinations. Of course with our proper context and meaning it will not work but the emotes I’ve seen are very transparent, telling you the exact sql query being generated, which columns it knows and understands and which ones it still does not know for sure.
This will definitely change how we work, but we need to ask ourselves, did we get into this business in order to deliver insights or do we do it because we likes to work with the tools that allows us to deliver the insights. If a better hammers comes along (when proven better), we should all switch to the better hammer right?
0
u/white_tiger_dream Apr 02 '26
Hahahaha I would literally say “ok I’ll help and by the way we are raising prices but only for new customers so if you want to keep us at same rate that’s good but if you come back in 6 mos because AI fucked your shit the price will be 4x” diplomatically of course
-8
u/thatwabba Apr 01 '26 edited Apr 01 '26
Data analytics is dead. What’s left is to pivot into DE and connect (fix, define) a company’s semantic layer to an LLM and govern it until one also realizes that it works really well and you don’t have to govern it no more…
5
u/GamingTitBit Apr 01 '26
Graphs baby. Actual graphs, semantic tools are just duplo compared to building your own graph!
1
u/akkiannu Apr 01 '26
Say more? Can i read about this somewhere? Implementing dbt right now and want to see how this intersects with it.
2
u/GamingTitBit Apr 01 '26
Look up ontologically driven RDF graphs. They're not vendor locked (unlike Neo4j) and they require a human to build an ontology of their knowledge. This helps AI actually understand your data landscape. Even helps vector rag (Microsoft research released a paper in November on this)
1
5
u/akkiannu Apr 01 '26
Not really though. Claude lacks critical thinking and still has limited context.
1
u/Zimbo____ Apr 01 '26
I just got laid off today, and I was the entire analytics team at my company. You're not entirely wrong.
•
u/AutoModerator Apr 01 '26
If this post doesn't follow the rules or isn't flaired correctly, please report it to the mods. Have more questions? Join our community Discord!
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.