How data scientists talked about AI — 2022 to 2026

For people that have new — everyone's excited about ML and AI. But there should be some sort of detailed — or an educated guesstimate — of ROI for the projects that people want to put in. And that's really hard to do, even for experienced developers and data scientists.

I think things that are happening in AI, and generative AI specifically, are exciting. But I feel technology companies and technologists sometimes feel like, we should do this because we can, or we should do this because it's cool, and haven't thought about the broader social, political, economic implications. I think we're going a little too fast.

I mean, maybe if ChatGPT becomes even better and then we become obsolete, OK, maybe. But I don't think that's going to go away. I think the way to prove the contribution, the benefit of a community — I used the word stuck and the other word unstuck — because that's really the only thing any business cares about. … There are obstacles in the way, they are stuck. How do they get unstuck? That's all data science is for, really.

So when I, as the boss, call you into my office and say I heard about this ChatGPT thing — I'm not talking about getting rid of you. I'm saying, you have been digging ditches with a shovel your whole career. Now I come and give you a backhoe. You're not going to use it? It's a force multiplier — if you can apply these quantitative tools to suddenly process far more data than you used to, that's a competitive advantage.

The public version-- just word of caution for all you out there-- never put company proprietary information or customer information into the public ChatGPT.

We're running towards a disaster on the AI governance front. You gotta know the data it feeds in. You gotta know how you're allowed to use it. You gotta have a policy. You gotta implement. You gotta audit. … And if we thought data governance was a problem in the past, AI governance is gonna be our nightmare in five years if we don't get a handle on it.

LLMs really do leak data. LLMs really do say crazy things, and they really do, not always have transparency. And even when they have transparency, it's not always real.

And what's funny is I'll get in there, and they're like, how can we integrate AI? And I'm like, you guys don't need AI. I was like, this is a bad idea. And in fact, I think it's gonna really slow you down. Some companies have use cases for AI where it makes sense, and there are certain tools that they can integrate. But a lot of companies don't.

Copilot is a big deal for developer productivity — you can use it to generate unit test cases for you and help with repetitive tasks that you might otherwise have to do a bunch of typing on and maybe make a lot more errors. Obviously there are copyright concerns and IP concerns … about misuse of LLMs.

I personally pay for Copilot, and I find it's very useful for my coding, to get things done quicker. But I have to double, triple check everything that comes out of it.

I find the answers that I get out of LLM models to often be wrong or imprecise. But often helpful. It gets me closer, and then I can iterate. Usually, by the time I get to the right question, I understand what the answer is. It's funny how that works — you're like, oh, my question was wrong. So I do think that for senior or skilled analysts, that's gonna be an incredibly valuable tool that's gonna accelerate development time.

In terms of stuff that I'm less excited about — the AI hype is something that causes me to ruthlessly unfollow people on LinkedIn — because it just makes everything so noisy. … I would say don't delegate your specialism. You need to stay as the expert in what you do.

And so having this sort of head of AI, people that are responsible for being that expert kinda prevents what I call AI grift. You basically prevent these grift organizations from getting in and extracting money, when in reality what they built is unsophisticated. They don't have any intellectual property. They don't have some secret data. They basically just glued a bunch of off-the-shelf stuff together.

I think sometimes there is a tendency to think that AI can do everybody else's job but not your own so I've seen a lot of people talking about how it can be useful for cleaning up data.

I'm old school — I still call it machine learning. There's no such thing as artificial intelligence. I think it's very useful.

With Gen AI, we now have tools at our disposal that are gonna be able to do this even more. My goal has always been, can I automate myself out of a job and then move on to another one? So it's always — can I get to the point of it doing what I would do?

But, you know, I think one of the biggest things right now is every senior leader wants us to do more with AI. It's like, well, then upskill your people, because there's only so many out there. You can do a lot if you train people. Every person that I know who's been really successful in analytics has learned outside of normal working hours.

All these companies had ethical AI teams inside them. And maybe a year or two years ago, those teams were the first one to get laid off. So there is no team anymore, pretty much, in those companies where they are thinking and they are vetting.

I would probably say that, if you're not in a situation where you have to actively build AI workflows right now, I would limit the amount of time you're spending learning how to code agents and things like that. Because if you're not doing it right now, you don't really have a use case to use it on. And by the time you do have a use case, it could have changed a lot between now and then.

But where we are, where that was then, we are with generative AI now. Almost everyone is at the same place. I think if you're young or mid career and you're interested in that — especially if you're somewhere else and planning to move to that — this is your moment, in my opinion. Get really up to speed as much as you can and go for it.

So early on in the LLM hype cycle, I was super skeptical because I was like, I can write R code much better than this thing can, like, much better. But my attitude changed a lot when I applied it to problems where I need to write Rust or TypeScript, which I'm substantially less good at. Part of one's skepticism can come from — well, how much are you expanding your comfort zone? If you stay in your comfort zone, the relative gain might be small depending on what your comfort zone is.

But I think if you use LLMs in that way, like, really think about the context, and really critique each answer, they end up acting like a giant rubber duck. Forcing you to say those things actually forces you to think those things. … I think certain types of habits in your LLM use can actually force you to refine your own thinking, whereas before you might have just started coding.

They've since said the one thing — everyone thinks they have to know the latest AI thing these days. Most organizations are doing their statistics and predictions in Excel — they've got a spreadsheet with two columns of data. A lot of companies are actually looking for people who are really solid on more conventional data analysis techniques. Maybe they can use AI tools to work faster, but they're not putting 'prompt engineer' on their resume.

The Shiny assistant — almost all the intelligence is just Claude. There's very little custom. We know there are a few common mistakes that it makes with Shiny that we instructed it not to. But other than that, it's mostly just Claude doing what it knows about Shiny.

This week, I was building an app with Claude and Positron that was showing all the different use cases we have across different industries. It's really about different ways where I can bring the customer voice into things that we do at Posit.

And so I started a conversation with Claude. I said: how can you achieve computationally reproducible research results when you're using a nondeterministic tool like AI? It said, well — that's not really the problem. The problem is that people overclaim the impact or significance of their results. But in a domain that rewards positive results, how is a tool optimized to give confident, comprehensive answers going to shape the direction of the science? Won't that just reinforce positive overclaiming? And it said — this was the most disarming answer I've ever gotten from an AI — yeah. That's a really good question. This might get worse before it gets better.