The dizzying experience of "vibe coding"

Moving away from Claude Code and tools like it

I remember more than ten years ago laying in bed in Lansing, Michigan, reflecting and looking over code I wrote. I think it was for an R package, basically what I’d call research software. Not professional software, but software for researchers to solve some task — in this case, one related to data analysis. It probably involved multiple scripts, files with code that took some inputs, and produced some output. And it worked. It worked! I couldn’t believe it. I wasn’t trained in any kind of software engineering; I was self-taught. It felt magical and gratifying.

I built on this experience over many years, getting better at research software engineering, building software that has been used, well, lots. Kind of astonishingly, the tidyLPA software I started to build as a doctoral student has now been referenced in more than 1,500 publications — probably being used by tens of thousands of researchers and analysts, and downloaded well over 100,000 times. And it has (thanks in large part to my co-developer C.J. van Lissa), “held up,” in that it hasn’t caused unintended issues for its uses, like incorrect output. I’m not a professional software engineer, but I have worked on software before the use of AI that has, then, been widely-used, and I have enjoyed the process of building tools like it.

Claude Code, of course, and other agentic AI tools like it (Codex, Replit, OpenCode, and many, many more of different flavors and with different goals) do what my colleagues and I did with tidyLPA, and lots more. This is often a part of what was termed by Karpathy as “vibe coding.” Such agentic AI tools are good for data analysis, too. Spin it up (I wrote a bit on it here, several months ago), give it a prompt and some task, and, yeah, it will, almost certainly, do it. And so I’ve probably used Claude Code for around 20 projects, and it’s often useful — more or less depending on the project, and, maybe interestingly, depending on how many times I iterate on the project. But the biggest difference I’ve noticed is in the experience of vibe coding.

Coding is hard, but, for the right task, it is enjoyable. Relaxed concentration? Working with a few data files for some topic or context I know well almost felt like taking a run, reading a book, or a good conversation. . .

library(tidyverse) 
d <- read_csv("data-file-1.csv")
d %>% select(stud_id, outcome_1)

I smile writing code like I’ve probably written 1,000 times.

Claude Code isn’t like that. It’s dizzying. And, worse, at the end, I feel like I don’t understand what I’ve produced; I don’t understand the data, which I inevitably do as I work through which variables to select, which files to load, and what the data might mean. And I am not sure of what I’ve produced; is it right? Did I miss something? As I iterate on the data with Claude Code, I can get to something usable, but it often comes at the expense of roughly the same amount of time it would have taken to do the analysis without it; and, the experience is more removed. Dizzying is the word I keep coming back to; and disorienting.

I think part of the reason this is an issue for data analysis, specifically, is that checking the output of individual data analysis steps when we are writing code is a part of the process. To get scripts, especially (but also research software) to work, one has to constantly check output and understand the data or inputs and the analysis in the process. But with an AI tool, it becomes a separate step that often doesn’t get done (or it’s a pain in the butt to do). Some of the most important parts - validating what you've done and making sense of what you’ve done - don’t, uh, happen, at least not as a part of how I’ve often used Claude Code. In short, doing the analysis is a way to understand and make sense of data. Simple. I think I forgot this over the last year. It’s not impossible to do this with tools like Claude Code, but I think (h/t Brian Kim for this point) we’ll need to innovate new workflows, approaches, methodologies, techniques, software, etc., to do this.

And so after adopting AI tools for coding relatively early, I think I now want to step away from them as they continue to get better, or, at least, to use them more selectively, as a part of the process of coding and analyzing data I knew well before the last few years. Some ideas include:

1. Using these tools only for discrete tasks you can write you, e.g. “Create this graph based on this data showing Xx and Yy”

2. Always having an editor / IDE (RStudio, Positron, whatever you use) open

3. Using these tools in code completion form or using an agent alongside the text editor, ensuring you at least copy and paste the code into your project

4. Manually checking that the code runs and that the output is correct and that you understand it

5. Manually documenting what you understand the code does for your benefit and to check alignment with what these tools are doing (CLAUDE.md is a useful file for such a thing)

I’d be curious to hear others’ thoughts here.