Recently, I gave the inaugural presentation for the newly launched GenAI + Teaching Discussion Sessions, hosted by AI Carpentry. You can watch the 15-minute talk, “Wait, is everybody using Claude Code?”. This was followed by a lively community discussion. I recommend joining future sessions if you’re teaching scientists of any sort about programming.

In short: I expected that scientific programming would get more uniform as scientists adopted AI tools, since LLMs drag you toward whatever’s modal on the internet. I think I was wrong and there’s still plenty of interesting variance. That leaves me torn between using natural language interfaces to push scientists toward developer norms and letting idiosyncratic workflows be. And if I did want to teach agentic coding—a distinct skill from copy-pasting out of a browser chat—what setup would I use? What’s the Google Colab of Claude Code?

After the session I answered some follow-up questions asynchronously. Here are a few, lightly rewritten.

Should students be expected to be productive with AI, or is slowness a feature of learning?

“Highly productive” may not be the right goal for a lot of learners. In a recent survey study, we looked at what factors are associated with perceiving yourself as most productive with AI coding assistance. Interestingly, having less programming experience and using fewer development practices (like Git version control, code review, testing, etc.) were associated with feeling MORE productive when you code with AI. What seemed most strongly associated with perceived productivity was actually getting a lot of lines of code out of your AI tool. It’s possible these relationships aren’t causal, but they give me pause. So at the very least, I’m cautious about having beginners use their own perceptions of productivity as a good metric for accomplishment, and as instructors, we might want to explicitly demonstrate how shallow cues like “lines of code written” or “my code runs without errors” can be misleading.

How can educators help students use AI coding tools without losing the skills computational research requires?

To put it shortly, I don’t know. One way I am thinking about this is teaching people to answer a meta-question: is vibe-ing okay for what I’m trying to do? How would I know when it isn’t?

For example, a lot of scientists need quick help writing code for a one-off visualization that gives them a view into a particular slice of their data. The visual isn’t the goal; it’s part of an iterative process of developing their own understanding of what their data “looks like” from a bunch of different “views” and slices. So here, I think a lot of people might rationally choose to offload some of that kind of quick-and-dirty coding to an LLM-based assistant (or any kind of available tool, really). But if they’re running the statistical test that is the crux of their analysis… that had really better be written correctly. So I’m interested in whether people can learn to pick and choose where to invest that precision, and it will vary a LOT by workflow and context and field. Of course, even in this example, there are moments when you need confidence your visualization code does what you think it does (you don’t want a visual in your published manuscript in which the code silently filtered data in an unwanted way before plotting).

How do I teach this? For now, I try to live code and explain my thinking to show explicitly when I’m investing a lot of cognitive effort to understand the code and where I’m operating more on a vibe. And I try to show the informal “stress tests” I’m putting my code under, modeling that I have an expectation for what should be coming out of every block. I doubt this is sufficient; I just don’t know the right answer here.

Which programming skills do we still need to teach scientists?

It’s probably an uncontroversial take, but I think it will depend on the context. There are areas of computational science where you really, really must understand code at the line level quite well. There are other kinds of research and data science programming where it’s more important to understand what libraries and abstractions exist (“This looks like a pandas-type project… did you know there’s a handy function for doing that data transformation?”). There’s an argument that we won’t need to even think about library APIs in the future, but as far as I can tell from my ongoing interviews and observations, scientists often go out of their way to require agents to contribute code from specific libraries that are familiar so they can understand it (although they might choose not to read it).

So right now, I would consider it a win if my students could identify: what are the load-bearing parts of their code, the parts that really must be valid (under some reasonable definitions for their goal)? And how would they detect if those load-bearing elements were quietly failing? I really don’t want them to get fixated on memorizing syntax or APIs—but that’s not new.

Can LLMs work as tutors that improve learning, rather than letting students outsource their thinking?

I am both (1) absolutely certain there are some folks who are learning meaningful things about how to code with LLM assistance, and (2) pessimistic about doing this at scale in a class with grades. Khan Academy recently overhauled their Khanmigo tutoring chatbot; Sal Khan said that for a lot of students, the bot was “a non-event.” That article suggests a number of possible reasons; the interpretation I find most persuasive is that on an internet where ChatGPT is freely available and will give you the answers directly, an LLM that scaffolds you to do the work yourself is not all that appealing for a lot of people. Maybe the idea sounds nice, but when homework’s due in a few hours… ChatGPT is going to win. I doubt I have any ideas the Khan Academy learning analytics team hasn’t already had.

I think anyone who wants to avoid outsourcing their learning to LLMs needs some external feedback on how they’re doing. In the classroom this has been done through assessments of artifacts like problem sets, a lot of which are way less meaningful as signals now. Unfortunately, well-designed assessments of people’s knowledge, not the artifacts they produce, are hard to run at scale. I feel most sure of where a student is at when I can talk with them about their code, but haven’t figured out how to make this scalable.

Should students confront the problem of managing large amounts of LLM-generated code early on?

I have had experiences where students are generating a lot of code, and they’re showing it to me, but now it’s a huge unversioned wall of code where no one can explain how it got here or why parts of it exist. So for hiring research assistants, I am now more concerned if they can manage code than write it—can you make a PR I could ever hope to understand and review? Can you show me the provenance of how you’re working? I haven’t had time to develop any teaching modules around this, but I wonder what others are doing. Maybe there’s a way to lightly adapt the Software Carpentry modules about Git for this?

Have you found evidence of actual productivity gains in your own work?

I didn’t see any meaningful gains until early 2026, when I switched to using Claude Code. All bets are off on whether it helps me accomplish more or better science, but I have seen benefits for certain kinds of coding (making browser-based interfaces for collaborative data annotation, getting a workflow running on a cluster where documentation was a bit limited). For data analysis, it’s more of a wash, because I spend a lot of time thinking through what I’m trying to do at each step.

Even though a lot of people have concerns about reproducibility and AI, I think using an agent has helped me stick to better reproducibility practices. I commit more regularly, I write tests more, every project has its own virtual environment… maybe this is because I already wanted to be doing these practices and was often annoyed with myself for falling short. I think someone saying “I had AI analyze my data and here’s the answer” is a wreck scientifically, especially if by this they mean “I uploaded a .csv to ChatGPT and here’s what came out.” But I now also think there is a possibility that certain uses of agents could support code habits that are good for reproducibility.

For that to work, though, there has to be some kind of version control at the heart of a codebase, which has historically been a pain point for a lot of scientists. So either the classic Git Software Carpentry lessons are exceptionally needed, or we need some scientific-context-aware ways to support less fluent users interfacing effectively with Git (Jon Udell’s Bram offers some possibilities).