Blogmark

Supervisory Engineering Work

We propose a new category of work we term supervisory engineering work, encompassing the direction, evaluation, and correction of AI output.

The productivity-experience paradox of working with AI-coding assistants:

We also identified a productivity-experience paradox: productivity perceptions held stable, with 84% reporting improvement at both time points, yet among matched participants, the proportion reporting worsened developer experience in at least one dimension nearly doubled from 14% to 27%, with flow state and cognitive load eroding while feedback loops improved. These findings suggest that AI coding assistants are impacting both the nature of software engineering work and how engineers experience it.

There have been studies on other aspects of LLM using in software as well as productivity for students, but not of software professionals.

Further, what remains largely missing is an understanding of how professional software engineers experience these tools, and whether those experiences change over time.

It strikes me that we need a classification system for the different forms of AI-assisted software development. These tools are used in widely non-uniform ways. From tab-completion for the current line or next couple lines to series of small prompts where the developer views and confirms each changeset to “auto mode” flows where an entire task or feature is completed by an agent harness to multi-loop agentic flows and beyond. Productivity gains and developer experience must vary widely across these types of AI development.

LLM tooling can tighten the feedback loop during development.

However, the usefulness of this feedback depends on its quality and relevance, and developers still report needing to validate AI output carefully.

Cognitive Fragmentation:

Overall, prior work indicates that AI coding assistants can improve aspects of developer experience while simultaneously introducing new sources of cognitive effort and fragmentation.

Concerns about maintainability had a big jump between the initial survey and subsequent survey. Which I interpret as the more these tools are used, the more maintenance becomes a growing concern.

Quality was the dominant primary concern at both time points, though it declined from 44% to 36%, followed by security. The most notable shift was in maintainability, which rose from 3% to 19% as a primary concern between Q1 and Q2.

Learning, onboarding, and code understanding are improved by LLMs and coding harnesses.

Sub-theme 2B: Accelerated learning, onboarding, and code understanding. Participants in both time points used AI to accelerate understanding across unfamiliar languages, frameworks, or legacy codebases. For example, one participant wrote: “Much easier to pick up new tools, libraries, languages and get productive.” (P132, Q1). Another wrote: “It has been particularly helpful in explaining legacy code and detecting bugs more efficiently.” (P2, Q2). Others reported gaining conceptual clarity: “Definitely felt like it’s given me a level up on understanding new concepts” (P5, Q2).

This supervisory work doesn’t really map onto existing tasks. It’s often compared, in my experience, to being a lead who is reviewing and giving feedback to others on the team, but I don’t think this is quite the same activity.

This supervisory work replaced portions of hands-on implementation effort but does not map cleanly onto traditional software development task categories. As one participant explained, “Sometimes, I’ll not be happy with the original reference, and will ’coach’ the AI to produce something that works better for my use-case”

Interesting, there is a “Use of Gen AI” disclaimer at the end of the paper.

We used Claude Code (Anthropic) to assist with developing R scripts for statistical analysis and reviewing written text for clarity and readability. All outputs were critically reviewed and validated, and responsibility for the correctness of the analysis and interpretations rests entirely with the researchers.