Did the Agile Manifesto, Scrum and open source kill the programmer?

A lot of what we call modern software development is really the management of change. Not code, change. Who asked for it, what it was supposed to do, what we tried, what broke, what got accepted, what got rejected, and what was shipped? The output of the modern software development process has become a large set of artefacts: man-made entities that provide a trace of what the activities were. They come in the shape of issues, tickets, pull requests, reviews, merges, releases. But this was not always what the artefacts of programming were. It belongs to modern software development culture. How did we end up here?

In the beginning of software development, upfront documentation and specification ruled, and learning too late that you had built the wrong thing reigned supreme. Many software development projects failed or got cancelled. Modern software development culture is a response to the risks and challenges in the waterfall software development model, which tried to ensure a complete view of what to build before starting to program the solution. Needless to say, by then the initial set of requirements would invariably have changed while the project either got built according to stale requirements or stalled to update the specifications.

The Agile Manifesto does not mention tickets, issue trackers, epics, or velocity. It is a preference for how to work and it makes a set of trade-offs. Less guesswork, more feedback. Less documentation, more incremental work. Less resistance to change, more willingness to adapt. Work is optimised for short loops that reduce lead time from business demand to shipping code. Increased business velocity became the organisational selling point for the new ways of modern software development. However, short loops necessitate small work items, which in turn create a rather large set of other artefacts.¹

Scrum also gained traction by turning the short loops into a coherent routine. Time was boxed and co-ordination became a set of recurring events, which reduced the cost of aligning the group. It also made work legible to people who were not reading the code, "the business", that is.² Requirements stopped being long documents and were reduced into smaller packets. A packet got split again until it could fit inside the cadence. In many organisations this ended up as epics, stories, and eventually tickets. Navigating, feeding, and reading ticket systems and work tracking interfaces became the job.³

Open source had a long era before the advent of modern software development practices. Being distributed by nature, patches and managing code in version control systems like CVS and Subversion were the norm. There were mailing lists and web forums, and bug trackers like Bugzilla for community and co-operation.⁴ It was a decentralised affair with different tools for different jobs and different tools for different projects. Although there had been previous attempts at integrating the different tools, Git became the versioning system of choice in the community and GitHub became the repository of choice with its free open source hosting. Git came later.¹⁰ In 2009 GitHub added an integrated issue tracker, making issues a first-class object inside the repository.⁵ Pull requests make proposed changes reviewable and discussable before merge, with the pull request page acting as a central hub for the work.⁶ The GitHub open source archive, built for humans, turned out to be a treasure trove for machines, especially machines training machine learning models to do ticket-based programming.

If you want to train systems to do ticket-based programming, you do not only want code. You want examples of transformations. You want the text that explains why the transformation was requested. You want the metadata that says whether the transformation was accepted. It is not surprising that code assistants are trained on public code. GitHub Copilot states that it was trained on publicly available sources, including code in public repositories on GitHub.⁷ The Stack is another example of what it looks like when you treat the public code commons as a training corpus and curate it at scale.⁸ Agile, Scrum, and GitHub did not just make more code. They created necessary and sufficient conditions for learning the ticket-based part of the job at scale. In hindsight it was a perfect storm. Modern software development emits aligned pairs of intent and transformation, with issues and pull requests sitting next to diffs, plus behavioural signals in the form of reviews, approvals, merges, and releases. That made modern software development a prime target for the first wave of large language model based automation.⁹

However, the automation of code does not imply that less code or software will be produced going forward, au contraire. Ticket-based programming has become a task a machine can complete increasingly well, given enough artefacts. The non-programming part becomes more valuable and more clearly software engineering, programming in this context being reduced to ticket-based work with code output. Choosing boundaries, deciding what to build and not to build, deciding what to delete or keep, and owning long-term planning and long-term consequences will still be needed. It just does not compress into neat work items, and it is not available online in abundance. This entails that we can have a future where the profession called programmer shrinks while software development and software engineering expands. More people will develop software without being programmers. Meanwhile, a substantial subset of ticket-based programming will be automated.

Agile, Scrum, and open source did not set out to do this. But they did help turn software development work into tickets. Killing the programmer?

Andreas Påhlsson-Notini

a@nial.se