# The Age of Agile Hasn't Even Begun Yet

Back when I worked at Rackspace, I once used to work with someone who, over
time, I came to be good friends with.  We shared many similar thoughts on
software development practices, and frequently had engaging conversation
about project management and evolution.  Recently, he posted a link to an
article which I have some disagreement with, and I'd like to address these
disagreements here.  The article is titled, "The Age of Agile Must End",
posted February 12, by Michael Burnett.

I don't think it's any secret that Michael Burnett doesn't like agile
development methods.  He writes,

> Agile development has been the #1 operational principle of tech for
> over 20 years, unchecked, unquestioned.

Of course, Michael must not have bothered to search on the Internet for any
of this, because if he did, he would have found the
HUGE religious wars that occurred between advocates of
heavyweight, formal development methods, such as those put forth by the SEI,
and the extreme programmers and early Scrum advocates like Kent Beck, Ward
Cunningham, and Ken Schwaber.  But, I am not Google, so I won't bother to
list those debates here.  Suffice it to say that, eventually, even the
mighty SEI had to come around and see the light, and even accounted for
agile methods when they released their CMMI framework.

Unchecked and unquestioned, indeed.

Still, in an attempt to provide evidence for his thesis, Michael strongly
insinuates, in a not-so-covert way, that waterfall and lean approaches to
software development are essentially equivalent, only further demonstrating
his willful ignorance of the history of agile and so-called non-agile
methods.

After discussing the introduction of lean production systems in America,
Michael then writes:

> The Waterfall Scapegoat
> 
> The burgeoning software industry largely followed these processes;
> applied to digital products, they became known as Waterfall
> methodology.

No.

They didn't.

In fact, software engineers here in the USA weren't even aware of the lean
production system until at least the late 70s/early 80s (and I'm being
deliberately generous here, as I wasn't in the software engineering field at
this time; it's more likely to be around 1988 based on texts I've read
since), while we've been basically following the Waterfall process in one
form or another since at least the mid-50s.

In fact, Waterfall didn't even acquire its name until 1970 when Winston W.
Royce wrote "Managing the Development of Large Software Systems" for IEEE
WESCON (Wikipedia, accessed 2023-Feb-19) as an example of what not to
do in software engineering.  Funny, really, considering Royce didn't even
use the name Waterfall in his paper.  That was adopted organically by the
greater development community.  You're welcome.

In fact, the whole idea of iterating on software wasn't even remotely
popular at the time; it wasn't until Boehm's paper, "A Sprial Model of
Software Development and Enhancement", that the idea of iteration became
common vernacular.  That was in 1986, more or less contemporary with
Scrum and Lean, in case you were wondering.  But, we'll get to Scrum later.

For the record, lean software development, as a thing people would actually
start talking about, didn't start to get a foothold until after the
introduction of Kent Beck's Extreme Programming made alternative models of
software development popular enough to talk about without a high probability
of actually getting fired.  That's a whole decade (more or less)
after thinking on lean software engineering started, and two and half
decades after Waterfall was coined.

Yup, that's right -- agility was hotly debated as far back as 1970.
Possibly even earlier.  But, we won't let facts stand in the way here.
Michael continues,

> The term Waterfall has become a catch-all to describe a linear
> development process that culminates in a fully realized product release.

Yes and no.  It really depends on what it is we're linearizing here.

The author is correct that lean depends on a linearized set of
production steps.  The existence of swim lanes implies this, and even a
cursory study into lean's history with the Toyota Way will demonstrate it.
However, these production steps are arranged in one or more pipelines (each
given its own swim lane), just like a real factory would be, so that
multiple components of a project can be worked on concurrently, and
integrated to form larger components.  This could be one big integration
like this:

    Task A ---------.
                    |
    Task B -------. `--|
		  `----|
    Task C ------------|----> output
                  .----|
    Task D -------' .--|
                    |
    Task E ---------'

    Time ----------------------------->

which I emphatically do not recommend; or, it could be more gentle, like
this:

    Task A ---.
              |
    Task B ---+--------|
		       |
    Task C ------------|----> output
                       |
    Task D -------+----|
                  |
    Task E -------'

    Time ----------------------------->

Thanks to lean's pull-based workflow, each task can proceed at its own
natural rate of progress.  Contrast against Waterfall, where each task's
scope and progress are rigidly defined a priori by requirements analysts and
project managers (resp.) first.  If any of these needs to change, the
whole process starts over from scratch (more or less), which is
demonstrably extremely expensive.  It's right there in Royce's paper.  So,
it's natural that companies tend not to do that, which is why usually
companies compromise on delivery dates, quality, or both.

Even someone incompetent in the art can tell you that these approaches
differ strongly.  With Waterfall, teams work on exactly ONE component
through a distinct succession of top-down refinement steps, only the last
couple of which are related to actual coding effort.  Granted, some
allowance for prototyping is made; but, nothing like what you would find in
any typical agile team.

Moreover, with lean development, any blockage in the flow of development of
a component is intended to be addressed by on-demand, proactive offers of
assistance from other team members to help unblock the stages involved.  If
you're just sitting around, twiddling your thumbs, because your supplier for
a component (be it another company or the person in the cube across from
you) is not delivering on time, then it is in your best interest to reach
out and offer to help.  This is strongly similar to Extreme Programming's
rule of Collective Code Ownership.  This can happen along a single swim lane
or across swim lanes, as necessary, thanks to its pull-based nature.

Note that there's no discussion about blaming your coworkers here.
One should not feel guilty for being slow on the team, as long as you can
learn from the experience and grow.

The Waterfall method, in contrast, is extremely push based and highly
regimented.  If there's a blockage, production usually doesn't stop while
the blockage is cleared.  Instead, resources might (if you're lucky)
be re-allocated to different projects in an effort to unblock, and failing
that, the slower team mates are frequently just terminated as
ineffectual and/or a drag on team progress.  In the worst cases, such as on
government contracts, you might even find yourself getting paid for
literally doing nothing due to the strong compartmentalization involved.  If
you've ever heard the phrase critical path or bottleneck
(typically used in phrases like don't become the bottleneck, or,
we don't want to be on the critical path) while working on a
commercial project, that's a good indicator you're actually using some
aspect of Waterfall methodology somewhere in your project planning.

This kind of misunderstanding appears frequently in many critiques of agile
processes (note: lowercase-A), and frankly, it rather pisses me off.  We in
the agile community have been trying to communicate these differences for
decades now (30+ years in my case), but we seem to be summarily ignored for
the convenience of thought-leadership articles like the one linked above.
Eventually, we just stopped caring, and now we're stuck with the spread of
misinformation as authors play the game of Telephone, first having gotten
their information via that same game of Telephone.

Unfortunately, this game has real consequences on us as software engineers.

Please note: this blog article is not intended to denigrate Waterfall and to
push agile methods as an unconditional truth above all else.  While it is no
secret that I strongly prefer more agility in software development than
less, there are successful counter-examples on my resume.  My current
employer, for example.  But, one of my most fondly remembered experiences as
an engineer was while working at a company called Plum District.  When I was
there, we incorporated the V Model (traditionally associated with heavy-weight
methodologies like Waterfall) successfully with Scrum.  For our team, V
worked fantastically, allowing us to deliver new feature requests and bug
fixes on-time and within budget in almost all cases.

In fact, of all the places I worked which had truly successful, happy teams
working on products, none of them wore a badge of honor just for
using agile or non-agile processes.  None.  All of them are happy to admit,
"We take a little of this, and a little of that, and a little from those,"
approach.  They are quick to adapt and react to problems that the team as a
whole is having.  It is the team that defines the processes used, not
some consultant, and definitely not some manager who hasn't touched a line
of code in thirty years.  The paper certification they have is ultimately
worthless, be it from the International Scrum or the Software
Engineering Institutes.

Reading further into Michael's essay shows the gradient of misinformation
into disinformation.  But, before he does, he writes the following, which I
believe unconditionally:

> The Agile vs. Waterfall argument is a false dichotomy: a company can have
> a long-term strategy and carry it out while still releasing the product
> incrementally, iteratively.

Michael's entire essay could have been replaced with the above paragraph,
and all that needed to be said would have been.

But, Michael goes on about Scrum, again getting some very significant
details wrong.  Knowing these details is important to know when to, and when
not to, apply Scrum in a project.

> As product owners and engineers attempted to apply lean manufacturing
> principles to software development, they found ways to mimic an assembly line.

Unfortunately, that's not true.  Companies have been treating programmers as
an assembly line since the 1950s, as I discussed above.  Scrum evolved
contemporaneously with lean.  You might even say that Scrum and lean are
outright competitors in the agile market.

Scrum was intended to address the needs of companies that needed to pivot
fast.  The idea was to gather a cross-functional team into a
war-room, and to bang out a product that met some new requirements before a
competitor could move into their market of interest first.

The fact that Scrum has escaped the war-room and become de rigueur is the
fault of both product management and those consultants I spoke of above.
See, Scrum was exceedingly successful at what it did, so naturally,
it was applied at wider scales of production: the on-going day to day
maintenance of a project, for example.  Here, Scrum languishes like a
fish out of water.  It is, literally, wholesale, unequivocally, and
indisputably the wrong tool for that sort of job.  Lean is head and
shoulders superior at this task, precisely because on-going
maintenance is so much more like running a production line.

Michael then goes on to list all the bullet points making Scrum, well,
Scrum.  (Not really; these largely derive from Extreme Programming, but I
digress.)  But, one item sticks out as wholly false to me:

> There's a Daily Standup to have on-the-fly "operational improvements."

No.

Just, no.

I mean, yes, but not in the way that Michael is scare-quoting
operational improvements.  So, in the end, just, no.

That's not the purpose of this meeting.  It is intended to be a status
report by the developers to report what they've done, what things are
blocking them, and what they intend on working on next to their fellow
developers.  This is literally text-book Extreme Programming here, and
has as much to do with organization-wide "operational improvements" as
literally any other programming methodology on the planet.  The fact
that Michael quoted this phrase tells me Michael has little grasp on the
utility of a methodology at all. Even Waterfall is intended to
improve operations.  Otherwise, what's the point?

But, he goes on:

> (Only developers are allowed to speak and discussion is not allowed
> because that leaves the machinery idle.)

Wha??!!

No.

Again, just, no.

The reason why developers are allowed to speak is, because, it's a status
meeting.  Discussions happen all the damn time.  That's how one
engineer can respond to another engineer's report of a blockage.  That's how
junior engineers can gain insight from senior engineers.  That's what
operational improvements is all about.  If discussions turn out to be
lengthy, then you schedule future meetings on the topic.  This is just
normal, every-day, good meeting etiquette.  Michael seems to be upset
that good meeting etiquette was codified.  Based on this reaction, I
strognly suspect Michael would be equally enraged by a project Code of
Conduct.  If so, Michael is expressly not the kind of project manager you
want on your team, agile or otherwise.

The reason only engineers can discuss things is because stake holders
have their own meetings in which they can voice requirements changes.
If a stake holder were to have a voice during a stand-up, then what's the
point of planning sessions with a customer?  Of requirements analysis?  This
is something that both the SEI and every agilist alike actually agrees on:
change requests that come too frequently will kill a project dead.

Dead.

Period.

This is not open for debate.  This is documented fact.  As I write this,
we'll have, what, 67 years of research on this topic somewhere in
SEI's library?  I encourage you to read some of the reports SEI has written
about large projects, both successful and failure alike.  It is a literal
Hollywood trope to have management come in and just randomly change a
requirement, forcing engineers to rip out and re-work software on an
ever-running treadmill of change requests.  Hasn't anyone seen Office Space
recently?  That's not just a comedy; it's also a documentary.

Finally, there's this gem.

> The Backlog isn’t core to the Scrum practice, it’s an artifact of the
> collateral damage and detritus of everything jettisoned during the sprint.

I ... just don't know how to respond to this except to say that I've stopped
reading the article critically right here.  Literally, there is no
further point in continuing.  Michael has made plain for the world to
see he has no understanding what-so-ever of development methods and the
tools they employ to manage the process.

At the end of the six bullet points defining Scrum, Michael writes,

> You can see how Scrum is fairly illogical already, but then it gets so
> much worse.

Actually, if you study literally every other software methodology on the
planet (including Waterfall!), you'll find that they also have:

1. Planning sessions where stake-holders are allowed to periodically review
   progress on the product.  They're the ones spending the money to fund the
   development, after all.

2. Planning and status sessions where teams interact to discuss feature
   development.  In agile terms, this would be your daily stand-up and sprint
   planning sessions.

3. Change and configuration management are strongly encouraged.  Granted,
   this came a bit later in Waterfall's evolution, but by the time V and
   Spiral models came out, this was just a given.  In practical terms, this
   amounts to the use of things like "backlogs" (which Michael says is an
   afterthought in Scrum), revision control systems, ticketing systems, etc.

4. Retrospectives are highly encouraged by SEI for all of
   their heavy-weight processes and light-weight processes alike.  Even
   Watts Humphrey himself, in his books on the Personal Software
   Process, encourages self-reflection on a periodic basis to re-evaluate
   what is and isn't working for you, the developer.  And he admits
   openly in his books, if you can't get PSP, don't bother with TSP.

With this knowledge, one starts to see how Scrum is not only extremely
logical, but also how it fits into the larger ecosystem of development
methods one can choose from for a project.

To further illustrate how far removed from reality Michael is, he writes
down below:

> The combination of Agile principles and Scrum practices is disastrous
> for startups.

Really?  I openly invite Michael to try and convince the boards of directors
of the venture firms backing Meta (nee Facebook), Alphabet (nee Google),
Amazon, and Space-X that their clients' respective methods of software
development is disastrous.  Even Microsoft, which once invested
heavily in PSP and TSP back in the Windows 2000 era, has since switched
towards more agility, at least for some projects.  Please!  I really want to
see what happens.

I do agree with what Michael wrote above in his essay, which I'll repeat
here:

> The Agile vs. Waterfall argument is a false dichotomy: a company can have
> a long-term strategy and carry it out while still releasing the product
> incrementally, iteratively.

I cannot emphasize how much I agree with this single statement.  But, that's
only because a broken clock can still be right twice a day.  So, Michael, I
emplore you to follow your own advice on this topic:

> Just stop.

You're hurting more than you're helping.