If you're middle or late in your career, you've noticed a pattern. The majority of software shops seem to struggle with two things:
Bad managers at the top
A tendency towards waterfall development
If you're early in your career, you've undoubtedly seen enough complaints on various subreddits to expect this.
There's a big question here: Why?
The accepted answer tends to rely on where you are in your career and how cynical you are.
Early in your career, you may think, "These people just have not heard of Agile yet; I can educate them."
In the middle of your career, you know the above is naive. Moreover, you've been through enough bad faith agile rollouts to be skeptical of agile yourself. After all, you think, if there truly were something better, it should outcompete this typical pattern, right?
What I'm hoping to show you in this post is:
Why the answer to this question isn't obvious
Why the answer has to do with culture
Why Does Everything Look the Same?
Okay, so let's adopt some Cartesian doubt and pretend we don't already know why Waterfall development and bad management pop up repeatedly.
It's not because it's superior.
As argued from this blog, agile development has been outperforming waterfall development since at least the 1980s.
The evidence for cross-functional teams, iterative development, and high trust environments is all over the literature. When companies adopt these things authentically, they do find success.
So we've got to clarify this question: Why is it that many software shops still struggle with bad management and Waterfall development despite agile methods leading to success in the marketplace?
It's Not Just Software
But even this misses the point of Takeuchi and Nonaka's research on the roots of Scrum.
They found two kinds of entire business processes - whole ways of running organizations. They didn't see a difference in just software departments. In other words, their original formulation of Scrum applied to the enterprise at large (which is one reason you don't have to do anything to scale Scrum).
They called the standard way of development - which will look suspiciously like Waterfall - "gated" development.
These gates don't just exist between UX and backend, QA and IT. In addition, they cross finance, HR, marketing, and every other business function. The Waterfall goes a lot higher than just what the software department sees.
So, another iteration on our question: Why is it that many organizations still struggle with bad management and gated development despite agile methods leading to success in the marketplace?
Let's Define Bad Managers
There's one more iteration we need to frame this question that will make it clear. What is "bad management" here?
Now, there are probably a lot of ways to define this. But there's one definition that will make the answer seem obvious. We will define bad management as the overuse of the "command and control" management style.
Managers who adopt this style use aggressive language, and power plays to get work done through people, whether you want to call it micromanaging or bullying. They usually don't seem to care about the consequences.
We're going to call these folks "authoritarian."
This finishes our question.
Let's take another crack...
Why Do Many Organizations Still Struggle with Authoritarian Managers and Gated Development Despite Agile Methods Leading to Success in the Marketplace?
Our question is now well formulated, but the answer will take one more jump of insight.
You may have tried working cross-functionally before; you may have even succeeded. After all, the research shows cross-functional teams are faster and produce fewer errors than departmental teams.
Whether it worked out for you or not, you almost certainly noticed one overriding issue: working cross-functionally in a departmental organization is incredibly difficult. It nearly always fights how the organization "wants" and "expects" to work. Unless there's a champion driving it constantly, the organization will devolve into gated development.
Successful cross-functional transformations get rid of functional departments themselves. This is somewhat rare. But, living alongside functional departments, no cross-functional work lasts long. Either cross-functional teams win, or departments win.
In other words, working truly iteratively (which requires a cross-functional team) is very hard to do in organizations with strict functional hierarchies. So, to a degree, the answer to our question is:
Gated development happens because it's the natural workflow of rigid, departmental hierarchies.
But this just begs the question: if agile methods are superior, why do we find so many strict department hierarchies in businesses?
Why do authoritarian managers prefer strict departmental hierarchies?
Authoritarian is a leadership style, but the tendency to be authoritarian is also a personality trait.
These folks tend to...
View everything as a zero-sum game. If you win, they lose.
Be very competitive.
Seek power.
Work for themselves rather than the group.
Does this sound like your organization? Do you want coaching on what you can do to personally survive or consulting on how you can make your team thrive? Contact us!
Authoritarian people seek power, while more honest and humble folks tend to eschew it. All other things being equal, you'll naturally find your leadership posts occupied by more and more authoritarians.
The rest of the functional hierarchy falls out from that: authoritarians will tend to produce elaborate hierarchies beneath them for various reasons. But the hierarchy follows the authoritarian personality rather than vice versa.
In other words, if you find yourself in a situation where there's a hierarchy of departments, your pleas to senior leaders that this situation is inefficient will often fall on deaf ears. They may have built that hierarchy on purpose.
Not all functional hierarchies are run by authoritarian managers, but nearly all authoritarian managers build functional hierarchies.
Why?
For senior leaders, it divides and conquers rivals.
Having many different departments helps divide and conquer rivals for the very top. Many different sub-organizations with varying levels of 'clout' allow a lot of rewards via promotion and punishments via transfers to those at the top echelons of power.
It also makes each department head view the other as a rival rather than focusing their efforts on replacing the senior leader.
Authoritarians view all of life as a competition. They assume all of their subordinates are competing to oust them. Thus, they use their authority to shift the organization to one where their subordinates fight with each other. This spares the authoritarian leader.
For senior leaders, compartmentalization prevents any department from doing anything independently.
By creating departments, authoritarian senior leaders also make it tough to do anything without the senior leader's express knowledge and approval. In other words, they've locked the guns in one safe and the ammunition in another and made sure the two key masters to those safes hate each other.
This ensures no uprising forms against the senior leader and ousts them.
It will only be the senior leader who can rally the resources of finance, engineering, and IT to get a project through (or embarrass a rival). Each of the department heads of those groups won't be able to do anything except through the senior leader.
Authoritarians view all of life as a zero-sum game. The results of this aren't always intuitive. We take this to mean that there are winners and losers, but you have to take it a step further. If authoritarians don't know how to "win" in a certain situation, they focus on ensuring others lose.
In other words, if you win, they lose. To them, any cross-functional project "for the company" will hurt them if it wasn't their idea (they don't have to prove how or why), so they ensure all attempts to work together without their approval are quashed out.
For middle managers, it perpetuates a useful "us vs. them" mentality
.
It's not just senior leaders who benefit from this situation. Authoritarian middle managers favor it too. This is why few department heads seem to want to work cross-functionally, even though most of the benefits to that accrue to their bosses.
Departmental hierarchies provide middle managers with an "us vs. them" mentality. They're always able to blame any failings on another department, either up the chain or down the chain.
This is similar to why authoritarian regimes like North Korea need a perpetual war with "the West." It provides a distraction for their people to rally against rather than the person in charge.
So while a culture of blame and us-vs-them benefits the middle manager's control over their teams, it also benefits their ability to save face with their own bosses. Any failure becomes a game of finger-pointing and a chance to re-establish the pecking order.
Why Functional Heads Though?
So we've established why authoritarian personalities like these strict hierarchies: the setting of rivals against each other, becoming indispensable for any work and providing a steady supply of blame.
But why do we often see mostly functional departments like finance, HR, etc.?
Indeed, some of this is just due to tradition. There's momentum behind that organization. However, even if we entered a world tomorrow where no one had heard of functional departments, they'd slowly reform.
Keeping an eye on powerful people
As people get better and better at a skill (often called a "function" in an organization), they accrue power via their knowledge. They can "do" things that others can't via their mastery of the technology or method.
This scares their authoritarian bosses, who don't like others having power.
Thus, the most "impressive" people in an organization need to be looked after.
I say "impressive" rather than "skilled" because frequently, skilled implies impressive, but impressive doesn't always mean skill.
To authoritarians, looks and signaling are all that matter. That means people who are good at faking skills will also tend to get "looked after."
Ironically, the easiest way to look after an impressive subordinate is to promote that person.
Rewards for loyalty, quid pro quo
First, this is a test to see if the subordinate can "play ball," so to speak. Will they respond to rewards that the boss can dish out? Can they be controlled? If so, their knowledge power becomes the boss's power.
Additionally, a promotion means that the individual tends to work with the boss more, allowing the boss to keep a better eye on that person.
Make the talent identify with the leadership.
The other benefit of promotion is a similar use of the "us vs. them" mentality. By pulling an "impressive" individual contributor into the management fold, that person is more likely to identify with management rather than the individual contributors they sprang from.
You can see this for yourself - write down your current team members.
Did you list your boss? Most people don't. Being in management tends to make someone "other." And while many bosses might list their reports as parts of their team, they aren't welcome there and will often meet with other managers as much as (if not more than) their teams.
Again, this prevents "uprisings" from the individual contributors by ensuring the top individual contributors side with management over anything else.
This All Springs from Theory X Management
Theory X management is a mindset that says people only respond to rewards and punishments and are intrinsically lazy.
Employees will only do work if it is in their self-interest to do so - because they're being incentivized to do it, or they will be punished if they don't do it. Or so the theory goes.
Whenever a Theory X manager runs into someone who has an intrinsic motivation to work on a project, it doesn't compute for them. They assume that the project's goal, rather than just providing a sense of mastery or purpose for the individual, must be in that individual's self-interest. Since the manager usually can't figure out how; they assume it must be some ploy to get ahead and will hurt the manager.
Thus, the authoritarian manager shuts down all work they don't understand as benefiting them directly. Due to their zero-sum game assumptions, all work that isn't in their interest must be against their interest.
Authoritarian People are Out For Themselves
This last bit about the authoritarian personality type explains why despite agile being more successful in the marketplace, you see gated development tend to grow and stick.
Authoritarian managers don't want to maximize profit!
Profit is a good shared by many stakeholders within the organization: shareholders, employees, and others. Authoritarian managers aren't interested in that, other than the profit that flows to them.
Instead, authoritarians want to maximize personal gain. And the overall benefits of cross-functional work and high trust environments are personally far lower than strict functional hierarchies, even after the additional productivity is factored in.
Conclusion
Collaboration, cross-functional work, and other agile processes and values effectively produce results.
These results often aren't valued by authoritarian people who don't see how this additional value benefits them.
In contrast, strict functional hierarchies allow authoritarian managers to easily control and pit their people against each other and reduce the risks of rivals usurping them.
All other things being equal, authoritarian managers will dominate the top levels of your organization over time since they're the only ones trying. Most organizations follow this slow and steady decay into factional departments, blaming each other for what went wrong.
What are you supposed to do about it? That's a big question and one to which there aren't easy answers.
If nothing else, try and take away from this that, to have a good culture:
Start at the beginning
Start at the top
Understand the values you're trying to defend (keep authoritarians out of power)
If you don't have a good culture:
Understand what you're up against
Manage your expectations about what can be done
Realize early on that your 'rivals' are probably a lot more powerful than you
Understand that organizations are dysfunctional on purpose
Was this hot take atomic or merely mango habanero? Let us know! Or...
Click "Subscribe to the Soapbox" below for more!
.png)