Why doesn't top-down agile work?
I've seen a few cases of successful top-down agile implementation. Contributing factors were smaller organization size, technical background of the leadership, lack of rigid constraints, enthusiasm of the individual team members.
I've also seen it turn into "agile waterfall" - when the process of reporting and celebrating velocity numbers becomes more exciting and important than actually getting good work behind those numbers done. Sometimes this results in half-empty iterations spent on chasing down requirements - where no code gets written and nothing can be tested. Other times it results in premature optimization of systems through excessive focus on system architecture.
Contributing factors: larger organization size, rapid growth, lack of quality lieutenants, Peter's Principle - with folks ultimately being promoted to a position of incompetence.
What's the right way to scale agile?
If by scaling we mean some sort of cookie-cutter one-size-fits-all process that can be easily applied throughout the entire organization, then I doubt there is such a thing as the right way to scale agile.
Agile at its heart is somewhat of of an anti-process. We disguise it as a process, but in its essence agile moves a group of people away from rules and regulations and toward being creative about getting the biggest impact possible with the resources they have at hand. And for different people, different groups within department and different departments within an organization those solutions are going to be very different.
Can you share an example from your own company?
The CEO of a company I worked for years ago read a book on Extreme Programming and personally handed a copy to each member of the IT department. Within days we started having stand-up meetings on every corner of the floor. Within a week the entire department was pair-programming and we didn't even have that many programmers on the team.
Everyone learned a whole lot and suddenly we became a DevOps department, long before that term was ever coined. Our customers developed a huge appreciation for the power of technology we placed into their hands.
How do you scale productivities instead of processes?
I know programmers who shudder at the world "agile" because they have grown to to associate it with getting interrupted all day with meaningless questions and with having to attend seventeen cross-functional team scrum meetings per day.
Each individual's contribution needs to have a huge, meaningful impact. Ultimately, each team member needs to be able to go home at the end of the day and be able to say to their spouse - "Honey, I did X, Y and Z today! I'm so excited, because this is going to make my world so much better than it was yesterday!"
You really need a lot of honesty to make this work. And the honesty has to start at the top. Can we honestly stand by the product and service we offer the world or are we just in it to make money? The enthusiasm the leadership feels for their place and their role in the world needs to be contagious for the rest of the firm. And that enthusiasm should empower every team member to break down the old organizational barriers that stand in the way of getting real work done.