Recently, I have encountered an interesting paper (2006) about Chubby – Google’s (Paxos based) distributed lock service.
I was especially amazed by the observations made on the Google engineering capabilities and mindset inside a “formal” research publication.

Although one can easily get into a cynical state of mind reading this paper… I feel that this “pragmatic view” which combines a deep architectural and algorithmic know-how with keen understanding of the social factor in software development is exactly the key to create legendary software.

Anyway, very well written – highly recommended reading…

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***

“Our developers sometimes do not plan for high availability in the way one would wish. Often their systems start as prototypes with little load and loose availability guarantees; invariably the code has not been specially structured for use with a consensus protocol. As the service matures and gains clients, availability becomes more important; replication and primary election are then added to an existing design.”

“Developers are often unable to predict how their services will be used in the future, and how use will grow.  A module written by one team may be reused a year later by another team with disastrous results … Other developers may be less aware of the cost of an RPC.”

“Despite attempts at education, our developers regularly write loops that retry indefinitely when a file is not present, or poll a file by opening it and closing it repeatedly when one might expect they would open the file just once.”

“Developers rarely consider availability. We find that our developers rarely think about failure probabilities.“

“Developers also fail to appreciate the difference between a service being up, and that service being available to their applications.“

“Unfortunately, many developers chose to crash their applications on receiving [a failover] event, thus decreasing the availability of their systems substantially”

When “The Moscow Cats Theater” came to New York, the Russian clown Yuri Kuklachev was interviewed:  “the secret of training them is realizing that you can’t force cats to do anything [...] If the cat likes to sit you can’t force her to do anything else [...] Each cat likes to do her own trick [...] Maruska is the only one who does the handstand. I find the cat and see what they like to do and use that in the show [...] I have a cat now that loves to be in the water…”

– REUTERS, 2006

__________________________________________

Personally, I think that managing engineers is much more complicated than herding cats (although I didn’t have the twisted pleasure to herd a cat yet)

When you go out of your way to hire the best people around than soon enough you will find yourself herding a superior, class A, hyper-developed mutant Ligers* who are much more knowledgeable than the herder (a.k.a. you)

In this environment you have to learn to simply trust your people (although this is not simple at all :), mark the vision, let them loose and only help to get rid of the stones in their way (this concept was best described as the Open Kimono** policy in Peopleware)

Well…. Managing the Delver Engineers is like Herding Legendary Ligers and you need to make a superior effort to see what these ligers “likes to do” and run fast enough to set the Vision and move the rocks out of the way.

__________________________________________

* The Liger, is a (huge) hybrid cross between a male lion and a female tiger

** Open Kimono Attitude: You take no steps to defend yourself from the people you have put in positions of trust.

By the way, The best answer I found on the origin of the term “Herding Cats” was in Google Answers

1^st Warning: Chaotic post below

Software project are chaotic system and are highly sensitive to their initial conditions (a.k.a. the butterfly effect) and dynamics (e.g. wrong design, vague requirements, team professionalism etc.).

To master (/control) a software project you must be able to breathe (/smoke ;-) the project – inhale the chaotic butterfly movements around you and exhale with the needed adjustments or you will be crushed on the nearest project failure shore with zillions of butterfly excuses.

2^nd Warning: Smoking software project is bad for you health

After a decade of software projects smoking I find myself easily doing a background-surfing on the chaotic edges of my projects like I drive my car in the same daily well known route back from work but since I am part of the same chaotic system I am trying to control, I know that my background-surfing is like forgetting my own butterfly wings.

Software project smoking isn’t a social event and can’t be easily shared but it is also one of the key factors in projects surfing – If you will not be able to share your surf experience with your team, your own butterfly wings will bring the next tsunami.

3^rd Warning: Don’t practice management if you don’t like the butterflies

2019-1972=47 (!!!)

Someone wise once gave this as a metaphorical example for a common engineers disorder, he called experience-anxiety-disorder, claiming that NASA stopped sending manned missions to the moon since they now know much more about the complexity and risk with doing this.

During the early seventies, it was a nice, naive working implementation but when NASA engineers started thinking about the next release they have built a five-decades-project-plan simply because they considered all the technological experience they have gained into large complexity buffers.

The Moral Lesson
Keep-it-simple can-do-approach and don’t over-complicate things with the long-tail-little-details when not needed or the project will take 5 decades to finish.

How do you know you are have an experience-anxiety-disorder?
If someone ask you to add a button to change the database schema and this make you feel a mixture of fear, apprehension, heart palpitations, nausea, chest pain, shortness of breath and headache.

What to do?
Sit down and relax, drink a cup of water and then add the damn button!

Last Long-Tail-Detail
Well… I don’t want to ruin this lovely moral lesson with the long-tail-little-detail but the real facts behind this 47 years gap were politics and money (as always) and not that NASA engineers got a severe experience-anxiety-disorder.

Google Trends (a.k.a. my experiment – part IV – Almost forgot…)

• Betty Casey
  
• Most Haunted Life
  
• Piercing & Tattoos
  
• Madonna
  
• World War III (…)
  

…Then came Scrum by “Natural Selection”:

“It is not the strongest of the species that survive, nor the most intelligent, but the ones most responsive to change” – Charles Darwin’s Origin of Species (1859)

…So I have evolved by Natural Selection to Agile and I can’t really go back to over-planned fantasy Gantt charts that try to capture every feature in advance and predict we will finish the project exactly in 666 days …

Why “Agile”?
For using all the right buzzwords e.g. Flexibility; Transparency; Short-term predictability; Long term vision ;-)

Why “Scrum“?
Scrum provides the mechanism for making the people and process problems apparent so they can be solved – It encompasses almost any good engineering technique; very simple, not overly prescriptive and relatively small set of interrelated practices and rules which can be learned quickly and is able to produce productivity gains almost immediately.

Why not???
The main reason as I see it now, is that it is extremely simple but very hard to implement successfully* – Mainly because short iteration cycles, rapid changes and transparency brings project management headache and programmer life to extreme optimization while traditional development processes (e.g. waterfall) give you the misleading euphoria** for very long time-frames (e.g. ~666 days in the above example ;-) inside the traditional project lifecycle

e.g. Transparency forces accountability, responsibility, prioritization discussions, trade-offs, and often scope reduction. Scrum requires that managers behave differently than in the past. Instead of reviewing status reports, managers should attend Sprint reviews and retrospectives. Instead of waiting for team members to prepare and present updates, management should go to the project room and see the project’s task board and burn down chart.

Scrum isn’t a silver bullet* but a simple yet powerful encapsulation of Peopleware mindset, project management patterns and development best practices which can put you on a good starting point when you face the software challenge…

—————————-
* Friendly reminder: no silver bullet in successful software management
** Although I did see few cases where a mixture of skilled project management & legendary engineers have managed to bypass that inherent misleading euphoria while allegedly practicing traditional development process but my claim is that if you look very closely they were actually practicing 90% scrum without even knowing it…

I think that human capital is the silver bullet* for successful software projects – productivity, personalities, teamwork and group dynamics will make or break a project.

Picking the right people is maybe the most important managerial task so on your next interviews please remember** that knowledge can be easily acquired but personality is there to stay.

Don’t spend 90% of your interview time on knowledge when personality (and potential) is the real key for successful recruitment.

——————————————————–
* Although “Peopleware” have a full chapter on how there is no silver bullet… but I partially agree since I never said it will be easy to get to the human capital silver bullet…
** Also remember: Somewhere today a project is failing… and I can personally guarantee that people were somehow involved in its failure!