Karmona Pragmatic Blog

“The further a project progressed, the more accurate the estimates for the remaining effort and time became”
(Barry Boehm, “Software Engineering Economics“, 1981)
 
NASA also came to the same conclusion that in the beginning of the project life cycle (i.e. before gathering of requirements) estimations have in general an uncertainty of factor 4. This means that the actual duration can be 4 times or 1/4th of the first estimations…
 
I felt very free to add my own interpretation of the different point-of-views with cool pastel colors as a sneak-peak cool-beta-preview of my next post.

Make sense?
 

I have found this “old” movie we made back in 2002 during my work at Mercury to “educate” new employees on our code review procedure.

p.s. Raz Yalov was nominated for best supporting actor that year ;)

Enjoy…

************************************************************************

I have also copy-pasted the presentation transcript below…

• The Golden Rules of Code Review
  • Do a self-review first
  • Stick to the requirements
  • If you need to explain it, document it!
  • It’s about the code – not about you!
  • Unit testing
• Do a self-review first
  • Have you checked all extreme cases?
  • Is your solution KISS?
  • Is your code self-explanatory?
  • Check it as if it was written by someone else
  • Ask yourself the questions you think you are about to be asked…
• Keep It Simple Stupid…
• Stick to the requirements
  • Make sure all aspects of the requirements are filled
  • Make sure you have tested both trivial and extreme cases
  • Document all thoughts and limitations in the code
  • Reference the requirements documents from your code
  • I18N – make sure the code supports multilingual capability where needed
• Documentation
  • Document your files and your functions
  • Give informative comments
  • Explain the logic behind your decisions
  • If you did something that is not logical or intuitive, explain why
  • If you learned historical info while learning the code, document it (be creative :)
  • Follow a standard documentation format
  • When possible, use known design patterns and document the design pattern you chose
• Unit Testing
  • Make sure your unit test is part of the change
  • If you can’t automate the test, check-in a manual procedure to run it
  • Make sure all aspects of the requirements are covered by the tests
  • Document all the tests you wanted to do but couldn’t
  • Test the I18N behavior of your code
• The Code Review Life Cycle
  • “Self-review” your code
  • Schedule a code review
  • Schedule enough time for both people involved
  • When possible, schedule for a time during the morning
  • Split the review into a few sessions, if needed
  • Switch your mind to “listener” mode
  • Write down all comments
  • After the code review is done, apply the necessary fixes and schedule a second review if needed
  • Check-in your changes
• Code Review Tips
  • When you program, remember the last code review
  • If you disagree with the reviewer, invite a 3rd person to review and decide
  • In special cases, invite special guests to review your code
  • When change is small/simple/intuitive, do an “e-review” by mailing your change to the reviewer(s)
  • When self-reviewing, fix mistakes on the spot… :)

This is my recommended check-list for high level design review.

***********************************************************************
*** keep it simple and make sure it answers all the requirements ***
***********************************************************************

• Reverify your Requirements
  • Functional specification, use cases and requirements are clear and publicly documented
  • Technical Requirements
  • Performance requirements
  • Security requirements
  • Resources (CPU, Memory, Storage, IO) consumptions limits
  • Audit requirements
  • Out of scope – What does the component NOT do? What NOT support?
  • Future extendibility of the component (options for future extension, generic features etc.)
• High Level Design
  • Main components involved
  • Main Data flows
  • Components and sub-systems and how they relate to the component
  • Which sub-systems does it communicate with (e.g. Relation Façade)
  • Communication mechanisms (e.g. HTTP, WCF)
  • Which subsystems it is dependent on (e.g. Database, Gigaspaces) – What are the requirements?
  • Which sub-systems depend on it (e.g. App.) – What is the expectation?
• Architectural Strategy
  • High availability and load balancing
  • Error detection, fault and recovery
  • Logging and statistics gathering
  • Capacity limitation, planning & resource management:
    • Memory consumption and management policies
    • CPU usage management
    • IO requirements
    • Storage requirement
• Technical Assumptions
  • Limitations
  • Compromises
    • What does this design compromise? (Security, high availability, capacity, performance, quality…)
    • What are the engineering tradeoffs of this design, and why was the current design chosen?
  • Risks and weak points
• Operation
  • Backward compatibility
  • Logging and Monitoring
  • Administration issues
  • Deployment issues:
    • How to deploy (e.g. can it be part of the regular release package?)
    • Required downtime?
    • Deployment risks?
    • Rollback capability
• Testing strategy
  • What to focus on (80/20)
  • Functional test plan review
  • Deployment, environments, and setups
  • Fault and recovery
  • Load and capacity planning
• Execution plan (phases, timeline, milestones, critical path, dependencies etc.)

Please comment if you think I  forgot something…

Recently, I have encountered an interesting paper (2006) about Chubby – Google’s (Paxos based) distributed lock service.
I was especially amazed by the observations made on the Google engineering capabilities and mindset inside a “formal” research publication.

Although one can easily get into a cynical state of mind reading this paper… I feel that this “pragmatic view” which combines a deep architectural and algorithmic know-how with keen understanding of the social factor in software development is exactly the key to create legendary software.

Anyway, very well written – highly recommended reading…

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***

“Our developers sometimes do not plan for high availability in the way one would wish. Often their systems start as prototypes with little load and loose availability guarantees; invariably the code has not been specially structured for use with a consensus protocol. As the service matures and gains clients, availability becomes more important; replication and primary election are then added to an existing design.”

“Developers are often unable to predict how their services will be used in the future, and how use will grow.  A module written by one team may be reused a year later by another team with disastrous results … Other developers may be less aware of the cost of an RPC.”

“Despite attempts at education, our developers regularly write loops that retry indefinitely when a file is not present, or poll a file by opening it and closing it repeatedly when one might expect they would open the file just once.”

“Developers rarely consider availability. We find that our developers rarely think about failure probabilities.“

“Developers also fail to appreciate the difference between a service being up, and that service being available to their applications.“

“Unfortunately, many developers chose to crash their applications on receiving [a failover] event, thus decreasing the availability of their systems substantially”

“You can always start a weekend experiment but you can never know how it will end”  (Moti Karmona, 2008 ;)

I was very curios and wanted to take the GAE for a quick test drive but the Google documentation have few inaccuracies and isn’t sufficient if it is your first Python encounter so I lost three hours of precious beauty sleep and compiled this blog-post-capsule for future generations.

How to setup your development environment to work with Google-App-Engine 

• Get yourself a GAE Account  
• Install Google App Engine SDK, Python and Eclipse.
• Install the PyDev Eclipse extension
  • Help –> Software Updates –> Available Software –> Add the http://pydev.sourceforge.net/updates site –> Install it –> Restart Eclipse
  • Configure Eclipse to use the Python interpreter by navigating to Window –> Preferences –> PyDev –> Interpreter –> Python –> New and select the location of python.exe on your system
• Create your first project
  • File –> New –> PyDev project and click next –> Name your project and make sure Python 2.5 is selected.  
  • Add necessary App Engine libraries to your project | Right click your Project folder –> Properties –> PyDev – PythonPath (see image below)
    • C:\Program Files\Google\google_appengine
    • C:\Program Files\Google\google_appengine\lib\django
    • C:\Program Files\Google\google_appengine\lib\webob
    • C:\Program Files\Google\google_appengine\lib\yaml\lib

• Create your project files (e.g. GAE examples)
  • app.yaml  
  • Your first python file 
• Test your Local application
  • Update your Run configuration (see image below)
    • Change the Main Module field to C:\Program Files\Google\google appengine\dev_appserver.py
    • On the Arguments tab, type in “${project_loc}/src” 
    • Name this configuration and click apply and click run.
  • Open http://localhost:8080 with your browser and enojoy your stupid app :)

• Upload your code to Google App Engine
  • Open a command prompt and navigate to your application directory.
  • Type appcfg.py –email=yourgmailaccount@gmail.com update src\ 
  • To publish code within Eclipse, open Run –> Run configurations and make a copy of your previous run configuration.  Change the Main Module to appcfg.py, and change the argument tab to –email=yourgmailaccount@gmail.com update ”${project_loc}/src”
  • Your code is now live! Test it by going to http://yourapp.appspot.com

Enjoy :)

P.S. More about the framework in future posts.

Last weekend I was wasting time on my Blog performance and all I got is 10 sec. and this lousy post.

Quick profiling with FireBug and YSlow FireFox extensions have done great wonders with my amazingly slow, not-really-that-interesting, hosted (webhost4life) WordPress blog.

Main action items:

• I have removed the sitemeter performance penalty widget
• I have installed the WordPress Super Cache Plug-in which wasn’t that simple with my webhost4life hosting limitations.
• I have removed couple of images and reduce the blog top image size from 80k to 5k (!!!)

Results:

• Main blog page returns after ~1 sec.
• I am still getting a lousy grade with the YSlow tests (62) but it seems like the YSlow blog post announcement have failed too… (57 :)
• Post pages are still slow (~2-4 sec.) but I know what I have to do the next time more than 1 million readers will complain…

Have fun!

Yahoo have released the Y!OS (Yahoo Open Strategy) 1.0 platform.

This is a cool set of simple APIs that can give you access to everything you ever wanted in Y! but was afraid to ask for…

Yahoo! Social Platform (YSP)
// The Yahoo Social Platform is a set of RESTful APIs for Profiles, Connections, Updates, Contacts and Status.

Yahoo! Query Language (YQL)
// The Yahoo Query Language is a web service that functions much like SQL (see example below)

OAuth Authentication
// OAuth is the authentication and authorization standard Yahoo has decided to use when giving third parties access to Yahoo user data.

Yahoo! Applications Platform (YAP)
// Currently very limited and in a restricted sandbox.

________________________________

Example: How to use YQL APIs to access MyBlogLog profiles?

Simply ask for all the community members of MyBlogLog community with this YQL:

select * from mybloglog.members.find where community_id in (select id from mybloglog.community.find where name=”Karmona Pragmatic Blog”)

And once you have the IDs you can ask for my personal profile by:

select * from mybloglog.member where member_id =”2008070609482910″

Well… together with the existing BOSS API, this set of APIs is a powerful enablers to the Y! development network and I am sure some cool stuff are going to emerge from this innovative move…

Amazing!!!

________________________________

* You can have more YQL experiments using the YQL Console

** Boss Hack Day is coming to Tel-Aviv | November 6, 2008 @ Feature (!!!)

I am very “proud” to introduce the ultimate geek widget: Base 64 Encode / Decode Online Widget

Q. Where can I see this dark magic?

A. Here… :)

Q. How can I add this cool Base64 widget to my blog?

A. Simply copy-paste this little script:
___________________________________________

<script type=”text/javascript” src=”http://blog.karmona.com/base64widget.js” ></script>

___________________________________________

Q. Does this blog widget support ALL blog platforms?

A. Sure… (including dasBlog :)

Please contact me if you have any issues / questions / suggestions,

Have fun!

When “The Moscow Cats Theater” came to New York, the Russian clown Yuri Kuklachev was interviewed:  “the secret of training them is realizing that you can’t force cats to do anything [...] If the cat likes to sit you can’t force her to do anything else [...] Each cat likes to do her own trick [...] Maruska is the only one who does the handstand. I find the cat and see what they like to do and use that in the show [...] I have a cat now that loves to be in the water…”

– REUTERS, 2006

__________________________________________

Personally, I think that managing engineers is much more complicated than herding cats (although I didn’t have the twisted pleasure to herd a cat yet)

When you go out of your way to hire the best people around than soon enough you will find yourself herding a superior, class A, hyper-developed mutant Ligers* who are much more knowledgeable than the herder (a.k.a. you)

In this environment you have to learn to simply trust your people (although this is not simple at all :), mark the vision, let them loose and only help to get rid of the stones in their way (this concept was best described as the Open Kimono** policy in Peopleware)

Well…. Managing the Delver Engineers is like Herding Legendary Ligers and you need to make a superior effort to see what these ligers “likes to do” and run fast enough to set the Vision and move the rocks out of the way.

__________________________________________

* The Liger, is a (huge) hybrid cross between a male lion and a female tiger

** Open Kimono Attitude: You take no steps to defend yourself from the people you have put in positions of trust.

By the way, The best answer I found on the origin of the term “Herding Cats” was in Google Answers

1^st Warning: Chaotic post below

Software project are chaotic system and are highly sensitive to their initial conditions (a.k.a. the butterfly effect) and dynamics (e.g. wrong design, vague requirements, team professionalism etc.).

To master (/control) a software project you must be able to breathe (/smoke ;-) the project – inhale the chaotic butterfly movements around you and exhale with the needed adjustments or you will be crushed on the nearest project failure shore with zillions of butterfly excuses.

2^nd Warning: Smoking software project is bad for you health

After a decade of software projects smoking I find myself easily doing a background-surfing on the chaotic edges of my projects like I drive my car in the same daily well known route back from work but since I am part of the same chaotic system I am trying to control, I know that my background-surfing is like forgetting my own butterfly wings.

Software project smoking isn’t a social event and can’t be easily shared but it is also one of the key factors in projects surfing – If you will not be able to share your surf experience with your team, your own butterfly wings will bring the next tsunami.

3^rd Warning: Don’t practice management if you don’t like the butterflies