Karmona Pragmatic Blog

In continue to my geo distance post, I have decided to post something on the simplest cloud architecture (RDS) for location based services (LBS)

IMHO, (to cut a long story short :) Amazon RDS is more than enough for most geo location applications.

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easy to set up, operate, and scale a relational database in the cloud.

Amazon RDS gives you access to the full capabilities of a familiar MySQL database. This means the code, applications, and tools you already use today with your existing MySQL databases work seamlessly with Amazon RDS.

Two preliminary steps:
* Start a small DB instance with the kind help of AWS management console (image above)
* Use the RDS instance as if it is your “disruptive”  MySQL instance e.g. manage it using MySQL Workbench 5.2.29

“Flirting” with MySQL spatial capabilities (which seems to be “fully” supported by AWS RDS)

* Create the (MyISAM) Table with spatial index

CREATE TABLE `locations` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`lat` decimal(10,6) DEFAULT NULL,
`long` decimal(10,6) DEFAULT NULL,
`loc` point NOT NULL,
`name` varchar(45) DEFAULT NULL,
PRIMARY KEY (`id`),
SPATIAL KEY `loc` (`loc`)
) ENGINE=MyISAM AUTO_INCREMENT=6 DEFAULT CHARSET=utf8

* Insert few values to populate your table

INSERT INTO locations VALUES(1,40.748433, -73.985655, GeomFromText(‘POINT(40.748433 -73.985655)’), ‘The Empire State Building‘);
INSERT INTO locations VALUES(2, 40.689166, -74.044444, GeomFromText(‘POINT(40.689166 -74.044444)’), ‘The Statue of Liberty’);
INSERT INTO locations VALUES(3, 40.758611, -73.979166, GeomFromText(‘POINT(40.758611 -73.979166)’), ‘Rockefeller Center’);
INSERT INTO locations VALUES(4, 40.757266, -73.985838, GeomFromText(‘POINT(40.757266 -73.985838)’), ‘Times Square’);
INSERT INTO locations VALUES(5, 40.7527, -73.9818, GeomFromText(‘POINT(40.7527 -73.9818)’), ‘New York Public Library’);

* Execute a simple test drive query which returns all the locations and their distance from ‘The Empire State Building’ + Comparing two distance calculation methods (1) MySQL euclidean calculation (2) Haversine calculation (results below)

SELECT locations.name,
astext(locations.loc),
GLength(LineStringFromWKB(LineString(locations.loc, GeomFromText(‘POINT(40.748433 -73.985655)’))))*100
AS euclidean,
6378.1 * 2 * ASIN(SQRT(
POWER(SIN(RADIANS(40.748433 – locations.lat) / 2),2) +
COS(RADIANS(40.748433)) * COS(RADIANS(locations.lat) )
* POWER(SIN((RADIANS(-73.985655 – locations.long)) /2), 2) )) AS haversine
FROM locations
— HAVING euclidean < 1
– ORDER BY euclidean ASC LIMIT 10;

Two Surprises/Issues

(1) SRID (Spatial Reference Identifier) support in MySQL is a disgrace – In MySQL, the SRID value is just an integer associated with the geometry value. All calculations are done assuming Euclidean (planar) geometry.

Possible Workarounds: (1) Euclidean calculation can be enough (2) Use Haversine function if you need the accuracy

(2) MySQL ‘Where’ clause can’t use column aliases for filtering

Possible Workarounds: (1) Use ‘Having’ clause (see above) (2) Use the explicit function or field and not the alias

*** *** *** *** *** *** *** *** *** *** *** ***

Related/Interesting reference - Geo Distance Search with MySQL (Presentation | 2008)

Important note: To help new AWS customers get started in the cloud, AWS is introducing a new free usage tier. Beginning November 1, new AWScustomers will be able to run a free Amazon EC2 Micro Instance for a year, while also leveraging a new free usage tier for Amazon S3, Amazon Elastic Block Store, Amazon Elastic Load Balancing, and AWSdata transfer – Very Exciting Times!!!

Disturbing unrelated fact: Starting in 1931, every graduate of the Japanese Naval Academy was asked: “How would you carry out a surprise attack on Pearl Harbor?”

It started like yet-another-weekend-experiment but once you start a weekend experiment you never know when or how it will end… ;)

I was very curios to take AWS for a quick test drive so I lost six hours of a precious beauty sleep and compiled this blog-post-capsule for future generations.

The Quest

* Run an “Hello World!!!1” Servlet on EC2 (less than $0.01 per hour)
* Create a local development environment using Eclipse + AWS Toolkit for Eclipse (seems like a really interesting toolchain)

Preliminary Steps

Issues with AWS Toolkit defaults

Plan B – Create a custom EC2 AMI with Tomcat 6.something and JDK 1.6

* Launch an EC2 instance using Amazon’s ami-84db39ed AMI.  (basic Fedora 8 image)
* Use Putty connect to your instance

* Install Java on EC2 Instance

* Download JDK (“Linux RPM in self-extracting JDK file”)

mkdir /usr/local/java

cd /usr/local/java

curl http://download.java.net/jdk6/6u23/promoted/b01/binaries/jdk-6u23-ea-bin-b01-linux-i586-30_aug_2010-rpm.bin > jdk1.6.0.23-rpm.bin

chmod 755 jdk1.6.0.23-rpm.bin # Change the permission of the file

./jdk1.6.0.23-rpm.bin #Install

updatedb; locate javac | grep bin  # this step merely serves to verify the installation

/usr/sbin/alternatives –install /usr/bin/java java /usr/java/jdk1.6.0_23/bin/java 100

/usr/sbin/alternatives –install /usr/bin/jar jar /usr/java/jdk1.6.0_23/bin/jar 100

/usr/sbin/alternatives –install /usr/bin/javac javac /usr/java/jdk1.6.0_23/bin/javac 100

/usr/sbin/alternatives –config java # Change the default JVM from gcj to Sun’s version (if needed)

* Install Tomcat on EC2 Instance

mkdir /usr/tomcat

cd /usr/tomcat

curl http://apache.spd.co.il//tomcat/tomcat-6/v6.0.29/bin/apache-tomcat-6.0.29.tar.gz > apache-tomcat-6.0.29.tar.gz

tar zxvf apache-tomcat-6.0.29.tar.gz

cd apache-tomcat-6.0.29

bin/startup.sh  # Start Tomcat

#!/bin/sh

# Tomcat init script for Linux.

#

# chkconfig: 2345 96 14

# description: The Apache Tomcat servlet/JSP container.

JAVA_HOME=/usr/java/jdk1.6.0_23

CATALINA_HOME=/usr/tomcat/apache-tomcat-6.0.29

export JAVA_HOME CATALINA_HOME

exec $CATALINA_HOME/bin/catalina.sh $*

chmod 755 /etc/rc.d/init.d/tomcat

chkconfig –level 2345 tomcat on

Are we there yet?

vi ~/.bash_profile

export JAVA_HOME=/usr/java/jdk1.6.0_23

export CATALINA_HOME=/usr/tomcat/apache-tomcat-6.0.29

ln -s /usr/java/jdk1.6.0_23/ /env/jdk

ln -s /usr/tomcat/apache-tomcat-6.0.29/ /env/tomcat

What next?

protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {

response.getWriter().println(“Hello World!!!1″);

}

That’s it – I hope this will help, it does take approx. 1 hour so if you know some other way to make it work, please don’t be shy and comment.

Additional references I used to make it this far:

Free VI Tip for Dummies

“The further a project progressed, the more accurate the estimates for the remaining effort and time became”
(Barry Boehm, “Software Engineering Economics“, 1981)
 
NASA also came to the same conclusion that in the beginning of the project life cycle (i.e. before gathering of requirements) estimations have in general an uncertainty of factor 4. This means that the actual duration can be 4 times or 1/4th of the first estimations…
 
I felt very free to add my own interpretation of the different point-of-views with cool pastel colors as a sneak-peak cool-beta-preview of my next post.

Make sense?
 

I have found this “old” movie we made back in 2002 during my work at Mercury to “educate” new employees on our code review procedure.

p.s. Raz Yalov was nominated for best supporting actor that year ;)

Enjoy…

************************************************************************

I have also copy-pasted the presentation transcript below…

• The Golden Rules of Code Review
  • Do a self-review first
  • Stick to the requirements
  • If you need to explain it, document it!
  • It’s about the code – not about you!
  • Unit testing
• Do a self-review first
  • Have you checked all extreme cases?
  • Is your solution KISS?
  • Is your code self-explanatory?
  • Check it as if it was written by someone else
  • Ask yourself the questions you think you are about to be asked…
• Keep It Simple Stupid…
• Stick to the requirements
  • Make sure all aspects of the requirements are filled
  • Make sure you have tested both trivial and extreme cases
  • Document all thoughts and limitations in the code
  • Reference the requirements documents from your code
  • I18N – make sure the code supports multilingual capability where needed
• Documentation
  • Document your files and your functions
  • Give informative comments
  • Explain the logic behind your decisions
  • If you did something that is not logical or intuitive, explain why
  • If you learned historical info while learning the code, document it (be creative :)
  • Follow a standard documentation format
  • When possible, use known design patterns and document the design pattern you chose
• Unit Testing
  • Make sure your unit test is part of the change
  • If you can’t automate the test, check-in a manual procedure to run it
  • Make sure all aspects of the requirements are covered by the tests
  • Document all the tests you wanted to do but couldn’t
  • Test the I18N behavior of your code
• The Code Review Life Cycle
  • “Self-review” your code
  • Schedule a code review
  • Schedule enough time for both people involved
  • When possible, schedule for a time during the morning
  • Split the review into a few sessions, if needed
  • Switch your mind to “listener” mode
  • Write down all comments
  • After the code review is done, apply the necessary fixes and schedule a second review if needed
  • Check-in your changes
• Code Review Tips
  • When you program, remember the last code review
  • If you disagree with the reviewer, invite a 3rd person to review and decide
  • In special cases, invite special guests to review your code
  • When change is small/simple/intuitive, do an “e-review” by mailing your change to the reviewer(s)
  • When self-reviewing, fix mistakes on the spot… :)

This is my recommended check-list for high level design review.

***********************************************************************
*** keep it simple and make sure it answers all the requirements ***
***********************************************************************

• Reverify your Requirements
  • Functional specification, use cases and requirements are clear and publicly documented
  • Technical Requirements
  • Performance requirements
  • Security requirements
  • Resources (CPU, Memory, Storage, IO) consumptions limits
  • Audit requirements
  • Out of scope – What does the component NOT do? What NOT support?
  • Future extendibility of the component (options for future extension, generic features etc.)
• High Level Design
  • Main components involved
  • Main Data flows
  • Components and sub-systems and how they relate to the component
  • Which sub-systems does it communicate with (e.g. Relation Façade)
  • Communication mechanisms (e.g. HTTP, WCF)
  • Which subsystems it is dependent on (e.g. Database, Gigaspaces) – What are the requirements?
  • Which sub-systems depend on it (e.g. App.) – What is the expectation?
• Architectural Strategy
  • High availability and load balancing
  • Error detection, fault and recovery
  • Logging and statistics gathering
  • Capacity limitation, planning & resource management:
    • Memory consumption and management policies
    • CPU usage management
    • IO requirements
    • Storage requirement
• Technical Assumptions
  • Limitations
  • Compromises
    • What does this design compromise? (Security, high availability, capacity, performance, quality…)
    • What are the engineering tradeoffs of this design, and why was the current design chosen?
  • Risks and weak points
• Operation
  • Backward compatibility
  • Logging and Monitoring
  • Administration issues
  • Deployment issues:
    • How to deploy (e.g. can it be part of the regular release package?)
    • Required downtime?
    • Deployment risks?
    • Rollback capability
• Testing strategy
  • What to focus on (80/20)
  • Functional test plan review
  • Deployment, environments, and setups
  • Fault and recovery
  • Load and capacity planning
• Execution plan (phases, timeline, milestones, critical path, dependencies etc.)

Please comment if you think I  forgot something…