Karmona Pragmatic Blog

Pragmatic Software Management, Internet Trends, Life and more…

Karmona Pragmatic Blog

The GeoSpatial Cloud

November 1st, 2010 by Moti Karmona | מוטי קרמונה · 3 Comments

In continue to my geo distance post, I have decided to post something on the simplest cloud architecture (RDS) for location based services (LBS)

IMHO, (to cut a long story short :) Amazon RDS is more than enough for most geo location applications.

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easy to set up, operate, and scale a relational database in the cloud.

Amazon RDS gives you access to the full capabilities of a familiar MySQL database. This means the code, applications, and tools you already use today with your existing MySQL databases work seamlessly with Amazon RDS.

Two preliminary steps:
* Start a small DB instance with the kind help of AWS management console (image above)
* Use the RDS instance as if it is your “disruptive”  MySQL instance e.g. manage it using MySQL Workbench 5.2.29

“Flirting” with MySQL spatial capabilities (which seems to be “fully” supported by AWS RDS)

* Create the (MyISAM) Table with spatial index

CREATE TABLE `locations` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`lat` decimal(10,6) DEFAULT NULL,
`long` decimal(10,6) DEFAULT NULL,
`loc` point NOT NULL,
`name` varchar(45) DEFAULT NULL,
PRIMARY KEY (`id`),
SPATIAL KEY `loc` (`loc`)
) ENGINE=MyISAM AUTO_INCREMENT=6 DEFAULT CHARSET=utf8

* Insert few values to populate your table

INSERT INTO locations VALUES(1,40.748433, -73.985655, GeomFromText(‘POINT(40.748433 -73.985655)’), ‘The Empire State Building‘);
INSERT INTO locations VALUES(2, 40.689166, -74.044444, GeomFromText(‘POINT(40.689166 -74.044444)’), ‘The Statue of Liberty’);
INSERT INTO locations VALUES(3, 40.758611, -73.979166, GeomFromText(‘POINT(40.758611 -73.979166)’), ‘Rockefeller Center’);
INSERT INTO locations VALUES(4, 40.757266, -73.985838, GeomFromText(‘POINT(40.757266 -73.985838)’), ‘Times Square’);
INSERT INTO locations VALUES(5, 40.7527, -73.9818, GeomFromText(‘POINT(40.7527 -73.9818)’), ‘New York Public Library’);

* Execute a simple test drive query which returns all the locations and their distance from ‘The Empire State Building’ + Comparing two distance calculation methods (1) MySQL euclidean calculation (2) Haversine calculation (results below)

SELECT locations.name,
astext(locations.loc),
GLength(LineStringFromWKB(LineString(locations.loc, GeomFromText(‘POINT(40.748433 -73.985655)’))))*100
AS euclidean,
6378.1 * 2 * ASIN(SQRT(
POWER(SIN(RADIANS(40.748433 – locations.lat) / 2),2) +
COS(RADIANS(40.748433)) * COS(RADIANS(locations.lat) )
* POWER(SIN((RADIANS(-73.985655 – locations.long)) /2), 2) )) AS haversine
FROM locations
— HAVING euclidean < 1
– ORDER BY euclidean ASC LIMIT 10;


Two Surprises/Issues

(1) SRID (Spatial Reference Identifier) support in MySQL is a disgrace – In MySQL, the SRID value is just an integer associated with the geometry value. All calculations are done assuming Euclidean (planar) geometry.

Possible Workarounds: (1) Euclidean calculation can be enough (2) Use Haversine function if you need the accuracy

(2) MySQL ‘Where’ clause can’t use column aliases for filtering

Possible Workarounds: (1) Use ‘Having’ clause (see above) (2) Use the explicit function or field and not the alias

*** *** *** *** *** *** *** *** *** *** *** ***

Related/Interesting reference Geo Distance Search with MySQL (Presentation | 2008)

Important note: To help new AWS customers get started in the cloud, AWS is introducing a new free usage tier. Beginning November 1, new AWScustomers will be able to run a free Amazon EC2 Micro Instance for a year, while also leveraging a new free usage tier for Amazon S3, Amazon Elastic Block Store, Amazon Elastic Load Balancing, and AWSdata transfer – Very Exciting Times!!!

Disturbing unrelated fact: Starting in 1931, every graduate of the Japanese Naval Academy was asked: “How would you carry out a surprise attack on Pearl Harbor?”

→ 3 CommentsTags: Cloud · Development · Disruptive Technology · Geo

Run “Hello World!!!1″ Servlet on EC2 using AWS Toolkit for Eclipse

September 15th, 2010 by Moti Karmona | מוטי קרמונה · 12 Comments

It started like yet-another-weekend-experiment but once you start a weekend experiment you never know when or how it will end… ;)

I was very curios to take AWS for a quick test drive so I lost six hours of a precious beauty sleep and compiled this blog-post-capsule for future generations.

 

 

The Quest

* Run an “Hello World!!!1” Servlet on EC2 (less than $0.01 per hour)
Create a local development environment using EclipseAWS Toolkit for Eclipse (seems like a really interesting toolchain)

Preliminary Steps

Issues with AWS Toolkit defaults

The plan was to use this tutorial but surprisingly enough this did not work out-of-the-box (apparently due to Tomcat/JDK versions on the default AMI the plug-in is using but I didn’t waste time in making sure this is the issue) so I moved to plan B

 

Plan B – Create a custom EC2 AMI with Tomcat 6.something and JDK 1.6

* Launch an EC2 instance using Amazon’s ami-84db39ed AMI.  (basic Fedora 8 image)
* Use Putty connect to your instance

 

* Install Java on EC2 Instance

Download JDK (“Linux RPM in self-extracting JDK file”)

mkdir /usr/local/java
cd /usr/local/java
curl http://download.java.net/jdk6/6u23/promoted/b01/binaries/jdk-6u23-ea-bin-b01-linux-i586-30_aug_2010-rpm.bin > jdk1.6.0.23-rpm.bin
* Install the JDK
chmod 755 jdk1.6.0.23-rpm.bin # Change the permission of the file
./jdk1.6.0.23-rpm.bin #Install
updatedb; locate javac | grep bin  # this step merely serves to verify the installation
/usr/sbin/alternatives –install /usr/bin/java java /usr/java/jdk1.6.0_23/bin/java 100
/usr/sbin/alternatives –install /usr/bin/jar jar /usr/java/jdk1.6.0_23/bin/jar 100
/usr/sbin/alternatives –install /usr/bin/javac javac /usr/java/jdk1.6.0_23/bin/javac 100
/usr/sbin/alternatives –config java # Change the default JVM from gcj to Sun’s version (if needed)

* Install Tomcat on EC2 Instance

* Download Tomcat 6.0
mkdir /usr/tomcat
cd /usr/tomcat
curl http://apache.spd.co.il//tomcat/tomcat-6/v6.0.29/bin/apache-tomcat-6.0.29.tar.gz > apache-tomcat-6.0.29.tar.gz
tar zxvf apache-tomcat-6.0.29.tar.gz
* Start Tomcat and to verify the installation, load the root page from a web browser: http://your_instance_name:8080
cd apache-tomcat-6.0.29
bin/startup.sh  # Start Tomcat
* Configure Tomcat to launch automatically – Create a file “/etc/rc.d/init.d/tomcat” with the following content:
#!/bin/sh
# Tomcat init script for Linux.
#
# chkconfig: 2345 96 14
# description: The Apache Tomcat servlet/JSP container.
JAVA_HOME=/usr/java/jdk1.6.0_23
CATALINA_HOME=/usr/tomcat/apache-tomcat-6.0.29
export JAVA_HOME CATALINA_HOME
exec $CATALINA_HOME/bin/catalina.sh $*
* Set the proper permissions for your init script and enable Tomcat for auto-launch:
chmod 755 /etc/rc.d/init.d/tomcat
chkconfig –level 2345 tomcat on
* Tomcat should now be automatically launched whenever your server restarts.

 

Are we there yet?

It could be but apparently the plug-in was poorly designed to use none-configurable command lines so I needed to add the following “tricks”:
* Set JAVA_HOME / PATH for your user – Login to your account and change your .bash_profile file
vi ~/.bash_profile
export JAVA_HOME=/usr/java/jdk1.6.0_23
export CATALINA_HOME=/usr/tomcat/apache-tomcat-6.0.29
* Create aliases to your Tomcat and JDK installation (these location are hard-coded in the plug-in)
ln -s /usr/java/jdk1.6.0_23/ /env/jdk
ln -s /usr/tomcat/apache-tomcat-6.0.29/ /env/tomcat
The EC2 instance is ready :)

 

What next?

* Create EBS Image AMI from your instance (it does takes couple of minutes to complete)
* Open your eclipse and start a new AWS project as described in the original link
* Define a new EC2 Server in Eclipse using your new AMI (reminder: the default didn’t work)
* Create your “Hello World!!!1” Servlet
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
response.getWriter().println(“Hello World!!!1″);
}
* Click Run… this will automatically deploy your Servlet and run it on the remote EC2 instance… Wow :)

 

That’s it – I hope this will help, it does take approx. 1 hour so if you know some other way to make it work, please don’t be shy and comment.

Additional references I used to make it this far:

 

Free VI Tip for Dummies

80% of knowing VI is just: ESC ESC ESC, i, Type-Your-Stuff, ESC ESC ESC, :wq!

 

 

→ 12 CommentsTags: Cloud · Development · Simplicity · Tools

Head in the Clouds

January 29th, 2009 by Moti Karmona | מוטי קרמונה · 2 Comments

La Condition Humaine | Rene Magritte, 1933It seems like everywhere I go these days, people are talking about cloud computing… Would it be accurate to say that almost everyone are “fairly optimistic” regarding Cloud Computing?

For your convenient, I have collected few cloud-buzz-quotes to  spice your cloud computing elevator pitch:

  • According to Gartner: “Cloud computing heralds an evolution of business that is no less influential than e-business”
  • IDC on Cloud Computing: “This is about the IT industry’s new model for the next 20 years“, Vernon Turner, head of enterprise infrastructure, consumer and telecoms research.

Cloud Computing The Latest Evolution of Hosting | Forrester Research

  • Merrill Lynch estimates that by 2012, the annual global market for cloud computing will surge to $95 billion and that 12% of the worldwide software market would go to the cloud in that period.
  • In January of 2008 Amazon announced that the Amazon Web Services now consume more bandwidth than do the entire global network of Amazon.com retail sites

Amazon Web Services Bandwidth

  • “… the new computing cloud age” Eric Schmidt (April, 2008)

Do you happen to have more???

→ 2 CommentsTags: Cloud · Conspiracy · Disruptive Technology