Karmona's Pragmatic Blog

Don't get overconfident… Tiny minds also think alike

Karmona's Pragmatic Blog

The GeoSpatial Cloud

November 1st, 2010 by Moti Karmona | מוטי קרמונה · 3 Comments

In continue to my geo distance post, I have decided to post something on the simplest cloud architecture (RDS) for location based services (LBS)

IMHO, (to cut a long story short :) Amazon RDS is more than enough for most geo location applications.

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easy to set up, operate, and scale a relational database in the cloud.

Amazon RDS gives you access to the full capabilities of a familiar MySQL database. This means the code, applications, and tools you already use today with your existing MySQL databases work seamlessly with Amazon RDS.

Two preliminary steps:
* Start a small DB instance with the kind help of AWS management console (image above)
* Use the RDS instance as if it is your “disruptive”  MySQL instance e.g. manage it using MySQL Workbench 5.2.29

“Flirting” with MySQL spatial capabilities (which seems to be “fully” supported by AWS RDS)

* Create the (MyISAM) Table with spatial index

CREATE TABLE `locations` (
`lat` decimal(10,6) DEFAULT NULL,
`long` decimal(10,6) DEFAULT NULL,
`loc` point NOT NULL,
`name` varchar(45) DEFAULT NULL,
SPATIAL KEY `loc` (`loc`)

* Insert few values to populate your table

INSERT INTO locations VALUES(1,40.748433, -73.985655, GeomFromText(‘POINT(40.748433 -73.985655)’), ‘The Empire State Building‘);
INSERT INTO locations VALUES(2, 40.689166, -74.044444, GeomFromText(‘POINT(40.689166 -74.044444)’), ‘The Statue of Liberty’);
INSERT INTO locations VALUES(3, 40.758611, -73.979166, GeomFromText(‘POINT(40.758611 -73.979166)’), ‘Rockefeller Center’);
INSERT INTO locations VALUES(4, 40.757266, -73.985838, GeomFromText(‘POINT(40.757266 -73.985838)’), ‘Times Square’);
INSERT INTO locations VALUES(5, 40.7527, -73.9818, GeomFromText(‘POINT(40.7527 -73.9818)’), ‘New York Public Library’);

* Execute a simple test drive query which returns all the locations and their distance from ‘The Empire State Building’ + Comparing two distance calculation methods (1) MySQL euclidean calculation (2) Haversine calculation (results below)

SELECT locations.name,
GLength(LineStringFromWKB(LineString(locations.loc, GeomFromText(‘POINT(40.748433 -73.985655)’))))*100
AS euclidean,
6378.1 * 2 * ASIN(SQRT(
POWER(SIN(RADIANS(40.748433 – locations.lat) / 2),2) +
COS(RADIANS(40.748433)) * COS(RADIANS(locations.lat) )
* POWER(SIN((RADIANS(-73.985655 – locations.long)) /2), 2) )) AS haversine
FROM locations
— HAVING euclidean < 1
— ORDER BY euclidean ASC LIMIT 10;

Two Surprises/Issues

(1) SRID (Spatial Reference Identifier) support in MySQL is a disgrace – In MySQL, the SRID value is just an integer associated with the geometry value. All calculations are done assuming Euclidean (planar) geometry.

Possible Workarounds: (1) Euclidean calculation can be enough (2) Use Haversine function if you need the accuracy

(2) MySQL ‘Where’ clause can’t use column aliases for filtering

Possible Workarounds: (1) Use ‘Having’ clause (see above) (2) Use the explicit function or field and not the alias

*** *** *** *** *** *** *** *** *** *** *** ***

Related/Interesting reference – Geo Distance Search with MySQL (Presentation | 2008)

Important note: To help new AWS customers get started in the cloud, AWS is introducing a new free usage tier. Beginning November 1, new AWScustomers will be able to run a free Amazon EC2 Micro Instance for a year, while also leveraging a new free usage tier for Amazon S3, Amazon Elastic Block Store, Amazon Elastic Load Balancing, and AWSdata transfer – Very Exciting Times!!!

Disturbing unrelated fact: Starting in 1931, every graduate of the Japanese Naval Academy was asked: “How would you carry out a surprise attack on Pearl Harbor?”

→ 3 CommentsTags: Cloud · Development · Disruptive Technology · Geo

Random Rumbling on Technology Triggers

August 10th, 2009 by Moti Karmona | מוטי קרמונה · No Comments

Monopoly Go to Jail“In the future everything will be augmented reality!”

I might be getting a little too old, visionless, pragmatic or pessimistic for this but I find it very hard to travel to the promised lala land, Gartner’s calls “peak of inflated expectations”.

When I encounter a new “Technology Triggers”, I skip right to the “Trough of Disillusionment” without really passing through the promising “peak”…

e.g. Hype Cycle for Emerging Technologies | Gartner, 2009 – Am I missing something here?

Hype Cycle for Emerging Technologies | Gartner, 2009

p.s. I do think Technology Triggers are very good for SEO and I will update you if it will work… ;)

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Human Augmentation, 3D Flat Panel Displays, Quantum Computing, Context Delivery Architecture, Video Search, Behavioral Economics, Mobile Robots, Surface Computers, Augmented Re4ality, 3D Printing, Internet TV, Wireless Power, Cloud computing, E-Book Readers, Social Software Suites, Micorblogging, Green IT, Video Telepresence, Mesh Networks, Online Video, Home Health Monitoring, Public Virtual Worlds, RFID, Social Network Analysis, Web 2.0, Idea Management, Tablet PC, Wikis, Corporate Blogging, SOA, Location Aware Applications, Speech Recognition etc.

→ No CommentsTags: Conspiracy · Disruptive Technology

Backup your Life with Amazon S3

February 1st, 2009 by Moti Karmona | מוטי קרמונה · 10 Comments

Head in the CloudsS3 (Simple Storage Device ) Overview

“Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.” (http://aws.amazon.com/s3/)

What can I do with this S3 thingy?

Many things… but I will focus on a pragmatic and more common use case – You can use S3 as the ultimate network drive to share your music collection, backup your documents or to store your blog images (a.k.a. CDN for the masses) etc.

Is it free???

Almost…   $0.15 per Giga (+ you can estimate your monthly bill using the AWS Monthly Calculator)

Amazon S3 Pricing

How do I start?

To use Amazon S3 service, you’ll simply need to open an Amazon account and register to the S3.

Interesting tools to simplify your S3 experience:

  • Amazon S3 Firefox Organizer – Simple Firefox add-on that provides an FTP like interface (Windows Explorer) to upload and manage files on S3 – “S3Fox Organizer helps you organize/manage/store your files on Amazon S3. It is easy to install and use as it is integrated into the browser…”
  • DropBox – New Amazon-S3-backed-storage service (thanks to Shlomo for the introduction) – Very simple to use and with 2GB storage limit on the default (free) account and paid upgrade to 50GB of space for $9.99 / month which is not that much above the $7.5 they need to pay to Amazon (before optimizations ;)

P.S. If you liked DropBox you might also like many others

→ 10 CommentsTags: Disruptive Technology · Simplicity · Tools

Head in the Clouds

January 29th, 2009 by Moti Karmona | מוטי קרמונה · 2 Comments

La Condition Humaine | Rene Magritte, 1933It seems like everywhere I go these days, people are talking about cloud computing… Would it be accurate to say that almost everyone are “fairly optimistic” regarding Cloud Computing?

For your convenient, I have collected few cloud-buzz-quotes to  spice your cloud computing elevator pitch:

  • According to Gartner: “Cloud computing heralds an evolution of business that is no less influential than e-business”
  • IDC on Cloud Computing: “This is about the IT industry’s new model for the next 20 years“, Vernon Turner, head of enterprise infrastructure, consumer and telecoms research.

Cloud Computing The Latest Evolution of Hosting | Forrester Research

  • Merrill Lynch estimates that by 2012, the annual global market for cloud computing will surge to $95 billion and that 12% of the worldwide software market would go to the cloud in that period.
  • In January of 2008 Amazon announced that the Amazon Web Services now consume more bandwidth than do the entire global network of Amazon.com retail sites

Amazon Web Services Bandwidth

  • “… the new computing cloud age” Eric Schmidt (April, 2008)

Do you happen to have more???

→ 2 CommentsTags: Cloud · Conspiracy · Disruptive Technology

Social Search Model

January 9th, 2009 by Moti Karmona | מוטי קרמונה · 5 Comments

Mechanical TurkTwo months ago, Brynn M. Evans and Ed H. Chi have published a very interesting article – Towards a Model of Understanding Social Search.

They have ran a small survey using Amazon Mechanical Turk (which is a pretty cool concept for itself) asking ~150 users to describe their search experience.

IMHO, their data analysis resulted in a very intriguing social search model.

“As we outlined through the model, social inputs may help users throughout the search process. Before searching, social interactions may help establish the requirements for the actual search task. During search, especially for self-motivated informational searches, users may talk to others for advice, feedback, and brainstorming to improve their search schema and query keyword selections. After search, users may still wish to engage with others to collect additional feedback or to share knowledge gained during the search.”
(Brynn M. Evans and Ed H. Chi – Towards a Model of Understanding Social Search | 2008)

Social Search Model

P.S. It seems like we are on the right direction… :)


Quick (a.k.a. too late) Y.A.S.B.P. (a.k.a. Yet Another Shameless Blog Plagiarism) update: I had a strange (yet familiar) feeling that I have read about this research somewhere before but I didn’t remember where so after I submitted this post, I have found that Ofer had picked it up way before me and as always, wrote a much better post about it…

→ 5 CommentsTags: Disruptive Technology · Search