Karmona Pragmatic Blog

Don't get overconfident… Tiny minds also think alike

Karmona Pragmatic Blog

The GeoSpatial Cloud

November 1st, 2010 by Moti Karmona | מוטי קרמונה · 3 Comments

In continue to my geo distance post, I have decided to post something on the simplest cloud architecture (RDS) for location based services (LBS)

IMHO, (to cut a long story short :) Amazon RDS is more than enough for most geo location applications.

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easy to set up, operate, and scale a relational database in the cloud.

Amazon RDS gives you access to the full capabilities of a familiar MySQL database. This means the code, applications, and tools you already use today with your existing MySQL databases work seamlessly with Amazon RDS.

Two preliminary steps:
* Start a small DB instance with the kind help of AWS management console (image above)
* Use the RDS instance as if it is your “disruptive”  MySQL instance e.g. manage it using MySQL Workbench 5.2.29

“Flirting” with MySQL spatial capabilities (which seems to be “fully” supported by AWS RDS)

* Create the (MyISAM) Table with spatial index

CREATE TABLE `locations` (
`lat` decimal(10,6) DEFAULT NULL,
`long` decimal(10,6) DEFAULT NULL,
`loc` point NOT NULL,
`name` varchar(45) DEFAULT NULL,
SPATIAL KEY `loc` (`loc`)

* Insert few values to populate your table

INSERT INTO locations VALUES(1,40.748433, -73.985655, GeomFromText(‘POINT(40.748433 -73.985655)’), ‘The Empire State Building‘);
INSERT INTO locations VALUES(2, 40.689166, -74.044444, GeomFromText(‘POINT(40.689166 -74.044444)’), ‘The Statue of Liberty’);
INSERT INTO locations VALUES(3, 40.758611, -73.979166, GeomFromText(‘POINT(40.758611 -73.979166)’), ‘Rockefeller Center’);
INSERT INTO locations VALUES(4, 40.757266, -73.985838, GeomFromText(‘POINT(40.757266 -73.985838)’), ‘Times Square’);
INSERT INTO locations VALUES(5, 40.7527, -73.9818, GeomFromText(‘POINT(40.7527 -73.9818)’), ‘New York Public Library’);

* Execute a simple test drive query which returns all the locations and their distance from ‘The Empire State Building’ + Comparing two distance calculation methods (1) MySQL euclidean calculation (2) Haversine calculation (results below)

SELECT locations.name,
GLength(LineStringFromWKB(LineString(locations.loc, GeomFromText(‘POINT(40.748433 -73.985655)’))))*100
AS euclidean,
6378.1 * 2 * ASIN(SQRT(
POWER(SIN(RADIANS(40.748433 – locations.lat) / 2),2) +
COS(RADIANS(40.748433)) * COS(RADIANS(locations.lat) )
* POWER(SIN((RADIANS(-73.985655 – locations.long)) /2), 2) )) AS haversine
FROM locations
— HAVING euclidean < 1
— ORDER BY euclidean ASC LIMIT 10;

Two Surprises/Issues

(1) SRID (Spatial Reference Identifier) support in MySQL is a disgrace – In MySQL, the SRID value is just an integer associated with the geometry value. All calculations are done assuming Euclidean (planar) geometry.

Possible Workarounds: (1) Euclidean calculation can be enough (2) Use Haversine function if you need the accuracy

(2) MySQL ‘Where’ clause can’t use column aliases for filtering

Possible Workarounds: (1) Use ‘Having’ clause (see above) (2) Use the explicit function or field and not the alias

*** *** *** *** *** *** *** *** *** *** *** ***

Related/Interesting reference – Geo Distance Search with MySQL (Presentation | 2008)

Important note: To help new AWS customers get started in the cloud, AWS is introducing a new free usage tier. Beginning November 1, new AWScustomers will be able to run a free Amazon EC2 Micro Instance for a year, while also leveraging a new free usage tier for Amazon S3, Amazon Elastic Block Store, Amazon Elastic Load Balancing, and AWSdata transfer – Very Exciting Times!!!

Disturbing unrelated fact: Starting in 1931, every graduate of the Japanese Naval Academy was asked: “How would you carry out a surprise attack on Pearl Harbor?”

→ 3 CommentsTags: Cloud · Development · Disruptive Technology · Geo

You Yong Wu Mou (“Having Courage but No Strategies”)

October 20th, 2010 by Moti Karmona | מוטי קרמונה · 3 Comments

I have just finished reading a very interesting book (!!!) “Predictably Irrational” by Dan Ariely and came across a very interesting historic story.

“In 210 BC, a Chinese commander named Xiang Yu led his troops across the Yangtze River to attack the army of the Qin (Ch’in) dynasty. Pausing on the banks of the river for the night, his troops awakened in the morning to find to their horror that their ships were burning. They hurried to their feet to fight off their attackers, but soon discovered that it was Xiang Yu himself who had set their ships on fire, and that he had also ordered all cooking pots crushed.”

“Xiang Yu explained to his troops that without the pots and the ships, they had no other choice but to fight their way to victory or perish. That did not earn Xiang Yu a place on the Chinese army’s list of favorite commanders, but it did have a tremendous focusing effect on his troops (as they grabbed) their lances and bows, they charged ferociously against the enemy and won nine consecutive battles, completely obliterating the main-force units of the Qin dynasty”

Prof. Ariely is making a point about the advantage of making a choice to focus by closing other doors/options/opportunities.

Joshua Baer had an interesting allegory to the startup world in his “Necessity is the mother of Invention” post

“This is similar to when a bootstrapper enters the Valley of Death* and commits to their venture, but before they are making money and cash flow positive. They are forced to figure out how to make it work with what they’ve got. The timeline is not completely in their control.

We’re always tempted to leave ourselves an escape route or path of retreat. And usually that’s a good idea. But sometimes there aren’t enough resources to mount the attack and cover the retreat. In order to be successful sometimes you have to commit the resources to what you believe in because the retreat option isn’t acceptable. Sometimes once you head down a path there is just no turning back, so you might as well commit all of your resources to getting to the end”

Well… this is true but since I am a notorious pessimist and usually like my options open, I have continued reading about this fine gentlemen (a.k.a. Xiang Yu)

I learned that indeed in the beginning of the civil war Xiang Yu was winning but with his rude manners, arrogance and lack of political vision, the tide turned against Xiang Yu and in the end he lost the war to Liu Bang.

In 202 BC, when Xiang Yu and his remaining men had their backs against the river while surrounded by Liu Bang’s troops, a boatman on a raft persuaded Xiang Yu to go with him across the river so he can prepare a comeback.
Xiang Yu said, “When I crossed the River and went west, I took with me 8,000 sons and brothers from east of the Yangtze. Now none of them has returned; how can I face the elders east of the Yangtze?” After declining this offer, Xiang Yu turned around, charged against the Han troops, killed over a hundred men, and finally cut his own throat.
Shortly after his death Liu Bang established the Han Dynasty.

Three concluding facts about Xiang Yu:

  • Xiang is popularly viewed as a leader who possesses great courage but lacks wisdom, and his character is aptly summarized using the Chinese idiom “Yǒu Yǒng Wú Móu” (有勇無謀) – “Having Courage but No Strategies” (or  to be foolhardy or to be more brave than wise or to have reckless courage…)
  • Xiang’s battle tactics were studied by future military leaders while his political blunders served as cautionary tales for future rulers
  • Xiang Yu is also the kind general that raided the Terracotta** tomb less than five years after the death of the First Emperor – Xiang’s army was looting of the tomb and structures holding the Terracotta Army, as well as setting fire to the necropolis and starting a blaze that lasted for three months.

“Yǒu Yǒng Wú Móu” (有勇無謀) – “Having Courage but No Strategies” – Think about it…! ;)

* Valley of Death – A slang phrase to refer to the period of time from when a startup receives an initial capital contribution to when it begins generating revenues.
During the death valley curve, additional financing is usually scarce, leaving the firm vulnerable to cash flow requirements.

** The Terracotta Army or the “Terra Cotta Warriors and Horses“, is a collection of terracotta sculptures depicting the armies of Qin Shi Huang, the First Emperor of China
The figures, dating from 210 BC, vary in height according to their roles, with the tallest being the generals. The figures include warriors, chariots, horses, officials, acrobats, strongmen, and musicians.
Current estimates are that in the three pits containing the Terracotta Army there were over 8,000 soldiers, 130 chariots with 520 horses and 150 cavalry horses, the majority of which are still buried in the pits.
There is also a legend that the terracotta warriors were real soldiers, buried with Emperor Qin so that they could defend him from any dangers in the next life.

*** *** *** *** *** *** *** *** *** *** *** *** *** *** ***

p.s. Prof. Ariely also recommends another role model for door closing – Rhett Butler for his supreme moment of unpredictable rationality with his astonishing elan, “Frankly my dear, I don’t give a damn”

→ 3 CommentsTags: Conspiracy · Leadership · People · Psychology · Strategy

Karmona Labs on Geo Distance

October 9th, 2010 by Moti Karmona | מוטי קרמונה · 6 Comments

Well.. Everyone are talking about Location-Location-Location so this weekend was all about Geographical distance

I will quickly preview few basics (geographic coordinate system, earth radius), introduce and compare four distance calculation models (Pythagorean, Law of Cosines, Haversine, Vincenty), finalize with a pragmatic recommendation (use Law of Cosines! :) and random quote for desert.

Start with the basics…

A Geographic Coordinate System Indicate location using lines of longitude and latitude

Latitude is the angle between the equatorial plane and a line that is normal to the reference ellipsoid, which approximates the shape of Earth to account for flattening of the poles and bulging of the equator.

Longitude is the angle east or west of a reference meridian between the two geographical poles to another meridian that passes through an arbitrary point. All meridians are halves of great circles, and are not parallel. They converge at the north and south poles.

Exempli Gratia
The Status of Liberty is located on 40° 41′ 21″ N , 74° 2′ 40″ W with the following Decimal representative: 40.689167, -74.044444

40.689167 = Degrees + Minutes/60 + Seconds/3600 = 40 + 41/60 + 21/3600 = 40.689167
-74.044444 = -1*(74 + 2/60 + 40/3600) = -74.044444 // the minus is used to represent South & West

What is Earth Radius? (hint: we will need later for the calculation)

To cut a *very* long story short, Google, IUGG and Karmona labs thinks it is 6378.137 (3963.19 miles)

Because the Earth is not perfectly spherical, no single value serves as its natural radius.

Distances from points on the surface to the center range and regardless of calculation model, the radius falls between 6,357 km and 6,378 km

Earth Radius (km) based on different models


Geographic Distance Calculation Models

There are quite few geo distance calculation models but I will focus on the four I found most relevant:
* Pythagorean
* Law of Cosines
* Haversine
* Vincenty

I have done a little excel experiment (downloadable here):

  • I  have compared the distance between “The Empire State Building” and 15 other locations
  • I have used three geo distance calculation models (beside Vincenty)
  • Modeling this into Excel – I had two locations A (latA, longA) and B (latB, LongB) with R as Earth Radius (coordinates are used in their decimal representative)
    • Pythagorean =SQRT((111.2*(latA-latB))^2+(85.2*(LongA-LongB))^2)
    • Law of Cosines =ACOS(SIN(RADIANS(latA))*SIN(RADIANS(latB)) + COS(RADIANS(latA))*COS(RADIANS(latB))*COS(RADIANS(longA-longB)))*R
    • Haversine =2*ASIN(MIN(1, SQRT(SIN(RADIANS(latA-latB)/2)^2 + COS(RADIANS(latA))*COS(RADIANS(latB))*SIN(RADIANS(longA-longB)/2)^2)))*R
  • Additional Notes:
    • The reason I have used 111.2 and 85.2 in the Pythagorean equation is the fact that 1° latitude ≈ 111 km and 1° longitude can vary but the average is ≈ 82.2 km (the right thing to do actually is to choose the exact longitude/km conversion base on the degree)
    • The conversion from the original Geo Location representative to a decimal one was using this excel formula =IF(Degree<0,Degree-Minutes/60-Seconds/3600,Degree+Minutes/60+Seconds/3600)
  • The Results
    • Pythagorean is easy to compute but not that accurate
    • Law of Cosines and Haversine are almost the same
    • See comparison table below…
ModelPythagoreanLaw of CosinesHaversineVincenty
Formulad = sqrt((X2 – X1)^2 + (Y2 – Y1)^2)a = sin(lat1) * sin(lat2)
b = cos(lat1) * cos(lat2) * cos(lon2 – lon1)
c = arccos(a + b)
d = R * c
dlon = lon2 – lon1
dlat = lat2 – lat1
a = sin^2(dlat/2) + cos(lat1) * cos(lat2) * sin^2(dlon/2)
c = 2 * arcsin(min(1,sqrt(a)))
d = R * c
Too long… ;)
AssumptionsFlat Earth… :)Spherical EarthSpherical EarthEllipsoidal Earth
Accuracy (Worst | 1-5 | Best)(1) Estimated distance (good enough for less than 20km)(4) Good!(4) Good! + The Haversine formula is more robust to floating point errors(5) Great! The most accurate…
Computability (Slowest | 1-5 | Fastest)(5) Fastest!(4) 5-6 trig. calls(3) 5 trig. calls + SQRT (+ Floating Point)(1) Requires iteration (+ “The rest”)

Final note:

  • Simple pragmatic recommendation – Use Law of Cosines to calculate geographic distance – It will be suffice in 90% of your usages !
  • Complex pragmatic recommendation – It really depends – Call me…

*** *** *** *** *** *** *** *** *** *** *** *** *** ***

Random Quote“There’s no sense in being precise when you don’t even know what you’re talking about” | John von Neumann

→ 6 CommentsTags: Geo

American-Israeli Cultural Misinterpretation

September 30th, 2010 by Moti Karmona | מוטי קרמונה · 8 Comments

*** Warning: this post might contain cultural bias material ;) ***

As most of my friends and colleagues, I had the pleasure of working with Americans for most of my professional life and this post is only the tip-of-the-iceberg trying to capture one tiny angle of the American-Israeli cultural gap.

We (Israelis) think we know enough about English since we have watched many hours of American TV, studied the language from early age and use it constantly and “fluently” during our adult professional life but we should be aware about our tendency to interpret English phrases literally, which results in amusing/frustrating/interesting/challenging misunderstandings – We don’t see things as they are, we see things as we are.

Executive summary:

  • Israelis consider themselves creative, direct and honest but being perceived as arrogant, stubborn, negative, rude and pushy (did I forget any other negative adjective here? ;)
  • Americans are professionals, positive, polite and showing respect but being perceived as bureaucrats, artificial and square

Few examples:

Original (American) WordingOriginal IntentPossible (Israeli) Misinterpretation
American: “This is a challenge”This is a problem!Israeli: Great!, we are always looking for interesting challenges
American: “I would appreciate if you could get this done by Friday”The deadline is Friday – Make it happen!Israeli: It is nice-to-have to get it done by Friday, but next week is also an option…
American: “You might want to consider…..”There is a problem here – This need to be changedIsraeli: IMHO, there are other options but yours is also good
American: “This is ok but…”This is bad!Israeli: This is OK (Thanks! :)
American: “I do have my concerns”There are severe problem hereIsraeli: This is OK – Need some more discussions but nothing serious…
American: “I’ll make an effort”Just being polite… No real commitment here, it’s optional whether I’ll try harder than usualIsraeli: You can count on me doing everything possible for this

Original (Israeli) WordingOriginal IntentPossible (American) Misinterpretation
Israeli: “I don’t agree”I disagree with your point of view and want to discuss it some more so that we can reach agreement on the best approach (this is almost the default :)American: (Rude) There’s no room for discussion
Israeli: “Your presentation was OK”Your presentation was really goodAmerican: I didn’t like your presentation.
Israeli: “Why don’t you do it another way?”I’m giving you a helpful suggestion because I’m interested in improving the result.American: (Rude) I am insulting your work
Israeli: “I am OK” (direct answer to “How are you?”)I feel great – if I wasn’t, be sure you will be the first to knowAmerican: Nothing is OK
Israeli: “This will not work!”I need you to explain the way it suppose to work since it seems like I am really missing something hereAmerican: (Rude) I am insulting your work again…
Israeli: “I think” (pronounced as sink :)I think…American: Ha???

German Coastguard | “What are you s(th)inking about?”

Do you have more examples to share?

Relevant books:

Very cool link:

*** *** *** *** *** *** *** *** *** *** *** ***

Random Quote: “Don’t get overconfident. Tiny minds also think alike” (Pasha Bitz, May 2010)

→ 8 CommentsTags: People

The Pointy-Haired Boss Alter Ego

September 28th, 2010 by Moti Karmona | מוטי קרמונה · 1 Comment

Robert (Bob) Sutton wrote insightful post about “Good Bosses” which I feel it worth more than just a tweet

With an evidence-based (experience studying and consulting to managers in many settings) approach he have identified a list of key beliefs that are held by the best bosses — and rejected, or more often simply never even thought about, by the worst bosses.

Here are the half dozen I liked most, you can read the rest (+ dedicated post on each one) on his HBR blog post.

  • I have a flawed and incomplete understanding of what it feels like to work for me.
  • My job is to serve as a human shield, to protect my people from external intrusions, distractions, and idiocy of every stripe — and to avoid imposing my own idiocy on them as well.
  • I aim to fight as if I am right, and listen as if I am wrong — and to teach my people to do the same thing.
  • Innovation is crucial to every team and organization. So my job is to encourage my people to generate and test all kinds of new ideas. But it is also my job to help them kill off all the bad ideas we generate, and most of the good ideas, too.
  • How I do things is as important as what I do.
  • Because I wield power over others, I am at great risk of acting like an insensitive jerk — and not realizing it.

@Bob, very insightful – Thanks!

By the way, just added his new book “Good Boss Bad Boss” to my Delver Wish-List… please feel more than free to surprise me ;)

*** *** *** *** *** *** *** *** *** *** *** ***

Random Quote“my comprehension can only be an infinitesimal fraction of all I want to understand” | Ada Lovelace (a.k.a. the *first* programmer)

→ 1 CommentTags: Leadership · Management · People