What is GeoMesa? | Hacker News Reader

What is GeoMesa?

geomesa.org

70 pointsbased210 comments

“GeoMesa is an Apache licensed open source suite of tools that enables large-scale geospatial analytics on cloud and distributed computing systems, letting you manage and analyze the huge spatio-temporal datasets that IoT, social media, tracking, and mobile phone applications seek to take advantage of today”

This is a geospatial focused library and framework that uses space-filling curves to collapse space/time dimensions so that it can run on any sorted key/value database (HBase, Cassandra, bigtable, etc) for storage and querying.

I would be interested in knowing more about how the z-curve based indexing performs compared to R-trees, for example as used in PostGIS.

Intuitively, if you look at the z-curve map below, some parts of antarctica which are adjacent, are not close on the z-curve.

http://www.geomesa.org/documentation/_images/Zcurve-HiRes.pn...

They are used for different data models. Z-curve indexing only works well for point geometries, and in most quality implementations is effectively equivalent to a quad-tree; sequential locality is not a useful optimization in modern geospatial storage engines. R-tree indexing generalizes well to non-point geometries like polygons, but doesn't scale so you can only use it for small-ish data sets. The best indexing algorithms for geospatial are analogous to adaptive quad-trie (not quad-tree) algorithms, though usually in 3-space and with additional features to ensure uniform shard sizes given highly biased data.

The most critical scalability feature of good geospatial stores has nothing to do with indexing algorithm per se: the ability to adaptively reshard and migrate data/load between cluster nodes. Geospatial data models are inherently prone to hot spotting which becomes worse as data models grow.

One thing I'd add about GeoMesa's use of space-filling curves is that most of the databases used are key-value stores. Given that, it is somewhat unwieldy to manage R-tree indexes.

Also, GeoMesa implements an 'XZ'-curve schema which gets around a few of the problems just using a Z-order curve runs into.

To the point about hot-spotting, there's a configurable sharding strategy that can help spread the data around more of the nodes in the cluster.

I haven't seen an easy/obvious way to apply adaptive quad-tries on top of key-value stores like Accumulo and HBase. Managing complex data structures in a cloud seems to be rather difficult.

We are interested in going down this path, would love to hear from anyone using it.

Also we need to deploy portable versions of the whole server side offline. Ie in docker or a VM or just on a linux box.

Here's a case study from CCRi using GeoMesa on BigTable from the GCP Next 2017 videos: https://youtu.be/KaRbKdMInuc?t=40m32s

You can find an assortment of Docker containers that have a mix of related packages and dependencies at https://github.com/geodocker

One quite cool aspect of GeoMesa is as an early use of JS versions of Arrow for, I believe, showing a bunch more in the browser than you normally see: https://github.com/apache/arrow/tree/master/js .

The Graphistry team has collaborating on the Apache side as part of how we pull off nodejs->GPU clusters on the server side, and been following as a cool proofpoint for the client one. Our early client-only versions looked similar (compiled down to typed arrays over web workers), yet much uglier without it, so I see this as quite promising for the web.

Hmm, this looks interesting.

Reply on news.ycombinator.com