Opinions I have formed about the “geospatial industry”
twitter.com
twitter.com
FOSS GIS software is pretty good but suffers from the usual lack of budget. Nevertheless, I’ve been able to push robotics + GIS packages for QGIS with ease. Reaching out to my ESRI contacts about “where’s your indoor and outdoor robotics suite offerings” received a response of only crickets.
As for Google Maps. The biggest tell is that there’s almost no geospatial capability in the software. They don’t want you to analyze and understand. They just want you to consume “where is X”. It’s a standard Google vector for ads approach.
I’m going to rant for a long time if I don’t stop here so I’ll leave it at: stop teaching so damn much ESRI and start teaching way more programming. The biggest regret of my education was that I was mostly taught how to use Geospatial tools rather than how to make my own.
https://roscon.ros.org/2018/presentations/ROSCon2018_Unleash...
It was never meant for that. That is/was Google Earth Pro.
Not to be confused with Google Earth Engine, which is the most shocking piece of technology I’ve come across in my career and is absolutely free if you can get access: https://earthengine.google.com/
Earth Pro = prosumer Google Earth.
Earth Enterprise = private, offline globe.
Earth Engine = research-oriented massively parallelized ML-based imagery analysis platform.
FOSS GIS is incredible these days. You can do everything in open source - it just sometimes requires a bit of work. Between qGIS (+SAGA/Grass/etc.), postGIS, python tooling like GeoPandas, leaflet/cesium, etc. there's plenty of tools for different things outside the ESRI ecosystem, you just have to learn to use them. This is a long way from the dark ages 10+ years ago when much of this capability was closed source or via some ridiculously complex text interface of GRASS.
ESRI's market dominance on GIS analysis is still a big thing though but only really a hangover from the days when ArcGIS had much more capability and lock in coroporate training was a thing. Now qGIS has essentially most of the same capability, but ESRI's corporate training structures etc. are still there to some extent. It is losing its stranglehold though, especially amongst GIS 'dabblers' who don't want to sign up and lock in to a massive corporate account for whom the ESRI older-style value proposition isn't really there. qGIS is even an accepted option for local government these days in many places.
Geospatial is both complex and diverse though. There's a lot of industries there and while there's big easy simple 2d GIS markets in mining, government zoning and military, there's also a lot of other stuff going on and its all moving fast. 3d, temporal data, drone photography, lidar and satellite etc. There's lots of innovation happening there right now and new ESRI monoliths being built.
For example, few consider going with Oracle anymore now that you have PostgreSQL but that took a long time. So, looking perhaps as far as 10 years, which FOSS GIS software might even have a shot to start displacing ESRI in the future?
- Web - ESRI never had a major horse in this race. Google Maps is still there but more common now is Leaflet and Cesium for 3D is basically the only option. Both Leaflet and Cesium are FOSS.
- Server - As you say postGIS is now the industry standard
- Programming - There's some people using ArcPy if you are in the ESRI ecosystem but the plethora of FOSS libraries out there are probably bigger in scope and fit together better than ESRI. PyQGIS is another option but in my experience not as heavily used as other open source libraries (all of which are based on GDAL, OGC standards and the Java Topology Suite)
- Desktop analysis - This is ESRI's core stranglehold and I think qGIS is now a competitive option for most users (certainly it has most of the core functionality) however there's a lot of inertia to move from big companies especially those that want ongoing external support/training.
This, a thousand times over. During my first geography degree I had a lot of GIS modules. I loved it, the capabilities it offered and so on. But it soon became clear that if you wanted to become genuinely good at it, you needed to learn programming, or at the very least have a very basic understanding of it. In this respect, my course was woefully inadequate; professors would perhaps touch upon it rather tangentially, and my requests to include programming in the course modules were brushed off. "You'll do more of that during your Masters!" they said. The reality is that even the MSc had very little of it. If I could say something to all those fresh-faced GIS undergrads, is this: learn programming; it will give you a much better understanding of how it works, it will greatly increase your capabilities in terms of what you can do (as Waterluvian so eloquently said).
For those of you wondering, I did my undergrad in the UK, graduated in 2009.
Also, greetings from a fellow geographer!
Challenges presented in spatial data are really really fun, and interesting, you can extract so much more information if you also include their spatial relationship.
Sadly the industry itself is severely underpaid. If not for my excellent colleges, challenging but fun problems i would swap to typical soul crushing webdev job for thrice my pay.
TL;DR: Lean programming, and also learn GIS - they work perfectly together.
Reminds me of the opioid industry!
I could rant about sas for ages. Its like it was purpose built to encourage anti patterns and Its own docs show code style is a foreign concept to the company.
Echoing your point though, you have to be willing to code. However, I've been working with geospatial data for going on four years now in various academic settings, and it still blows my mind that I can process a very large amount of publicly available data, very quickly, on old equipment and make meaningful contributions to research & policy.
Source: Worked through ~1.3K hdf5 files in the last couple of hours
One thing I’d like to note—-I have learned a tremendous amount from other people responding to the opinions enumerated in that thread. Some things I’ve even started to question. So I’d love to hear your ideas—-especially if you disagree with me. Just please do it constructively. Sometimes seeing your ideas thrust onto HN without being prepared for it can be a harrowing experience and I’d rather this time around be a productive one.
Much love to the brilliant minds of HN -Joe
If not a game of thrones fan, it's adaptable to something real world.
I get where it comes from, but what I observed when I was in the military was if you asked staff in the operations center about data they would say, something like "we've got two drone feeds, GPS trackers, field reports.. we're drowning in all this data we can't look at it all..." Okay, great we got some money from the theater, we can get anything you want, what do you want? "Another drone feed." It seems that even if you can't use all your data right now, the long run answer to today's unanswered questions is usually more data, or did I not understand something about that take?
If you take a time machine back to the year 2000 and talk to people who collaborate on documents and ask them what would make their lives easier, very few of them will say "invent an online browser-based real time collaborative document editing system" - many will just want more time, and consistent file naming when e-mailing back and forth modified word documents.
I've heard it described by the NGA as the "success-catastrophe of big data." They are struggling to figure out wtf to do with a data source like Planet that is daily, everywhere. Change detection has turned out to be WAY harder than anticipated, so most of it goes not only unseen but also un-analyzed. It's a giant waste of valuable information that is still locked up in servers of AWS and Google Cloud.
So I think every community is interpreting the post with their own bias (it also showed up on Reddit). The context that is missing is that (1) you work for a company that writes amazing open source software to analyze satellite imagery and that as many other companies in the space have found out the hard way, (2) competing with ESRI marketing is absolutely difficult - you have some battle scars already from it. Cheers!
My biggest realizations came using tools like PostGIS, GeoPandas, Rasterio, and Numpy where geography is just another data type (albeit with some specialized functions to work with them) and well integrated with the rest of the software ecosystem.
You can see it reflected in the salaries - A "GIS Programmer" will often make a fraction of what a "Software Engineer" makes. There's a reason for that - for most use cases, geospatial isn't unique enough to warrant a walled garden of technology.
A critical distinction that explains much of what we see today is that geospatial analytics has almost no relationship to making maps at a technical level. There is a superficial similarity because they sometimes share presentation layer. All of the popular infrastructure and tooling was designed, either implicitly or explicitly, for making maps.
If you want to analyze some physical world behavior using multimodal sensor data, the paucity of viable data infrastructure is insurmountable for most companies. Unfortunately, this is what most organizations want to do these days.
I'm working on a startup trying to solve this problem now. The idea is to "crowdsource" aerial imagery using a network of camera arrays on 80,000 commercial flights/day. We stream the video feed back to earth using in-flight wifi and overlay the visual diff to create a map that shows minute-level changes. There are huge built-in cost savings that come with not having to develop, launch and maintain a satellite constellation, and with a vastly higher revisit rate (as high as 4,532/day in some areas). We're at notasatellite.com
[i] ground sync, which would be fine for a lot of use cases where higher revisit rate and sometimes less cloud cover is sufficient to make the imagery valuable, and open up non-wifi flights. Ground sync can be over regular 4G or even swapping flash drives [some W-IFE boxes update this way, though infrequently].
[ii] Getting dedicated bandwidth from the satellite/a2g internet provider, who can potentially allow you a lot more than is made available to an individual consumer, but obviously at a hefty charge.
[iii] using streaming codecs designed for low kbps rates - though this is obviously lossy, and pixel perfection matters for a lot of EO use cases.
Even if global flights slowed to post-2001 levels, we'd still be capturing vastly more data than the ~300 or so commercial imaging satellites. Our margins are fairly wide in that sense.
Google map is indeed a commercial success, but it's far from “the most successful and ambitious” when it comes to mapping itself, quite the opposite. State projects (even those developed before satellites where a thing[1]) are way better, and even openstreetmap achieves a better coverage and higher maps quality.
It's the “most successful and ambitious” commercial project, which is coherent with the rest of the thread.
[1]: for instance: https://www.sovietmaps.com/
Google doesn't even attempt to map most or all of these things here, much less expose this information. OSM attempts to build a much better map, and for my region, easily succeeds.
That's not to discredit Google Maps. Almost all of the time, I don't need the best map, I need the kind of yellow pages, yelp, sightseeing guide, live traffic and transport, navigation mashup Google is projecting onto their map with great skill. I think it's their best product.
And never ever use Google maps if you're planning a bike trip, unless you really want to get into troubles. As a host on warmshowers.org for the past four years, I've heard tons of horror stories…
If I want to ask "What exactly are the traffic restrictions for roads through Hyde Park" only street view has the data.
Agree Google Maps can't compete with OS if you want building outlines.
That's why the data can be different if you view Google Maps in the US or Europe. Some licenses are only allowed in Europe for example.
A) What technologies should I learn to allow me to get my foot into the industry? I have historically focused on being a UX focused web engineer. I'm sure that experience could surely transfer, but am interested in learning the technologies and/or understanding that is more core to geospatial problems.
B) As someone who loves maps and imagery, but does not want to work for the military industrial complex, how do I avoid point #3. "In geo, you either die a hero or live long enough to make the majority of your revenue from defense and intelligence."
For the latter, I’d focus on Python to start. Look at GDAL (and it’s various Python interfaces/wrappers like rasterio), Cloud Optimized Geotiffs, and the SpatioTemporal Asset Catalog spec.
In both cases, watch what the leading companies are putting out (Mapbox, Azavea where I work, Development Seed, etc.) and not what OGC is doing. One is a leading indicator, the other is a lagging one. Sorry OGC, I still love you and you play an important role.
B: Get comfortable with the idea of working with Defense. For better or worse, they’re the ones with the budget to take on humanitarian work, disaster response work, even climate change adaptation work. That’s my advice. Possible to avoid it altogether, but only if you work on an app that is domain-focused (eg Trulia) and incidentally geospatial.
Epidemiologists are nice, but they're expensive, rare, and usually bad at GIS programming. Most governments will happily hire a GIS analyst. They can't get enough maps and data dashboards. And most of the ones they do have are terrible (misleading, slow, convoluted, or just plain not useful).
For 8. Openstreetmap, GDAL and associated open source ecosystem offers a solution to proprietary format and API of the month problem to a degree. It is not surprising that there is no money flowing if mapping is not a core business of the users.
Aerial and satellite imagery is the heavily regulated and lucrative part of any mapping solution. If you are allowed and able to obtain that, you can probably get by just selling the raw imagery without geospatial processing.
OpenLayers was selected after Google started charging 10x for their maps API, grayed out the existing tiles, then required a credit card on file. Since the API key can be "view sourced", that was totally a non-starter, along with the always-changing ToS. Way too many moving goal-posts.
(Part of the reason non-free is a problem is that any testing ends up costing money, and the more testing you do, the more $$, which is a disincentive to do testing. But just staying on top of lopsided ToS gets old fast when you can just replace it.)
Using a combination of OpenLayers and SO sample code, I've been able to do everything I wanted to do and am happy with the final result, but usually it's some of the toughest programming of the month.
For me my GIS job was mostly a time to learn programming and get a real it job
I wanted to be that Geospatially focused programmer. So I graduate with my B.S. in Geography and another in CS. I get interviews with a couple of random defense contracts and directly for federal government, but I turn them down because honestly I don't want my work going to wars I might not agree with and I would need to leave my family. I also happened to get an interview at ESRI for fun when I still was deciding if I should leave my metro or not and they laughed at me because I didn't have an internship--something that is very common in Toronto, LA, NYC, but not common in my mid-US metro CS scene.
So I now have a nice programming career solving "big data" problems completely away from geospatial, because the industry is tiny. I don't really know where I was going with this but this industry lacks opportunities for programmers compared to all the other stuff they could be doing. I'm someone who put years into wanting to break into the industry and do the tech work, if I wasn't the prime candidate for a junior geospatial programmer role then I don't know who is and the industry still couldn't get me in it.
Mostly offtopic: I initially misread that as "Professor of Think Tank" and I immediately understood it to mean something within the disciplines of military affairs, macroeconomics, some areas of law, a few others. This actually seems like an interestingly useful grouping, possibly more so than "STEM."
I'm a statistician with a state health department and do some GIS work. If I were more skilled with geospatial analysis, there's no doubt the decision makers I work with would be delighted.
If somebody wants to use GIS skills, they should keep an eye on IT or analysis roles. They might not specifically ask for it, but you can still sell it in the interview if it'll help solve their problems.
If you're hoping to make the big bucks or work on cutting edge problems, then staying local will be difficult.
[0] https://en.wikipedia.org/wiki/Point_in_polygon#Ray_casting_a...
And I don't know what GIS is. I see it talked about _a lot_ but I don't understand how there can be so much money and interest in geography.
I understand web stuff being huge, because I'm online almost all day. But what _is_ GIS? Is it something businesses do and consumers don't see?
GIS is everything location. From modelling pandemics to rainforest loss, to logistics, to Tinder and Pokemon Go.
I think some of the sort-of-obscurity/confusion comes from the fact that, as the OP tweet gets at, you don't really see a very visible "GIS Industry"/products most of the time, you just see tools that happen to have maps embedded in them, and those often require a lot of backend GIS work to power them.
Finding a rideshare and planning your next oil well don't, at first, seem to have a lot in common, but on the backend there may be a surprising amount of overlap.
And less details, much less objects, more grander scale. Only points, polygons, polylines and markers.
The sacro-saint trilogy, for me at leat, is GDAL/OGR, PROJ and GEOS. They are the building blocks behind QGIS, PostGIS and most of spatial libraries in various programming languages (Rasterio and Fiona in Python for example).
On a more GIS analyst perspective, most of it can be done with ESRI or QGIS and if you can get you hand dirty PostGIS is an amazing tool that I which I had the time to invest way more (or even Spatialite, sqlite flavour of GIS in database). I really which than Shapefiles goes away for good and be superseded by GeoPackage coupled with GeoJSON.
The web had a hard time to be a proper tool for visualisation on GIS/remote sensing due to the big size of the various files but Cloud-optimized GeoTiff and Vector Tiles are the game changers but came too late.
Commercial vendors tends to go for the cloud and platform lock down as Planet and their super-powered jupyter notebook and access to the catalog. Same for Google Eart Engine.
We see also stuff like CartoDB and the Database As A Service for geodata.
In a world when Geodata are highly available with better resolution than never, they are so big to store somewhere than it can be crazy. Working on time series at country scale (or even world scale) require now a lot of store space and calculation power for high resolution images (Even just Sentinel or Landsat at 10m and 30m resolution respectively). Small-scale is easier than ever and lower resolution can be managed on a laptop. I remember when I had to work with time series dataset of Sentinel-1 and -2 images when we were preprocessing 10-12 Tb of data with a pretty long chain of operations to do before handing it out to another team for furter data analysis. Some academic lab are storing Petabytes of remote sensing data gather during the time.
"Geospatial Industry" is way more than looking at ESRI or Google Maps/Earth. A lot of things can be and are done with various research and industry projects than require high integration with the spatial field and the temporal analysis. And the quality of what you have as tools vary highly between field, actually I am working for a PhD on a Bayesian spatio-temporal modelling with lattice data. It is the wild west and you need to cure a lot of R libraries to get what you need. I don't know how much you can do in Python because we had to avoid any MCMC method due to constraints on processing power available : low in fund and no easy access to calculator, we settled for R-INLA who is very interesting on its own.
Replicate that across nearly every single political locality in the US, probably the world. The industry is huge.
Also, "long form twitter" is dumb.
Things may have changed since then (this was nearly a decade ago, before ArcGIS Pro licensing style was implemented), but probably not by much, you can probably extrapolate that ballpark and figure how much any actually decent sized org spends on Arc suite of products (a lot)
The thread format forces people to break their thoughts up into chunks - a little like bullet points on slides but 280 characters makes the chunks much more information-full than just a bullet.
I can like or retweet specific interesting points as I read them.
Individual tweets in a thread can include images or link previews or link to other tweets.
What platform are you reading them on? I find them fine to read in the Twitter iPhone app - almost indistinguishable from a article elsewhere. Is that not the case outside of the app?