Comment by tomhoward | Hacker News Reader

Comment by tomhoward | Hacker News Reader

Something like this is so sorely needed.

I've been doing some work in ag-tech for the past few years (having previously worked in ISP/telco, general SME web/mobile development, then consumer travel tech).

The tools available to farmers/growers for what should be quite basic things - i.e., web-connected weather stations and monitoring systems for soil moisture, frost alarms and irrigation control are terrible.

The industry is full of small players who take a look and think "I know how to build stuff with [Arduino/Raspberry Pi/Zigbee etc], I can build an ag-tech monitoring/control product easily".

So everyone builds a hardware product from scratch, then as a quick afterthought, a web app with an SQL database and Highcharts to show data graphs. You have an MVP within a few months, then a few paying customers in your local community, then suddenly your customers start asking for features you'd never thought of, like sensor inputs you didn't know existed, or combined displays of different kinds of data you'd never considered anyone would want, or needing it to be faster (SQL turns out to be really slow for this kind of data) or more reliable (bugfixing is hard when your devices are hundreds of miles away and connected via weak 3G/4G data links), and you realise your hardware and software architecture doesn't support any of this, and you'd have to completely start over to actually deliver something that customers really need, but you're out of money and energy.

The result is a lot of farmers/growers dissatisfied/frustrated at never being able to get the monitoring systems they need (some of them having tried 3+ different vendors), and a lot of new companies trying to build new solutions and going bust.

I've been thinking there needs to be some widely accepted open standards in the industry for hardware and software platforms, so solutions providers can avoid trying to re-invent everything and instead focus on integration of tried-and-true building blocks.

If this initiative brings that about, it would be a big win for everyone in the industry.

The best work I've ever done was for one of the biggest energy/agg companies in the world. They wanted an outside contractor to build something from scratch and ignore their normal corporate bureaucracy. They thought it was insane how much better the system we (I) built for them was than their normal in house tooling. The crazy part was the solutions were mostly just normal cloud tech. Postgres, DynamoDB, some AWS tools. They paid over a million dollars (nothing to them) for a set of tools that weren't technically advanced at all.

In my experience, the problems are a result of the manufactured constraints on the physical hardware/sensor devices. Mountings and power and connectivity. Receiving data from an array of devices into a shared storage system for analysis is a well solved problem. Receiving data from multiple sensors on a fixed interval, when the sensors may be made by different companies/work totally differently, is where the complexity lives. Combined with each company trying to build their own awful closed source proprietary data system on top of their sensors, you've got a really terrible time.

> SQL turns out to be really slow for this kind of data

I think this is just poor modeling. SQL is just fine for the work you're talking about.

> Combined with each company trying to build their own awful closed source proprietary data system on top of their sensors, you've got a really terrible time.

The more I think about right to repair, the more I become convinced it's a symptom of hazy interface specifications.

Radical idea: Ban sensor / actuator companies from building software on top of them in-house.

They're welcome to offer a turn-key solution to market, but it must (1) have hardware and software built by two separate, independent companies & (2) publish its interface specs, between those two companies, to all customers or end users.

Things would cost more. But I'm not convinced this would be a worse world, in aggregate.

Kind of simpler to me, is just making closed-source drivers illegal. I like the idea of being able to buy something cheap because they were too lazy to actually write a spec for how it works. But not being able to read the de facto spec - the source code for the driver - is just silly. I've paid for the device, the driver is basically useless except for that device. Give me the code.

> I think this is just poor modeling. SQL is just fine for the work you're talking about.

I'm as big a booster of good old SQL as anyone, but there's a lot to be said for more targeted time series solutions when it comes to sensors.

I'm working on a platform for monitoring water water tank levels. It slices Grafana and influxdb horizontally to share the resources between multiple users and multiple tanks.

The productivity of such a stack is high, when it comes to getting beautifully rendered, interactive graphs of e.g stacked water levels. And with influxdb flux language, you can write joins that join data from the time series database and the rdbms (for more reference data, like the names and calibration data of individual tanks).

Yes you can do anything with SQL but there's a reason for the presence of dedicated time series databases.

> I'm as big a booster of good old SQL as anyone, but there's a lot to be said for more targeted time series solutions when it comes to sensors.

https://www.timescale.com/

Sensors aren't really different from any other timeseries data.

> The productivity of such a stack is high, when it comes to getting beautifully rendered, interactive graphs of e.g stacked water levels. And with influxdb flux language, you can write joins that join data from the time series database and the rdbms (for more reference data, like the names and calibration data of individual tanks).

Your productivity being high with a given tech stack does not disqualify an alternative tech stack from having equally high (or much higher) productivity for equally trained users.

> Yes you can do anything with SQL but there's a reason for the presence of dedicated time series databases.

The reason you're describing is "marketing"

> Your productivity being high with a given tech stack does not disqualify an alternative tech stack from having equally high (or much higher) productivity for equally trained users.

Try implementing classic timescale features like down sampling in your straight RDBMS.

Certainly you can do it, just as you can build a house with a hammer and nails rather than a nail gun.

But you'll spend lots of time building undifferentiated infrastructure that you could have got out of the box.

https://blog.timescale.com/blog/how-to-proactively-manage-lo...

Also InfluxDB used to support InfluxQL (SQL). And I believe its still partially supported in v2.

Not completely related, but I also briefly consulted at Monasanto in an entire department that seemed composed of consultants hired to build something from scratch and ignore the normal bureaucracy.

It was fun because we were hired to do things that in house employees had failed to do for whatever reason, so when I built the little dashboard I was hired for, everyone was so amazed. I just completed the job I was given, but to them I was a superstar. Consulting can be fun like that.

There is a bit of progress towards open standards: e.g., MIAPPE, BrAPI. But you're right, we have a long way to go. Programmers and software/hardware engineers are sorely needed but ag struggles to attract them, and the startup model is not very well suited for the space imo- we need to build a collaborative ecosystem of open source tooling that farmers, biologists, breeders, and others in ag/plant sciences can hack for their particular use case

Check out Phenome Force, they do weekly webinars featuring the open source/DIY tools people are building: https://phenome-force.github.io/PhenomeForce/

Is remote work common in AgTech?

I can't comment on the situation prior, but post-covid it's pretty common afaik. I've been entirely remote since last March, in both software developer and grad student roles, and although word is students will be back in the office/classroom this fall at my institution I think there is likely a lot of flexibility w.r.t. software-oriented positions

I know a few people who work outside of academia, e.g. agtech companies under the Flagship Pioneering umbrella (Inari etc, same parent as Moderna and various other ag/biotech) and the software teams are very remote friendly I believe

If you're interested in a staff position at a university there's a chance my PI might be hiring for full time dev positions later this year (dependent on luck with grant applications), lmk if interested

You appear to have had the exact same experience as I have. I work in the fresh vegetable side of things and the tools that are available are completely closed source and locked in. Even silly things like humidity controls and whatnot. there's so much potential for the ag sector to benefit from all these open data sources and systems.

You can't make money selling hardware with open API, though. It'll get cloned and manufactured in China, and people buying in bulk will naturally choose the cheapest option.

Nobody makes money in open-source selling software, either. Everything is either consulting, or SAAS.

Wow. I worked in an agtech startup and this is exactly how it went. Hopefully your comment stays on top for others to read.

You know what I'd like to see is there can be other non-dev support for these types of efforts. For example:

* technical writers

* security pentesting/code reviews

* hosting and infrastructure support services for open source projects

Pretty sure that there are a lot of people who would be happy to donate their time to opensource projects that benefit farmers, so there is a bigger opportunity for matchmaking here.

That's the issue indeed. The entry threshold and the amount of minimum needed features for agtech products are very high. Moreover, the seasonality of ag business makes the situation for agtech startups even more difficult (you can usually sign new test clients only in between ag seasons)

Interested question : would those folks be interested in low bandwidth / long distance network over the 800khz band ( Lora ) ?

There is incentive to deploy those network now. A agreement to deploy is all what is needed ( private property style ). Electricity is a plus but that work well on solar too.

Can you elaborate on the kind of data and queries that SQL is slow for?

Likely the sensor data being stuffed into a "standard" SQL schema -- loads of these bespoke solutions aren't fully dialed in with tools like timescaleDB, or Prometheus for these metrics. Even with slower (eg 240s interval) sensors the data builds up -- and slows the systems (w/o indexes).

The problem that arises with a lot of these "pull data from sensors, pump it into a database" is schemas and data integrity have to be kind of a second-class problem behind storage. When you can't push an update to whatever is ingesting data, and that ingestion tool is also ingesting with an invalid format, you can't just ignore the data (or fix the problem). So your store has to accommodate semi structured and unstructured data gracefully.

I do not agree that SQL is "slow" for these types of problems. I've built a number of systems that support this issue effectively. You _could_ use a tool that has schemaless/unstructured data as a first-class feature, but if your goal is to reduce complexity a Postgres instance is just fine. As with all data projects, indexing is important and needs to be thoughtful (from the beginning). For sensor data, it's also a good idea to think about data retention and removal policies immediately (keep your metrics/aggregates, move raw data to cold storage after a while).

Are you aware of any promising companies or start ups in this area?

I'll do a bit of self-marketing, but we at GeoPard Ag https://geopard.tech work on this. Have some big ag companies as clients

Reply on news.ycombinator.com