HNHacker News
TopNewBestAskShowJobs

polemic

4,778 karma · joined August 31, 2011

Full stack web dev, geospatial specialist, man about town for Koordinates.com / KartProject.org

@polemic

http://polemic.nz

[ my public key: https://keybase.io/polemic; my proof: https://keybase.io/polemic/sigs/cYLikrIQUZYBGhp8ZBQ750Z3xfKUW14bEmwajriwXUI ]

submissionscomments
polemic··on Extreme video compression with prediction using pre-trainded diffusion models
Unproven - but yes like I say - a new type of compression
polemic··on Extreme video compression with prediction using pre-trainded diffusion models
Meh, AI doesn't break information theory. The relationship between the prompt size to the "similarity" of the result will be such that it doesn't beat traditional compression techniques.

At best we might consider it a new type of lossy (or... replacey?) compression. Of course if storage / RAM / bandwidth keeps increasing, this is quite likely the least energy efficient technique available.

polemic··on Extreme video compression with prediction using pre-trainded diffusion models
How big is the SORA model itself?
polemic··on Netflix: Piracy is difficult to compete against and growing rapidly
Music rights management is independant of production and distribution. This means that Spotify, Apple Music and any other music services are able to negotiate rights to large libraries of music quite easily.
polemic··on Lapsus$: GTA 6 hacker handed indefinite hospital order
[According to the article] He has acute autism, is violent in custody and expressed intent to continue comitting crimes. This isn't just "some kid with potential" – he will end up in prison sooner of later, which will be an especially hellish place for someone like that to be.

I hope he gets the help he needs. Maybe courtesy of certain British spooky agencies.

polemic··on [dead]
Insanity. Just because Paul - or anyone - thinks something "is plausible" does not mean that you should say "it is fact". It's the very root of everything wrong about social media and modern "conspiracy theory culture". Why are these "thought leaders" so bad at thinking?
polemic··on Kart: DVC for geospatial and tabular data. Git for GIS
I touched on this a little in this comment https://news.ycombinator.com/item?id=38075723 - but tldr; OSM has it's own versioning model, and their data model isn't a great fit for the flat-table structure of most of the GIS data we commonly work with.

That said, the general issue of "data supply chains" and keeping track of who did what, and when, is largely unsovled outside of OSM, and if you're looking after your own GIS data and you're concerned with tracking and detecting unwanted changes, Kart is a great option. You get all the cryptographic verifyability of git for free.

polemic··on Kart: DVC for geospatial and tabular data. Git for GIS
This is a great question, and one of the first things we had to answer before deciding to build Kart. These are my top 3:

1. We found Geogig hard to get running. I'm a software engineer but I struggled to get Geogig installed correctly. Typical GIS people are technical, but maybe not in command lines or building software. Kart is designed to be 'batteries included', installs easily on Windows, Mac and Linux.

2. Geogig was "inspired by" git, but isn't built "on top of" git. That meant rebuilding a lot of things that git is already _really_ good at. Kart teaches git how to work with spatial data, so we're getting all of the benefits of many thousands of human-hours in optimizations in git. It also means a lot of git tooling works with kart today, leaving us free to focus on GIS-specific enhancements. There are a few specific examples of later additions to git, like git filters, that have made critical features possible (e.g. spatially filtered clones).

3. Geogig was a great project largely sponsored by Boundless Geo before they got bought by Planet, and it more-or-less died at that point. Ultimately, these projects have a big bootstrapping problem. You need data, the tool, and willing users. Kart is sponsored by Koordinates.com - a platform that's been doing GIS data delivery for over 10 years. We have a lot of data that we're been 'mirroring' into Repos and will make avaiable soon, we have a lot of users with specific use cases (fetching updates to large, regularly updated datasets) who are already using it, and we're making a long term committement to Kart as an OS project.

polemic··on Kart: DVC for geospatial and tabular data. Git for GIS
I haven't, but it looks like it's a Git repo hosting solution? This issue with using Git with data directly, is you generally loose the per-row/feature change information. With common binary GIS data formats, just putting them into Git looses a lot of the utility and will blow out the size of the repo as you apply changes.

Kart gives you row-level tracking, so you can see who made what change & when, and diffs small and fast to apply.

polemic··on Kart: DVC for geospatial and tabular data. Git for GIS
Yes, you can do shallow clones with `--depth` as well. This is incredibly useful - it means we can publish massive Kart repositories of spatial data with lots of versioning info, but still allow users to work with small subsets of the most recent changes. Very important for typical GIS use cases.
polemic··on Kart: DVC for geospatial and tabular data. Git for GIS
One of the benefits of building on Git is a lot of people have put a lot of time into make it work really well with lots of objects. And even though we say "files", Git abstracts that into packfiles etc very efficiently.

So, we're seening pretty good performance. We're maintaining a number of repositories with several millions features, with a decade of weekly updates of ~10,000+ rows. It _does_ take some time to push that data around, but it's _vastly_ better than old ways, and once you have your clone, maintaining updates becomes extremely trivial - a _major_ unsolved problem in the GIS/data world.

I'd add - Kart has GIS specific features that nullify some of these issues. The ability to spatially index the objects, then filtering them on Clone, means I rapidly clone a tiny subset of the data to work with.

polemic··on Kart: DVC for geospatial and tabular data. Git for GIS
Dolt is a neat project, but it's tightly coupled to MySQL. Kart supports MySQL as a working-copy format but MySQL has some limitations around geometry support that make it unsuitable for most GIS usage - see our docs for more info: https://docs.kartproject.org/en/latest/pages/wc_types/mysql_...

Kart works with GIS working copies that are more familiar to GIS people - e.g. GeoPackage, Postgres/PostGIS & MSSQL databases. Differenet users can use different working copies, and still collaborate together too.

polemic··on Kart: DVC for geospatial and tabular data. Git for GIS
Hi there,

OpenStreetMap has it's own versioning mechanisms (and a fairly specific-to-OSM data model) and Kart isn't really designed to work with OSM data as such. Kart adds version control to the GIS data that planners, academics, architects, civil engineers, etc, use day-to-day. There's a lot of data out there!

"Large" is relative, but Kart works well with quite big vector datasets for these typical use cases. For example, we're regularly working with datasets that have over 2 million features, with a decade of weekly data changes.

Kart includes some feautres specifically for working with small geographic areas. We can spatially filtering cloned data so you're working with a small subset of a much larger dataset, but you still retain the abilityt commit/merge/push to the source repo.

polemic··on Kart: DVC for geospatial and tabular data. Git for GIS
Hi there,

Kart supports points, lines & polygons, as well as GeoTIFFs for imagery and LAZ for point clouds.

Kart is a CLI tool, but provides fully machine readable outputs. You can use the QGIS plugin to get a visual diff of vector feature changes though.

polemic··on Kart: DVC for geospatial and tabular data. Git for GIS
Hi everyone, I'm Hamish, PM for KartProject here. If you want to learn more about Kart:

  * Our CTO Rob Coup presenting on Kart at FOSS4G 23: https://www.youtube.com/watch?v=1B-HB2Z3Vlc
  * Docs are available at https://docs.kartproject.org/en/latest/
  * We also have a QGIS plugin! This gives you visual diffs of vector feature changes. https://plugins.qgis.org/plugins/kart/
Happy to answer any questions!
polemic··on Kart: DVC for geospatial and tabular data. Git for GIS
Hi! It's neat to see Kart making HN. These are great points, we'll get that Docs link much more visible.
polemic··on Kart: DVC for geospatial and tabular data. Git for GIS
Kart repositories are also Git repositories - they're 'interopable' in the sense that there is a lot of tooling that will work, but the storage structure for vector data differs, and using Git on a Kart repository won't work very well.

Kart serializes vector/tablular data into datasets in the repository, and manages the process of writing them out to useful working copies (GeoPackages, or into databases).

For large binaries - rasters and pointclouds - we're using LFS. We include some additional spaital information into pointer files to enable some very useful GIS functionality, like spatially filtered clones (this works for vector data too).

polemic··on Safety benefits of roundabouts
Roundabouts are great (and common where I live), but a downside is they can be difficult for pedestrians to cross. You need crossings that are set back from the exits, but this also makes it a longer path for the pedestrian.
polemic··on Striking auto workers want a 40% pay increase–the same rate their CEOs’ pay grew
Alternatively, as many argue, wages are not only set by supply and demand.
polemic··on “The biggest Ponzi of all time”: why Ben McKenzie became a crypto critic
Note that this page hijacks the back button, and is paywalled.
polemic··on US Supreme Court needs more scrutiny, not less
> It's interesting that the complaint for scrutiny comes now when loud liberal voices are upset with current rulings.

FWIW, the scrutiny first came about because Justice Thomas's wife was involved in an insurrection against the government.

polemic··on Overture Maps Foundation releases open map dataset
It's unclear to me how Overture manages to "license wash" OSM data here.

> Transportation: The OMF’s Transportation layer represents a worldwide road network derived from data in the OpenStreetMap project. This community-built data has been recast into the Overture data format which provides consistent segmentation of the data and a linear reference system to support additions of data such as speed limits or real-time traffic.

The OSM ODbL is crystal clear that OSM contributors have to be credited. I don't believe that CDLA Permissive v2.0 magically allows Overture to bypass it.

--EDIT: I missed that they're using different licenses per dataset, the transport theme is OBDL, which I'm sure will trip up users who are not careful.

polemic··on Why Host in Kosovo?
FOSS4G (Free and Open Source for Geospatial Information Systems) - https://2023.foss4g.org/

It's a yearly global conference series run by volunteers. There are also regional versions, e.g. FOSS4G NA will be in Baltimore this year (https://foss4gna.org/), Asia will be in Seoul (https://foss4g.asia/2023/) and Oceania will be Auckland (https://2023.foss4g-oceania.org/). It's a great conference if you're interested in Mapping / Geo tech.

polemic··on Why Host in Kosovo?
I've just spent a week in Kosovo for an international tech conference and it's pretty clear that the article is shit-posting. Kosovo might be small and somewhat 'wild west' but you'd be very silly to take any of that advice.

Even if you believee the claims, the EU - especially Germany - is heavily involved in economic development of the country. Kosovo is very keen to improve their EU integration. So, you might be able to side-step Kosovar authorities, but gain the attention of the EU and you're not going to be protected by incompetance or bribery of local law enforcement.

As an aside - it's a great place to visit. Friendly, safe, cheap (for now), good weather. Looking forward to returning one day.

polemic··on Bill would ban the teaching of scientific theories in Montana schools
The article states facts. It doesn't call anyone evil, or stupid.

But at face value, outlawing the teaching of well established scientific theory in science class is exactly the sort of thing you'd do if you were a theocratic bumpkin who despised knowledge.

polemic··on Gptcommit: Never write a commit message again (with the help of GPT-3)
The very last thing you should do is commit a GPT-3 generated commit message for a fairly simple reason: if GPT-3 can interpret and and explain the change as written, there is no reason to commit that message. You will always be able re-run the generator at any later date, over any range of changes, to get the same or (presumably, in future) improved results.

As pointed out by other comments, the commit message should be telling you facts about the change that are not evident from the change itself. GPT-3 can't tell readers why the change happened.

polemic··on New Zealand passes legislation banning cigarettes for future generations
As we'd say here: "Yeah, Nah". This is a caricature of parochial English thinking of the colonies from the 50s, and is entirely divorced of what Australia and New Zealand are like today.
polemic··on New Zealand passes legislation banning cigarettes for future generations
> They trust technocratic experts to know what's best for people. And implement.

It helps to be small & pragmatic. People (mostly) follow the rules and politicians (mostly) only implement rules with broad support. Police have wide discretion to apply laws and (mostly) do this to benefit public good.

That's not to say we don't have problems, of course, but this particular law is seen as a compromise approach to public health that more-or-less works for everyone.

polemic··on Show HN: We scaled Git to support 1 TB repos
There seem to be a lot of data version control systems built around ML pipelines or software development needs, but not so much on the sort of data editing that happens outside of software development & analysis.

Kart (https://kartproject.org) is built on git to provide data version control for geospatial vector & tabular data. Per-row (feature & attribute) version control and the ability to collaborate with a team of people is sorely missing from those workflows. It's focused on geographic use-cases, but you can work with 'plain old tables' too, with MySQL, PostgreSQL and MSSQL working copies (you don't have to pick - you can push and pull between them).

polemic··on Why Twitter didn’t go down: From a real Twitter SRE
Exhibit B: non-code policy changes have non-zero effects on your infrastructure. https://twitter.com/atax1a/status/1594905195737419777

> we don’t think so. the prod incident we heard about involved someone making an ill-advised choice to reactivate a large account, causing a huge load on the social-graph system, on the night before a prolonged high-traffic event.

← PreviousPage 2 of 19Next →