HNHacker News
TopNewBestAskShowJobs

michalc

816 karma · joined April 7, 2016

https://charemza.name/
submissionscomments
michalc··on I am not a supplier (2022)
> If you use this, I owe you nothing.

Sure the legalese states I owe nothing, but if I’ve shared the code, written some documentation, and encouraged others to use it as I have on a few projects…

… I feel as though I do owe something. I feel like I’ve made a sort of social contract to provide a bit of support on what I’ve, er, “supplied”.

Not really sure why.

michalc··on Historical trends in the usage statistics of Python version 3 for websites
I was surprised to see that Python 3.6 is still the most popular version, even though it's been over 2 years since the end of security releases for it.
michalc··on Show HN: pg-bulk-ingest – Now with multi-table support
I didn't know about this - thanks!!
michalc··on Show HN: pg-bulk-ingest – Now with multi-table support
Ah the name - you're not the first to mention it! Do you (or anyone lurking...) have any suggestions as to what it might better be called?

On what it does/why it exists, we've kept the README quite light to avoid duplication, with the main bits of the docs at https://pg-bulk-ingest.docs.trade.gov.uk/

But to try to answer the question here:

A set of insert statements - there are lots of cases where this would be fine, so pg-bulk-ingest (/its future name ;-) would be unnecessary, and so you might as well use insert statements.

But there are lots of things that pg-bulk-ingest does that a set of insert statements don't:

- It uses COPY, which in many cases is (much?) faster than INSERT

- It creates the table if it doesn't exist

- It migrates an existing table to match the definition if it does exist (pg-bulk-ingest can't handle all possible changes, but enough to make it useful to us)

- And it migrates without holding a long ACCESS EXCLUSIVE lock on the table

- It offers a fairly structured way to do high-watermarked ingest - it stores and retrieves a high watermark so you can fetch the data "since" some previous ingest, with the exact meaning of "since" custom to each source you're ingesting from.

- And it (now) combines COPYs with multiple tables, without loading all incoming data into memory at once, and keeping the ingest within a single transaction, so other clients see all the data, or none of it.

(The high watermarking I think is maybe the least clear of these in the docs, while also one of the most important things. Would really like an example that is somehow standalone that also makes it clear what it does...)

michalc··on OpenTTD
I thought it was time I got off my... and starting making an AI (https://github.com/michalc/SupplyChainLabAI - but it does nothing but log a debug message repeatedly at this point, I feel I have a long journey...)

But it makes me realise there is another use case for OpenTTDLab that I don't think I considered: regression tests for OpenTTD AIs?

Edit: And it has been mentioned to me separately that maybe it could be used as part of regression tests of OpenTTD itself https://github.com/OpenTTD/OpenTTD/discussions/11863#discuss...

michalc··on OpenTTD
Short answer: yes, exactly

Long answer: trying to make it much easier to do simulation-based research using OpenTTD, and for any results found to be as reproducible as possible. There have been a few papers/dissertations that use OpenTTD, but it's tricky to do exactly what the authors did, or even in a few cases I suspect it was hard for them to repeat their own experiments or extract results from them. OpenTTDLab seeks to make these easier.

And, such research seems to often (/always?) use AIs, so that's the focus.

I also would like to be able to train an AI in some automated(/formal?) way, and maybe use OpenTTD to investigate something a bit more... supply-chain-y, but that's a work in progress (I've not even written an AI yet!)

But also a bit of a less formal "just run an AI/some AIs over some seeds and see" is also very much a use case. Am trying to strike that balance between something quite... exploratory, but still reproducible.

michalc··on OpenTTD
Related: I recently made OpenTTDLab https://github.com/michalc/OpenTTDLab that can run OpenTTD headlessly from Python and extract data from its savegames in a hopefully fairly straightforward way.

(Very recently posted on ShowHN, so maybe this is cheeky…)

michalc··on Getting things done in small increments (2022)
Very much on board with practicing this sort of thing and getting to know what works for you.

For me, it’s really about deciding about what bit of micro “value” I can do next, doing it, and leaving things in a working “done” state. And interestingly not so much leaving something “unfinished” as some commenters say.

And also maybe not knowing exactly the next thing to do. Tomorrow can bring unexpected things, new priorities (stemming from me sometimes depending on the situation) and that’s ok with me. Leaving things “done” makes this all fine and fairly non-painful.

Suspect this takes practice and a certain amount of confidence. Submitting a PR that doesn’t do all that has been planned/asked of you might be scary.

michalc··on Show HN: sqlite-memory-vfs - Open a SQLite db in Python without hitting disk
Have now implemented a version of this at https://github.com/michalc/sqlite-memory-vfs/blob/74a24ce567...

I think different to yours in two ways:

- If doesn't go via PENDING if it doesn't need to. Not sure what's best really - but since RESERVED can already be skipped in some cases (recovering from a hot-journal) then so far happy with "skipping is a thing", and PENDING is private to the VFS anyway from what I can tell, so nothing else should care I think

- It doesn't do the blocking in the lock function, but depends on SQLite's core code doing it via the busy handler/timeout

michalc··on Show HN: sqlite-memory-vfs - Open a SQLite db in Python without hitting disk
Ah thanks for this (maybe not so much for the code itself admittedly, but for the push to implement locking)

So after a few attempts, I _think_ I have something reasonable: https://github.com/michalc/sqlite-memory-vfs/blob/64a55c3491...

(I'm still a but stuck on PENDING I have to admit... not really sure how that fits in)

michalc··on Show HN: sqlite-memory-vfs - Open a SQLite db in Python without hitting disk
> and the associated sqlite3_serialize and sqlite3_deserialize APIs

So... I have to admit I missed the existence of these. So for many cases, using sqlite3_deserialize is probably a very reasonable choice over sqlite-memory-vfs

Still, as you say, sqlite-memory-vfs doesn't require the memory to be contiguous, unlike sqlite3_deserialize

> You don't support concurrency at all (xLock/xUnlock/xCheckReservedLock are noops).

Well, you can read concurrently, just not write

michalc··on Show HN: sqlite-memory-vfs - Open a SQLite db in Python without hitting disk
No benchmarks right now. And in fact, I'm not even really sure what would make good ones, so suggestions welcome.

In terms of the difference from pointing SQLite to /dev/shm, my answer is somewhere between "I don't know", and "I suspect the VFS is more portable because not all systems have /dev/shm?"

michalc··on The big TDD misunderstanding (2022)
> Now, you change a little thing in your code base, and the only thing the testing suite tells you is that you will be busy the rest of the day rewriting false positive test cases.

If there is anything that makes me cry, it’s hearing “it’s done, now I need to fix the tests”

michalc··on Shshsh is a bridge connects Python and shell
My own attempt at bridging the Python and, well, maybe not quite shell but more subprocess, boundary: https://github.com/uktrade/iterable-subprocess.
michalc··on What if OpenDocument used SQLite? (2014)
Shameless plug of a couple of Python libraries I’ve been involved with that work around memory issues of ODS files (for very specific use cases):

https://github.com/uktrade/stream-read-ods https://github.com/uktrade/stream-write-ods

michalc··on For Python packages, file structure != API
You can also make a single file Python package that achieves the same API.

https://www.stefaanlippens.net/single-python-module-packagin... has info on how to do this (not my page, but seems clear)

michalc··on AsyncIO
Shameless plug for a short post/guide to “the good bits” of asyncio from a few years ago: https://charemza.name/blog/posts/python/asyncio/I-like-pytho...
michalc··on Portugal has a divorce rate of 94%
I'm sad that it was. I think it's interesting pretty much any way you slice it - either from a sort of social point of view, or from a statistical point of view.
michalc··on Programmer interrupted: The cost of interruption and context switching (2022)
I do something very similar - I didn’t realise this had a name. I plan that I will essentially have to context change in a short amount of time, so try to achieve some tiny thing before that.

It really works - at least for me.

michalc··on Design of GNU Parallel (2015)
I have previously thought a bit about using something like Supervisor. And if I was running something a bit closer to the metal, with no other infrastructure to restart stuff, then I would be much more pro.

But if inside Docker when something else already has the job of restarting things if they fall over, then it feels a bit over complicated in that there are multiple ways of doing the restarting. Plus, I think there is a touch more visibility - it's all just command line arguments to parallel:

    parallel --will-cite --line-buffer --jobs 2 --halt now,done=1 ::: \
        "some_proc some args" \
        "another_proc some more args"
michalc··on Design of GNU Parallel (2015)
I've never been sure if it's too much of a hack, but I've used GNU parallel in Docker containers as a quick and easy way of getting multiple processes running for web applications.

And with the `--halt now,done=1` option (that I think is relatively recent?) it means that if any of the parallel processes exit, parallel would exit itself, the whole container will shut down, and external orchestration would start another one if needed.

michalc··on Ask HN: HN people who write meaningful software, how did you learn to program?
I have a very minor example of something similar myself:

A quick 1-liner Stack Overflow answer that addresses a common typo is my highest rated answer. Those that I spent hours (or maybe even days!) answering: almost nothing.

michalc··on Welcome to Your Airbnb, the Cleaning Fees Are $143 and You’ll Still Have
One thing seems odd to me... about a guest's rating (not a host's):

"she got a three-star review because the cleaning wasn’t up to the mark. Her perfect five-star rating dipped to 4.1."

Can guests see their own star rating? I didn't even realise guests had them on AirBnB. I can't seem to find mine anywhere

michalc··on Show HN: Query SQLite files stored in S3
Aha! This is done in https://github.com/uktrade/mbtiles-s3-server, but using an alternative library to query SQLite on S3, https://github.com/michalc/sqlite-s3-query

(Full disclosure: I wrote most of both of these)

michalc··on Show HN: Query SQLite files stored in S3
Ah even if this were the case, in some situations you can’t mount filesystems (e.g. Docker without certain capabilities enabled?), and so something more in the application layer would be the only way to access SQLite on S3?
michalc··on Real World Divorce
The many references to children as instruments of profit is… pretty bleak
michalc··on The world map that reboots your brain
I think other projections have been on my mental todo list for about 5 years now… one day!
michalc··on The world map that reboots your brain
Thanks!
michalc··on The world map that reboots your brain
Ah for me the lack of sense of what’s going on is sort of the point of it. The rotated projections are just as right as the regular Mercator projection in an objective way, but it really doesn’t feel that way.
michalc··on The world map that reboots your brain
I’ve posted this on HN before, but my own “brain reboot” came from making

http://projections.charemza.name/

that allows you to rotate the world before applying the Mercator projection. The “crazy looking” distortions make me realise just how distorted the usual projection itself is

← PreviousPage 2 of 4Next →