syastrov

Probably more likely inefficient data access patterns, such as N+1 queries, inefficient algorithms, lack of parallelism

It is super interesting! I was at a talk where a genetics company used the Smith-Waterman algorithm (1) to perform sequence alignment. Still I believe it took a long time (24 hours?) to perform the calculations. They were working on optimizing the code as doing so could mean a faster turnaround.

1. https://en.m.wikipedia.org/wiki/Smith–Waterman_algorithm

I did an experiment of switching an app which uses Postgres to use SQLite. I ran into issues where I had a long running transaction in one process doing writes and another process doing reads and writes. As soon as the second process tried to do any writes, I would often get the queries running into locks and aborting. Otherwise, it worked great for reads and sped up the app a lot as there were lots of N+1 queries issues.

This is also after playing with settings about WAL mode.

It would not be a viable solution to split these tables into separate databases to avoid the concurrency issue. Also, most of the issues involve writes to the same tables.

Is there honestly any way to get that working well with current SQLite or do we need to wait for hctree or begin concurrent to mature?

I would be concerned that to get this working in its current state would require a large refactoring of the app which would require a lot of effort and would introduce limitations that make coding more difficult.

One way to have the best of both worlds for situations where you have complicated read queries is to create database views for the queries. You can then define them as an unmanaged table in Django (requires some boilerplate) and query them in a simple way as if they were a normal ORM model.

Similarly, we have amazingly powerful and reliable computer hardware, but continue to have software that is extremely slow and buggy and hard to use.

I theorize that since software is so invisible/intangible, these complexities are not understood by many.

Anyone tried any of the alternative full-text search extensions mentioned? Zombodb sounds good, except I would like that it didn’t work synchronously and fail transactions if there is a network error talking to ElasticSearch. I think most people will use Postgres’ built in FTS up until a point or go straight to an application level integration with ElasticSearch.

> That's where things got a bit hairy. Lila is built on Play Framework which is not yet ported to Scala 3.

> So I forked it and butchered it to remove everything we don't need - which is actually most of the framework.

I guess there is hope that Play framework itself will be migrated to Scala 3 and that the dependency on the fork can be removed, but this is taking on a risk - what if there are security updates to the upstream in the mean time?

About 15 years ago, my brother used to run a shoutcast (Internet radio) server. He wasn’t getting many listeners since the list of stations people browse was sorted by popularity - number of listeners. So I disassembled and then hexedited the shoutcast server binary so that the initial number of listeners was 60-something (which meant the listener count would never drop below that). Then he actually started getting a few listeners :)

I was just curious if I could improve our PHP-based site’s performance. So I attached strace to an Apache process and followed the log of syscalls and counted milliseconds between them. Sure enough, I discovered that 20 ms was being spent each time on a DNS lookup to a statsd metrics collector service over UDP (I remember being told that this was lightweight, since it was UDP). PHP didn’t cache DNS lookups and this was sometimes happening many times per request. I added a static entry in /etc/hosts and the overall latency improved by 30% across all endpoints.

Another hack: I once was consulting for a client who was running Drupal and was going to launch their new site the next day, but suddenly it started crashing on some of the pages. I found out that you can take a core dump of Apache and load it into gdb. Then if you run some gdb macros, you can see the PHP stack trace at the time the crash occurred. Turns it it was some module (tokens?) they had recently enabled which was recursively calling into itself. Not sure why it didn’t hit some stack limit, though. We disabled the module, which fixed it and the client was super happy. If I knew more about Drupal, I probably would have disabled modules in a form of binary search as a first troubleshooting step. But I did know a little about gdb, so that came in handy.

Haha, that made my day! (It’s a reference to That Mitchell and Webb Look).

Yes, you are right. I was just mentioning the name of the principle as used in the article.

I experienced a case of zero one many that would have been better if YAGNI was applied.

We were building a messaging system for a system that was being rewritten, where the only requirement was direct messaging between 2 users. It was suggested to generalize it to support multi-user conversations since that was a product desire some time ago. This complicated the backend implementation significantly, including performance and maintainability. The APIs and the frontend only supported 2 user messaging still.

And the system never needed multiuser chat.

I think there’s a trend to replace applications having information-dense designs with all actions at your fingertips with ones that add so much padding that you can only see a little bit of information at a time and in order to perform any action, you have to hover or click several times.

e.g. postgraphile

Someone mentioned it on a Django forum thread about async ORM [0]. It seems to have been ignored. Super cool idea and execution! I hope it gets some more exposure.

[0] https://forum.djangoproject.com/t/asynchronous-orm/5925/51

There are a lot of comments about async ORM, but to me the support for application-level (e.g. if you use forms, admin, DRF, Graphene) validation of DB-level constraints is a super cool feature that solves a real pain point - having to duplicate this logic at both levels, or forgetting to do so.

Or use imgix for that part

For context: Heroku also has their own Redis add on.

I wrote an article [1] a few years ago to vent about how messed up and backward the experience has been for especially foreign filers. The “free” options provided by commercial companies are as TFA states unusable by large portions of the population. Besides those, the IRS provides “free fillable forms online”, which looks like a scam website and also has many limitations (one of which, in my case, was that you must provide a US phone number). It’s all beautifully messed up.

So there wouldn’t be this problem with Intuit if the government got their shit together. Why not spend efforts to improve the system rather than litigation? The answer? Lobbying.

This is all in stark contrast to the system which exists in Denmark, where I live, where taxes are all filed online and the government automatically fetches most information so that you mostly only need to review it and add deductions.

[1] https://medium.com/@syastrov/us-tax-system-a-fractal-of-bad-...

Another gem: http://www.christianpoulet.com/iemac/en/indexf.htm

JetBrains IDEs can generate methods for undefined names if you use an intention activated by ctrl+enter (or alt, I can’t remember)

Imgix has been quite nice: unlimited variants, pricing per master image plus bandwidth charges, and you can keep your images on S3. They even contacted us about setting up a custom agreement when they noticed we were going over the allotted plan.

Most likely the actual work done by this extension is little compared to the actual work required to perform the queries…but…

If you want to scale horizontally you can run multiple read replicas with something which distributes connections evenly to the replicas.

You can also put some books under the laptop and use it as a screen while connected to an external keyboard at desk height.

I thought of this idea as well, but don’t think this is really the right approach. Many of these issues relate to specific features which developers subsequently avoid using. I think these types of initiatives should be more broadly targeted and probably curated by the maintainers/users of the software. E.g. the faster-cpython project is focusing on improving general Python interpreter performance: https://github.com/faster-cpython/ideas

Wish I had a tool that could suggest things to do like the “cartoon” at the top of the article.

Kudos for open-sourcing it. Here’s to hoping some of the changes make their way into upstream CPython.

What’s the compatibility story like? Is there a list somewhere of unsupported features?

A bit off-topic, but as the article notes, Stripe is using AWS. Could someone share their experiences with AWS versus Google Cloud for a smallish business using mostly compute plus managed Postgres? The pricing differences shouldn’t be an issue, it’s more a question of whether they make a lot of breaking changes, have too many limitations, or have a bad UI, or too much complexity.

If a company was starting fresh, which would you choose?

I’d like a tool that automatically suggests which indexes to add/remove, including use of different types of indexes and partial indexes, based on the queries being performed. Even better if it can automatically test the results of that on a production workload.

Or suggest schema changes that would improve normalization or performance.

I’m just thinking about how much CO2 emission was spared by optimizing performance instead of scaling horizontally.

It would be nice to see the effect on the environment as part of the decision-making process for deciding whether to optimize versus scale.

I think developers have an innate desire to fix performance issues as optimization produces a better understanding of the code and the problem and developers would like to solve the problem in the best possible way. And it helps to know that the problem won’t bite again later.