HNHacker News
TopNewBestAskShowJobs

thomasmg

654 karma · joined June 19, 2018

Original author of Hypersonic SQL and H2 database (Java relational databases): h2database.com
submissionscomments
thomasmg··on CUBOTino: A small, simple, 3D printed, inexpensive Rubik's Cube solver robot
There is a Lego Mindstorms Rubik's Cube solver robot as well: https://mindcuber.com/

It is a bit slow, but it seems to work. Here are some videos: https://www.youtube.com/user/IAssemble/videos

thomasmg··on FC8 – Faster 68K Decompression (2016)
I wonder how it compares against LZ4 for the same platform: https://github.com/arnaud-carre/lz4-68k . Specially because this LZ4 implementation doesn't use a table, it seems.
thomasmg··on Trinary Decision Trees for missing value handling
Well, it seems "ternary" is more common than "trinary", according to https://www.reddit.com/r/Iota/comments/cepkv7/difference_bet...

Update: there's also the "Ternary tree", the "Ternary search tree", the "Ternary heap", and the "Ternary numeral system". Most have even Wikipedia articles. There is no Wikipedia article for anything "Trinary" related to computer science.

thomasmg··on Successful room temperature ambient-pressure magnetic levitation of LK-99
Nice! For show, I would try to levitate. Probably using a stronger magnet, inside a small (tiny?) glass tube, so it slides up a bit. I wouldn't do "rock surgery" just yet, to remove dead weight. Levitation with 4 magnets in a checkboard fashion, as usually done for pyrolytic carbon sheets, maybe doesn't work as the sample is not flat.

Also, turning the magnet upside down seems useful. And then, heating up to show that it drops at a certain temperature. I wonder what would be needed in this case; I guess less than 100° C.

In any case, the "show" part is important. Good video quality is important.

thomasmg··on Observation of zero resistance above 100 K in Pb₁₀₋ₓCuₓ(PO₄)₆O
Lead is still used for example in car batteries (lead-acid), bullets, and solders (electronic components).
thomasmg··on Huazhong University demonstrates LK-99 diamagnetism
Not if 2 magnets are flipped. Over just one magnet, the pyrolytic carbon (or LK-99) would slide off to the side. Over 4 magnets arranged correctly, it stays in the middle. This is described here: https://www.imagesco.com/magnetism/graphite-levitation-kit.h...

Some superconductors will float and stay over just one magnet, but it's a different story: https://en.wikipedia.org/wiki/Flux_pinning - LK-99 is not of this type as far as I know.

thomasmg··on Huazhong University demonstrates LK-99 diamagnetism
I think they didn't show that yet because for scientists, what they did already proves diamagnetism: they show that both sides of the magnet lift the sample.

They could more or less easily demonstrate levitation even with the current sample, by using four (maybe slightly stronger) magnets. Four magnets are needed so that the sample doesn't float to the side. Typically four magnets are used to demonstrate diamagnetism of e.g. pyrolytic carbon, see the picture in https://en.wikipedia.org/wiki/Diamagnetism

So in summary, they don't show levitation because that's scientifically not needed. It would be much more impressive for the non-scientist however, so I guess eventually they will show this. I assume another reason they didn't do that so far is that they are afraid to lose the tiny sample :-)

thomasmg··on Invertible Bloom Lookup Tables with Less Randomness and Memory
When I started reading such papers this also confused me.

Yes, it's about randomness and probability of collisions. Adding two characters might be OK or not, it depends on the hash function and the use case. For example, the Java "String.hashCode" method return "abc-1": 92597638, "abc-2": 92597639; the hash code of the "-2" is (almost) always one greater than the "-1" variant. So the two are not independent. Anyway 32-bit hash functions might not be good and you need 64 bit. Then you should use something like MurmurHash.

But you don't need to compute MurmurHash twice. It is probably enough to compute one MurmurHash, and then hash this hash code using an integer hash function [1]: h1=integerHash(hash + 1), h2=integerHash(hash + 2). This is almost twice as fast for string data.

For certain use cases, for example for Bloom filters, you need more hash "functions". For Bloom filters, it is even simpler: you can split a 64-bit hash code into a 32-bit start, and a 32-bit offset. And then use: h(x)=start+x*offset. That means h(3)=h(2)+offset and so on. It is really fast.

https://stackoverflow.com/questions/664014/what-integer-hash...

thomasmg··on I don't need your query language
I have written 4 query optimizers: HypersonicSQL, H2 database, PointBase Micro, and Apache Jackrabbit Oak. I know procedural language don't have such optimizers. But I argue that you don't always need them. I argue that the database engine query optimizer is often more a risk than a help. I have seen missing indexes far too often. SQL doesn't require that the programmer thinks how the data is accessed, so the result is that too many programmers don't think about it, and so don't add indexes, request too much data, and so on. The SQL statements work fine with small (development) data sets, so the same statements are used with production databases, and you run into problems too late.

I don't think it's hard to write procedural queries that are always fast. You write a loop or map/filter/collect method. You just explicitly need to mention which index to use, is all.

thomasmg··on I don't need your query language
We write programs with Python, Java, Rust, Javascript and so on. Yet we use a very different language, eg. SQL, to query and modify data. Why? Why don't we use eg. Python as well? SQL is different from other languages: it is declarative, meaning it doesn't dictate how to do it, but what the response should be. Maybe that's the reason? But if that's the case, why are declarative languages not more popular? SQL and other language are not compatible, and that's often a problem. You have this hard border (related to impedance mismatch).

SQL (or GraphQL) is often used as a remote API: The client sends a SQL statement, the server processes it and sends the response. SQL is very powerful, but also dangerous: the statement might be very expensive, for example because an index is missing. Sure, you can shoot yourself in the foot also with Python or Java, but I argue it's harder. SQL is one more technology in your stack, one more thing to learn. And actually, there are many many SQL dialects.

I wish databases have better, faster, and standardized support for a fast procedural language. So that clients can send programs, the server processes it and sends the response. A way to access tables and indexes like a hash table or ordered map. That way, there is no border. There is no slow query due to a missing index. You have to think about how data is access, which indexes are needed. But you have everything under your control. There is no risk of a missing index, or risk of the database not picking the index it should.

(I wrote 3 relational database engines and 4 SQL parsers: HypersonicSQL, H2 database, Apache Jackrabbit Oak, PointBase Micro. I also wrote a GraphQL parser and engine, and a Key-Value store. It's not that I hate SQL.)

thomasmg··on FreeBSD spends 7% of its boot time running a bubblesort on its SYSINITs
I was talking about "exponential algorithms" in general, and about business logic. I know FreeBSD isn't business logic, but low-level kernel code. I don't know the details of the FreeBSD problem.
thomasmg··on FreeBSD spends 7% of its boot time running a bubblesort on its SYSINITs
For each page, you have a select statement. The first without offset, then with offset 10 limit 10, then offset 20 limit 10, offset 30 limit 10 and so on. The database engine will, for each query, read all entries up to the offset + limit. This is a staircase pattern: first it reads 10, then 20, then 30, and so on. So the sum of read entries is n * n / 2, which is O(n^2).

One could argue that the database engine should be "smarter", but it's not. Note that data could be added or removed in the meantime, so the database engine can't really cache the result. See also https://stackoverflow.com/questions/70519518/is-there-any-be...

thomasmg··on FreeBSD spends 7% of its boot time running a bubblesort on its SYSINITs
O(n^2) algorithms often cause performance issues. The main cases I have seen in business logic are: (A) offset pagination (select ... offset n) and then paginate over all entries, and (B) read a text value, append something, store, repeat.
thomasmg··on FreeBSD spends 7% of its boot time running a bubblesort on its SYSINITs
Why do you think qsort is not nice?

I could imagine that qsort is not an option in FreeBSD for example because it is recursive and there is no stack at that level, or there is a risk of stack overflow. If that's the case, shell sort should be fine I guess.

thomasmg··on The grid isn’t ready for 300M EVs by 2030
Non-fossil diesel is going to be very expensive... The alternative is battery swap stations. Nio seems to be successful with that.
thomasmg··on The Smallest Hash Table
Actually I think it would help quite a lot in what I struggled with!

> as LLMs are prone to BS

I agree. The output sounds very convincing, and on a high-level it might make sense. If there are 3 layers: (a) high-level idea, (b) more detailed ideas, and (c) choosing the right words, tone, syntax, grammar, then LLMs seem to be very good in (a) and (c), but quite bad at (b). For somebody not familiar with the topic, it's hard to detect this.

thomasmg··on The Smallest Hash Table
Let me know if I can help! If you are interested in new developments in this area, I can recommend reading the papers of Stefan Walzer. His work is quite amazing.
thomasmg··on The Smallest Hash Table
Yes! That's the implementation from the paper. There's also a Java implementation: https://github.com/thomasmueller/minperf
thomasmg··on The Smallest Hash Table
Exactly. The reviews where blind in my case (even thought I heard that's not always the case, specially for journals). In my case, there were multiple problems, like missing proves (probabilities and examples are not enough), which I agree. A lot of great feedback! Actually in retrospect, it seems amazing on how much feedback I got. Only one of the 6 reviews was about the tone, and rather useless.
thomasmg··on The Smallest Hash Table
About finding a minimal perfect hash function for a small set by brute force ("you'd be surprised to see how often you get lucky"): A few years ago, I spend (far too much) time in this rabbit hole, and invented a new algorithm and data structure for minimal perfect hash functions of an arbitrary size. I invented what I think is still the worlds best algorithm in terms of space usage. I then tried to publish a paper. As I'm not in academia (bachelor degree is all I have), I had no chance, my papers were rejected. Finally, a professor from Italy picked this up, together with his student, and then, finally, a paper was published: https://epubs.siam.org/doi/pdf/10.1137/1.9781611976007.14 - I was even able to present it at the conference, a few weeks before the pandemic started.

The paper goes into quite many details about the probability to find a minimal perfect hash function for small sets. I hope it is somewhat readable... the section for small sets is 5.2 "Searching for bijections": the average number of trials is (m^m / m!), with m being the size of the set. The rest of the paper is to extend this idea to larger sets, by splitting the sets recursively until they are small enough to be mapped in this way.

thomasmg··on Estimating square roots in your head
In the movie "Gifted" a girl calculates the approximate square root of a 4-digit number in her head. Here the scene on YouTube: https://www.youtube.com/watch?v=37meAwQqPsE

Later, it is said she is using the Trachtenberg system: https://en.wikipedia.org/wiki/Trachtenberg_system

The square root method seems to be described in the book "The Trachtenberg speed system of basic mathematics", the text is available on archive.org: https://archive.org/stream/TheTrachtenbergSpeedSystemOfBasic... (search for "square root").

thomasmg··on Australians scour desert for dangerous radioactive capsule smaller than a penny
I don't know if something like that would help, but in Switzerland there are annual measurement flights (with a helicopter): https://blog.alertswiss.ch/en/categories/focp-news/annual-ne...

Part of this seems to be (sometimes?) to find radioactive sources: "The goal of this part of the exercise was to find two radioactive sources which, for training purposes, had been placed somewhere in an area of about 2,500 km2"

thomasmg··on Sorting algorithms that don’t hate you
You are right, heap sort doesn't need recursion, and is more efficient than shell sort for large arrays. But if you compare the code size, it is hard to beat shell sort. This is a Java version:

    static <T>void shellSort(T[] a, Comparator<T> c) {
        int n = a.length;
        int h = 16, g = 1;
        while (n > g) {
            h = h + h + h / 4 + 16;
            g = (h + 15) / 16;
        }
        do {
            h = (h - 16) * 4 / 9;
            g = (h + 15) / 16;
            for (int i = g; i < n; i++) {
                T t = a[i];
                int j = i;
                for (; j >= g && c.compare(a[j - g], t) > 0; j -= g) {
                    a[j] = a[j - g];
                }
                a[j] = t;
            }
        } while (g > 1);
    }
I was wondering, is there anything as simple as that, and similarly efficient? I doubt it.
thomasmg··on Sorting algorithms that don’t hate you
I wonder which type of Shell sort you were looking at. In my experience, it is fairly simple to implement gap sequence that is faster than O(n^2). Sure it is not a stable sort. But it is really simple (a few lines of code), doesn't use recursion, and is fairly fast. In fact, I'm not aware of an algorithm that is as simple as Shell sort, and at the same time much faster than O(n^2).
thomasmg··on Cheap, sustainable hydrogen through solar power
Natural gas already contains hydrogen. Up to 20% of hydrogen in natural gas doesn't sound all that dangerous. The article you linked says "if a hydrogen economy replaced the fossil fuel-based energy system and had a leakage rate of 1%, its climate impact would be 0.6% of the fossil fuel system.". So the climate impact would be 100 times smaller(?)

Yes there are many delaying tactic for switching to electricity (the promise of "synthetic fuel" for example, to power existing gas cars). But using hydrogen to replace fossil fuels is needed anyway, for example for ammonia production and steel manufacturing. So I don't understand why you think hydrogen will delay the switch to electricity.

thomasmg··on Cheap, sustainable hydrogen through solar power
Natural gas infrastructure already deals with hydrogen: 2% of natural gas is hydrogen. For town gas which was used before (made from coal), hydrogen content was around 50%, so this isn't entirely new. Of course you can't re-use the existing infrastructure and use 100% hydrogen.

This isn't black-and-white. Leakage is a disadvantage, as it is for fossil fuels (natural gas, oil, coal,...). Other technologies have other problems: windmills kill birds and bats! Pumped storage kills fish! Dams cause methane leaks! It is important to quantify how much of a problem those disadvantages really are.

thomasmg··on Cheap, sustainable hydrogen through solar power
You can add hydrogen to natural gas (using existing pipelines and storage), and so slowly replace that. No changes to existing infrastructure needed up to maybe 20% I think.
thomasmg··on Mazda has tested a synthetic fuel in an unmodified MX5 on a 1000-mile road trip
The grid can handle everyone using electric cars. Yes some upgrades are needed, but they are needed anyway. See for example https://www.scientificamerican.com/article/why-electric-vehi...

Electric car batteries are even seen as part of the solution to move to renewables.

thomasmg··on PRQL: a simple, powerful, pipelined SQL replacement
I don't want to write everything myself. (Well, I did write 3 relational database engines, and 4 query optimizers... Hypersonic SQL, PointBase Micro, H2 database, and the Apache Jackrabbit Oak query engine.)

I want a good abstraction (API) that allows to write programs that access and manipulate data in an efficient and simple way. But I argue it shouldn't be a declarative set manipulation language like SQL, but a language that is more low-level and makes it hard to (by mistake) retrieve or update millions of rows, or makes it hard to write statements that are very inefficient, and makes it hard to have systems that lack encapsulation, so you end up having hundreds of tables that are entangled in a big mess.

I do like the fact that network roundtrips can be reduced: you can send SQL (or GraphQL) statements from the client to the server, where the statements are processed and the result is returned. But it's too easy for the result to become large, or statements too slow to be processed. With a key-value store (like Redis) you can't do that. Or can you send a list of commands to Redis (or some other key-value store) that are executed in a safe and efficient way (with guaranteed runtime) and the result is returned? That would be what I look for.

thomasmg··on PRQL: a simple, powerful, pipelined SQL replacement
> What we have is generally good enough

Well, that can be said about any programming language (assembler, C,...) but yet there is innovation in this area still. And that's good. But in the database space, there is little innovation. And no, I don't think what we have (SQL, GraphQL) is good enough.

> SQL is not a programming language

Yes and no. SQL is an API that allows changing the state. But (same as programming languages such as C) it has advantages and disadvantages.

> if you take the time to learn it

Again, that can be said about assembler as well :-) Or Excel. You can do a lot in Excel.

> SQL is as low level as you're going to get in regards to that.

Absolutely no: SQL is a high-level language. And I argue you don't need a declarative set-manipulation language at all, to store and retrieve data. Computers are not set-manipulation machines: they are instruction-based machines. And the popular programming languages reflect that: the most popular languages (Python, C, C++, Java, C#, Visual Basic, Javascript) are _all_ imperative languages, which is close to how computers operate. And that's relatively easy for people (developers) to understand. Functional languages are not that popular. So, SQL is an exception here.

> SQL has to be long and detailed

I argue the reason is that SQL is not a very good abstraction, if you need extremely large statements that are hard to understand and almost impossible to maintain. Same for Excel.

> I will concede that I've seen some monstrous SQL in the wild... mostly written by people who don't really know SQL

Yes. I argue that SQL _favors_ (or at least allows for) writing complicated, messy code that lacks encapsulation. In a larger company, you will have some percentage of people that are not all that great, and you don't want to pick a language that will easily result in a complete mess. Languages such as Java favor modular, well encapsulated code: it is hard to make a complete mess in Java.

> Permissions can be tweaked

Permissions are not an encapsulation mechanism, in the way "private" fields in Java / C++ are protecting other programs for accessing internal.

>> SQL statements often don't use indexes > This is a trivial problem

Well, for the experts, yes. But not for the novice programmer. And you will have some percentage of novice programmers. In a larger shop, you typically (still) need a "database administrator" or a group of (experienced) people that ensure indexes are made correctly.

> various libraries do have their own optimizations

Libraries typically don't have optimizers. Compilers have, yes, but those are not picking indexes or such that can result in one million times slower or faster runtime. The compilers typically can speed up things by a factor of 10, at most (constant factor). Query optimizers are very different. (Again, I wrote 3 relational databases, and 4 query optimizers.)

← PreviousPage 7 of 10Next →