phiresky

https://casual-effects.com/markdeep/ has a similar idea but different execution.

I use something similar for my blog (I don't have a WYSIWYG editor for the widgets though, that's pretty neat): The posts are written in markdown, but then there's a few "widgets" that are basically markdown code blocks with a special "language" tag set that will render a React component with the text content as a input prop. I also use pandoc-flavored markdown (https://pandoc.org/) which already has many features that make formatting a bit more complex things easier (tables, latex math, ...)

For example a bar chart widget ( https://phiresky.github.io/blog/2019/rga--ripgrep-for-zip-ta... ):

    [...] it will be much slower:
    
    ```barchart
    title: Searching in 65 pdfs with 93 slides each
    series: run time (seconds, lower is better)
    data:
       pdfgrep: 19.16
       rga (first run): 2.95
       rga (subsequent runs): 0.092
    ```

becomes a colored bar chart with three bars. So the file is both kinda readable in a plain text editor, in a browser with js disabled (with React server side rendering), and in the richest version with JS enabled.

Full example:

Markdown:

https://github.com/phiresky/blog/blob/master/posts/2021/host...

Corresponding post with interactive widgets:

https://phiresky.github.io/blog/2021/hosting-sqlite-database...

Alternatives (without judgement):

https://github.com/cantino/mcfly

https://github.com/jcsalterego/historian

https://github.com/larkery/zsh-histdb + https://github.com/m42e/zsh-histdb-fzf

https://github.com/ellie/atuin

All of these except the OP (I think) use SQLite databases.

Personally I use zsh-histdb, which is great but only for ZSH. I'm working on adding a rich SQLite history to nushell so I can finally try nu: https://github.com/nushell/reedline/pull/401 .

The popular pg-promise library for PostgreSQL in NodeJS has a similar issue - for some types of queries ("Formatting Filters") it interpolates parameters itself instead of using real parameterized queries.

This is especially bad because it uses it's own escaping function and escaping in PG depends on a server configuration variable (standard_conforming_strings) that the client doesn't know about.

This behavior is barely mentioned in the docs, and the author does not really accept any suggestions or criticism.

The shutter-open-for-half-a-frame or "180° shutter angle" is commonly recommended as a default for shooting video. It is the result of having to compromise between inaccurate recording and a blurry video when your frame rate is low.

With 24fps and a small "shutter angle" you are capturing wrong information (e.g. resulting in wheels turning backwards) and you get a choppy look. With a large shutter angle you get too much motion blur. 180° is neither really correct nor really sharp, but it's the most commonly used compromise.

These problems are really the result of a too slow frame rate though - with 60fps you can shoot with a 1/60 shutter, and you get 100% accurate playback while still having frames sharper than a 24fps movie (with 1/48 s shutter) and a smooth video.

When you capture with a high frame rate (e.g. 120fps) and a 360° shutter angle (1/120s) you can also easily reduce the frame rate later without any ridiculous hacks like 3:2 pulldown by combining (averaging) frames or leaving them out, depending on the motion blur amount you want.

Applying the "180-degree-rule" to frame rates other than 24 or 30 fps just comes from missing knowledge or misunderstandings from what I can tell.

This only got better (at least here) because booking.com had to change their practices after legal action from the EU consumer protection commission.

> As a market leader, it is vital that companies like Booking.com meet their responsibilities in this area, ensuring that online accommodation reservation systems are free from manipulative techniques such as hiding sponsoring in ranking, unduly putting time pressure on users or misrepresenting rebates.

> Booking.com has committed to make the following changes to their practices by 16 June 2020 at the latest:

* Make clear to consumers that any statement such as “last room available!” refers only to the offer on the Booking.com platform;

* Not present an offer as being time-limited if the same price will still be available afterwards;

* Clarify how results are ranked and, whether payments made by the accommodation provider to Booking.com have influenced its position in the list of results;

* Ensure that it is clear when a price comparison is based on different circumstances (e.g. stay dates) and not present that comparison as a discount;

* Ensure that price comparisons presented as discounts represent genuine savings, e.g. by providing details about the Standard Rate price taken as a reference;

* Display the total price that the consumers will have to pay (including all unavoidable charges, fees and taxes that can reasonably be calculated in advance) in a clear and prominent way;

https://ec.europa.eu/commission/presscorner/detail/es/ip_19_...

According to the docs, looks like the ANY type in a strict table might do that:

> In a STRICT table, a column of type ANY always preserves the data exactly as it is received. For an ordinary non-strict table, a column of type ANY will attempt to convert strings that look like numbers into a numeric value, and if successful will store the numeric value rather than the original string

There's still much less data types than in most SQL databases though, e.g. float4, boolean, timestamp etc are missing.

You can also try sqlite-zstd [1], which is an sqlite extension allows transparent compression of individual rows by automatically training dictionaries based on groups of data rows.

Disclaimer: I made it and it's not production ready

[1] https://github.com/phiresky/sqlite-zstd

There's a free and opensource program called `diffpdf` that can compare both visually and by text. It has a GUI and works great, though it doesn't specially handle layout changes. It's included in the normal package sources in most Linux distros:

    apt install diffpdf
    pacman -S diffpdf

https://gitlab.com/eang/diffpdf

Looks like Netlify changed something since I wrote this article regarding what headers they send. Detecting support for Range-requests is kinda tricky and relies on heuristics [1]. Not sure why it still works in Chrome though.

You can go to this version of my blog, it should work there:

https://phiresky.github.io/blog/2021/hosting-sqlite-database...

Maybe the link could be updated?

Except for the DOM demo, since those need some Cross-Site-Isolation Headers you can't set on GitHub Pages (that's the reason I mirrored it to Netlify originally)

[1] https://github.com/phiresky/sql.js-httpvfs/issues/13

If you're talking about a normal local SQlite DB and your dataset is less than maybe 100GB of plaintext then SQLite FTS will work fine regarding performance.

You can chunk the file into e.g. 2MB chunks. The CDN can then cache all or the most commonly used ones. That's what I did in the original blog post to be able to host it on GitHub Pages.

Author of the referenced blog (and library) here. This is great!

The full text search engine in SQLite is sadly not really good for this - one reason is that it uses standard B-Trees, another is that it forces storing all token positions if you want BM25 sorting, which is a huge overhead for articles as long as Wikipedia's.

But that doesn't mean full text search isn't possible in a very efficient manner with statically hosted data! I wrote a proof of concept of making the Rust-based tantivy library work in the same way, which has a lot of internal things that can make the index much smaller and more efficient than SQLite's. It's also >10x faster in creating the search index.

Here's the demo also for Wikipedia: https://demo.phiresky.xyz/tmp-ytccrzsovkcjoylr/dist/index.ht... I'm not sure if it's more efficient than the SQlite version in this form, but it definitely has more upward potential and is more fun to work with.

And the corresponding draft PR: https://github.com/tantivy-search/tantivy/pull/1067

I sadly haven't gotten around to working on it more and writing an article about it.

Other people are also working on using this stuff to make Sci-hub and LibGen more available by using this in combination with IPFS for distributed and "uncensorable" hosting which is pretty awesome.

Edit: Just realized that the OP demo only searches in article titles, while mine searches in full article contents by default. You can search in only the titles in my demo by querying `title:harry title:potter`

Wikidata is a great resource, but the SPARQL query language seems more annoyingly complicated and confusing than it could be.

I'm using Wikidata to automatically categorize visited websites and used programs for time tracking purposes.

For example, here's a query to get the entity (e.g. company) that has a specific domain (news.ycombinator.com), and get the categories that entity is in that that are a descendent of the "service on internet" category:

    SELECT distinct ?service ?website_url ?outer_category ?outer_categoryLabel WHERE {
        ?service wdt:P856 ?website_url.
        optional {
            ?service wdt:P31 ?inner_category.
            ?inner_category wdt:P279* ?outer_category.
            ?outer_category wdt:P279+ wd:Q1668024.
        }
        VALUES ?website_url { <https://news.ycombinator.com/> }
        SERVICE wikibase:label { bd:serviceParam wikibase:language "en" }
    }

Returns

- social news website - online service - website

Relations have to be written as `wdt:P279` even though they have names [1], and the query engine just silently accepts lots of stuff it shouldn't. For example if above you do `"url"` instead of `<url>`, it will just not return any results because URLs is a separate type and URLs never match strings. And if an entity doesn't exist it will also just not return anything. Then there's this hacky "label service" thing that implicitly creates new output variables (outer_categoryLabel) to make it actually return text.

The UX of the query site [2] is also pretty bad.

I feel like Wikidata would be much more used if it had a easier query language. Or better, real bindings to common languages with full intellisense for properties etc). Something like this:

    const service = wikidata.variable();
    const inner_category = wikidata.variable();
    const outer_category = wikidata.variable();
    const results = await wikidata
        .filter(service.official_website("https://news.ycombinator.com"))
        .filter(service.instance_of(inner_category))
        .filter(inner_category.subclass_of(outer_category).anyDepth())
        .filter(outer_category.subclass_of("service on internet").anyDepth());

    console.log(results.map(row => row.get(outer_category).getLabel({language: "en"})));

It's also really hard to find good answers about the SparQL language without reading hundreds of pages of dry documentation.

[1]: https://phabricator.wikimedia.org/T196450 [2]: https://query.wikidata.org

From what I understand, to really get spatial audio from stereo headphones, you need to use a HRTF (head-related transfer function) specific to the person.

There's an open dataset of 50 different HRTFs [1] and a long video to compare them [2]. For me personally, only 2 of those 50 samples actually vaguely sound like audio is ever coming from in front of me. Left, right, and behind mostly works, but for most samples when it should be coming from in front of me it either comes from behind me or from above me.

So it shouldn't be really possible to get good spatial audio without 3d-scanning a persons head or at least making them go through a calibration step where they rate "where is the sound coming from?" . And it definitely shouldn't be possible to get any good results by pre-mixing the audio down to stereo before a user-specific transform is applied.

[1]: http://recherche.ircam.fr/equipes/salles/listen/index.html [2]: https://www.youtube.com/watch?v=VCXQp7swp5k

I never understood why the `sizes` attribute has to be so complicated. Why do I have to decide on which image to choose based on the display size instead of it automatically choosing the best image based on the size the image is displayed as?

It makes it impossible to choose the "correct" image without using JS if the amount of space the image will take up of the viewport is determined in some relative manner (i.e. basically always).

For example, If I have a container with width: 10em, and then in there an image with width: 50%. I want to just e.g. specify 3 alternative images with widths 200px, 400px, 800px. The browser knows the exact pixel size the image will actually be, so why can't it just choose the best one? Why do I have to specify which one to use based on the viewport width instead of the actual image width? It's impossible to calculate that unless your image is a known fixed ratio of the viewport or by asking the layout engine for the computed width (with JS).

There's an answer about this being impossible by one of the spec authors here: https://stackoverflow.com/questions/38684928/correct-sizes-f... But it doesn't explain at all why it was done this way, I don't even know when I would ever prefer to specify the chosen image in the way the spec currently requires.

Redis EXPIRE doesn't actually delete any data after it expires though. Active deletion happens at random, so you can easily still have expired values in memory months later:

> Redis keys are expired in two ways: a passive way, and an active way.

> A key is passively expired simply when some client tries to access it, and the key is found to be timed out.

> Of course this is not enough as there are expired keys that will never be accessed again. These keys should be expired anyway, so periodically Redis tests a few keys at random among keys with an expire set. All the keys that are already expired are deleted from the keyspace.

> Specifically this is what Redis does 10 times per second:

1. Test 20 random keys from the set of keys with an associated expire. 2. Delete all the keys found expired. 3. If more than 25% of keys were expired, start again from step 1.

So really it's not much better than doing `SELECT value from keys where key=? and expires > now()` with manual deletion. Though I agree that it can be more convenient.

Thank you, I really appreciate it. It's pretty fun to do this kind of thing for yourself, but it's really rewarding to be able to share it with other people.

The actual transferred data for the sqlite code should only be 550kB (gzip compression).

Stripping out the write parts is a good idea. SQLite actually has a set of compile time flags to omit features [1]. I just tried enabling as many of those as possible, but it didn't seem to reduce wasm size much, though I might be doing something wrong. There's also no easy flags to disable CREATE / UPDATE / INSERT .

[1] https://www.sqlite.org/compile.html#omitfeatures

I think it wouldn't change much - SQLite is already pretty optimized towards reads, for example a write always replaces a whole page and locks the whole DB. The free pages can easily be removed by doing VACUUM beforehand which should be done anyways to balance the b-trees.

The storage of SQLite is already really efficient, for example integers are always stored as varints so small ones only take a byte. The only thing I think could maybe be improved for this use case is changing the structure of the b-tree to be more columnar - since right now all the data from different columns is intermingled with the btree structure itself, querying a subset of columns has a high overhead.

That's true, but it also means that random access will always use at least that amount of data even if it only has to fetch a tiny amount. I did a few (non-scientific) benchmarks on a few queries and 1kB seemed like an OK compromise.

And note that the request chunk size is bound to the SQLite page size, and to change that page size you have to rewrite the whole DB. So it can't be set on the fly unless you have multiple copies of the database.

Yeaah I felt like at that point the article was already long enough so I didn't bother describing the DOM part too much - even though I spent more time implementing that than I did implementing the rest ;)

Basically SQLite has a virtual table mechanism [1] where you have to define a few functions that figure out how to scan your "fake" table / which indices to use and then how to read / write the actual data. I hook into this mechanism and redirect the request to DOM functions like querySelector() etc. Then there's the issue about SQLite being fully synchronous, but I have to run it in a WebWorker - and the WebWorker can't actually access the DOM and it can only communicate asynchronously with the main thread... So I have to do some weird stuff with SharedArrayBuffer and Atomics.wait to make that work [2].

[1] https://www.sqlite.org/vtab.html [2] https://github.com/phiresky/sql.js-httpvfs/blob/master/src/v...

The B-Tree is a tree that in this case is perfectly balanced. So if you do a query with an index in a database it will fetch an logarithmic amount of data from the index and then a constant amount of data from the table.

For the example the wdi_data table is 300MB and an index on it is 100MB in size. This index has a tree depth of 4 - which means SQLite has to read exactly 4 pages (4KiB) to get to the bottom of it and find the exact position of the actual row data.

you can check the depth of the b-trees with `sqlite3_analyzer`.

Git does actually diff binaries and stores them very efficiently :) If you do a single small UPDATE in your db file git will only store the changed information. It's just kinda slow, and for most binary files the effort git spends to try and compute deltas for binary files is useless - which is why it has a bad reputation for binary files.

Note that the diffs that git shows you are completely unrelated to the deltas it uses to compress it's database - which are always "binary deltas" and not line-based diffs.

Also I'm not sure why you mean that db corruption possibility has something to do with whether or not it stores diffs?

Since random accesses across the internet are really slow, for this kind of fairly small table (where SQLite stores the row data inline within the B-Tree of the table) it basically fetches the whole content for each row - so even if you query only the long_name and country_code column, it will in fact fetch the data of all 29 columns in that table.

If you want it to fetch less data for querying a subset of columns, you could create create an index on those columns - then SQLite will do an COVERING INDEX scan and thus read only the necessary data (with the B-Tree itself and the start / end page alignment being the only overhead).

Huh, that's actually kind of a worst case I didn't think about: Since you're doing a reverse table scan my "sequential access" detection doesn't kick in. If you do the same query but with a forward scan it should fetch roughly the same amount of data but only do like 5 HTTP requests since the request size doubles for every sequential access.

e.g.:

select country_code, long_name from wdi_country where rowid >= 164 order by rowid asc limit 100;

Yeah, that was one of the inspirations for this. That one does not work in the browser though, would be a good project to do that same thing but with sqlite in wasm and integrated with WebTorrent instead of a native torrent program.

I actually did also implement a similar thing fetching data on demand from WebTorrent (and in turn helping to host the data yourself by being on the website): https://phiresky.github.io/tv-show-ratings/ That uses a protobufs split into a hashmap instead of SQLite though.

The main restriction is that the DB really needs well fitting indexes, otherwise querying is really slow and fetches a lot of data.

Regarding writing:

You could of course implement a writing API with POST requests for changing pages of the database - but then you would lose most of the benefits of this (not requiring any special kind of server).

I also thought about implementing a kind of overlay filesystem, where chunks that are written to the file are stored in a local storage so the modified data is available locally while still reading everything else from the remote database.

Interestingly in SQLite that's already exactly what the WAL mode does: It's a second file next to the database that's just a set of pages that are overlaid over the main file when read queries happen - which allows concurrent readers and writers since the database itself isn't in an undefined state even when write transactions are happening.

So you could enable WAL mode and disable WAL auto checkpointing, then you get a downloadable WAL file that can be read by normal SQLite and written back to the main file. It would be neat, but I'm not sure what the actual use case would be ;)

Note that Frink is proprietary, and that data file is directly taken from the GNU Units project in violation of its GPL license, including many of the textual descriptions.

I'm working on a time tracking tool inspired by Arbtt, written in Rust, that also tracks visited websites including metadata (e.g. YouTube video category), tracks shell directories etc, and also is able to import data from your phone and other sources like Google Fit.

It's not complete, but it's already working great for myself:

https://github.com/phiresky/timetrackrs

I've added a comparison to Arbtt and ActivityWatch in the readme.

> I don't know why dd remains the standard for so many tutorials

You can also do something like

    dd if=file.iso of=/dev/disk/by-id/ata-Samsung_SSD_840_EVO_120GB

Where the last part is basically the brand and name of your USB stick, not much guess work required. Though I've never seen that in any tutorials.