RobinL

Similar or a bit better per mile to an efficient European ICE car with a single passenger. But:

- you travel vastly more miles in the plane

- you could fit four people in the car to achieve quarter the per person fuel consumption

Also assumes you're riding in economy.

Electric cars use around quarter the energy per mile than ICE, so therefore beat the plane easily even with single occupancy

Agreed. Highly recommend giving it a go for anyone who hasn't tried it. I'm a heavy (paid) chat gpt user, but for anything that I know in advance the answer will benefit from a web search (because it needs recent data), I use perplexity.

A public good is a commodity or service that is non-excludable (no one can be effectively excluded from using it) and non-rivalrous (one person's use of the good does not reduce its availability to others)

Examples include clean air, national defense, and public parks.

This isn't to say they shouldn't be provided by the government, but they're definitely not a public good

The performance is pretty good because the prediction is ultimately just adding up match weights. Much of the performance is dictated by

(1) The blocking approach you choose (how wide you cast the net in searching for matches. This is actually somewhere were minhash can be used in conjunction

(2) whether you choose to use complex fuzzy matching functions and how many - this is chosen by the user.

There's some benchmarking results here: https://www.robinlinacre.com/fast_deduplication/

Overall it's an approach which makes a pretty good trade off between speed and accuracy. That's why it's used by many national stats institutes (UK, US, Aus, Germany etc.) - because it's capable of working on country-population sized datasets.

The Fellegi Sunter model is able to estimate the importance of different types of information from the data itself (i.e. unsupervised learning).

For instance, a match on a date of birth column lends a greater weight of evidence in favour of a match than a match on first name (since dob has higher cardinality).

The method is also able to estimate weights for fuzzy matches (how much evidence in favour of a match is close match on dob with one character difference), and also how much evidence against a match a mismatch is.

For instance, if you have very high data quality on gender, then a match on gender doesn't tell you much, but a mismatch on gender is quite strong evidence against the idea two records match.

I have a blog post here that delves into this a bit more: https://www.robinlinacre.com/fellegi_sunter_accuracy/

One useful technique here could be to use text embeddings and cosine similarity: https://simonwillison.net/2023/Oct/23/embeddings/

For deduplicating people, the Fellegi Sunter model is also a powerful approach. Splink[0] is a free Python library that implements this for big datasets. Probably you could combine parts of both approaches as well. Full disclosure: I am the lead author.

I've also written up some interactive tutorials on how the method works [1] if anyone's interested

[0]https://github.com/moj-analytical-services/splink [1]https://www.robinlinacre.com/intro_to_probabilistic_linkage/

Beautiful, I love it!

The article provides zero insight beyond anecodes. The fact a a specialist ebike repair company sees lots of broken ebikes tells you practically nothing about the frequency of problems. The anecdote about 5 failures in a warranty period sounds a bit suspicious and if true is probably a combination of bad luck and extreme usage, or possibly a design flaw in a specific bike model. It's also from a forum post from 5 years ago . That said, as a Bosch ebike owner myself, I'd love to see some decent stats on reliability of different systems. Had no problems myself so far and love it.

You may know this already but the postgres extension[1] may help:

If I understand it correctly, when you use it it:

- Pulls the minimal data required (inferred from the query) from postgres into duckdb

- Executes your query using duckdb execution engine

BUT, if your postgres function is not supported by DuckDB I think you can use the `postgres_execute` [2] to execute the function within postgres itself

I'm not sure whether you can e.g do a CTE pipeline that starts with postgres_execute, and then executes Duckdb sql in later stages of the pipeline

[1] https://duckdb.org/docs/extensions/postgres.html#running-sql... [2]https://duckdb.org/docs/extensions/postgres.html#the-postgre...

Feels like there could be anti fake news applications. It's pretty common for photos to be recycled and purported to be of current events, when what they actually show is far away both in distance and time

In Britain, 'fired' usually implies a problem with individual performance whereas ,'made redundant' could be part of corporate restructuring. 'let go' is more ambiguous

I don't understand why the government don't do more to support these kind of tariffs that incentive demand shifting.. it seems such a powerful way to make the grid greener without huge infrastructure projects

Oh - nice, thanks - so it looks like the interactive window (which is effectivey the same as the output in a jupyter notebook) is also possible, but not (yet) 'properly'/'officially' supported

https://github.com/REditorSupport/vscode-R/issues/1412

I'm specifically referring to: https://code.visualstudio.com/docs/python/jupyter-support-py

The support for R looks a bit different (to me at least?): https://code.visualstudio.com/docs/languages/r

In the screenshot the window on the right does not look comparable to the output in a jupyter notebook. It looks more like a standard terminal. e.g. does it support interactive charts, html tables etc?

The Python interactive window uses the ipykernel package to allow rich outputs like that.

I still might be wrong and would like to be corrected on this, since it would mean R support in VS Code is now better than I thought (I haven't tried it fora. while)

It looks like, as far as I can tell, VS Code doesn't support the interactive window for working in R, which was a bit of a surprise to me when i looked it up.

The python interactive window has pretty much fully replaced my use of jupyter, since it gives you notebook-style output without the annoyance of the notebook format. My usual workflow is highlighting lines of code and shift-enter to execute (there's also a cells syntax).

I'm surprised by this because it _is_ possible to use R in Jupyter (although I never really liked the experience, R Studio was far superior).

Lead author here! Nice surprise to see Splink being posted here. Readers may be also interested in this recent blog post which delves into the performance of Splink in more depth: https://www.robinlinacre.com/fast_deduplication/

In addition, I guess a public defender is likely to work harder in a case where they believe the client is innocent.

This interpretation of 'eleven magic words' also suggests it matters: https://news.ycombinator.com/item?id=33471299

I value how useful I am to the world far more highly than how profitable I am to my employer. The primary reason I love open source is that it maximises how useful my work can be to others.

An important secondary consideration is paying it forward.

Amazing work as always.

Is it just me who sees diagonal lines on the first interactive graphic? Are they a bug? They appear after a little while, fade out, then reappear

https://gist.github.com/RobinL/aa4a5e14d35e61e46c1e99c8198d0...

I wrote an article a while back about why I think it's the best default choice for vis in Python: https://www.robinlinacre.com/backing_vega_lite/

To save to an image in Altair, it's just chart.save("myimage.png"). This works much better since version 5 (the syntax is the same but the dependencies are less fiddly)

Check out suni.ai, especially outputs from the new v3 model released a couple days back. The quality seems to have gone way beyond background music. E.g https://app.suno.ai/song/efc4e137-3464-49dc-9409-445b9943e3b...

I wonder if th result changes if you put a high quality medical reference in context. Feels like there might be an opportunity for someone to try and cram as much medical knowledge as possible in 1m tokens and use the new Gemini model.

For me, it's about having a job I enjoy and feel like I'm good at. Which would apply to most of the jobs I've done. So in those circumstances I'd just get a new job, there are lots of roles and industries that interest me.

The problem would occur if I was unemployed and unemployable because my skills (or any new skills I could conceivably learn) were no longer valuable.

I think this is subjective. I get a sense of purpose from my job. I don't feel like 'just' a labourer, I feel like by working I'm contributing to society and helping others.

I don't want infinite leisure, I want to feel useful. A good day's work is very satisfying. But at the point machines can do my job better than I can, it might be difficult to feel useful.

Observable is such an incredible, powerful and enjoyable tool. I use it heavily, including to power my blog. I love it, but I've always had a slight concern about needing to rely on the Observable notebook website. So this has really made my day.

To give a sense of the kind of performant, statically hosted interactive content that has only really been within my reach since using Observable, here are some examples:

Highly interactive vis: https://www.robinlinacre.com/visualising_fellegi_sunter/

Editable computations that flow through the document https://www.robinlinacre.com/computing_fellegi_sunter/

Multiple synced representations of data: https://www.robinlinacre.com/prob_bf_mw/ https://www.robinlinacre.com/partial_match_weights/ (half way down, under 'Understanding the partial match weight chart and waterfall chart)

Of course, there are a huge number of additional examples: https://observablehq.com/trending but i think the whole thing makes much more sense to the end-user when embedded (sadly at which point they don't even know it's observable!)

I generated the lyrics using ChatGPT 4 and the suno model attempts to follow them.

It generally does a good job, but I have noticed it's fairly common in a second chorus for it to ignore the direction and instead use the same lyrics as the first chorus

I don't have any special insight into how it works, but I suspect it is largely synthesizing audio from scratch. The more I've thought about it, the task of generating music feels very similar to the task of text-to-speech with realistic intonation. So feels like the same techniques would be applicable.

Suno do have an open source repo here that presumably uses similar tech: https://github.com/suno-ai/bark

> Bark was developed for research purposes. It is not a conventional text-to-speech model but instead a fully generative text-to-audio model, which can deviate in unexpected ways from provided prompts. Suno does not take responsibility for any output generated. Use at your own risk, and please act responsibly.

I've generated probably >200 songs now with Suno, of which perhaps 10 have been any good, and I can't detect any pattern in terms of the outputs.

Here's another one which is pretty good. I accidentally copied and pasted the prompt and lyrics, and it's amazing to me how 'musically' it renders the prompt:

https://app.suno.ai/song/d7bad82b-3018-4936-a06d-8477b400aae...

Here are a couple more which are pretty good (i use it primarily for making fun songs for my kids):

https://app.suno.ai/song/a308ca8a-9971-47a3-8bb3-a95126ff1a8...

https://app.suno.ai/song/3b78a631-b52a-4608-a885-94f2edc190b...

And this one's kindof interesting in that it can render 'gregorian chant' (i mean it's not very good): https://app.suno.ai/song/0da7502b-73cf-4106-88e8-26f4f465a5f...

But this is one reason it feels like these models are very similar to text-to-speech but with a different training set

Here is a silly song I generated using suno.ai, which I have found to be incredibly impressive (at least, a small percentage of its outputs are very good, most are bad). I think it's good enough that most humans wouldn't realise it's AI generated. https://app.suno.ai/song/8a64868d-9dd3-46db-91af-f962d4bec8b...