HNHacker News
TopNewBestAskShowJobs

RobinL

2,261 karma · joined February 9, 2013

Data scientist/engineer. Blog: robinlinacre.com Twitter: https://twitter.com/RobinLinacre Lead author of Splink, for record linkage at scale: https://github.com/moj-analytical-services/splink
submissionscomments
RobinL··on Automatic Programming
Agree. Much of the value of devs is understanding the thing they're working on so they know what to do when it breaks, and knows what new features it can easily support. Doesn't matter whether they wrote the code, a colleague wrote it, or an AI.
RobinL··on Vibecoding #2
> saw absolutely zero value in it

At the very least, it can quickly build throwaway productivity enhancing tools.

Some examples from building a small education game: - I needed to record sound clips for a game. I vibe coded a webapp in <15 mins that had a record button, keyboard shortcuts to progress though the list of clips i needed, and outputted all the audio for over 100 separate files in the folder structure and with the file names i needed, and wrote the ffmpeg script to post process the files

- I needed json files for the path of each letter. gemini 3 converted images to json and then codex built me an interactive editor to tidy up the bits gemini go wrong by hand

The quality of the code didn't matter because all i needed was the outputs.

The final games can be found: https://www.robinlinacre.com/letter_constellations https://www.robinlinacre.com/bee_letters/ code: https://github.com/robinL/

RobinL··on Running Claude Code dangerously (safely)
Does anyone have direct experience with Claude making damaging mistakes in dangerously skip permissions mode? It'd be great to have a sense of what the real world risk is.
RobinL··on Why DuckDB is my first choice for data processing
I think that's totally fine for individual work, but in larger data engineering teams it's less good to switch between tools because other people may have to maintain your code.

That said, polars is good, and if the team agree to standardise on it then that's a totally reasonable choice.

I guess one of my reservations is I've been (historically) burned by decisions within data eng teams to use pandas, causing all sorts of problems with data typing and memory and eventually having to rewrite it all. But I accept polars doesn't suffer from the same problems (and actually some of them are even mitigated in more recent versions of pandas)

RobinL··on Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
Worse in some ways, better in others. DuckDB is often an excellent tool for this kind of task. Since it can run parallelized reads I imagine it's often faster than command line tool, and with easier to understand syntax
RobinL··on If writing the code is the easy part, why would I want someone else to write it?
Found myself nodding along. I think increasingly it's useful to think of PRs from unknown external people as more like an issue than a PR (kind of like the 'issue first' policy described in the article).

There's actually something very valuable about a user specifying what they want using a working solution, even if the code is not mergeable.

RobinL··on Why DuckDB is my first choice for data processing
Exactly - these huge machines are surely eating a lot into the need for distributed systems like Spark. So much less of a headache to run as well
RobinL··on Why DuckDB is my first choice for data processing
Author here. I wouldn't argue SQL or duckdb is _more_ testable than polars. But I think historically people have criticised SQL as being hard to test. Duckdb changes that.

I disagree that SQL has nothing to do with fast. One of the most amazing things to me about SQL is that, since it's declarative, the same code has got faster and faster to execute as we've gone through better and better SQL engines. I've seen this through the past five years of writing and maintaining a record linkage library. It generates SQL that can be executed against multiple backends. My library gets faster and faster year after year without me having to do anything, due to improvements in the SQL backends that handle things like vectorisation and parallelization for me. I imagine if I were to try and program the routines by hand, it would be significantly slower since so much work has gone into optimising SQL engines.

In terms of future proof - yes in the sense that the code will still be easy to run in 20 years time.

RobinL··on Why DuckDB is my first choice for data processing
Author here. Re: 'SQL should be the first option considered', there are certainly advantages to other dataframe APIs like pandas or polars, and arguably any one is better in the moment than SQL. At the moment Polars is ascendent and it's a high quality API.

But the problem is the ecosystem hasn't standardised on any of them, and it's annoying to have to rewrite pipelines from one dataframe API.

I also agree you're gonna hit OOM if your data is massive, but my guess is the vast majority of tabular data people process is <10GB, and that'll generally process fine on a single large machine. Certainly in my experience it's common to see Spark being used on datasets that are no where big enough to need it. DuckDB is gaining traction, but a lot of people still seem unaware how quickly you can process multiple GB of data on a laptop nowadays.

I guess my overall position is it's a good idea to think about using DuckDB first, because often it'll do the job quickly and easily. There are a whole host of scenarios where it's inappropriate, but it's a good place to start.

RobinL··on Why DuckDB is my first choice for data processing
It is true that for json and csv you need a full scan but there are several mitigations.

The first is simply that it's fast - for example, DuckDB has one of the best csv readers around, and it's parallelised.

Next, engines like DuckDB are optimised for aggregate analysis, where your single query processes a lot of rows (often a significant % of all rows). That means that a full scan is not necessarily as big a problem as it first appears. It's not like a transactional database where often you need to quickly locate and update a single row out of millions.

In addition, engines like DuckDB have predicate pushdown so if your data is stored in parquet format, then you do not need to scan every row because the parquet files themselves hold metadata about the values contained within the file.

Finally, when data is stored in formats like parquet, it's a columnar format, so it only needs to scan the data in that column, rather than needing to process the whole row even though you may be only interested in one or two columns

RobinL··on Ask HN: Share your personal website
https://robinlinacre.com
RobinL··on Find a pub that needs you
> In her November Budget, Chancellor Rachel Reeves scaled back business rate discounts that have been in force since the pandemic from 75% to 40% - and announced that there would be no discount at all from April.

That, combined with big upward adjustments to rateable values of pub premises, left landlords with the prospect of much higher rates bills.

RobinL··on Ozempic is changing the foods Americans buy
I was curious for a UK comparison so I looked it up.

At the start of 2025, about 3% of adults in UK had used GLP-1 drugs in past year in the UK. And "most GLP-1 for weight loss in the UK is from private, rather than NHS provision" [1].

[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC12781702/

RobinL··on ChatGPT Health
Another interesting aspect is that the NHS app makes all your detailed health history (doctors notes, scan results etc.) available to you as the patient.

Which in turn means you have the option of feeding it into ChatGPT. This feels potentially very valuable and a nice way of working around issues with whether doctors themselves are allowed to do it.

I'm not sure this applies to every surgery, but certainly my dad had access to everything immediately when he had a scan.

RobinL··on Total monthly number of StackOverflow questions over time
I'm hoping increasing we'll see agents helping with this sort of issue. I would like an agent that would do things like pull the spark repo into the working area and consult the source code/cross reference against what you're trying to do.

Once technique I've used successfully is to do this 'manually' to ensure codex/Claude code can grep around the libraries I'm using

RobinL··on Tiled Art
The art creation tool is great: https://tiled.art/en/create/
RobinL··on Gemini 3 Flash: Frontier intelligence built for speed
Yeah - agree, Anthropic much better for coding. I'm more thinking about the 'average chat user' (the larger potential userbase), most of whom are on chatgpt.
RobinL··on Gemini 3 Flash: Frontier intelligence built for speed
Feels like Google is really pulling ahead of the pack here. A model that is cheap, fast and good, combined with Android and gsuite integration seems like such powerful combination.

Presumably a big motivation for them is to be first to get something good and cheap enough they can serve to every Android device, ahead of whatever the OpenAI/Jony Ive hardware project will be, and way ahead of Apple Intelligence. Speaking for myself, I would pay quite a lot for truly 'AI first' phone that actually worked.

RobinL··on Is it a bubble?
I've found Opus 4.5 in copilot to be very impressive. Better than codex CLI in my experience. I agree Copilot definitely used to be absolutely awful.
RobinL··on AI Is Breaking the Moral Foundation of Modern Society
Price and scarcity go hand in hand, not value and scarcity.

Diamonds are pretty worthless but expensive because they're scarce (putting aside industrial applications), water is extremely valuable but cheap.

No doubt there are some goods where the value is related to price, but these are probably mostly status related goods. e.g. to many buyers, the whole point in a Rolex is that it's expensive.

RobinL··on Geothermal Breakthrough in South Texas Signals New Era for Ercot
More coverage of the broader geothermal renaissance: Geothermal’s time has finally come https://www.economist.com/interactive/science-and-technology...

And i found this interesting: https://www.complexsystemspodcast.com/episodes/fracking-aust...

Seems fairly promising

RobinL··on Python is not a great language for data science
Out of the current options, I strongly agree - I even wrote a blog post! https://www.robinlinacre.com/recommend_sql/

But on the other hand, that's doesn't mean SQL is ideal - far from it. When using DuckDB with Python, to make things more succinct, reusable and maintainable, I often fall into the pattern of writing Python functions that generate SQL strings.

But that hints at the drawbacks of SQL: it's mostly not composable as a language (compared to general purpose languages with first-class abstractions). DuckDB syntax does improve on this a little, but I think it's mostly fundamental to SQL. All I'm saying is that it feels like something better is possible.

RobinL··on Python is not a great language for data science
I would argue that's about how the data is stored. What I'm trying to express is the idea of the programming language itself supporting high level tabular abstractions/transformations such as grouping, aggregation, joins and so on.
RobinL··on Python is not a great language for data science
I think a lot of this comes down to the question: Why aren't tables first class citizens in programming languages?

If you step back, it's kind of weird that there's no mainstream programming language that has tables as first class citizens. Instead, we're stuck learning multiple APIs (polars, pandas) which are effectively programming languages for tables.

R is perhaps the closest, because it has data.frame as a 'first class citizen', but most people don't seem to use it, and use e.g. tibbles from dplyr instead.

The root cause seems to be that we still haven't figured out the best language to use to manipulate tabular data yet (i.e. the way of expressing this). It feels like there's been some convergence on some common ideas. Polars is kindof similar to dplyr. But no standard, except perhaps SQL.

FWIW, I agree that Python is not great, but I think it's also true R is not great. I don't agree with the specific comparisons in the piece.

RobinL··on Nano Banana Pro
Thanks - this worked for me (some errors, some success).

Last week I was making a birthday card for my son with the old model. The new model is dramatically better - I'm asking for an image in comic book style, prompted with some images of him.

With the previous model, the boy was descriptively similar (e.g. hair colour and style) but looked nothing like him. With this model it's recognisably him.

RobinL··on Gemini 3
- Anyone have any idea why it says 'confidential'?

- Anyone actually able to use it? I get 'You've reached your rate limit. Please try again later'. (That said, I don't have a paid plan, but I've always had pretty much unlimited access to 2.5 pro)

[Edit: working for me now in ai studio]

RobinL··on UK gov to ban selling show tickets above face price
Here is an alternative idea: https://aeturrell.com/blog/posts/one-trick-to-stop-ticket-to...

The key idea is that only the original ticket buyer is eligible for a large rebate when attending the event. It prevents touting, but does not mean everyone who wants a ticket gets one.

Though in practice it is perhaps to techie, and in the end not dramatically different to what Glastonbury Festival does, which is that the ticket is only valid for entrance by the original purchaser, using photo id.

RobinL··on A new book about the origins of Effective Altruism
> many of EA's most notorious supporters.

The fact they're notorious makes them a biased sample.

My guess is for the majority of people interested in EA - the typical supporter who is not super wealthy or well known - the two central ideas are:

- For people living in wealthy countries, giving some % of your income makes little difference to your life, but can potentially make a big difference to someone else's

- We should carefully decide which charities to give to, because some are far more effective than others.

That's pretty much it - essentially the message in Peter Singer's book: https://www.thelifeyoucansave.org/.

I would describe myself as an EA, but all that means to me is really the two points above. It certainly isn't anything like an indulgence that morally offsets poor behaviour elsewhere

RobinL··on Britain's railway privatization was an abject failure
Salaries don't tend to be strongly correlated with bad working conditions or stress. In most industries (like software development) it's just supply and demand, and I imagine there are more people willing and able to work for £65k as a train driver than as a software developer. It's a bit different for train drivers because of the strong unions; my guess is that explains their high salaries more than lack of supply.

(Median total reward for TOC train drivers is £66,043) https://www.orr.gov.uk/sites/default/files/2022-10/review-of...

RobinL··on Britain's railway privatization was an abject failure
Is that true? My understanding is they command very high wages because their unions are strong and they have a lot of leverage: by striking they can impose extremely high costs on the wider economy (not to mention bad press for politicians).
← PreviousPage 3 of 18Next →