RobinL

Stepping back a bit, having these on the road rather than cars or petrol scooters must be a big win even if they don't last as long as we'd like

I got several people wanting to send a courier last time I listed something on Facebook. Checking their pages, they were all from eastern Europe with no obvious connection to my city. Good to know the mechanics of the scam, I wondered what they were up to. Don't understand why Facebook couldn't have auto detected the messages though - seemed like a pretty major failure of marketplace that the majority of the messages I got were scams.

We're still using parquet. So we use the native duckdb format for intermediate processing (during pipeline execution) but the end results are saved out as parquet. This is partly because customers often read the data from other tools (e.g. AWS athena)

I'd be interested in hearing about experiences of using duckdb files though, i can see instances where it could be useful to us

Interesting. So what does that look like on disk? Possibly slightly naively I'm imagining a single massive file?

I submitted this because I thought it was a good, high effort post, but I must admit I was surprised by the conclusion. In my experience, admittedly on different workloads, duckdb is both faster and easier to use than spark, and requires significantly less tuning and less complex infrastructure. I've been trying to transition as much as possible over to duckdb.

There are also some interesting points in the following podcast about ease of use and transactional capabilities of duckdb which are easy to overlook (you can skip the first 10 mins): https://open.spotify.com/episode/7zBdJurLfWBilCi6DQ2eYb

Of course, if you have truly massive data, you probably still need spark

My son loves openttd. Fun fact: the original transport tycoon was primarily written in assembly by Chris Sawyer

It's actually a fourth power law, not a linear relationship. So trucks are polynomially more damaging. https://en.wikipedia.org/wiki/Fourth_power_law

With apologies for the X link, here is an example from Suno which felt very musical to me: https://x.com/sunomusic/status/1857501332560818342

Here's another example on the Suno website: https://suno.com/song/fc991b95-e4e9-4c8f-87e8-e5e4560755e7

Yeah, absolutely agree with that. Definitely has the potential to be particularly harmful in educational settings of users blindly trusting.

I guess it's just like many tools, they can be used well or badly, and people need to learn how to use the well to get value from them

I think this is pretty good advice.

I think often AI sceptics go too far in assuming users blindly use the AI to do everything (write all the code, write the whole essay). The advice in this article largely mirrors - by analogy - how I use AI for coding. To rubber duck, to generate ideas, to ask for feedback, to ask for alternatives and for criticism.

Usually it cannot write the whole thing (essay, program )in one go, but by iterating bewteen the AI and myself, I definitely end up with better results.

Same! (asking it to write code for the foss lib I maintain). This is immediately very useful.

Can anyone comment on the viability of using an external SSD rather than upgrading storage? Specifically for data analysis (e.g. storing/analysing parquet files using Python/duckdb, or video editing using divinci resolve).

Also, any recommendations for suitable ssds, ideally not too expensive? Thank you!

I use cursor (cursor.com) and it's fantastic

The whaling industry as depicted in Moby Dick is pretty high on my list of the most unbelievable things that really happened. Particularly the range of the boats and the method of catching the whales. Up there with the Spanish conquest of the Aztecs and how the East India company managed to dominate the entire country in terms of sheer implausibility.

In the short run, the trains are running anyway, so you might as well fill them up.

This is something that particularly frustates me in the case of buses the UK. They're ridiculously expensive (well, except for the temporary current £2 scheme), and empty. There seems to be a political consensus that we should have them, so it would make far more sense for them to be free in all instances, except when demand is so high they're full (e.g. rush hour)

By way of a single example, we've been migrating recently from spark to duckdb. Our jobs are not huge, but too big for a single 'normal' machine. We've gone from a 2.5 hour runtime on a cluster of 10 machines (40,vCPU total) to a 15 minute runtime on a 32vCPU single machine. I don't know for sure, but I think this is largely because it eliminates expensive shuffles and serde. Obviously results vary hugely depending on workload, and some jobs are simply too big even for a 192 core machine. But I suspect a high proportion of workloads would be better run on single large machines nowadays

I wonder what percentage of 'big data' jobs that run in clusters would now be far faster on a single big machine with e.g. duckdb rather than spark

To take one example of where this is valuable:

- Take some dense research paper or other material that is unsuitable for listening to aloud

- Listen to it (via NotebookLLM) whilst commuting/washing up or whatever

This way you'll have a big headstart on what it's all about when you come to read the details.

I imagine in future we'll see a version of this where the listener can interject and ask questions too, that feels like a potentially very powerful way to learn.

Yesterday the rain was torrential when I picked up my 5yo from school with my 2yo. I brought wellies and we walked up and down the streams of water running down the hill. The kids were more excited than anything we've paid for recently.

In general I'm very optimistic about the usefulness of AI to programmers but I can't think of much worse than an Excel sheet that's bundles a bunch of AI generated Python code from a quality assurance/correctness point of view!

Agreed. The sweet spot is people who have product owner skills _and_ can code. They are quickly developing superpowers. The overhead of writing tickets, communicating with the team and so on is huge. If one person can do it all, efficiency skyrockets.

I guess it's always been true to some extent that single individuals are capable of amazing things. For example, the guy who's built https://www.photopea.com/. But they must be exceptional - this empowers more people to do things like that.

I am cautiously optimistic. So much of building software is deciding what _should_ be built rather than the mechanics of writing code.

I you like coding because of the things it lets you build, then LLMs are exciting because you can build those things faster.

If on the other hand you enjoy the mental challenge but aren't interested in the outputs, then I think the future is less bright for you.

Personally I enjoy coding for both reasons, but I'm happy to sacrifice the enjoyment and sense of accomplishment of solving hard problems myself if it means I can achieve more 'real world' outcomes.

Another thing I'm excited about is that, as models improve, it's like having an expert tutor on hand at all times. I've always wanted an expert programmer on hand to help when I get stuck, and to critically evaluate my work and help me improve. Increasingly, now I have one.

Have to say, whilst this tech has some creepy aspects, just playing about with this my family have had a whole sequence of laughs out loud moments - thank you!

Does anyone know of examples of something like this that can be bought in the UK?

Just recently released version 4 of our free library for data deduplicating/linkage at scale. After four years working on it, finally getting to the stage where I'm really happy with both the performance and the API design.

https://github.com/moj-analytical-services/splink

The effect of this should be to reduce the price of used peletons by around $95 - i.e. effectively seller pays. I imagine that's the thinking behind this (ie it only affects the customer who's leaving the service), but it probably leaves a bad taste in the mouth of the buyer anyway, which doesn't feel like great business

Also worth noting there are benefits to exposure that are often not full unaccounted for when considering exposure: https://www.ed.ac.uk/news/2024/uv-rays-may-boost-health-in-l...

Overall perception: there are a lot of inefficient and ineffective charities, you have to be careful about who you give to if you want to have an impact

Ever given? Yes, I give through a regular deduction from payroll (I'm a giving what you can pledgee)

What keeps me committed? Knowing that I'm having an impact. I give mostly to GiveDirectly and find it almost impossible to read about the recipients and not want to give more: https://live.givedirectly.org/

She's got a great blog too: https://susan-stepney.blogspot.com/

Everything is a video. An image is the special case of length 1 frame