michaelmarkell

You're 100% right, thanks! Updated my comment.

That's approximately ~15.3M USD per person in the top 1% assuming 340 million americans

Not really. Take for example:

item, date, price

abc, 01/01/2023, $30

cde, 02/01/2023, $40

... 100k rows ...

subtotal. $1000

def, 03/01,2023, $20

"Hey Claude, what's the total from this file? > grep for headers > "Ah, I see column 3 is the price value" > SUM(C2:C) -> $2020 > "Great! I found your total!"

If you can find me an example of tech that can solve this at scale on large, diverse Excel formats, then I'll concede, but I haven't found something actually trustworthy for important data sets

IMO, a real solution here has to be hybrid, not full LLM, because these sheets can be massive and have very complicated structures. You want to be able to use the LLM to identify / map column headers, while using non-LLM tool calling to run Excel operations like SUMIFs or VLOOKUPs. One of the most important traits in these systems is consistency with slight variation in file layout, as so much Excel work involves consolidating / reconciling between reports made on a quarterly basis or produced by a variety of sources, with different reporting structures.

Disclosure: My company builds ingestion pipelines for large multi-tab Excel files, PDFs, and CSVs.

Danya was like the Mr. Rogers of chess. He had a way of making you feel accepted into the chess community even if you were a beginner, and was such a clear thinker. I strive to be more like him, and am devastated by this loss.

The way my company uses Clickhouse is basically that we have one giant flat table, and have written our own abstraction layer on top of it based around "entities" which are functions of data in the underlying table, potentially adding in some window functions or joins. Pretty much every query we write with Clickhouse tacks on a big "Group By All" at the end of it, because we are always trying to squash down the number of rows and aggregate as aggressively as possible.

I imagine we're not alone in this type of abstraction layer, and some type-safety would be very welcome there. I tried to build our system on top of Kysely (https://kysely.dev/) but the Clickhouse extension was not far along enough to make sense for our use-case. As such, we basically had to build our own parser that compiles down to sql, but there are many type-error edge cases, especially when we're joining in against data from S3 that could be CSV, Parquet, etc.

Side note: One of the things I love most about Clickhouse is how easy it is to combine data from multiple sources other than just the source database at query time. I imagine this makes the problem of building an ORM much harder as well, since you could need to build type-checking / ORM against sql queries to external databases, rather than to the source table itself

From the Readme:

Given a model with pre-trained transcoders, it finds the circuit / attribution graph; i.e., it computes the direct effect that each non-zero transcoder feature, transcoder error node, and input token has on each other non-zero transcoder feature and output logit. Given an attribution graph, it visualizes this graph and allows you to annotate these features. Enables interventions on a model's transcoder features using the insights gained from the attribution graph; i.e. you can set features to arbitrary values, and observe how model output changes.

The blog post: https://www.anthropic.com/research/open-source-circuit-traci...

Syncopate | NYC (Hybrid ~3d/week) | Full-time | Senior Full Stack Engineers / Focus on AI + Finance

Syncopate builds tools to help automate financial diligence and management of long-tail financial assets.

We've found product market fit with ETL/analysis tools for niche financial data, starting with music rights, and we're looking to build out our capabilities across more Excel + PDF-based workflows.

What we're looking for: A full-stack engineer with experience building data-heavy applications. Experience with analytics databases like Clickhouse and data pipelining is a plus. Required to be knowledgeable in Typescript.

Big bonus points for: 1) High agency (previously a founder or built side-projects to completion) 2) Some knowledge of finance 3) Skill in Rust

You can reach out to me here https://www.linkedin.com/in/michael-markell-377b4221a/ or via email (michael at syncopate dot ai)

More about Syncopate (geared towards our music rights segment): https://syncopate.notion.site/

In our use-case we have many gigabytes of PDFs that contain some qualitative data but also many pages of inline pdf tables. In an ideal world we’d be “compressing” those embedded tables into some text that says “there’s a table here with these columns, if you want to analyze it you can use this <tool>, but basically the table is talking about X, here are the relevant stats like mean, sum, cardinality.”

In the naive chunking approach, we would grab random sections of line items from these tables because they happen to reference some similar text to the search query, but there’s no guarantee the data pulled into context is complete.

It seems to me like chunking (or some higher order version of it like chunking into knowledge graphs) is the highest leverage thing someone can work on right now if trying to improve intelligence of AI systems like code completion, PDF understanding etc. I’m surprised more people aren’t working on this.

If I were to guess, most (adult) humans could not add two 3 digit numbers together with 100% accuracy. Maybe 99%? Computers can already do 100%, so we should probably be trying to figure out how to use language to extract the numbers from stuff and send them off to computers to do the calculations. Especially because in the real world most numbers that matter are not just two digits addition

Can someone with more expertise help me understand what I'm looking at here? https://crt.sh/?id=10106356492

It looks like Deepseek had a subdomain called "openai-us1.deepseek.com". What is a legitimate use-case for hosting an openai proxy(?) on your subdomain like this?

Not implying anything's off here, but it's interesting to me that this OpenAI entity is one of the few subdomains they have on their site

Archive link: https://web.archive.org/web/20240330143422/https://www.nytim...

Timing of moves

Artists do not get paid per stream on Spotify + many other DSPs. The platform sums up all of the ad revenue and divides it pro rata among all of the streamed artists. So the fraudulent streams dilute the pie for legitimate streams.

The Monty hall problem is a great example of something I’ve been educated into believing, rationalizing, whatever you want to call it…but I would still never claim I “understand it.” I think that’s maybe the source of disagreement here, there are many truly unintuitive outcomes of statistics that are not “understood” by most people in the most respectful sense of the word, even if we’ve been educated into knowing the formula, knowing how to come to the right answer, etc.

It’s like in chess, I know that the Sicilian is a good opening, that I’m supposed to play a6 in the najdorf, but I absolutely do not “understand” the Najdorf, and I do think it’s fundamentally past the limit of most humans understanding.

can you run the whole task as a postgres transaction? like if i want to make an idempotent job by only updating some status to "complete" once the job finishes.

From the article it seems like the content wasn’t entirely AI generated, it “digitally superimposed the faces of child actors onto nude bodies”, which makes this a lot worse than purely AI generated content

> which implies what casual mechanisms you're controlling for.

Without thinking terribly deeply about it, it seems like this could be modified to "what correlated mechanisms we're controlling for", in which case, yes I think if we controlled for the highly correlated variable of "does this cow live on a meat farm" with the mean age, we would be able to make broad claims about cattle lifespan without needing to be privy to the internal mechanisms of the agriculture industry

This reads like it was written by ChatGPT

Treatments like exercise also have high variance - depending on the circumstance and how diligently you implement it, it may have no effect, a big effect, or even cause injury in the worst case.

Insurance likes low variance outcomes because they’re easier to model accurately

I'm building https://royal.io, which helps turn music royalties into an asset class people can invest in. When investors (fans, mostly) "buy" a portion of a song on our platform, they become entitled to a percent of the recurring royalty revenue stream. It's one of the few applications of crypto that I think genuinely makes sense right now, and I especially like it because crypto is not the star of the show, it's just the backend implementation detail

I right a lot less now then I did as a kid, so maybe it’s about just staying sharp

But saying “I don’t want to be wildly unkind” implies that there is something about the author and their claim that is deserving of meanness. I think that is an unkind thing to say.

As a dev, you build on mainnet because that’s where the liquidity is. Your project will get less trade volume on any other chain (even compared to an EVM compatible chain like polygon, where transaction times are faster and gas fees are lower.)

The story may be different on Solana but it’s not really my area of expertise. My guess is many ethereum devs haven’t moved to solana because of its lesser decentralization.

Anecdotally I know a lot of ethereum devs are excited about building on layer 2s like zksync / arbitrum but there’s a learning curve and relatively little liquidity there too

No, In Ronin for a while there was no ether (they wanted free transactions) so they had a special rpc endpoint that could accept “free” transactions with gas priced at 0. They still process 4/9 of the transactions on ronin even though they introduced a paid gas now — RON

Feels like as a seller, I should report every other seller's similar products on fb marketplace to get mine more clicks...

The general advice is "do things that don't scale" so rather than trying to post about it and get some analytical data, just go talk to 20 people in your target market for an hour each and see what resonates with them, what problems they have etc -- all before building anything (except maybe a simple mockup if you feel it will get your point across better.)

I've just done the same. If you want to talk through strategy, I'd love to grab 20 minutes because I'm grappling with the same.

If you want to support < 5 person calls without things like hd screensharing, you can just do peer to peer webrtc connections.

Once you get above that, there’s a variety of FOSS you can self host, like jitsi and mediasoup (what pop.com uses.) There’s also some relatively cheap paid APIs like twilio and opentok.