mritchie712

it depends... but yes, you could likely set it up in a way that matches (or beats) snowflake performance.

the compute would happen on your machine if you were running duckdb locally.

And yes, to get more compute, you'd want to spin up a VM.

What's cool is you can do both (run locally for small stuff, run on VM for heavy workloads).

Here's a step-by-step setup. It's using S3 and RDS, but I wouldn't be hard to swap in a local sqlite instead.

https://www.definite.app/blog/cloud-iceberg-duckdb-aws

> an earlier Opus was highly committed to animal welfare and they’re not sure why; other versions and models have not been

I caught that too, very interesting.

Many people treat their pets as part of their family. Many people make statements online like: "I like dogs / cats more than most people". So it's not surprising that this sentiment would be picked up in training data. I'd wager animals on average are thought of more highly on a site like Reddit than humans are. i.e. the average sentiment of a post on cats is more positive than a post about a human. Cats, in the training data, may be perceived as "more important" or "better" than humans.

It is surprising Opus holds on to this tighter than other models.

it's coming. they already have hive style parquet writes. Iceberg is more complicated than that, but it's certainly doable.

looks like this is one-shot generation right?

I wonder how much the results would change with a more agentic flow (e.g. allow it to see an error or select * from the_table first).

sonnet seems particularly good at in-session learning (e.g. correcting it's own mistakes based on a linter).

we were using it for our agent in https://www.definite.app/ and I've been expecting it to die for almost a year considering the lack of updates.

We switched over to https://ai.pydantic.dev/ which I really like. LLM agnostic and the team is very receptive to feedback.

LLMs are far more reliable at producing something like this:

    {
      "dimensions": [
        "users.state",
        "users.city",
        "orders.status"
      ],
      "measures": [
        "orders.count"
      ],
      "filters": [
        {
          "member": "users.state",
          "operator": "notEquals",
          "values": ["us-wa"]
        }
      ],
      "timeDimensions": [
        {
          "dimension": "orders.created_at",
          "dateRange": ["2020-01-01", "2021-01-01"]
        }
      ],
      "limit": 10
    }

than this:

    SELECT
      users.state,
      users.city,
      orders.status,
      sum(orders.count)
    FROM orders
    CROSS JOIN users
    WHERE
      users.state != 'us-wa'
      AND orders.created_at BETWEEN '2020-01-01' AND '2021-01-01'
    GROUP BY 1, 2, 3
    LIMIT 10;

The JSON compiles to SQL. Have you used a semantic layer? You might have a different opinion if you tried one.

the short answer: use a semantic layer.

It's the cleanest way to give the right context and the best place to pull a human in the loop.

A human can validate and create all important metrics (e.g. what does "monthly active users" really mean) then an LLM can use that metric definition whenever asked for MAU.

With a semantic layer, you get the added benefit of writing queries in JSON instead of raw SQL. LLM's are much more consistent at writing a small JSON vs. hundreds of lines of SQL.

We[0] use cube[1] for this. It's the best open source semantic layer, but there's a couple closed source options too.

My last company wrote a post on this in 2021[2]. Looks like the acquirer stopped paying for the blog hosting, but the HN post is still up.

0 - https://www.definite.app/

1 - https://cube.dev/

2 - https://news.ycombinator.com/item?id=25930190

no, those are two examples of many many possible circumstances. I intentionally made it two very specific examples so that was clear. Seems it wasn't so clear.

were you immediately more productive in Cursor specifically?

my point is exactly inline with your comment. The tools you get immediate value out of will vary based on circumstance. There's no silver bullet.

my stance is the opposite of all-or-nothing. The note above is one example. How much value you get out of CURSOR specifically is going to vary based on person & problem. The Python dev in my example might immediately get value out of o3 in ChatGPT.

It's not all or nothing. What you get value out of immediately will vary based on circumstance.

How useful the various tools will be depends on the person and the problem. Take two hypothetical people working on different problems and consider if, for example, Cursor would be useful.

IF you're a:

* 10 year python dev

* work almost entirely on a very large, complex python code base

* have a pycharm IDE fine tuned over many years to work perfectly on that code base

* have very low tolerance for bugs (stable product, no room for move fast, break things)

THEN: LLMs aren't going to 10x you. An IDE like Cursor will likely make you slower for a very long time until you've learned to use it.

IF you're a:

* 1 year JS (react, nextjs, etc.) dev

* start mostly from scratch on new ideas

* have little prior IDE preference

* have high tolerance for bugs and just want to ship and try stuff

THEN: LLMs will to 10x you. An IDE like Cursor will immediately make you way faster.

duckdb is primarily a query engine. It does have a storage format, but one of it's strengths is querying data where it already resides (e.g. a parquet file sitting in S3).

There are some examples[0] of enabling DuckDB to manage distributed workloads, but these are pretty experimental.

0 - https://www.definite.app/blog/smallpond

Databricks started in 2013 when Spark sucked (it still does) and they aimed to make it better / faster (which they do).

The product is still centered Spark, but most companies don't want or need Spark and a combination of Iceberg and DuckDB will work for 95% of companies. It's cheaper, just as fast or faster and way easier to reason about.

We're building a data platform around that premise at Definite[0]. It includes everything you need to get started with data (ETL, BI, datalake).

0 - https://www.definite.app/

> No they wouldn't.

haha, you're giving people way too much credit. Tons of people make bad software purchasing decisions. It's hard, people make mistakes.

> they're saving over 60% off of their BigQuery bill

how big is their data?

A lot of BigQuery users would be surprised to find they don't need BigQuery.

This[0] post (written by founding engineer of BigQuery) has a bit of hyperbole, but this part is inline with my experience:

> A couple of years ago I did an analysis of BigQuery queries, looking at customers spending more than $1000 / year. 90% of queries processed less than 100 MB of data. I sliced this a number of different ways to make sure it wasn’t just a couple of customers who ran a ton of queries skewing the results. I also cut out metadata-only queries, which are a small subset of queries in BigQuery that don’t need to read any data at all. You have to go pretty high on the percentile range until you get into the gigabytes, and there are very few queries that run in the terabyte range.

We're[1] built on duckdb and I couldn't be happier about it. Insanely easy to get started with, runs locally and client-side in WASM, great language features.

0 - https://motherduck.com/blog/big-data-is-dead/

1 - https://www.definite.app/

is this using pg_duck?

The "datalake" is becoming a bit of a commodity. It's getting pretty easy to spin one up yourself[0] using completely open source components.

Databricks and Microsoft (thru Fabric) are trying to build a complete data platform, i.e. ELT + datalake + BI

My bet with Definite (https://www.definite.app/) has been this is too hairy for a large company to do well and we can do it better.

0 - https://www.definite.app/blog/cloud-iceberg-duckdb-aws

Delta Lake is not catching on, but no worries, they bought Iceberg[0] (the competing standard).

I'm joking, but only a bit. Iceberg is open source (Apache), but a lot of the core team and the creator worked at Tabular and Databricks bought them for $1B.

0 - https://www.definite.app/blog/databricks-tabular-acquisition

supabase lured me in with built-in oauth, real-time, and some nice client side features in their JS lib, but I do worry about the latency sometimes.

It'd be a lot of work to run an apples to apples test with a Google Cloud Postgres db vs. Supabase and see what the difference is.

this is a tarpit idea I've fallen into multiple times. It's really hard to make money from a desktop SQL client, let alone now that DuckDB has a good, free UI.

we have a discount for startups, which would qualify many on HN

Yes, I've looked at them all. Most AI analytics tools are doing "text to SQL", but writing SQL is a small percentage of data work.

We built an entire stack so the agent can operate across that whole stack (e.g. create pipelines, model data, build reports, etc.)

Yes, we've had many bigquery / snowflake converts. The reality is, most companies don't have 100tb of data (which is what those platforms are optimized for). Motherduck has a good post[0] on this:

> There were many thousands of customers who paid less than $10 a month for storage, which is half a terabyte. Among customers who were using the service heavily, the median data storage size was much less than 100 GB.

I'm a fan of what motherduck is doing. We're building something different (opinionated, instant data stack), but yes, we both use duckdb under the hood.

0 - https://motherduck.com/blog/big-data-is-dead/

The best open source options are Airbyte and Meltano / Singer. But it's hard to keep them running. If you self-host them, you'll hit issues at least a few times a month which can each take a few hours to solve.

It's not like running Postgres which "just works". When you self-host Airbyte, you're still building a good bit.

I felt the same way about the cost of data tools. Paying $1,000 for Fivetran, $2,000 for Snowflake, $2,000 for Looker seemed crazy. We bundle all three for $500 / month at https://www.definite.app

there's going to be more consolidation in data tooling this year. Many of the stand alone tools raised too much money and no one wants to buy 5 really expensive tools to assemble a "data stack" anymore.

if you want a data platform that's built to work as one cohesive unit, we got you: https://www.definite.app/

Definite has a data lake, ETL, and BI in one app.

I'd guess a little north of $500 million.

- Census last raised $60M Series B at a $630M valuation (upper bound)

- Census’s estimated annual revenue is $31.6 million with ~200 employees.

- Median private-SaaS EV/ARR multiple is 7× (7 * 31 = 217 = lower bound)

- Hightouch raises $80M on a $1.2B valuation(at ~60× ARR)

- Twilio completes $3.2B acquisition of Segment at ~21× ARR (upper multiple bound)

Yeah, I was always curious why Fivetran didn't build this themselves when reverse-ETL started to take off.

I built a company[0], SeekWell, in this space (launched before Census), but was mostly focused on Sheets and Slack as destinations. SeekWell was acquired a few years ago too.

0 - https://seekwell.io/