sa46

Does anyone have experience using client-side DuckDB with WASM? It looks really compelling to ship analytics to the browser by running DuckDB over parquet files.

My most popular StackOverflow question is that phenomenon exactly: https://stackoverflow.com/questions/58277794

No root cause after 3 years. The best guess is the overhead of IO somewhere in the stack.

Link? Sounds neat.

I like the mnemonic that five 9s is five minutes a year.

10x seems extreme. ESR went from 14k lines in Python to 21k in Go. http://www.catb.org/~esr/reposurgeon/GoNotes.html

Would this speed up capturing stack traces? The use case I have in mind is stack traces for logging. The Zap logger has a mild warning [1] about stack traces being relatively expensive.

[1]: https://pkg.go.dev/go.uber.org/zap#Stack

I think the problem is there are too many trade offs Postgres would have to make.

- should soft-deleted rows move to another partition to avoid bloat?

- what’s the column name and type (boolean, timestamp, tstzrange)?

- what’s the syntax for a soft delete versus hard delete?

- what’s the behavior of updating a soft deleted row?

- do soft deleted keys prevent inserts with the same key?

Bitemporal tables mentioned in the thread are a more general solution. Alternately, you could use a statement-based trigger to prevent deletes to the table.

Helps reduce storage space by better aligning columns to reduce padding.

https://www.2ndquadrant.com/en/blog/on-rocks-and-sand/

One pattern I happily use is to capture the error into a named return error, using a multi-error if the named error is already non-nil.

    func fetchFoo(ctx context.Context) (_ Foo, mErr error) {
        conn := pool.Acquire(ctx)
        defer errs.Capture(&mErr, conn.Close, "close conn for fetch foo")
    }

I have a couple of useful variants:

        defer errs.CaptureLog(logger, conn.Close, "close conn")
        defer errs.CaptureT(t, conn.Close, "close conn")

You might like the book, Influence: The Psychology of Persuasion. I found the book helpful to reframe disengaging with sales folks from:

Before: "I'm acting uncharitably by not reciprocating attention."

After: "I recognize this person is attempting to influence me. I'm choosing not to engage."

> you'd still basically always want to make the changes incrementally, system by system, and with a sensibly sequenced rollout plan rather.

Depends. It's significantly faster to deploy everything at the same time and accept that unlucky requests might end up in a weird state than to safely sequence changes.

In SRE phrasing, I'm choosing to spend our error budget to maximize change velocity by giving up on compatibility during deploys by skipping a multi-stage rollout plan. In return, I can condense a rollout to a single commit and deploy. A 99.9% availability target yields up to 86 seconds per day to pretend that deploys are "atomic".

When do you find reusing a promise useful? I know one canonical example is a loadable hashmap as in Concurrent Programming in Java, but I was curious if you knew of other examples.

> Doesn't Go have a similar serialization/deserialization library?

There isn't anything as flexible or widespread as serde.

> In this case, what about stored procedures?

That works but moves logic out of the query and into the DB. Keeping logic in the query is nice to keep server releases independent. The lifecycle of app (and query) code is easier to manage than DB migrations.

> The trouble with templating is that dynamic queries are hard to do in a type-safe manner.

Where-clauses and order-by-clauses are straightforward since the structure of those clauses doesn't affect the structure of the query results. Group-by and table clauses are troublesome since a query might work during codegen but fail at runtime.

Cornucopia is neat. I wrote a similar library in Go [1] so I'm very interested in comparing design decisions.

The pros of the generated code per query approach:

- App code is coupled to query outputs and inputs (an API of sorts), not database tables. Therefore, you can refactor your DB without changing app code.

- Real SQL with the full breadth of DB features.

- Real type-checking with what the DB supports.

The cons:

- Type mapping is surprisingly hard to get right, especially with composite types and arrays and custom type converters. For example, a query might return multiple jsonb columns but the app code wants to parse them into different structs.

- Dynamic queries don't work with prepared statements. Prepared statements only support values, not identifiers or scalar SQL sub-queries, so the codegen layer needs a mechanism to template SQL. I haven't built this out yet but would like to.

[1]: https://github.com/jschaf/pggen

I’ve been meaning to start using JetBrains http files as integration tests with their recent free-as-in-beer jar [1]. There was also an open-source Go implementation floating around that I can’t find.

[1]: https://blog.jetbrains.com/idea/2022/12/http-client-cli-run-...

Slightly off topic, would someone mind providing a down to earth summary of the benefits (or disadvantages) of dbt?

To ape the infamous iPod and Dropbox middlebrow dismissals: dbt looks like a control loop with a nice UI to make a database schema match templated SQL files.

More seriously, at what level of maturity would a company find dbt useful and which teams benefit the most?

I wrestled with a graph traversal problem and Postgres docs for a hot minute before realizing that the "recursive" working set can only access the results of the previous iteration. With that in mind, I took a crack at your problem with the following approach:

1. Build the JSON from the bottom up starting at the deepest path.

2. At each step, handle two cases: a) build the parent of the working set, and b) collect any sibling paths at the same depth.

3. The last step aggregates all children together since all paths descend from '/'.

Solution: https://www.db-fiddle.com/f/qTcG5BUjM8GzuzBfHamnVJ/4

There was a fair bit of trickiness. I used a CTE in the recursive step to reference the recursive table multiple times and to enable aggregation in the recursive step. I used a few helper functions for my own sanity.

> What does single-tenant SaaS even mean?

I'll attempt to explain to clarify my own understanding.

SaaS is a delivery mechanism. Tenancy is an isolation model. To your broader point, the isolation model is implementation specific.

Multi-tenant SaaS means a single deployment for all tenants, and the data is delivered over the internet.

Single-tenant SaaS means a separate deployment for each tenant. I think common usage means a separate database per tenant. Single-tenant can also include entirely new infra for each tenant with private networking which is what Gitlab describes.

HN thread on single-tenant DB experiences with lots of useful tidbits: https://news.ycombinator.com/item?id=23305111

> I can create them all with one round trip rather than multiple.

I see. I'd lean towards using common table expressions. The first insert statement returns the primary key and other inserts can depend on the key. I do understand it's not a panacea and composing queries can be problematic.

> a sequence generator for PKs, it becomes a resource around which there is contention

As I understand, Postgres sequences don't block which leads to a different problem [1].

[1]: https://news.ycombinator.com/item?id=27843084

What's the benefit of having the primary key before the row is stored in the database?

Maybe concurrent inserts to multiple data stores so you don't need to wait for an initial ID from the database. You'd have to trust the client to give you a “good” UUID as well as the normal distributed problems of one of the RPCs failing. I know both Twitter (called Snowflake iirc) and Google have unique ID services for this use case.

Is there any tooling you use to squash the commits and use the PR message?

Because transactional workloads (many small reads and writes that require consistency) look different from analytical workloads (giant reads with a small number of columns with looser consistency guarantees).

OLAP databases typically use a column store which is amazing for reading a subset of columns because of much better compression and use of vectorized execution as opposed to Postgres’ tuple-at-a-time execution. The tradeoff is it’s expensive to update a column store since you have to rewrite a chunk of the column at least.

Postgres is moving towards better OLAP. AlloyDB is a recent commercial DB that swaps out the Postgres storage engine to better support OLAP.

That said, you can coerce Postgres into doing a reasonable job at OLAP for a surprising amount of data.

I'm also confused.

2. Agreed. 150 ms is extraordinarily slow for a point lookup from a single table. A simple lookup should take around 1 ms since it'll be cached in memory.

3. Also agreed. The actual database time looks to be the first flame. Hovering over it shows PG::Connection::exec which accounts for 2.5% of the time.

I was curious about date parsing and dug up the source [1]. Seems like you could gain a ton of speed back by using a postgres specific timestamp parsing routine. In Go, it's 40 lines [2]

[1] https://github.com/ruby/date/blob/d21c69450a57a1931ab6190385...

[2]: https://github.com/jackc/pgx/blob/a968ce3437eefc4168b39bbc4b...

Also former infantry: To expand the point, in US Army doctrine, an obstacle has one of four effects, block, turn, fix, or disrupt.

Typically, you use obstacles in engagement area development as part of a defense. An obstacle that's not observed isn't valuable because it's easy to circumvent. You can probably use a bulldozer (guessing, I was light infantry, not a tanker) to breach the dragon's teeth relatively quickly so the obstacle doesn't have the intended effect. However, it's much harder to breach the dragon's teeth if someone is shooting at you. Now you need an armored bulldozer, or enough suppressing fire to cover the breach.

I like protobufs but I was also disappointed at the JS protobuf options. I disliked both the JS object representation and RPC transport.

grpc-web in particular requires an Envoy proxy which seems absurdly heavyweight. I ended up using Twirp because Buf connect wasn't yet released or planned.

I rolled my own JS representation. The major differences from Connect:

- Avoid undefined if the message is not present on the wire and use an empty instance of the object instead. For recursive types, find the minimal set of fields to initialize as undefined instead of empty.

- Transparently promote some protobuf types, like google.protobuf.Timestamp to a proper Instant type (from js-joda or similar library). This makes a surprisingly large difference on reducing the number of jumps from the UI to the API.

I prefer reading proto files with services over OpenAPI yaml. Here's the pet store example to compare.

- https://github.com/project-flogo/grpc/blob/master/proto/grpc...

- https://github.com/OAI/OpenAPI-Specification/blob/main/examp...

I also went down the bespoke path after briefly investigating the official Go generator. I ended up forking https://github.com/deepmap/oapi-codegen. The official generator felt very "Java".

Sysadmin 101 doesn't involve separating the dynamic portions of similar, but unstructured log lines to dramatically improve compression and search performance.

> Zstandard or Gzip do not allow gaps in the repetitive pattern; therefore when a log type is interleaved by variable values, they can only identify the multiple substrings of the log type as repetitive.

> Curious wouldn't `go run` give you the same?

We use protobufs and pggen. Bazel transparently manages the codegen from proto file to Go code.

> would you say your experience with golang IDEs / gopls is degraded?

Yes, that's our biggest pain point with Go and Bazel. I haven't been able to coax IntelliJ into debugging Bazel managed binaries. To enable IntelliJ code analysis, we copy the generated code into the src directory (with a Bazel rule auto-generated by Gazelle) but don't add it to Git.

I've tried the IntelliJ with Bazel plugin a few times but I've always reverted back to stock IntelliJ.

Old comment with some details: https://news.ycombinator.com/item?id=28472062

In addition to the techniques in the linked comment, I'm currently investigating managing the entire Postgres data dir as a Bazel output directory. That way, we avoid executing the schema for every test. Instead, we'd start a new Postgres cluster against the existing data dir without re-executing the schema or creating from a template DB.