With something like service_binary, how do you wire up services? For example, If main.cc depended on Redis, how would you get the host and port of the Redis dependency from main.cc?
1,425 karma · joined December 2, 2015
Paratrooper to programmer.
http://github.com/jschaf
hn@schafer.dev
How you setup LISTEN and NOTIFY is app dependent. In a multi-tenant database, you could have 1 NOTIFY channel per tenant.
As you scale, you probably do something that listens in a smarter way, maybe 1 Go channel per client with a single LISTEN instead of 1 database connection per client. The downside is that now the app code is responsible for tenant isolation instead of the database.
The common workaround is to use a connection pooler like PGBouncer so that clients reuse connections. This approach doesn’t work for LISTEN because typically a client will listen for its entire lifecycle so you can’t share connections in a pool.
Not having 64 bit integers is a large annoyance. There’s workarounds but none of them are especially ergonomic.
The expression n | 0 is a signed 32 bit int. The expression n >>> 0 is an unsigned 32 bit int.
In my experience it’s a pretty good heuristic. It fails badly if the organization knows what pieces will be inspected (queue Goodhart). It can also be fantastically inefficient if there’s a ton of checks or the checks are ambiguous.
Being a stickler at inspections sets a useful precedent that organizations will do a better job for you in the future.
Napoleon took advantage of this during his first inspection of a combined arms unit (Napoleon was an artillery officer). He inspected the shit out the artillery unit first. Then he could be reasonably confident the light infantry and Calvary units would take the inspection seriously.
Even if you do get it to work, you usually have to wait for the keypress timeout to detect the command. The sequence is something like:
- Enter insert mode.
- Press "j"
- Nothing appears until 40 ms later, when keypress timeout passes and the logic falls back to insert a "j" character.
I think Vim and Emacs use a fancier undo-mechanism, where they always insert the "j" and then delete the "j" if a command completes before the timeout in order to hide the latency.
INSERT INTO bq_table SELECT * FROM EXTERNAL_QUERY('');
- If you write the same message multiple times, protobuf implementations should merge fields with a last write wins policy (repeated fields are concatenated). This includes messages in oneofs.
- For a boolean array, you're better off using a packed, repeated int64 (if wire size matters a lot). Protobuf bools use varint encoding meaning you need at least 2 bytes for every boolean, 1+ for the tag and type and 1 byte for the 0 or 1 value. With a repeated int64, you'd encode the tag and length in 2 varints, and then you get 64 bools per 8 bytes.
- Fun trivia: Varints take up a max of 10 bytes but could be implemented in 9 bytes. You get 7 bits per varint byte, so 9 bytes gets you 63 bits. Then you could use the most significant bit of the last byte to indicate if the last bit is 0 or 1. Learned by reading the Go varint implementation [2].
- Messages can be recursive. This is easy if you represent messages as pointers since you can use nil. It's a fair bit harder if you want to always use a value object for each nested message since you need to break cycles by marking fields as `T | undefined` to avoid blowing the stack. Figuring out the minimal number of fields to break cycles is an NP hard problem called the minimum feedback arc set[3].
- If you're writing a protobuf implementation, the conformance tests are a really nice way to check that you've done a good job. Be wary of implementations that don't implement the conformance tests.
[1]: https://github.com/protocolbuffers/protobuf/tree/master/conf...
[2]: https://github.com/golang/go/blob/master/src/encoding/binary...
[3]: https://en.wikipedia.org/wiki/Feedback_arc_set#Minimum_feedb...
Given that there’s multiple companies that exist to scale out Postgres, someone sees business value.
Here's a list of more advanced functionality built into Intellij that would be difficult to replicate:
- Bullet-proof navigation and advanced refactoring that works every time across a wide variety of languages.
- Database integration: syntax highlighting for SQL code in strings, runnable sql, database introspection
- Debugging: consistency and polish across languages, not relying on external tools that might not be installed, visual debugger
Text-editors struggle with the "integrated" aspect of an IDE. It's not terribly difficult to coerce text editors to perform well for a single language. It's much harder to integrate plugins and provide a consistent experience.
There's good reasons to use text editors (learn how it all works, extreme customization, magit, lightweight) but there's many development tasks Intellij is just better at.
At a meta-level, using unethical as a qualifier seems like an attempt to bolster an argument without having to provide a logical argument. I think most discussions are cleaner without broaching the thorny topic of ethics. Such discussions usually devolve into ideological battles, which by HN guidelines, "trample curiosity".
My guess is that the lag is a consequence of the single version policy in the monorepo. Meaning, you need to upgrade all dependencies on IntelliJ without breaking Google engineer workflows.
Would you mind enumerating them? I have an idea of what they are but curious about other perspectives.
My sense is that code sharing is not that common between a Typescript frontend and backend. You mostly need generated request/response data types but I don't think there's that much shared behavior because you can't import any of your backend-y logic (database, auth, external APIs) transitively into your frontend.
I think primary gain is what you've hinted at: 1 ecosystem and it's easier to onboard fullstack devs.
For projects I lose interest in, I unsubscribe from everything and either mark the project as maintenance mode or try to find a maintainer. To find maintainers, I've had good success granting write access to anyone who lands a few good PRs and telling them they have free reign over the code.
Does grpc-web work well? Is there a way to skip the proxy layer and use protobufs directly if you use websockets?
[1]: https://github.com/grpc/grpc-web [2]: https://github.com/grpc/grpc-web
Maybe number of air squats in 2 min?
Decentralization can help with sprawl (aka scale) but it makes software more complex so it’s not a way to tackle complexity. Decentralization is only tolerated because, at some point, it’s the only method left to scale. As an example, distributed databases are significantly more complex to build than a single node database.
You’ve also ignored all of the downsides of using your OLTP database for OLAP queries. The most important is that adhoc queries can impact production. In my experience, that cost dwarfs the costs of maintaining an ETL pipeline as soon as you have your first outage. Another downside is that with a federated, pull based model you need to keep the mapping (FDW definition) up to date. In essence, the maintenance cost moves from an ETL pipeline to an FDW definition.
- Explain plans for slow queries, via auto_explain in Postgres. You could get really fancy here and convert the pieces of an explain plan (Aggregate, ModifyTable, FullTableScan) into proper distributed traces. Hard to tell, but it looks like GCP might offer that.
- Various errors logs associated with queries that require more detail than a SQL state code. For example, lock timeouts.
- Debugging, especially root cause analysis when you're trying to figure out why something broke. The trace IDs help build up context.
The primary benefit of pggen and sqlc is that you don't need a different query model; it's just SQL and the tools automate the mapping between database rows and Go structs.
Datadog is tracing the request from the application akin to something like
openTrace("Query Name", queryStr)
db.Query(queryStr)
closeTrace()
I would love for a better mechanism than SQL comments for distributed tracing all the way to the database.
Strong disagree. Uber is better in pretty much every way over taxis. It's hard to overstate how much better Uber is compared to taxis. It's (generally) cheaper, cleaner, more predictable, and more pleasant.
- Outside of major cities, you can order a pickup but there's no guarantee that anyone will show up. I've had taxis cancel after waiting for 40 minutes.
- Cleanliness in taxis ranges from meh to horrifying.
- Manipulative techniques like pretending that the credit-card machine stopped working to force a cash transaction.
- In Vegas taxis, there's an obnoxious screen 3 inches in front of your face.
> I don't think I've really ever had any issues with Uber/Lyft, aside from I wish I could rate drivers honestly without worrying about ending their career.
The big difference between uber and taxis is that uber drivers have a direct incentive to improve via ratings. The incentive for taxi drivers to improve is much more indirect.
Join us as the founding engineer at our two-person, fully funded startup. We're looking for a senior frontend engineer well-versed with the intricacies of building large React applications. You will develop and architect a greenfield application from the ground floor and help shape a world-class engineering team.
SimpleCircle helps craft brewers distribute their products to local points of sale, with an operations management platform that automates clerical chores, while supercharging sales with superior mobile tools and product analytics. Our customers look to us to take care of a large chunk of running the business side of their breweries so they can focus on the beer.
Tech stack:
- React & Typescript
- Go backend using gRPC
- Postgres
We're based in SF, but remote until COVID conditions allow us to be in-person.
hn.careers@simplecircle.io
- App Engine has a bunch of weird limitations and slow deploy times. Qualitatively, it feels like the spotlight has moved on.
- Running my own compute instances felt like reinventing Kubernetes, especially once you roll your own deploy mechanism and throw load balancing in the mix. I also don't buy that it's easier.
- Cloud run is promising but for database heavy apps it's a non-starter.
- GKE was pretty smooth. It feels like it gets a lot more love than App Engine. The UI was functional with lots of depth. Once I push a docker image, GKE updates the nodes to serve the latest version. Load balancing was a matter of ~3 yaml files at 10 lines a pop.
Interesting. I'm looking Cloud SQL right now and the advice seems to lean in the opposite direction: use public IPs for ease of connecting. Can you quantify the decrease in latency? All I can find is bits about reduced network hops.
To be fair, it looks like GCP supported Postgres 13 (Nov 5, 2020) before AWS did (Nov 27, 2020) and AWS currently marks Postgres 13 as a preview. Maybe GCP had a large initial engineer-cost to support multiple versions of Postgres and now the incremental cost to add new versions is small?
> It's just a lot of money to pay for a crashy, outdated version of Postgres.
Have you looked at other options? I'm evaluating GCP SQL and the comments in this thread are scary. Seems like Aiven might be a good way to go. I've also briefly looked at CrunchyData's Postgres Operator [1] for Kubernetes but it's a lot of complexity I don't really want.