Hacker News Reader

rorymalcolmon Sub-transactions throttled our Postgres-backed queue

Wrote this; the MultiXact system in Postgres is really pretty complicated but not something you come across _that_ often (unless you're running workloads like this) - it's worth understanding exactly the mechanics of what's happening when you do hit scaling issues like this and if something like the leasing we do here is possible.

rorymalcolmon We turned off Pub/Sub and nobody noticed

Disclaimer: I work at the incident company

This is one of those projects where the end goal is to build the scariest button possible. You cannot complete the project until you’ve pressed that button. You do all the work to build the button, then you have to press it.

Never not scary, never not eventually fun, but type 2 fun for sure.

rorymalcolmon Don't add a read replica until you've read this

Disclaimer: I worked with Johanna on this

I had a lot of fun “long tail whackamoling” our way through edge queries that our model of “this is eligible for a read replica” wasn’t accurately catching. Honestly just one of those “keep reading traces and working through them” development processes that just keeps drive a graph down and to the right.

rorymalcolmon The AI Innovator's Dilemma

Building higher-touch AI products that are applicable across the entire customer base of an existing enterprise SaaS provider on release does feel impossible.

It's very obvious when you look at the AI companies that have 'exit-velocity' in this space that they are playing in that they are either one of the following:

1. Targeting a need within that org that's narrow enough to be MVP-able enough across a diverse customer base

2. Targeting IC work directly

As we add more complexity here, it's going to be a long, long slog to market - even with model advancements.

rorymalcolmon Who watches the watchers? How we page ourselves if incident.io goes down

(Full disclosure, I work at incident.io!)

We recently released our On-call product, and as part of that, had to think a lot about redundancy and 'failing safety'.

Here's how we achieve it - and how we're thinking about it. Interested if any other examples of this exist in the wild - I'd love to know more about how eg: Datadog achieve this.

rorymalcolmon Building a multi-platform on-call mobile app with React Native in 2024

Yes - it is!

rorymalcolmon Building a multi-platform on-call mobile app with React Native in 2024

(Full disclosure, I work at incident.io!)

We recently released our On-call product, and as part of that, had to build a mobile app.

We'd noticed there's been some debate recently regarding React Native, and if you can build 'native tier' apps using it. We'd seen a lot of this but as part of an initial hackathon tried Explo / RN / Nativewind and were completely blown away by both the quality of app we could produce and the developer experience that let us do this quickly.

If anyone's in this space / thinking about making a technical decision here I couldn't recommend this stack strongly enough!

rorymalcolmon Building and scaling Notion's data lake

Found this comment (from Fivetran's CEO, so, with that in mind) regarding this article enlightening regarding the costs they were facing here https://twitter.com/frasergeorgew/status/1808326803796512865

rorymalcolmon SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue

Were Prequel using RaabitMQ to stay cloud platform agnostic when spinning up new environments? Always wondered how companies that offer managed services on the customers cloud like this manage infrastructure in this regard. Do you maintain an environment on each cloud platform with a relatively standard configuration, or do you have a central cluster hosted in one cloud provider which the other deployments phone home to?

rorymalcolmon TypeScripting the technical interview

Having the type system this complicated is mostly for library builders, makes the developer experience of tools like tRPC, Zod and Prisma possible. An engineer writing business logic in TypeScript will probably never have to learn how to write (or even read tbh) complex TypeScript signatures, but benefit significantly from the solutions the type system complexity is a necessary precursor for.

rorymalcolmon Databases in 2022: A Year in Review

I think it makes sense in the context of the poential of plain old SQL (and additions such as dbt) winning out vs Python or other general purpose programming solutions as the analytics workbench (although this is controversial, there's a school of thought that backs it).

rorymalcolmon Show HN: Airbyte, data integration platform with 300+ open-source connectors

Totally agree - in fact from the issues pages it seemed like there had been a conscious decision from Airbyte not to support a Terraform provider and instead build out Octavia, which as you say leaves a lot to be desired. Deploying Airbyte is still far too hard, I'd be happy to pay for an 'on prem' version like Gitlab where it's a bit more managed. I also think the failure modes for Airbyte on prem are too hard to debug in comparison to other data 'MDS' tools.

rorymalcolmon Show HN: Airbyte, data integration platform with 300+ open-source connectors

Switched to Fivetran after experiences with Airbyte, operating Airbyte was extremely tedious, even primetime connectors had bugs that I personally had to fix, we keep Airbyte around for the "long tail" of SaaS tools that Fivetran doesn't have a connector for but managing an Airbyte deployment in a stable manner for the long term was not easy.

rorymalcolmon Database drivers: Naughty or nice?

Which Lambda competitors (which aren't v8 isolate based) don't have cold starts?

rorymalcolmon How PlanetScale Boost serves SQL queries faster

They acknowledge it is based off Noria[1] in the longer blogpost

[1] https://pdos.csail.mit.edu/papers/noria:osdi18.pdf

rorymalcolmon Deno 1.28: Featuring 1.3M New Modules

It is popular in newgen tooling but I feel its more a function of the slowness of `tsc` than anything else. I get why `tsc` is slow, and have a lot of respect for the team and the constraints they work under but I can't help but feeling if we were to get a faster type checker (potentially `stc` from the creator of `swc` which also does this? https://github.com/dudykr/stc), this choice would be less popular. However if I'm honest I don't know if TypeScript's type system means that there are some natural constraints on how fast it can be validated.

rorymalcolmon $5.6B cloud company Fivetran acquired its way to survival

https://technical.ly/startups/dbt-labs-series-c/ They changed their name semi-recently to dbt Labs

rorymalcolmon Why we’ve decided to decommission Gov.uk PaaS

I feel the biggest PaaS offerrings in Europe are internal deployments within the enterprise!

rorymalcolmon Ask HN: What Happened to Big Data?

Those algorithms and improvements in large data processing got bundled away into a platform/infra layer a developer or user interfaces with unaware of what's going on in the background to produce the results they want.

rorymalcolmon CircleCI Down

Hosting any significantly scaled CI/CD system is very difficult work and since every team in your organisation usually touches it you often have a large blast radius. I've seen downtimes take a long time to recover from even when self hosted, compliance is also hard. It's a good candidate for SaaS IMO.

rorymalcolmon A magical AWS serverless developer experience

You can trivially resolve this with Organisations now, multi-account is incredibly common even within environments, it's a smart isolation boundary.

rorymalcolmon It’s fine, Rewind: Revert a migration without losing data

If you drop a 'title' column from a users table, for example, you can revert and have the title column reappear with the dropped data, new users added during this time (while the column was dropped) will not have a title.

rorymalcolmon Epic Games acquires Bandcamp

They aren't shutting down? Bandcamp's model is so directly connected to DRM free lossless downloads it'd be an incredibly bold decision to remove it.

rorymalcolmon State of JavaScript 2021

I think with React the core functionality / idea is much more simple than you'd imagine. The complexity evolves out of the add-ons and library situation, some do simplify the situation though, I find Next.js React apps incredibly powerful and easy to build on top of.

rorymalcolmon Show HN: No-code alternative to Retool, Appsmith, Internal, etc.

This is retool, retool sometimes gets marketed as 'nocode' but really benefits from past developer experience, I think you could build exactly what you want using it.

rorymalcolmon Gov.uk Platform as a Service

Totally agree, an unbelievably good service that does what it says on the tin in the best and most intuitive way.

rorymalcolmon Gov.uk Platform as a Service

This is part of a wider suite of tools developed and run by the Government Digital Service including Notify (lets you send texts, letters and emails, https://www.notifications.service.gov.uk/) and Pay (lets you take inbound payments quickly, https://www.payments.service.gov.uk/).

rorymalcolmon TypeScript Features to Avoid

NestJS makes pretty heavy use of decorators and I really dislike it because of it, it reminds me of reading Spring Java code.

rorymalcolmon BreakingFormation: AWS CloudFormation Vulnerability

https://twitter.com/colmmacc/status/1481670721449385984

There appears to be a denial of components of this from senior AWS figures, not a blanket denial but I think some of the statements could be a potential overreach.

rorymalcolmon Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

This is just not true for Terraform at all, they do not aim to be multi cloud and it is a much more usable product because of it. Resource parameters do not swap out directly across providers (rightly so, the abstractions they choose are different!).