HNHacker News
TopNewBestAskShowJobs

lawrjone

907 karma · joined June 18, 2015

submissionscomments
lawrjone··on GitHub Is Down
I wouldn't say this applies to cloud providers, they have a very different business on their hands.

But for SaaS in general I think the trend is noticeable? Twitter led the way with massive layoffs in engineering often in the roles around reliability. The industry as a whole have aimed to cut costs however possible, and reliability/ops is usually seen as a cost-centre that gets hit hard.

I've watched this in my own space (start/scale-ups and larger companies, I work in incident response tooling) as people start talking very differently about reliability and engineering investment. You hear "do more with less" about five times every day and spend that was previously greenlit by default around reliability/redundancy is under much more scrutiny now.

I see this as a silent mirror of the reduction in open-source efforts from companies now there's been a refocus on business impact and bottom-line.

lawrjone··on GitHub Is Down
I'm not totally certain you can attribute it to this. I know there's a bunch of work going on to migrate things into Azure after the MS acquisition but it feels like more of an industry trend that we cut engineering spend at the cost (often) of lower quality output and outages like these.

GitHub laid off 10% of their staff in 2023 and like you say, won't have slowed down to account for that.

Ignoring all that though... other than Copilot, what big feature changes have you seen in GitHub? My experience of using their product has been broadly unchanged for years.

lawrjone··on GitHub Is Down
The industry made a general decision after ZIRP ended to deprioritise availability after years of historic levels of engineering investment.

It's no surprise we're now feeling those effects but damn, GitHub and other services like Slack have been really bad lately.

lawrjone··on Context should go away for Go 2 (2017)
This guidance is actually super important, as contexts are expected to be modified in a code flow and apply to all functions that are downstream of your current call stack.

If you store contexts on your structs it’s very likely you won’t thread them correctly, leading to errors like database code not properly handling transactions.

Actually super fragile and you should avoid doing this as much as is possible. It’s never a good idea!

lawrjone··on Context should go away for Go 2 (2017)
Yeah most people talking here are unlikely to have worked on large scale Go apps.

Managing a god-level context struct with all the fields that ever could be relevant and explaining what they mean in position independent ways for documentation is just not scalable at all.

Import cycles mean you’re forced into this if you want to share between all your packages, and it gets really hairy.

lawrjone··on Who watches the watchers? How we page ourselves if incident.io goes down
Author here!

It’s a fun problem to solve and one I’ve come across before when trying to alert on your monitoring tool being down, but slightly different when it’s your product.

Hopefully interesting if you’ve hit similar puzzles before.

lawrjone··on Observability as a Superpower
I’m one of the engineers on the team with Sam and can vouch for how impactful all the work he’s done has been to creating a great Observability setup.

Well worth a read!

lawrjone··on Upgrading Uber's MySQL Fleet
Yeah I found this really off putting: it’s not possible for you to have several goals that are all ‘paramount’, and the word ‘seamless’ adds nothing in every place it appears!

I wish it didn’t turn me off the content as much as it does but it’s very jarring.

lawrjone··on Show HN: I am building an open-source incident management platform
Don't these people already have the option of running open-source tools like Netflix's Dispatch or Monzo's Response? If they have the open-source options why would they pay for the hosted services?
lawrjone··on Learn One Thing at a Time
Thanks for the repost!
lawrjone··on Incident.io On-Call
That's for posting this! I'm one of the engineers on the team who built On-call, so if you have any questions about how we did it (like how we handle redundancy, observability, etc) then I'd be well up for answering.

It's been an intense 9 months of work but think anyone on-call is going to really love this.

lawrjone··on Almost every infrastructure decision I endorse or regret
Fwiw as a medium spend (250k to eventually grow into 1M+ year) GCP customer we had the same deal with product roadmaps shared up-front under NDA etc.

And never did I miss something in GCP that I could find in AWS. Not sure the breadth is adding much compared to a simpler product suite in GCP.

lawrjone··on Lessons learned from building our first AI product on OpenAI
Not the author but a colleague of Milly's: it's been awesome watching the team build this and everything they've learned along the way.

I think these features are a great example of how AI can fit alongside expert human work without necessarily displacing it, just helping people with the tasks they'd otherwise be distracted by doing. Writing incident summaries is a great example of work that should never be done by a human but is valuable and should happen, but AI can entirely remove the need for anyone to consider.

There's loads of opportunities for other products to add features like these and this post gives a bit of colour about how!

lawrjone··on A modern data stack for startups (2022)
It would work for most of our business reporting, but BigQuery is so good at what it does and cheap that it doesn't make much sense to avoid it.

We also have some parts of our dataset that are complicated to compute.

An example is that we estimate the hours that people have spent working on incidents, with a method that we describe here: https://incident.io/guide/insights/workload

That requires us to do some expensive cross-joining of every action ever taken in an incident from messages sent to the channel to GitHub PRs being merged. We could make this incremental and optimise it for performance but using BigQuery by default means we don't need to worry yet and can leave the optimisations for when we're bigger and the engineering resource we'd dedicate wouldn't detract as much from customer-focused work.

lawrjone··on A modern data stack for startups (2022)
I don’t think we did consider this, probably because we have a preference to buy instead of build with tools like these and prefer a team who can respond to our feedback.

It looks like a promising tool though! I’m sure we’ll blog about our experience with the new tooling once we’ve moved over, the team will no doubt have a lot to say about it.

lawrjone··on A modern data stack for startups (2022)
This sounds really awesome, I will note that I put this data stack together by myself in about 1 week when we were just ten people in the company.

Obviously very different resource constraints than Meta, so worth considering which situation you may be closer to when picking an implementation plan.

lawrjone··on A modern data stack for startups (2022)
Data like this allows us to be extremely customer focused and help direct investment for the business, such as what features we build on and when.

We also use the same data pipeline to power a lot of our data product features which customers pay us for.

So it’s extremely worthwhile as an investment for us. It’s also why we have about five people hired into data adjacent roles, as it’s so key to us running the business correctly.

lawrjone··on A modern data stack for startups (2022)
The product provides login and sign up with Slack which means all product plans include SSO under whatever scheme the Slack workspace is provisioned with.

The SAML feature in the Pro and up plans is about hooking us up directly to your identity provider to do things like RBAC and group syncing, and we pay WorkOS a monthly fee of several hundred dollars per customer that uses this, so makes sense to gate in the higher plans.

So I’m not sure this is quite right?

lawrjone··on A modern data stack for startups (2022)
We do very similar things ourselves: our insights product (https://incident.io/learn) uses Metabase to power the dashboards.

The data that goes into those insights can be quite complex and the queries are actually threading JSON parameters through into BigQuery SQL queries using JavaScript UDFs to power the filters in the dashboard (show incidents with these custom field values). This works pretty well with signed Metabase dashboard links.

We have hit limitations with Metabase though. Performance of the instance can be a bit unpredictable and their support has been poor when things do go wrong, with very little willingness to take our feedback into account for new product features.

For that reason and more (such as more flexible dashboards) we’re going to move ourselves to Omni (https://omni.co/) for internal business analytics use cases, and will reconsider Metabase for our customer facing product dashboards when we do. Omni may work for these or we might build them bespoke, we’ll see at the time.

lawrjone··on A modern data stack for startups (2022)
This is an article from Jan 2022 when we were a company of 10, and now are a company of ~80.

Worth some observations that:

- We're still using Fivetran for the EL stages. Costs are much more significant than they were before and we're looking (for the high volume sources) into options like DataStream as cost savers, but it's not unmanageable.

- dbt is still working great, even if we've done a lot of investment having now built a 5 person data team (BI, DA, DE) around it.

- Still use Metabase but have some frustrations and are considering other options.

- We no longer use Stitch :tada:

There's a post that followed this on improvements we made to our setup that may be interesting: https://incident.io/blog/updated-data-stack

The OP is still full of relevant, useful information, though (imo, of course).

lawrjone··on A modern data stack for startups (2022)
I'm not sure it's so steep: Starter is $16/user, Pro is $23/user (with the $10k minimum) and then Enterprise is expecting 100+ (into the many thousands) of users and companies often want to discuss that pricing alongside support constraints etc.

The Pro plan contains a load of features like insights, audit logs, API access, etc that you don't get in Starter, hence the increase per-seat bill.

Would be interested if that makes more sense to you? (I work there, always keen to hear feedback)

lawrjone··on Tracking developer build times to decide if the M3 MacBook is worth upgrading
We do frequently adjust the codebase to improve build times, but the developer productivity hit we’d take by splitting our app into several microservices would be much more substantial than much slower build times than we currently see.

We’re super happy with 30s as an average build time right now. Assume we spent about £50k for the entire teams laptops, how much time would an engineer need to break our monolith into microservices? How much extra investment would we need to make it work as well as our monolith, or debug production issues that are now much more complex?

I’d wager it’s much more than half an engineers time continually, at which point you’ve blown the money you saved on a solution that works far less well for dev productivity and debugging simplicity in production.

lawrjone··on Tracking developer build times to decide if the M3 MacBook is worth upgrading
I think we’ll definitely see AI find its way to the notebook tools. Funny enough, you can ask the model to give you an iPython notebook of its workings if you want to move your analysis locally, so in a way it’s already there!

On the process: we’re using OpenAI’s assistants feature alongside the ‘code interpreter’. This means the LLM that you speak to is fine tuned to produce Python code that can do data analysis.

You upload your files to OpenAI and make them available in the assistant. Then you speak to the LLM and ask questions about your data, it generates Python code (using pandas/numpy/etc) and runs the code on OpenAI infra in a sandbox, pulling the results out and having them interpreted by the LLM.

So the plots you see are coming direct from Python code the LLM generated.

On how many times did I resubmit: quite a few. I’d ask for a graph and it would give me what I needed, but maybe the layout was bad so you’d say “repeat the above but show two histograms a row and colour each machine model differently”.

I was using a very recent model that’s in preview. It was a bit slow (30s to 2m to get a response sometimes) but that’s expected on a preview and this stuff will only get faster.

Hope that answers your questions!

lawrjone··on Tracking developer build times to decide if the M3 MacBook is worth upgrading
Hahaha yep, Milly won the lottery on this one. We chose the person who was doing most builds that week to give it to, so it’s well deserved!
lawrjone··on Tracking developer build times to decide if the M3 MacBook is worth upgrading
Am OP: the Docker for Mac daemon can claim a lot of system resources and the FS can have a lot of latency.

This means devs have less system memory available for their host because it’s in the VM, battery is often harmed because the VM box is constantly churning, and as we run our Postgres database in docker the FS latency makes for a noticeably slower dev environment.

Docker is great for standardising the dev env but honestly, running Postgres locally isn’t hard, and we don’t have many other deps. If the quality of experience is better at the cost of the occasional “let me help you out with this pg_upgrade” then I’d consider that a win.

lawrjone··on Tracking developer build times to decide if the M3 MacBook is worth upgrading
Hahah good question: 2.5 days all in, including building the new hot-reloader which fixed a number of outstanding bugs.

- 1 day for hot-reload (I don’t think this should count tbh)

- 1 day for messing with data

- 0.5 day for write-up

Assuming my day rate is £1k/day (it is not) we land at £1.5k, less than a laptop and less than I’d pay for the impressions I got from my post sharing this on LinkedIn, so feels worthwhile!

lawrjone··on Tracking developer build times to decide if the M3 MacBook is worth upgrading
The machines haven’t arrived yet but I wouldn’t expect we’d have the data to prove this out. As you’ll know, number of tickets complete is a weak proxy for value delivered, and can be so wrong as to imply a team that you know is 2x as productive is half as much as another, or worse.

The best way (imo) to measure this is setting a threshold that we agree is bad (in our case, this is 1m+ builds) and continually measuring it so we catch when that starts to happen more frequently. And listening to devs when they say things have got slow/have become irritating.

lawrjone··on Tracking developer build times to decide if the M3 MacBook is worth upgrading
Author here: this probably took a 2.5 days to put together, all in.

The first day was spent hacking together a new hot reloaded but this also fixed a lot of issues we’d had with the previous loader such as restarting into stale code, which was really harming people’s productivity. That was well worth even several days of effort really!

The second day I was just messing around with OpenAI to figure out how I’d do this analysis. We’re right now building an AI assistant for our actual product so you can ask it “how many times did I get paged last year? How many were out-of-hours? Is my incident workload increasing?” Etc and I wanted an excuse to learn the tech so I could better understand that feature. So for me, well worth investing a day to learn.

Then the article itself took about 4hrs to write up. That’s worth it for us given exposure for our brand and the way it benefits us for hiring/etc.

We trust the team to make good use of their team and allowing people to do this type of work if they think it’s valuable is just an example of that. Assuming I have a £1k/day rate (I do not) we’re still only in for £2.5k, so less than a single MacBook to turn this around.

lawrjone··on Tracking developer build times to decide if the M3 MacBook is worth upgrading
Author here!

We’re using the new OpenAI assistants with the code interpreter feature, which allows you to ask questions of the model and have OpenAI turn those into python code that they run on their infra and pipe the output back into the model chat.

It’s really impressive and removes need for you to ask it for code and then run that locally. This is what powers many of the data analysis product features that are appearing recently (we’re building one ourselves for our incident data and it works pretty great!)

lawrjone··on Tracking developer build times to decide if the M3 MacBook is worth upgrading
They’re normalised histograms so the y axis is deliberately adjusted so you can compare the shape of the distribution, as the absolute number of builds in each bucket means little when there are a different count of builds for each platform.
← PreviousPage 2 of 8Next →