83 karma · joined May 12, 2014
Bottom line, every single post requires approximately 10 different prompts to refine the extraction.
ChatGPT does an incredible job parsing, but then lots of effort goes into normalizing and deduping each field. Long story short, your results look quite good to me!
Remote: yes
Willing to relocate: maybe
Technologies: Go, JavaScript, TypeScript, Python, Postgres, SQLite, SQL, PyTorch, FastAI, LLM, NextJS, React, GCP
Résumé/CV: https://u-turn.dev
Email: dudley@u-turn.dev
Product engineering leader helping build great teams. Twenty-plus years of software development. Over the past decade have helped turn around multiple products and teams in crisis.
Currently focusing on helping organizations apply deep learning and LLMs for information extraction from unstructured sources. A recent project in that vain: https://hnjobs.u-turn.dev
There's a small write-up on how we went about processing HN Who Is Hiring with ChatGPT:
> Interestingly, those who felt the greatest boost in mood also experienced the biggest drop in heart rate variability.
A drop in HRV isn't generally good. So I check the paper, and found this:
> No significant changes were found in heart rate variability or resting heart rate over the course of the study in either of the groups (Figures 4C and 4D)
There was a reduction respiratory rate for those an increase in daily positive affect. Bottom line, unclear if this particular study points to a positive health outcome other than feeling happier.
When subscribing to StackAid, you indicate that you'd like to support your dependencies. It's hopefully clear that their dependencies have to participate to receive their allocation. We're also upfront that not all OSS projects are interested or currently have a relationship with StackAid. As a result, allocation amongst dependencies will shift depending on the OSS project interested in participating, and that's an important feature that sets it apart from other funding models.
All of that said, we've tried to be clear in the messaging and setting expectations. We're very much open to feedback to improve what we communicate.
To give some context, if SPI were interested in receiving money for repositories associated with SPI, they would work with the developers to claim those projects on StackAid and associate their Stripe account. SPI's governance model can then decide how to allocate those funds.
We are thinking through how to give subscriptions control over projects and organizations to exclude, so you could choose to exclude tiny dependencies in your subscription.
That said, defaults matter, and so it's still worth considering the implications of small projects getting a large amount of funding:
1. Subscriptions/open source repositories might be ok with a rimraf getting a lot of money, especially if it funds those developers to build other things that are high impact.
2. It might also inspire other open source projects to potentially compete for those funds in terms of offering something more or encouraging tooling that incorporates that functionality, thus creating a smaller set of dependencies.
Re: Electron getting much less than these smaller projects in the simulation. Electron getting less money is a function of the NPM packages we discovered and sampled. That could or could not be a representative set. You could argue that Electron shows up more in non-public repositories. It doesn't mean that Electron would get more money than rimraf, but the gap could be significantly less.
We are busy thinking through the best way to provide controls around allocation including specifying repositories/organizations you don't want to fund. This feedback is what we wanted to hear before we embarked on an approach.
Some organizations might also specify that they don't ever want to receive money, and allowing us to avoid waiting to reallocate the money intended for their repositories. This is another option we're planning on exposing.
Previously, the MySQL solution had many rows for each ranking spanning several tables with all kinds of other data not related to showing the aggregate statistics that were relevant to Moz's users.
The solution was processing the raw data in batch and applying categorization to have an integer representation for all values. These changes led to a very compact representation that could quickly be loaded into memory and then filtered/aggregated.
Storing the results as a CSV wasn't important. It just turned out that having the static data allowed me to effortlessly scale-out serving since the data was only updated once a day or once a week, and it was append-only.
How big of a difference did it make? All of the CSVs individually compressed was less than 20GB. The production system served all user data off of a ~60 node MySQL cluster, with rankings being the most costly in terms of processing and 2nd in terms of disk usage (from what I remember).
Also, Pandas was blazing fast at loading several megabytes of CSV data (< 80ms @ the time). If I had to do it again today, I'd probably use Apache Parquet instead.
The most important insight that carried through the various solutions was to pre-process the data so that it was easily consumable for the task at hand. The languages (Python/Elixir) didn't make a difference, in my opinion. That said Pandas is fantastic, it made working with that data in-memory very easy, at least for this prototype.
Having looked through the set of extensions, wal2json (https://github.com/eulerto/wal2json). That makes sense since there's no way at the moment to get access to replication.
However, the use case for wal2json in addition to the access to replication is that it's the best way that I know of to get reliable change notifications from the database. I personally use this to pipe the JSON blobs into message queues to be consumed by applications.
As a short aside, I don't use PostgreSQL's NOTIFY/LISTEN because, as I understand it, you will not receive messages if there are no connections to the DB -- this is a show stopper for me.
AWS RDS allows this and Google's PostgreSQL does not. I've personally abandoned Google's version first partly because it seems to have stalled in terms of upgrades, and AWS for being way too expensive for what you get. Although it doesn't work for me at the moment, I'm hoping the DO option will work for me in this regard in the not so distant future.