HNHacker News
TopNewBestAskShowJobs

icyfox

1,616 karma · joined April 6, 2021

pierce [at] freeman.vc despite the email i mostly code
submissionscomments
icyfox··on My AI costs went from $100 to less than $1/day: Fine-tuning Mixtral with GPT4
I've done some research involving parameterized summarization recently. GPT-4 does a really good (like, pretty outstanding) job in following the tone and density of a few one-shot example inputs. It's a lot more difficult to encourage the OSS models to do the same so you're going to have to experiment with a lot of different techniques to get it to what you want. In the worse case it could actually require some serious R&D investment to achieve the same results as an API out of the box.

All that to say, my general philosophy is to worry about vendor costs / scale once you start approaching an adoption threshold where it matters. If the core idea doesn't have legs there's no point wasting time in developing your own summarization layer.

icyfox··on My AI costs went from $100 to less than $1/day: Fine-tuning Mixtral with GPT4
Favorite part of this piece:

> We were initially skeptical whether we would get to 10,000 results. But with nightly leaderboard gamification, we managed to break 15,000 results within a week. Out of fear of eating into our productivity, we closed the contest.

I've hosted a few of these corporate data labeling events. If sufficiently gamified / there's a good enough UX, they can be surprisingly engaging. It helps a lot if you have a large employee base though. Distributing results over 5000 employees is exponentially easier than even 50 - in practicality, even larger than the orders of magnitude.

icyfox··on Clay Foundation Model: An open source AI model for Earth
Right now it seems that they've trained a true foundational model. It can recreate its training inputs, but that only matters as far as downstream performance goes. Based on a brief scanning of the repo & website:

- Their training process is a self-supervised MAE to recover the original satellite images and some additional metadata.

- They're trying to then use this for plastic pollution detection, deforestation, mining, etc.

I'd liken this to a llama-base. It's exposed to a lot of satellite data, but to do something useful really has to be fine-tuned (or clustered in the case of unsupervised embeddings). No word yet on the performance against these specific downstream tasks.

icyfox··on GigaGPT: GPT-3 sized models in 565 lines of code
Well thankfully my day job is actually ML research. :)

Lines of code is not strictly speaking a bottleneck for the next generation of models, but it ties with other objectives that are: researcher productivity, hardware efficiency, and model verifiability. GPT-5 might be another case of simply scaling up the existing transformer models, but the next step-function change of model quality are going to involve a lot more R&D about the right architectural primitives to take before scaling it up. And in that case lines of code do matter - because they allow simple concepts to be robustly tested, optimized, and iterated against. Doing that against a 50k monolith is a much harder task.

icyfox··on GigaGPT: GPT-3 sized models in 565 lines of code
I'm a big believer in the approach you're laying out too. Bugs are much easier to diagnose, crash reporting is more straightforward, and you don't need augmented services to consolidate everything at the output layer. That said - I've been predicting a shift back to simple architectures for awhile now and they haven't really come to pass. Maybe there's too much pressure or financial incentives for increasing complexity to solve increasing complexity?

At the end of the day I do believe ergonomics are going to win out. I think that's in large part why pytorch won out over tensorflow and jax; it provided the just-in-time computation that would allow people to more easily find bugs & visualize results without having to `compile()` everything down to a static computation graph. Hardware seems like a natural place for that abstraction layer - but maybe the silver bullet will really be on the software side, since we already have too many "low-RAM" equivalent ML devices in the wild. Cheaper to string things together after the fact vs. shipping net new hardware.

icyfox··on GigaGPT: GPT-3 sized models in 565 lines of code
For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) so it can fit more transistors and memory onto one chip. They achieve this small LOC count because they don't need to shard across multiple devices / backprop consolidate on a central CPU / etc. These tricks are usually what blows up a project from a single architecture proof of concept to a robust training pipeline that can handle the billions of parameters on modern models. This is more akin to training a whole model on a single GPU because... it kind of is.

Even with a wafer scale chipset this approach has limits. You eventually will still need to shard to fit more parameters / use different training modalities / etc. I'd look at this more as a proof of concept for the ergonomics of what LLM training can look like when you have access to a much larger compute primitive versus a new state of the art in feature-equivalent clean code.

Disclaimer: I'm a small investor in Cerebras.

icyfox··on Paving the way to efficient architectures: StripedHyena-7B
Common wisdom in most industries is to release bad PR announcements on a Friday and good ones towards the start of the week. It's interesting how the advent of twitter communication has shifted the ML ecosystem to publishing work whenever it's ready versus trying to find an optimal weekday. Or maybe they're optimizing for the weekend hackers that will take improvements released late in the week and put them into practice on Saturdays and Sundays.
icyfox··on Investing in new vector database development vs enhancing existing databases
The only production case I've seen of someone switching to a vector database totally in-lieu of a RDBMS is when their raw data lives in parquet / arrow files and they need fast read-only searches in production. Then a periodic indexing daemon with some additional metadata bolted onto the vector definitions worked well and saved them from having to dip their toes into classic database optimization.

Every place where I'm deploying vector databases, though, I'm using them as an alternative index just for searches (indexing embedding/id only) and doing the full object value lookup in the RDBMS. It's easier to optimize their vector clustering / sort performance outside the constraints of postgres - which is perhaps why pgvector suffers more under higher load.

icyfox··on Vtracer: Next-Gen Raster-to-Vector Conversion
This is the best I've found as well. They also have a new offering (different internal model, similar quality evidently) that I've used back when it was in a free unlimited trial. Not sure the status right now: https://vectorizer.ai/
icyfox··on What has changed in CPU cores in M3 chips?
Sometimes gradients are small but meaningful, if you constrain them to too few bits / degrees of freedom they'll be unable to backprop successfully. This can hamper training and therefore results quality.

You can also think about it as compounding errors - at any one weight index the bit values might not be too meaningful, but cascaded over a lot of tensor multiplications they will be.

icyfox··on What has changed in CPU cores in M3 chips?
Practically speaking, most models today infer at 8bit or 16bit (sometimes, rarely 32). You don't see an empirical lift at more bits of precision. Size of the memory is far more important.
icyfox··on Emmett Shear becomes interim OpenAI CEO as Altman talks break down
This will be an interesting test to see how fast you can bootstrap GPT-4 level performance with unlimited funds and talent that already has deep knowledge of the internals. With the initial adoption of ChatGPT alongside Copilot, OpenAI's data moat of crawled data & RLHF is pretty vast. And that's not leaving the walled garden of OpenAI. You can simulate a lot of this using other off-the-shelf LLMs (see Alpaca) but nothing is a substitute for real world observed usage.

In a related note, has this meaningfully broken through to the mainstream yet? If a ChatGPT competitor comes out tomorrow that is just as good - but under a different brand - how many people will switch because it's Altman backed? I'll be curious to find out.

icyfox··on Summing columns in remote Parquet files using DuckDB
It fits a pretty nice niche between small data (32GB in-memory, pandas level) and truly huge data where the disk files can't even fit on one machine (2TB+). Most data that people want to process on a daily cadence fit somewhere in between those ranges.

As for whether to use a database or not, I think that's a more fundamental question about where your different read and write loads are coming from. Scaling a database to support billions of rows and have high query performance is not trivial; if your use-case fits more in a write-infrequently, read-often then OLAP is a pretty good choice of index structure.

icyfox··on Cursorless is alien magic from the future
I've actually seen this work quite well, in terms of voice recognition. The commands are specific enough and a lavalier / directed microphone can filter out most of the surrounding noise. Whether other people will be as happy with you speaking gibberish, that I can't say.
icyfox··on Montreal’s new rapid transit line saved millions per mile
I'm really happy that Montreal made this work. We need more alternative examples of public transit build-outs in the west that actually work and come in on-budget. Copenhagen is another good exemplar here with their public/private collaboration for new lines.

I do think this article has a rather unrealistic tone for what the rest of us can learn from this though.

> One advantage is that CDPQ Infra was able to take advantage of existing rights of way to create the route, rather than needing to dig costly tunnels or demolish buildings.

This is huge. In many cities where we most desperately need public transit in the US, this just isn't realistic. We need net new lines either over or underground, and no matter how you slice that they're going to need right of way allowances and NIMBY disagreements. Plus most urban settings that will benefit the most from transit will need tunnel development as part of the cost projections.

> Quebec passed a law that requires municipalities to respond in a timely manner to CDPQ Infra’s requests for permits and other forms of cooperation.

Reading about California's highspeed rail project[^1], it seemed clear that there was deep government buy-in about the end state but the interim goals were over legislated. Counties in the central valley traded their buy-in for the project to starting the line build-out in their counties, even though population centers in LA and SF could have benefited way more from early wins. One thing this article missed was that we need to set more of a precedence for transit agencies / bureaucrats on the ground to make decisions that will further the end goal and circumvent the horse-trading at the legislative level.

[^1]: https://www.nytimes.com/2022/10/09/us/california-high-speed-...

icyfox··on Ask HN: How are you using your datasets with LLMs?
My general rule of thumb at the moment is:

- Tasks requiring knowledge synthesis between multiple different datapoints (in your case wiki pages) should be fine-tuned so the model's able to do some basic chain of reasoning to reach a new conclusion. Often just fine tuning on the text itself and not (query, text) pairs are sufficient for basic memorization and therefore lookup. The con with this approach is you don't know where the original information came from - the pretrained model, your wiki, or if it's just a hallucination.

- Tasks that require some source of truth for reliability benefit more from the RAG approach, since the summarization layer can explicitly reference the input sources. I use markdown annotations for this output format since it provides inline and easily parsable references to the retrieved content.

RAG is effectively a layer on top of classic information retrieval, like what happens with search engines. The question of how to do the retrieval itself could be semantic embedding-based like what you get back from OAI, tfidf-based, or some other heuristic approximation.

If you have a few initial queries of what people are looking for, I'd start simple with a jupyter notebook, the OAI fine-tuning API, and numpy before jumping to off-the-shelf solutions that promise to solve this problem for you. It will build more of an intuition of your data and the tradeoffs required.

icyfox··on Why You Should Write Your Own Static Site Generator
I'm not sure I'd advocate for writing a static site generator, although I'm certainly guilty of writing a few myself.

Instead I always encourage people who are trying to start blogging to do the writing first. Figure out a workflow that works for you - what time of day you prefer writing, what editor, do diagrams naturally come up in your thought stream, etc. It's way easier to get this workflow dialed in when you're doing things locally since the switching cost between solutions is that much lower.

Only when you know that you A) enjoy writing and B) have something worth sharing should you invest the time in translating your workflow to something that can deployed. That might mean writing your own SSG - that might mean just spinning up a wordpress blog.

icyfox··on Show HN: Internet Speed Test
Fast can also measure some more connection metadata if you click "show more info". A bit annoying it requires the extra step to start measuring ping and upload speed, but it is there.
icyfox··on ML Experiments Management with Git
I have a similar harness going for my recent experiments, except instead of hosting with huggingface I have a dataframe with pointers to the files on S3 and then just download them during local preprocessing.

Every time I see DVC mentioned I always feel like the idea was so close (and perhaps right in intuition to use git for everything) but the execution had just enough friction that I looked elsewhere. Small DX improvements really do cascade pretty far.

icyfox··on Show HN: Open-source background removal in Node.js
Last time I looked into this their training code is kept internal and they're just releasing the model weights.
icyfox··on OpenAI and Jony Ive in talks to raise $1B from SoftBank for AI device venture
There was an Information article a few weeks ago on Apple increasing its daily spend to millions of dollars in ML training costs per day. With their historic desire for control over the whole stack, I'd put money on them rolling their own architecture.
icyfox··on My favourite API is a zipfile on the European Central Bank's website
That's the interesting part of this story; Congress didn't think this requirement existed, neither did the lobbyists. But the language that congress adopted (with the consultation of this lobbying group) made the agencies _think_ it was what congress wanted. So the agency was left holding the bag for something no one actually wanted in the first place. Like a big game of telephone.

I agree with your broader point however. Congress needs to do a better job of owning outcomes.

icyfox··on My favourite API is a zipfile on the European Central Bank's website
I can almost assure you it was an explicit requirement in an RFP that was copied from some master template. So not a good justification - but a justification in the eyes of the bureaucracy.

The book Recoding America has a lot of anecdotes to this effect; most of these situations reduce to a Congressional mandate that got misinterpreted along the way. My favorite was for an update to Social Security. The department in charge of the implementation swore that Congress was forcing them to build a facebook for doctors (literally where doctors could friend other doctors and message them). Congress had no such intention; it was actually 3rd party lobbying that wanted the requirement so they could build their own solution outside of government. Really crazy stuff.

icyfox··on Uninstall the NightOwl app
Thanks for linking to these. They are certainly in the right direction although they're a bit vague on how much notice to give:

> Even if Douglas’s continued use of Talk America’s service could be considered assent, such assent can only be inferred after he received proper notice of the proposed changes. Douglas claims that no such notice was given. (Douglas v. Talk America)

> But the Court also concludes that, even in light of their agreement to the Special Terms at the time of registration, customers' assent to the revised Terms cannot be inferred from their continued use of safeway.com when they were never given notice that the Special Terms had been altered. (Rodman v. Safeway)

Both cases seem focused pretty narrowly on situations where notice was not given. Is continuing to use an app after an update notification enough? Glancing over a GDPR-like popup? An email? I'd prefer an explicit opt-in to changes once they've occurred.

icyfox··on Uninstall the NightOwl app
I know this happens with some frequency, I wonder how frequently the companies update the TOS with language like this. The very idea of a self-updating TOS that will govern all usage into perpetuity feels like it should have been legally stuck down years ago. This company's current language on indistinct modification rights:

> We reserve our right to alter the terms in this Agreement and/or the pricing information and method detailed in NightOwl app's website at any time. In case the Agreement is amended as described, we will post an updated version of it in our website, at which time it becomes active and binding.

> In case NightOwl app alters the Agreement in a way which will be deemed material to the relations and/or obligations of the parties by NightOwl app's sole decision, we will inform you of these changes on our website or via our social media accounts and other established communication channels.

Great, a website update for a locally installed application. Definitely going to subscribe to your social feed to get an update.

icyfox··on Ask HN: Visiting SF on a Budget?
How long are you staying?

I don't know of any co-living spaces / hacker-houses that have short term opportunities, most are commitments on the order of a few months to a year at a time. Personally I would try to find a few roommates (in your network or through some common forums) who are collectively interested in the ML space and go from there.

To your broader question about keeping costs low, I'd recommend some general strategies that are probably true anywhere:

- Host dinner at your house instead of eating out. Raw ingredients to cook at home in SF are no more expensive than other places in the country but eating out can get really pricy really quickly.

- Get a bike, Clipper card, or sign up for Revel. People love to hate on Muni but if you're willing to wait a bit for the bus they typically get you where you need to go. You also feel more connected to the city versus looking out of an Uber window. Or even cheaper just walk the hills are not that bad.

- If you end up not going the hacker home route, there are a lot of SF Housing facebook groups where people try to find people to fill temporary sublets. They're usually a bit cheaper than rent prices and way cheaper than AirBnBs.

- If you already have an apartment back home I also know companies like Kindred let you swap housing 1:1 so you might have some luck this way.

Welcome to the city! See you around.

icyfox··on Launch HN: Roundtable (YC S23) – Using AI to Simulate Surveys
I think we're using aggregate statistics in two different ways.

Technically speaking - the whole idea with a language model is that you're learning to generalize underlying patterns of text, not just memorize the input text. Otherwise language models would be very good at echoing back training data but fail miserably during validation. If we go back to the training sequence - it's trying to maximize the posterior given the conditional probabilities in the sequence:

P(y1, y2, ... yn ) = P(y1) * P(y2|y1) ... P(yn|y1...yn-1)

That probability is by definition an aggregate; it's the best fit given potentially competing inputs of the training set that all have the same input conditional chains.

Where generative LLMs have a leg-up is because they have such a large parameter space, large context windows, and coherent sampling strategies. This helps them stay internally consistent with their response data. But at the end of the day what they are learning are still patterns. That's why they aren't able to link content back to the exact source of origin; parameters fuse inputs from different places into one hybrid.

Seeding a generative chain with Einstein or someone down the street doesn't change the fact that what's next is some fused learning from a lot of different training set inputs.

icyfox··on Launch HN: Roundtable (YC S23) – Using AI to Simulate Surveys
Congrats on the launch!

I have an interesting dissonance with this. On one hand, I understand how huge parameter sets can and do model specific personas well. I've also read some of these cited papers and _know_ intellectually that predicted results can be close to actual survey data. The other part of me is screaming at my laptop that language modeling is about aggregate statistics, revealed preference counts for a lot, and how could a language model actually substitute for market research?

I imagine the biggest hurtle you're going to face are research teams that:

- A. Want to see actual proof behind data

- B. Disbelieve a LLM could generate statistically significant insights about real people that would make individual decisions

- C. Need to justify their own existence / organizational clout with boots on the ground facilitating surveys

A and C might be surmountable, but I'm not sure of a good way of tackling B.

icyfox··on Building Boba AI: Lessons learnt in building an LLM-powered application
Sure - I'm using it in a few different internal tools and know others are using it in production. The API should be relatively stable at this point since I intentionally kept the scope pretty limited. The main changes over time will be improved robustness and error correction as issues report different JSON schema breaks that we can fix automatically. Let me know if you see more cases that can be addressed here, would love to collaborate on it.
icyfox··on Building Boba AI: Lessons learnt in building an LLM-powered application
There's no bulletproof solution to this. JSON5 (https://www.npmjs.com/package/json5) gets you slightly more leniency, as does plugging the currently streamed content into another smaller LLM. I also wrote a deterministic parser more tailored towards these partially-complete LM outputs. Not perfect certainly but handles the 99% of cases well: https://github.com/piercefreeman/gpt-json. In particular the "streaming" functionality here might be of interest to you.
← PreviousPage 3 of 5Next →