gamegoblin

I had written this comment[0] about our pattern here a few months ago and we decided to turn it into a full blog post.

It's crazy how many weird bugs you can detect with such a trivial method.

For applications like a spreadsheet that have a million overlapping features, it's impossible to manually write unit tests for every combination of features, so randomized tests do a lot of heavy lifting.

[0] https://news.ycombinator.com/item?id=40876726

Tools like GPT and Copilot still generate lazy_static a lot since it's the majority of the training data

I agree that fine-tuning isn't good at imparting knowledge, though in my experience, K-shot prompting is very close to as good as fine-tuning at getting output formats (for recent models, at least)

So except for cases where you are really trying to perfectly match a particular writing style and want to fine-tune on loads of some author's text, I think prompt caching mostly dominates fine-tuning for most real-world workloads.

I'd still fine-tune if I wanted my model to have a really particular "voice" though.

Seems like their competition (Google, Anthropic) have both shipped prompt caching, which is a lot more developer-friendly and gets a lot of the same benefits as fine-tuning.

You just cache a prompt with a ton of examples that you would have otherwise fine-tuned on. You can trivially update that prompt whenever, no asynchronous fine-tuning job needed.

Wonder if OpenAI will stick with fine-tuning or go towards prompt caching (or both). Fine-tuning has uses, but prompt caching gets you 99% of the benefits for 1% the effort.

I appreciated the comment (author here)

Yes, the llama.cpp work supports arbitrary CFGs, not just JSON

Language models like GPT output a large vector of probabilities for the next token. Then a sampler decides which of those tokens to pick.

The simplest algorithm for getting good quality output is to just always pick the highest probability token.

If you want more creativity, maybe you pick randomly among the top 5 highest probability tokens or something. There are a lot of methods.

All that grammar-constrained decoding does is zero out the probability of any token that would violate the grammar.

It's essentially an Earley Parser[0]. It maintains a set of all possible currently valid parses, and zeroes out the probability of any token that isn't valid in at least 1 of the current potential parse trees.

There are contrived grammars you can give it that will make it use exponential memory, but in practice most real-world grammars aren't like this.

[0] https://en.wikipedia.org/wiki/Earley_parser

I'm glad they gave up on their "fine-tuning is all you need" approach to structured output. It's possible fine-tuning will work in the long term, but in the short-term, people are trying to build things, and fine-tuning wasn't cutting it.

Surprised it took them so long — llama.cpp got this feature 1.5 years ago (actually an even more general version of it that allows the user to provide any context free grammar, not just JSON schema)

I’ve cancelled Prime and just buy things from Target, Costco, Home Depot, etc now

Amazon is just full of junk, the 2-day shipping too often becomes 3-4 for me, and general concerns about the health and safety of products on there (I have no idea how people trust them enough to buy things like toothpaste or other stuff that goes in your body)

If you happen to be coding in Rust, for really robust concurrency testing, I cannot recommend enough the AWS Shuttle library (https://github.com/awslabs/shuttle) which can find insanely complicated race conditions.

What the Shuttle library is doing is basically automatically going through all the permutations of the schedule diagrams described int his blog post.

We used it at AWS to verify the custom filesystem we wrote to power AWS S3.

If you're curious, I wrote a little tutorial on it here: https://grantslatton.com/shuttle

These concepts would make for a killer PVP real time strategy game, good write-up

Yes exactly, though not exhaustive, so more like beam search actually

That's why my outer loop goes from low-complexity cases to high-complexity cases, it has basically the same effect as shrinking (without actually having to do any work)

I do a lot of manual proptesting in Rust that all look something like:

    let mut rng = rand::thread_rng();
    for action_count in 1..4 {
        for _ in 0..10_000 {
            let seed = rng.gen::<u64>();
            eprintln!("let seed = {seed};");
            let mut rng = ChaChaRng::seed_from_u64(seed);

i.e. top level true randomness, then a bunch of nested loops (only 2 here, but some tests have more) to go from low-complexity cases to high-complexity

then generate a seed to seed a deterministic PRNG, and print it out so if the test fails, I just copy and paste the error seed to replay the error case

I have found doing this manual proptesting to be faster, more flexible, and generally less fuss than using any frameworks or libraries

That said, for really robust concurrency testing, I cannot recommend enough the AWS Shuttle library (https://github.com/awslabs/shuttle) which can find insanely complicated race conditions. I wrote a little tutorial on it here: https://grantslatton.com/shuttle

We used it at AWS to verify the custom filesystem we wrote to power AWS S3.

Depends entirely on how you intend to get a job

If you post cool projects on HN or Twitter, people will absolutely look at your personal site and send you job offers if you look promising

I've gotten 10x more serious job offers from posting projects on HN and Twitter than via resumes and LinkedIn

All that said, the content of the projects is an order of magnitude more important than the design of the personal site, unless you want frontend roles

Adding image support from Python is on our roadmap

Thank you for all the feedback!

1. Our frontend guy has been dreading this but it is high on the list nonetheless :)

2. Somewhat lower down the list but we have heard this before too

3. I have array functionality on a branch, still ironing out some rough edges, might be a month or two before we land it for real. We can add those builtins hlookup & edate very quickly.

4. We haven't heard anyone request vector drawing before. Can you elaborate on the use case? Sounds interesting.

5. Re: Row 0, we have funnily heard a lot of Excel people always wanted it, because they tend to have headers at the top, and it always bothered them referring to their data starting at A2 instead of A1. It's not a compatibility issue for import from Excel, because we just import all the data starting at A1. It can be a problem for export to Excel (which we don't have yet, but will), but we will just shift everything down by 1 row as if you had done a giant cut-paste.

6. For string types we have "contains" but we don't have "does not contain", will add it. Excel allows you to do "contains" on number types too which we thought nobody actually wanted, but if you do want it, we can add it. There are some performance implications to this because it means you have to convert numbers to strings to do the check.

7. More customization is probably farther down the line when we grow more.

8. Yes, right now we can only output to CSV. We have had some people talk about how they want to output to Powerpoint slides, PDF, etc. This is definitely on our radar.

We have a super fast iteration cycle, so most times a user asks us for a feature parity thing like this, we can ship it within 24 hours. This particular one might be slow for us, a few days :)

If we waited until we had full Excel parity, we would never ship. Google Sheets has been around for 17 years and still doesn't have full Excel parity on features or functions, and that's with dozens of engineers working on it full time.

Luckily now that we are launched we have a bunch of very knowledgable people like you requesting all these features, and this gives us a really good signal for prioritization.

E.g. a bunch of other people also requested the alt-shortcuts, so they are pretty high priority

Highest priority are the bugs like jkaptur pointed out above, and some other people had some crazy CSV files that we couldn't parse. So I'll fix those today and get to the copy-paste-formula-to-range thing in the next day or two. Alt shortcuts is very high on our frontend guy's list.

I don't have any great book recommendations because I learned on the job mostly.

For a project that will teach you literally everything there is to know about backend development:

1. Write a simple PUT/GET/DELETE REST API network layer, this should be a few hundred lines of code max depending on how much you choose to lean on libraries vs write it yourself

2. Then, write a simple in-memory blob store behind the PUT/GET/DELETE REST API. Just use a very simple hashmap of keys to data. You now have an in-memory S3 mock!

3. Now, rather than in-memory hashmap, start saving the files to disk. Write code that allows you to start and stop the process and recover all the data from disk. Now you have a persistent S3 mock.

4. Now, start finding the limits of this application. Write a program that calls your API with different usage patterns to load test it. Find the limits.

5. Now start optimizing. You can go as far as you want here. This optimization process was my whole career at AWS S3, and most of it at Row Zero.

6. If you want to really learn some advanced data structures and algorithms, stop using the filesystem directly, and write your own storage mechanism. Allocate 1 giant 16GB (or however large) file and store all of your blob data inside that 16GB "partition". You have to do all the serialization and retrieval etc yourself.

Your filesystem is already doing this for you when you store as files (look up how the ext4 filesystem works, inodes, pages, etc). But your filesystem is probably optimized for consumer use, not this blob-storage system use.

So you can implement your own much faster version if you want. If you want to really learn datastructures and algorithms, try to code a simple Log Structured Merge Tree. This will teach hashes, trees, bloom filters, serialization, deserialization, etc, all with high performance in mind.

We didn't know about this formula-paste-to-range functionality, we will add it in the next few days as we grind down this post-launch backlog. Let us know any other stuff we are missing please!

In the mean time you can still paste the formula to the top cell then drag it down or ctrl+D as a workaround, but we want to support the full experience, so check back in a few days and it should be there :)

We use Apache Arrow which is what Data Fusion and Polars also use

We will add a little tasteful AI stuff later, but it's not the highest priority. Top priority is big data and fast data. AI is relatively easy to layer on later.

We have the capability to run it as a desktop app. How much do you think we should charge for it, and how do you think we should do support, feature additions, etc? We are open to the idea.

We were fortunate to work in AWS which has some of the gnarliest datastructures and algorithms problems in the world, so we got really good there. Spreadsheets are really just more datastructures and algorithms on the backend.

Our frontend guy is brilliant and built a lot of awesome stuff at Airtable, but I can't speak to the craziness that is high-complexity high-performance frontend. But it's absolutely necessary to a good product too!

I implemented an Excel format processor and the Excel format does not lend itself well to super-high-performance for large datasets unfortunately

Didn't know who I was talking to, looked you up now, amazing ;)

Thanks a lot for the bug report. We actually used to implement the range-extension logic in Excel/Sheets but removed it during a cut-paste refactor due to the complexity (I'm sure you know...) and resolved to add it back when someone asked for it.

I actually found edge cases where Sheets and Excel don't do range extension logic the same way.

So we don't do the range extension for now, but the 4 not getting recomputed to 3 is definitely a bug, will fix!

Conditional formatting is high on the to-do list

If you're open to it, email us at contact[at]rowzero.io and we would love to do a call to talk about your use case, what features you want, etc. We love doing customer calls and adding delightful features.

Email us at contact[at]rowzero.io, we should probably set up some sort of newsletter now that we are launched

I just tested in Excel and Google Sheets and our behavior matches them here, unless I am misunderstanding the repro steps.

For me, A3 becomes =SUM(A1:A2)+A1+A2 and remains 4 as it should be.

The action I am taking:

    A0=1
    A1=1
    A2=SUM(A0:A1)+A0+A1 (evaluates to 4)

Right click A0 and click "insert row above"

Now I have:

    A0=empty
    A1=1
    A2=1
    A3=SUM(A1:A2)+A1+A2 (evaluates to 4)

Thank you for trying to find edge cases! I have put literally hundreds of hours into stuff like this. Let me know if you have different repro steps.