joelburget

I wrote an OCaml implementation of this paper a few years ago, which I've now extracted into its own [repo](https://github.com/joelburget/constructive-reals/blob/main/C...)

The link in the paper to their Java implementation is now broken: does anyone have a current link?

And more recently, [Language Models Use Trigonometry to Do Addition](https://arxiv.org/abs/2502.00873)

This is a good idea, though one problem is that Einsum notation (as realized in Numpy and Pytorch) doesn't support the notion of co-contravariance, and the site is based on their Einsum notation. I could potentially add the variances for the examples, though that would move away from how the site currently works (where the information about the reduction comes only from the einsum input).

A couple of these I'd like references on if anyone happens to have them.

1. "current science suggests that the actual health impact from consuming most types of plastic might well be essentially zero"

2. "the (weak) evidence we have now suggests running strengthens your knees"

o1 is an application of the Bitter Less. To quote Sutton: "The two methods that seem to scale arbitrarily in this way are search and learning." (emphasis mine -- in the original Sutton also emphasized learning).

OpenAI and others have previously pushed the learning side, while neglecting search. Now that gains from adding compute at training time have started to level off, they're adding compute at inference time.

Vision Transformers do a shocking amount of compression in the tokenizer. In the [Chameleon paper](https://arxiv.org/pdf/2405.09818) they say the tokenizer "encodes a 512 × 512 image into 1024 discrete tokens from a codebook of size 8192". That's 256 pixels per token (512 * 512 / 1024). If we assume that a pixel is 24 bits (3x 8 bit channels), this implies that they've compressed 256 * 24 = 6144 bits into 13 = (log2(8192)). [An Image is Worth 32 Tokens for Reconstruction and Generation](https://yucornetto.github.io/projects/titok.html) pushes this even further. If these models work similarly, it's no wonder they struggle with some vision tasks.

Vision transformers should be our default guess as to how GPT-4o works, yet this article never mentions them.

It works on all human languages, just inefficiently. I ran it over a sample I found on wikipedia:

    sample = "ฟองมันฟันหนู, ฟันหนูฟองมัน, ฝนทองฟองมัน"
    len(sample), len(enc.encode(sample))

This returns `39, 40` so it's just encoding one character at a time. It's probably like this for almost all non-English text.

A few interesting findings:

* the cl100k_base tokenizer has ~100k tokens -- previous tokenizers had ~50k. (enc.n_vocab gives 100277 but some numbers in that range don't work, starting at 100256)

* it has exactly 1110 tokens which are just digits. 10 1 digit tokens, 100 2 digit tokens and 1000 3 digit tokens! (none have preceding spaces). this is a huge improvement from GPT2's tokenizer, which was a huge mess.

* there are <|fim_prefix|>, <|fim_middle|>, and <|fim_suffix|> tokens (see Efficient Training of Language Models to Fill in the Middle)

The biggest news to me is the improved handling of numbers. This could explain some improved performance on arithmetic. One disappointment is that it tokenizes from the front, e.g. "1000000" -> 100|000|0. This is one of those "so close!" moments -- I would work for free to fix this.

For those looking to run this on a Mac, the following seems to have worked for me (M1, Big Sur 11.2.3):

``` brew tap gcenx/wine brew install --cask --no-quarantine wine-crossover brew install winetricks winetricks corefonts vcrun6 vb5run native_oleaut32 vcrun2010 richtx32 comdlg32 comctl32ocx wine BookStory_en.exe ```

You have a keen eye -- thanks for the close reading.

Thanks for the heads up. It never occurred to me the svg might render differently. I'll see if we can fix this.

I'll use a bit of Haskell in my response, since that's what I'm most comfortable with and what we actually use for our work. Hopefully that's okay.

We can define a data type of expressions in our language, with addition, multiplication, etc.

  data Expr = Addition Expr Expr | Multiplication Expr Expr | ...

Then we need the type of an interpreter which consumes program inputs and produces an output. I'll leave this abstract for now.

  data Interpreter = ...

By map, we just mean that we can write a Haskell function with this type

  interpret :: Expr -> Interpreter
  interpret (Addition a b) = (+) <$> interpret a <*> interpret b
  interpret (Multiplication a b) = (*) <$> interpret a <*> interpret b

If you're curious, here's some real symbolic evaluation code I wrote doing the same https://github.com/kadena-io/pact/blob/234ba3dd01f0df8b4e462...

Stick around for part 2. We haven't gotten to the good stuff yet :)

Co-author here. We're planning to turn this into a series, with additional posts on how symbolic execution works at a lower level, how to present counterexamples to users, and ideas for future programming tools built on symbolic execution.

Similarly, I once wrote a triangular maze generator https://www.khanacademy.org/computer-programming/triangle-ma...

Hi acbart, thanks for the suggestion. This is now fixed. https://codewords.recurse.com/issues/three/algebra-and-calcu...

I built something similar. It's meant to duplicate the live editing functionality on the React homepage. My repo has components for live compilation of react code, the codemirror editor, and using both of them together.

demo: http://joelburget.github.io/react-live-editor/

code: https://github.com/joelburget/react-live-editor

Some great previous discussion on thorium based nuclear reactors:

http://news.ycombinator.com/item?id=2723675 http://news.ycombinator.com/item?id=1009869

I think the first article was in reaction to the Fukushima disaster. It's hard to imagine this technology will actually make it to cars, whether it's safe or not, because of the r-word. Intriguing stuff, though.

I agree somewhat. Yes, a lead has more responsibility, and will work as hard as the other team members. Sometimes they may have to push to get the product shipped.

That is a completely different thing than four hours of sleep a night for years.

I wish people would stop glorifying how many hours they work and how little sleep they get, "You need to live without sleep (4 hours a night every day for years baby!)." Sounds juvenile to me.

I'm guessing he means looking back it was a mistake. I would love to hear more details from the OP though. That phrase definitely piqued my interest as well.

"And most of all, Haskell makes programming fun!" - If you're doing a startup make sure you choose a language you enjoy. You're going to be spending a lot of time using it.

(Also applies to cofounders and the application itself.)

Honest question: Is this really HN material? I can't remember seeing a photo gallery on here before.

Really, my reason for saying I wouldn't learn anything is because I personally learn better from concrete examples. That's why I like the RWH chapter. It gives you knowledge of how GHC represents your program, and helps you use that to change your program accordingly. Perhaps I should have reserved my criticism for the end of the series. Hopefully you will show us how to apply your model to our real programs. For instance, I think it would be neat to have a simple example of, say, folding a list. Show us what happens when we make a field strict. Show why `True || undefined` doesn't blow up. Just some suggestions.

I agree with you on the wikibooks article. I included it so someone reading my comment would have a place to go for the rest of the story that RWH doesn't talk about.

(By the way you might want to reformat your links, HN seems to have misinterpreted them.)

As someone who is relatively familiar with evaluation in Haskell I'm enjoying this series, but if I didn't already understand most of it I doubt I would get anything out of it. Real World Haskell has a more concrete chapter on profiling: http://book.realworldhaskell.org/read/profiling-and-optimiza.... The Haskell wikibook also has a fairly comprehensive chapter on laziness: http://en.wikibooks.org/wiki/Haskell/Laziness.

I've been reading some blog posts here http://existentialtype.wordpress.com/ about the curriculum at Carnegie Mellon and I've been very impressed. I can't believe they're phasing out object-oriented programming! Here the faculty seem to think object-oriented is the only way. Actually I can't think of any functional programming we've been introduced to. (Or any other paradigm for that matter)

I was really surprised when I got to college and the only languages people cared about were C++ and Java. Ok, you'll also find some C#, Python, and Ruby, but nobody has heard of any of these "advanced" languages. The only thing they care about are the number of job opportunities they can get. I sometimes try to advocate for Haskell, or Scheme, or even Prolog. The response is usually "Pascal?" It's sad that these students never wonder if there's a better language, or a more fun language, or anything other than Java. It's a desert out here.

I tend to agree with Mark Twain on this point: "I have never let my schooling interfere with my education."

For me, I have learned more outside of school than I have in it. Really I'm in it for the diploma. I wish companies would look at your software rather than your certificates, I think they would tend to find better developers. I know some companies are good about this but the reality is that programmers do need to go to college if they want to compete for jobs.

I will say this for school: it fills in gaps in education so companies can expect some standard knowledge from everyone. Also, it can be helpful for people that don't have the motivation to learn on their own.

The article asserts that heavy containers placed on top are more likely to be flung into the sea. Is that really true? What if it was actually better to load heavy containers on top? Or heavy containers around the edges with lighter ones in the middle. More valuable cargo in the middle? What is the optimal shipping container arrangement?

Do shippers have software that tells them where to place the containers or is it more or less random?