Obscure audio codec is a perfect fit for the N64 | Hacker News Reader

Obscure audio codec is a perfect fit for the N64

phoboslab.org

268 pointsfrostbyte731 comments

This codec would have been great 27 years ago! I suspect it wouldn't have gotten past the patent lawyers in 1999 though. It seems cute today but the entire codec space was a patent minefield, the N64 mp3 decoder was implemented twice (once based on the reference codec, once clean) with Nintendo/Rare then paying RAD game tools to license their decoder... of which none of their code was used but they had a license from the mp3 patent holder (Fraunhofer) which then shielded Nintendo from lawsuits. Fraunhofer wanted something like $250k to license directly which wasn't going to fly.

I'll also note the frame rate issues in Conker were mainly down to raw RDP fillrate issues, which was for sure the weakest part of the N64 hardware, it could draw beautiful triangles (for the time) but oh so so slowly. There was a decent amount of speech during gameplay using the RSP mp3 codec. The cartridge size was still an issue, 64Mb got a good amount of mono speech but was still very much at a premium.

>It seems cute today

Maybe in audio, but video still seems like an absolute nightmare. Look at the absolute silliness with like having to pay $.99 to play h265 video in Windows.

> having to pay $.99 to play h265 video in Window

what?

https://www.free-codecs.com/guides/navigating-hevc-video-ext...

Almost nobody actually pays for this, since every OEM includes a volume license that's baked into the price tag. VLC includes FFMpeg so it just works ootb.

Video codecs have historically been like this, to some extent or another. It's really just a manner of if the manufacturer has absorbed it into the price tag or pushed it to the consumer.

No one likes paying for a codec license anyways, but since patents are at play, they kind of have to.

That leads to absolute idiocy like this https://www.reddit.com/r/technology/comments/1p3p1hy/dell_an...

AV1 is royalty free and better!

Real example, I recorded the world series using an ATSC antenna, and I re-encoded a 90 second sample using a bunch of different settings on a 32-thread znver5:

  # Codec        Flags                     Size         Speed
  - -----        -----                     ----         -----
    Original     (Raw ATSC mpeg2video)     77MB (100%)  N/A
  1 libx265      -preset slower -crf 24    37MB  (48%)  0.245x
  2 libsvtav1    -preset 3 -crf 44         36MB  (25%)  1.87x
  3 libsvtav1    -preset 1 -crf 35         19MB  (25%)  0.556x

(NB: The same CRF results in a higher quality video at slower presets, this was for archival so we're using the some of the slowest presets).

(1) and (2) are completely indistinguishable from the original unless I go frame by frame looking for transient motion artifacts. (3) is only distinguishable in one shot where a dark background was washed out. Not only does AV1 produce results just as good as H265, it encodes quite a bit faster for the same size/quality.

That said, I've watched so much h264-encoded video over the past decade I think I've been trained to perceive the subtle visual "snow noise" as a sign of a high quality video: I find how AV1 blends backgrounds more visually objectionable than the dithering in H265, even though I think it's objectively higher fidelity. Similar to how I find high frame rate video "fake" after having watched NTSC TV for the first half of my life.

These were the CRFs necessary to get the same visual quality with libx265 at faster presets and the resulting sizes:

  hevc-slower-24          37MB (used above)
  hevc-slow-21            56MB
  hevc-medium-20          57MB
  hevc-faster-19          52MB
  hevc-veryfast-18        60MB

you used to be able to get it for free on windows[0] but it seems that might not be available anymore.

Of course there is always VLC

0: https://apps.microsoft.com/detail/9n4wgh0z6vhq

You can get it for free though: https://www.free-codecs.com/guides/how_to_download_hevc_vide...

>raw RDP fillrate issues, which was for sure the weakest part of the N64 hardware

I thought weakest part was the size of the texture buffer.

Nobody expects the N64! Our two weakest parts are our raw RDP fillrate, and our tiny Texture buffer.

Ok, from a performance perspective the RDP fillrate was the biggest issue. For artists the texture buffer was a massive challenge.

If the RDP had been clocked 4x faster and had 4x the texture size then it would have been incredible… it makes beautiful pixels!

i've heard the rambus dram latency was also a big deal but does that have any effect on the fillrate as well? or is that mostly a bottlneck affecting the cpu?

and memory bandwidth. Our three weakest parts are our raw RDP fillrate, our tiny texture buffer, our paltry memory bottleneck...and compiling in debug mode... Our four... no... *Amongst our weakest parts... are such elements as... I'll come in again.

Don't shit on the N64 from too high. It introduced 1998 (and 2026) real-time rendering concepts at a time when two years was an eternity in PC hardware.

It's why modern programmers can make it do things that would have given John Carmack sleepless nights if they had accomplished them back in 1996.

In case anyone else got to the demo thinking the quality is awful -- the vocals are incomprehensible and keep cutting out -- it turns out it's like that in the original track:

https://www.youtube.com/watch?v=i1gVxKhdGPs&t=57s

i got confused at first, and then noted that this was an awful piece of music to pick for a compression demonstration.

not the music taste itself, but perhaps he should have gone for something more known that would be easy for A/B tests.

Tom’s Diner by Suzanne Vega would've been a neat choice, as it was the track used to assess and refine the MP3 compression algorithm during it's development.

Good point! Replaced the video.

Didn't Top Gear Overdrive (1998) use MP3 tracks? It had several tracks by Grindstone that would play during races. The audio quality isn't that great but lyrics are mostly comprehendible.

https://www.youtube.com/watch?v=kyXsWd7DpEs

Wipeout 64 had some music by Fluke and Propellerheads but I'm pretty sure it used some tracker format to re-create the tracks from short recordings and honestly, they sound awful, not just because the audio quality but because the repetitiveness and lack of progression.

It was all great until they mentioned LLM usage.

The demoscene and homebrew scene is dying as we speak.

No it's not, but it's splitting into pro-AI and anti-AI segments.

Think about it, if someone embedded GPT-2 into a demo in 2019 the same scene would've found it hilarious. It's not the AI, it's how we use AI and everyone being sick of it.

Is it such a sin that he used an LLM to transpile some code?

Great read, thanks!

Pardon my ignorance but one thing I'm curious about, if the decoder fills almost the entire 4k RSP instruction memory (IMEM) - wouldn't that be something games need for graphics? If so, then is this still usable for say a game?

Like the DMEM, you can replace the IMEM at any time. In this case, when you're done with processing a block of music, you can DMA the code for 3d transforms or anything else to the RSP.

Libdragon handles this transparently for you with a queue[1].

[1] https://libdragon.dev/ref/rspq_8h.html

I wonder if this would also be a great fit for the MAS3507D! Supposedly you can replace the program with your own in a 1-kiloword RAM (the default MP3 decoder is in ROM, and probably larger). If you can fit this codec in 1024 MIPS instructions, perhaps you can do the same in 1024 of these DSP instructions…

Too bad there is seemingly zero public documentation on the format of these programs, though (and no emulator or similar).

Beautiful! I long wanted to write my own audio codec, this might be a nice challenge.

Sorry but what? Half-Life on an N64?? it looks incredible!

There's an explosion of N64 ports lately, with a huge mix of new libraries (libdragon) pure fan passion, and LLM assistance.

Some I'm tracking:

- Duke Nukem 3D (ie, the fully-functional PC version) https://github.com/BigChat88/N64DukeNukem3DPort

- Portal 64 Still alive https://github.com/mwpenny/portal64-still-alive

- Resident Evil 64 (ie, not RE2, but the first one) https://www.youtube.com/watch?v=_xRiwy8Z6S0

- Silent Hill (still an early WIP) https://www.youtube.com/watch?v=v7fM06jmHso

I haven't personally tested, but these look interesting:

- Simpsons Arcade https://www.youtube.com/watch?v=zH9JYQVzKDE

- X-men Arcade https://www.youtube.com/watch?v=ah3nB8MP4CM

- TMNT Arcade Collection https://www.youtube.com/watch?v=CiR5-ZpLtyI

At this point, they're coming out faster than I can play them. I'm in the middle of the 2nd Episode of Duke 3D right now and it's quite nice. The original 64 version was always fun, but mid-level saves are really a game changer.

> There's an explosion of N64 ports lately

There's an explosion of ALL ports lately. It's crazy.

Ocarina of Time only got released today officially but it's already playable on ARM Macs. It's not even a port per se but executing the Switch ARM code natively. And of course it's also playable on emulators regardless of the platform

libdragon certainly isn't new; I distinctly recall talking to dragonminded (the original author) about her efforts to reverse-engineer the RSP code that came with the n64 devkit back in 2008 or so and her subsequent efforts to reverse the RSP. I can totally believe the increased rate of games though

>"Time-domain codecs (such as ADPCM) need significantly higher bitrates for this quality. Subband codecs, such as MP2, have much worse quality at comparable bitrates, and more “modern” transform codecs, such as MP3, usually need at least an 80–100 MHz CPU. For this GBA demo, no existing codec would fit the bill.

So Aikku93 not only wrote a decoder in ARM assembly, but also created a whole new codec: The Ultra Low Complexity Codec (ULC).

ULC is a transform codec and somewhat similar to MP3. ULC transforms slices of PCM data into the frequency domain. Like MP3, it uses the Modified Discrete Cosine Transform (MDCT) to do so. But unlike MP3, it does not split the signal into multiple bands (ULC's MDCT is “full-band”). Where MP3 uses classic entropy encoding (Huffman specifically), ULC does… not. Instead, transformed coefficients are stored with a very simple nibble-based syntax.

In short: ULC is what MP3 could have been. The same core idea, but without the cruft around it."

Nice!

This would make a great audio compressor / decompressor to run on older computers, on older operating systems, on older more resource-constrained hardware (not just the N64) !

Anyway, really nice!

https://en.wikipedia.org/wiki/Modified_discrete_cosine_trans...

Reply on news.ycombinator.com