Would be interesting to read about tools used and more technical information on RE process.
Lots of tools were written from scratch. I wrote otgdbg for probing the device; this program has tons of subcommands to let me manipulate the device in various ways, get/set registers, boot a program on the MIPS side from memory, boot a program on the APE side from memory, copy a new image to flash, etc.
otgimg examines firmware images and prints information about them, like the MAC addresses in the configuration block, etc. apeimg shows information about APE firmware images and can decompress them.
Since the image formats are custom, I had to use linker scripts to build the images, but some fixups could only be done programmatically, like calculating CRC fields. These fixups were done with small C programs which the build system runs afterwards.
The APE used a more sophisticated image format with section headers, etc. The fixup program for the APE had to compress some of the sections, etc. before setting the CRCs.
These tools are all available in the repository, but most of them link to small amounts of proprietary/reversed code which is automatically scrubbed from the public release. It's not a large amount of code which would need to be replaced, though, if someone wants a tool like otgdbg to probe Broadcom NICs in arbitrary ways.
Oh, I should also mention that using clang and lld rather than gcc/binutils made targeting different architectures a breeze. It's long been a bone of irritation to me that you have to recompile gcc to retarget it; with clang, I could target both MIPS and ARM without compiling a new toolchain. https://github.com/hlandau/ortega/blob/master/cc_mips https://github.com/hlandau/ortega/blob/master/cc_arm
Other than that it's a question of building job-specific scaffolding as you go.
> I decided to “emulate” x86 real mode inside C by translating x86 real mode disassembly into C very directly, modelling segment registers explicitly in C
(from the section described as "mentally draining")
Once he'd figured out the algorithm:
> the particular LZSS format used here turns out to originate from some public domain DOS code which someone posted on a Japanese BBS in 1988
Ah, the glory of the Internet long tail.
> After writing the shellcode to facilitate this mode of access, I finally had a way to access the APE's address space.
(relating to the use of provided functions to inject code into the APE, allowing inspection of its point of view and boot code)
> Although the diagnostic tool was quite helpful, ironically this is not because I ever managed to run it. Neither the DOS, UEFI nor Windows versions have ever worked for me. Instead, the diagnostic tools are useful because they contain various routines to probe APE registers, and then print the contents of these registers along with their names. It's not much information, but it's all I have, and it makes all the difference. Pretty much everything I know about the APE that isn't guessed is from the dry reverse engineering of this diagnostic tool.
More reverse engineering heroics, this time on the support tools.
Rather like marathon running or the ascent of Everest, a prime attribute for doing the thing is a refusal to give up long after the process has become painful and unrewarding.
See for example almost any screenshots of ghidra in mainstream-ish media which show completely nonsencial i386 disassembly interspersed with meaningfull looking autogenerated labels and comments. That is what reliably happens when you feed .NET binary into the thing, as it sees embedded debug info and proceeds to disassemble the .NET bytecode as i386 code...