Hacker News Reader

Armv8 has an opt-in feature you can turn on to ignore the top byte: https://en.wikichip.org/wiki/arm/tbi This is also where the pointer authentication code goes for arm pointer authentication: https://lwn.net/Articles/718888/

On x86_64 and arm without those features enabled, the top bits of the pointer must be sign extended. This means that x86_64 by default gives you the top two bytes to play with as long as you don't use the values 0xffff or 0x0000. Attempting to access a pointer whose top 16 bits aren't a sign extension of bit 48 will fault. You can still safely play this game as long as you fix up the pointer before dereferencing it.

"Attempting to access a pointer whose top 16 bits aren't a sign extension of bit 48 will fault."

Currently. Coming soon to an Intel chip near you is 57-bit virtual addressing and 5-level page tables [1]. It would be quite a bug that would only crash on new Intel hardware on probably quite full memory maps where your pointer fix up code wouldn't restore bits 48-57 correctly.

[1] https://www.phoronix.com/scan.php?page=news_item&px=Linux-De...

It'll probably be quite a while before operating systems rush to turn on extra level of page walk fun, increasing TLB miss penalty even more.

If only x86 could have 64 kB pages...

Of course you're right it's not a great idea to use those bits. Eventually they will be in use, although it's probably 10+ years.

The linux kernel has patches for it submitted now, ready for when the hardware arrives https://www.phoronix.com/scan.php?page=news_item&px=Linux-De...

My point was that very few need more than 48 bits of virtual address space. Having extra layer of lookup will reduce page walk performance.

52-bit physical & 57-bit virtual address space is a no brainer if you have more than 128 TB of RAM installed, of course. :-)

Isn't the tendency to use more and more large pages (1MiB) on x86 anyway? And then just use user space allocators to split them up for malloc.

x86 large pages are 2 MB (64-bit) or 4 MB (32-bit). Even larger page variety is 1 GB. They do save a lot of TLB misses, but can be painful to reliably allocate.

> Currently. Coming soon to an Intel chip near you is 57-bit virtual addressing and 5-level page tables [1]. It would be quite a bug that would only crash on new Intel hardware on probably quite full memory maps where your pointer fix up code wouldn't restore bits 48-57 correctly.

Yeah, I'd hope that code that uses such tricks would have an error check for pointers that use the bits it wants to use. Better a clean crash than corruption.

It'd be neat if a program could tell the OS what address range is acceptable. Linux has mmap(..., MAP_32BIT, ...) but obviously that's pretty limited. Maybe something like map(addr, ..., MAP_MAXADDR, ...) which would tell it addr represents the maximum acceptable address to return. So if you intend to use the top 16 bits, you could tell it this mapping can't use those.

Edit: oh, actually, I see they do something kind of like this. https://lwn.net/Articles/717293/ "An application that needs [virtual address beyond 48-bits], and which does not play games with virtual addresses, can provide an address hint above the boundary in a call to mmap(), at which point the kernel will understand that mappings in the upper range are accessible." Still not quite as flexible as I was imagining but not bad.]

The coolest thing this enables is a way more precise ASAN: https://clang.llvm.org/docs/HardwareAssistedAddressSanitizer...

Back around 2001 I was part of a webdev shop that had its own proprietary application server. It pre-parsed HTML files for <script language="perl"> or <script language="python"> and would run whatever you put there. We linked in slightly patched perl/python libraries so we didn't have to start a new interpreter every request. There were a couple in-house RPN languages too, and other comment-based markup for easy loops/interpolations. One design goal was to let you round-trip between developers' editors and designers' Dreamweaver, so that even after adding code the designers could still see real HTML and usually even change it. I haven't seen any system so successful at that part since. (I love Haml, but design tools don't. :-) Anyway, I got to help out with porting the app server from Solaris to Linux, and one of the tricks they were using was to stuff some extra info into unused areas of the Solaris file struct, basically to smuggle it around without needing to rewrite anything. As long as you could get to the struct, you could retrieve what you put there. It was a brilliant-but-horrifying trick, and of course it was a big challenge for the porting effort.

This is majestic. Is your current work anywhere near as interesting? :P

It was pretty cool. The company was founded by MIT folks, so I guess they weren't afraid of mixing C & scripting languages. :-) I was just a junior developer, so I barely understood what was going on. The company tried to build an open source version, but it suffered a lot from second-system effect IMO.

I don't often get to do work as cool as that, but I've really enjoyed pursuing more researchy things in my spare time. Right now I'm working on adding SQL:2011 temporal features to Postgres, so I guess I'm still getting my C fix from somewhere. :-)

Was this https://openacs.org/ ?

No, but it looks like it's still around (at least the open source successor, which was not as nice or stable as the original): https://sourceforge.net/projects/integratis/

> "Repurposing Top Bits" - don't do that. Honest.

+1

I once shrugged this of as paranoid BS and got bitten by it in the very same project. It did cost me some all-nighters to fix the results of my arrogance.

Honestly, you may as well use the bits if they are available, because as you say too many will break and will break backwards compatibility.

With x86_64, the top bits have to be all 0, or all 1. You can still reuse them as long as you clear them before actually using them as pointers. Of course that might break at a later date is memory spaces increase in size and they are used, but I'll deal with that when it happens.

You'll have to deal with APIs that do return full 64-bit pointers with real information in the top bits. You won't be able to play games with these bits. Stuff is going to break unless you are careful. If you are shipping a platform of some kind then you must also ensure that your customers cannot get into trouble (if you tell them "Hey, these upper bits are up for grabs" then you are doing them a disservice, in several ways).

"We know this is wrong, but we'll deal with it when it happens" is a demonstrably expensive road. I can't think of many clearer cases of refusal to learn from history in computing.

I think as with everything it’s about context. If you play games deep within some internal component that is very limited in scope, easy to rewrite, and it buys you performance that is truly worthwhile, that’s one thing. If you’re doing this with pointers beyond some region of local control, you’re asking for it.

Nothing on x86_64 is returning "full 64 bit pointers", unless they are also playing games, as CPUs only support 48 bits of address space.

Also, your suggestions are a bit strict, in my opinion. Every VM or interpreter I know about uses this kind of trick, usually to squish things like integers. Not doing this requires adding an extra allocation for every integer.

> Nothing on x86_64 is returning "full 64 bit pointers"

Well, I'm going from the processor reference manuals from Intel and AMD. Existing implementations are 48 bits, and future implementations WILL have more. I trust these folks to make good on their promise. (Similarly, the 68000 had 24 bits, and when the 68020 came out it had full 32 addressing, and an MMU, and badly written software broke).

There are pretty safe ways to do tagging; common techniques usually distinguish limited range integers and pointers with a low bit, and pointers often have extra bits since those are relatively safe to manipulate. I have not seen runtimes that use high-order bits for tags in quite some time, probably the late 80s; you haven't seen that software recently either, because it doesn't work anymore.

The x86 segmented architectures (which is all we had from Intel until the 386's flat address space came out) had several variations on pointers, basically a menu of terrible tradeoffs between fast and small code versus small and large address spaces. Far pointers were formed with a segment and an offset, and the game was all about making these multipart pointers efficient for things like comparison. In the face of pointer normalization, you mucked with high bits at your peril.

The one thing I'm sure of in this industry is that software is always going to be getting bigger. :-)

The responsibility for this lies not with just the programmers, but the address decode logic. The hardware should have seg faulted on non-zero upper bits (which would have required the addition of a few OR gates, hardly a problem).

The hardware - at least, x86-64 hardware - does raise an exception on use of a non-canonical address.

So to use these dirty tricks successfully, you have to mask out those bits before you dereference your pointer.

Apple’s market cap is $983B and IBM’s is $125B. So somehow they’ve survived this lowbrow hack.

Seriously, with 64 bit addresses available on the iPhone I’m typing this into, this is an excellent trick, especially for applications. Even ARM’s TBI leaves 56 bits which is more address space than a data center’s DRAM.

  log2(16 billion * 100,000) is about 50

You have a point at 32. You’re just wrong at 64.

Right, because the quality of every lesson one learns about programming practice should be evaluated by the market cap of the firm where it was learned.

The parent post mentioned those companies and the corporate 'cost' of recovering from this mistake. Take it up with him.

"You see, these guys hit this boulder , and they still ride their 60-ton battle tank! Why should I not try the same with my car?"

See also: survivor bias.

That’s nice rhetoric but I didn’t pick the survivor examples.

You did, by mentioning the companies that are alive and kicking.

OTOH DEC placed a bet on new and cool Alpha architecture, and died switching (bought by HP).

Those examples and companies, IBM (the IBM 360) and Apple (the Macintosh), were picked by the parent post.

DEC was acquired by Compaq not HP. Compaq was later acquired by HP but before that happened Compaq sold the Alpha line to Intel because it would have competed with the Itanium. The Alpha was then licensed by Intel to the Chinese (according to Gustafson) and was used as the basis for the Sunway Taihu Light.

Can I help you with anything else?

Meh, when you get to the point where you're exceeding your available address space, there's probably a million other refactorings you had to make to address all of the other architectural changes.

Or, you play by the rules and your old code continues to just work. Might not be optimal, but at least it will run. Your customers will thank you . . . well, we all know they won't, but they won't be speculating about your ancestry in public forums.

I don't remember how many Mac applications were '32-bit clean' and ran without modification when VM was turned on. I do know that some apps had to do significant rework, and that there was a whole generation of abandonware where developers didn't think it was worth the effort and just walked away from their customers.

Can't you just make sure that your allocator only uses the bottom x bits? (Of course you still need to be careful when interfacing with code that maps memory in some other fashion)

If your allocator calls mmap or an equivalent, and the OS gives you back a pointer that uses all 64 bits (or more than 48, anyway) then what do you do? I guess the allocator has to retry and pray, or maybe just give up and fail. In any event your clever code is not going to work well, since you have made a choice that is non-portable.

I imagine that the guy on the MacOS team who stuffed some desk accessory bits into the high byte of a pointer parameter thought he was being clever. Four or five years down the line he was paying for it, and it took years to fix his original moment of convenience. He could have added a second parameter on the call in question, or allocated another byte in a struct, and the original system would have been slightly bigger but he wouldn't have hatched a nightmare. Go read about '32-bit clean' Mac applications if you don't believe me. It sucked real hard.

A bunch of companies with a bunch of smart people have made the decision to use "unused" high bits in pointers and have later regretted it, to the tune of a ton of expensive remedial engineering. I see a lot of talk here along the lines of "b-b-b-but we know what we're doing" and "oh, 48 bits is enough, and if it's not then we'll deal with it later" and I'm thinking, "This is precisely how the software industry just never learns."

I figure that I have about 20 years left in my career writing software. I've seen address spaces grow from 16 bits (and we used to think "wow, that's BIG") to 64 bits [okay, 48 bits if you're still in denial :-) ] and wow, that's big. But I'll bet I'll see 64 bits generally considered kind of tight before I retire, and I'll bet there are at least half a dozen people on the planet who are up against a 64-bit wall today. (High probability at least a couple of those folks are on HN. Any hands?).

You can ask mmap to map memory to an exact address. You could have an allocator that decides itself what addresses get allocated.

Interesting history, thanks.

Sorry if this is a silly question but is the "upper" in these examples the most significant bit end of the address then?

The most significant bits, yes.

(The world numbers these from 0 to N, least significant to most significant. IBM numbers them from 1 to 32, with 1 being most significant. I guess that made sense to accountants).

Exactly. We more frequently use the lower two bits for this. This also works on 32bit systems. NaN tagging is also frequently used.