Yes this would have been mind-blowing.

Ps there was a game that had a similar FPS view in those days. It was a maze game with polygon graphics but no textures. I forget the name. No shooting though!

I could tell something was off, but couldn't put my finger on it. Very cool

IIRC, the Atari ROM routines used a different 6-byte BCD floating point format.

I doubt there are any floating point numbers because FP is very slow to emulate (for example, just to add two numbers you have to shift mantissas and 6502 has no fast way to do it). If I was writing a game for an 8-bit CPU, I would use fixed point numbers (for example, 8.8 numbers which use 8 bits for integral part and 8 bits for the fractional part, or maybe 10.6, or 12.4 numbers).

6502 cannot multiply or divide (division is a costly operation even on modern CPUs) so I would use adding or subtracting logarithms for this purpose. 12-bit precision logarithm table requires just 8 Kb RAM (and you can get 13-bit precision with interpolation).

The disadvantage of this approach is that every time you convert a number from linear to logarithm or vice versa you get approximately up to 0.3% error (for 12-bit logarithm). So multiplying or dividing several numbers in a row is fine, but if you have to alternate multiplication with additions you will accumulate an error. So I would look for formulas that avoid this. But for a game a little error in calculations is not noticeable.

Also one might think that the most time-consuming part for pseudo-3D game is math and calculations. I doubt that. The most of CPU cycles are usually spent in rasterisation and applying textures. Is is easy to calculate positions of 3 vertices of a triangle, but it takes a lot of time to draw it line by line, pixel by pixel, and if you want to have textures this time can be multiplied 5x-10x.