You're right about the bonkers 3D! https://www.youtube.com/watch?v=qwcN9FraNjQ https://www.youtube.com/watch?v=KatdrdEVEwY&t=532s
I think the main thing was that Atari (pre83 company) abandoned the 8 bit line too early, and didn't make the 5200 cross-compatible.
One problem is that these kinds of architectures that relied on special custom chips have inevitable obsolescence built in. When your "API" for graphics programming is a custom chipset at a certain clock rate with certain capabilities, it's just not going to scale up past a certain point. You get initial superpowers, but then Moore's law just makes it pointless.
See also: Amiga.
I agree that the custom chips were a dead end. However the ST line could have beat the mac if management was any good. (as could/should the Amiga, though I was Atari locked at the time and so I didn't pay attention to what they were doing)
But I also think there's just no way Atari Corp could have taken on e.g. Apple regardless of management. They simply didn't have the capital, market penetration, etc. to carve out a segment that big. Apple was already a big deal and although the Macintosh was not really a big success at that point, there was a giant company behind it with top tier talent, and lots of cash.
And it seems those are Polish guys (https://www.atari.org.pl/forum/viewtopic.php?id=4520)
-- EDIT --
I found it: only version for Atari 130XE (i.e. 128KB RAM - this is not the standard, I guess it could run with cartdrige on 65 XL/XE maybe?), year 2006/2007 (fresh :) )