The Visual Workstation line failed because you could get 90% of the performance for <50% of the cost by buying/building a dual-PII/PIII workstation and throwing an NVIDIA graphics card in it. The architecture SGI used for the initial pair of Visual Workstations was proprietary and didn't provide enough performance gain to justify its proprietary-ness, and the later models were just bog-standard PCs with bog-standard NVIDIA graphics cards slapped in pretty SGI cases with huge price tags. The Visual Workstation line would've failed no matter what operating system they ran.

But the Visual Workstation line's failure was merely one gut punch in a succession of gut punches that destroyed SGI. SGI had two core competencies: graphics workstations and HPC. The availability of cheap (comparatively) multi-processor x86 workstations running ports of formerly IRIX-only software killed their workstation lines, and the combination of Itanium being late, Itanium trailing way behind x86 in $/FLOPS and watts/FLOP, and x86 clusters with fast Myrinet (later 10GigE and Infiniband) interconnects becoming in vogue killed most (not all, but enough) of SGI's HPC market share.

(I type this all with a bit of sadness: part of my first job out of college was running and porting software to a 20 CPU, 20 GB RAM SGI Power Challenge 10000, and I still have a soft spot for exotic big iron.)