In large part the problem was the vision that the language offered was beyond what consumer-accessible hardware could handle well by at least 10 years. The Alto was a bespoke minicompter-esque machine custom built by Xerox and was far beyond what landed on regular office desks until the late 80s. Tektronix and others tried really hard to optimize Smalltalk-80 to run on 8mhz/68000 class machines at reasonable spec and it was still pretty sluggish. On top of clock speed, RAM and disk space was super expensive in the 80s.
And Smalltalk was also heavily blockaded by gatekeepers commercial and academic until it was too late, as well. There was no serious open source Smalltalk-80 until Squeak in the late 90s.
That, and, yeah, interoperability.