We looked at Smalltalk early on but found a number of problems. They included slow execution speed (there was no JIT), lack of static typing (hard for maintenance), and difficulty to strip the VM environment to create an efficient runtime. Finally it had very limited ability to manage storage and memory directly. Without these it's difficult to build fast database systems.
Sybase tried to acquire a Smalltalk vendor in order to fix the VM problems but could not reach a deal so we ended up moving to other technologies. When Java came along we felt it was the language we had been looking for even with the early performance problems.