What if you click an action in a popup and closed the popup or navigate to another page in the SPA all-together. Without a toast you have no way of knowing if an async action failed.
I agree though that we should try to put the info closer to the user's attention, but still Toasts should be used as a fallback.