Ah sorry, I was only checking same domain XHRs, which doesn't seem to send the Origin header. That combined with the fact that Origin isn't sent for <script> tags seems like it can't be used to prevent CSRFs.
Obviously this is only true for read operations: anything that updates state would still be vulnerable to CSRF (since you don't need to be able to read the result to make the request).
So since you can't force an Origin header to be on all of your legit API requests, you won't be able to differentiate them (using the Origin header) from an attacker with a <script> tag.
Edit: I was confusing X-Requested-With and Origin. My apologies.