For embedded devices you may not have access to Python. You can precompile the graph to the target device in such cases.
Note that in practice, PyTorch is as fast or faster than Tensorflow and, and newer version allow you to "post compute" the graph and export to ONNX to allow embedded inference using Caffe2
A relevant essay: "On Machine Learning and Programming Languages" https://julialang.org/blog/2017/12/ml&pl
In the maximally abstract sense, Python isn't necessarily the best choice for this, but as the closest manifestation to the correct way to do this that I know of is probably Haskell, that seems unlikely to beat out Python any time soon. There's certainly a lot of worse languages.
Ultimately, you'll want a language that has the necessary primitives built in natively, but it doesn't sound like we're to that level of maturity in the field yet.