But then they said 250/second when running multiple inference? Again I don't know if their assertions about running multiple inference are correct but why focus on the wrong number instead of addressing the actual claim?
I know it's hard to understand, but you can achieve a throughput that is a few orders of magnitude higher.