I’d wager it can probably a few percents of the full runtime. But no matter the variation in the time it takes for LLMs to generate outputs (depending on the nb of tokens produced/input size) likely drowns it completely.
Like 5s+/-1s vs 4.95s+/-1s
Like 5s+/-1s vs 4.95s+/-1s
No comments yet.