HNHacker News
TopNewBestAskShowJobs

kanyesrthaker

12 karma · joined August 30, 2021

building Hyper
submissionscomments
kanyesrthaker··on Show HN: Hyper – A stupidly non-corporate voice AI app for IRL conversations
hey all, I'm the other cofounder of Hyper -- we'll be around to answer any questions!
kanyesrthaker··on Show HN: NLP Flashcards for Most of the Internet
One thing that we're trying to hit is the actual quality of the generated questions/answers; there's an immense amount of variance in internet content at large and it's difficult to ensure high-quality content without some kind of supervised QA (aside from some kind of hyperparam grid search). We were able to achieve some control over quality through rule-based heuristic filters on our generated questions, but we're trying to make this more robust.
kanyesrthaker··on Show HN: NLP Flashcards for Most of the Internet
Hi, Kanyes here from Ferret. Starting the discussion by sharing an unsolved technical hurdle that may be of interest. We made a decision early in development to perform all inference on CPU to avoid unfriendly production costs and inefficiencies processing single inputs instead of batches.

Sequential models like T5 tend to be large (300mb >), and we observed high latency per inference of approx 8s. We've masked this latency on the frontend, mainly sending concurrent requests with async code (4 at a time) and preloading content early. However, this is kind of hacky and we'd (ideally) want to reduce inference time.

To this end, we've demonstrated roughly 1.7x speedup by converting our model weights in pytorch to a quantized ONNX graph. However, we've found a lot of friction in trying to deploy ONNX graphs to AWS. We understand there are a variety of potential solutions (training smaller distilled models, deploying ONNX, contesting our rationale to use CPU etc), so we're looking for suggestions for the optimal method to make inference faster!

← PreviousPage 2 of 2