briggers

Machine learning productionization: https://paulbridger.com

Absolutely yes.

I write fairly deep ML performance tuning articles at https://paulbridger.com and the (many) hours I've spent on each article have been hugely worth it.

Many people reach out to me via this work, and when we talk they already see me as an expert or already want to work with me.

I need to blog more, thanks for the reminder.

Before you start optimizing runtime performance: measure, trace, inspect, or whatever is appropriate to understand current performance.

Perhaps you are looking for something like this?

docker run -d -p 5000:5000 --name registry registry:2

https://docs.docker.com/registry/#:~:text=The%20Registry%20i....

I used to love doing this with Clojure, it’s an awesome way to increase productivity by a good chunk.

It’s less about saving the time to re-run something, and more about removing conceptual overhead (I think).

Yes! I do this and love it. Works locally or via SSH with no difference at all.

Nice one! I've long been interested in the ONNX serving path.

Yeah. A 2080Ti doesn't fit in your pocket or in your AR glasses but the same techniques and tools scale down.

BTW, this is pumping the same video file through the network - not just a single file. I don't measure latency, but this is not a deep pipeline so it's easy to calculate.

Mobile phones definitely since these days most of them have pretty powerful GPUs.

Great question, now I wish I'd recorded power consumption for all these experiments. Judging from cumulative hours of watching the output of nvidia-smi I've definitely seen a linearish relationship between utilization and power draw (with a non-zero floor of 30-40W).

Very practical question :) Exactly as you say, multi-stream throughput. Also for faster than realtime offline processing of video. Check the caveats section at the end of the post - DeepStream is probably not well suited to high throughput single-stream inference.

I looked into the GIL saturation as measured with gil_load (https://github.com/chrisjbillington/gil_load), but perhaps I should have included more numbers here.

To me, seeing the GIL held for 40% of time and significant time spent waiting on GIL by other threads was a fairly strong indicator. Keen to hear your thoughts/experience on it.

Great point - dependencies between frames are inherently problematic for many of these techniques.

Everything lostdog says. I've had experience speeding up tracking immensely using the same big hammer I talk about in the article - moving the larger parts of tracking compute to GPU.

Also, in a tracking pipeline you'll generally have the big compute on pixels done up front. Object detection and ReID take the bulk of the compute and can be easily batched and run in parallel. The results (metadata) can then be fed into a more serial process (but still doing the N<->N ReID comparisons on GPU).

Author here. As other commenters are saying, the Pytorch JIT and torchscript might be your friend here.

Alternatively, there are some quite fast OSS libraries for object detection. Nvidia's retinanet will export to a TensorRT engine which can be used with DeepStream.

Author here. I really appreciate your feedback.

Completely agree that almost none of the SoTA github repos are really ready for production and making this stuff work can be pretty hard.

Getting this done on C++ and moving up to the next level of performance is the focus of my next article :)

Nice one.

More useful to me would be something similar that operates on tensors on the GPU.

Doing image annotations on host/CPU often becomes a bottleneck.

It’s important to note this is just within visual range dogfighting/BFM. Also just guns, and also perfect enemy state information. Important progress but it’s super early. Beyond visual range/BVR, coordination with wingmen and many other higher-level tasks still to come (or are still secret).

Similar thing happened to me. Lost access to a perfectly set-up forwarding account. The account recovery process is impossible because I nolonger have the same phone number from 10 years ago.

I think I understand your point about declaring and then later using your layers.

Are you aware of the Sequential module? It allows you to chain together layers into a single variable, making this repetition disappear into a single forward/__call__ on the Sequential.

Good stuff.

In my experience inputs to MTCNN tend to be full frames, so the uniform dimension requirement is usually met.

I see from the MTCNN code that this repo (like all others I've seen) is still bouncing tensors between GPU and CPU while passing between the P/R/ONets.

So many ML repos make this mistake in pre/post-processing and end up bottlenecked on CPU.

Anyone know of an MTCNN that's been ported to run more or less fully on GPU? (Or even that does batching instead of an image-by-image approach?)

In the spirit of balanced anecdata, I've flown to the US about 10 times in the last 2-3 years on business and pleasure. Multiple airports and it has always been efficient and friendly.

Well, in Singapore you'll need at least one SG-resident board member. Probably not you if you'll be travelling constantly.

On the other hand, if you can manage this I've found forming and dissolving companies in SG to be very easy, not to mention the tax and accountancy processes.

Likewise. AirBnB.

Awesome. I'd do it if there was some overlap between people I wanted to work around, could tolerate living with, and who had my freedom.

Pretty interesting. This is what I've been doing for the last 12 months throughout Europe, but with a new location every 1-2 months.

At first the organisational details were the frustrating part, but after almost 12 months and 8 cities the lack of longer term friendships is more of a problem.

In regards to "the headaches of looking after 100 people", surely that's something that can be addressed by limiting the scope of services provided? I think handling 1) accommodation and 2) work would be more than sufficient for most responsible people.

I've been moving around with a single carry-on suitcase for a year now. It seemed like a challenge at first, but turned out to be very easy, with a few restrictions. Sure, I can't take up kite-surfing unless I'm willing to rent gear but i can live with that. Living simply is very relaxing.

To give him the benefit of the doubt, I would guess he meant products rather than startups.

Great resource, thanks.

I've spent the last year staying mostly in EU capitals for a month or two, and this site will be a good source of inspiration for the future!

I'm sure the numbers will improve over time with more data. :)

One of my unaddressed desires is to spend more time in small towns and less in bigger cities, but I don't know a good way to find them.