sabalaba

Memory pooling is irrelevant for DL training. 24 GB is enough to run batch size of 1 for Bert-Large so honestly this is a good choice. Some folks are saying that 2x 2080 Tis would have been better and that's true if you're doing convnets but any large scale language model fine-tuning you'll want to have at least 24 GB of vRAM.

Nothing really has changed in the last two years in terms of training cost. I think the author is making unreasonable extrapolations based on changes in performance on the Dawn benchmarks. A lot of the results are fast but require a lot more compute / search time to find the best parameters and training regimen that lead to those fast convergence times. (Learning rate schedule, batch size, image size schedules, etc.) The point being that once the juice is squeezed out you aren’t going to continue to see training convergence time improvements on the same hardware.

Also, because you cited our GPU benchmarks, I also wanted to throw in a mention our GPU instances which have some of the lowest training costs on the Stanford Dawn Benchmarks discussed in the article.

https://lambdalabs.com/service/gpu-cloud

Lambda | Senior Software Engineer | San Francisco, CA (Dogpatch) | Onsite | Full Time

We're looking for a software engineer who wants to write code at our Deep Learning workstation and server company. You will:

* Be responsible for the full stack

* Build reliable, scalable, highly available systems

Required Experience:

* Bachelor's degree in technical field (CS, Physics, EE, Math, etc.)

* Extremely comfortable in a Linux/Unix shell

===

We're located in San Francisco 5 minutes away from the 22nd street Caltrain.

Apply to:

https://lambdalabs.com/careers

Great tutorial. Now instead of the 8x V100 $24/hr p3 instance, you can use a Lambda GPU Cloud 8x V100 instance for $12/hr (https://lambdalabs.com/service/gpu-cloud). It’s crazy how much Amazon still charges for older hardware.

In addition, when you’re first starting off with Deep Learning you should just build a GPU machine under your desk and use that to train. The suggestion in the post to do horizontally scaled distributed training is a great way to horizontally scale your monthly bill too. I’ve seen dozens of people spend $500 on an AWS training experiment only to realize they could have bought a 2nd hand 1080 Ti with that same money and been able to train for the next 2 years.

No, I already multiplied 624 TFLOPS / GPU * 8 GPU = 4992 TFLOPS (the 5 petaflops number).

I'm saying that you are still missing another 8x on the way from 19.5 TFLOPS / GPU to 624 TFLOPS / GPU. 19.5 (base FP32 theoretical peak performance) * 2 (FP16 instead of FP32) * 2 (counting sparse matrix ops as dense ops) * 8 (unknown) = 624 TFLOPS.

No, the A100 has a 19.5 TFLOP theoretical peak for SGEMM[1], real world benchmarks will likely achieve 93% of that, and so the DGX A100 will be 145 TFlops of FP32 SGEMM performance or 0.145 FP32 PFLOPS. Maybe in 72 FP64 TFLOPS. FP64 is what the TOP500 benchmaks count.[2]

The 5 "petaflops" number is a creatively constructed marketing number based on FP16 TensorCore "flops", sparse matrix calculations, and then multiplying by 8x for some reason. They basically take the 19.5 FP32 TFLOPS number and multiply it by 32x to get to the claimed 624 "TFLOPS" for a single A100. 8 * 624 = 5 "petaflops". I see they get 2x by actually using FP16 instead of FP32, 2x from counting sparse matrix ops as dense ops, and 8x from somewhere else that I have no idea.

[1] https://devblogs.nvidia.com/nvidia-ampere-architecture-in-de...

[2] https://www.top500.org/resources/frequently-asked-questions/

Lambda | Senior Software Engineer | San Francisco, CA (Dogpatch) | Onsite | Full Time

We're looking for a software engineer who wants to write code at our Deep Learning workstation and server company. You will:

* Be responsible for the full stack

* Build reliable, scalable, highly available systems

Required Experience:

* Bachelor's degree in technical field (CS, Physics, EE, Math, etc.)

* Extremely comfortable in a Linux/Unix shell

===

We're located in San Francisco 5 minutes away from the 22nd street Caltrain. Right now everything is remote!

Apply to:

https://lambdalabs.com/careers

Lambda | Senior Software Engineer | San Francisco, CA (Dogpatch) | Onsite | Full Time

We're looking for a software engineer who wants to write code at our Deep Learning workstation and server company. You will:

* Be responsible for the full stack

* Build reliable, scalable, highly available systems

Required Experience:

* Bachelor's degree in technical field (CS, Physics, EE, Math, etc.)

* Extremely comfortable in a Linux/Unix shell

===

We're located in San Francisco 5 minutes away from the 22nd street Caltrain.

Apply to:

https://lambdalabs.com/careers

It’s likely some type of segmentation network like Mask RCNN or SegNet. Look up Mask RCNN for some state of the art segmentation results. This stuff has been able to run on mobile phones in real time for years now.

Lambda | Senior Software Engineer | San Francisco, CA (Dogpatch) | Onsite | Full Time

We're looking for a software engineer who wants to write code at our Deep Learning workstation and server company. You will:

* Be responsible for the full stack

* Build reliable, scalable, highly available systems

Required Experience:

* Bachelor's degree in technical field (CS, Physics, EE, Math, etc.)

* Extremely comfortable in a Linux/Unix shell

===

We're located in San Francisco 5 minutes away from the 22nd street Caltrain.

Apply to:

https://lambdalabs.com/careers

- It frequently stops working and needs to have the chrome extension re installed (at least on Linux).

- It’s sluggish.

- The password sharing experience sucks.

- The drop down menus often get obfuscated in weird ways.

Lambda | Senior Software Engineer | San Francisco, CA (Dogpatch) | Onsite | Full Time

We're looking for a software engineer who wants to write code at our Deep Learning workstation and server company. You will:

* Be responsible for the full stack

* Build reliable, scalable, highly available systems

Required Experience:

* Bachelor's degree in technical field (CS, Physics, EE, Math, etc.)

* Extremely comfortable in a Linux/Unix shell

===

We're located in San Francisco 5 minutes away from the 22nd street Caltrain.

Apply to: https://lambdalabs.com/careers

Yea we’re seeing this all over the place at Lambda (https://lambdalabs.com). Most people running consistent GPU training or inference jobs are building on-prem clusters or even groups of workstations.

It just doesn’t make financial sense to use the big the cloud service providers for those with consistent workloads. I always hear stories where folks have saved hundreds of thousands in infrastructure costs with owning + co-lo.

Only BSL4 lab in Mainland China. Built with French assistance. Was controversial because a culture of open communication is important to maintaining BSL4 standards.

Very impressive work. I would love to see what this looks like when you start to drop more and more frames. I’d be curious to see how this technique starts to degrade as the input FPS goes down.

Seems like it already can dramatically reduce the amount of work involved in making an animation. Very cool to see pixel art animated at 60 FPS!

The text generated by GPT-2 is far superior to HMMs. GPT-2 was able to perform unsupervised machine translation and answered more than 5x as many questions correct on the SQUAD Q&A dataset than the previous best pure neural model.

Not to mention that the text generated by GPT-2 can often fool an online reader whereas HMMs have the problem of being long-term incoherent and don’t reference back to subjects of the sentence like GPT-2 often does.

I’m not staying you should believe the AI hype in news media. But the paper does contain a lot of thorough analysis and comparison to the previous state of the art.

https://cdn.openai.com/better-language-models/language_model...

It’s not 10x more expensive. For example, Cloud TPU costs $12 for ImageNet training. AWS is $42. Lambda GPU Cloud is $19.

You also need to take into account the cost of changing and tuning your code to run on a TPU.

https://dawn.cs.stanford.edu/benchmark/#imagenet-train-cost

Having worked on a similar product to what you're describing, an always on camera embedded in a baseball hat that uses face recognition to catalog everybody you meet, I don't think it is going to happen any time soon. Primarily because it's not very useful. You end up with a bunch of data that you never look at again.

There's a reason why that stuff has remained a niche for people interested in trying to implement Vannevar Bush's "As We May Think".

Perceived pace of progress when seen from the inside of a field is significantly slower than the perceived pace of progress from the outside. That said, I will agree that the progress in AI and face recognition since 2011 has been phenomenal.

Yes, go to scholar.google.com and search the name of the network. You’ll find the original paper (it will likely have many citations) which will describe the architecture in detail and have a useful diagram for you.

I have always absolutely hated this diagram and think it should go away. I have also never seen somebody who understands the content of the diagram share it as a useful pedagogical tool.

For example, with CNNs, you are building up feature activation volumes based on the entire previous layer. The edges between layer N only connects to two nodes in layer N-1. What are the nodes even supposed to represent? This is not how CNNs works. This explains nothing and is actually just confusing.

This entire diagram should be re written using the block diagrams from the actual papers.

Extreme alarmism is going to make the Internet an over-regulated hellscape that mostly benefits incumbents and further reinforces monopoly positions. We would all be worse off if something like what you’re describing was implemented.

I completely agree with all of your points, having built a Face Recognition API in the past at Lambda. If it’s a feature, high willingness to pay customers will just build in house. Low willingness to pay customers will want to use it for free or not enough money to keep the lights on. It’s difficult to build an API business with pure CV/AI/ML for that reason.

I know this is unrelated, but I want to thank you for your work on Kessel Run. It’s inspiring to meet folks in the DOD working hard on making software work for our national defense. I ran into some of your colleagues and was very impressed and excited about that program.

Don’t know about the Philip K Dick work but William Gibson has this in Neuromancer.

I’m not sure if this is the same person but it sounds about right. The cowboy-pilot whose ranch he landed on died about 20 years later in a plane crash:

https://oklahoman.com/article/2151107/cowboy-hall-figure-die...

Fascinating read. Thankfully the puritans won out. Imagine a ‘demartialed’ colony. Not one that would be very fit to rebel.

Hey HN, just wanted to post here to say that if you need more than 6 hours training (free tier limit) we offer a 4x GPU machine that is only $1.5/hour. Just wanted to let everybody know what kind of competition is out there while you're doing your price comparisons:

https://lambdalabs.com/service/gpu-cloud

Exactly. AWS is simply too expensive. You can buy a Lambda Quad GPU workstation and pay it back in a couple of months. If you want to save more you can just build it yourself.

https://lambdalabs.com/blog/8-v100-server-on-prem-vs-p3-inst...

Software. Software. Software. Just two companies, Google and NVIDIA, have publicly launched a viable service or software stack. Just two companies have successfully written a "sufficiently advanced compiler". Just two companies actually have a product. And Google refuses to step into the arena and actually compete with NVIDIA. Man, what a time we live in.

And no, AMD doesn't count. ROCm is a mess.

Feel free to apply if up your alley: https://lambdalabs.com/careers