agcat

So cool!

I found the article helpful!

That's true. Traditional single-tier storage can not meet the throughput and latency demand. My cofounder wrote this piece on a three-tiered storage architecture optimized for both performance and cost - https://nilesh-agarwal.com/three-tier-storage-architecture-f...

The best way to learn is just by running small experiments and being curious about the field.

Just a few weeks ago, along with my partner i started this Sunday experiment of building and learning robotics as a software engineer.

We ordered a one hand robotic arm with orin nvidia gpu to start running experiments.

to commit and make it a habit, we also started documenting our learnings and experience.. if this motivates you you can check it out here - https://youtu.be/pAwkl6zd0ws

I agree! the moment i switched my narrative to do it for myself building and writing content gives me more joy and fulfillment

Love the idea. All the best!

This is a cool product. I carry a JBL speaker with me everytime i use the bathroom in the morning. Haha!

I agree. Documenting your thoughts, ideas on a notebook is one of the best ways that helps you think clearly.

Great piece. I agree its easy to pin point VCs generally in situations like these but ambitious ventures comes with risks.

I came across this new benchmark leaderboard focused on evaluating AI models for education use cases — found it helpful and well-structured

Nowadays, I am working on learning robotics from a software engineer's lens on weekends. I got a yahboom one hand robot with Nvidia jetson nano orin gpu. So far i have setup the robot and next i am going to run basic apps on it. I am also going to document video and blog the journey. If someone else is also playing with robots would love to learn

This is pretty cool!

such a timely advice

Congrats on the launch!

Thanks for sharing. Its a great writeup

You can check out this technical deep dive on Serverless GPUs offerings/Pay-as-you-go way. This includes benchmarks around cold-starts, performance consistency, scalability, and cost-effectiveness for models like Llama2 7Bn & Stable Diffusion across different providers -https://www.inferless.com/learn/the-state-of-serverless-gpus... .Can save months of your time. Do give it a read.

P.S: I am from Inferless

Hey community: In this deep dive, analyzed LLM speed benchmarks, comparing models like Qwen2-7B-Instruct, Gemma-2-9B-it, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, Phi-3-medium-128k-instruct across Libraries like vLLM, TGI, TensorRT-LLM, Tritonvllm, Deepspeed-mii, ctranslate. All independent on A100 GPUs on Azure, no sponsorship.

You can check out this technical deep dive on Serverless GPUs offerings/Pay-as-you-go way.

This includes benchmarks around cold-starts, performance consistency, scalability, and cost-effectiveness for models like Llama2 7Bn & Stable Diffusion across different providers -https://www.inferless.com/learn/the-state-of-serverless-gpus... .Can save months of your time. Do give it a read.

P.S: I am from Inferless.

This is true especially if you are deploying custom or fine-tuned models. Infact, for my company i also ran benchmark tests where we tested cold-starts, performance consistency, scalability, and cost-effectiveness for models like Llama2 7Bn & Stable Diffusion across different providers - https://www.inferless.com/learn/the-state-of-serverless-gpus... Can save months of evaluation time. Do give it a read.

P.S: I am from Inferless.

An internal hackathon project to help you deploy with Triton easily. You just write your model logic, run a command, and Triton Co-Pilot does the rest. It automatically generates everything you need, uses AI models to configure inputs and outputs, and handles all the tedious parts. You get your Docker container ready to go in seconds.

Yes, we made this as part of an internal hackathon project. We are looking for contributors to make this better.

Triton Co-Pilot: A quick way to write glue code to make deploying with NVIDIA Triton Inference Server easier. It's a cool CLI tool that we created as part of an internal team hackathon.

Earlier, deploying a model to Triton was very tough. You had to navigate through the documentation for the Python backend, figure out how to get your inputs and outputs right, write a bunch of glue code, create a config.pbtxt file with all the correct parameters, and then package everything up. It could easily take a couple of hours.

But with Triton Co-Pilot, all that hassle is gone. Now, you just write your model logic, run a command, and Triton Co-Pilot does the rest. It automatically generates everything you need, uses AI models to configure inputs and outputs, and handles all the tedious parts. You get your Docker container ready to go in seconds.

Check out our GitHub repository and see how much easier deploying to Triton can be! It would be great if you folks try it out and see if it works for you.

Technical Product Marketer at Inferless.com - You will be taking care of all things content marketing, SEO and Social Media. Understand the needs of our ideal customer persona and craft relevant content. It's 50% Content - 50% Distribution. At Inferless we are building a reliable Serverless GPU Inference for AI/ML Companies.

DM me on Twitter if interested - @aishwarya_08

This is a good way to do math. But honestly, how many products actually have 100% utilisation. I did some math a few months ago but mostly on the basis of active users, on what would be the % difference if you have 1k to 10K users/mo. You can run this as low as $0.3K/mo on Serverless GPUs and $0.7K/mo on EC2.

The pricing is outdated now.

Here is the piece -https://www.inferless.com/learn/unraveling-gpu-inference-cos...

Technical Product Marketer at Inferless.com - You will be taking care of all things content marketing, SEO and Social Media. Understand the needs of our ideal customer persona and craft relevant content. It's 50% Content - 50% Distribution.

At Inferless we are building a reliable Serverless GPU Inference for AI/ML Companies.

DM me on Twitter if interested - @aishwarya_08

This is a great post. Specially the tricks section. I have been organizing Casual Breakfasts for Developers for last 4 months every 2 weeks. And it's been amazing, have learned so many new things.

You can leverage Nvidia Triton Inference server to do this.

In case you want a managed Infra service look at this cookbook on how you can use Transformers, Inferless to serve multiple LLMs - https://docs.inferless.com/cookbook/serverless-voice-chatbot

One of our customers partnered with us to use Serverless GPUs for production workloads.

They saw benefits like:

1. Dynamic Scaling 2. Reduced Cold start Times consistently at scale 3. Were able to go live in less than one day 4. Maintain separate environments for production, non-production, and development at no additional cost.

Check this out - https://www.inferless.com/learn/the-state-of-serverless-gpus...

Btw, I am the founder of Inferless.

You can also find the same if you know the tokens/sec for different input and tokens variation.

In case you are interested to see results for speed ( tokens/second)

I Ran some tests between LLama2 7Bn, Gemma 7Bn, Mistral 7Bn to compare tokens/second on 6 different libraries with 5 different input tokens range (20 to 5000) and three different output tokens (100,200 and 500) on A100.

These are the results : https://inferless.com/learn/exploring-llms-speed-benchmarks-...