Hacker News Reader

Comment by Legend2440 | Hacker News Reader

> I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU.

>These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold.

This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle.

You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.

View on HN

> This is just benchmaxxing.

I believe the normal term is "gaming a benchmark test".

Nothing in what you quoted is related to that. It's a normal way to train and test a classifier.

The only way to determine if it's brittle is to test on a different dataset and look at the sensitivity and specificity.

Even so, there is nothing wrong with a model that is trained to a narrow domain when it doesn't need to deal with "wild" images.

> benchmaxxing

Jesus Christ the techbros are stumbling upon the fundamentals of fields they’re unfamiliar with and treating them as completely novel discoveries. No wonder they’re so high off their own farts in delusions of grandeur.

Training a task-specific small model is just good developer practice.

That's just good developer practice, not "benchmaxxing".

Benchmaxxing is when someone presents a model as being generally smart, but actually quietly overtrained it on some benchmarks that were supposed to be proxies for general information.

If you train a small model on a specific task that's not "benchmaxxing" because there's no deception involved. It's just training a model for a task.

TIL: benchmaxxing, aka overfitting?

It is benchmaxxing because the test distribution in these kind of image dataset generally matches the training much more than real world image. It used to be the biggest issue in real AI in most companies that the test data and real data is either different or risk getting drifted apart.

So it is not solving a task, it is solving the same test set. You can easily see these in simple OCR datasets like mnist. Draw a number yourself. It would be much less accurate than the 99% that good architecture claims.

Ok first of all. benchmaxxing, is not a well defined term. It's a word made up by the twitter/reddit crowd to describe overfitting.

That is not overfitting, that is training for a specific task. The measure for overfitting is how well it works on previously unseen data, hence the 200 test images removed from training and validation. That set serves as a second order of validation for the researchers to check themselves they have not overfit the model to match only the training data.

What would be an unbenchmaxxed model here? A general image classifier like Inception v3, without the fine-tuning?

The 200 test images here are 'wild'. Fine-tuning in-domain is the whole point. This is what model training is. A skin cancer diagnosis tool doesn't also need to generalise to dog breeds to be valid, and will almost always perform worse if it does.

This is standard practice.

I developed many models from scratch deployed in various fields.

When you want to automatically detect a pathology from a give scan, you want it to be model that you own, train, use, and update, and not a generic vision capable LLM.

This is NOT what benchmaxxing is. You want the model to be accurate in terms of your cared metric- depending on the problem.

This is a good point and was an issue for skin cancer AI in the early days but has largely been solved now - newer models use ensembles of vision transformers and much greater number of training images and have been proven accurate in clinical trials. See for example https://skin-analytics.com/ (I am not linked in any way)

Could you explain why this is benchmaxxing?

This is the same way LLMs are trained. Are they benchmaxxing and useless out of domain?

Being good at the intended application is not benchmaxxing.

Reply on news.ycombinator.com