Comment by godelski | Hacker News Reader

Comment by godelski | Hacker News Reader

I tried this, with an example image they give[0] and it failed. It shows a value of 1.9, which is where the shadow is pointing but not the actual needle.

I am suspect that the training data is good. Many tasks highly depend on labeled datasets and unfortunately these need to be very large (given current ML methods). The major problem is that these labels are generated from cheap or free labor where labelers are also expected to know the nuance and label appropriately. Nearly (if not every) dataset you've seen has major errors in them that matter. Honestly, one of the only ways to combat this is be insistent on viewing benchmarks as guides. There's been countless times I've demonstrated to others how I can perform worse the training dataset but absolutely demolish results when using customer datasets or real world data (i.e. generalization). Remember, it is absolutely possible to overfit your model even if your validation accuracy does not decrease. There are many ways to overfit and evaluation is unfortunately substantially harder than looking at results on a test set. There's a lot of complexity to this and if you want an effective product you should probably have at least one person on your team that cares about statistics and will nitpick stuff (though you can also over-saturate your team with these people too. I say that as one of these people).

Edit: Actually I believe every example shows an incorrect reading.

[0] https://synanthropic-reading-analog-gauge.hf.space/--replica...

Though I haven't effort into analyzing or tweaking the training, some of the perditions does look like it points at the shadow instead of the tip. This synthetic dataset is tiny (not even 100k) - but still larger than what I could find. I think there's a lot of room for improvement. What do you think is the most important requirement?

That's all fair, and I'm a big believer in context for proper evaluation. My issue is really about the presentation. There's a pervasive issue in research that exaggeration compounds as if one work exaggerates its effectiveness that subsequent works have a higher bar to meet to show improvements and the frequent result is additional exaggeration (even in just small amounts).

> What do you think is the most important requirement?

Data quality and augmentation. I looked at the dataset[0] and the quality is very poor. There is very low diversity and it appears that it is composed of a few images and increased through augmentation. Augmentation should not be handled through dataset generation but dataset processing. Torch will even move bounding box keypoints when cropping or resizing images.

Data quality is one of the most important factors and I know it is common to just get labels to Turk or cheap services but you absolutely get what you pay for.

[0] Please don't harvest emails and usernames to require downloading. I created a dummy account for this because that's absolutely not acceptable. DO NOT SEND PEOPLE SPAM. DO NOT HARVEST EMAILS AND SELL THOSE EMAILS. I happily flag emails that come to my account through means like this until Google identifies all emails from that sender as spam. DO NOT DO THIS. It will just make people hate you and build a devoted base against you. There is no justification for this. You should absolutely rescind the gate and delete all emails you harvested. Your licensing is already an agreement. Don't be shady.

Reply on news.ycombinator.com