Hacker News Reader

Comment by metropolisdelaq | Hacker News Reader

Interesting post, with many insightful comments but I think regarding data validation, this fella needs a bit of strong typed language training and functional programming skills, a lot of the problems from the current ML ecosystem comes because many use python, simple as that.

View on HN

Data validation in this context means checking the data you're training your model with (or validating the data you're about to predict something on). Things like checking if the format of the data matches what you encountered during development (same columns, same values in categorical columns, ...), if the proportion of missing values is roughly the same than the original training set, if the means and variances of numerical values are also roughly the same than what the original training set had, ...

I believe what he means by "data validation" is to validate that your data is matching up with reality. For example if you do real estate predictions but your data set doesn't include any low-price sales because your broker didn't list them. Then your data doesn't cover the lower-price part of reality, so all of your predictions are going to drift upwards, due to the gap in your data.

This is correct, and strongly typed languages don't help.

Often your data validation process does things like running descriptive statistics (means, standard deviations, etc) on particular columns and making sure they are within reasonable ranges.

Strongly typed languages don't have any particular advantage for this kind of thing.

Reply on news.ycombinator.com