Hacker News Reader

Comment by thechao | Hacker News Reader

Any time you can convert a problem from an `N x K` problem to an `N + K` problem, there's some asshole administrator trying to turn an `N + K` problem into an `N x K` problem. It wouldn't surprise me if there's huge amounts of redundant information in there.

View on HN

> In the newly-released data, each "negotiated rate" (or simply "price") is associated with a lot of metadata, but it boils down to: who's paying, who's getting paid, what they're getting paid for, plus some extra fluff to keep track of versioning. The hundreds of billions of prices in the dataset (probably over a trillion) result from all the possible combinations of these things.

They basically denormalized all the dimensions.

Imagine you have a function which takes 5 arguments and returns one value. You could give me the source code and let me run this function. Or you could give me a mapping of every possible combination of the 5 inputs to the returned value. The former could be quite small, but the latter would be a massive number of rows.

> You could give me the source code and let me run this function

If I understand correctly, in this case, that function's source is highly distributed in wetware. It's about as closed-source as it gets; nobody has anywhere near the full source. Each hospital is its own fiefdom!

Yeah this is part of the problem.

But even if you had like 10M rows of pricing and then gave a 2% discount to entity A, 3% discount to entity B, 4% discount to C, etc.

You could publish these discount rules.

Or you could just multiply the 10M rows by the number of different entities giving 10*n M rows.

And then let the consumer of the data try to figure out the rules from the output...?

It would be interesting to see whether it's possible to reconstruct the rules by comparing the negotiated prices to a baseline like Medicare price, and doing classification into discount buckets to recover the categories insurers negotiated.

Reply on news.ycombinator.com