> “does my model perform worse than the true model?”
What is "true model"? I can't make head nor tail of that term. I've never heard this before, nor does it make sense to me when I take just the word meaning. > “does my model perform worse than the true model?”
What is "true model"? I can't make head nor tail of that term. I've never heard this before, nor does it make sense to me when I take just the word meaning.In practice, though, these systems are often well-described by distributions. In common parlance for statistics, people thus sometimes like to talk about that large set of unknowns as though it really were just some arbitrarily complex distribution which we don't know: the true model.
From a frequentist perspective, the idea of the true model is very important. Many arguments arise by considering it and trying to understand how to reduce the distance between your estimators and the unknown truth.
Bayesian foundations avoid this entirely, refusing to posit an unknown and instead talking about refinements of what we know.
- Polling for presidential election: The "true model" is the voting preferences of all 300 million Americans. A (uniformly) random sample of N Americans can be used to estimate the true model roughly with standard error 1/sqrt(N).
- Particle physics: The "true model" is the decay probabilities of various particles as computed by quantum mechanics. Different models (and parameters) yield different decay probabilities, and experiments can be used to choose between models and/or estimate model parameters.
Of course, oftentimes the true model is philosophical as you describe.
- In polling, it's a bit of an ideal world idea to think that the true voting preferences of all Americans are (1) fixed, (2) consistently measurable, or (3) even relevant given a lot of people won't vote. These are all sources of unknowns and variances which make me pretty unhappy with the idea of the idea of a true model even here.
- In physics there's definitely a notion of a true model taken under the assumption that one set of equations and models is "correct" (which is its own philosophical problem!) but even then experiments don't measure this true model perfectly. They're also based on correction for variance in measurement and tooling which is assumed to be eventually ignorable. In any case, this has a much more definite notion of a true model, but I still find it difficult to swallow all together.
This bit seems trivially resolved by interpreting "voting preferences" to include likelihood of voting.
Say you have a coin that might be unfair, and you want to estimate its bias. You flip it a bunch of times, and it mostly lands heads.
Predictions of the coin's bias based on this observed data are usually going to be that it's biased some % towards heads. (unless maybe you have a strong prior, but that's a different topic)
But there is also a chance that really the coin is fair, or even biased towards tails and you just got unlucky in your flips.
There's a mismatch between the true model (the coins actual bias) and the observed data (the result of your flips) because of this chance of `unluckiness'
Semantics is the essence of communication.
Suppose I take the function y = log(x) and add random white noise. The function log() and the parameters on the random white noise process are the data generating process. We could then fit a model y = \beta X + \epsilon, and then compare the "true" (first) model to our second model. When the natural world generates our data, the idea behind all this is the same: there is a process which generates the data, and the data reveals information about that process to an approximate degree.
Some reads:
[1] http://www.rimini.unibo.it/fanelli/econometric_models2_2012....
Edit: I don't know if people don't like the grammar, or what?
How about this:
Can you give a non-synthetic, ie. natural, example of a data generating process?
The notation is pretty sloppy for someone handing out statistical advice.
The main point of his argument is that R² compares to observed mean and not into true model behind the data.
Terms "true model", "true distribution" and "true population" are well defined concepts in statistics – unless Jorma Rissanen is nearby.
y = log(x) + u
Where u is an error/noise term, and in this case takes a normal distribution. The error term is meant to Hoover up all the stuff you can't account for because it's impossible to measure everything. As ever, what you're trying to estimate is the parameters for everything but the error term.He's maybe being a little sloppy with the notation, but TBH it's a pedantic distinction that I haven't seen anyone make a big deal of since I last took intro stats. In R, for example, you'd specify the model as "y ~ log(x)" and leave it at that.