They are not wrong. I passed Duolingo Spanish course (it might have been extended since then) and barely remember anything now. I was also studying German in school, and later French by myself to the point where I could even speak with native speakers a little. Hardly remember anything either. English is the only foreign language that I haven't forgot because I use it constantly. But if you're not using a foreign language, after passing the test you will completely forget it in 2 years.
I spent a lot of time barely using certain foreign languages (+ than 2 years). Of course you get 'rusty', but far from forgetting everything.
And it came up back quickly
This tell me Duolingo doesn't make you learn it, but rather memorize their game.
Remember this: everyone is taught their perfect grammar and spelling and quizzed on it for years in school. How many people can say what a past participle is, or even what an adverb is? How many people who passed geometry can still do that level of math from the top of their heads?
Of course you can't permanently learn something. You have to keep it in practice.
It's hard to master and complete a Duolingo course without getting a grip on the language. You'd have to pretty much ignore the "Strengthen your skills" button and never repeat anything.
Even after finishing a course you should use the repetitions for weeks and months to keep refreshing your memory. That's the only way it may stay forever.
I'd love to see a independent study into its reliability/retention. But even if it is completely useless, it is still no more wasteful than any other mobile game people play daily.
I have seen the effects myself. After a month or so, even though I had learned some spanish before, native speakers suddenly slowed down (in my perception).
An important clue: Use the "Strengthen your skills" button frequently to keep the tree golden. Duolingo itself doesn't nag you enough to do that, but it's essential for mastery of the language.
You usually take this test because you will have to speak this language, so the case where you take the test and then do nothing with it seems pretty unlikely. Also, I disagree with your assumption that you will completely forget another language in 2 years (even when not used). I cannot confirm this from my own experience.
I can guarantee you that you didn't forget everything. If you completed the Spanish course, your brain has been rewired for Spanish...
I was thinking at the time (it was in 2005) that they were really out of touch with common sense, and right now I wonder if anybody outside academia gives some credibility to the TOEIC or the TOEFL.