...
When you don't formalize the challenges, the kids just kind of meander, don't get much accomplished and things start to devolve into chaos. I know there's been a bunch in the press lately about how if you just poke a hole in a wall and stick a computer in it the kids will teach themselves differential equations or something and then of course teach it to a bunch of the other kids (believe me, I'd love it if it were that easy), but that's not been our experience at all. Not remotely. We've tried that a number of times in a number of different ways (because you know - it's easy) and when you're feeling burned out from work or didn't have time to write up a bunch of challenges, it's something you can rationalize trying, but we've NEVER seen that work. Occasionally, one of the kids will look something up on the web or figure something out on their own and tell the other kids about it, but that rarely gets them very far.
...
Just to add to this a bit in the form of a slightly different technology. I bought a few Mindstorm kits last year for the class, wrote up a bunch of easy to read documentation and examples and basically told them to go at it, but even with that and the occasional soft intervention, not a whole lot happened in terms of learning. And we tried this a number of times throughout the year. Sure, they learned a few basic concepts, but the efficiency was extremely low, and nothing much was learned that was transferable. This is also pretty much the same story I heard from every well-meaning, thoughtful and supportive parent who bought their "gifted" kid a Mindstorm kit. They basically played with it a few times, mostly with the legos and then gave up and left it in the corner.
...
In regards to the concern about teaching to the test, I make up new challenges every week based on how the kids are progressing, what they're struggling with, what they're excited about, what the next logical learning step might be, etc., so it's not like there's something static that they can optimize against or cynically "game" without actually learning how to code.
Also, while we're very focused on whether they're having fun, exploring and being creative, at this stage the primary goal is to give them enough fundamental skills to be able to do ANYTHING interesting. I mean it takes a lot of work to get them even to that point, and we're not there yet. And the thing is that we're competing against very attractive distractions like Mindcraft and YouTube, as well as all the homework they have to do and the other activities they're involved in, so it's not like being relentless task masters would even fly with them. They'd simply quit.
To sum it up, what we're attempting to do is extremely hard. We're not idiots. And we're really trying. I've probably read just about every related article that's popped up on HN over the past two years, plus 100 others on learning theory and creativity, but a lot of these articles are just people spouting off who've never really tried it in the real world for any length of time or with kids this young, and I can tell you that a lot of it just doesn't work, or not to the degree that you may have been lead to believe. Look, I can't speak for what works for a every group of kids in every context, but what we're doing seems to be working, the kids are having a blast, and if we keep up this pace I think the kids might actually be able to do some real stuff within the next few months, which, in my opinion, would really be something.