This is an awesome post! What a fantastic idea! I really like the way it includes a social interaction aspect, and I agree it is virtually no work for you, so very much a win, win!

What I liked most though, is a real benefit of this activity that you did not actually articulate: by asking kids to choose a sampler of questions that “represent” the concepts of the unit, they are having to consider:
-what are the concepts we learned
-how can I best include them all within 5 problems
-what level of difficultly should I make them
… and I am sure lots more.

I would be really tempted to try this as well, but in addition, get them to blog about it: what questions did they choose, and why? This would really force them them to reflect thoughtfully on the unit as a whole.

Another great benefit that you did not mention: differentiation. By getting students to select questions (or you could allow them to invent them also!) they will choose varying levels of questions depending on their own level. This could lead to nice opportunities for differentiation in the math classroom.

I am not that familiar with the Khan Academy program, but there are so many important skills we teach that can’t happen by that model — how to speak, to listen, to write, to be part of a community, etc….. these are also hard to test, so they don’t register.