How do they react when they fail Fizzbuzz?
> [I] let them think they made it all the way through the interview
Maybe then, although they get rejected later, they still won't be particularly upset at your company -- since the in person interaction was friendly and positive (i suppose), and you can take some time and write a friendly rejection email or phone call too (i suppose you have a template)