I'm not sure the author really understands the math (from his comments) but he's right that it's significant, where significant means "I am at least 95% confident that the observed improvement is caused by this change". The test is fine (assuming he didn't just check the results every day until he found one that would make for a fun blog post).

The reason it feels intuitively broken is because the conversion rate is so low. But there were about 10k impressions in his test.