The Aha! moment for me came when playing with SimianArmy, the wonderful Netflix OSS project and in particular, Chaos Monkey.
Rather than build redundancy into your system, build failure in and force failures early and often. This will surface architectural problems better than any whiteboard.
https://github.com/Netflix/SimianArmy
Also, check out boto and aws cli.