1. Time
2. One-way communication
1. You sometimes have to waste a lot of time to get articles published. The way the current process works, it is very easy for referees who are either competitors, incompetent, or simply assholes to effectively filibuster the review process. Even when things go smoothly the process is often agonizingly slow.
2. The peer-review process operates under the obsolete limitation of snail-mail correspondence. There is no two-way communication between the referees. Referees often return either unprofessional or obviously wrong (to everyone but the editor!) comments. If the referees could talk to each other this would be sorted out quickly. It doesn't help that editors are typically very reluctant to reconsider rejection verdicts even when they are based on clear cases of referee error.
How would I address this? Use the #$%@!ing internet people! Journals should abandon this one-way snail-mail process and build a private message board for each paper being reviewed. Referees are invited to post their comments anonymously on the forum. Other referees and, ideally, the authors could then reply to those comments. After a reasonable time period has elapsed, perhaps a couple of weeks, the editor should take stock of what valid concerns remain and reach a verdict.