So I can't tell easily (mea culpa, I'm new to Go) but were the crypto speedups also timing-attack-safe?
If you want a timing safe cipher, use (X)Salsa20 from go.crypto repository.
Probably the easiest mistake to make is on DH/RSA exponentiation. Using the simple square-and-multiply algorithm gives away plenty of timing data; you usually need to explicitly protect against it by either using blinding or always performing both the square and multiply at each iteration.