This is such a weird part of cryptography... you can have two implementations, which both always produce the exact same output... and yet, one could be vulnerable because of side channel or timing attacks.
Use battle tested frameworks. For crypto, for sites, for everything.
Likewise, with your second example, the storing clear text and bcrypted passwords are different outputs.
When I say the two pieces of code have identical outputs for identical inputs, I mean everything, not just what an end user sees.
Here is a toy example to show what I mean: say you have a function that takes a username and password, and compares it against a stored username and password (lets ignore the insecurity of storing a plain text password for this example). Now, you have two implementations of the function, and we know that both work as desired; they both will accept all correct username/password combos, and will reject all incorrect username/password combos. We have exhaustive tests proving that both return the same result when given the same input.
Normally, this is good enough - both functions are interchangeable, because they are guaranteed to always output the same data with the same input. You would probably choose the implementation that is faster, or used less memory, or whatever.
However, for crypto, this isn’t enough. Lets say in our exmaple, one implementation starts checking each bit of the password. If the first bit of the submitted password matches the first bit of the stored password, it moves to the next bit. If they don't match, it returns false because it knows that the passwords don't match. What a great optimization! You don't have to keep checking once you hit the first wrong bit. The second implementation checks every bit, even after it finds an incorrect match. So inefficient!
However, even though they both work perfectly in the sense that they will always return the correct result, the one that returns false on the first mismatched bit is vulnerable to a timing attack. An attacker can keep submitting passwords, and wait for the response to get a little bit slower... now they know they have the first bit correct. Repeat this for a while, and you can figure out all the bits.
This is obviously a toy example, but this technique has been used over and over again to break crypto implementations. You have to care about more than just the output when you deal with crypto!
Here is a paper about using this technique to break an earlier version of OpenSSL: https://crypto.stanford.edu/~dabo/papers/ssl-timing.pdf
But that's exactly the trouble with side channels: you've just defined "outputs" and "inputs" in a way that excludes them. If you widen your lens a bit to include, say, the timing of the code as an output, or even further to the physical changes of bits in hardware, then the outputs and inputs are no longer identical.
You can even extend that to the lock metaphor: when you pick a lock, you're using the tumbler itself as an "extra" input, and the feel of the pins as an "extra" output, instead of just the position of the pins. The tumbler and the feeling were there all along; it's not that they're really extra inputs or outputs, we just don't normally pay attention to them. Unless you take up lockpicking.
> 2 identical websites, one checks your username+password for real, the other always logs you in.
In that case, it's not a side channel. Presumably the intended use of authentication includes turning away bad credentials.
> one stores bcrypted passwords, then other cleartext ones...
That is an implementation detail. In itself, it's not a side channel because there's no way in the scenario to get the passwords. That cleartext implementation would appear at least as tough against timing attacks as bcrypt, in fact, because it would always decrypt all cleartext passwords in the same time - no info leak there.
My only point is that it is not something you normally have to deal with when you program. Normally, you are only worried about the inputs and outputs that you are intentionally interacting with. Side channels involve ones that you might not have even known existed.
For example, two search algorithms give identical results, but one is O(n) and the other is O(n^2). They’ll both appear to work fine but one will catastrophically break down on certain inputs.
A similar one is where you have an O(1) hash table, but the hash function is vulnerable to collisions and sometimes explodes. That one starts to overlap with crypto in that it could be the basis of a DoS attack.
Besides running time, you might also care about memory usage, or designing an API that’s easy to use and robust against user error.
(1) Zip bombs -- Do you accept compressed user content by first unzipping it and then examining the data? Good luck with the OOM.
(2) Catastrophic backtracking -- Are you still using regex to parse urls/emails? You're almost certainly using a library with worst-case exponential run-time rather than linear.
And so on. If the natural way to solve a problem is self-referencing you're almost certainly going to expose yourself to exponential memory/time consumption for malicious inputs.
The same could be said when using a native application with an undiscovered use-after-free vulnerability or something, but in that case there are some theoretical inputs that'll cause the program to behave in a way you don't expect, like crashing or corrupting data, whereas with flawed cryptography the flaw may never give the slightest hint of its presence no matter how much or what kind of input you throw at it.
[1] For the sake of argument, let's say there's no risk of any kind of backdoors or "Reflections on Trusting Trust"-style hijinks
In other words, don't roll your own… anything?
Just, no. Down that path is lost knowledge and the decline of our civilization. Enough of us must know the fundamentals. https://www.youtube.com/watch?v=pW-SOdj4Kkk
In the Tux data maybe 03038F12128F0303 turns into 49DC195026F3300A and that looks pretty random. It's only visible as the Tux image because every 03038F12128F0303 turns into 49DC195026F3300A, thus preserving the very bold overall pattern of the image.
If you shove a photograph of a kitten through it, that looks like noise, but Tux is made of these big bold shapes and so the resulting pattern is big enough to "show through" the ECB encryption.
As usual, there’s even an xkcd on it: https://xkcd.com/1286/
But that means that ECB isn't actually passing the smoke test, because the casual observer isn't analyzing AES+ECB. They're only analyzing AES.
Recognizing that this is a problem is another matter.