Shebang specifically refers to using an exclamation point to define where data should be directed. Same as it is in Bash scripts, just a different context and slightly different syntax.
https://en.m.wikipedia.org/wiki/Shebang_(Unix)
The word "shebang" has different uses outside of computing, but I can't find any use of the word that refers to a ! without a #