Why would the tax people care at all? They are more likely to take your salary from the two jobs, add them up, and charge taxes on the results.

Where I work, a side job is legit (with the obvious caveats, of course!)