Is there a reasonable use case for linking to line-numbers on branch/tag on github? If not, it might be good UX to remove that option. i.e. only honour such links to immutable urls.
Further, even if you're looking at a mutable reference (like master), the line number links could/should be to the underlying immutable url (based on the commit).
If there is a marginal use case for linking to line numbers of mutable refs, perhaps it should be the non-default case, discoverable by keyboard shortcut like this...