It's hard to tell without seeing a screenshot or video of the game in question, but there are almost certainly ways to optimize this problem.

If there are on the order of even several dozen enemies on the screen at any given time, you could simply maintain runtime sets of different factions and only do line-of-sight checks between hostile groups within a certain proximity.

There's probably hundreds of better ways to do this than by spamming raycasts. I have a VR space game with dozens of spaceships, and there are generally never more than num_spaceships raycasts getting fired every frame