There are also clear safety advantages to recumbents, like harder breaking without flipping over and an unobstructed view without sacrificing aerodynamics, so I do not think they should automatically be considered less safe.
I am also pretty confident that the 'manoeuvrability' disadvantage is completely irrelevant for any individual track event. And do you think that it would be an important factor for time trials?