Generally it isn't something to worry about. Should we have a star nearby go super nova on us, or if a gamma ray burst occurred nearby (from a black hole eating a big chunk of matter for example) it would be more problematic.
[1] Cosmic rays are distinguished from "regular" solar flux by the energy of their particles (not all of them have higher energy, but the ones that do are much more likely to be of cosmic rather than solar origin). A simple cosmic ray detector can be constructed by shielding a cloud chamber such that shielding blocks any events below your desired cut off level. Detection events are then most likely cosmic rays.
Extremely high-energy cosmic rays are sometimes detected, however. The so-called “Oh-My-God particle“, detected in 1991, apparently carried as much energy as a baseball moving at 58 mph [2].
> Can this be used as a source of randomness, like for example time as a Poisson?
Yes. Although from an electronics point of view, lower energy photons are _much_ easier to deal with. That's how random.org started -- https://www.random.org/history/
Subsequent investigations believe that a cosmic ray induced bit flip is the most likely explanation.
My son was most likely hit by a cosmic ray in the early stages of development in the womb. It will have effected only a small number of cells (maybe even a single one) but the error replicated and caused a malformation in his heart.
https://en.wikipedia.org/wiki/Cosmic_ray#Effect_on_electroni...
https://en.wikipedia.org/wiki/Health_threat_from_cosmic_rays...