Gorse in a small garden can be complicated to manage. Too spiny and it reseeds itself. Rosmary or Leptospermum can take that job.
If that were the case you would expect to see large growths of it in the wild, right? Whilst I do see it in the wild, I've never seen any situation where it looks to be taking over. I just see individual plants occasionally.
(I live in the South of Germany, but the UK, where it was originally introduced in the 19th century, seems to also have a huge problem with it: https://www.cabi.org/invasivespecies/species/himalayan-balsa...)
https://vibrantgardenblog.wordpress.com/2016/12/02/buddleia-...
Each one of those individual plants can produce 40.000 seeds each year, so give them a decade alone and you will see. Is very invasive on river beds and disturbed soils.