This has been done many times before. Composers who explicitly based their rhythms on the Fibonacci sequence include Per Nørgård and Sofia Gubaidulina. (While Bartók is sometimes said to have done so, evidence is lacking.) Brian Ferneyhough makes use of prime numbers in one scene from his opera Shadowtime.
In the popular-music world, BT wrote a Fibonacci-sequence tune that was a club hit in 1999.