Bitcoin miners affect the difficulty level directly, by mining or not.
Furthermore, there are a finite number of bitcoins to be mined. The higher the difficulty, the faster they are being mined. Therefore, the scarcity of each individual coin is going UP as they are mined. Let me say that another way: before a coin is mined, it is still available for anyone to get by mining. After it is mined, it is in a private wallet number, worth whatever the market says the owner should reasonably receive for it. As coins are mined, the supply of total available coins goes DOWN. As long as demand continues, price will go UP. BTC become more scarce with each coin mined, because there is one fewer to mine in the future, and every one to be mined after it will need to be done so at the new (assumedly higher) mining cost.
The difficulty level might be thought of as a direct index of the COST of mining a new coin. As that cost increases, supply for coins goes DOWN. Ceterus paribus, supply DECREASES, price INCREASES.