The wood definitely matters. Beginner kits are usually made out of poplar and even with good heads and tuning they just sound bad compared to a kit made with a wood known to sound better. Maple and birch are quite popular, oak, bubinga, and ash all make appearances on high-end drum sets. I used to work in a music store and saw a lot of drum kits come through, and high-end kits even with the same heads and tuning as low end kits do just sound better.
Also, in the rest of musical instruments, particular woods are valued for their acoustic properties especially on string instruments. So I don’t think it is that far-fetched to say that it makes a difference in drums too, unless you also want to argue that an acoustic guitar made of particle board vs one made of maple are the same. (They do not sound the same at all)