I generally prefer free cash flow yield (fcf / dollar invested), although both PE and FCF can be fucked with depending on how certain expenses are classified. That being said, if I had to choose one of the two, I would go with FCF yield every time.
P/E = stock price / (earnings / # of shares oustanding)
And it actually does matter because a company can steadily grow EPS while not growing revenue meaning that something might be amiss, but it can easily be overlooked because EPS has been growing. Also, there are plenty of reasons not to want buybacks. Many times companies will buy their stock when it's at highs, thus destroying value for shareholders.