You may be right here. The tariffs might not directly cause enough pain for the average person to matter.
The big unknown still is the impact on the economy and job market. His actions may reduce competitiveness of American companies globally due to retaliatory tariffs and resentment.
If the tariffs cause a recession, people will punish him for it.