Post should probably be titled "macOS Photos keeps a huge wallpapers cache", since it's not actually tied to the macOS wallpaper mechanism.
Yes maybe but Photos only creates a copy of the file/image in order to be used from macOS (due to Sandboxing). So it's more a macOS connected to Photos feature than Photos alone.
...in the end, it's just a mess.
That explains it.
Also, the path that I have is as follows.
~/Library/Containers/com.apple.Photos/Data
Mine's a killer 55 MB.
Correct, mine is few MB (now). Just don't set the wallpapers via Photos in the future, or remember to clean this folder.