The cost of the flight & hotel is pretty trivial in the grand scheme of things. By that I mean relative to stuff like going to a place you want to see, the value of spending a vacation day, etc...
I do agree that the meeting smart people is fun... but the full on crash-and-burn on an algorithm + whiteboard interview would really have a massive negative impact on my confidence & self-esteem. Not worth it.