It's immoral to visit such a country. It has concentration camps. It tortures on a large scale. It's citizens suffer and visiting the country gives money to an evil regime.
That said, I struggle with this question. In some ways you're absolutely correct. My personal struggle: North Korea has a chain of restaurants abroad, and I walked by one quite often while I was in Bangkok. Out of (morbid) curiosity, I would have loved to dine there and talk to the (very attractive) North Korean waitresses. I couldn't stomach giving the regime the $3 for the meal though.
In some ways, going to the NK restaurants supports the quazi-slavery of the workers in the restaurant. But in other ways, that spending enables the restaurant workers to live in Thailand and not North Korea. In the end, I decided not to visit, as I couldn't help but think of how I would feel about someone dining at a Nazi restaurant in the 40s. Difficult ethical questions though.
People often boycott and don't like visiting circuses or shows that generally abuse and exploit animals (elephant shows in Thailand come to mind). I can't see why visiting NK is any different.
EDIT> Is it moral to give money to a regime that has killed millions of its own citizens? I mean, unless you're secretly giving medical supplies to the poor, your money is only benefiting the elite.