Too much can be made of the idea of sharing state via channels. If it's simpler to express a piece of code using a mutex, use a mutex. A thread-safe set is one such situation. Don't feel bad about doing so; there is nothing non-idiomatic about using locks in Go.
When using channels, the choice between buffered and unbuffered depends on the situation. There are cases where a buffered channel is required to avoid deadlocks. For example, consider the case where you start N goroutines and use a channel to collect the results. If you return before collecting all results, the remaining goroutines will block forever trying to write to an unbuffered channel. Using a buffered channel with size N avoids this possibility.