It often is, but sometimes those rights are only imagined rights. Does a right extremist have a right to have his race free from other races?
You say yourself that rights are sometimes at conflict with each other. That requires that sometimes rights are restricted to a point that other rights are not infringed. For instance we lock up dangerous people. Why do you think is freedom of speech is some kind of super-right that triumphs over every other right? Is freedom of speech more important even when it’s used to incite genocide?