Regardless of your level, just remember to set your pacing expectations. It's perfectly okay if you are going slow, it's really information-dense.

I have no background in abstract math. It took me three days to make it through the first paragraph. I'd still recommend it.

I think the prerequisite for "Category Theory in Context" is a grasp of basic category theory ((co-)limits, Yoneda lemma, adjoint functor theorems, etc). You should also have some interest in an abstract maths field that requires all these category theory machinery. So a graduate student in abstract maths sounds about right.

I went through it with some background abstract algebra and functional programming