If consciousness is just matter, why isn't yours accessible to me. There is something deeper at play.

I have a hypothesis that a consciousness is a function of a universe. Each universe has just one consciousness.

Another person's consciousness is inaccessible because it's in a different universe. Only the matter part of that is projected in your universe, not the actual identity.

Without a hypothesis like this, we are left with postulating hidden variables: some hidden context pointer which resolves to you or me or such.