Most mutable objects are easy because they don't support concurrent modifications. Therefore pattern matching would not make them harder to implement. Pattern matching such an object while it is being concurrently modified is simply a programmer error just like any other use of the object.
Not allowing pattern matching of mutable objects would be similar to not allowing equals and hashCode methods on such objects and for similar reasons.