That's true if you're passing messages between identical models. There's a question to ask as to whether different models trained for different tasks would be better than single, multipurpose models though. My gut feel is that eventually multipurpose models will win because you don't have the embedded cost of relearning what syntactic structure is, but for a given training time and number of weights it's not clear whether that's true today.