According to that recent The Information report, Orion is supposed to be just a regular LLM except trained with synthetic data generated via Strawberry. Anthropic et al. have also been working on ways to generate synthetic data (as seen in the success of Sonnet 3.5) so I don't really know if that's going to be a big lead.

And of course the ever-hyped strawberry is supposed to be some sort of tree-of-thought type thing I think, or maybe it's related to https://arxiv.org/abs/2203.14465. Either way, nothing so far has come out that it's a completely novel training technique or architecture, just a gpt-4 scale model with different post-processing.