That was with a 6S!
Where the iPhone shines is getting nicely automatically post-processed pictures. Olympus has an iAuto mode, which also aims for that. There are also a lot of in-camera settings where you can tweak the produced jpegs, if you don't want to post-process. The best quality you would get though by post-processing raw images on your computer.
Before I bought my E-M5.2 I had an E-PL2, looking at some of the pictures I took with the kit lens they are generally still better than the pictures I've taken on my phone.
Of course the camera gives a lot more creative freedom, but it felt pointless to carry a body + lenses for little improvement in quality, light as they were. I switched to a Fuji XT-2 and am much, much happier.