They require a higher angle of attack on take-off and landing, and, with passengers sitting much further from the centerline of the aircraft than on a conventional design, passengers are subject to much more g-forces during roll maneuvers.
The second issue can be mitigated by avoiding rolling the aircraft and/or by performing less aggressive rolls, but the first issue is something passengers will just have to deal with if BWB passenger aircraft become a thing.