Nothing is wrong with byte buffers. Use them where appropriate. The advice is to stop when you find yourself implementing a full-blown memory manager / quasi-malloc in user code on top of byte buffers...