To me sure there is things you think about but generally memory performance is less of a mental burden.
Also, swift 4 is supposed to support a rust like memory model. Not sure if it's opt in or how it will work but my impression is that it's memory model is about to support some new use cases to make it appropriate for more memory critical tasks.