Yes - genetic modification is not doing anything that couldn't happen by normal mutation/selection - but it accelerates the process by multiple orders of magnitude (that's the whole point).
Making the debate one of 'show us evidence of harm or shut up' always felt fairly short-sighted.
I always felt that there is a huge 'unintended consequences' factor lurking somewhere. The need for a deep sense of caution when working on a new frontier struck me as more rational than a stance that basically boiled down to "let's assume we haven't overlooked any potential risks".