Ie we will be able to describe how a function of `add(1,2)` is most clearly definitely not intelligent. We will understand the internal makeup of these neurons better, the whole system will be deterministic and understood - just a very large `add(1,2)` .. so it's definitely not intelligent, of course.
Yet it will still produce output that is fully and truly turing-proof. If you let it run, it retains, stores, "behaves", etc. It's clearly just `add(1,2)` in a loop after all.. we will understand it so well that it's obviously just a trick of our mind that it seems real.
The conclusion in that space will not be one of the machine .. but of us. If we can't tell it apart from us.. and if we know it is most definitely not alive or even mysterious.. what does that say about us? The existential crisis will be neat.