The example I recall was a "ribbon" in the OS that slide out to reveal the dialog. If a dialog presented itself but the ribbon remained along the edge you could immediately tell it was spoofed. Of course this requires the OS not allow untrusted code to reposition/hide the ribbon or present a full screen display without prompting the user.
Another example is iOS grays out the background (including status bar at the top) when presenting a modal password prompt. However this could easily be spoofed by a full screen native app. The only way to solve that is to require authorization to enter full screen mode.
Browsers are improving. At least Chrome shows the URL at the top of all popup windows. Entering full screen mode requires user authorization.
That of course doesn't solve the OP's problem of spoofing a floating window purely inside a webpage, but that really needs to be solved at the OS level.