Copper, scrolling, sprites etc., sure won't actually touch the frame buffer. What you're forgetting is the blitter, which indeed is a way for the Amiga to redraw the whole framebuffer faster than its CPU alone could handle, AFAIK well within the time span of a frame. Add game logic and other I/O to that and the difference could very well cross the line between not enough CPU and enough CPU.