The ballot box is a very low resolution feedback tool and an election cycle is a very long iteration cycle.
Imagine you only had performance reviews every two years and the feedback from your boss was either "you're not as good as the next best choice, so you're fired" or "you're not fired right now."
Voter willpower is diffuse (different people care about different things at different times) and only infrequently expressed. Special interest willpower can be much more acutely and frequently applied. It's hard to see how you can correct the imbalance between these two things given the current system and the inertia of incumbents who benefit from it.
Term-defined performance goals. Before elections begin, each citizen can define a list of priorities, metrics, and multipliers of expected performance. These are used in a function to calculate a number that determines whether an elected official is performing according to the standards of the citizens.
These "expectations" are collected, anonymized, and split up into groups of similar expectations. A candidate for office can select to align themselves to a specific group of expectations, or define their own and run on that as a platform. Either way they're going to have to live up to that set of expectations. Citizens vote for the candidate they feel best represents those expectations. At this point we're just "hoping" they come through on their promises; there's no guarantees.
Once a candidate wins and begins their term, their performance is continually reviewed by the function. If a candidate's performance during their term drops below 50%, a performance review can be ordered, and more specific short term goals can be drawn up to see if they can rise to meet them. If their performance drops below 25%, a suggestion is automatically flagged to inform citizens they should vote this rep out at the next election cycle.
The end results should be that citizens can more effectively inform representatives of what they want, they can track the progress of these representatives, and stay informed so that at the next election cycle they aren't just re-voting for the incumbent even if the incumbent didn't live up to expectations. In addition, a citizen no longer needs to vote according to a party; they simply define their personal expectations for government, and "parties" may naturally form according to real issues, rather than rhetoric, tradition, etc.