CO2 goes down due to chemical weathering - basically it binds to the rock and is then washed into the sea, where it is subducted into the mantle due to plate tectonics. It comes back up from volcanoes - the deep carbon cycle.

The amount of weathering depends on the access the atmosphere has to the rock. If there is a thick ice layer, the rate goes down. Water plays a role as well, and the fastest weathering happens in the tropics and not the deserts (where you have mechanical weathering which does not bind CO2). Accordingly, the configuration of continents can play a role as well, which is interesting if you go back far enough. On the other hand, the volcano emissions appear somewhat constant over time.

In total though, there are still many open questions. Why does CO2 appear to go up so fast after an ice age? Does CO2 or temperature go up first? In the records, the temperature increase appears rather sudden, on the timescale of decades. How does this happen? (Note though that that has no huge importance for our current problem since the situation is different. But the reaction speed of the system is relevant, for example.)

Note: If we wait long enough (~tens of thousands of years) the CO2 level should go down again from weathering. There are proposals for carbon capture that uses this mechanism (by grounding the rocks and distributing them openly so the atmosphere can reach a lot of the rock surface, thus increasing the weathering rate massively).

Note2: You need silicate rock, which is not distributed evenly.