> It seems to be able to only be able to control the bottom 12-bits of the address to recover (but I didn't fully get why)
2^12 = 4096, or the x86 page size in bytes.
2^12 = 4096, or the x86 page size in bytes.
(As an aside, this also limits the size of the L1 cache, which is why it hasn't grown much despite the L2 and L3 cache growing a lot; an 8-way set-associative VIPT cache with a 4KiB page size is limited to 32KiB, absent tricks like page coloring. Perhaps this will change if 64-bit ARM servers become popular, since they can only address the largest amount of memory when using a 64KiB page size, and this would make enterprise distributions default to that page size.)
A huge L1 would be slower to access. You might end up slowing down all memory accesses by a cycle (or more) just for accessing the large cache. You also have to find some place to put the thing in your floorplan, and route everything else around it. This may result in timing challenges in other parts of the chip, which might require additional delay to resolve.
A huge L1 would take up more space. This would increase area, reduce yield, and increase cost. Since in a multi-core chip each core has its own L1, you will have to pay this cost multiple times. Also, L2 caches are typically inclusive, so you would potentially need a much larger L2 to be able to accommodate all the extra information in these L1s.
These tradeoffs have to be studied with simulated experiments to make the right call. For programs with huge working set, maybe the added latency pays for itself because you have have fewer cache misses. For programs with good locality, maybe you end up losing performance. Maybe you save power by reducing misses, or maybe you waste more power because the cache is using more power. Maybe it's an insignificant area increase, or maybe it completely blows up your budget.
> To prevent the confusing situation where the same physical address has more than one index in the cache
Ah yes, of course. Is this something that actually happens (e.g. because of processes sharing memory, OS hands out different virtual addresses pointing to the same physical memory) or theoretical in nature?
For example, a common trick to implement an efficient circular list for objects of varying size is to map the same physical addresses twice in adjacent locations. Then you can just use a pointer to the start of each object and have no special handling for the case where an object straddles the end of the allocated area.