At least one drawback I can see is that apparently the user has to manually opt-out some macros (the ones that aren't pure) of the caching. Which would require everyone using the rust compiler to know which macros of which crates are pure and which ones not. I guess that the ones writing the macros could do that, but then you rely on library maintainers to know about this specific optimization.

It really should be opt-in so the maintainers of libraries that know about the optimization and know that their macro is pure can annotate them as so.