I can't seem to find anything like it, neither as a Fuse fs. Is there really no file system, that just evicts the oldest entries once it's full? I mean, it could be a simple queue.
This would have been the ideal tool for cache; no bothering with size limits and background daemons.
Sounds like you want to automate Trash folder management. Why would number 1 require complex rules if number 3 can work with only the rule of delete the oldest file? Wouldn't the same rule work in both cases?