You are misunderstanding.

The file cache is never written out to the swapfile, because files are already on disk, like you say. The file cache is kept in memory and the kernel may decide it's more advantageous to swap out unused anonymous memory pages to disk than flushing a file from the cache. You can use the vm.swappiness parameter to finetune this behavior to your liking btw. Lower values favor keeping more anonymous memory pages in memory, higher values favor file backed pages.

To give an extreme example of where this is useful: I have a use case where I process a number of large video files (each 2GiB-10Gib in size). The job I'm doing involves doing several passes over the same file sequentially. You can bet your ass that caching them in memory speeds things up dramatically: the first pass, where it has to read the file on disk is at 200x speed (relative to the video's duration), the second pass at 15000x speed.

Even in less extreme circumstances it also helps by keeping frequently accessed files in your home directory in memory, for example your browser profile. Your browser and desktop environment would be much more sluggish if it had to reach out to disk every time for every file it touched.

And you are free to disagree, but it's not my opinion but the opinion of kernel developers of just about every operating system built the past 4 decades. So I'd say: take up the argument with them and see how far you get.