RAM is the big bottle neck here.
Here's the comparison:
PS5: 16 GB/256-bit GDDR6 SDRAM
512 MB DDR4 RAM
Series X: 10 GB/320-bit & 6 GB/192-bit (16 GB total)
Series S: 8 GB/128-bit & 2 GB/32-bit (10 GB total)
But more than that, it's the speeds involved:
PS5 Peak Bandwidth 448 GB/s
Xbox Series X 10 GB 560 GB/s and 6 GB 336 GB/s
Xbox Series S 8 GB 224 GB/s and 2 GB with 56 GB/s
So not just less RAM, but at 1/2 the speed.