To copy things into GPU memory from system memory, the CPU needs an address to copy it to. There are some ways around the 4GB limit with e.g. PAE, but the limitation that a single application can only use 32 bits of memory addresses applies regardless. With CUDA there's definitely ways to fill the memory of the GPU with something - you might even be able to make limited use of it - but it's not all addressable at once from any general context, and it won't be useful for gaming purposes in any way I'm aware of.

I guess it's also worth a mention that some Pentium 4 systems didn't have PCIe - just the prior PCI, so there might need to be some kind of adapter thing in the picture for this to work out at all, and I don't know whether those do some kind of DMA, impersonation of the older standards, or something else entirely.

It probably just isn't a good idea to connect a modern GPU to a 25 year old processor.