but I mostly work with infrastructure and optimizing build systems.

Besides source code, games also need to compile source assets. An example of this is textures; a dev might store them as PNGs in their source control, but the texture will need to be converted to a different format depending on the target hardware. On mobile, that might be something simple like PVRTC, or maybe supercompressed KTX. Geometry and animations also need to be compiled, usually from something like FBX to whatever runtime formats the engine uses, not to mention all the other copious amounts of custom data needed for the game.

Shaders are a challenge because you can't fully compile them ahead of time. They're usually authored in some DSL (maybe HLSL/GLSL, but sometimes also an engine-specific custom language), and depending on the target graphics API they can be compiled to intermediate bytecode (eg SPIRV). However, that bytecode needs to be compiled by the user into the internal format needed by the graphics implementation (which can even change between driver versions on the same machine), which is why modern PC games need to compile shaders on first launch. Consoles with fixed hardware can avoid this problem.

Beyond that, the shader permutation issue leads to games requiring a shitload of shaders for performance reasons, exacerbating the issues described above, both for the developer performing a release build, and for the user launching the game. This article describes the issue well.

So add all that on top of the usual time it takes to (cross-) compile the engine and game code, which is often a massive C++ codebase of dubious quality, and sometimes also an outdated proprietary toolchain of dubious quality.

Worth noting: That 4 to 6 hour number is probably for a build from scratch. A good build system will be able to only recompile things that have changed and so the develop -> compile -> test cycle is usually much faster.