I think that there are two pieces to it. There's tradition, of course, but I don't think that that's a motive. Also, some folks will argue that not taking hands off the keyboard, not going to a mouse, is an advantage; I'm genuinely not sure about that. Finally, I happen to have decent touch typing; this test tells me 87 WPM @ 96% accuracy.

First, I don't spend that much time at the text editor. Most of my time is either at a whiteboard, synchronizing designs and communicating with coworkers, or reading docs. I'd estimate that maybe 10-20% of my time is editing text. Moreover, when I'm writing docs or prose, I don't need IDE features at all; at those times, I enable vim's spell check and punch the keys, and I'd like my text editor to not get in the way. In general, I think of programming as Naur's theory-building process, and I value my understanding of the system (or my user's understanding, etc.) over any computer-rendered view of the system.

Second, when I am editing text, I have a planned series of changes that I want to make. Both Emacs and vim descend from lineages of editors (TECO and ed respectively) which are built out of primitive operations on text buffers. Both editors allow macro-instructions, today called macros, which are programmable sequences of primitive operations. In vim, actions like reflowing a paragraph (gqap) or deleting everything up to the next semicolon and switching to insert mode (ct;) are actually sentences of a vim grammar which has its own verbs and nouns.

As a concrete example, I'm currently hacking Linux kernel because I have some old patches that I am forward-porting. From the outside, my workflow looks like staring out the window for several minutes, opening vim and editing less than one line over the course of about twenty seconds, and restarting a kernel build. From the inside, I read the error message from the previous kernel build, jump to the indicated line in vim with g, and edit it to not have an error. Most of my time is spent ~~legitimately slacking~~ multitasking. This is how we bring up hardware for the initial boot and driver development too.

Third! This isn't universal for Linux hackers. I make programming languages. Right now, I'm working with a Smalltalk-like syntax which compiles to execline. There's no IDE for execline and Smalltalks famously invented self-hosted IDEs, so there's no existing IDE which magically can assist me; I'd have to create my own IDE. With vim, I can easily reuse existing execline and Smalltalk syntax highlighting, which is all I really want for code legibility. This lets me put most of my time where it should go: thinking about possibilities and what could be done next.

Consider the shell itself to be "the IDE"

Everything is interoperable, extensible, scriptable, and more. CLI tools are designed to run fast/instant, have keyboard shortcuts for everything, and be deeply customizable. The openness and variety cannot be overstated, Google "CLI file explorer" and you'll easily find at least 10. Nobody has the same exact setup because it gets molded to match how your brain works. Go for popular tools, niche setups, or both.

Graphical IDEs could also run fast/instant and have keyboard shortcuts for everything, but their users don't demand it. If you wished the file explorer/git integration/debugger/etc worked a bit differently, there might be a plugin, if you're lucky. Many operations can only be invoked manually via sequence of dialog boxes or mouse clicks or both.