They don't need the entire project to fit in their token windows. There are ways to make them work effectively in large projects. It takes some learning and effort, but I see it regularly in multiple large, complex monorepos.

I still feel somewhat new-ish to using LLMs for code (I was kinda forced to start learning), but when I first jumped into a big codebase with AI configs/docs from people who have been using LLMs for a while, I was kinda shocked. The LLM worked far better than I had ever experienced.

It actually takes a bit of skill to set up a decent workflow/configuration for these things. If you just jump into a big repo that doesn't have configs/docs/optimizations for LLMs, or you haven't figured out a decent workflow, then they'll be underwhelming and significantly less productive.

(I know I'll get downvoted just for describing my experience and observations here, but I don't care. I miss the pre-LLM days very much, but they're gone, whether we like it or not.)

So far, there is serious cognitive step needed that LLM just can't do to get productive. They can output code but they don't understand what's going on. They don't grasp architecture. Large projects don't fit on their token window.

There's a remarkably effective solution for this, that helps both humans and models alike - write documentation.

It's actually kind of funny how the LLM wave has sparked a renaissance of high-quality documentation. Who would have thought?