You can have a system that just gives you a relevance-sorted list, or a system that then ingests the chunks of those documents that are most relevant and gives a human readable summary in addition to the list.

The indexing step requires a lot of processing, by which you chunk reference documents into couple hundred word chunks and cook each of those chunks into a high dimensional embedding vector. Once you have done that and saved the database of vectors, you can put in your query, embed it, check its dot product against all the other embedding vectors, and get your ranked list (potentially quite fast) without need to produce text out the other end.

Making those embedding vectors works much faster on a GPU rather than a CPU, but can be done on either. Since I have one system set up to ingest a bunch of web documents and embed them and search them, I have that system already set up using the GPU to go faster, so there's no real disadvantage to having it also set up to spit out a small natural language summary using a shitty tiny local LLM to go along with the list. When running it on local files, you can totally ingest the local document stash slowly on even just a CPU and just store that and run the semantic search on CPU, no LLM required.