Shared use of servers is probably the main issue

It appears that users in this case include agents such as software. A bit confusing for the general public.

For instance, a malicious app obtained from an app store could use the Downfall attack to steal sensitive information like passwords, encryption keys, and private data such as banking details, personal emails, and messages.

—Official website

It can theoretically even be exploited via a browser:

[Q] What about web browsers?

[A] In theory, remotely exploiting this vulnerability from the web browser is possible. In practice, demonstrating successful attacks via web browsers requires additional research and engineering efforts.

—FAQ at the official website

I think it also means software running can access other software's memory which is probably bad but personally I'm not keen for that performance hit on my desktop