Oh, you're using it for just bridging group chats.
For one to one you need to link your matrix account to your discord account using the !login command. After this you can access everything that your discord account can access. Manually managing anything except for which server you want bridged is completely unnecessary.