Ok this is the closest thing i have found but i haven't tried setting it up yet: https://forums.libretro.com/t/official-release-thread-for-windowcast-core/40464

From an user "the only way to run non-retroarch programs within retroarch is via the Windowcast-Core, but all that really does is allow you to use shaders and overlays. it also adds some latency, so I wouldn't really recommend it anyway."