You can restrict casual directory access if you're careful, and well-informed, and diligent about the (often overly generous) permissions chosen by each Flatpak's maintainers, and those chosen by runtimes and base applications, and about changes over time.
However, files and directories are only a small fraction of what hostile code can exploit. There are also d-bus services, abstract sockets, shared memory areas, display server protocols, and device nodes, for example. In the fingerprinting and tracking department, we have things like hardware peculiarities and installed font lists. If you're concerned not only about privilege escalation and data exfiltration, but also about collection of your biometrics, then you should consider keystroke timing, mouse movement patterns, voice samples, and the like. These are just examples, not exhaustive lists.
More fundamentally, Flatpak sandboxes are made of Linux namespaces, which have a long history of vulnerabilities and exploits. They're pretty good at containing buggy software, but software that wants to circumvent them can find a way, and if its developers are sufficiently skilled, will find a way.