Reportedly, the app uses an on-device machine learning model to classify the content.
Using your shiny new phone to train AI...
For now its only supposed to check Google Messages, so not if you use a different text app.
But one vulnerability already is side loaded versions with super high version numbers so they won't be auto updated. Those will scan everything and do whatever with it. Considering the apps point is to scan for nudes, likely sending all your nudes to the hacker.