Likewise, it's possible that it may be able to generate AI child porn without any actual child porn in its training dataset, but if it knows what children look like and what nudity looks like, then it could figure out what nude children look like.

This is true and it's what happened to OpenAI, there's a section in Karen Hao's Empire of AI about it. Remember, these companies (under) paid Kenyan workers and gave them PTSD to make sure CSAM and other undesirable content wasn't in their training set.