Stephen Wolfram's article on how ChatGPT works was enlightening: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/

Like you said, it's just text prediction, using online content as the training ground.