I've heard that phrase my whole life and only looked it up now because of your comment. For anyone else that's curious:
"Let them eat cake" is the traditional translation of the French phrase "Qu'ils mangent de la brioche",[1] said to have been spoken in the 18th century by "a great princess" upon being told that the peasants had no bread.
...
The quote is taken to reflect either the princess's frivolous disregard for the starving peasants or her poor understanding of their plight.