OK, so there's something called "context", which in this case is the person being questioned being Irish.
All Europeans eats potatoes so singling out the Irish in this way means you are referencing the famine.

the joke was in reference to the irish potato famine which was caused by the uk subsidising the export of food in ireland so potatoes were literally the only thing they were able to eat because the british would force them to hand over everything else

is every use of the word “yellow” racist? no but context matters