No, no. It's an isekai for sure, it just isn't one that revolves around the protagonist trying to get home. The "getting home" plot line is the thing I haven't seen much and could only think of those two examples off the top of my head.
Plenty of isekai where the main character doesn't care to go home or can't go home (usually because they died and the other world is some kind of reincarnation or afterlife). Such as with Inuyasha, Futurama, and nearly all those animes with the super long titles about being overpowered in another world.
Heck, the ending of the orginal Fullmetal Alchemist turned it into an isekai when Edward ends up in the real world's WW2 era Germany.