I don’t think that change would negatively impact the experiment, but for what they were trying to measure I’m unsure of how necessary that step would be.
They weren’t measuring how smart or good at math the participants were, they were specifically measuring changes in focus and persistence (so basically the amount of effort put forward). The findings show that people tend to just give up when you take the LLM away, and I think that with your proposed change you’d still get largely the same findings.
I would agree that LLMs aren’t the entire issue, but they are making the issue far worse at an exponential rate.
I’d be interested to read a study about what factors and/or traits predispose someone to trust an LLM, or allow themselves to become dependent on it.