nothing other than doing it maliciously

Malicious compliance generally doesn't involve technicalities, just following the rules. The maliciousness comes from knowing that following the rules, exactly as they are written down, will cause problems and disruption, and then doing it anyway, allowing the rules to override things like common sense.

As I understand it, it's generally any form of compliance. Typically, it's abusing technicalities, but what makes it "malicious" is that it subverts the intention behind the rule or order.

This law was designed to harass women. The transphobia is the vehicle used to deliver that intent, but for all its bigotry, it clearly isn't the ultimate point. All the discourse is about "protecting (cis-)women", about transfem athletes, and so on. Frankly, I'm not sure the type of politician to push this type of politics even remembers that transmascs exist. Even if they do, it's not like they could do anything about it that wouldn't risk collateral damage among cis-men.

Imagine how it would to if you tried to apply the "bathroom genital check" to the gender that invented etiquette about not standing too close to others when pissing. The geriatiric shrimpbearers in power would have a meltdown over the idea of someone manhandling their dick to make sure it's real.

(Nothing wrong with a small dick, just with making your insecurity about it someone else's problem.)

Now, unlike the transphobia, they can't make the misogyny explicit in the law (or at least they aren't sure they'd get away with it yet, so why risk it?). The law as written technically also applies to transmascs, even though they weren't the target and it realistically would never be enforced against them.

Thus, the malicious compliance is that the transmascs carry out the letter of the law rather than its spirit, fully expecting the predictable result and provoking the authorities into tipping their hand.

they are going out of their way to create situations where they follow the law and cause disruption. that makes it malicious.

the intent of the rule is irrelevant.

that is one method of performing it, yes. that is not the only method.