Your thinking is off, the GPL and derived licenses like the AGPL are viral on purpose. They apply to everybody who uses, downloads, or accesses the software (in the case of the AGPL) and they are explicit about this:
Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License.