97
‘It’s terrifying’: WhatsApp AI helper mistakenly shares user’s number (www.theguardian.com)
submitted 1 year ago by to c/privacy@lemmy.dbzer0.com
12 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] -5 points 1 year ago (10 children)

An LLM cannot generate random numbers. It has to pull from a list of numbers its model was built to include.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 10 child comments
    •
  • [–] 9 points 1 year ago* (6 children)

    I mean the latter statement is not true at all. I'm not sure why you think this. A basic GPT model reads a sequence of tokens and predicts the next one. Any sequence of tokens is possible, and each digit 0-9 is likely its own token, as is the case in the GPT2 tokenizer.

    An LLM can't generate random numbers in the sense of a proper PRNG simulating draws from a uniform distribution, the output will probably have some kind of statistical bias. But it doesn't have to produce sequences contained in the training data.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 1 point 1 year ago (5 children)

    It could have the whole phone number as a string

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 2 points 1 year ago (4 children)

    A full phone number could be in the tokenizer vocabulary, but any given one probably isn't in there

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 1 year ago (3 children)

    No way, tokens are almost always sub-word length.

    Using larger tokens means that you need way more tokens to represent data and so encoders always learn to use short tokens unless they're specifically forced not to.

    Just to put it in perspective. Imagine that you were trying to come up with a system for writing down every phone number. The easiest system would be to have a vocabulary of 10 items (digits), with such a vocabulary you can write down all phone numbers. While storing entire phone numbers as a single 'word' would require a vocabulary of 10 billion items in order to write down all phone numbers.

    That's why encoders learn to use the smallest token sizes possible.

    LLMs can't generate random numbers, but the process of selecting the next token involves selecting a random (using a pseudorandom number generator) next token from the distribution of possible next tokens. The 'Temperature' setting alters how closely the random selection is to the distribution in the vector describing the next token.

    An extreme example would be, on one end of the Temperature scale it always chooses the highest probability next token (essentially what the person you're responding to is thinking happens) and on the other end of the scale it completely ignores the distribution and chooses a completely random token. The middle range is basically 'how much do I want the distribution to affect my choice?'

    In the end, the choice of the next token is really random. What's happening is that the LLM is predicting the distribution among all possible tokens so that the sentence fits into its model of how language works.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 1 point 1 year ago (2 children)

    I know, I'm just saying it's not theoretically impossible to have a phone number as a token. It's just probably not what happened here.

    the choice of the next token is really random

    It's not random in the sense of a uniform distribution which is what is implied by "generate a random [phone] number".

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 1 year ago* (1 child)

    Unless the person is use math terms elsewhere, I always assume people mean 'unexpected' then they say random.

    It’s not random in the sense of a uniform distribution which is what is implied by “generate a random [phone] number”.

    Yeah, true.

    There, I was speaking more to the top level comment's statement that an LLM cannot generate random numbers. Random numbers are pretty core to how chatbots work... which is what I assumed they meant instead of the literal language model.

    You could say that they're technically correct in that the actual model only produces a deterministic output vector for any given input. Randomness is added in the implementation of the chatbot software through the design choice of having the software treat the language model's softmax'd output as a distribution from which it randomly chooses the next token.

    But, I'm assuming that the person isn't actually making that kind of distinction because of the second sentence that they wrote.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 1 year ago

    The point of my second statement is that if you made an AI that stores and retrieves phone numbers that the model could reasonable use phone number chunks in its random number generation. A phone number can normally be broken into 3 to 6 chunks of 1 to 5 numbers which is reasonable sizes to tokenize. If you then asked it for a random number I think it is reasonable that it would be as likely if not more likely to use the data from the phone number list as it would to use the core 0 to 9 tokenized number list unless you specifically tried to split the two.

    This is a WhatsApp AI so I think asking it for Tim's number is a use case they trained on. It needs to be a phone book. My guess is they said that list A is a list of public numbers for training things like what a phone number looks like, and list B is a list of private user numbers. Now while a random number could be a random string of numbers it could also be that the LLM is too likely to pull a combination that is actually a real number.

    So is this a case where it randomly pulled together 11 digits that magically hit the roughly 1 in in 100 chance that a random string of numbers shaped like a UK phone number would be a number of a user. Is it a case where it pulled from a public combo list of 4 tokens and randomly reformed a real number that was both public and private? That seems more likely to me. We probably won't ever get to know.

    If I was making this AI chat bot I would have it check against the most critical data I have for privacy before it shared it as a random number though. WhatsApp phone numbers are its users IDs. Even if it truly randomly generates one it should verify that it is a private number and not output it as it showed it could do when questioned where the number came from.

  • permalink
  • fedilink
  • source
  • parent