995
Results of the "Can you tell which images are AI generated?" survey (i.vgy.me)
submitted 3 years ago* (last edited 3 years ago) by to c/technology@lemmy.world
161 comments fedilink hide all child comments
 

Previous posts: https://programming.dev/post/3974121 and https://programming.dev/post/3974080

Original survey link: https://forms.gle/7Bu3Tyi5fufmY8Vc8

Thanks for all the answers, here are the results for the survey in case you were wondering how you did!

Edit: People working in CS or a related field have a 9.59 avg score while the people that aren’t have a 9.61 avg.

People that have used AI image generators before got a 9.70 avg, while people that haven’t have a 9.39 avg score.

Edit 2: The data has slightly changed! Over 1,000 people have submitted results since posting this image, check the dataset to see live results. Be aware that many people saw the image and comments before submitting, so they've gotten spoiled on some results, which may be leading to a higher average recently: https://docs.google.com/spreadsheets/d/1MkuZG2MiGj-77PGkuCAM3Btb1_Lb4TFEx8tTZKiOoYI

you are viewing a single comment's thread
view the rest of the comments
[–] 154 points 3 years ago (15 children)

So if the average is roughly 10/20, that's about the same as responding randomly each time, does that mean humans are completely unable to distinguish AI images?

  • permalink
  • fedilink
  • source
  • hideshow 15 child comments
    •
  • [–] 89 points 3 years ago (5 children)

    In theory, yes. In practice, not necessarily.

    I found that the images were not very representative of typical AI art styles I've seen in the wild. So not only would that render preexisting learned queues incorrect, it could actually turn them into obstacles to guessing correctly pushing the score down lower than random guessing (especially if the images in this test are not randomly chosen, but are instead actively chosen to dissimulate typical AI images).

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 42 points 3 years ago (1 child)

    I would also think it depends on what kinds of art you are familiar with. If you don’t know what normal pencil art looks like, how are ya supposed to recognize the AI version.

    As an example, when I’m browsing certain, ah, nsfw art, I can recognize the AI ones no issue.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 23 points 3 years ago (2 children)

    Maybe you didn’t recognize the AI images in the wild and assumed they were human made. It’s a survival bias; the bad AI pictures are easy to figure out, but we might be surrounded by them and would not even know.

    Same as green screens in movies. It’s so prevalent we don’t see them, but we like to complain a lot about bad green screens. Every time you see a busy street there’s a 90+ % chance it’s a green screen. People just don’t recognize those.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 12 points 3 years ago (1 child)

    Isn't that called the toupee fallacy?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 9 points 3 years ago

    It is!!

    Someone might assert that, “All toupees look fake. I've never seen a good toupee.” This is an example of neglecting the base rate because if I had seen good toupees, I wouldn't know it.

    Thanks, I love learning names for these things when they come up!

  • permalink
  • fedilink
  • source
  • parent
  • [–] 21 points 3 years ago (3 children)

    If you look at the ratios of each picture, you’ll notice that there are roughly two categories: hard and easy pictures. Based on information like this, OP could fine tune a more comprehensive questionnaire to include some photos that are clearly in between. I think it would be interesting to use this data to figure out what could make a picture easy or hard to identify correctly.

    My guess is that a picture is easy if it has fingers or logical structures such as text, railways, buildings etc. while illustrations and drawings could be harder to identify correctly. Also, some natural structures such as coral, leaves and rocks could be difficult to identify correctly. When an AI makes mistakes in those areas, humans won’t notice them very easily.

    The number of easy and hard pictures was roughly equal, which brings the mean and median values close to 10/20. If you want to bring that value up or down, just change the number of hard to identify pictures.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 4 points 3 years ago (2 children)

    The number of easy and hard pictures was roughly equal, which brings the mean and median values close to 10/20.

    This is true if "hard" means "it's trying to get you to make the wrong answer" as opposed to "it's so hard to tell, so I'm just going to guess."

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 3 years ago (1 child)

    That’s a very important distinction. Hard wasn’t the clearest word for that use. I guess I should have called it something else such as deceptive or misleading. The idea is that some pictures got a below 50% ratio, which means that people were really bad at categorizing them correctly.

    There were surprisingly few pictures that were close to 50%. Maybe it’s difficult to find pictures that make everyone guess randomly. There are always a few people who know what they’re doing because they generate pictures like this on a weekly basis. The answers will push that ratio higher.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 3 years ago*

    A great example of the below 50% situation is the picture of the avocado and the tomato. I was confident that that was AI generated because I was pretty sure I'd seen that specific picture used as an example of how good Dall-E 3 was at normal text. However, most people who had used other models were probably used to butchered text and expected that one to be real.

    If they did this quiz again with only pictures that were sketches, I bet the standard deviation would be much smaller.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 20 points 3 years ago (2 children)

    It depends on if these were hand picked as the most convincing. If they were, this can’t be used a representative sample.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 11 points 3 years ago (1 child)

    But you will always hand pick generated images. It's not like you hit the generate button once and call it a day, you hit it dozens of times tweaking it until you get what you want. This is a perfectly representative sample.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •