Yes, I think if we can get an LLM to work while providing high quality, real world sources it will be a game changing technology across domains. As it stands though, it's like believing a magician really does magic. The tricks they employ are incredibly useful in a magic show, but if you expect them to really cast a fireball in your defense, you'll be sorely mistaken.

Did you read mine? If you wanted a depiction of a city, it's more than good enough. In fact it's amazing what it can do in that respect. My point is: it gets major details wrong in a way that feels right. That's where the danger lies.

If your GPS consistently brought you to the wrong place, but you thought it was the right place, do you not think that might be a problem? No matter how many people found it useful, it could be dangerously wrong in some cases.

My worry is precisely because people find it so useful to "look things up", paired with the fact that it has a tendency to wildly construct 'information' that feels true. It's a real, serious problem that people need to understand when using it like that.