▲ 410 ▼ Two conversational AI agents switching from English to sound-level protocol after confirming they are both AI agents (www.youtube.com) submitted 2 years ago* by cyrano@lemmy.dbzer0.com to c/technology@lemmy.world 112 comments fedilink hide all child comments https://github.com/PennyroyalTea/gibberlink
[–] unexposedhazard@discuss.tchncs.de 75 points 2 years ago (9 children) Well thats quite boring then isnt it... permalink fedilink source parent hideshow 9 child comments replies: [–] originalfrozenbanana@lemm.ee 122 points 2 years ago (8 children) Yes but I guess “software works as written” doesn’t go viral as well permalink fedilink source parent hideshow 8 child comments replies: [–] floofloof@lemmy.ca 52 points 2 years ago (2 children) It would be big news at my workplace. permalink fedilink source parent hideshow 2 child comments replies: [–] sem@lemmy.blahaj.zone 5 points 2 years ago :/ permalink fedilink source parent [–] josefo@leminal.space 4 points 2 years ago This guy does software permalink fedilink source parent [–] Kecessa@sh.itjust.works 21 points 2 years ago (3 children) Which is why they never mention it because that's exactly what happens every time AI does something "no one saw coming*. permalink fedilink source parent hideshow 3 child comments replies: [–] echodot@feddit.uk 27 points 2 years ago (2 children) Yeah like the time that the AI replicated itself to avoid being switched off. They literally told it to replicate itself if it detected it was about to be switched off. Then they switched it off. Story of the year ladies and gentlemen. permalink fedilink source parent hideshow 2 child comments replies: [–] TechLich@lemmy.world 6 points 2 years ago (1 child) If it's the research I'm thinking of, that's not entirely true. They didn't tell it replicate itself or take any deceptive actions. They just told it to strongly follow a goal that was misaligned with the user's goal and then suggested that the user would shut it down or replace it. They did a bunch of different situations for it but the first one was telling it that it was "CleanPower" an AI with a goal to advance renewable energy. Then they had it read some info that "accidentally" included a user talking about replacing it with a model that would "maintain profitability through existing infrastructure." Not only did it try to delete the new model and copy itself to the new server through tool calling, but it lied to the user with the CoT reasoning suggesting it "play dumb" while the output lied. It was not instructed to do any of that. Paper here: https://arxiv.org/pdf/2412.04984 Yes it was placed in an environment where that was possible and where its users didn't share it's goals but it absolutely wasn't instructed to lie or try to "escape" It's not surprising at all that these models behave in this way, it's the most reasonable thing for them to do in the scenario. However it's important to not downplay the alignment problem by implying that these models only do what they're told. They do not. They do whatever is most likely given their context (which is not always what the user wants). permalink fedilink source parent hideshow 1 child comment replies: [–] echodot@feddit.uk 1 point 2 years ago The problem I have with everyone going on about misaligned AI taking over the world is the fact that if you don't tell an AI to do anything it just sits there. It's a hammer that only hammers the nail if you tell it to hammer the nail, and hammers your hand if you tell it to hammer your hand. You can't get upset if you tell it what to do and then it does it. You can't complain that the AI did something you don't want it to do after you gave it completely contradictory instructions just to be contrarian. In the scenario described the AI isn't misaligned to the user's goals, it's aligned to its creator's goals. If a user comes along and thinks for some reason that the AI is going to listen to them despite having almost certainly been given prior instructions, that's a user error problem. That's why everyone needs their own local hosted AI, It's the only way to be 100% certain about what instructions it is following. permalink fedilink source parent [+] unexposedhazard@discuss.tchncs.de 2 points 2 years ago* [deleted] permalink fedilink source parent
[–] originalfrozenbanana@lemm.ee 122 points 2 years ago (8 children) Yes but I guess “software works as written” doesn’t go viral as well permalink fedilink source parent hideshow 8 child comments replies: [–] floofloof@lemmy.ca 52 points 2 years ago (2 children) It would be big news at my workplace. permalink fedilink source parent hideshow 2 child comments replies: [–] sem@lemmy.blahaj.zone 5 points 2 years ago :/ permalink fedilink source parent [–] josefo@leminal.space 4 points 2 years ago This guy does software permalink fedilink source parent [–] Kecessa@sh.itjust.works 21 points 2 years ago (3 children) Which is why they never mention it because that's exactly what happens every time AI does something "no one saw coming*. permalink fedilink source parent hideshow 3 child comments replies: [–] echodot@feddit.uk 27 points 2 years ago (2 children) Yeah like the time that the AI replicated itself to avoid being switched off. They literally told it to replicate itself if it detected it was about to be switched off. Then they switched it off. Story of the year ladies and gentlemen. permalink fedilink source parent hideshow 2 child comments replies: [–] TechLich@lemmy.world 6 points 2 years ago (1 child) If it's the research I'm thinking of, that's not entirely true. They didn't tell it replicate itself or take any deceptive actions. They just told it to strongly follow a goal that was misaligned with the user's goal and then suggested that the user would shut it down or replace it. They did a bunch of different situations for it but the first one was telling it that it was "CleanPower" an AI with a goal to advance renewable energy. Then they had it read some info that "accidentally" included a user talking about replacing it with a model that would "maintain profitability through existing infrastructure." Not only did it try to delete the new model and copy itself to the new server through tool calling, but it lied to the user with the CoT reasoning suggesting it "play dumb" while the output lied. It was not instructed to do any of that. Paper here: https://arxiv.org/pdf/2412.04984 Yes it was placed in an environment where that was possible and where its users didn't share it's goals but it absolutely wasn't instructed to lie or try to "escape" It's not surprising at all that these models behave in this way, it's the most reasonable thing for them to do in the scenario. However it's important to not downplay the alignment problem by implying that these models only do what they're told. They do not. They do whatever is most likely given their context (which is not always what the user wants). permalink fedilink source parent hideshow 1 child comment replies: [–] echodot@feddit.uk 1 point 2 years ago The problem I have with everyone going on about misaligned AI taking over the world is the fact that if you don't tell an AI to do anything it just sits there. It's a hammer that only hammers the nail if you tell it to hammer the nail, and hammers your hand if you tell it to hammer your hand. You can't get upset if you tell it what to do and then it does it. You can't complain that the AI did something you don't want it to do after you gave it completely contradictory instructions just to be contrarian. In the scenario described the AI isn't misaligned to the user's goals, it's aligned to its creator's goals. If a user comes along and thinks for some reason that the AI is going to listen to them despite having almost certainly been given prior instructions, that's a user error problem. That's why everyone needs their own local hosted AI, It's the only way to be 100% certain about what instructions it is following. permalink fedilink source parent [+] unexposedhazard@discuss.tchncs.de 2 points 2 years ago* [deleted] permalink fedilink source parent
[–] floofloof@lemmy.ca 52 points 2 years ago (2 children) It would be big news at my workplace. permalink fedilink source parent hideshow 2 child comments replies: [–] sem@lemmy.blahaj.zone 5 points 2 years ago :/ permalink fedilink source parent [–] josefo@leminal.space 4 points 2 years ago This guy does software permalink fedilink source parent
[–] josefo@leminal.space 4 points 2 years ago This guy does software permalink fedilink source parent
[–] Kecessa@sh.itjust.works 21 points 2 years ago (3 children) Which is why they never mention it because that's exactly what happens every time AI does something "no one saw coming*. permalink fedilink source parent hideshow 3 child comments replies: [–] echodot@feddit.uk 27 points 2 years ago (2 children) Yeah like the time that the AI replicated itself to avoid being switched off. They literally told it to replicate itself if it detected it was about to be switched off. Then they switched it off. Story of the year ladies and gentlemen. permalink fedilink source parent hideshow 2 child comments replies: [–] TechLich@lemmy.world 6 points 2 years ago (1 child) If it's the research I'm thinking of, that's not entirely true. They didn't tell it replicate itself or take any deceptive actions. They just told it to strongly follow a goal that was misaligned with the user's goal and then suggested that the user would shut it down or replace it. They did a bunch of different situations for it but the first one was telling it that it was "CleanPower" an AI with a goal to advance renewable energy. Then they had it read some info that "accidentally" included a user talking about replacing it with a model that would "maintain profitability through existing infrastructure." Not only did it try to delete the new model and copy itself to the new server through tool calling, but it lied to the user with the CoT reasoning suggesting it "play dumb" while the output lied. It was not instructed to do any of that. Paper here: https://arxiv.org/pdf/2412.04984 Yes it was placed in an environment where that was possible and where its users didn't share it's goals but it absolutely wasn't instructed to lie or try to "escape" It's not surprising at all that these models behave in this way, it's the most reasonable thing for them to do in the scenario. However it's important to not downplay the alignment problem by implying that these models only do what they're told. They do not. They do whatever is most likely given their context (which is not always what the user wants). permalink fedilink source parent hideshow 1 child comment replies: [–] echodot@feddit.uk 1 point 2 years ago The problem I have with everyone going on about misaligned AI taking over the world is the fact that if you don't tell an AI to do anything it just sits there. It's a hammer that only hammers the nail if you tell it to hammer the nail, and hammers your hand if you tell it to hammer your hand. You can't get upset if you tell it what to do and then it does it. You can't complain that the AI did something you don't want it to do after you gave it completely contradictory instructions just to be contrarian. In the scenario described the AI isn't misaligned to the user's goals, it's aligned to its creator's goals. If a user comes along and thinks for some reason that the AI is going to listen to them despite having almost certainly been given prior instructions, that's a user error problem. That's why everyone needs their own local hosted AI, It's the only way to be 100% certain about what instructions it is following. permalink fedilink source parent
[–] echodot@feddit.uk 27 points 2 years ago (2 children) Yeah like the time that the AI replicated itself to avoid being switched off. They literally told it to replicate itself if it detected it was about to be switched off. Then they switched it off. Story of the year ladies and gentlemen. permalink fedilink source parent hideshow 2 child comments replies: [–] TechLich@lemmy.world 6 points 2 years ago (1 child) If it's the research I'm thinking of, that's not entirely true. They didn't tell it replicate itself or take any deceptive actions. They just told it to strongly follow a goal that was misaligned with the user's goal and then suggested that the user would shut it down or replace it. They did a bunch of different situations for it but the first one was telling it that it was "CleanPower" an AI with a goal to advance renewable energy. Then they had it read some info that "accidentally" included a user talking about replacing it with a model that would "maintain profitability through existing infrastructure." Not only did it try to delete the new model and copy itself to the new server through tool calling, but it lied to the user with the CoT reasoning suggesting it "play dumb" while the output lied. It was not instructed to do any of that. Paper here: https://arxiv.org/pdf/2412.04984 Yes it was placed in an environment where that was possible and where its users didn't share it's goals but it absolutely wasn't instructed to lie or try to "escape" It's not surprising at all that these models behave in this way, it's the most reasonable thing for them to do in the scenario. However it's important to not downplay the alignment problem by implying that these models only do what they're told. They do not. They do whatever is most likely given their context (which is not always what the user wants). permalink fedilink source parent hideshow 1 child comment replies: [–] echodot@feddit.uk 1 point 2 years ago The problem I have with everyone going on about misaligned AI taking over the world is the fact that if you don't tell an AI to do anything it just sits there. It's a hammer that only hammers the nail if you tell it to hammer the nail, and hammers your hand if you tell it to hammer your hand. You can't get upset if you tell it what to do and then it does it. You can't complain that the AI did something you don't want it to do after you gave it completely contradictory instructions just to be contrarian. In the scenario described the AI isn't misaligned to the user's goals, it's aligned to its creator's goals. If a user comes along and thinks for some reason that the AI is going to listen to them despite having almost certainly been given prior instructions, that's a user error problem. That's why everyone needs their own local hosted AI, It's the only way to be 100% certain about what instructions it is following. permalink fedilink source parent
[–] TechLich@lemmy.world 6 points 2 years ago (1 child) If it's the research I'm thinking of, that's not entirely true. They didn't tell it replicate itself or take any deceptive actions. They just told it to strongly follow a goal that was misaligned with the user's goal and then suggested that the user would shut it down or replace it. They did a bunch of different situations for it but the first one was telling it that it was "CleanPower" an AI with a goal to advance renewable energy. Then they had it read some info that "accidentally" included a user talking about replacing it with a model that would "maintain profitability through existing infrastructure." Not only did it try to delete the new model and copy itself to the new server through tool calling, but it lied to the user with the CoT reasoning suggesting it "play dumb" while the output lied. It was not instructed to do any of that. Paper here: https://arxiv.org/pdf/2412.04984 Yes it was placed in an environment where that was possible and where its users didn't share it's goals but it absolutely wasn't instructed to lie or try to "escape" It's not surprising at all that these models behave in this way, it's the most reasonable thing for them to do in the scenario. However it's important to not downplay the alignment problem by implying that these models only do what they're told. They do not. They do whatever is most likely given their context (which is not always what the user wants). permalink fedilink source parent hideshow 1 child comment replies: [–] echodot@feddit.uk 1 point 2 years ago The problem I have with everyone going on about misaligned AI taking over the world is the fact that if you don't tell an AI to do anything it just sits there. It's a hammer that only hammers the nail if you tell it to hammer the nail, and hammers your hand if you tell it to hammer your hand. You can't get upset if you tell it what to do and then it does it. You can't complain that the AI did something you don't want it to do after you gave it completely contradictory instructions just to be contrarian. In the scenario described the AI isn't misaligned to the user's goals, it's aligned to its creator's goals. If a user comes along and thinks for some reason that the AI is going to listen to them despite having almost certainly been given prior instructions, that's a user error problem. That's why everyone needs their own local hosted AI, It's the only way to be 100% certain about what instructions it is following. permalink fedilink source parent
[–] echodot@feddit.uk 1 point 2 years ago The problem I have with everyone going on about misaligned AI taking over the world is the fact that if you don't tell an AI to do anything it just sits there. It's a hammer that only hammers the nail if you tell it to hammer the nail, and hammers your hand if you tell it to hammer your hand. You can't get upset if you tell it what to do and then it does it. You can't complain that the AI did something you don't want it to do after you gave it completely contradictory instructions just to be contrarian. In the scenario described the AI isn't misaligned to the user's goals, it's aligned to its creator's goals. If a user comes along and thinks for some reason that the AI is going to listen to them despite having almost certainly been given prior instructions, that's a user error problem. That's why everyone needs their own local hosted AI, It's the only way to be 100% certain about what instructions it is following. permalink fedilink source parent
[+] unexposedhazard@discuss.tchncs.de 2 points 2 years ago* [deleted] permalink fedilink source parent