Sir Jim Mann. Photo: Billy Wong/University of Auckland
A professor who was impersonated to sell a bogus diabetes treatment is urging New Zealanders not to fall for deepfake scams.
Sir Jim Mann, a leading endocrinologist, was mortified when he discovered his face and voice were being used to scam patients with type 2 diabetes.
A video posted on Facebook in November claiming to be a lecture delivered by Sir Jim urged patients to stop taking the medication metformin and instead buy a scam gummy product.
In reality, the video was using AI deepfake technology to impersonate his voice.
"The AI was so effective, it looked like I was actually saying those words... I (the impersonator) was saying that there was a discovery relating to a new natural medicine for diabetes, and the current standard medication for type 2 diabetes was inappropriate and should no longer be used."
Sir Jim feared his expertise was being used to harm the very people he had dedicated his life to helping.
"It was particularly serious because it was telling people to stop taking a medication. Given that I have been in practice as a diabetes physician for an incredibly long time, I'm a patron of Diabetes New Zealand... it was potentially a dangerous recommendation," he said.
Stacey Edmonds, a deepfake scam educator working for Mastercard, said the scams were becoming more common and advanced.
"At least 29 percent of Kiwis have been targeted by deepfake scams in the last 12 months, and of course these are just the people who have reported it and know it's a deepfake."
The scams could circulate for days at a time because social media users did not bother to report them, she said.
"How many times have you seen a deepfake and gone 'that's probably a scam' and just ignored it?"
When the video of Sir Jim was reported and removed, he said it popped back up under a different account.
"It was taken down quite rapidly, but then it popped up again repeatedly. Every time there was a complaint it seemed to come back again very quickly.
"I would never have believed this could have happened. I didn't think people bothered to do this for people like me."
Photo: 123RF
But Edmonds said it was vital New Zealanders understood this technology existed and was being used to target them.
"The number one [tip] is know that they exist, because once you know they exist, if you see something and think it's a bit odd or a bit dodgy you'll [recognise it]."
Though the technology had advanced significantly in recent years, she said there were still ways to identify the impersonators.
"Does their face match what they're saying? Often the bottom half of the face is mismatched, so it looks a bit odd. Does the audio sound a bit weird?
"So you've got this combo of: does the face match the mouth? Does the mouth match the words? Is the audio sounding natural?"
Those flaws could be ironed out in the future, but certain anti-scam tactics would continue to be relevant, she said.
"Scams work on social engineering, so they want to make you feel afraid, excited, worried, because emotion kicks in and you're unable to make rational and critical decisions. So if it sounds too good to be true, it probably is."
Sir Jim urged social media users to be more discerning.
"There should be a higher level of suspicion. People need to be aware there's a very wide range of scamming going on. It's not just people trying to get into your bank account, phoning you up and asking for your pin number.
"A number of people have said to me, 'Gosh this has been a good lesson for us.' … That you shouldn't be taken in just because you hear someone well known talking about something."
Sign up for Ngā Pitopito Kōrero, a daily newsletter curated by our editors and delivered straight to your inbox every weekday.
