Pondering AI

KIMBERLY NEVALA: Welcome to Pondering AI. I'm your host, Kimberly Nevala.

In this episode, we are pondering synthetic media with the fabulous Sarah Barrington. Sarah is a Fulbright scholar who is currently in her final year as a PhD candidate researching generative AI, deepfakes, and synthetic media in Professor Hany Farid's UC Berkeley lab. And I have to mention she is also a graduate research fellow at Berkeley's risk and security lab.

Sarah, it's awesome to have you on the show. Welcome.

SARAH BARRINGTON: Thank you so much for having me. I'm so excited to be here.

KIMBERLY NEVALA: All right, so before we talk about the work itself, I would love to know what sparked your interest in this area of research in the first place.

SARAH BARRINGTON: Yeah, I think a lot of it stems back to being the generation that, I guess, grew up with the internet and with social media. Seeing this kind of evolving ecosystem of the online sphere growing around me and seeing myself and other people subject to just a range of online harms. To just kind of going about our lives, trying to exist online, and feeling out what that was looking like with the growth of social media and as things became more digital.

So that that's kind of happening around me as I'm a teenager and as I'm going into my degree. I was actually a mechanical and manufacturing engineer at university and then started to look at software in my job. My first job working at a Formula One company called McLaren took me into this kind of world of data science and processing data to do very powerful things with it.

So these two things are happening in parallel. I'm kind of developing these technical skills where I'm understanding how to make sense of what we now refer to as AI - but at the time was kind of analytics and data science and these things. And I'm growing ever more passionate about what it means to protect people online. So I was able to combine those two things by doing this PhD with the wonderful Professor Farid and the rest has been history.

KIMBERLY NEVALA: Yeah, so you moved from the literal fast track to the fast track of synthetic media. That's amazing.

SARAH BARRINGTON: That's right. Yes, yes. Very different speeds, I must say, academia and that industry. But both fun nonetheless.

KIMBERLY NEVALA: That's excellent. So let's maybe start by giving folks a sense of the overall story of the work, if you will: where this all started and what it's become today.

SARAH BARRINGTON: I think, really, you would need to go back to the start of photography to think about--

KIMBERLY NEVALA: Wow.

SARAH BARRINGTON: --when we talk about this idea of synthetic media or what it means to engage with a piece of content and really understand where it came from, who's behind it, what it means, and whether it's authentic or not. This has been a topic that has been a focus for centuries, if not millennia, when it comes to people impersonating each other, stories, myths, these sort of things.

When you bring photography into that, all of a sudden, there is, of course, going to be people who are trying to use that for bad. So the birth of photography means also the birth of image editing. So people are going to try and do things to make those photos seem inauthentic. And there's a very well-documented and very interesting history of how traditional photography and traditional photographs have been manipulated in really sort of, like, prominent historical situations to kind of rewrite history. Some really fascinating examples of those online. But we're talking hundreds of years ago at this point.

So fast forward to today: what's changed? Well, of course, digital photography comes into our sphere of existence in the public domain. Then we get things like Photoshop. So we get the ability to manipulate images digitally. This means that the barrier to creating something that looks very authentic is, all of a sudden, a lot lower. But still, there's some skill involved. Then you fast forward to today's generative AI, which is just AI models: the ChatGPTs, the Geminis, the OpenAIs. These models are able to create things that look like they were built or designed by a human from just a few input sentences.

So at this point, we're able to create photographs really easily. We're able to manipulate those, and they look very perceptually realistic. But of course, they're inauthentic. So there's been this evolution. Humans have wanted to do bad things with this technology for a long time, but generative AI, what's really changed here is that it kind of gives people the ability to do that really easily and really cheaply. And that's kind of where we're at today.

KIMBERLY NEVALA: And it is so interesting because you and I have talked about this propensity of us all to talk about AI as if it's just something net new. And that some of the challenges that it brings with it are somehow really just unique and never before seen. But coming back to the start of photography really just puts paid to that narrative, doesn't it? It really says, hey, this is, unfortunately perhaps an escalation of what's been happening. But it plays into our innate human nature.

But to that end, you have said-- I guess is it true - let me ask this as a question - that synthetic media and what we can do today with generative AI platforms is really supercharging existing harms more so than creating net new?

SARAH BARRINGTON: I think that's a really good way to put it. And we love this phrase in our lab, supercharging existing harms. Because, again, any time a new technology comes out there is unfortunately a side of humanity that will try and use it for bad. And with this technology in particular the bad use cases are really, really apparent. And often, the use cases that are good are less apparent and they are starting to emerge, which is really exciting.

But when we saw these technologies first start to emerge-- So by the way, the term deepfake, which is often what we talk about when we say generative AI videos, fake videos. This term deepfake actually came from Reddit in 2017 when deepfake technology, so using AI or deep learning to create these fakes, was being used to basically declothe with actors. So putting a female actor's face on a kind of pornographic image and doing these really horrible, scary things. So the roots of this technology, are kind of bad when we think about synthetic media specifically.

So emerging today, these good use cases. But of course it's really hard to forget where the technology came from. And these harms still exist very much so in the present day as well. So that's one example. But of course, there's fraud. We're seeing such a rise in fraud being, again, supercharged charged by this ability to create really human kind of appearing content. And, of course, disinformation-- we can now put, at scale, disinformation out there that is really convincing for people, and we can create new narratives in the world. So these things have all existed before.

But again, just coming back to what's different now is the ease and the democratized access to this technology. So that anyone can create something that's really convincing very easily and very quickly.

KIMBERLY NEVALA: And when we look at some of those harms like deepfake scamming and fraud, what are some of the more concerning, I was going to say use cases or applications, but that feels like it's putting a positive spin on it. How is the way-- what are some of the ways that people should be aware that this is being used that is really so harmful?

SARAH BARRINGTON: I think honestly the way you've described use cases and this kind of product language is probably how scammers - well, it is how - scammers and fraudsters talk about these technologies. Like, you will know this really well because of this amazing podcast and all the amazing guests you have on. There is genuinely sort of a cybercrime industry like, almost mirroring a lot of legitimate industries and how they function. And there's markets and there's talent but in this really horrible, nefarious corner of the internet.

So that's a very valid way to frame it. And the same things that you think about in general software as a service - scale, ease, and targeting of people, so knowing your customer, that kind of thing - that's where AI is really helping on the fraud side as well.

So the two really big things here, I think, are scale. So now we're able to generate sort of this fraudulent content at a scale that just was not previously there. If you wanted to create a phishing email that was really convincing, you'd have to have someone sit there, type it out, come up with a narrative, design this whole campaign, to effectively get someone on the other end to believe it. And now you can turn up to ChatGPT. And of course, there are guardrails for some of these models. But you could turn up to ChatGPT and pretty easily get it to write you that phishing email. So all of a sudden you're able to very quickly kind of replace humans in that scamming process. So you have that scale.

And of course, you've got this micro-targeting. So, really big thing that we worked on in the lab is these kinds of AI voice clone scam calls. So if you have come across this before, this is when someone receives a call from a purported family member. I've been in an accident. I've been in a car accident. I need you to transfer me $20,000 on Venmo or else I'm going to die. I need that for my medical bills. And it sounds like the relative, but it's actually an AI deepfake of them.

Not to mention, of course, that you're also in this elevated emotional state when you receive that call. Which much really good research has shown that that also decreases your chances of being able to spot that something's fake. But these things are so realistic anyway that it's really hard for someone to understand that is, in fact, a fraudulent call.

So these things are happening, and it's really scary. It's very personalized, very targeted. And again, that's what makes it so much easier for these scammers to come out with these realistic campaigns now. So, yes, it would be that scale and that targeting, I think, that AI really is supercharging.

KIMBERLY NEVALA: And you had mentioned earlier about rewriting historical narratives. And to some extent, as I understand it, some of your research is about folks really trying to influence current narratives. Things that are soon to be history. And I know that you've done some research on this in the context of wartime and conflict. What are we seeing there and who's wielding these tools for nefarious purposes?

SARAH BARRINGTON: Definitely nefarious. Very interesting and important time at the moment. Of course, with all of the tensions around the world, there's often information warfare campaigns that are happening alongside those. Ranging from traditional propaganda to sort of much more divisive or polarity inducing narratives to stir up anger and whatever that might be. But this time, particularly with the current Middle Eastern conflict, we're really seeing very sophisticated falsified content coming out from various different locations onto the public online sphere.

So often, these videos of things like alleged or purported explosions are circulating on X, for example, or on Facebook. And after doing the rounds and being viewed by millions of people, when we actually analyzed these things, it turns out many of them are false. Of course, this is made worse by the fact that we also have these, I guess, verified accounts now that can be very easily slapped onto a legitimate-- to make a social media profile look legitimate, you can slap one of these blue badges on that you can buy for $5 a month, whatever it is. And all of a sudden, you have a very realistic looking piece of news.

And in war zones as well, of course, it's so important that people have access to timely, accurate information. And people are crying out for that. So when they're seeing this breaking news shared online, perhaps they don't have the capacity to fact check that to the n-th degree. And perhaps we let some of our sort of critiquing of content slip. So this is a really perfect storm to spread misinformation and disinformation.

But the thing that really is alarming about this current information ecosystem associated with the Middle Eastern conflict is really the quality that's coming out. We are seeing this kind of step change in how realistic these videos look. And I think a piece of research that you and I have spoken about previously that's really relevant here is some work that we did recently led by my colleague Catherine. We were looking at this idea of how people can differentiate a real and a fake video.

And she led this fantastic piece of work where we showed people these videos ranging from different durations, from a still image up to 8, 10. And in those experiments, we asked people to say, do you think this is real or fake? And we observed that for the fake videos, watching for longer actually did help people spot these kinds of cues or these things that might tell them that this is a fake video and give something away. The six fingers on a hand, all those classic things. But for the real, actually the increased duration didn't increase their confidence in how confident they felt about predicting that it was real.

So it's kind of like we're living in this liar's dividend, it's called in the literature, where we're in a world where we're questioning everything. Because we now don't know what's real and we don't know what's fake. And anyone can say anything, and it's very hard to prove.

KIMBERLY NEVALA: And I find this really interesting, in a somewhat scary way, that we can try to train people up to do this in a way, to start to spot fakes. Although, to what you were saying earlier, it is going to get harder and harder to do that unless you're really being maybe very sophisticated in how you're researching. Are these all different pieces that you can trace back to other pieces of media and things? And that's time-consuming. It takes a minute. The tools to help us do that don't seem to be evolving as fast as the tools to create them from my outside in.

But you would think, or someone might logically conclude, if you're better able to spot fakes that means you have more trust in your own ability to know what's real. But those two things are completely disconnected, huh?

SARAH BARRINGTON: They are completely just-- they're largely disconnected. They're different questions. And I think this is something that one of the fundamental beliefs we have in the lab is the question of what is real and the question of what is fake, they're separate. And therefore, when you think about interventions for this: so what would be a good technological intervention for this problem? Well, perhaps it would be content labeling. So when things get shared, social media largely being the place where these things get disseminated, so when we share these things, having a label that says this was created by AI, or this was real, would be really helpful.

But when we first, as a field, started thinking about building these detectors to detect fake content or to do this real-fake prediction, very quickly it became apparent that there are limits to this approach. So on the more nerdy technical side, when we think about building these kinds of machine-learning approaches - like using AI to detect AI-generated content - to develop a detector that can actually give us a label and say this is real or fake, we come across these problems. Where no matter how well you think train your algorithm and you show it millions of images and data, if it comes across data that's out of distribution - so things it hasn't really seen before or come across before - it really struggles to generalize.

So on top of that, also there's a kind of issue of explainability. If we label something as real, can we really explain why that is? It's some algorithm that was trained. We can't really give a good answer to, oh, yes, it's because there's five fingers or the six, whatever, right? So we have these issues, this kind of generalizability and this explainability issue, that makes it really hard to develop a really good detector.

So then when you take that online and this thing is inaccurate. We can't live in a world where we're inaccurately labeling the provenance of content because then we're losing trust in anything that we see online. And that's a disaster too. So it's almost as if there's no simple silver bullet for this problem. It's a very multifaceted problem. And sometimes that can be a kind of misbelief about the work that we do.

KIMBERLY NEVALA: And how does this then tie into, there's a - I'm going to call it a trend for lack of a better word - about this idea of verify you're a human, right? Or providing proof of personhood. So talk to us about the difference between verifying something or someone - not something, someone - is a human versus something is a machine.

SARAH BARRINGTON: Yeah, and again, it's these two different questions. And the question of is this a human is really interesting because there's actually two parts to that as well. There’s is this a human, yes or no? And there's also is this a human with the identity that I was expecting? So if you join a call as me, sure, it's a human. OK, so it passes the human check. Great. But the identity is wrong still.

So it's not as simple as just is this a machine or is this not. There's actually this question of who is speaking as well. So we're in a world where we already have sort of captures and two-factor authentication for our general digital lives. So there's some sort of good digital hygiene around those things. But when it comes to our faces and our identities and our voices, we don't have that same kind of protocol in place for how we deal with that.

So we are seeing this increase in these tools and these services that will kind of verify that there's someone human on the other end of the line. That's super, super important. And I'll shout out here Get Real Labs, Get Real Security, who are a fantastic company doing really great things in this space. And I've worked for them a little bit, and they're co-founded by my advisor, Professor Farid.

And these companies and these tools are going to become so important now because our identities really aren't safe online. Face, voice, the content of how we speak, all of that now can be packaged up together into an avatar and, in real time, used as your digital identity. I know this because we've just finished a big study or sort of data set curation project that we've been doing called DeepSpeak.
And we're on the third version now and this is when we engage crowd workers from the general public online and they interact with various different types of deepfakes. In this most recent version, what we've done is we've actually got these people to talk to AI avatars. And we didn't tell them that they were AI avatars until after the call. At which point we said, when did you realize? And there are people who went through an entire 10, 15-minute call who didn't realize they were talking to an AI rather than a human.

And that's today. So can you imagine, can you imagine, where we're going to be in a year's time? The trifecta of digital identity has been solved.

KIMBERLY NEVALA: Yeah, and I've been thinking about - when I've been reading some of the ways that folks are trying to even lean into this - there seems to be almost a Faustian bargain that's being created. Which is a trade-off between having this idea that you have to give your biometrics to support things like continuous authentication. Because you've also said you can't just do sort of point in time. But that, by definition, means you have to give up your biometric information and, to some extent, rights to it. How do we think about that problem and that is there a clean way out of that?

SARAH BARRINGTON: That's a really great question. I think the biggest challenge for me when thinking about this stuff is who. Who is collecting that data?

And it's now a kind of inevitability that some sign of life, some kind of proof, is going to be required to show this is you. Now we all want this. Where we're able to prove that it is indeed me on the end of this call and not someone else or an AI or whatever it might be. So there will have to be tools that can do this. But who is controlling them? Who is taking that data? How are they capturing that data? These are the questions we need to be asking. So it's not the existence of these things full stop, but it's about the control and the power that the people developing them wield.

So I would urge people really to think about that question of who, how, what with regards to data. I think it's amazing that you're asking these questions, and I hope everyone else listening will as well. But I really worry about this. And I worry about this generally as well with the rise of agentic AI right now. Where we're very excited, all of us in research and beyond, about these agentic systems.

So AIs that aren't just ChatGPT living in your browser but can actually get in your computer and control things and act as a human would in a lot of ways as an agent. We are kind of just handing over our entire digital lives to these things in a way that right now isn't particularly controlled or understood, shall we say. We can put various kind of security measures in place to try and ring fence data and all these kinds of things. And companies like Anthropic are doing great work with that. But we still don't really know what the boundaries are of that kind of problem.

So handing over data, handing over our digital identities, it's going to become more commonplace. But who, how, and what are the key things we need to ask about that.

KIMBERLY NEVALA: And I want to talk a little bit about the policy landscape and some of the narratives that are happening on that side. But I have two questions before we go there. And one is around something, again, that I saw in your work, in some of the research that I don't know if it was your group specifically or folks that were associated with you. It was something that, I guess a concern, or I don't even know if it's a risk, that I hadn't ever thought about and I think is maybe underappreciated.

And I wrote it down, so I didn't forget. And you were talking about, I think it was around voice cloning, and David Attenborough. And the quote was: this raises deeper questions about identity and the cultural significance of a voice. And not, as you said earlier, just real versus fake. Can you tell us a little bit about that and what are some of those deeper questions?

SARAH BARRINGTON: Yeah, I think this is a really interesting case. It's a fantastic article written by, I think, it's Danny Chow at The Ringer. Was really happy to be a part of this piece because it explored effectively the kind of importance of David Attenborough's voice in cultural, I don't know, in kind of the culture of the UK and beyond. And what it looks like to associate a voice with not just a person but kind of a set of beliefs. Or a community. Or maybe even kind of link it to values as a civilization.

So David Attenborough's voice: he's a kind of documentarian. He is always working on things to do with nature and climate and these really wonderful projects. And his voice and his identity, therefore, is so prolific. But in a way that is not just, oh, I recognize that person, but that person's kind of cultural identity has a lot of weight as well.

So the questions we kind of think about in the lab. Not just does this voice, with the voice cloning stuff, does this voice sound like this person. But it's also: and what do you do with it? So if you're using that voice to scam someone then, of course, that voice, that cultural identity, has a lot of weight to the person who's receiving it. If it's meant to be a family member versus just a random fake voice that sounds highly realistic but doesn't have that relevance to them.

And similarly, if we took, stole, David Attenborough's voice, created a fake of it, and decided to disseminate disinformation with it - which is exactly what happened, by the way - and spread that online. It was associated with political disinformation for a while. People were using David Attenborough's voice to push sort of sketchy political messaging. What does that mean?

That voice has some sort of trustworthiness to it in the cultural zeitgeist. So you're not just causing a harm by deceiving someone. You're causing a harm because you're also playing into these kinds of greater questions of truth and trust and what it means to be a human that has a recognizable identity in that way.

KIMBERLY NEVALA: Yeah, and as you said, that import of just their expertise or, as you said, the cultural significance. And we used to say you'll know it when you see it. Or you would know him if. Once you hear him, you'll always know him. If you've heard David Attenborough, when you hear him again, you'll know it. And we just can't say that anymore, can we?

SARAH BARRINGTON: No, I don't think we can. I don't think we're safe from that and I don't think anyone's safe. And again, the kind of warning we come back to in the lab is if you've got so much as five seconds of your voice online - which most people have and if you think you don't, your voicemail probably does or your phone or something - if there is five seconds of your voice available digitally somewhere you are not safe from this. So it's a problem we all need to be aware of.

KIMBERLY NEVALA: So maybe before we go to the policy landscape and the narratives there, which I'd like to say there's some bright spots, but I'm not sure there are. You did mention early on that we're now seeing the use cases for ill became apparent very, very quickly. But we are seeing more use cases for good. So maybe we'll add a spot of bright here and tell a little bit about some of the use cases you've seen that are really positive before we continue down this path.

SARAH BARRINGTON: I know. It's a great idea. I always love having a reminder for that. When I do presentations and talks, I always try and add in something positive at the end. And I have to say on the whole as well, I'm an AI optimist. It really doesn't come across in my work all the time but there is a huge amount of good that can be done with this.

And I think that we need to separate the distant future of AI, which is AGI - so AGI is like that distant idea of recreating human intelligence in a machine - and we're not there yet. What we currently have is today's AI, which is effectively language models, video models, photo models. Things that we can input a few lines of text about and what will be returned is a kind of piece of content that looks like a human created it. That's very different from this dystopian future.

So thinking about what we even just have today, there are so many amazing things we can do with this. Big thing for me in my everyday life is kind of automating software engineering tasks and these technical tasks that we have to do. Really is quite astounding out here in San Francisco and in the Bay how incredible these tools are. These LLMs are at creating really good working code. And there are problems associated with it; we haven't solved the problem completely yet. But these things are very, very powerful, and are able to build digital infrastructure in a way that is super impressive.

And of course, we also have, on the cybersecurity side, big developments coming. The likes of Anthropic and Mythos and something I know you're very passionate about. But these models coming down the pipeline that can help secure our existing digital infrastructure and our tools. And do it in a way that's really kind of systematic and can replicate some of the best kind of hackers in the world and actually use that to patch holes in our ecosystem online and make it kind of much more safe and secure. So that's great.

And then, of course, there are these breakthroughs as well happening in just kind of making human life better. And a really beautiful use case that I love coming back to is giving people a voice back when they might lose it due to illness. And I know that the voice cloning company, ElevenLabs, have worked with ALS patients who have been able to kind of preserve their voice and use that moving forward as they lose it due to illness. And that is just the most beautiful way to use a technology like that. So that's a really apparent good use case, and it's a shame that these good use cases often have so many bad use cases as well paired with them.

KIMBERLY NEVALA: Yeah, and I have to give a shout out. We did an episode early with Lama Nachman from Intel on the ALS use case. And it was really interesting because there are a lot of ethical guardrails that need to be put in place there. But the transformative power when that is wielded correctly is really phenomenal.

So now you mentioned Mythos. Which is maybe a good turning point into the policy landscape and some of the narratives that we see there. Mythos was interesting to me because I have to say, out loud, admittedly, I'm a bit skeptical about the motivation behind that. Those announcements and where they're really trying to push. But when you look at something like Mythos and the narrative around that, how does that then tie back to the work that you're doing? Or how do you put that into context in the overall landscape that we're dealing with today?

SARAH BARRINGTON: Yeah, I think we watch these sorts of things with great interest. So I wouldn't be working directly on something like a Mythos technology. That's very much a kind of cyber security, hacker-style tool. So while it's not directly something that I would be tinkering with myself, what I really find interesting about it is how the company, so Anthropic in this case, but generally speaking how a company releasing something like that handles the release and who they're releasing it to.

So Mythos has been very interesting because there has been this kind of slow or controlled rollout. At the same time, Anthropic talking very loudly about the power of the model and using that actually as the kind of reason to justify the slow rollout of it and the controlled rollout. And so there are issues with this. We are still putting a kind of a model that maybe isn't fully understood into a very high criticality situation or use case. And Anthropic will be actively working on what that looks like and how to make sure that it's doing the right thing and it's in the right hands.

But what is fascinating is who gets access and why. And I think the other thing kind of that worries me is that once a technology is out there - even if you're controlling the release and you're trying to limit who gets access to what - once the technology is known and is out there, there's not much that's able to stop it really ultimately being able to proliferate elsewhere through various different mechanisms. So I think I look at this as an interesting and useful step in the right direction but will be very intrigued to see how this plays out in the long run.

KIMBERLY NEVALA: Yeah, and I think where my skepticism comes from is, to me, it feels like something that is an announcement of we're doing something for good. While we are loudly proclaiming that, A, we've got a system that we clearly tested, possibly trained, to be able to do this on purpose. And, we are loudly telling you why it could, but you shouldn't, use it for this. And that, to me, just seems like an invitation to actors to say that. And you don't need to comment on that. I'll just make that my own personal statement. So I am struggling with that skepticism about the effect of that narrative, regardless of what the intention was, for good or ill.

SARAH BARRINGTON: I think, really, Mythos is a great example of just how AI can be the ultimate dual use technology. Meaning that it can be used for good and bad. And this is an absolutely classic example of that. Where it's really an arms race in cybersecurity, and we've developed the ultimate tool that can play on both sides so that you really just have to pick a team. And Anthropic have picked the defense side.

But you can imagine if this got into the wrong hands, it could be really terrifying. And this is what's going to be interesting and scary for how this plays out. But yeah, that being said, I do think the alternative here could have been let's just release this or let's just not be very careful with this. And that would have been an absolute disaster. So it was a step in the right direction.

KIMBERLY NEVALA: I mean, I guess there's the alternative not releasing it as well. But yeah, anyway, so I've banged on that drum long enough, I suppose.

SARAH BARRINGTON: I would fully agree with that, actually. But then I think you're giving Silicon Valley companies too much credit. Where there's profit to be made, these things are getting released. I don't think not releasing that was ever an option.

KIMBERLY NEVALA: Yeah, I don't know. I would tend to agree. And I think it was you that maybe said to me that you can't talk loudly about the tech and what it shouldn't be used for and not expect that others are going to get there immediately.

But, that being said, I think when we look at the regulatory landscape more broadly, you've said when we explore some of the anti-regulation narratives that they really lean on the premise of how unprecedented AI is. And I'm wondering if you can color in a little bit about the implications of that and how we go about trying to regulate or corral this? Especially relative to these existing harms that are being supercharged here.

SARAH BARRINGTON: Yeah, so I think this is a question we've been thinking about a little bit recently in the lab. Which is, how do we regulate AI? So there's a huge question.

Lots of people are thinking about that in the field. But from our work we see these possible solutions. But we also see the complexities of these solutions are things that we spoke about before. You can't just label content with a detector. That's very hard to do. So we see both sides of this. And we see policymakers in a lot of ways having a valid struggle with what to do about these harms. Because they are complicated and there's no simple solution. And so we've been thinking a little bit about how to bridge that gap.

But some of the narratives that get pushed, particularly in Silicon Valley or by these model developers, will be around AI is too new and unprecedented to regulate. So while it is undeniable that these things, these models, have got novel capabilities and are unlocking technological breakthroughs every week, and that's very impressive, we have seen kind of dual use technologies before. We've seen technologies that enable scams before. We've seen these harms for decades at this point, and how are we going to deal with those harms is actually a question that is different from the technology.

So by making it a discussion around, oh, AI is unprecedented, and it's too difficult to even think about regulating something this big and this broad, is not really a kind of valid way to frame it. And on the breadth thing as well: when we see language about AI, often, it gets conflated with this idea of AGI. Which is what we mentioned earlier, Artificial General Intelligence. We sort of abstract away from, oh, this LLM, which, by the way, lives in the same browser interface, the same apps, the same software products we've been engaging with for 20 years. When we abstract away from that, actually, these things are, very complicated, too broad, that we don't know what they're capable of. Really, they're just products being used for use cases right now. So that's one narrative.

And then the second thing that happens a little bit is the regulation of AI gets tied to the kind of national security agenda in a way. And this phrase of we must win the AI race against our adversaries or whatever that might be. And while that is true, it's pushed in a way that implies regulation will stifle innovation, which we don't think is true at all, actually. And I think there are examples where regulation can actually kind of encourage safe and very directional innovation, which is a good thing for everyone.

And then the final thing that happens a lot in this world of online harms is that discussion around regulation gets tied to content moderation, i.e. the First Amendment, free speech. So by trying to label content, even though we know that's not a perfect solution, if we're trying to label something or we're trying to prove that something circulating on social media is fake, we actually run into questions around, is that moderating speech? And it becomes very hard to have that discussion because that's a very high constitutional level. And it's difficult to bring that back down to: we're not talking about all that. We're just talking about can we get better at controlling the first part of this, which is people shouldn't be able to create pornographic imagery when they go to one of these models. But these two get kind of pushed together with this anti-regulation narrative.

But yeah, just to come back to the fact that in a lot of cases right now, these AI models in their current form, are software products and things we've been engaging with for decades. And we can get through a lot of regulatory kind of barriers by just treating them in that way.

KIMBERLY NEVALA: Yeah, well, something else that's coming through in this whole discussion is that a lot of times, we put - or even the questions that I'm asking - is we put them out there as if they're a single thing. And actually, you're saying no, look. These are two different questions.

SARAH BARRINGTON: Yes.

KIMBERLY NEVALA: Is it a human? Or is it the human that you think? And which human is it and is it a human, these are different questions. And can you create this kind of harmful content is different than free speech. And so us being critical and careful about how we ask those questions and disentangle those separate issues seems really key.

And you've also called out that it's fairly in vogue at the moment to look at - and again, maybe this is because people are foreshadowing what they see as this dawn of AGI. But the way that we need to think about this, or regulating this tech, to the nuclear analogy. And you said there are assumptions there that we're relying on that don't hold true. That's my recapitulation of that, not your exact words. But talk to us about why people are drawn to that analogy in your view and where it falls short.

SARAH BARRINGTON: Yes, so I think people are drawn to that narrative because it really highlights the kind of transformative nature of this technology. AI is a huge technology and if we think about it in that context of AGI, which is ultimately the future vision, that really is like civilization-changing technology. And how many of those are there to compare against? Well, nuclear being the really obvious first example. So I think people are drawn to it because it really implies that transformative nature and that scale.

But it's not always valid for a bunch of reasons. And one of them is I can't sit at home and build a nuclear weapon. Whereas I can sit at home and access AI pretty readily. And I can use it to enable me to do lots of other things. The barriers to entry with AI are just so much lower, and in fact, the technology and the products are designed to be as such. We are going through this typical tech playbook of getting the whole world to adopt something and then picking up the pieces afterwards. Again, that's absolutely not what's happened with nuclear at all. And these things aren't directly comparable.

I think it is valid to raise, kind of, awareness in that way and often using this analogy as to create some urgency. And that can be a useful tool but the analogy needs to stop at a certain point. And I think it's also used to kind of skirt over these really important questions we have about what the harms currently are by focusing on these big existential threats, the AI threats that are at the scale of nuclear weapons. And perhaps that takes away from the conversation about your grandma who's getting scammed because she received a phone call.

So again, that can be tied, that can also be tied to this regulatory paralysis. Where we start focusing on these big existential risks, which, don't get me wrong, are very important and scary, but can often be associated with a black swan probability. And we have harms happening today that we need to protect against. So what are we doing about those?

KIMBERLY NEVALA: Yeah. So what, in your mind, might be the antidote - or the start of an antidote - for some of these harms and what do you see as some of the most promising developments in the space? And I have to imagine that this goes beyond just tech alone.

SARAH BARRINGTON: Yes, I think the bigger picture of all of this is that, one, we need the public who are experiencing these harms to have an awareness and understanding of what this technology is and what these harms are. We need policymakers to be holding industry accountable, and we need industry and academia to be developing methods that can be real safeguards.

So not one of these components is going to solve this problem alone. It really is this big picture of those three forces coming into play. And all of our work ties into this idea of how can we get these three to work together to solve this problem? So by having industry developing these models but also thinking about safeguards that go alongside them. And there has been great work started in this: not just immediately releasing products, as we talked about with Mythos, but actually giving some thought into how to do that in a safe and controlled way.

But policymakers ensuring that this is actually happening. And really, a lot of our work is also trying to give policymakers the language to have that discussion. And our work on looking at these anti-regulation narratives is designed to be for a policymaking audience. To equip them with the vocabulary to have these discussions with industry so they're not bamboozled by the overwhelming nature of this technology and the harms.

And then, yeah, on the public side, so for listeners as well, what can we do about this? And in your everyday life, what does this look like for you? My biggest piece of advice is just to get really good at questioning everything you see. And I think we got a little bit lazy as a society in we no longer have to seek out our news. Hany always says never get your news from social media. If you're getting your news from social media, then you've already just failed at step one.

But we got kind of lazy as a society with the internet bringing news and current affairs and our world right to our door, right to our phone screen. And we need to get back to a society where we're being a bit more conscious about how we seek information. And the first step to that, really, is just questioning everything you see online. A really obvious thing to do, now we're in a world where the content is basically indistinguishable, real and fake. Best thing to do is look at who's posting it and why. If it's coming from a legitimate news outlet, it's a pretty good chance we know we can trust it. But if it is being tweeted by some strange account that has zero followers, immediately it should be raising a red flag. And that check, by the way, took maybe two seconds. So just getting a lot better at critiquing things.

And then when it comes to tangible things you can do, particularly with voice clone work, you can actually have a safe word with your close friends and family. So that if someone gives you a call and you're suspecting that it might be a deepfake, that you can say code word, sorry, would be whatever. And also, if you're receiving what you think might be a scam call, just call back. Hang up and call back.

So these are the basic things. But it's no longer just about looking at a piece of content and it's going to be obvious that it's fake or real. It's about looking at the wider picture of that piece of information: who, what, why, and what was the intent. So lots of things we can do. Really, all of us just knowing this is a problem is a really good first step, I think.

KIMBERLY NEVALA: Yeah, so really quickly, I have to shout out that I think I pronounced Professor Farid's first name incorrectly earlier and didn't even notice I was so excited. So Hany, I apologize if that was the case. And I may have just done it again. So--

SARAH BARRINGTON: He's going to be angry.
[LAUGHING]

KIMBERLY NEVALA: Back to the actual point here. So as we are looking forward and there's just so many things happening. And they're so fast, and as you said, the bad things just seem to grow faster than the good ones. Which do come around, but they're not always as obvious. What just keeps you motivated and what are you looking forward to as we move forward here?

SARAH BARRINGTON: I think a world with AI and humans coexisting could be absolutely wonderful. Don't you just love the idea of automating the things that you find really boring or you don't enjoy? Or parts of your job that seem monotonous to you or whatever it is? Whatever it might be, all of us will have parts of our life that it would be lovely if we had an AI assistant to help with.

But I think what's going to be the challenge is getting the balance right. What is AI? How much is AI? What is human? What remains human? What makes us human? That's going to be the really tough question, but I remain really optimistic that there's going to be a wonderful future.

Well, wow. Wonderful is a strong word, but I believe there is a possibility for a really exciting, beautiful future where humans are completely empowered by technology. And able to really harness it for good. And there are sort of safeguards that can be put in place to make sure people are using it properly. And I'm not saying they will always be 100% effective; nothing in cybersecurity ever is. But if you can get 99% of the bad out, typically, you're in a pretty good place with that.

And that's how the rest of society works as well. We're never going to get rid of all the bad things happening at once. So I just dream of that future where we have technology being rolled out in a careful and considerate way. And we are able to empower humans to live their best lives because of this wonderful, transformative technology and not be hindered by the bad.

KIMBERLY NEVALA: Well, I think that is a wonderful vision to leave ringing in all of our ears and hopefully just motivates us all to really make that manifest. I think if we can't imagine it, it can't happen. So that is fantastical and I so appreciate not only your time and insights today but all of the work that you have been doing and I know will continue to do in the future. So thank you so much.

SARAH BARRINGTON: Thank you so much for having me. It's been wonderful talking to you.

KIMBERLY NEVALA: Awesome. Now, if you would like to continue learning from thinkers, doers, and advocates such as Sarah, you can find us wherever you listen to podcasts and also on YouTube.

SPEAKER 1: This has been a SAS podcast.