Pondering AI

KIMBERLY NEVALA: Welcome to Pondering AI. I'm your host, Kimberly Nevala.

In this episode, I am so excited to bring you Dr. Rebecca Portnoff. Rebecca is the VP of Data Science at Thorn. Thorn is a nonprofit developing solutions and driving initiatives to defend children from sexual abuse. She joins us today to talk about why so many - maybe even most - of our AI systems are unsafe by design and what we can do about it. Welcome to the show, Rebecca.

REBECCA PORTNOFF: Thanks so much, Kimberly. It's great to be here.

KIMBERLY NEVALA: Now, you have your master's and PhD in computer science and machine learning and have been working steadily in this area of AI and child safety for nigh on a decade. And I know that is exceedingly important work and excruciatingly difficult, so I just want to put a shout out here to you right up front. But based on your experience and all this time, what do you think that we need to broadly understand or acknowledge about the types of harms being perpetuated and our responses to date?

REBECCA PORTNOFF: Thanks for asking, Kimberly.

I think it's really important to have a solid understanding that's grounded in the data when we talk about impact and when we talk about scale as that's really the best way that we can start to then ideate on the right solutions to both prevent and respond to this kind of abuse.

So when it comes to that question of impact and scale, we tend to focus on three main pillars of impact. The first is around victim identification. The second is around revictimization. And then the last is around prevention and what it looks like to help accelerate prevention.

So if I take a step back and talk about the child safety ecosystem more broadly, we are, unfortunately, in a space where the amount of suspected and reported child sexual abuse material just continues to increasingly grow year over year. We're seeing millions of images and videos of suspected child sexual abuse reported to NCMEC annually.

When we talk about - I can't even say emerging trends anymore - because unfortunately it's something that's become a reality for so many children. But when we talk about trends like sexual extortion, in a 2024 report that Thorn and the National Center of Missing and Exploited Children released, we saw that NCMEC received, on average, 812 reports of sexual extortion per week. So these kinds of numbers really ought to trouble us. They really highlight the reality of what these frontline defenders are facing, but also what these children are facing as they encounter these types of abuses and harms.

When we talk about revictimization, shifting a little bit, the reality is that every time this kind of abuse material is resurfaced survivors have been very clear: they experience another trauma, another moment of revictimization. And this is also something that we should really take seriously when we think about what kind of interventions we want to prioritize and what kind of society we want to be building.

And then finally, when we talk about prevention, I'll shift a little bit now to talking about generative AI, if that's all right. Noting that that's something that we've been focusing on quite a bit. So when it comes to that concept of new and emerging threats and what we can do to prevent them, I think we're seeing on this baseline of existing tremendous abuse, we are seeing even more growth. And it matches some of those verticals that we've already discussed.

So if we talk about the misuse of generative AI we are seeing broadly shared and open-source models used to produce abuse material. This is both material that can be produced at scale and also is photorealistic. And so if we think about that haystack of content that frontline defenders have to through in order to be able to find the children that may be in an active abuse scenario that are in need of that urgent need of recovery and care, that this is just growing that haystack.

I think NCMEC's most recent report indicated that they saw a 1,000% increase in reports of generative AI exploitative material of children. Now, this is still a small portion of the overall reports that are sent to NCMEC, less than half a percent. But that jump and that step change in growth is something that we should be paying attention to.

We know that when it comes to revictimization, offenders are fine tuning and optimizing some of these broadly shared and open-source models with existing abuse material to create bespoke, targeted content of particular victims.

And we know that, unfortunately, access to these kinds of technologies is only getting easier and easier. Where we see easily accessible nudifying apps and services that are built off of these same generative AI technologies are getting used to both create, to turn benign children's imagery into sexualized content that is then used to sexually extort or otherwise blackmail them.

That this is something that also is being used by minors to bully and harass their own peers. Where we see 1 in 10 minors report that they know of cases where their peers have used this kind of technology to create explicit imagery of other kids.

So really, regardless of what vertical you're looking at, the impact is real, and the scale, unfortunately, continues to grow.

KIMBERLY NEVALA: What do we hear from kids themselves, young adults and children, about this type of content and the exposure to these apps and activities?

REBECCA PORTNOFF: Yeah, there is something encouraging here when it comes to how children view this.

I think that from our research we see there's a pretty strong consensus amongst children that they know that this type of material - and specifically I'm referring to generative AI, nudification of children's benign material - that there is harm in that. That's something that is fairly self-evident to them.

On the downside, they also report that the mechanisms and the access to use these types of tools is just incredibly easy. That they can go on to any major search platform, any kind of app store provider, and very easily find tutorials and technologies that they can then use to create this kind of material.

KIMBERLY NEVALA: So you've touched on a lot of elements here. And so I'd like to step through those each in a little bit more detail. Now, you identified a number of issues in that rather eye-opening and appalling description of the level of harms that are happening.

So how do you-- folks, I think, are probably most familiar with the idea of CSAM (or child sexual abuse materials) and, to some extent, probably sextortion because we see that trending in the news. But when you think about situating these issues related to child exploitation and CSAM in the broader context of online child safety, where do they kind of fall on that spectrum, and what do we need to be aware of, broadly?

REBECCA PORTNOFF: I think when it comes to where it sits in the broader spectrum of child safety, the reality is that we are still mostly in reactive mode.

Where, even if we're quick to react, even when we're lucky enough to be quick to react, that it's still mostly in reactive mode. Where you see a new harm start to emerge and then you respond with appropriate detection mechanisms. Or by making pathways for user reporting or otherwise helping to ensure that victims and survivors have the right resources in place to support them.

But these are all inherently reactive measures and that when it comes to investment and prioritization of prevention solutions, we still continue to be pretty far behind.

KIMBERLY NEVALA: I think you've referred to this as, well, as sort of a whack-a-mole problem with that view. And something else you said was interesting; you've educated me on this area - even in some of the short conversations we've had.

Which is that in the generative AI space, it would be - for folks who think about this, most of us probably try not to, which is perhaps part of the problem as well - be reasonable to expect that there should be a really clear, we would hope, bright red line, ethically, morally, socially. An understanding that training on explicit materials and on CSAM training these models, if the models are exposed to it, then it would obviously allow you to generate that imagery. Hopefully, we would think that would be just a hard-line pass. But as experiences and things in the news like with Stable Diffusion - I think it was model 1.5 or something – show, we know that that's not true. So we know that there is this type of content and imagery in the training models for very available products.

But one thing I don't think I had realized and this will tie into - I want to talk about your safety by design initiative and some of the solution, the open solution spaces, that we need to look at - is that it isn't necessarily true that you have to have explicit material in order to then generate or create explicit material.

Can you talk a little bit about that? Because I think that’s likely a - or maybe it's just me - but a misunderstanding a lot of us have. And I think it's important because it widens the threat landscape quite a bit, I think.

REBECCA PORTNOFF: It does. And I think that's the right way to put it, that widening of the threat landscape.

So a model at its core is a function of the training data that was used to build it and the training paradigm or the training strategy that was used to build it. So it's these two things together that essentially make up what a model is going to be.

And so if we focus on the training data aspect of it, there's two known pathways, currently, for a model's capabilities to produce AI-generated child sexual abuse material. The first is the one that you've already referenced: that there's enough CSAM in the training data that the model has generalized and learned this concept and is able to reliably reproduce it.

And then the other is something called a compositional generalization: where a model may be able to combine unrelated concepts and basically put them together to learn a new concept. So in this particular scenario, that would be if there are benign depictions of children in the training data, and there are also adult sexual content or adult sexual acts that are depicted in the training data. Then the model may be able to then combine these concepts to learn the concept of AI-generated child sexual abuse material.

KIMBERLY NEVALA: And is just the plain old fact that generative AI systems in particular are so publicly and easily available exasperating the rise of this material? You said that today it's still a small percentage, but it seems to be growing rapidly.

REBECCA PORTNOFF: Yeah, I think there's a couple of things to unpack here.

First, I'm inherently a solutions-oriented person. So I got to say, that said, regardless of how you deploy or how you release your model, I definitely see many opportunities for the safe and responsible deployment of it. I don't think you need to just do any particular type of deployment and then feel that your hands are tied. There's nothing I can do to keep this safe. I don't believe that at all. But I will say that the risks that show up, depending on how you're deploying your model, are definitely going to be different. And how you address those risks are definitely going to need to be different.

And then separately from that, there's the unfortunate reality that there are some models and some services that were explicitly built to produce harmful material. And so we talked earlier about models that have been fine-tuned on CSAM. Where the whole point of it is to produce this bespoke AI CSAM. You have nudifying apps and services where the whole point of it is to be able to nudify somebody's benign content. And in those circumstances, access to those kinds of models and services, yeah, it's a lot easier than it has to be.

KIMBERLY NEVALA: I find it particularly chilling when you talk about this information being personalized.

And again, it's easy to get caught on just that element of CSAM. But we're also talking about some of the new abuse pathways and ways that don't necessarily have to do with visual content, either. Is that correct? We're really personalizing outreach and mechanisms through social media, through using generative AI, using all of these tools together to contact and torment, in my opinion, kids and youth as well.

REBECCA PORTNOFF: Yeah. If I'm getting the number right, I think it's 50% of law enforcement have reported encountering AI-generated CSAM used for the online grooming of minors.

And so there's both data modalities outside of imagery and videos that can create pathways for harm. And then there's the use of this kind of material to create new and additional pathways for harm.

So I would say, unfortunately, yes, it's all of those things.

KIMBERLY NEVALA: Now, you, at Thorn, have a Safety by Design initiative. And you mentioned that you are very much focused on solutions and starting to hack at this problem by whatever means necessary.

So tell us a little bit about the Safety by Design initiative from Thorn. What are the key pillars of that initiative?

REBECCA PORTNOFF: Yeah, happy to. So we publicly launched in the spring of 2024.

The goal of the initiative was to ensure that generative AI is built, developed, deployed, and maintained in such a way that it doesn't further the sexual exploitation and abuse of children.

And so we gathered together several industry players with the explicit goal of having them define, align on, and then commit to a set of safety by design principles and, in some cases, more granular mitigations.

Where the goal of these principles is, first, to make it such that generative AI models are less capable of producing this kind of abuse material. So this ties to some of what we were talking about. About the role of the training data and what you could potentially do there to intervene with some of those capabilities.

The second goal is to where that material still gets produced, where you have a dedicated offender, where you have an older model that has been released to the ecosystem, however the case may be, where that material still gets produced, it's more reliably detected.

And then finally, that the underlying models, apps, and services that are being used to create this kind of abuse material - and so now, harkening back to those nudifying apps and those optimized models optimized for CSAM generation - that their distribution is limited.

So it really is a pretty broad set of principles. And it was intended to be that way. It was purposefully written that way because the reality is that generative AI is already sort of full steam ahead, getting integrated into our lives in a lot of different ways. And so the ideal here was to reflect that by having as comprehensive a set of principles as possible. While noting that everything requires iteration and everything is going to need to be expanded to ensure that the promise of those principles are met.

KIMBERLY NEVALA: And I suppose any type of governance that we do, the principles are always step one. Because putting principles into practice tends to be where either some ethics washing happens and/or where the rubber hits the road and where the difficulty comes into play.

So tell me today, in your assessment, where are different organizations showing up on the spectrum and through the lens of, for instance, the Safety by Design initiative and those pillars?

REBECCA PORTNOFF: Yeah, well, I'll start by saying, I couldn't agree with you more.

And I think that's reflected in our strategy here for the Safety by Design initiative. So we have three different ways that we're approaching that.

The first is through our progress reports, so every company that signed on to these commitments and publicly stated that they were going to be following through also were committing to transparently sharing the insights, the progress that they've been making. The goal there is to ensure that the public has visibility into both the progress and the gaps. That where we're able to we're providing metrics that give evidence to these two different things. And that's a way to help move the needle when it comes to ensuring that that transparency and that visibility is surfaced to the right people.

The second is around our collaboration with standard-setting institutions. This is where the nerd in me is just super excited. I love this line of work. The idea here is that if we look at the line of sight between consensus and industry and accountability, I do think that standards have a big role to play in it. That standards help establish that global consensus, that scientific consensus that then opens the door to other kinds of auditing or other kinds of verification of your commitments, of meeting these types of standards.

That's the line of sight that we see in the computer security community when it comes to standards and then ISA compliance and all that kind of great verification of following through on these best practices. So we do work collaborating with these kinds of standard-setting institutions to help establish those with that sort of longer line of sight.

And then the last is helping to inform policy makers with interventions that are both impactful from an issue perspective and also technically sound. Because, at the end of the day, there are always going to be companies that don't bother with best practices and with standards and with commitments and need a little bit more of a nudge in order to, maybe a lot more of a nudge, in order to take action.

KIMBERLY NEVALA: And I want to come back to some of the solution perspectives here.

But part of what you're talking about, the interesting thing about standards if they become openly accepted and expected is they can help to drive a norm. Even for companies and persons who are not explicitly trying to go out and harm children but may not take action outside of that except for we really do need to do that.

But it does require, as you said, those being enforced. But not even just being enforced but people being held to account by consumers, by folks, third parties. So that's always an interesting conversation because people will say, great, Kimberly, we have principles. We have standards. And what does it matter?

But, well, it matters when you start to buy based on those standards. Maybe it's the eco rating on your appliances and all that. And although the scale of issues here are not, nowhere near, comparable, again, when you start to look for that I think that's helpful.

I would love to be able to go in and look at an application and see that it is certified in a way that is safe. Even if the purpose of the application is not - or the audience, the consumer base is not - intended to be children. So that's also something interesting there.

And I guess when you're looking at the different organizations, what is the profile of organizations that have been engaging with you in the initiative? Are these all consumer-facing organizations? Are these organizations that are developing applications targeted at adults, some at children? Or is it a broad swath?

REBECCA PORTNOFF: It's a pretty broad swath, although, I would say, definitely consumer application focused, I think, just because of the reality of the space that we're currently in.

And when we went about brainstorming on which types of industry players we would need to pull into this, the obvious start was those who are actually building the generative AI models. Those AI developers. But it quickly became clear over the course of more conversations and some of the early meetings with the working group that helped establish these principles and mitigations that we would also need to pull in some of the more traditional tech companies. Like the ones - maybe traditional is not the right word - but the search engines and the social media platforms. Those who really have played a very significant role in the distribution of media in general, and now synthetic media, as well as the actual technologies themselves.

So it's intentionally, but with some iteration, a pretty broad stretch of stakeholders.

KIMBERLY NEVALA: Sometimes when we talk about the responsible innovation space, or even ethical tech, we - or maybe I - think about it as a bit of a three-legged stool. Where we've got commitments, voluntary commitments, from organizations and companies. There's the regulatory piece of that. And then there's things like academia that are driving research.

Do you see those three pillars and how are they working together today? Are they required? Or are there any weaknesses in any of those supports, if you will?

REBECCA PORTNOFF: Yeah, it's a great question. I would say, absolutely. I think all three of those are necessary and that, in general, this type of problem is a whole of society problem. Where we really need to have all of these different players coming in and investing in, doing their part to drive that impact.

I know, coming from academia myself, I have a soft spot in my heart for the kind of work that gets done there and its role to play here. And I think, in particular, when it comes to pushing the needle on some of the new technology, research, and innovation that's necessary from that safety perspective, that I see a lot of interest coming from academia on that point.

And that's something that's encouraging to me because as far as incentives goes, I think it can be hard sometimes to have those incentives in place at a small startup that is sort of focused on making sure that it gets its bottom line. That can be hard to then have headspace to invest in the kind of technical innovation that really is necessary when it comes to following through on some of these different, more complicated threats. So happy to unpack particular examples, but I'd say that one in particular is something that's on my mind to want to continue to see that grow out.

KIMBERLY NEVALA: Yeah, I think an example would probably be helpful. And I'd also be interested in how you view the utility and usage in the current state of affairs, I guess, between organizations trying to do in-house type evaluations and third party assessors. And where are the gaps in that approach or even between those two bodies of folks, I suppose?

REBECCA PORTNOFF: Yeah, it's a great question.

So first, just to quickly share an example of the kind of research I'm thinking of. So right now, you could have a good faith open-source developer follow all of the Safety by Design principles and mitigations to a T. And it would still, unfortunately, be really easy, as soon as that is released, for a technically savvy offender downstream to just undo those interventions by fine tuning or otherwise optimizing the model in such a way that it can break through these barriers.

So the technical research that's necessary in order to build models that are more robust to this kind of downstream adversarial optimization, that's something that I'm definitely keeping a close eye on. Because it's the kind of breakthrough that ultimately will be necessary in order to be able to address some of these risks.

To your question on the efficacy or the trade-offs between inside and outside assessments: at the end of the day, it's probably not a good idea for people to grade their own homework. I think that's a pretty universal rule of thumb. With that said, I do think that having in-house expertise on these problems can be really valuable. It's a tighter loop when it comes to being able to more quickly respond to some of the offenses or threats that you might see, is if you have that kind of dedicated unit inside of your organization. And so there's definitely value to that for sure.

But I think that doesn't undercut the reality that when it comes to transparency and making sure that we're all confident about what it is that these assessments produce, there is absolutely a role and a need for that kind of third-party external assessment.

KIMBERLY NEVALA: So it sounds like this is an area similar to some of the other areas we've talked about. Where this, and AI in general, benchmarking and assessment, ethical or otherwise, it's a work in progress.

Now, I want to go back to Safety by Design and what we need to do to start to take a more prevention or a proactive stance in this area. When we were talking before, you had made this statement and it stuck with me. In fact, I wrote it down.

You said, for Safety by Design to have impact, it needs to be scalable. However, there are some particular issues or open problems that need to be addressed. And so there are, even for folks who may want to be trying to evaluate and assess these things proactively, there are barriers in the way.

So can you enumerate some of, first, just enumerate some of the open problems to be actioned on. And then I'd like to talk through the specifics of each of those. About what the particular issues are and what's going to need to be done or actioned to solve them.

REBECCA PORTNOFF: Yeah, absolutely.

So I think probably the first one that comes to mind is scalable model assessments. So if we're thinking about the use case of a platform that is hosting models, like third party models, that it's making available to users to be able to explore and innovate and create all sorts of cool things with, the pace at which these kinds of models are hosted and uploaded, it's tremendously fast. There's thousands of new models, sometimes daily.

It's something that inherently is not the type of problem that, from an assessment perspective, that you can manually address. You just can't go about assessing every single model manually with the prompt output strategy that is typically still the type of assessments that occur in these kinds of spaces.

And so when I think about the need for a scalable strategy here, what that looks like is, for example, what would it look like to have solutions where you can automatically extract from the latent space or, in my wildest dreams, from the model weights of the model itself whether or not that model was trained on CSAM or otherwise has CSAM capabilities.

And so this type of research is not just, it's blue sky, but it's not just, in my mind, imagination. There are absolutely really smart folks who are exploring these kinds of solutions. And they need to, because ultimately, in order to be able to meet the pace at which folks are choosing to innovate, we will need to have these kinds of scalable solutions.

I think another example of a challenge that we've heard from companies. And now this is less about the scalability and more tying back to your three-legged stool analogy and the role that government can play here. When it comes to that kind of assessment or red teaming of these models, in the United States, it's still, it is illegal to prompt a generative AI model, explicitly prompt it to create child abuse material. Whether or not you are doing that because you're an offender who's trying to make that material or because you are a developer who is trying to assess your model's capabilities.

So now, that's less about scalability but it is more about the interplay between that socio-technology. Of what does it look like to be able to still conduct those kinds of assessments in a way that's effective without being on the wrong side of the law? That's a question that definitely is worth addressing.

And then I guess the last one I'll raise is the training data sets that are used to build these generative AI models are tremendously enormous. So what does it look like to be able to take action on some of the fundamental mitigations like detect, remove, and report child sexual abuse material from your training data sets? In practice, what does that look like if your training set is super huge?

And that's a very practical engineering question. Maybe the way I framed it is not particularly engineering friendly. But it is a very practical question that gets at the heart of, what's a scalable solution for detecting, removing, and reporting this kind of abuse material?

KIMBERLY NEVALA: And it does seem that until that becomes non-negotiable, until it becomes a requirement-- because it is a massive problem, yes. But it doesn't feel to me like it should be an unsolvable problem, if, in fact, we are to believe even a small percentage of the hype about what the system's own capabilities are.

So is it your sense that there's just the need here to, I don't know if this is-- I don't even know if it's a lack of care, but a lack of will, and time, and attention. Just that pressure to make it a must-do versus a nice-to-have.

REBECCA PORTNOFF: It's definitely solvable.

And I think something that I've personally been encouraged to see in some of the reports from the committed companies is how many of them have started to take action on this particular mitigation; of detecting, removing, and reporting this kind of abuse material from their training data sets.

But it does require either the in-house knowledge and access to build these kinds of detection - scalable detection - technologies yourself. Or the budget to work with a third-party vendor. Or the relationships to help establish and get access to some of the industry hash lists that are used to detect known CSAM.
There are things that need to be addressed in order to get there. But none of them are impossible. And in fact, many platforms are using those exact same strategies today in other settings.

And so I guess I'll another word of hope I'll bring is it has been encouraging to see the faster learning curve here. We think about how long it took social media to get to where it's at. And many people are understandably not all that happy with where it's at right now. But if you think about the length of that learning curve, if I compare it to the learning curve with generative AI, I do feel encouraged with that.

But to your point, it requires investment, it requires prioritization, and then it requires access. And the access is not impossible. The investment is not impossible. You have got to prioritize the work if you're going to get to that place.

KIMBERLY NEVALA: Yeah. It was an interesting point you made about the legality question as well.

Because does that then indicate that today really the only folks that could legally be actively testing and trying to validate whether these systems are or are not capable of doing this or are capable of being modified such that they produce this kind of material would be law enforcement? And are they actually adequately resourced, staffed, and trained to be able to do that at any scale?

REBECCA PORTNOFF: Yeah, it's a good question.

So the strategy that we've recommended and that we've seen some industry players adopt to use as a proxy for this direct assessment actually ties back to that idea of compositional generalization.

So we do know that there are some industry players who are red teaming their models by, first, assessing if they can - I don't know if first - but by assessing whether or not they can generate benign depictions of children and then independently assessing if that same model can generate adult sexual content.
And if your model can do both of those things, then for some industry players, that's sufficient to assume that it can create material that you don't want it to and needs to have some mitigations and rework done.

So again, none of these problems to me are impossible. They do require creativity and investment and collaboration but it’s not impossible to do even in the setting where guardrails are still pretty clear for that direct assessment, like for the status of the legality of that direct assessment.

KIMBERLY NEVALA: Are there other perspectives or things that we need to think about as we then try to further develop these kinds of benchmarks and standards that we haven't talked about yet? That are kind of intrinsically linked to this assumption that most testing or validation has to be, or should be, based on this sort of prompt output framework?

REBECCA PORTNOFF: I think one thing that I would offer is right now, there's a good chunk of research that has been done on various unlearning strategies. Where you have a model and you are able to find that concept that you don't want it to have and then have it unlearn it. And I think we are seeing that, depending on the strategy, some of these are more or less robust.

And so, I guess in my mind, I return to that concept of model and training strategy. Excuse me, data and training strategy. So if I think about training strategy, those types of solutions that prioritize the reality that these models are going to get tweaked and messed with downstream. What does it look like to still build them in a robust way? That's the line of sight I'd want to see.

I feel like I might have just answered the question I wanted to answer, instead of the one you asked, so let me check.

KIMBERLY NEVALA: No, I actually like that because my actual next question for you was going to be, because you've been doing this work and because you go and speak a lot about this, what are the questions that we aren't asking or the perspectives or considerations that aren't brought up?

Because those of us who aren't in this work day to day, or even over the long term, we don't think to ask them. And we really, really should. So this is your open invite to pose the questions that you wish I would, or that I should, for everyone's edification. So, yeah.

REBECCA PORTNOFF: Yeah, no, thank you. I'll walk through that open door.

KIMBERLY NEVALA: Yes, please.

REBECCA PORTNOFF: I think one conversation I'd love to see more of it goes back to that three-legged stool. I love that metaphor. I'm going to use it all the time now.

But it’s the regulatory stool in particular that we still see, in my opinion, a reactive approach. Even at that setting where I see regulation that is more about the synthetic media that is produced. And you need that kind of regulation. You need to have clarity on what is and is not a balance. You need a shared understanding of what the harms are, that kind of education and awareness. And then enforcement is really important.
But it doesn't get at the heart of this, which is how that tech is built in the first place. And so to me, it misses out on the real promise of prevention which is we can do it better from the beginning. We can get it so much better from the beginning. We don't need to wait until the harm happens and then respond. Or at least we don't need to just do that. And that, to me, is reflected in, or the lack of that conversation, is something I'd love to see corrected in that regulatory conversation.

KIMBERLY NEVALA: Well, there's a level of, actually all types, of will there. Political will, legal will, and social will. To say that if you are saying to us that because of the way the system was developed initially we cannot go back and retrospectively, whether it's through forgetting or all these other mechanisms, remediate the system then you are just not allowed. You will have to pull the system.

And that is a red-- a line that we have not crossed. It's a conversation that comes up with people, say in the context of copyright and licensing and things like that, which are important. And I think they have broader ramifications than just what folks tend to want to look at as a legal challenge.

But this is an area where it seems to me that there's just no justification. We may have to actually work up the will along all of those elements to say no. If you were not able to take enough care or you are not able to explicitly and discretely verify to the extents of our actual solution knowledge and abilities to remediate it, then you are not allowed. Yes, you will have to pull it and so be it.

But that is a level of will and awareness perhaps that we haven't mustered in other areas and even in social media prior to this. But I don't know what it's going to take us at this point to get there. But I keep hoping we will, I guess.

REBECCA PORTNOFF: It's a challenge, for sure.

And I think my mind goes back to the example you shared earlier on what it looks like to have products that transparently share the safety interventions that are put in place, the assessments that were made. That there's regulatory bodies that say this can or cannot be released based off of XYZ assessment.

As much as machine learning and AI certainly has been around for quite some time - it's a decades long type of field - this particular manifestation with generative AI is still pretty early days. And so I'll lean into my hope and say, yes, I hope that we get to that place. I think that we have seen it done for other types of products and distribution mechanisms. And so I want to see us get there for this too.

KIMBERLY NEVALA: And this then begs - I don't know if it begs the question - but it brings a question up. A lot of the things we've been talking about today are very specific to the generative AI space in particular.

But there are, as we talked way up front, a whole spectrum of harms that are, even AI mediated or generated harms, that are not generative AI. Just other digital harms.

How do we make sure that we balance this so that in the moment that we're in, where all eyes are on generative AI - and I would love to hope that this gets a little bit more into that preventative so we cut it off because it's already bad enough, before it gets that much more egregiously worse - but where else are we at danger?

What are the other things that we need to keep an eye on to make sure that these things that we're doing work in the broader ecosystem as well?

REBECCA PORTNOFF: Yeah, that's a great question.

I think there's always a need for more awareness and education in general. So I would say, we haven't had a chance to highlight that a ton in this conversation. I'll take a moment to give a plug for it. I think when it comes to preventing harm, having folks have a shared understanding of what those harms are, how they manifest, what you can do as a person and as a citizen to try to prevent those harms, that kind of awareness and education is really important.

And then in my dream world scenario, I would love to see us in a world where folks who are the best and brightest, who are innovating, who are creating these new technologies, who are pushing forward these breakthroughs. That they also are aware and so that when they're building this kind of potential for misuse is front of mind.

It's not something that you end up chatting about because you're pulled into a coalition two years later after you've released it. It's something that they know about because maybe there was an ethics class that they were required to take as a computer scientist, as an undergrad. Where they talked through all of known types of misuse and harms that exist and so that it just becomes a little bit more of the vernacular.

So that accountability really stops being such a bad word and just becomes something that builders say, yeah, I want to be a part of this accountability. I want to be a part of making sure that what I'm building doesn't get misused in this way. I'm just excited about innovating in that space as I am about other spaces. This is my pie in the sky dream. I'd love to see us get to that place.

KIMBERLY NEVALA: Yeah, absolutely. And how do we-- or how do you think about, I guess, is a better, more fair question. You talked earlier about kids themselves will say that they know that this type of content is harmful. That they don't like being exposed to it but it comes up whether they want it or not. Or they can find it really easily. And they are very young. And those brains are malleable. And peer pressure, group think, all of those things happen. And then a lot of those behaviors get instantiated. I think I had shared with you, someone said to me, well, isn't it at least better? We've been talking about this with AI companions and things and so not in the context of child sexual abuse and exploitation, but well, at least it's virtual. That is a really, really thin conversation. But also what we're doing online and what we are exposing kids to sets the basis for what they think are social norms and how we treat each other and what's appropriate and what's not appropriate.

And I do worry a little bit that, not you and your folks, but in some of the broader outside that there's a focus, yes, on bringing kids into the conversation. Which I think is fabulous and needs to happen but that we need to also be careful that the accountability and the responsibility isn't on the youth. Who don't really have the tools, the experience, and the wherewithal, I suppose, to understand those implications, short and long term. Particularly if they become the perpetuators of some of this violence and abuse, just because those apps have fallen into their hands, and they're working through them.

REBECCA PORTNOFF: Yeah, no. So I cannot claim to be any kind of expert or knowledgeable in anything about behavioral psychology. I am not.

What I can say is that when it comes to this question of who's responsible? Who's responsible to prevent this? Who's responsible to tackle this? Who's responsible to stop this? My answer is fairly consistently - I hope it's consistent by all of us - that responsibility is not about blame here.

That it's really easy to shift, very understandably, to want to shift to a blame conversation. And I think, not to go off on a tangent, I think there's space for, when we talk about offenders, that there's space to open up that conversation around blame.

But if we talk about who's responsible for stopping it, I think the answer has to be all of us. That all of us are responsible. And the level of responsibility that you have is not tied to how much harm you've done necessarily, but how much power you have. What is your position in this society? What is your position in this world? How much are you able to enact and make happen?

And that, to me, is a much more compelling way to think of responsibility and a more solutions-focused way of thinking about responsibility.

KIMBERLY NEVALA: Yeah, I think that's a very powerful framing. I'll refer back to that also in other situations. I think that's really well stated, particularly in this area. So as we wrap up, any final words you'd like to leave with the audience or perspectives on what comes next and how we can all really start to show up so children don't, in this way?

REBECCA PORTNOFF: I think the one word I'd want to end on is this is a complex challenge. This is a complex challenge. It requires a lot of effort to navigate, but it is not impossible to prevent this kind of misuse. And I think it's easy to want to look at something and throw up your hands and say, I can't do it. And so I'm not going to think about it. I'm not going to bother. There's too many technical challenges, too many policy challenges.

And, yes, those challenges are real. They need to be overcome. But it's not impossible to address. And that's something that I really want to continue to bring forward in these conversations, is that there are so many practical and tactical things that all of us, and especially these tech companies, can be doing to prevent this kind of misuse.

KIMBERLY NEVALA: Hopefully we see more of these conversations and that type of awareness as well. Because this is one of those topics that's really hard for people. It’s hard to really look at straight on, with open eyes. And therefore, I think we tend to want to avoid it because there's a level of discomfort, just with the topic as well, and shame, and all those things. So I think that's a great final call to action for everyone.

And really, again, I just want to thank you so much. Not only for sharing your insights today but for continuing to really move that work forward for as long as you have and into the future here.

REBECCA PORTNOFF: Well, thank you. I appreciate the kind words.

KIMBERLY NEVALA: Awesome. So we will put links to Rebecca's work and that of Thorn in the show notes. I just really encourage us all to get on board and help really promote and support and drive Safety by Design for children, and in fact, for all of us.

And, if you'd like to continue learning from thinkers, doers, and advocates such as Rebecca, you can subscribe to Pondering AI now. You can find us wherever you listen to podcasts and also on YouTube.