Transcript
00:00:04 Veena McCoole
Hello and welcome to the Human Interface, brought to you by the Oxford Internet Institute at the University of Oxford.
00:00:10 Veena McCoole
This is a podcast about how developments in technology impact our work and life, as told through the research and insights of a brilliant roster of experts and industry practitioners from the university and beyond.
00:00:22 Veena McCoole
I'm your host, Veena McCoole.
00:00:24 Veena McCoole
Today's episode explores the evolving role of human oversight of AI systems in the workplace and beyond.
00:00:31 Veena McCoole
With so much of our professional lives being permeated by AI-enabled tools, where do checks and balances come in, and what might they look like?
00:00:39 Veena McCoole
Dr.
00:00:39 Veena McCoole
Johan Laux is a departmental research lecturer at the OII who leads the Emerging Laws of Oversight project, in which he investigates how human oversight of AI systems can be implemented effectively.
00:00:52 Veena McCoole
Ali Shah is the Global Managing Director of Accenture's Responsible AI Practice, a board member at the Ada Lovelace Institute, and is focused on AI governance and safety.
00:01:03 Veena McCoole
Ali's previous roles include positions at the UK Information Commissioner's Office and the BBC.
00:01:10 Veena McCoole
Maxine Setiawan is an associate manager at Accenture's Global Responsible AI team and a data scientist embedding trust, accountability, and fairness into the development and deployment of AI systems.
00:01:23 Veena McCoole
She's also a graduate of the OII's MSC program.
00:01:26 Veena McCoole
Welcome, Johan, Ali, and Maxine.
00:01:28 Ali Shah
Thank you.
00:01:29 Ali Shah
Hey, really nice to be here.
00:01:31 Veena McCoole
Thanks for joining us.
00:01:32 Veena McCoole
So I'd love to kind of start with the basics here.
00:01:34 Veena McCoole
What even is human oversight?
00:01:37 Veena McCoole
Johan, let's start with you.
00:01:38 Johann Laux
Human oversight is at the same time very old and very new.
00:01:43 Johann Laux
So it's about how humans oversee automated systems such as AI during their runtime.
00:01:49 Johann Laux
So when the system is in use,
00:01:51 Johann Laux
And one of the major reasons for having it is that we want humans to mitigate risks that AI systems may pose, because AI systems are not perfect.
00:02:00 Johann Laux
They might make errors.
00:02:01 Johann Laux
For example, if you think of an AI system that looks at lung scans and whether it shows a cancerous cell or not, it might get it wrong.
00:02:09 Johann Laux
So you want to have a human expert, like a trained radiologist or oncologist, to also check on the recommendation of the AI system.
00:02:16 Johann Laux
And at the same time, AI systems can also be biased, right?
00:02:20 Johann Laux
If you want to predict whether someone
00:02:21 Johann Laux
is likely to commit a crime in the future, then the AI system might pick up on the fact that this person lives in a neighborhood that is very high crime or very low crime that might affect the risk level that this person got attached with their system.
00:02:35 Johann Laux
So we might find that unfair on an individual perspective.
00:02:37 Johann Laux
So again, we want humans to look over the shoulder of the AI system and monitor its behavior and intervene.
00:02:44 Johann Laux
And the way that the term human oversight has really made its
00:02:50 Johann Laux
entry into the AI governance field, in which I guess we're all in, is prominently the AI Act of the European Union.
00:02:57 Johann Laux
So there's one particular article which claims that for certain very high-risk AI systems, humans are supposed to mitigate risks to safety, health, and fundamental rights.
00:03:07 Johann Laux
So it's quite broad.
00:03:08 Johann Laux
It's quite a lot what humans are supposed to do and how they're supposed to monitor AI systems.
00:03:13 Johann Laux
And I think, so last thought here is that really think of humans sometimes as the last line of defense
00:03:20 Johann Laux
in our risk mitigation system or in a risk management system.
00:03:24 Johann Laux
And that is both a very challenging and very important task.
00:03:28 Johann Laux
And I guess that's what we're here to, talk about how humans can do this.
00:03:32 Ali Shah
I love that opening, Johan, because I think it really puts the role of human oversight as why it's so important at the heart of the conversation that we're going to have.
00:03:40 Ali Shah
Maxine and I work for Accenture.
00:03:41 Ali Shah
It's a technology professional services firm.
00:03:44 Ali Shah
We go and help organizations, whether they're public sector, private sector,
00:03:48 Ali Shah
make sense of data and AI, figure out how to use it.
00:03:50 Ali Shah
And the work that Maxine and I do is to help them figure out how to use it responsibly.
00:03:55 Ali Shah
Part of that is human oversight, part of that is driven by regulation, but other factors come in.
00:04:00 Ali Shah
The notion that we should have somebody accountable for what is being done to us, you know, whether that's delivering a product or a service or the app that you have on your phone or if you're going into a bank and you're applying for a loan, that there's somebody there who has helped make that decision, I think is so central
00:04:15 Ali Shah
to the social covenant we have with each other around what it means to rely on somebody else and rely on a business or a service or an individual.
00:04:24 Ali Shah
Businesses want to use AI because it's an amazing technology.
00:04:27 Ali Shah
They can do amazing things with it, but they also want to have confidence.
00:04:31 Ali Shah
And the way they want to have confidence has to include some aspect of humans having a view around, is this working the way we intended?
00:04:38 Ali Shah
Sometimes that's to really deliver the amazing productivity or the creative use case.
00:04:44 Ali Shah
sometimes that's to understand, will this go wrong?
00:04:46 Ali Shah
And if so, how can we avoid it doing some harmful things or things that we're not expecting?
00:04:52 Ali Shah
And I think the role that the human plays in that process, I think, is fundamentally important, but also one that's evolving.
00:04:59 Ali Shah
The pace of AI technology is changing so quickly and evolving so quickly that it's forcing all of us to think about how do you make human oversight work in practice in all of these different ways that AI is entering into our lives.
00:05:10 Maxine Setiawan
Coming back to the questions of why does it matter, I think the concept of human in the loop and human oversight comes into the picture because of the inherent risk of AI.
00:05:20 Maxine Setiawan
Different to other technologies, AI is not a deterministic platform.
00:05:24 Maxine Setiawan
It can create other things that you didn't expect it to be.
00:05:28 Maxine Setiawan
So that's why human in the loop is so important or human oversight.
00:05:31 Maxine Setiawan
And even the actual words of human in the loop is interesting because why is it the loop?
00:05:39 Maxine Setiawan
in the technological world, the loop means like, these agile iterative mindset of AI developments and AI use, and where does human come into that?
00:05:48 Johann Laux
If I might just add to what both of you said, because I think this mandate for human oversight, right, mitigate risks and make AI safe, with that comes this idea of trustworthiness, right?
00:06:00 Johann Laux
And if you look into the research, if you look into empirical studies, if you tell people that there's
00:06:06 Johann Laux
a human in the loop, then they actually tend to trust the AI system more.
00:06:09 Johann Laux
Which now, from a business side or a government that uses AI, this is kind of what you want if you implement the system, if you want people to use it.
00:06:18 Johann Laux
One good predictor for use and adoption rate is actually trust.
00:06:21 Johann Laux
So if people trust it, they use it more.
00:06:24 Johann Laux
At the same time, this is why it makes us, I guess, here connect between your work and the research.
00:06:31 Johann Laux
If the human is not effective at overseeing AI systems, then that trust is misplaced.
00:06:36 Johann Laux
right?
00:06:37 Johann Laux
So this is also where questions of accountability come in, where if you roll out AI systems without any accountability, then the risk is very high that the AI system will cause some harm.
00:06:47 Johann Laux
And again, it might be very hard to rectify that if no one's accountable.
00:06:51 Johann Laux
At the same time, there is an interesting concept or an interesting paper titled by Madeline Elish, who calls the humans in the loop kind of like the moral crumple zone.
00:07:02 Johann Laux
So in a car, the crumple zone is the part of the car that gets
00:07:06 Johann Laux
crumpled in an accident to protect the human from harm.
00:07:09 Johann Laux
Now, in a moral crumple zone, the idea is that the human takes the blame and protects the technology from harm.
00:07:15 Johann Laux
And there are certain cases, and it's not just with AI, it's generally also in aviation, right?
00:07:20 Johann Laux
Recently, there were two big crashes of a Boeing that came down.
00:07:24 Johann Laux
And first reaction was to blame the pilots until we all understood that it was an automated system.
00:07:30 Johann Laux
that kept on pushing the nose of the plane down, the pilots had no chance.
00:07:33 Johann Laux
So with AI, there's also a risk that the design of the system is such that there is a false or a mistake somewhere, an error somewhere, it's not working the way it's supposed to.
00:07:42 Johann Laux
And you mentioned it, Maxine, earlier about the loop, right?
00:07:46 Johann Laux
That it might be hard to find the iteration where things went wrong, but now you have a human and might be a frontline worker, right, that then absorbs, like in a crumple zone, the blame.
00:07:58 Johann Laux
So there's a lot of questions
00:08:00 Johann Laux
to untangle, I guess, in that topic of human oversight about who is held accountable for what exactly.
00:08:06 Ali Shah
We start in our advice, and the thing that Maxine and I do every single day when we speak with companies and organizations is, let's really look at what is it you're trying to achieve out of this product or service.
00:08:17 Ali Shah
And before we even talk about the technology, what are you trying to actually deliver as a service?
00:08:21 Ali Shah
If you are starting to think about AI, is it the right thing for you to use?
00:08:25 Ali Shah
How might it work effectively?
00:08:26 Ali Shah
Do you have a good sense of the objectives of this system and the way that it will deliver the benefits to your customers or individuals or maybe to your employees or others?
00:08:34 Ali Shah
Can you describe that?
00:08:36 Ali Shah
And then I think the thing that really helps you understand what the blueprint should become for governance and oversight is how might this break?
00:08:42 Ali Shah
How might this fail?
00:08:44 Ali Shah
What are the ways that if this went wrong, you would be impacted?
00:08:47 Ali Shah
How can we look at that and then try and help you understand where should humans be added to the loop in the way that Maxime was describing?
00:08:54 Ali Shah
Or what might you need to do to meet some of the regulatory requirements in the way Johan was describing?
00:08:59 Ali Shah
All of those things sit around the central question that businesses are asking us every single day, which is, there is no one type of AI that we are using.
00:09:08 Ali Shah
across the things that we do, there are different types of AI.
00:09:10 Ali Shah
They are entering our operations in different ways.
00:09:14 Ali Shah
The level of mix of humans plus machine is different depending on what we're trying to do.
00:09:18 Ali Shah
So what should we do in response to that?
00:09:20 Ali Shah
So no one blueprint, I think, can really solve this, but you kind of need to have a pattern library of blueprints that you can take out and really then adapt to your context.
00:09:29 Ali Shah
That I think is the most practical way that we've helped organizations.
00:09:33 Maxine Setiawan
There's no one-size-fits-all because of the way that AI has been evolving and will continue to evolve compared to the three years ago, for example, and what's the next three years will be looking like.
00:09:44 Maxine Setiawan
I think there's a lot of different approaches for different businesses to deploy AI, whether it's developing in-house, buying third-party AI, or a mix of both.
00:09:54 Maxine Setiawan
And that's why that combination of what works before and what will work in the future would change.
00:10:02 Ali Shah
Let's take an example of, say, a bank.
00:10:04 Ali Shah
Banks are really using the governance structures that I already have.
00:10:07 Ali Shah
They're thinking about model risk management, the three lines of defense mode.
00:10:10 Ali Shah
So for listeners who maybe are not familiar, this is really an established approach that banks have around how do they govern the use of models in the organization.
00:10:19 Ali Shah
But they're having to adapt to that for the type of AI technology that they are interested in using now, whether it's generative AI or Gentic.
00:10:26 Ali Shah
And so for them,
00:10:27 Ali Shah
Many of the things they are doing to evolve come from that original position of we have the MRM approach, we have the three lines of defense.
00:10:34 Ali Shah
We need to pick and choose and borrow from each of these different industry contexts.
00:10:39 Ali Shah
Actually, the lessons from finance and banking around model risk management and how you adapt that for AI can apply and do apply to the work that we do with companies in media or retail or telecommunications.
00:10:52 Ali Shah
But we also see the experience that organisations that have lots of consumer facing services really care about maybe the more normative or ethically driven questions around what will people feel about this?
00:11:04 Ali Shah
Will they trust us?
00:11:05 Ali Shah
What does it mean to be trustworthy?
00:11:07 Ali Shah
Like how can we prove that?
00:11:08 Ali Shah
We need to not just talk the talk, we need to walk the walk.
00:11:11 Ali Shah
How do you take those notions and turn them into more concrete steps?
00:11:15 Ali Shah
How do you build in the controls so that you can evidence that we have been acting appropriately?
00:11:19 Ali Shah
I think that is something that as an experience
00:11:22 Ali Shah
feeds back from industry and practitioners, back into those who are thinking about the bigger frameworks around what it means to govern.
00:11:29 Johann Laux
Yeah, and if I may add, like the political view on that.
00:11:33 Johann Laux
So imagine you're the regulator, you're a lawmaker, you want to, you said like, oh, like with the AI Act, now we have, we demand human oversight, right?
00:11:40 Johann Laux
The problem is that technology really changes exponentially fast, right?
00:11:46 Johann Laux
Versus institutions and laws, they're like more incrementally changing.
00:11:50 Johann Laux
So if you look at human oversight,
00:11:52 Johann Laux
site.
00:11:52 Johann Laux
So you have maybe a recommender system that recommends, for example, a certain treatment maybe in medicine, right?
00:12:01 Johann Laux
You still have a highly trained medical doctor checking in on that recommendation.
00:12:07 Johann Laux
So that makes the case both super important because it's a medical field, but at the same time, you already have a human in the loop who is super well trained in their domain.
00:12:16 Johann Laux
Then there might be other users of AI where we also have a human in the loop.
00:12:20 Johann Laux
Think of
00:12:21 Johann Laux
using ChatGPT, right?
00:12:23 Johann Laux
It only spits out certain texts that you may use or may not use.
00:12:26 Johann Laux
And depending on whether it's what kind of industry it is, there's still a human in the loop, maybe less specialized because it's not a field where you need a license like in medicine or in law to operate.
00:12:36 Johann Laux
There already cannot be one fits all, right?
00:12:40 Johann Laux
Because you have different humans you're actually dealing with.
00:12:42 Johann Laux
And you need to study how those humans behave.
00:12:45 Johann Laux
And now if you think of the technology moving forward with agentic AI, where AI systems now can pursue multiple step goals, so they don't just make a recommendation, they might actually themselves act upon that recommendation, then we might not have a human place in the loop, but then technically we place them on the loop, so it's a bit different, right?
00:13:04 Johann Laux
So we need to come up with all different, and I like the idea of the library that you mentioned earlier, with kind of like an entire toolbox.
00:13:12 Johann Laux
And investigating that toolbox is
00:13:15 Johann Laux
Interesting and at the same time really challenging because we see the development of the technology being so fast.
00:13:21 Johann Laux
And usually we call it like in political science, we call it the Collingridge dilemma or the pacing problem really, that you're always like behind.
00:13:29 Johann Laux
And if you look at what's happening in the EU now with their AI regulation, with the AI Act, it is in force, but the section on human oversight should have been in force next year.
00:13:40 Johann Laux
And now they want to delay it further because what they're lacking are standards.
00:13:45 Johann Laux
like technical standards, which are like the library that you mentioned, right, of actually, okay, I understand we need to mitigate risks.
00:13:51 Johann Laux
Yes, a human should do this, but how are we doing this?
00:13:54 Johann Laux
So we're lagging very much behind in this development.
00:13:58 Johann Laux
And it's not just an EU problem that's globally, like global standardization organizations that are ISO is writing a standard on human oversight and they're not near completion.
00:14:07 Johann Laux
I think that is fair to share and say that.
00:14:10 Johann Laux
And it also leads to the question of how we want to define what human oversight should
00:14:15 Johann Laux
achieve?
00:14:15 Johann Laux
Do we want like a checklist where we say, well, the human oversight person should be instructed well.
00:14:22 Johann Laux
And then you see that, oh, yeah, I've assigned you now, Maxine, to be the human oversight person of my AI system, and I've given you 2 sheets of paper in which I told you that bias is bad and that errors are bad, and this is what you look out for and what fundamental rights might be involved in this decision making here.
00:14:37 Johann Laux
And then I can check it off because, yes, I've instructed you well.
00:14:41 Johann Laux
actually, if you really, really, and if I may speak as a researcher here, if you really want to know if that works, I would actually have to test you afterwards and find out whether you actually work with the instructions that I gave you the way I intended for you to work with it.
00:14:55 Johann Laux
Now, I don't know what the appetite of your clients is to actually have empirical testing of instructions, right?
00:15:02 Johann Laux
It's a very costly and a very slow process, right, to evaluate every policy that you have, every governance policy.
00:15:09 Johann Laux
It is, of course, much faster
00:15:11 Johann Laux
cheaper and more scalable to just have these checklist systems.
00:15:14 Johann Laux
But since we are still in the early days of certain forms of human-AI interaction, right, we still don't know necessarily how humans will behave.
00:15:23 Johann Laux
So if it was my world, right, I would want every policy to be evaluated by human-AI interaction researchers, right, before we really trust it.
00:15:32 Johann Laux
But I understand that this is not necessarily the pace which business can deal with.
00:15:37 Johann Laux
But probably, yeah, I would be really interested to hear your view.
00:15:41 Johann Laux
doing this here.
00:15:41 Ali Shah
My big lesson from being a regulator and really thinking about this was getting across one central point, which is, have you placed the individual at the heart of your deliberation?
00:15:51 Ali Shah
In data protection, the individual is known as a data subject.
00:15:54 Ali Shah
The question was always, is the data subject at the heart of your deliberation around what this AI system might do and how you're going to use their data?
00:16:01 Ali Shah
And have you put in place all of the necessary oversight and controls around that?
00:16:05 Ali Shah
Let's extend that to AI and what the EU is suggesting.
00:16:08 Ali Shah
Again, I think there is a notion at the heart of what they're trying to do, which is to say, have you really thought about the impact of what this AI system might deliver, how it might work, what can that mean?
00:16:17 Ali Shah
Now, everyone else in industry, all of us, are then left to interpret, well, what do we practically do about that?
00:16:22 Ali Shah
And I do think organisations like SEN CENELEC and ISO are going to and are trying to produce the sorts of standards that can be applied practically
00:16:32 Ali Shah
There's other frameworks more globally.
00:16:33 Ali Shah
If you look at what NIST has done, if you look at what is being developed in Singapore, they're all really helpful.
00:16:38 Ali Shah
But the human oversight aspect has to be part of a broader framework, a framework of how do you manage risk, a framework of how do you help employees become more literate so they understand risks and can understand when and where they should be taking action versus not taking action.
00:16:54 Ali Shah
A broader sense of understanding around when is it right to use certain types of AI technology and when is it not.
00:17:00 Ali Shah
You gave the scenario earlier around medical decisions.
00:17:02 Ali Shah
In that example, it's really important that you've tested the system and the level of confidence you have that it will deliver the results you're expecting is high.
00:17:12 Ali Shah
If the technology is not ready for that use case, then maybe it's not the right technology in this moment in time.
00:17:17 Ali Shah
And so I think that's where the human value judgment comes in.
00:17:20 Ali Shah
We see that in work with healthcare authorities.
00:17:22 Ali Shah
We see that in work with insurance providers.
00:17:24 Ali Shah
We see that in work with retailers.
00:17:27 Ali Shah
They are making these choices around when is the right time for us to use certain types of technology and to make it accountable.
00:17:34 Ali Shah
And so they have good control.
00:17:36 Ali Shah
They are also then trying to decide when is it right for us to educate our teams and staff members, those who have a governance role around how to test, how to make a decision, how not to overly, how not to have overconfidence.
00:17:48 Ali Shah
how to really judge whether actually the AI might be more accurate than a human if a human was in that position.
00:17:53 Ali Shah
So I think Maxine has learned some points on this.
00:17:56 Ali Shah
So I'll pass to you, Maxine, but I just wanted to make that point in Johan first.
00:17:59 Maxine Setiawan
No, I think I wanted to touch upon the thing about checklists first.
00:18:02 Maxine Setiawan
I think that's a very interesting point because is there someone there saying approve versus decline?
00:18:07 Maxine Setiawan
That's a very old way of thinking human oversight when we used to have reinforcement learning with human feedback.
00:18:14 Maxine Setiawan
We're now today in
00:18:16 Maxine Setiawan
AI deployment world, you don't really see that as much anymore.
00:18:19 Maxine Setiawan
You see more of the AI agents, AI assistants that are everywhere, essentially.
00:18:24 Maxine Setiawan
Within the realms of how do we put this into processes and practice, I think there's two ways to see this.
00:18:32 Maxine Setiawan
There's the preventative and there's the detective bit.
00:18:35 Maxine Setiawan
The preventative is the whole literacy point where you have to make sure that the humans in itself are equipped, they know what to
00:18:42 Maxine Setiawan
see, they know what to look for, they know what to identify errors, if there's any.
00:18:46 Maxine Setiawan
And there's a detective bit where it's like, can we check that it's actually working?
00:18:50 Maxine Setiawan
Which what you said earlier, Johan, drifting away from that medical example, for example.
00:18:56 Maxine Setiawan
There's more and more.
00:18:57 Maxine Setiawan
more AI assistance going everywhere in different industries, different corporate worlds, different roles and jobs where it's more peripheral.
00:19:06 Maxine Setiawan
And that part is not necessarily high risk, but it's everywhere.
00:19:10 Maxine Setiawan
And how does human play a role in that?
00:19:13 Maxine Setiawan
And there's a huge...
00:19:14 Maxine Setiawan
kind of like piece on over-reliance risk here, where there's two ends of the spectrum.
00:19:19 Maxine Setiawan
One is people who are very knowledgeable in their domains, high-risk use cases where people like the doctors, people who have licenses around this use cases, they need to be there, versus the other end of the spectrum where it's the general person, the general audience, the usual worker using an AI assistant, and how does human
00:19:41 Maxine Setiawan
oversight and effective human oversight pays in that.
00:19:44 Johann Laux
It's about value judgments, right?
00:19:46 Johann Laux
And especially when it comes to domains such as fairness, then we can see that people might make different judgments about what's fair.
00:19:54 Johann Laux
And it's very hard to find common ground on that, except for like very clear cases of unfairness.
00:20:00 Johann Laux
And usually when we look at AI systems and the deployment, there are different actors involved, right?
00:20:06 Johann Laux
You have an AI developer and say you have an AI developer who builds an AI system for human
00:20:11 Johann Laux
resources for who to select for an interview.
00:20:14 Johann Laux
And this AI developer is very fairness conscious and says, well, you should have a human oversight person here, but I give them a metric, right, that tells them when something is unfair.
00:20:25 Johann Laux
And say you have two groups of applicants, say males and females, and our system is designed such that the group that gets the best selection rate, that gets invited to interviews the most, the other group should at least have the selection rate of 80% of that, right?
00:20:39 Johann Laux
So now,
00:20:41 Johann Laux
you have that threshold, 80% is the fairness threshold now.
00:20:44 Johann Laux
And then, but now think of the user side.
00:20:47 Johann Laux
Now, so you buy, you have an organization, you buy that AI system, keep the 80% in mind.
00:20:52 Johann Laux
So you have 10 applications for male candidates, you have 10 applications for female candidates, and the system suggests 4 males and three females to invite for the interview.
00:21:01 Johann Laux
Now, if you do the math, the female group now has 75% of the select rate of the males.
00:21:06 Johann Laux
Now, the human in the loop should say, hold on, AI developer, you told me
00:21:12 Johann Laux
the system is fair, so we should have selected more females, but we can't divide them, really.
00:21:17 Johann Laux
So they might still think, well, actually, it's only 75%, so it's below the numerical threshold that you told me, but three against four is still kind of fair for me.
00:21:27 Johann Laux
Or the supervisor might say, you might flag that as a human oversight person, right?
00:21:31 Johann Laux
And the supervisor says, oh, come on, three and four, that's roughly the same, right?
00:21:34 Johann Laux
So it just goes to show that we still make value judgments along the use of the technology,
00:21:41 Johann Laux
even if we've maybe before agreed on even a numerical value, but will people respond to that, right?
00:21:47 Johann Laux
Because a lot with AI is, just to finish that thought, it's a lot of like metrics driven, right?
00:21:51 Johann Laux
And it's engineers to develop those systems, but then how do people who are in human resources decide on that?
00:21:58 Johann Laux
Will they be impressed by the 80% threshold where they think that 75% selection rate is still good enough?
00:22:04 Johann Laux
Is that still fair?
00:22:04 Johann Laux
So there's a lot of value judgments that will still have to be made
00:22:09 Johann Laux
In different scenarios, so defining the ground truth of what is fair is complicated and complex and requires ongoing reconsideration.
00:22:17 Johann Laux
So.
00:22:17 Ali Shah
Yes, I love your point there, but and I was going to try and be contrary just to spice it up, but just by saying I don't think that's the conversation we're having in industry today.
00:22:27 Ali Shah
I think that's the conversation we were having three, four, five years ago around how do we set metrics.
00:22:31 Ali Shah
Actually, if you look at how the US approaches recruitment targets versus what the UK does, what the EU does, there's variations.
00:22:37 Ali Shah
So judging what is fair, how you might equitably and fairly use AI in recruitment, I think varies.
00:22:43 Ali Shah
So you've got to make sense of that.
00:22:44 Ali Shah
And to try and simplify this, can we set some numbers, whether it's 80% or whatever it might be, and then that leads to the outcomes that you're describing.
00:22:51 Ali Shah
But I think that example doesn't match reality now.
00:22:54 Ali Shah
I think actually this is not a technology driven question.
00:22:57 Ali Shah
It becomes a question around what is the organisation's goals and values.
00:23:01 Ali Shah
And lots of work was done over the last few years around organisations setting principles and policies around how they want to use AI, what their values are.
00:23:08 Ali Shah
The hard yards now are how do you translate that into a way of delivering services.
00:23:13 Ali Shah
And let's take your example.
00:23:15 Ali Shah
The question might be, what is our goal around making sure our workforce represents the sorts of clients and customers that we are serving?
00:23:22 Ali Shah
represents the sorts of profile that we want as an organization.
00:23:26 Ali Shah
And then does AI have a role?
00:23:27 Ali Shah
So this goes back to my point earlier that actually human oversight and governance doesn't always start with the technology, it starts with the overall objective that many of our clients have, whether it's in HR or finance, whether it's in a particular industry.
00:23:39 Ali Shah
That's where they're starting.
00:23:41 Ali Shah
It's no longer the case that actually the metrics are set by the technical team or the data science team.
00:23:46 Ali Shah
Actually what happens is there's a broader discussion around how will all of these
00:23:50 Ali Shah
suggestions that we have around how we want to evolve our business in HR recruitment using AI as an example.
00:23:56 Ali Shah
How do we want to go away and make sure that we represent those values in each of our functions?
00:24:00 Ali Shah
So HR can really think about and recruitment can think about what does it mean to have people applying?
00:24:05 Ali Shah
How will we do CV sifting?
00:24:07 Ali Shah
What's the sort of approach we want to take?
00:24:09 Ali Shah
How do we want to do outreach and build a relationship that data science and technical teams might be thinking about?
00:24:15 Ali Shah
Actually, if we were to build a tool,
00:24:17 Ali Shah
What do we want to build in that allows it to sift CVs really fast and accurately?
00:24:22 Ali Shah
But also, how might this go wrong?
00:24:24 Ali Shah
So in your example, is there a way that actually we might end up localizing around the wrong metrics?
00:24:30 Ali Shah
Maybe we haven't thought about that.
00:24:31 Ali Shah
Maybe there's unintended consequences.
00:24:32 Ali Shah
So now the conversation moves on.
00:24:34 Ali Shah
And because it's a team sport in terms of accountability, the question then becomes, how do you avoid this getting overly complex?
00:24:40 Ali Shah
Actually, the loop is the feedback loop to allow
00:24:43 Ali Shah
decision makers to take action.
00:24:45 Ali Shah
Fundamentally, this is a technology that allows us to really quickly iterate and learn.
00:24:50 Ali Shah
But how do you take action based on that feedback?
00:24:52 Ali Shah
I think that's the biggest challenge right now, the pace of developments, creating this sense of complexity.
00:24:58 Ali Shah
But actually, there are ways to simplify that.
00:25:00 Ali Shah
A lot of the work we do is how can you help simplify these processes?
00:25:04 Maxine Setiawan
Yeah, I think the loop is no longer a loop.
00:25:07 Maxine Setiawan
In reality, I would agree with you, Ali, that this conversation of
00:25:11 Maxine Setiawan
Developers and the users are very close.
00:25:14 Maxine Setiawan
In reality, it's not.
00:25:15 Maxine Setiawan
In reality, it's very far removed.
00:25:16 Maxine Setiawan
And the developer sits within an AI company that sells this platform of software for clients to use, where the end users who might be recruiters or people in HR, any...
00:25:29 Maxine Setiawan
Anyone who's using this tool, they're so far removed from the design and development of the AI systems.
00:25:34 Maxine Setiawan
They themselves might not know how to spot errors or how to identify, oh, this is not fair.
00:25:39 Maxine Setiawan
This is the predefined metrics that I want.
00:25:42 Maxine Setiawan
They might not know that.
00:25:43 Maxine Setiawan
And how do we capture all of these different roles?
00:25:46 Maxine Setiawan
and improve all of the understanding of how AI can go wrong or can do very well.
00:25:53 Maxine Setiawan
Therefore, it's the literacy question.
00:25:55 Maxine Setiawan
How do we make sure people know about what AI does and cannot do very well?
00:25:59 Johann Laux
The way I started to think about human oversight is that we don't want to create people who are then scapegoats for blame, right?
00:26:07 Johann Laux
And because eventually the harm will occur somewhere, and that might allow the public to localize a point
00:26:17 Johann Laux
at which to direct the blame.
00:26:19 Johann Laux
And I give you like an early example that I sometimes use to actually get the discussion started on human oversight is when Uber tested self-driving cars in Arizona in 2018, there was an accident where a person crossing the street was run over and actually killed.
00:26:35 Johann Laux
Now, the
00:26:35 Johann Laux
The problem is that there was a safety driver on board of that car, and that person was using their phone to stream an episode of The Voice because they were utterly bored with checking on that car, right?
00:26:47 Johann Laux
You sit in the car all day, and maybe there's also a function of trust, right?
00:26:51 Johann Laux
The longer you see the car driving on its own very well, you think, oh, it's working well, so I might just stream something, which is something that the person should have, of course, never, ever done.
00:27:01 Johann Laux
You should not stream an episode
00:27:03 Johann Laux
of TV while you're the safety driver in a car.
00:27:05 Johann Laux
Absolutely not.
00:27:06 Johann Laux
What happened actually then was that the person was found to be like legally responsible for that and that person, a woman took all the blame.
00:27:15 Johann Laux
I don't want to make excuses for that absolute neglect of the job.
00:27:19 Johann Laux
At the same time,
00:27:21 Johann Laux
I wonder why is that person even able to have a phone in the car?
00:27:25 Johann Laux
Was she allowed to have the phone?
00:27:26 Johann Laux
Was she checked to have a phone?
00:27:29 Johann Laux
Why do you design a system in a way that people get bored doing the job, right?
00:27:32 Johann Laux
So it's that kind of like diffusion of real responsibility if you think it through.
00:27:37 Johann Laux
But if you have something like a car crash or in the medical domain, right, the occurrence of harm will be very localized and it will be very easy to point fingers and spot a person, versus there's other fields where it might be
00:27:51 Johann Laux
more diffuse, right?
00:27:52 Johann Laux
If I apply at a company and I'm not invited for the interview, I don't know.
00:27:55 Johann Laux
Is this the HR professional's fault?
00:27:57 Johann Laux
Is this the AI developer's fault?
00:27:58 Johann Laux
Is it the CEO's fault?
00:28:00 Johann Laux
I don't know.
00:28:00 Johann Laux
Is it my fault?
00:28:01 Johann Laux
I don't know.
00:28:02 Ali Shah
I'm so glad that you're sort of bringing our attention to this point around, is the role that humans play in human oversight meaningful?
00:28:09 Ali Shah
That's what you're getting.
00:28:10 Ali Shah
Are we meaningfully introducing those checks and balances and safeguards?
00:28:14 Ali Shah
I've talked earlier about how we're seeing actually it's becoming more involved and more complex in many ways to try and ensure accountability works in practice.
00:28:22 Ali Shah
It's a shared team sport rather than the simple sort of workflow that you were describing earlier.
00:28:27 Ali Shah
And to deal with that complexity whilst making sure that the intervention of a human is meaningful, I think also requires us to reimagine
00:28:35 Ali Shah
what we need to make this work in practice.
00:28:37 Ali Shah
And I think AI itself is part of the answer.
00:28:40 Ali Shah
Part of the answer is, how do we make sure that those of us who have a role in governance and oversight and accountability are also augmented with the tools that allow us to understand what are the risks within the system?
00:28:51 Ali Shah
How is the AI working?
00:28:52 Ali Shah
Is it working the way it should be working?
00:28:53 Ali Shah
Is it working in ways that are getting close to a failure?
00:28:56 Ali Shah
Is there going to be misuse of this technology happening?
00:28:58 Ali Shah
What are other issues are emerging that we might see from the outputs?
00:29:02 Ali Shah
The level of information we're generating, I think, very quickly becomes more than any one individual or even a team or even a whole department working around this can handle.
00:29:11 Ali Shah
So bringing in technology to support that assessment, I think, becomes really critical to judge whether the human role was meaningful.
00:29:20 Ali Shah
If we can define what it means to have human
00:29:23 Ali Shah
intervention that is meaningful, then I think around that we can communicate what it means to then be trustworthy.
00:29:29 Ali Shah
Because to be trustworthy, you have to have indicators to demonstrate.
00:29:32 Ali Shah
So if you think about how decisions are made in, say, insurance, one of our insurance clients is trying to judge whether they use AI technology around what they do to process insurance claims, there they have thought about reimagining
00:29:46 Ali Shah
the whole human-in-the-loop process because they really want to use agentic technology because it can really improve the experience for the end user.
00:29:53 Ali Shah
It doesn't matter, typically, if a user maybe had a language barrier where they weren't able to communicate easily.
00:30:00 Ali Shah
Actually, agentic-based system and generative systems can allow you to communicate in multiple languages.
00:30:04 Ali Shah
You can allow people to have a much better user experience.
00:30:08 Ali Shah
But by introducing that technology, you also have to really think about
00:30:12 Ali Shah
will we even understand what the customer was saying?
00:30:14 Ali Shah
How will we have confidence around the information they gave us?
00:30:17 Ali Shah
We're relying more on AI to do that.
00:30:19 Ali Shah
And the answer is bringing in different AI, independent AI systems to help you understand.
00:30:25 Ali Shah
And I think that then gives the information to the human overseers to judge actually, have we got the balance right here?
00:30:32 Ali Shah
Are we focused on the right types of risks?
00:30:33 Ali Shah
Are the risks high enough that we need to have more dedicated human resources focused on that?
00:30:38 Ali Shah
Or are the risks low enough that actually we've got confidence that harm is not going to happen?
00:30:43 Ali Shah
And in that example, the insurance provider we're working with, they were able to move faster in helping some of their customers who were maybe not, English wasn't their first language or native language,
00:30:54 Ali Shah
get access to help and support a lot faster than the tradition used to.
00:30:58 Ali Shah
At the same time, they had a much better overview of when might the technology not work exactly.
00:31:03 Ali Shah
That's why we need to invest our human energy to spend more time trying to fix those issues.
00:31:07 Veena McCoole
So Ali, it sounds like here we have almost a situation where we were talking about humans in the loop, but now there's actually a world in which AI can actually help support humans.
00:31:17 Veena McCoole
in their journey for accountability in these kind of like partially automated systems, if that makes sense.
00:31:23 Ali Shah
It does.
00:31:23 Ali Shah
And we are going through another phase of evolution with Agentic systems.
00:31:29 Ali Shah
70% of the companies that we work with who are thinking about or using Agentic right now are incrementally improving the workflow that they've got.
00:31:37 Ali Shah
You know, they're trying to think about how can Agentic help us
00:31:40 Ali Shah
make this workflow slightly more efficient, make it move a bit faster, what can we do?
00:31:45 Ali Shah
And that's great if that delivers some benefits.
00:31:47 Ali Shah
But the real power of this next phase of AI, this next wave of AI technology, is if you totally reimagine that workflow, if actually you think, well, the objective is to help customers reach our services a lot quicker, how might we totally reimagine that workflow?
00:32:01 Ali Shah
There, we're going to have to also rethink what governance really looks like and accountability looks like.
00:32:06 Ali Shah
And the only way, I think,
00:32:08 Ali Shah
will be able to offer the same level of confidence that people expect of us, an industry needs to deliver, is if AI is part of the answer around human oversight.
00:32:17 Ali Shah
At the scale that we're talking about, you know, Maxime was talking earlier around, it's no longer simple checklists and it's no longer simple workflows.
00:32:24 Ali Shah
Actually, we've got not one or 10s or hundreds, dozens of models, but potentially hundreds of AI models and systems running in businesses.
00:32:31 Ali Shah
At that sort of scale, you also have to really think about how the technology can help you as a human,
00:32:37 Ali Shah
have that confidence.
00:32:38 Ali Shah
We should also, of course, I think, be humble in saying that not all of the big issues are easily resolvable, like over-reliance, overconfidence.
00:32:48 Ali Shah
How do you avoid cognitive buy-ins towards the technology always being right?
00:32:53 Ali Shah
There's the human frailty aspect to this that we need to sort of manage.
00:32:55 Ali Shah
But I think even with those things said, the technology will be part of our toolkit in helping make sure that human oversight is robust.
00:33:04 Ali Shah
And needs to be robust in the ways that give people confidence around using and interacting with this technology.
00:33:09 Johann Laux
Yeah, it's kind of, if you think of the idea of human oversight that it's supposed to monitor systems during runtime and allows intervention during runtime, then with agentic workflows, the pace is so fast that there is no way you can...
00:33:25 Johann Laux
do that meaningfully without yourself as a human oversight person relying on AI tools, technical tools, right, that support you in your role, that give you indicators of something potentially going wrong.
00:33:36 Johann Laux
I think we're maybe entering a world in which the human control of AI, the human oversight of AI, it's kind of like in a hall of mirrors, right, where you see technology everywhere because otherwise it's just impossible to do.
00:33:49 Johann Laux
And that's, I think, why research on human-AI interaction, right, you mentioned cognitive bias,
00:33:55 Johann Laux
right?
00:33:56 Johann Laux
Humans have been shown to have all kinds of tendencies towards technologies, right?
00:34:00 Johann Laux
To over-rely on them, to under-rely on them, to selectively rely on them whenever it matches their pre-existing biases, right?
00:34:08 Johann Laux
So we're still early in our journey.
00:34:09 Johann Laux
We're still adding to, I like the idea of a library that you mentioned, right?
00:34:13 Johann Laux
We're still adding knowledge to this library.
00:34:15 Johann Laux
We're adding it fast, but at the same time, I think the technology is still faster at the moment.
00:34:19 Johann Laux
So I think there's a call for all of us, both researchers and industry practitioners, right, to just
00:34:25 Johann Laux
add knowledge to society in terms of how people interact with AI systems, how they can control AI systems where errors and harm occur and not bury that.
00:34:34 Johann Laux
So I think one of the preconditions for this being a successful enterprise of human retaining control over AI is also both the accountability part that we spoke about, to have humans being accountable, but also transparency, right?
00:34:46 Johann Laux
So that we can learn from each other, we can learn from the uses of AI and the oversight policies that we've tried to implement or implement.
00:34:55 Johann Laux
implemented and then evaluated.
00:34:57 Maxine Setiawan
The good news is we're not as clueless as we were three years or five years ago when AI just started and started getting into the workplace.
00:35:06 Maxine Setiawan
Most of us are using AI in our day-to-days, whether it's in your personal life or work life.
00:35:11 Maxine Setiawan
This is...
00:35:12 Maxine Setiawan
solvable, hopefully, in the near future.
00:35:15 Veena McCoole
So we're unfortunately running out of time.
00:35:17 Veena McCoole
I feel like we could talk about this topic for hours more.
00:35:20 Veena McCoole
I'd love to finish by understanding from each of you in one line, what does human oversight of AI mean for the future of work?
00:35:29 Johann Laux
In one way, just listening to what our colleagues here have said, is that maybe all of us will be in some form of overseeing AI in the future of our own work.
00:35:38 Johann Laux
And the challenge will be to keep it not just meaningful in the sense of it being effective, but also meaningful to us, that we still get like meaning out of our own work, that we're not just feeling as like maintaining the operation of a machine, that we still fulfill tasks that are meaningful for us.
00:35:56 Maxine Setiawan
I think it's good to understand what humans do very well and what AI and machines do very well and combine them into like a human and AI interaction at work, not just an interface, but how do you productively utilize your systems and your tools?
00:36:13 Maxine Setiawan
Because at the end of the day, AI is a really good tool.
00:36:16 Maxine Setiawan
It is a tool.
00:36:17 Maxine Setiawan
And how do we use it very well is the question.
00:36:20 Ali Shah
We have to anchor ourselves to today
00:36:24 Ali Shah
and ensure that the way that we're introducing AI into society more broadly takes consideration of all of the ways that society requires us to build in the right sorts of controls.
00:36:35 Ali Shah
We can't live in the future, we live today.
00:36:38 Ali Shah
But I do think we need to keep one eye on the future and really recognize that actually we're in the foothills of our societal journey with AI.
00:36:46 Ali Shah
And actually AI will be part of the answer there.
00:36:48 Ali Shah
It's going to be part of the toolkit that we need as humans to make sure that we
00:36:52 Ali Shah
keep human accountability central to what it means to really govern and deploy AI.
00:36:58 Ali Shah
Every single organization that I work with places their customer, the individual, the citizen at the heart of their deliberation.
00:37:04 Ali Shah
Like that's the big shift I see.
00:37:06 Ali Shah
You know, what does this mean for AI?
00:37:08 Ali Shah
Joe Public?
00:37:08 Ali Shah
How do we really think about that?
00:37:10 Ali Shah
And I think keeping that in mind through all of these iterations and evolutions that we're going to see, I think will anchor as to what is right.
00:37:16 Ali Shah
You know, what is the right balance around governance?
00:37:19 Ali Shah
How do we make sure it works effectively?
00:37:20 Ali Shah
How do we make sure it works at the pace of our lives?
00:37:23 Ali Shah
All of these questions are complex and knotty, but if you don't lose sight of the fact that it's the people who matter at the heart of this, then I think we're going to be okay.
00:37:30 Veena McCoole
Johan, Ali, Maxine, thank you so much for joining us on this episode of The Human Interface.
00:37:36 Veena McCoole
Make sure to subscribe to be alerted when new episodes drop on Spotify, Apple, or anywhere else you listen to your podcasts, and follow the Oxford Internet Institute on LinkedIn and Instagram to stay abreast of our latest research and updates.
00:37:49 Veena McCoole
Until next time.