JUSTIN NORDEN: Welcome back to another episode of the Stanford

Health Care AI podcast.

And we're thrilled to be joined by Dr. Shantanu Nundy, who

is a physician, technologist, worked at some of the biggest

health care technology companies over the past decade

and is now an advisor to the FDA on AI.

Welcome, Shantanu.

SHANTANU NUNDY: Thanks so much, Justin.

So many things coming out.

There's never a dull moment.

But one of the places to start actually

was something we found fascinating,

which is, our people are actually using these AI

tools in their daily lives.

So it was amazing, actually.

OpenAI shared conversations and published

on millions and millions of conversations.

And on the right of this chart, you

can see how people were starting to use this.

So I'll call out just a couple of things first

and then Matt and Shantanu, please jump in.

As we're talking about health care,

one of the things we've talked on bits and pieces

about and heard rumors about from the companies,

but at 5% to 10% of ChatGPT conversations relate to health.

Now, they published some of these stats.

They claim practical guidance, 6% here.

We've heard claims similarly, Google, other companies,

health-related queries are top of mind for consumers.

And so you're seeing this now here in the data

that they've published.

Some of the other things, writing, seeking information

is something very top of mind.

People are doing all the time.

And then interestingly, on the left,

they also talk about in the work setting,

and there's slightly different uses

that people are using at work.

Lots and lots of writing.

So the thing I joke about is expect all those personalized

emails to be written by AI.

That's just the expectation as we're moving forward.

But anything else to stand out to you both?

SHANTANU NUNDY: I mean, for me, I mean,

I'll just say my personal journey, right.

I sort of joke, it's like it started off as a copy editor

for me.

Then a little bit of like a ghostwriter.

And now it's like an analyst.

And I think that's kind of reflected

in a lot of these things.

I'm actually almost like building on top of these tools

as obviously the tools are getting more mature.

But honestly, I think most of it's me

and just getting used to figuring out

how to create more complex prompts that build up

to more higher order tasks.

So this resonates.

MATT LUNGREN: I mean, from my side, again,

we are terminally online.

We are definitely following this, maybe

more so than we should even, I would say.

But this is totally what I would expect

to see, especially on the writing side to see, 40%.

You could almost argue that there are behaviors that are--

our internet behaviors that are starting to transfer over I

think in a large degree, like seeking information

to the model.

But the other part that my mild critique or it's a weird thing,

but you mentioned you're moving up

this ladder of sophistication.

I can't wait for the rest of the world to get there, too.

Because every time I open up LinkedIn, or Twitter,

or whatever name is, I am me because we all

have worked with these models.

So you can immediately spot just the complete cut and paste

of an AI.

And I have to say, it's starting to--

I don't like it.

I don't know how else to say it.

I feel like it's--

and now I'm turning into someone who's like,

searching for the post that is like maybe has a spelling error

or has a little bit of--

clearly not an AI-written cut and paste because I--

SHANTANU NUNDY: The key is the double hyphen.

If you see the double hyphen.

You don't know it's AI.

[LAUGHS]

MATT LUNGREN: It's not this, it's that.

It's like it is so bad, it almost

like you lose credibility, which is an interesting phenomenon

that I did not expect.

I didn't expect to be saying this, but to some extent

I'm wondering.

There's two things I'm wondering as a part of that.

One is I think as people move up the sophistication ladder

and the models become better, whatever

that looks like, personalization, memory, that

may start to change, hopefully.

Maybe this is just a passing moment.

But the other part is like, who are these--

what is this content ultimately written to do?

Because if the consumers of that content

eventually will be another model that's

just grabbing that and providing your end,

then do I even- am I even ever looking at the source again?

So those are kind of things that are bumping around in my head

as I'm seeing this adoption curve.

And this again, I know we like to- we hear a lot of AI slop.

I am full up of AI slop, if I'm being honest.

And I'm begging for just a little bit more, at least

as a reader, more content taste on the internet.

JUSTIN NORDEN: Totally interesting.

Interesting, yeah.

The video content, again, we haven't

talked a lot about the video and image generation,

as it's been a little less relevant on the health care side

so far.

But that's just another piece you talk about,

Matt, AI slop that's happening and just how easy it

is now to create a video of you with any famous person

that you want, doing whatever you want.

We saw OpenAI Dev Day, all this Sora 2 announcements,

Google's Veo 3.

That is moving so fast.

And curation of content is just going to be something.

Again, we've talked about it not a ton here in the health care

context, but it's coming.

And actually, maybe the one health

care related example just that I saw recently is actually

doctors starting to get faked, real doctors that people

are starting to use now to use their image and name

and likeness.

So I haven't seen that yet of you, Matt or Shantanu,

but that's how I guess you'll know you made it.

SHANTANU NUNDY: How do you know I'm not one of those right now?

[LAUGHTER]

JUSTIN NORDEN: Exactly, exactly.

MATT LUNGREN: I had a suspicion.

No, I'm just kidding.

But on that topic too, one of the things

I noticed on the video say maybe less on the writing side,

just to maybe pivot from that former point I was making.

On the Sora 2, for example, great, interesting,

where it almost immediately, you can sense

someone's inherent creativity.

Some of the things that people are thinking up

are pretty freaking cool.

But the signal to noise is difficult.

There's a ton more noise now.

But the signal when it's good is like, wow, that's amazing.

So I don't know where that's going to head,

but I have a lot more respect for the creatives of the world.

We're able to take that tool and still apply

their inherent genius and creativity to really conjure up

something amazing.

I don't happen to be one of them.

I tried.

My kids called it all cringe.

I gave up.

JUSTIN NORDEN: Well, that democratization piece

is something that is paramount on that.

And I think the other piece and Matt,

maybe you want to go through this, is just the cost changing.

These models used to be so expensive to create the videos.

You used to only get a couple seconds.

But Matt, walk us through this.

MATT LUNGREN: Yeah, just two seconds

on this and then happy state of AI

report to all of you who celebrate because a lot of us

wait around for this report.

Shout out to Nathan for putting this out every year.

This is massively consumed by the entire industry.

If you haven't seen it before, check it out.

State of the AI, you can see the link on the bottom.

And this is just such a great public service

because it does encapsulate in a very difficult time what's

happened for the past year.

This is really just to show the point, I think,

that yes, we talk about the capabilities getting better,

but we don't often talk about how much the capabilities are

getting cheaper and more democratized.

And I don't know if either of you think about this,

but as we start to pivot this conversation

towards the direction of health care

and we've talked before about the increasing capabilities,

the access of this technology, I think,

is unlike anything I've maybe ever seen.

I'm sure I know the internet's a great analogy,

but it just seems faster to me and cheaper to me than ever

before.

SHANTANU NUNDY: Yeah.

No, I think it's paramount.

I mean, I think obviously, like affordability

is a huge, huge, huge, huge issue

and equates to access, especially in this country.

And so I think that part of it, you're right,

not a part of the story we hear enough about.

And I think it's pretty incredible.

We had the OpenAI team in recently and just saying

like the model that they made to be the cheaper model

is now performing at the same level as their best model,

three or four months ago.

And so that happening, I think, does open up

a lot more possibilities.

So yeah.

JUSTIN NORDEN: Yeah.

And the access point and the cost as these

are now ubiquitous.

And the state of AI report also talks

about the fully open-source models in that race

that's happening.

The world is moving fast.

And while once, oh, you can't do that, cost prohibitive.

I have this conversation all the time with folks

in our team or health systems.

The cost will come down.

That is the benefit of the competition

we see in the space is people are racing.

And when that happens, cost will drop.

One of the other things we were starting to talk to you about,

though, is access.

And we talked about this some before.

And I know, Shantanu, you were thinking

about a number of problems here.

Well, what happens when you give people unfettered access

and you see a lot of amazing use cases?

But we haven't talked as much about some

of the areas or problems or things that are coming up.

But Shantanu, I know you had a couple you wanted to go in.

And I'm super curious as how you're seeing this.

SHANTANU NUNDY: Yeah, I mean, to me,

what's so interesting is the juxtaposition

between these exciting progress and then some of the stories

that you hear.

So there was one just a couple of days

ago in The New York Times.

I think that the title of the article

was person came to the ER with the flu

and died a few days later.

So it was basically about a college-aged kid, totally

healthy, had flu-like symptoms, went to the ER,

was discharged home with the viral illness,

and then a couple of days later came back

and then ended up passing away.

And there's a lot of pieces to this story, but one piece of it

was that there was actually a sepsis alert that

was at that hospital.

And that sepsis activated and actually was a positive alert,

but it was ignored by the provider.

And I laughed, but I've totally been there.

I see patients still.

I usually turn off the allergy drug-drug interaction stuff.

So I think that this brings up like a whole host of issues.

And I think zooming out, I think those issues are now going

to become like the gatekeepers.

It's not going to be the accuracy as much.

It's not going to be the cost as much to Matt's point.

I think it's going to be the social, technical side.

And some of those specific issues, where, OK, well, they

happen to have the alert set up, which is great,

but we know that all alerts aren't created equal.

And then in terms of workflow, doctors

are getting tons of alerts for different reasons,

so it was ignored.

But now there's other interesting issues that

are coming up, which is OK, well of

all the hospitals, since it's a lawsuit,

the key is what's the standard?

What do other doctors and hospitals do in that area?

Do they have an alert?

And then do doctors generally follow the alert?

And now there's even people asking,

maybe I should turn off my alerts as a health system

because if we don't have the alert,

then there's no risk of our doctors ignoring the alert

and then creating this problem.

And it's like, that's the exact opposite, I think,

of what this technology and those graphs are showing us.

We should be doing, but we have to understand

all those different components of the health care system,

if ultimately we want to solve the problem.

MATT LUNGREN: Yeah.

I mean, I struggle with this too.

And sometimes when I'm explaining

to folks that aren't in health care,

like how something like this could happen because on its face

this is like- it seems how the doctor ignore that.

Alert fatigue, I think this has been

talked about since I think Bob Watson [INAUDIBLE]

a wonderful book on this.

And we've known about this for a long time.

I try to relate this to someone who's not in health care.

It's like if you ever go to a city

that they always honk their horn,

no matter-- like even for the slightest infraction,

they lay on their horn.

You have to become like, OK, they're honking,

but that's just like [INAUDIBLE] it

fades into the background a little bit,

and especially then if you go to a place that doesn't do that

and then it's like a huge event that someone's honking you.

It has a little bit of that vibe to me,

like when you pay attention and then if--

SHANTANU NUNDY: Or maybe the simplest

is the boy who cried wolf.

MATT LUNGREN: 100%, this is a tale zone.

[INTERPOSING VOICES]

SHANTANU NUNDY: It's all tale zone's time.

MATT LUNGREN: But to your point about that-- but with the worry,

though, I share this worry is that either that becomes

the background noise even like it's happened here,

or it leads to a reaction that says no more AI, no more

decision support.

It's clearly, causing harm.

And I don't think that's right either.

JUSTIN NORDEN: Totally.

Well, Shantanu you're seeing this at scale.

You're interacting with scientists at the FDA, reviewers

every day.

You're seeing so many issues for these things coming up,

but tell us a little more.

What are the conversations you're hearing?

what are the topics that keep coming up?

Is it this?

Is it others?

And how are you hearing those conversations going?

SHANTANU NUNDY: Yeah I mean--

well, first of all, let me--

I'll say a couple like preamble things.

I think one is-- it's been really incredible to get

to meet a lot of the scientists and reviewers that work at FDA

and have made their careers there.

I mean, a lot of them came straight from their PhDs or MDS

into the FDA and have been there for 10 or 20 years.

And so there are really incredible people thinking

about this problem already, and it's really inspiring

to have a chance to work with them.

The other thing is, when I was coming in as an advisor,

I thought a lot about--

I do feel this urgency.

I think stories like that example in Columbia, which

was the hospital involved-- creates this urgency.

And I thought a lot about what do I

want to say to have everyone share that urgency?

Because at the FDA, the mission is really

to promote and protect.

But I think sometimes, there's a focus on the protect side

and not as much on the promote side.

And so when I came in, I thought a lot about, OK, maybe I

could use science and numbers.

And I came up with five numbers that I

talk about in every meeting that's become a running joke now

at FDA, which is that there's a hundred

million people in our country with no regular medical care.

There's 75 million that live in a desert.

Medical error is the third leading cause of death.

95% of rare diseases have no FDA approved treatment,

and life expectancy is largely flat

and four years behind other OECD countries.

And we all have our favorite statistics,

but I start with that every single time because I

think what sometimes gets lost is like,

what's the counterfactual?

So like this example, we just talked about of, OK, well, there

was an alert.

Maybe the alert was wrong or right or it was annoying

and it was ignored.

But the counterfactual is we've known for decades since

the Harvard Medical practice study in the 1970s, I think,

or 80s, that there's a jumbo jet every day that's crashing due

to medical errors.

But it's hard because I think the way

that a lot of the folks look at it

is OK, well, is this device causing harm as opposed

to saying what's the harm that's already existing because people

can't get access or that access isn't

as high quality as we want.

JUSTIN NORDEN: And I think this is the fascinating example we're

starting to see in so many fields is

we've talked a lot about benchmarks on tests and exams.

And while it's exciting, we know none of those

are the real comparator as we're talking about.

What is that?

What is the ChatGPT?

Is the answer right?

Is it wrong?

It's pretty good.

Oh, it made a mistake, and that's going to lead to harm,

and it will.

How does that compare to the Google search,

compared to nothing?

And these are just the fundamental questions

we have to wrestle with.

And I think that the interesting field, you all

probably annoyed with me for how much I point back to it,

but it's fascinating starting to see the data come out

in the autonomous vehicle world.

Waymo has released a lot of data now on their crash events,

and events-- and actually, I've seen

a lot of now doctors starting to comment on whoa, whoa, whoa,

if you extrapolated these numbers,

this would be saving thousands and thousands of lives.

And it's not just the accident rates, the fatalities,

as you can react in different ways.

And there's not a right answer to be super clear.

There's this fundamental fear which I think is real in myself

and I think the country that you start

to see on an AI related error that causes harm.

We're going to count as more than the physician we know

harming someone, or then the car crash that

was caused by a drunk driver because we're almost numb to it.

[INTERPOSING VOICES]

SHANTANU NUNDY: I think everyone has that video of that car.

It's like it stays on you, which is interesting.

I think the other thing you brought up, which is--

I think you're right.

A lot of the testing right now-- this

is something that we're thinking a lot about right now.

And the scientists at FDA are thinking about,

is a lot of the testing is in these very simulated

hypothetical environments.

And so we think, OK, that's happening, that's great.

Now we need to figure out how to do that when

it comes to real world prompts and real world scenarios.

Because the other thing I think people

think a lot about is their performance, which

is the median or the mean versus the long tail.

Part of the FDA'S job is to think about the long tail.

And so when you simulate and people are like, oh, yeah,

we tested 100 cases with thousand and thousand cases,

you're like these things are being used by hundreds

of millions of people.

So that tail gets extremely long.

Somebody from the Google team unrelated to LLM,

shared with me that even today--

and I'm going to get this wrong a little bit-- but even today,

every day of all the Google searches, something like--

and again, I'm going to get the number a little bit wrong,

but 10% to 15% is a search that Google's never seen before.

Even today-- I'm not talking about their LLM.

I'm just talking about their normal search.

And so that long tail--

So anyway, so that's the second part.

And then the third part is now how

do we actually begin to understand the outcomes piece?

And I think the other thing that relates

to your point about the self-driving cars

is unless we let these things drive,

how can we iterate fast enough.

So if we force everyone to stay here, and then we control here,

and we really tightly control what

can be implemented with patients,

the cycle times to get that learning,

and this is where I think health has historically really, really

fallen down.

And not been able to improve quality and things.

I think that's something that I'm

thinking a lot about at least.

MATT LUNGREN: Yeah, I mean, well and then to--

and for every one of these-- to your point,

they stick in your head.

There does seem to be a different standard.

My gut says that I don't know if I disagree

with that different standard.

It's for some reason.

I don't know whether it's because I know it can scale,

whether I know that it's probably just a blip that

is maybe representative of something that we're not seeing.

I don't know-- it doesn't.

So that's why I struggle with it too,

but I also feel like there's the other stories that are like,

hey, I went to the health system, they told me X, Y, Z,

I put it in GPT and they told me something different.

And then later it confirmed what GPT said and not my--

and I feel like we have these competing narratives.

I don't know what's going to win out, but at some level,

again, reflecting back on the beginning of this conversation

around consumer behavior at some level and the stats,

by the way, that you pointed out about the desert and the lack

of access, these things are two speeding trains heading

at each other to me, and I don't know how it's going to turn out.

SHANTANU NUNDY: Absolutely.

JUSTIN NORDEN: You mentioned Shantanu

this idea of how do we get started and learn and iterate?

And actually just it was last week,

the FDA just put something out about real world monitoring.

And so tell us a little bit more about that.

It sounds similar to the concepts, but what can

you say at this point?

SHANTANU NUNDY: Yeah it's really exciting.

I mean, I think look--

I think in general even if you Zoom out of software

for a second and you think about the canonical drug pipeline.

So you have phase 1, 2, 3 and then phase 4 is post-market.

I mean, I think that in today's environment where

we have so much data, when I look at a patient in front

of me, they look a lot different than the patients that

were eligible and not eligible in a clinical trial.

And that's why people say, half of health is not evidence based

and half of evidence based medicine isn't done.

On some level that's fair.

On some level, it's like, well, the evidence-base

has limitations.

It doesn't apply to most of the patients sitting in front of me.

So those challenges were already there,

but now you add to the fact that unlike a pill

that doesn't really change once it's in the market.

We know that this stuff is changing really

rapidly, let alone the fact that the underlying technology itself

is advancing rapidly.

So I think there's this new idea that's maybe not super new,

but it may be becoming more common to say, how do we

focus more of our attention on how these things are performing

in the real world?

For all those reasons, we may not

have the best test to know if it's valid or not.

We know these things are going to change.

We know the underlying technology is going to change.

We know how it's deployed matters

so much, and so why not focus our attention there?

And we realize that's a science too that's evolving.

So the whole idea behind what the FDA put out last week,

which is really exciting, is like, hey, we

want to learn in that area.

And we recognize is that-- that evolving let's

call it like the regulatory science or the science of how

do you evaluate and monitor these things.

No one's quite cracked the nut on that.

And that work is really distributed across academia,

industry, regulatory bodies, and so it's a chance for us

to really be able to understand what's working

and what's not when it comes to that.

MATT LUNGREN: I think this is really important because well,

first of all, I want to make a meta

comment that the level of discourse and this is, again,

back to the credit of those that are serving in these roles--

the level of discourse has, I would say,

elevated beyond, I think, anyone's wildest expectations.

I mean the fact that we're thinking about these things,

and the other topic I wanted to bring up in addition to this

is we are--

at least I'm hearing--

maybe you can confirm some of these conversations

that we could get to a place where some of the trials

can be performed digitally.

We've seen some recent work that's

continuing to show us that we're getting to a level of scale

and sophistication, and this is somewhat orthogonal to just

the ChatGPT's of the world.

But we can start to think about data and then projecting out

with a great degree of confidence matching trials

that we have already plenty of data

on to show that we done this digitally,

the same outcome would have been reached.

So therefore, can we accelerate the market, the bottleneck

that we have in our traditional structures

without sacrificing safety.

This is science fiction stuff.

At least as someone who's been around long enough to know

that these have been floated for-- these ideas

have been floated for a long time.

It feels tantalizingly close.

And again, that goes back to that level of sophistication,

the level of discourse that I think

the community is having with the regulators in a way

that I personally haven't observed before.

SHANTANU NUNDY: Yeah, no and that and that's honestly

why I wanted to be on with you guys is,

I think what's been incredible just generally

about this community, not just the amazing folks at the FDA,

is that everyone I talk to is like, how can we help?

And again, I haven't been in FDA very long

and I haven't worked in other spaces within FDA.

But I think usually there's, oh, the FDA is going to regulate us,

and like I don't know what they want.

But I feel like in this space--

I think folks that are really at the frontier,

to use an overloaded term-- on the frontier,

I think have a lot of humility around,

how can we do this together?

We're trying to answer honestly, the most important

health-policy question of the century, which

is how do we ensure access to safe and effective AI?

And that's been really incredible to see.

And part of what I'm shining a light on is like, OK, yeah,

there's that whole post-market space.

That's where we need help.

We need really pragmatic ways across a very wide range

of technologies, whether it's an imaging-based AI, whether it's

a pathology based, whether it's purely

conversational primary care or specialty-based,

we need ways of benchmarking and monitoring those tools.

And then underneath that there's a whole scaffolding.

So I'll pick up example of something

that most people aren't aware of is, so let's say hypothetically,

we had like all EHR data in the country

flowing through a place like the FDA.

Amazing, well guess what?

We still wouldn't be able to evaluate and monitor any AI.

And why is that?

Because all of us are docs.

We operate-- we work in EHRs.

That data is not even in there.

So when you prescribe a pill, you have the pill,

you have the NDC.

It's E prescribed.

It's in the claim file.

It's in the EHR file.

Where is the fact that I use OpenEvidence last Friday?

Where is that in the EHR.

And so it's like you can have all the data

in the world, all EHR data, but if it's literally not there--

and if it is there, do we have a unique identifier?

So every pill of every formulation

and every generic and every type has a different code.

Do we have a code that says that this was--

I'm not going to keep picking on open, but this algorithm

versioned on this day and this thing-- and without that,

how can we use real world evidence to be able to do this?

So there's the methodological sexier stuff.

And then there's the plumbing part

of this, which again, I believe that if you

put that in front of the innovation community,

we can solve it, but we're not really talking

about those things, and instead we're

really focused on those hypothetical testing

environments, as opposed to saying, guys,

what is it going to take to support

an entire ecosystem to do this?

JUSTIN NORDEN: And just, is this just related

to one of the multiple conversations

I had, and I won't name names, but just

talking with health systems of we

don't even know what AI tools we're using.

But let alone the who's using it where and how.

And there's been a couple instances

where even a CISO has been let go because they

were asked this question.

This question was presented, and it's like, how can we

not even have that answer?

I bring it up because I actually would--

most and it's not even a fault on trying to-- most hospitals

are trying to-- are starting to put together

lists of where their AI is getting used

or starting to put together governance communities.

But software is moving so fast.

This old tool you bought and provisioned,

a health system might have 2 to 20,000 software systems that

they work with.

As these things are put together.

And rightly so, many CIOs want to reduce that list.

But every one of those software companies,

if they're not sleeping, should be implementing AI.

But we're still grasping as health

care, how do we track this?

How do we understand this?

How do we move through this?

And to your point, I think this is, again,

part of the reason we started talking about this

is how we can bring people together

to try to solve these problems.

SHANTANU NUNDY: And like you said, and it's got to be-- you

have to know it at the system level,

but it has to be time stamped.

I'm going to use an old school term,

time stamp in the individual, patient encounter.

Because if you want-- let's say you're

looking at a clinical-decision support tool,

and you're trying to understand in a post-market, real-world way

how it's working.

Well, you have to know that it was used on Mary Smith.

These were the inputs into the model.

These are what the outputs of the model is.

And this is the version of the model,

just very simply so that you can roll the tape, and say, OK,

did Mary Smith actually have that diagnosis?

Did Mary Smith actually benefit from that treatment?

But if we don't have that, even if we

timestamp that it was used.

And we don't know the inputs and outputs, how can we

know if it was right or wrong.

I mean, there's a lot of foundational stuff

that has to happen in addition to

again, what are the benchmarks?

What's the tooling?

What's the framework?

And then how do we get access to the data?

So there's a lot of pieces there,

which I think is all solvable, but we

have to be paying attention.

MATT LUNGREN: I mean, with such a broad landscape, I mean,

this again goes back to the prior comment I made,

which is just that this is the conversation that

bringing this stuff to light.

Even hearing from you now, I actually

hadn't considered some of this and the stat about the CIO,

imagine the old model, which is still

in operation of running a tech shop in a health system

is how are my data centers doing?

How is my software all patched and up to date?

And what am I licensing?

But to Justin's point, each of those solution providers,

their new version could have nine or 10 or dozens and dozens

of AI-based capabilities.

That's one side.

And then there's this-- again, this consumer part

that I keep bringing up.

That's not just the patient as the consumer,

but it's also the health-care workforce

that is maybe using their phone or whatever

the solution of the data.

So this is a really hairy problem.

I guess one of the things I was going to ask if at some level,

do you have to just prioritize because it seems like there's

a lot here, a lot of big rocks, and part of that prioritization,

obviously you're hearing from the community.

Is there also just some registry or some way that we collectively

are reporting signal back to you in some way,

as opposed to waiting or having to have the outreach be

extraordinarily proactive.

I don't and maybe you're going to tell me

this is already a thing, but I would love to be able to say,

hey you know what?

Like a feedback, this was a dangerous output or a flagging.

This alert is insufficient for the gravity of what

the alert is for or some--

I don't know if there's an answer here

because there's so many players involved,

but it almost feels like a way to prioritize

all the different things we have to tackle.

SHANTANU NUNDY: Yeah, absolutely.

So I think it is a huge challenge.

I don't want to minimize it, but also-- maybe we

could talk about some likes-- there's

a lot that we can build on.

So the FDA has always had a risk-based framework.

And so I think a big part of this

is really being very clear that, hey, these

are the things that are not regulated

and these are the things that are.

And then of the things that are regulated,

these are the things that we think it really matters

to do this post-market stuff.

And this is the stuff that we don't.

And because I think that this takes from 1,000 things to maybe

a couple dozen things.

So for example, ambient scribes, like wellness tools.

There's places where I think we've already

said these things are not regulated,

or these are places where we don't

need that level of oversight.

So that helps cut it down.

The other piece is that there are scaffolding to build on.

So in the implant world, so FDA regulates

like 20% of the economy.

Part of that includes knee implants and things like that.

There's a system called unique device identifiers.

And there's a whole system, if you're a surgeon

and you put in a device and you find that it's faulty,

you can put up the flag that says,

hey, I just put something in that I think really

harmed a patient.

And there's a whole process for when that recall happens.

Just like car recalls, there are device recalls.

And even though they don't change as frequently

as software, they actually change more frequently

than you think.

And the oversight for the changes

is not as much as you think.

The canonical example is screws, like ortho screws.

There's a ton of companies that make them.

Some of them are small.

They iterate on those screws.

And so we really--

I'm not going to say it but a big part of the signal

comes from individual surgeons saying, hey, I use this screw.

And it didn't seem to work the way it intended to.

And so part of it is just extending

some of those frameworks.

Obviously, there's some unique challenges,

but it's really building on some of those pieces.

JUSTIN NORDEN: Well on that optimistic note, and again,

thank you so much for joining us on the conversation again, which

was the reason we started this, which is how do we bring people

together at what I think we all believe

is the most transformative time for health care period.

And how do we set this up right?

Thank you so much--

SHANTANU NUNDY: For anyone out there

listening that has made some progress on any of these fronts,

go take a look at the request for information

that's out there and definitely patch in.

We're looking to collaborate and learn from the best.