Narrator: You're listening to the humans of DevOps podcast, a
podcast focused on advancing the humans of DevOps through skills,
knowledge, ideas and learning, or the SK il framework.
Jason Baum: Hey everyone, it's Jason Baum, Director of Member
experience at DevOps Institute. And this is the humans of DevOps
podcast. Happy New Year. And welcome back. Thank you for
joining us for season three of the humans of DevOps podcast. My
second one with you. I'm really excited to head into a new year,
new year new beginnings. And and we're going to shake things up
on the podcast, we're going to do things differently. And today
on the episode, we'll be chatting with founder and CEO of
unit Q. Christian Wicklund, on how DevOps should evolve their
decision making. And I love this topic, so I'm really excited to
talk to Christian about it. Christian believes that customer
feedback is the missing ingredient in how developer
teams prioritize and make decisions. His company unit Q
observes millions of quality issues for consumer tech
products, speaking to their product. Speaking to their
progress, rather, the company also raised 30 million from top
VC firm Excel. Prior to unit Q. He and Nicholas Lindstrom
founded Scout a social app with over 50 million users that was
acquired by the meat group in 2016. Christian is a native
Swede, an avid surfer, a prolific tomato grower and a
proud father of three. Cristian, welcome to the humans of DevOps
podcast.
Christian Wiklund: Thank you, Jason. Happy to be here.
Jason Baum: Awesome. Yeah. And thank you for taking, like a
second in your day with with three kids. I don't know how you
do. I've won. And I don't know how how it's done, especially
right now. So
Christian Wiklund: they say to kids is four times worth one.
And then the third, it doesn't make a difference. So
Jason Baum: just adding to the chaos adding? Yeah, yeah. Yeah.
Couple more, you got a basketball team? So there you
go. So you're ready to get human Christian? Yes. Awesome. Let's
let's just dive right into it. I'm really excited to talk about
this. Because the experience experiences are really, you
know, that's, that's the topic I love to talk about. With DevOps
Institute. I'm the Director of Member experience. And one of
the things that I like to talk about all the time is no more
anecdotal, let's get to the data. And but of course, how do
you get to the data? What are the goals, you should look at?
What are the metrics, you should be talking? Because you can
metric yourself into death? Right? There's so many metrics,
so many things to look at. What how do you get into it? So let's
just start with, why should DevOps prioritize real time
customer feedback in their decision making?
Christian Wiklund: Yes, so maybe I should start with a little
backstory here of why we're building unit Q and and how we
set out on this, this journey to build really the quality
companies. And it started years ago with me and my co founder,
Nicholas, we were building this company called scouts, which
still is a social network for meeting people. And if was a
mobile first company, which sounds not very interesting
today, but back then in 2009, and, and 10, it was pretty
special, you know, that you started with mobile. And what we
discover there was, you know, as we were moving from, sort of
more slow, slower release, Cadence, Cadence and more
predictability, more free QA into this ship that will
environment with, you know, in order to stay competitive, you
need to shift and be agile. We had many, many integrations, I
think 20 Plus integrations between authentication,
unfortunately, a bunch of AD, that's the case in their
analytics, and so forth. We have 25 languages to be supported. We
have Android, iOS web, we have big screen, small screen, you
have environmental variables, such as different connectivity
out there. And when you layer all of this together, in each of
these dimensions, you can have bugs that that leak out, and the
system is continuously morphing. So how do we make sure that
nothing is broken out there when we're operating a sort of a
global Multi Language multi platform experience, where our
users are thing that you're, of course, after 24/7, but also
that your features are working. And we found that we had really
good instrumentation sort of lower in the stack. So if you
were to look at monitoring machine data, we had, of course,
you know, data, dog and other solutions, that we're looking
for anomalies and trends. And when you climb up the stack for
the clients, we had app dynamics in there that could look at
what's going on on the client. But then the surface layer,
which is how the product manifests itself, well, you also
have signal you have signal where your user base is actually
telling you what may or may not be working, best part was not
monitored by machines, it was monitored by humans. And we
found that that didn't really scale. Once you get over
10 15,000 pieces of user feedback a month, it gets really
hard to extract signals in real time to the DevOps team. And
other teams in the company so that they can fix bugs faster.
So so that is really the the the idea here with what we're doing
to to provide monitoring, but instead of using machine data,
we're using human data user feedback,
Jason Baum: which at the end of the day, is it's the best
feedback, right?
Christian Wiklund: Well, I would say the one from when a user
takes time out of their day to tell you something, then it's a
pretty, it's a pretty special moment. And we should listen to
what they have to say. And a lot of feedback will be great, you
know, you're getting praised. And I love this product, that is
incredible. And some of it will be that, hey, I can no longer
reset my password, or the equalizer disappeared, or
whatever it may be. And so So we also have to realize that not
every person is going to report feedback. So they wants to do is
you need to make sure that you really take that into account
and make that trickle down into the organization, basically, so
that the right person in your org gets the right information
at the right time. So how do we build a basically a, an
interface between the user base and the company itself so that
we have better flow of information?
Jason Baum: So how did you do that?
Christian Wiklund: So what how we did it in our left company,
we have, when we look at teams that are dealing with user
feedback, we have of course, the support team, for the support
team, they will deflect tickets and resolve issues for customers
and users. And they work sort of in the support ticket buckets.
And then you will have the marketing team. And they are
probably looking at what's going on on Reddit and Twitter, and
other social media platforms. So they're trying to keep tabs on
what's happening there. You likely will have an app store
marketing team that's looking at reviews and trying to derive
insights from that. And then you may have user research user
insights team, or that's typically from some team in the
product management organization that will do surveys, and, and
so forth. And and what we had. And what we also found out there
in the market is that these teams are when it comes to data,
they're siloed. So there is no single source of truth around
all of this user feedback data. So that was the first thing that
we have to solve. So like we need to aggregate every every
channel where users are leaving feedback needs to end up in one
repository. And and then it comes down to how do we how do
we categorize all of this data? So we have, we have customers
such as chime, we have Pinterest, Spotify, and you
know, these these really large and incredible brands, they get
a lot of data or millions of user feedback come in every
month. So in order to make signals actionable, we need
granularity. So I can give an example where if I tell the dev
team, hey, we're seeing an increase in password reset
issues? Well, they're going to say, okay, what can I do with
that? So what is actually breaking? So two bucket eyes is
sort of more generic with password reset, not working.
It's hard to take action on and it creates confusion. It's easy
to brush it off and say hey, maybe they signed up with the
wrong password or the wrong email or whatever. Like what is
the root cause of this stop working? So what we have to do
is we have to break it down into the root cause of why password
reset is not working So that can be, the link didn't work, or I
couldn't pass capture or the email was never delivered. So
when you break it into the root cause, that's when you have
actionability. And in order to break it into those fine
buckets, we we bucket size data in up to 1500, unique packets
for one customer. So we call these quality monitors. So these
quality monitors can be really anything that breaks the
experience, I would say 80% is sort of diggit, some product
related. But we also have customers such as HelloFresh.
And we have Uber and sort of grocery delivery companies where
the bug is not going to be that password reset link broke, it
may be that I ordered bananas, but I got avocados. So we do
capture all of the and we caught we named them quality issues. So
a quality issue is basically the delta between the experience and
the expectation. So when I'm using a product, I expect
something to happen. And then it didn't happen. And I've reported
that we define as a quality issue. And then we bucket high
speed quality issues into these quality monitors. And what then
happens is we can alert on on these different buckets. So if
you see a trend, just as you would do with any monitoring
system going in the wrong direction, we can then ping
different teams on Slack channels and page duty channels
to make sure that they fix the stuff that's breaking out there.
Jason Baum: How do you break it down from expectation? Because
Okay, so that sounds great. Right? In, in, in concept,
that's amazing. You can actually do that. How do you get that
expectation data and match it to what the outcome? How is that?
How does that happen?
Christian Wiklund: So we use machine learning to first and
classify a piece of text as a quality issue, yes or no? So we
have the company. You know, it's interesting, when you look at
quality of product, if you have someone bodies quality of like
every
Jason Baum: so subjective, it's
Christian Wiklund: very subjective. Yes, something that
we live in experience every day. And it's something that we of
course, think is important. Like, if you look at quality of
the product, you know, how do we compete in today's marketplace?
So if if I if we have a music app together, how do we compete
like you compete with features anymore? Well, the feature set
is basically the same across competition. What about content?
Well, content is becoming commodity there, you don't have
different access to content on different music apps, it's
pretty much the same. What about pricing? Well, turns out, you're
going to pay 999 a month for music no matter what. So
quality, quality of products is something that's incredibly
important for how you stand out and compete. So we're sitting,
we're sitting on Zoom, we're not sitting in GoToMeeting. So
that's a good example of GoTo Meeting. They, they they have
less good marketing No, I don't think so. They they have
features that were not as not the same feature set the zoom,
no, it really comes down to that zoom works. Like it's a great
experience. And, and when you look at when you look at quality
impact, top of funnel, so high quality products, they spread
faster, organically. And even more importantly, it impacts the
conversion cycle inside of the product machine. So if you were
to look at sign up to activated user to second day return rates,
seven eight return rate conversion to paid, how many
times they log in per day, what's the session length,
quality is impacting all of those metrics. So we want to
make sure that what you put in the product machine has really
clear signal to the outputs and then you can reinvest sort of
those outputs like revenue and so forth into the beginning of
Jason Baum: today's episode of the humans of DevOps podcast is
sponsored by collide collide is an endpoint security solution
that sends your employees important and timely security
recommendations for their Linux Mac and Windows devices right
inside Slack collide is perfect for organizations that care
deeply about compliance and security, but don't want to get
there by locking down devices to the point where they become
unusable. Instead of frustrating your employees collide educates
them about security, and device management while directing them
to fix important problems. You can try collide with all its
features on an unlimited number of devices. Free for 14 days. No
credit card required. Visit callide.com/h o DEP to sign up
today. That's callide k olid.com/h. O DEP enter your
email when prompted to receive You're free collide gift bundle
after trial activation.
Christian Wiklund: So get back to how we do it. So first, it's
a binary classifier. Is this equality issue? Yes or no. And
we have defined sort of what what equality issue is. So we
have companies on that. And there, there are three layers of
this. Sort of, if you look at it as a pyramid, the bottom of the
pyramid is just functionality. Does the functionality work? Yes
or no? Can I reset password? Yes or no? Can I log in? Yes or no?
Can I upgrade to premium? Yes or no? The next layer is usability.
So there may be a mismatch that perceived as a bug, or a quality
issue. But it's implemented according to spec, but it's just
too hard to use. So that can be used ability that can be
slowness of the product, friction and so forth. And then
on top of the product of the quality pyramid, is just delight
that, hey, there's, there's really no friction, it's a
beautiful experience. And I never, I never seem angry, when
I use this product, it's great.
Jason Baum: Sometimes it just comes down to write it. It's,
they have a saying in football, American football. That's the
best ability is availability. And God that is like such a true
statement for almost anything. But yeah, like you were, you
mentioned, zoom. What happened to Skype? You know, not gonna
happen? Yeah, what happens? This guy go to me. I mean, it's
because Zoom is just, it just works. It's just as simple as
that.
Christian Wiklund: And you know, Jason, what's funny here is, you
ask any, anyone at any company, like if quality of the product
or service, is that important? They all gonna say? Yes. And
then you ask them, Well, how do you measure it? And they say, we
don't? We have no good metrics? And how do you align teams
around goals of quality of the product, if you don't have a
metric if you don't have a single source of truth of what's
impacting the user base. So that's also part of what we've
done here, we have to develop some quality metrics. So what is
the unit Q score, which basically, it's like a credit
score, so it gives you a score from zero to 100, and 100, you
have epic quality, and, and so forth. And that's been eye
opening for a lot of our companies. And to benchmark
themselves, we actually have on our website, you can check the
unit two scores for the 4000 largest apps out there.
Jason Baum: Tell me Tell me about the unit to score a little
bit.
Christian Wiklund: So the way it works is that it measures the
fraction of all public feedback data that we analyze what
fraction contains a quality issue. So if we look at 1000
people feedback, and we find no quality issues reported, then
you get score 100. If 100 of them contains a quality issue,
then you get score 90. So if it's sort of a direct
instrumentation of how much of the public use of feedback data
out there refers to a quality issue. And you know, what, we've
seen it with apps in particular that star ratings is not the
best measurement for quality. Because they are a snapshot in
time. So star ratings, there's an average over a longer period
of time. But also, I'm sure you've seen Jason, like when you
use an app four or five times, then it will say, Hey, can you
rate me five stars? Or hey, would you rate us? Do you like
Jason Baum: us? It's like, it's like a very basic statement,
like, like this app? And cool. Clearly, you're using the app.
So when you say yes.
Christian Wiklund: Do you like us? And the oldest trick in the
book, as you say, hey, please give us one to five stars in the
app. And then if you do three or lower, they will not send you
today, I will record it. Yeah, if you do five or four, they
will send you to the App Store to review it.
Jason Baum: Or on Amazon, people just make up reviews just to be
you know, funny.
Christian Wiklund: Yes. How many times since you ordered vitamins
and is that Oh, give us a five star review and take a picture
of the review. And we'll give you another another box for
free. I mean, it's there's so there's a lot of there's a lot
of stuff going on there. So but we said hey look like if we if
we can't measure something we can improve. So we need to
develop quality metrics. And the industry has been amazing at
building a set of growth metrics so that maybe what's your daily
active users over monthly active users? What your five over seven
retention metric look like? You know, all of these great metrics
to you know, what's your seven day churn look like your 30 day?
One, ADC 60. So I think the industry has been really good at
building out growth metrics that we have put in operations but
quality of service and product lesson there were two metrics,
we developed unit Q Score. And the other one is, we call this
time to fix, which is really like time to resolution. So what
we can see is when something impacts the user base, and we
see the signal that ops people are reporting it, then how
quickly can the company fix it, because the, of course, the
longer you have issues out in production, it impacts the
entire company. So let's take an example of what we had a scout,
we had this ethic bug, where in Polish language on Android,
longitude, latitude was reported in some different format than in
other languages. And our parser couldn't take that. So it
crashed, and scouted the location based app, which means
we asked for location every time you open the app. So no one in
Poland could use the app for six months. And they were reporting
it in the app store. So we got a lot of one star reviews from the
Polish Android users, they were emailing support, they were
clogging the support tickets, you know, they were adding more
tickets that didn't have to be there. If they can't use the
app, then we can't really make any revenue there. They're not
going to be very happy. So of course, we lost out on revenue.
And the signal was there, the issue was that the Polish
Android community was a small piece of our overall community.
And I actually stumbled upon it, I sort of asked the the human in
the loop, the monitoring machine for user feedback, so I stumbled
upon it. And I said, hey, something is going on in Poland.
And it was a parser bug, it took 10 minutes for an engineer to
fix it. And we made $800,000 in revenue from Poland in the next
12 months. So that was sort of the that's like a million dollar
bug. Expensive bug, a very expensive bug. So that's where
we saw these different bugs that were out. And we were always too
slow to react to it. And we started obsessing over it. So
like, I told the team, hey, we probably have $50 million bugs,
we don't know where they are, but let's go find them. So
that's where that's where you know, you need to be able to
measure it, you need to be able to track it. We have companies
using our unit to score as OKRs, they will say, hey, our score is
85. Let's get it to 90. So what are the what are the top 10
things we need to fix Strava is a great customer of ours, they
will do they've done like a bug's squash week where they
will take our data and and then you fix a bunch of bugs and
drive the score up. So they do that on a regular basis. But the
sooner in the cycle, you detect that something is broken. And
the faster you can align the teams that like, Hey, this is
not an anecdote. This is not just one user talking about it,
or this is not a support tickets that that were filed in JIRA.
And I have one example of like, why should I fix this, when you
can align the teams around a single source of truth and look
at the data and say, Well, up until today, no people reporters
that the password reset link was broken. And today, we had 10
people, okay, I get it, it's 10 people out of a million that are
using the product today, but we had zero a lot of days, so
something must have broke. And then if you don't fix it in
maybe five days, you will have hundreds of people every day
reporting it. So get in early fixes, align the teams to get it
done. And then of course, the thing I really love about this
is we give,
we give context to developers who are going to fix the bug. So
imagine you're a developer and someone threw over this JIRA
ticket from support with one example, saying, hey, please go
fix this. Well, I need some context. I want to see the data.
Is this happening on every platform? Is it happening on a
particular device? Is it Is there some evidence here that I
can gather so I can reproduce the bug, because a lot of times
when I'm going to when I'm tasked with fixing a bug, the
first step is how to reproduce it. And a lot of times I can't
reproduce it, it works on my device. So So that's really how,
you know, early detection of issues, aligning cross
functional teams quicker around what needs to be fixed. And then
the actual fixing of the bug, it gets them the evidence, so they
can reproduce faster. And
Jason Baum: so we talked about all the positives, right? So,
but what are the drawbacks of relying on that anecdotal
decision making are there? What are the biggest drawbacks
Christian Wiklund: so anecdotal to me? If so, if you were to
look at what we do is unique curiosity. Really, we take
qualitative data and make it quantitative. So we take all you
know you're sending an email and other people send emails to
support and then some attributes talk about One particular
quality issue. And so that's, that's qualitative data? Well, I
can't make decisions based on one data point. But when you
take this qualitative data and you actually make it into a
trendline, you create data out of it. Now, it's no longer
anecdotes, now we're looking at patterns. anecdotes are really,
really hard, because they, what is the scope of this bug? Where
is it happening? Why should I care? You know, it's one person
out of a million who reported it. So this may be a user error.
And, you know, it's something that that I like to call like,
the great wall between support and engineering, you know,
support from their perspective, they will file a JIRA ticket,
they throw it over the fence, and then they have no idea what
happens, like there is no, there's no feedback loop. And
they keep seeing the same ticket being reported by different
users over and over and over again. So I think there's been
there's been like a breakdown between how these different
teams communicate, and how do they align around a single
source of truth that like, Okay, if it shows up in this trend, in
this product, you need to then we got to jump on in and fix.
And that's what's been really cool with even some of these
really incredible big brands. When we go live, we see sort of
a before and after, where the unit to score always goes up.
And it's not because, you know, it's not because we fix their
bugs, it's because our data tells the company like, Hey,
here's your 1015 Things you need to fix right away. And then as
the as things keep breaking, and guess what every product is,
like a living organism. And a lot of changes is not even
because of your internal changes, you might have an API
that changed. We had a case with scouts where Ukraine banned SSL
on public hotspots. And we had no idea that 30% of our
Ukrainian users were on internet cafes. And our session handler
needs SSL to function. So they couldn't use it on public Wi Fi.
So how do we find that out? Well, the users told us, but
since our manual processes were were couldn't capture it, we
found that by also way, way too late. So So it comes down to
Jason like we're using machines to monitor large datasets lower
in the stack. So we should use it as well, in the surface layer
of the stack, how the product manifests.
Jason Baum: So from from one, one person who follows
experience trends to another one, well, actually a couple
metrics that I have always looked at our net promoter
score, customer satisfaction, and cost, the customer effort
score. Yep. And those are and of course, churn rate retention,
all that those desert and percentage new all that stuff is
obviously that you're going to do that anyway. But those those
three Net Promoter Score, customer satisfaction, customer
effort, to me shape and experience. So it sounds like
I'm missing something.
Christian Wiklund: Well, so the CSET is great, right? They will,
they will raise the experience of the support experience. So
that's a symmetric they, they should obsess over and very much
established their MPs, I think is it's a great snapshot in
time. Right? And there, there are some people out there and
they're like, well, NPS is the best buyers, you know, self
select, self selecting users who respond to the survey and so
forth. But I think
Jason Baum: isn't that true about anything, though? Yeah.
Every survey?
Christian Wiklund: Yeah. Surveys? Yeah. Like you
typically see the most responses on the number eight, you're
number one, right? So it's, it is what it is. But I think NPS
is very much established as as a as a crucial metric. And I think
if now the four operations, you know, how is how is quality of
the product and experience trending today on Android, this
hour on Android, with the latest release on iOS? To have a
snapshot once every quarter of how our NPS is trending. It's
not good enough for operations, like operations, we need data
now. So if we were to look at the unit Q score, and we've done
some studies on this, it's a leading indicator of both MCS
and star rating. Where if, if your score goes down and stays
down consistently, then the next NPS survey you do is going to,
you're going to see that you took a took a hit there, and
vice versa. If you get the units to score up, then you keep it up
consistently. You're going to see NPS going up in the next
survey as well. So So I think NPS is great for getting that
question. really check in. But for daily operations, we need
something that's daily.
Jason Baum: Awesome. I appreciate, you know, all the
time that you've spent with us and talking to us a bit about,
you know, these metrics, and certainly the unit Q metric, I'm
gonna have to check that out for myself. But I really appreciate
you coming on. And one, one last question, which is a question we
asked all of our guests in season two, and I think maybe in
season three will come? We'll have a new question. But since
you're our first guest on season two, or season three, I mean,
let's, let's ask, what is one thing that maybe you do that
that is unique to you, that maybe no one else knows
professionally?
Christian Wiklund: Well, I make quite a lot of music. So that's
one thing, I have been tinkering with analog circuitry since a
very young age. And when I was younger, I even built my own
analog synthesizers and stuff. So I, I love, I love the folding
iron and breaking, you know, when I was a kid, my dad was not
very happy when I dissemble the VHS recorder, and, you know,
then tried to put it back together. And I've never, I've
never, I've never managed to get it back together. But I know
electronics and I love music. And, you know, and that's
really, you know, for me, there were three paths, musician,
entrepreneurship, and I had in my early age, I wanted to be a
quant analyst on Wall Street, I wanted to build pricing models,
but I ended up in Silicon Valley. And that's also what I
love, Jason about software. And music, software. And music is
very similar. In Canada, they go hand in hand, you start with a
blank piece of paper, an empty class file, or an empty track in
Ableton, or Cubase, or whatever you use. You can create stuff,
and I just love that creative process. The most fun part about
software is you can build stuff from nothing. And then you have
the entire world is your marketplace, really, you can you
can, and you have no inventory. There's so much to love about
software.
Jason Baum: You know, it's funny, you're clearly not the
first person who has come on this podcast and is also a
musician, in addition to their life in in tech. So I think we
have talked about it ad nauseum about having like a DevOps
playlist or something like that. And I think we're gonna build it
out. And I might even do another one, another podcast with the
humans of DevOps musicians, because I feel like there's so
many out there. And by the way, what is with you Swedes and
synthesizers, it's like, sweet love.
Christian Wiklund: While here, I can tell you something
brilliant. And yeah, and I think music is top five export
industries for Sweden. For with 10 million people, how come
there are so many great musicians, and not just Abba,
but you know, you have Max Martin, Ace of ACE, Ace of Base,
and, you know, rocks. And of course, you know, the Swedish
House Mafia and all these these great artists. And it's very,
very simple. Some clever politician that way back when
said, Hey, shouldn't private tutoring and music education be
free of charge for every child in freedom. And that's what they
did. So if you want to learn to play the guitar, or the piano,
or the trumpet or whatever, then you get free tutoring from very
young age. And as a result of that, we have a lot of
musicians. And and so the investment they make in giving
free music classes, pays itself off many times older. So I
really love that one. That's awesome. Yeah, I think it's
super cool.
Jason Baum: That's great. Well, thanks again for coming on the
podcast. And it's great having you as a guest sharing. I did
not know that about, about musicians in Sweden. So now I
know. I hope I hope everyone listening learn something, too.
Christian Wiklund: Yeah. Thank you so much, Jason. And I hope
my DevOps team were a little nervous to show Oh, my goodness,
are you gonna hope he doesn't get grilled here, but this was
very pleasant. And thank you very much.
Jason Baum: Awesome. Yeah. No, I do. I one thing I do not do is
grill. You know, it's, I don't I don't believe in any gotcha
question. Maybe next time. I'll think of a few glad gotcha
questions. Just yeah. Thanks again, Christian. And thanks for
listening to this episode of the humans of DevOps podcast. Happy
New Year again. So glad we're back. We're going to have a
bunch of these. You know, we come at you every week with a
new podcast, new guests new topic. So until then, I'm going
to end this episode the same way I always do, encouraging you to
become a member of DevOps Institute to get access to even
more great resources, just like this one. Until next time, stay
safe, stay healthy, and most of all, stay human. We'll see you
on the next episode. Live long and prosper.
Narrator: Thanks for listening to this episode of the humans of
DevOps podcast. Don't forget to join our global community to get
access to even more great resources like this. Until next
time, remember, you are part of something bigger than yourself.
You belong