1 00:00:02,939 --> 00:00:05,819 Narrator: You're listening to the humans of DevOps podcast, a 2 00:00:05,819 --> 00:00:09,449 podcast focused on advancing the humans of DevOps through skills, 3 00:00:09,479 --> 00:00:13,829 knowledge, ideas and learning, or the SK il framework. 4 00:00:34,620 --> 00:00:37,140 Jason Baum: Hey everyone, it's Jason Baum, Director of Member 5 00:00:37,140 --> 00:00:41,310 experience at DevOps Institute. And this is the humans of DevOps 6 00:00:41,310 --> 00:00:45,420 podcast. Happy New Year. And welcome back. Thank you for 7 00:00:45,420 --> 00:00:49,470 joining us for season three of the humans of DevOps podcast. My 8 00:00:49,470 --> 00:00:53,700 second one with you. I'm really excited to head into a new year, 9 00:00:53,700 --> 00:00:57,900 new year new beginnings. And and we're going to shake things up 10 00:00:57,900 --> 00:01:01,500 on the podcast, we're going to do things differently. And today 11 00:01:01,560 --> 00:01:05,460 on the episode, we'll be chatting with founder and CEO of 12 00:01:05,460 --> 00:01:09,750 unit Q. Christian Wicklund, on how DevOps should evolve their 13 00:01:09,750 --> 00:01:13,230 decision making. And I love this topic, so I'm really excited to 14 00:01:13,230 --> 00:01:16,620 talk to Christian about it. Christian believes that customer 15 00:01:16,620 --> 00:01:19,710 feedback is the missing ingredient in how developer 16 00:01:19,710 --> 00:01:23,370 teams prioritize and make decisions. His company unit Q 17 00:01:23,370 --> 00:01:26,490 observes millions of quality issues for consumer tech 18 00:01:26,490 --> 00:01:29,730 products, speaking to their product. Speaking to their 19 00:01:29,730 --> 00:01:33,840 progress, rather, the company also raised 30 million from top 20 00:01:33,840 --> 00:01:38,430 VC firm Excel. Prior to unit Q. He and Nicholas Lindstrom 21 00:01:38,430 --> 00:01:42,120 founded Scout a social app with over 50 million users that was 22 00:01:42,120 --> 00:01:45,660 acquired by the meat group in 2016. Christian is a native 23 00:01:45,660 --> 00:01:49,290 Swede, an avid surfer, a prolific tomato grower and a 24 00:01:49,290 --> 00:01:53,610 proud father of three. Cristian, welcome to the humans of DevOps 25 00:01:53,610 --> 00:01:54,570 podcast. 26 00:01:55,350 --> 00:01:57,000 Christian Wiklund: Thank you, Jason. Happy to be here. 27 00:01:57,510 --> 00:02:00,000 Jason Baum: Awesome. Yeah. And thank you for taking, like a 28 00:02:00,000 --> 00:02:02,850 second in your day with with three kids. I don't know how you 29 00:02:02,850 --> 00:02:05,670 do. I've won. And I don't know how how it's done, especially 30 00:02:05,670 --> 00:02:06,930 right now. So 31 00:02:07,680 --> 00:02:12,660 Christian Wiklund: they say to kids is four times worth one. 32 00:02:12,690 --> 00:02:14,940 And then the third, it doesn't make a difference. So 33 00:02:16,139 --> 00:02:19,469 Jason Baum: just adding to the chaos adding? Yeah, yeah. Yeah. 34 00:02:19,499 --> 00:02:21,479 Couple more, you got a basketball team? So there you 35 00:02:21,479 --> 00:02:27,149 go. So you're ready to get human Christian? Yes. Awesome. Let's 36 00:02:27,149 --> 00:02:30,749 let's just dive right into it. I'm really excited to talk about 37 00:02:30,749 --> 00:02:35,969 this. Because the experience experiences are really, you 38 00:02:35,969 --> 00:02:39,299 know, that's, that's the topic I love to talk about. With DevOps 39 00:02:39,329 --> 00:02:43,289 Institute. I'm the Director of Member experience. And one of 40 00:02:43,289 --> 00:02:47,189 the things that I like to talk about all the time is no more 41 00:02:47,189 --> 00:02:51,029 anecdotal, let's get to the data. And but of course, how do 42 00:02:51,029 --> 00:02:53,459 you get to the data? What are the goals, you should look at? 43 00:02:53,459 --> 00:02:56,039 What are the metrics, you should be talking? Because you can 44 00:02:56,039 --> 00:03:00,119 metric yourself into death? Right? There's so many metrics, 45 00:03:00,119 --> 00:03:03,329 so many things to look at. What how do you get into it? So let's 46 00:03:03,329 --> 00:03:07,499 just start with, why should DevOps prioritize real time 47 00:03:07,499 --> 00:03:09,839 customer feedback in their decision making? 48 00:03:12,120 --> 00:03:15,150 Christian Wiklund: Yes, so maybe I should start with a little 49 00:03:15,180 --> 00:03:20,100 backstory here of why we're building unit Q and and how we 50 00:03:20,100 --> 00:03:23,910 set out on this, this journey to build really the quality 51 00:03:23,910 --> 00:03:28,470 companies. And it started years ago with me and my co founder, 52 00:03:28,470 --> 00:03:34,080 Nicholas, we were building this company called scouts, which 53 00:03:34,470 --> 00:03:39,720 still is a social network for meeting people. And if was a 54 00:03:39,720 --> 00:03:43,140 mobile first company, which sounds not very interesting 55 00:03:43,140 --> 00:03:47,670 today, but back then in 2009, and, and 10, it was pretty 56 00:03:47,670 --> 00:03:51,150 special, you know, that you started with mobile. And what we 57 00:03:51,150 --> 00:03:55,290 discover there was, you know, as we were moving from, sort of 58 00:03:55,290 --> 00:04:00,360 more slow, slower release, Cadence, Cadence and more 59 00:04:00,360 --> 00:04:03,870 predictability, more free QA into this ship that will 60 00:04:03,870 --> 00:04:06,960 environment with, you know, in order to stay competitive, you 61 00:04:06,960 --> 00:04:10,500 need to shift and be agile. We had many, many integrations, I 62 00:04:10,500 --> 00:04:14,670 think 20 Plus integrations between authentication, 63 00:04:14,730 --> 00:04:17,220 unfortunately, a bunch of AD, that's the case in their 64 00:04:17,520 --> 00:04:23,010 analytics, and so forth. We have 25 languages to be supported. We 65 00:04:23,010 --> 00:04:28,200 have Android, iOS web, we have big screen, small screen, you 66 00:04:28,200 --> 00:04:32,610 have environmental variables, such as different connectivity 67 00:04:32,610 --> 00:04:37,800 out there. And when you layer all of this together, in each of 68 00:04:37,800 --> 00:04:42,990 these dimensions, you can have bugs that that leak out, and the 69 00:04:42,990 --> 00:04:47,970 system is continuously morphing. So how do we make sure that 70 00:04:48,390 --> 00:04:53,190 nothing is broken out there when we're operating a sort of a 71 00:04:53,190 --> 00:04:59,220 global Multi Language multi platform experience, where our 72 00:04:59,220 --> 00:05:04,290 users are thing that you're, of course, after 24/7, but also 73 00:05:04,290 --> 00:05:10,710 that your features are working. And we found that we had really 74 00:05:10,710 --> 00:05:13,800 good instrumentation sort of lower in the stack. So if you 75 00:05:13,800 --> 00:05:18,210 were to look at monitoring machine data, we had, of course, 76 00:05:18,240 --> 00:05:23,970 you know, data, dog and other solutions, that we're looking 77 00:05:23,970 --> 00:05:27,300 for anomalies and trends. And when you climb up the stack for 78 00:05:27,300 --> 00:05:32,490 the clients, we had app dynamics in there that could look at 79 00:05:32,490 --> 00:05:35,250 what's going on on the client. But then the surface layer, 80 00:05:35,280 --> 00:05:38,730 which is how the product manifests itself, well, you also 81 00:05:38,730 --> 00:05:42,180 have signal you have signal where your user base is actually 82 00:05:42,660 --> 00:05:46,620 telling you what may or may not be working, best part was not 83 00:05:46,620 --> 00:05:52,470 monitored by machines, it was monitored by humans. And we 84 00:05:52,500 --> 00:05:57,480 found that that didn't really scale. Once you get over 85 00:05:57,570 --> 00:06:01,860 10 15,000 pieces of user feedback a month, it gets really 86 00:06:01,860 --> 00:06:08,040 hard to extract signals in real time to the DevOps team. And 87 00:06:08,040 --> 00:06:11,400 other teams in the company so that they can fix bugs faster. 88 00:06:11,400 --> 00:06:15,270 So so that is really the the the idea here with what we're doing 89 00:06:15,270 --> 00:06:19,260 to to provide monitoring, but instead of using machine data, 90 00:06:19,290 --> 00:06:21,600 we're using human data user feedback, 91 00:06:23,160 --> 00:06:25,410 Jason Baum: which at the end of the day, is it's the best 92 00:06:25,410 --> 00:06:26,190 feedback, right? 93 00:06:27,930 --> 00:06:32,400 Christian Wiklund: Well, I would say the one from when a user 94 00:06:32,400 --> 00:06:37,830 takes time out of their day to tell you something, then it's a 95 00:06:37,830 --> 00:06:40,980 pretty, it's a pretty special moment. And we should listen to 96 00:06:40,980 --> 00:06:46,560 what they have to say. And a lot of feedback will be great, you 97 00:06:46,560 --> 00:06:48,600 know, you're getting praised. And I love this product, that is 98 00:06:48,600 --> 00:06:53,730 incredible. And some of it will be that, hey, I can no longer 99 00:06:53,790 --> 00:06:58,620 reset my password, or the equalizer disappeared, or 100 00:06:58,620 --> 00:07:05,790 whatever it may be. And so So we also have to realize that not 101 00:07:05,850 --> 00:07:09,960 every person is going to report feedback. So they wants to do is 102 00:07:09,960 --> 00:07:12,240 you need to make sure that you really take that into account 103 00:07:12,270 --> 00:07:15,630 and make that trickle down into the organization, basically, so 104 00:07:15,630 --> 00:07:18,600 that the right person in your org gets the right information 105 00:07:18,600 --> 00:07:22,020 at the right time. So how do we build a basically a, an 106 00:07:22,020 --> 00:07:25,350 interface between the user base and the company itself so that 107 00:07:25,350 --> 00:07:26,970 we have better flow of information? 108 00:07:28,020 --> 00:07:29,250 Jason Baum: So how did you do that? 109 00:07:31,320 --> 00:07:34,860 Christian Wiklund: So what how we did it in our left company, 110 00:07:35,550 --> 00:07:39,750 we have, when we look at teams that are dealing with user 111 00:07:39,750 --> 00:07:43,050 feedback, we have of course, the support team, for the support 112 00:07:43,050 --> 00:07:47,010 team, they will deflect tickets and resolve issues for customers 113 00:07:47,010 --> 00:07:51,360 and users. And they work sort of in the support ticket buckets. 114 00:07:51,900 --> 00:07:55,950 And then you will have the marketing team. And they are 115 00:07:55,950 --> 00:08:01,590 probably looking at what's going on on Reddit and Twitter, and 116 00:08:01,590 --> 00:08:04,320 other social media platforms. So they're trying to keep tabs on 117 00:08:04,320 --> 00:08:08,370 what's happening there. You likely will have an app store 118 00:08:08,370 --> 00:08:11,640 marketing team that's looking at reviews and trying to derive 119 00:08:11,640 --> 00:08:15,810 insights from that. And then you may have user research user 120 00:08:15,810 --> 00:08:19,680 insights team, or that's typically from some team in the 121 00:08:19,680 --> 00:08:24,720 product management organization that will do surveys, and, and 122 00:08:24,720 --> 00:08:28,320 so forth. And and what we had. And what we also found out there 123 00:08:28,320 --> 00:08:32,580 in the market is that these teams are when it comes to data, 124 00:08:32,580 --> 00:08:36,090 they're siloed. So there is no single source of truth around 125 00:08:36,330 --> 00:08:39,690 all of this user feedback data. So that was the first thing that 126 00:08:39,690 --> 00:08:43,590 we have to solve. So like we need to aggregate every every 127 00:08:43,590 --> 00:08:46,830 channel where users are leaving feedback needs to end up in one 128 00:08:46,830 --> 00:08:51,990 repository. And and then it comes down to how do we how do 129 00:08:51,990 --> 00:08:56,040 we categorize all of this data? So we have, we have customers 130 00:08:56,040 --> 00:09:01,620 such as chime, we have Pinterest, Spotify, and you 131 00:09:01,620 --> 00:09:05,310 know, these these really large and incredible brands, they get 132 00:09:05,460 --> 00:09:10,770 a lot of data or millions of user feedback come in every 133 00:09:10,770 --> 00:09:18,090 month. So in order to make signals actionable, we need 134 00:09:18,090 --> 00:09:22,710 granularity. So I can give an example where if I tell the dev 135 00:09:22,710 --> 00:09:28,770 team, hey, we're seeing an increase in password reset 136 00:09:29,160 --> 00:09:34,770 issues? Well, they're going to say, okay, what can I do with 137 00:09:34,770 --> 00:09:39,150 that? So what is actually breaking? So two bucket eyes is 138 00:09:39,270 --> 00:09:41,910 sort of more generic with password reset, not working. 139 00:09:41,940 --> 00:09:45,600 It's hard to take action on and it creates confusion. It's easy 140 00:09:45,600 --> 00:09:48,900 to brush it off and say hey, maybe they signed up with the 141 00:09:48,900 --> 00:09:52,260 wrong password or the wrong email or whatever. Like what is 142 00:09:52,260 --> 00:09:55,230 the root cause of this stop working? So what we have to do 143 00:09:55,230 --> 00:09:58,320 is we have to break it down into the root cause of why password 144 00:09:58,320 --> 00:10:03,210 reset is not working So that can be, the link didn't work, or I 145 00:10:03,210 --> 00:10:07,140 couldn't pass capture or the email was never delivered. So 146 00:10:07,140 --> 00:10:09,570 when you break it into the root cause, that's when you have 147 00:10:09,570 --> 00:10:14,010 actionability. And in order to break it into those fine 148 00:10:14,010 --> 00:10:21,450 buckets, we we bucket size data in up to 1500, unique packets 149 00:10:21,480 --> 00:10:26,610 for one customer. So we call these quality monitors. So these 150 00:10:26,610 --> 00:10:30,180 quality monitors can be really anything that breaks the 151 00:10:30,180 --> 00:10:35,250 experience, I would say 80% is sort of diggit, some product 152 00:10:35,250 --> 00:10:38,880 related. But we also have customers such as HelloFresh. 153 00:10:38,910 --> 00:10:44,700 And we have Uber and sort of grocery delivery companies where 154 00:10:44,760 --> 00:10:47,790 the bug is not going to be that password reset link broke, it 155 00:10:47,790 --> 00:10:51,600 may be that I ordered bananas, but I got avocados. So we do 156 00:10:51,600 --> 00:10:55,410 capture all of the and we caught we named them quality issues. So 157 00:10:55,410 --> 00:11:01,170 a quality issue is basically the delta between the experience and 158 00:11:01,170 --> 00:11:04,860 the expectation. So when I'm using a product, I expect 159 00:11:04,860 --> 00:11:07,920 something to happen. And then it didn't happen. And I've reported 160 00:11:08,370 --> 00:11:12,600 that we define as a quality issue. And then we bucket high 161 00:11:12,600 --> 00:11:17,700 speed quality issues into these quality monitors. And what then 162 00:11:17,700 --> 00:11:22,170 happens is we can alert on on these different buckets. So if 163 00:11:22,170 --> 00:11:24,960 you see a trend, just as you would do with any monitoring 164 00:11:24,960 --> 00:11:27,630 system going in the wrong direction, we can then ping 165 00:11:27,630 --> 00:11:31,080 different teams on Slack channels and page duty channels 166 00:11:31,800 --> 00:11:34,770 to make sure that they fix the stuff that's breaking out there. 167 00:11:35,100 --> 00:11:38,340 Jason Baum: How do you break it down from expectation? Because 168 00:11:38,340 --> 00:11:42,840 Okay, so that sounds great. Right? In, in, in concept, 169 00:11:42,870 --> 00:11:47,310 that's amazing. You can actually do that. How do you get that 170 00:11:47,310 --> 00:11:52,440 expectation data and match it to what the outcome? How is that? 171 00:11:52,440 --> 00:11:53,340 How does that happen? 172 00:11:55,019 --> 00:11:58,319 Christian Wiklund: So we use machine learning to first and 173 00:11:58,319 --> 00:12:03,839 classify a piece of text as a quality issue, yes or no? So we 174 00:12:03,839 --> 00:12:06,029 have the company. You know, it's interesting, when you look at 175 00:12:06,029 --> 00:12:08,819 quality of product, if you have someone bodies quality of like 176 00:12:08,819 --> 00:12:09,479 every 177 00:12:09,630 --> 00:12:11,430 Jason Baum: so subjective, it's 178 00:12:11,429 --> 00:12:13,109 Christian Wiklund: very subjective. Yes, something that 179 00:12:13,109 --> 00:12:17,129 we live in experience every day. And it's something that we of 180 00:12:17,129 --> 00:12:20,759 course, think is important. Like, if you look at quality of 181 00:12:20,759 --> 00:12:24,389 the product, you know, how do we compete in today's marketplace? 182 00:12:24,389 --> 00:12:29,579 So if if I if we have a music app together, how do we compete 183 00:12:29,639 --> 00:12:33,659 like you compete with features anymore? Well, the feature set 184 00:12:33,659 --> 00:12:37,139 is basically the same across competition. What about content? 185 00:12:37,169 --> 00:12:40,019 Well, content is becoming commodity there, you don't have 186 00:12:40,019 --> 00:12:42,509 different access to content on different music apps, it's 187 00:12:42,509 --> 00:12:46,229 pretty much the same. What about pricing? Well, turns out, you're 188 00:12:46,229 --> 00:12:50,309 going to pay 999 a month for music no matter what. So 189 00:12:50,339 --> 00:12:54,029 quality, quality of products is something that's incredibly 190 00:12:54,029 --> 00:12:57,569 important for how you stand out and compete. So we're sitting, 191 00:12:57,899 --> 00:13:00,569 we're sitting on Zoom, we're not sitting in GoToMeeting. So 192 00:13:00,599 --> 00:13:04,619 that's a good example of GoTo Meeting. They, they they have 193 00:13:04,739 --> 00:13:07,739 less good marketing No, I don't think so. They they have 194 00:13:08,099 --> 00:13:12,839 features that were not as not the same feature set the zoom, 195 00:13:12,839 --> 00:13:16,619 no, it really comes down to that zoom works. Like it's a great 196 00:13:16,619 --> 00:13:20,549 experience. And, and when you look at when you look at quality 197 00:13:20,579 --> 00:13:25,529 impact, top of funnel, so high quality products, they spread 198 00:13:25,529 --> 00:13:30,209 faster, organically. And even more importantly, it impacts the 199 00:13:30,209 --> 00:13:34,229 conversion cycle inside of the product machine. So if you were 200 00:13:34,229 --> 00:13:40,109 to look at sign up to activated user to second day return rates, 201 00:13:40,109 --> 00:13:43,889 seven eight return rate conversion to paid, how many 202 00:13:43,889 --> 00:13:46,019 times they log in per day, what's the session length, 203 00:13:46,229 --> 00:13:50,459 quality is impacting all of those metrics. So we want to 204 00:13:50,459 --> 00:13:53,459 make sure that what you put in the product machine has really 205 00:13:53,459 --> 00:13:56,519 clear signal to the outputs and then you can reinvest sort of 206 00:13:56,519 --> 00:13:59,669 those outputs like revenue and so forth into the beginning of 207 00:14:01,470 --> 00:14:04,410 Jason Baum: today's episode of the humans of DevOps podcast is 208 00:14:04,410 --> 00:14:08,430 sponsored by collide collide is an endpoint security solution 209 00:14:08,430 --> 00:14:11,580 that sends your employees important and timely security 210 00:14:11,580 --> 00:14:15,420 recommendations for their Linux Mac and Windows devices right 211 00:14:15,420 --> 00:14:19,320 inside Slack collide is perfect for organizations that care 212 00:14:19,350 --> 00:14:23,310 deeply about compliance and security, but don't want to get 213 00:14:23,310 --> 00:14:26,100 there by locking down devices to the point where they become 214 00:14:26,130 --> 00:14:30,330 unusable. Instead of frustrating your employees collide educates 215 00:14:30,330 --> 00:14:33,990 them about security, and device management while directing them 216 00:14:33,990 --> 00:14:37,830 to fix important problems. You can try collide with all its 217 00:14:37,830 --> 00:14:43,200 features on an unlimited number of devices. Free for 14 days. No 218 00:14:43,200 --> 00:14:49,530 credit card required. Visit callide.com/h o DEP to sign up 219 00:14:49,530 --> 00:14:58,620 today. That's callide k olid.com/h. O DEP enter your 220 00:14:58,650 --> 00:15:01,860 email when prompted to receive You're free collide gift bundle 221 00:15:01,890 --> 00:15:03,450 after trial activation. 222 00:15:06,089 --> 00:15:08,609 Christian Wiklund: So get back to how we do it. So first, it's 223 00:15:08,639 --> 00:15:12,089 a binary classifier. Is this equality issue? Yes or no. And 224 00:15:12,089 --> 00:15:16,079 we have defined sort of what what equality issue is. So we 225 00:15:16,079 --> 00:15:19,469 have companies on that. And there, there are three layers of 226 00:15:19,469 --> 00:15:23,459 this. Sort of, if you look at it as a pyramid, the bottom of the 227 00:15:23,459 --> 00:15:29,129 pyramid is just functionality. Does the functionality work? Yes 228 00:15:29,129 --> 00:15:33,959 or no? Can I reset password? Yes or no? Can I log in? Yes or no? 229 00:15:34,229 --> 00:15:39,809 Can I upgrade to premium? Yes or no? The next layer is usability. 230 00:15:39,839 --> 00:15:45,659 So there may be a mismatch that perceived as a bug, or a quality 231 00:15:45,659 --> 00:15:49,199 issue. But it's implemented according to spec, but it's just 232 00:15:49,199 --> 00:15:51,689 too hard to use. So that can be used ability that can be 233 00:15:51,749 --> 00:15:55,499 slowness of the product, friction and so forth. And then 234 00:15:55,499 --> 00:15:59,939 on top of the product of the quality pyramid, is just delight 235 00:16:00,029 --> 00:16:02,759 that, hey, there's, there's really no friction, it's a 236 00:16:02,759 --> 00:16:07,559 beautiful experience. And I never, I never seem angry, when 237 00:16:07,559 --> 00:16:09,119 I use this product, it's great. 238 00:16:10,380 --> 00:16:13,440 Jason Baum: Sometimes it just comes down to write it. It's, 239 00:16:13,770 --> 00:16:19,050 they have a saying in football, American football. That's the 240 00:16:19,050 --> 00:16:23,190 best ability is availability. And God that is like such a true 241 00:16:23,190 --> 00:16:25,920 statement for almost anything. But yeah, like you were, you 242 00:16:25,920 --> 00:16:30,210 mentioned, zoom. What happened to Skype? You know, not gonna 243 00:16:30,210 --> 00:16:33,420 happen? Yeah, what happens? This guy go to me. I mean, it's 244 00:16:33,420 --> 00:16:38,280 because Zoom is just, it just works. It's just as simple as 245 00:16:38,280 --> 00:16:38,730 that. 246 00:16:39,600 --> 00:16:42,360 Christian Wiklund: And you know, Jason, what's funny here is, you 247 00:16:42,360 --> 00:16:46,800 ask any, anyone at any company, like if quality of the product 248 00:16:46,890 --> 00:16:51,000 or service, is that important? They all gonna say? Yes. And 249 00:16:51,000 --> 00:16:55,050 then you ask them, Well, how do you measure it? And they say, we 250 00:16:55,050 --> 00:17:00,120 don't? We have no good metrics? And how do you align teams 251 00:17:00,150 --> 00:17:03,570 around goals of quality of the product, if you don't have a 252 00:17:03,570 --> 00:17:06,360 metric if you don't have a single source of truth of what's 253 00:17:06,390 --> 00:17:09,420 impacting the user base. So that's also part of what we've 254 00:17:09,420 --> 00:17:13,470 done here, we have to develop some quality metrics. So what is 255 00:17:13,470 --> 00:17:16,230 the unit Q score, which basically, it's like a credit 256 00:17:16,230 --> 00:17:19,890 score, so it gives you a score from zero to 100, and 100, you 257 00:17:19,890 --> 00:17:25,140 have epic quality, and, and so forth. And that's been eye 258 00:17:25,140 --> 00:17:28,740 opening for a lot of our companies. And to benchmark 259 00:17:28,740 --> 00:17:31,500 themselves, we actually have on our website, you can check the 260 00:17:31,500 --> 00:17:35,130 unit two scores for the 4000 largest apps out there. 261 00:17:35,700 --> 00:17:38,400 Jason Baum: Tell me Tell me about the unit to score a little 262 00:17:38,400 --> 00:17:38,790 bit. 263 00:17:40,230 --> 00:17:43,920 Christian Wiklund: So the way it works is that it measures the 264 00:17:44,190 --> 00:17:48,510 fraction of all public feedback data that we analyze what 265 00:17:48,510 --> 00:17:52,740 fraction contains a quality issue. So if we look at 1000 266 00:17:52,740 --> 00:17:56,970 people feedback, and we find no quality issues reported, then 267 00:17:56,970 --> 00:18:01,230 you get score 100. If 100 of them contains a quality issue, 268 00:18:01,260 --> 00:18:05,940 then you get score 90. So if it's sort of a direct 269 00:18:05,970 --> 00:18:10,350 instrumentation of how much of the public use of feedback data 270 00:18:10,380 --> 00:18:14,850 out there refers to a quality issue. And you know, what, we've 271 00:18:14,850 --> 00:18:20,010 seen it with apps in particular that star ratings is not the 272 00:18:20,010 --> 00:18:25,620 best measurement for quality. Because they are a snapshot in 273 00:18:25,620 --> 00:18:29,040 time. So star ratings, there's an average over a longer period 274 00:18:29,040 --> 00:18:32,100 of time. But also, I'm sure you've seen Jason, like when you 275 00:18:32,100 --> 00:18:34,800 use an app four or five times, then it will say, Hey, can you 276 00:18:34,800 --> 00:18:38,760 rate me five stars? Or hey, would you rate us? Do you like 277 00:18:38,760 --> 00:18:42,150 Jason Baum: us? It's like, it's like a very basic statement, 278 00:18:42,150 --> 00:18:44,970 like, like this app? And cool. Clearly, you're using the app. 279 00:18:44,970 --> 00:18:47,430 So when you say yes. 280 00:18:47,790 --> 00:18:49,770 Christian Wiklund: Do you like us? And the oldest trick in the 281 00:18:49,770 --> 00:18:53,340 book, as you say, hey, please give us one to five stars in the 282 00:18:53,340 --> 00:18:56,760 app. And then if you do three or lower, they will not send you 283 00:18:56,760 --> 00:19:00,390 today, I will record it. Yeah, if you do five or four, they 284 00:19:00,390 --> 00:19:01,890 will send you to the App Store to review it. 285 00:19:02,220 --> 00:19:07,410 Jason Baum: Or on Amazon, people just make up reviews just to be 286 00:19:07,440 --> 00:19:08,760 you know, funny. 287 00:19:09,510 --> 00:19:12,270 Christian Wiklund: Yes. How many times since you ordered vitamins 288 00:19:12,300 --> 00:19:16,200 and is that Oh, give us a five star review and take a picture 289 00:19:16,200 --> 00:19:18,750 of the review. And we'll give you another another box for 290 00:19:18,750 --> 00:19:22,080 free. I mean, it's there's so there's a lot of there's a lot 291 00:19:22,080 --> 00:19:25,320 of stuff going on there. So but we said hey look like if we if 292 00:19:25,320 --> 00:19:28,110 we can't measure something we can improve. So we need to 293 00:19:28,110 --> 00:19:31,170 develop quality metrics. And the industry has been amazing at 294 00:19:31,200 --> 00:19:35,790 building a set of growth metrics so that maybe what's your daily 295 00:19:35,790 --> 00:19:40,590 active users over monthly active users? What your five over seven 296 00:19:40,890 --> 00:19:44,610 retention metric look like? You know, all of these great metrics 297 00:19:44,610 --> 00:19:48,750 to you know, what's your seven day churn look like your 30 day? 298 00:19:49,020 --> 00:19:52,230 One, ADC 60. So I think the industry has been really good at 299 00:19:52,410 --> 00:19:56,160 building out growth metrics that we have put in operations but 300 00:19:56,460 --> 00:20:00,540 quality of service and product lesson there were two metrics, 301 00:20:00,540 --> 00:20:04,830 we developed unit Q Score. And the other one is, we call this 302 00:20:04,830 --> 00:20:08,310 time to fix, which is really like time to resolution. So what 303 00:20:08,310 --> 00:20:12,030 we can see is when something impacts the user base, and we 304 00:20:12,030 --> 00:20:16,500 see the signal that ops people are reporting it, then how 305 00:20:16,500 --> 00:20:20,790 quickly can the company fix it, because the, of course, the 306 00:20:20,790 --> 00:20:23,970 longer you have issues out in production, it impacts the 307 00:20:23,970 --> 00:20:31,350 entire company. So let's take an example of what we had a scout, 308 00:20:31,380 --> 00:20:35,970 we had this ethic bug, where in Polish language on Android, 309 00:20:36,660 --> 00:20:41,370 longitude, latitude was reported in some different format than in 310 00:20:41,370 --> 00:20:45,570 other languages. And our parser couldn't take that. So it 311 00:20:45,570 --> 00:20:49,350 crashed, and scouted the location based app, which means 312 00:20:49,350 --> 00:20:53,220 we asked for location every time you open the app. So no one in 313 00:20:53,220 --> 00:20:58,380 Poland could use the app for six months. And they were reporting 314 00:20:58,380 --> 00:21:04,050 it in the app store. So we got a lot of one star reviews from the 315 00:21:04,050 --> 00:21:07,470 Polish Android users, they were emailing support, they were 316 00:21:07,470 --> 00:21:10,500 clogging the support tickets, you know, they were adding more 317 00:21:10,500 --> 00:21:14,250 tickets that didn't have to be there. If they can't use the 318 00:21:14,250 --> 00:21:18,420 app, then we can't really make any revenue there. They're not 319 00:21:18,420 --> 00:21:21,210 going to be very happy. So of course, we lost out on revenue. 320 00:21:21,840 --> 00:21:26,010 And the signal was there, the issue was that the Polish 321 00:21:26,040 --> 00:21:30,660 Android community was a small piece of our overall community. 322 00:21:32,130 --> 00:21:37,650 And I actually stumbled upon it, I sort of asked the the human in 323 00:21:37,650 --> 00:21:40,950 the loop, the monitoring machine for user feedback, so I stumbled 324 00:21:40,950 --> 00:21:44,190 upon it. And I said, hey, something is going on in Poland. 325 00:21:44,610 --> 00:21:47,850 And it was a parser bug, it took 10 minutes for an engineer to 326 00:21:47,850 --> 00:21:54,270 fix it. And we made $800,000 in revenue from Poland in the next 327 00:21:54,270 --> 00:21:56,760 12 months. So that was sort of the that's like a million dollar 328 00:21:56,760 --> 00:22:02,580 bug. Expensive bug, a very expensive bug. So that's where 329 00:22:02,610 --> 00:22:05,610 we saw these different bugs that were out. And we were always too 330 00:22:05,610 --> 00:22:09,870 slow to react to it. And we started obsessing over it. So 331 00:22:09,870 --> 00:22:13,980 like, I told the team, hey, we probably have $50 million bugs, 332 00:22:13,980 --> 00:22:18,780 we don't know where they are, but let's go find them. So 333 00:22:18,780 --> 00:22:21,270 that's where that's where you know, you need to be able to 334 00:22:21,270 --> 00:22:24,300 measure it, you need to be able to track it. We have companies 335 00:22:24,330 --> 00:22:28,200 using our unit to score as OKRs, they will say, hey, our score is 336 00:22:28,620 --> 00:22:32,340 85. Let's get it to 90. So what are the what are the top 10 337 00:22:32,340 --> 00:22:35,910 things we need to fix Strava is a great customer of ours, they 338 00:22:35,910 --> 00:22:40,440 will do they've done like a bug's squash week where they 339 00:22:40,440 --> 00:22:45,060 will take our data and and then you fix a bunch of bugs and 340 00:22:45,060 --> 00:22:48,330 drive the score up. So they do that on a regular basis. But the 341 00:22:48,330 --> 00:22:53,700 sooner in the cycle, you detect that something is broken. And 342 00:22:53,700 --> 00:22:57,630 the faster you can align the teams that like, Hey, this is 343 00:22:57,630 --> 00:23:01,380 not an anecdote. This is not just one user talking about it, 344 00:23:01,380 --> 00:23:06,030 or this is not a support tickets that that were filed in JIRA. 345 00:23:06,090 --> 00:23:10,170 And I have one example of like, why should I fix this, when you 346 00:23:10,170 --> 00:23:13,230 can align the teams around a single source of truth and look 347 00:23:13,230 --> 00:23:18,390 at the data and say, Well, up until today, no people reporters 348 00:23:18,390 --> 00:23:22,200 that the password reset link was broken. And today, we had 10 349 00:23:22,200 --> 00:23:25,950 people, okay, I get it, it's 10 people out of a million that are 350 00:23:25,950 --> 00:23:29,010 using the product today, but we had zero a lot of days, so 351 00:23:29,010 --> 00:23:32,670 something must have broke. And then if you don't fix it in 352 00:23:32,670 --> 00:23:35,700 maybe five days, you will have hundreds of people every day 353 00:23:35,700 --> 00:23:40,770 reporting it. So get in early fixes, align the teams to get it 354 00:23:40,770 --> 00:23:44,610 done. And then of course, the thing I really love about this 355 00:23:44,610 --> 00:23:45,300 is we give, 356 00:23:46,710 --> 00:23:49,740 we give context to developers who are going to fix the bug. So 357 00:23:49,740 --> 00:23:52,650 imagine you're a developer and someone threw over this JIRA 358 00:23:52,650 --> 00:23:56,160 ticket from support with one example, saying, hey, please go 359 00:23:56,160 --> 00:23:59,820 fix this. Well, I need some context. I want to see the data. 360 00:23:59,820 --> 00:24:03,600 Is this happening on every platform? Is it happening on a 361 00:24:03,600 --> 00:24:07,860 particular device? Is it Is there some evidence here that I 362 00:24:07,860 --> 00:24:10,890 can gather so I can reproduce the bug, because a lot of times 363 00:24:10,890 --> 00:24:13,800 when I'm going to when I'm tasked with fixing a bug, the 364 00:24:13,800 --> 00:24:16,830 first step is how to reproduce it. And a lot of times I can't 365 00:24:16,830 --> 00:24:20,820 reproduce it, it works on my device. So So that's really how, 366 00:24:20,820 --> 00:24:23,730 you know, early detection of issues, aligning cross 367 00:24:23,730 --> 00:24:27,420 functional teams quicker around what needs to be fixed. And then 368 00:24:27,420 --> 00:24:30,900 the actual fixing of the bug, it gets them the evidence, so they 369 00:24:30,900 --> 00:24:31,920 can reproduce faster. And 370 00:24:33,150 --> 00:24:36,060 Jason Baum: so we talked about all the positives, right? So, 371 00:24:36,270 --> 00:24:39,810 but what are the drawbacks of relying on that anecdotal 372 00:24:39,870 --> 00:24:43,200 decision making are there? What are the biggest drawbacks 373 00:24:44,430 --> 00:24:47,670 Christian Wiklund: so anecdotal to me? If so, if you were to 374 00:24:47,670 --> 00:24:50,490 look at what we do is unique curiosity. Really, we take 375 00:24:51,990 --> 00:24:55,890 qualitative data and make it quantitative. So we take all you 376 00:24:55,890 --> 00:24:58,290 know you're sending an email and other people send emails to 377 00:24:58,290 --> 00:25:00,540 support and then some attributes talk about One particular 378 00:25:00,540 --> 00:25:05,640 quality issue. And so that's, that's qualitative data? Well, I 379 00:25:05,640 --> 00:25:09,210 can't make decisions based on one data point. But when you 380 00:25:09,210 --> 00:25:13,410 take this qualitative data and you actually make it into a 381 00:25:13,410 --> 00:25:16,680 trendline, you create data out of it. Now, it's no longer 382 00:25:16,680 --> 00:25:21,600 anecdotes, now we're looking at patterns. anecdotes are really, 383 00:25:21,600 --> 00:25:27,780 really hard, because they, what is the scope of this bug? Where 384 00:25:27,780 --> 00:25:31,500 is it happening? Why should I care? You know, it's one person 385 00:25:31,530 --> 00:25:34,920 out of a million who reported it. So this may be a user error. 386 00:25:35,640 --> 00:25:38,520 And, you know, it's something that that I like to call like, 387 00:25:38,520 --> 00:25:42,030 the great wall between support and engineering, you know, 388 00:25:42,030 --> 00:25:45,750 support from their perspective, they will file a JIRA ticket, 389 00:25:45,810 --> 00:25:49,440 they throw it over the fence, and then they have no idea what 390 00:25:49,440 --> 00:25:53,220 happens, like there is no, there's no feedback loop. And 391 00:25:53,220 --> 00:25:55,710 they keep seeing the same ticket being reported by different 392 00:25:55,710 --> 00:25:58,710 users over and over and over again. So I think there's been 393 00:25:59,550 --> 00:26:01,920 there's been like a breakdown between how these different 394 00:26:01,920 --> 00:26:05,970 teams communicate, and how do they align around a single 395 00:26:05,970 --> 00:26:10,950 source of truth that like, Okay, if it shows up in this trend, in 396 00:26:10,950 --> 00:26:13,710 this product, you need to then we got to jump on in and fix. 397 00:26:14,430 --> 00:26:18,300 And that's what's been really cool with even some of these 398 00:26:18,300 --> 00:26:21,630 really incredible big brands. When we go live, we see sort of 399 00:26:21,630 --> 00:26:25,290 a before and after, where the unit to score always goes up. 400 00:26:25,530 --> 00:26:28,710 And it's not because, you know, it's not because we fix their 401 00:26:28,710 --> 00:26:31,470 bugs, it's because our data tells the company like, Hey, 402 00:26:31,500 --> 00:26:35,370 here's your 1015 Things you need to fix right away. And then as 403 00:26:35,370 --> 00:26:39,420 the as things keep breaking, and guess what every product is, 404 00:26:40,200 --> 00:26:44,100 like a living organism. And a lot of changes is not even 405 00:26:44,940 --> 00:26:47,970 because of your internal changes, you might have an API 406 00:26:47,970 --> 00:26:54,480 that changed. We had a case with scouts where Ukraine banned SSL 407 00:26:54,510 --> 00:26:59,280 on public hotspots. And we had no idea that 30% of our 408 00:26:59,280 --> 00:27:04,050 Ukrainian users were on internet cafes. And our session handler 409 00:27:04,290 --> 00:27:08,040 needs SSL to function. So they couldn't use it on public Wi Fi. 410 00:27:08,310 --> 00:27:12,480 So how do we find that out? Well, the users told us, but 411 00:27:12,480 --> 00:27:15,810 since our manual processes were were couldn't capture it, we 412 00:27:15,810 --> 00:27:20,700 found that by also way, way too late. So So it comes down to 413 00:27:20,700 --> 00:27:25,980 Jason like we're using machines to monitor large datasets lower 414 00:27:25,980 --> 00:27:30,900 in the stack. So we should use it as well, in the surface layer 415 00:27:30,930 --> 00:27:32,730 of the stack, how the product manifests. 416 00:27:34,230 --> 00:27:37,650 Jason Baum: So from from one, one person who follows 417 00:27:37,650 --> 00:27:42,720 experience trends to another one, well, actually a couple 418 00:27:42,720 --> 00:27:46,110 metrics that I have always looked at our net promoter 419 00:27:46,110 --> 00:27:52,440 score, customer satisfaction, and cost, the customer effort 420 00:27:52,440 --> 00:27:57,750 score. Yep. And those are and of course, churn rate retention, 421 00:27:57,750 --> 00:28:01,710 all that those desert and percentage new all that stuff is 422 00:28:01,740 --> 00:28:05,760 obviously that you're going to do that anyway. But those those 423 00:28:05,760 --> 00:28:10,080 three Net Promoter Score, customer satisfaction, customer 424 00:28:10,080 --> 00:28:13,800 effort, to me shape and experience. So it sounds like 425 00:28:13,800 --> 00:28:14,580 I'm missing something. 426 00:28:17,339 --> 00:28:20,999 Christian Wiklund: Well, so the CSET is great, right? They will, 427 00:28:21,029 --> 00:28:23,999 they will raise the experience of the support experience. So 428 00:28:23,999 --> 00:28:27,749 that's a symmetric they, they should obsess over and very much 429 00:28:27,749 --> 00:28:33,839 established their MPs, I think is it's a great snapshot in 430 00:28:33,839 --> 00:28:36,809 time. Right? And there, there are some people out there and 431 00:28:36,809 --> 00:28:40,229 they're like, well, NPS is the best buyers, you know, self 432 00:28:40,229 --> 00:28:43,679 select, self selecting users who respond to the survey and so 433 00:28:43,679 --> 00:28:45,179 forth. But I think 434 00:28:45,180 --> 00:28:47,730 Jason Baum: isn't that true about anything, though? Yeah. 435 00:28:47,760 --> 00:28:48,720 Every survey? 436 00:28:49,170 --> 00:28:52,200 Christian Wiklund: Yeah. Surveys? Yeah. Like you 437 00:28:52,200 --> 00:28:54,810 typically see the most responses on the number eight, you're 438 00:28:54,810 --> 00:28:59,580 number one, right? So it's, it is what it is. But I think NPS 439 00:28:59,580 --> 00:29:04,770 is very much established as as a as a crucial metric. And I think 440 00:29:04,800 --> 00:29:12,120 if now the four operations, you know, how is how is quality of 441 00:29:12,120 --> 00:29:15,990 the product and experience trending today on Android, this 442 00:29:15,990 --> 00:29:21,810 hour on Android, with the latest release on iOS? To have a 443 00:29:21,810 --> 00:29:26,280 snapshot once every quarter of how our NPS is trending. It's 444 00:29:26,310 --> 00:29:29,850 not good enough for operations, like operations, we need data 445 00:29:29,850 --> 00:29:32,850 now. So if we were to look at the unit Q score, and we've done 446 00:29:32,850 --> 00:29:37,380 some studies on this, it's a leading indicator of both MCS 447 00:29:37,410 --> 00:29:42,120 and star rating. Where if, if your score goes down and stays 448 00:29:42,120 --> 00:29:45,420 down consistently, then the next NPS survey you do is going to, 449 00:29:45,660 --> 00:29:48,390 you're going to see that you took a took a hit there, and 450 00:29:48,390 --> 00:29:51,810 vice versa. If you get the units to score up, then you keep it up 451 00:29:51,810 --> 00:29:54,060 consistently. You're going to see NPS going up in the next 452 00:29:54,060 --> 00:29:59,790 survey as well. So So I think NPS is great for getting that 453 00:29:59,790 --> 00:30:03,150 question. really check in. But for daily operations, we need 454 00:30:03,150 --> 00:30:04,740 something that's daily. 455 00:30:06,750 --> 00:30:08,940 Jason Baum: Awesome. I appreciate, you know, all the 456 00:30:08,940 --> 00:30:12,060 time that you've spent with us and talking to us a bit about, 457 00:30:12,060 --> 00:30:14,970 you know, these metrics, and certainly the unit Q metric, I'm 458 00:30:14,970 --> 00:30:18,990 gonna have to check that out for myself. But I really appreciate 459 00:30:18,990 --> 00:30:23,820 you coming on. And one, one last question, which is a question we 460 00:30:23,820 --> 00:30:27,090 asked all of our guests in season two, and I think maybe in 461 00:30:27,090 --> 00:30:30,210 season three will come? We'll have a new question. But since 462 00:30:30,210 --> 00:30:33,840 you're our first guest on season two, or season three, I mean, 463 00:30:33,840 --> 00:30:39,570 let's, let's ask, what is one thing that maybe you do that 464 00:30:39,600 --> 00:30:43,530 that is unique to you, that maybe no one else knows 465 00:30:43,530 --> 00:30:44,370 professionally? 466 00:30:44,730 --> 00:30:47,520 Christian Wiklund: Well, I make quite a lot of music. So that's 467 00:30:47,520 --> 00:30:51,690 one thing, I have been tinkering with analog circuitry since a 468 00:30:52,440 --> 00:30:57,480 very young age. And when I was younger, I even built my own 469 00:30:57,480 --> 00:31:02,580 analog synthesizers and stuff. So I, I love, I love the folding 470 00:31:02,580 --> 00:31:08,580 iron and breaking, you know, when I was a kid, my dad was not 471 00:31:08,580 --> 00:31:12,420 very happy when I dissemble the VHS recorder, and, you know, 472 00:31:12,420 --> 00:31:14,370 then tried to put it back together. And I've never, I've 473 00:31:14,370 --> 00:31:17,250 never, I've never managed to get it back together. But I know 474 00:31:17,280 --> 00:31:22,890 electronics and I love music. And, you know, and that's 475 00:31:22,890 --> 00:31:27,120 really, you know, for me, there were three paths, musician, 476 00:31:27,750 --> 00:31:31,080 entrepreneurship, and I had in my early age, I wanted to be a 477 00:31:31,080 --> 00:31:34,740 quant analyst on Wall Street, I wanted to build pricing models, 478 00:31:34,770 --> 00:31:39,270 but I ended up in Silicon Valley. And that's also what I 479 00:31:39,270 --> 00:31:42,390 love, Jason about software. And music, software. And music is 480 00:31:42,390 --> 00:31:46,380 very similar. In Canada, they go hand in hand, you start with a 481 00:31:46,380 --> 00:31:52,110 blank piece of paper, an empty class file, or an empty track in 482 00:31:52,140 --> 00:31:55,560 Ableton, or Cubase, or whatever you use. You can create stuff, 483 00:31:55,620 --> 00:32:00,210 and I just love that creative process. The most fun part about 484 00:32:00,210 --> 00:32:04,560 software is you can build stuff from nothing. And then you have 485 00:32:04,920 --> 00:32:07,860 the entire world is your marketplace, really, you can you 486 00:32:07,860 --> 00:32:10,650 can, and you have no inventory. There's so much to love about 487 00:32:10,650 --> 00:32:11,130 software. 488 00:32:11,910 --> 00:32:14,640 Jason Baum: You know, it's funny, you're clearly not the 489 00:32:14,640 --> 00:32:17,760 first person who has come on this podcast and is also a 490 00:32:17,760 --> 00:32:22,590 musician, in addition to their life in in tech. So I think we 491 00:32:22,590 --> 00:32:26,400 have talked about it ad nauseum about having like a DevOps 492 00:32:26,400 --> 00:32:29,460 playlist or something like that. And I think we're gonna build it 493 00:32:29,460 --> 00:32:33,570 out. And I might even do another one, another podcast with the 494 00:32:33,600 --> 00:32:37,020 humans of DevOps musicians, because I feel like there's so 495 00:32:37,020 --> 00:32:40,680 many out there. And by the way, what is with you Swedes and 496 00:32:40,680 --> 00:32:43,890 synthesizers, it's like, sweet love. 497 00:32:45,450 --> 00:32:47,340 Christian Wiklund: While here, I can tell you something 498 00:32:47,370 --> 00:32:54,060 brilliant. And yeah, and I think music is top five export 499 00:32:54,060 --> 00:32:57,930 industries for Sweden. For with 10 million people, how come 500 00:32:57,930 --> 00:33:01,500 there are so many great musicians, and not just Abba, 501 00:33:01,500 --> 00:33:05,850 but you know, you have Max Martin, Ace of ACE, Ace of Base, 502 00:33:05,880 --> 00:33:10,110 and, you know, rocks. And of course, you know, the Swedish 503 00:33:10,110 --> 00:33:13,860 House Mafia and all these these great artists. And it's very, 504 00:33:13,860 --> 00:33:19,590 very simple. Some clever politician that way back when 505 00:33:19,590 --> 00:33:24,840 said, Hey, shouldn't private tutoring and music education be 506 00:33:24,840 --> 00:33:29,490 free of charge for every child in freedom. And that's what they 507 00:33:29,490 --> 00:33:32,580 did. So if you want to learn to play the guitar, or the piano, 508 00:33:32,610 --> 00:33:37,920 or the trumpet or whatever, then you get free tutoring from very 509 00:33:37,920 --> 00:33:41,070 young age. And as a result of that, we have a lot of 510 00:33:41,070 --> 00:33:44,430 musicians. And and so the investment they make in giving 511 00:33:44,430 --> 00:33:50,850 free music classes, pays itself off many times older. So I 512 00:33:50,850 --> 00:33:54,030 really love that one. That's awesome. Yeah, I think it's 513 00:33:54,030 --> 00:33:54,540 super cool. 514 00:33:55,080 --> 00:33:57,510 Jason Baum: That's great. Well, thanks again for coming on the 515 00:33:57,510 --> 00:34:00,960 podcast. And it's great having you as a guest sharing. I did 516 00:34:00,960 --> 00:34:05,580 not know that about, about musicians in Sweden. So now I 517 00:34:05,580 --> 00:34:08,790 know. I hope I hope everyone listening learn something, too. 518 00:34:09,330 --> 00:34:11,610 Christian Wiklund: Yeah. Thank you so much, Jason. And I hope 519 00:34:12,750 --> 00:34:16,020 my DevOps team were a little nervous to show Oh, my goodness, 520 00:34:16,050 --> 00:34:20,250 are you gonna hope he doesn't get grilled here, but this was 521 00:34:20,250 --> 00:34:22,620 very pleasant. And thank you very much. 522 00:34:22,830 --> 00:34:25,500 Jason Baum: Awesome. Yeah. No, I do. I one thing I do not do is 523 00:34:25,500 --> 00:34:29,280 grill. You know, it's, I don't I don't believe in any gotcha 524 00:34:29,280 --> 00:34:31,860 question. Maybe next time. I'll think of a few glad gotcha 525 00:34:31,860 --> 00:34:36,660 questions. Just yeah. Thanks again, Christian. And thanks for 526 00:34:36,660 --> 00:34:40,080 listening to this episode of the humans of DevOps podcast. Happy 527 00:34:40,080 --> 00:34:42,540 New Year again. So glad we're back. We're going to have a 528 00:34:42,540 --> 00:34:45,240 bunch of these. You know, we come at you every week with a 529 00:34:45,240 --> 00:34:49,620 new podcast, new guests new topic. So until then, I'm going 530 00:34:49,620 --> 00:34:53,100 to end this episode the same way I always do, encouraging you to 531 00:34:53,100 --> 00:34:56,130 become a member of DevOps Institute to get access to even 532 00:34:56,130 --> 00:35:00,450 more great resources, just like this one. Until next time, stay 533 00:35:00,450 --> 00:35:04,650 safe, stay healthy, and most of all, stay human. We'll see you 534 00:35:05,010 --> 00:35:07,350 on the next episode. Live long and prosper. 535 00:35:09,690 --> 00:35:11,790 Narrator: Thanks for listening to this episode of the humans of 536 00:35:11,790 --> 00:35:15,330 DevOps podcast. Don't forget to join our global community to get 537 00:35:15,330 --> 00:35:18,690 access to even more great resources like this. Until next 538 00:35:18,690 --> 00:35:22,110 time, remember, you are part of something bigger than yourself. 539 00:35:22,470 --> 00:35:23,220 You belong