Narrator: You're listening to the humans of DevOps podcast, a
podcast focused on advancing the humans of DevOps through skills,
knowledge, ideas, and learning, or the skil framework.
Max Howell: We see this all over the place with open source. I
think a lot of the time the people who make successful open
source, making a hobby project, some kind, something they need,
and then it becomes an essential piece of internet
infrastructure.
Eveline Oehrlich: In December of 2004, a group of IIT students
from DAP University in Copenhagen has launched a open
source beer, the one that a version was described by those
students as a great tasting, energetic beer, and its world
first open source beer. It's based on a classic ale brewing
tradition. But with added Guarana for a natural energy
boost. The aim was to see what happens when an open source
structure is applied to a universally known product like
beer. Hello there. This is evolutionarily ish. We are not
here to talk about beer. But we're here to talk about open
source. And I brought this introduction because I think it
fits in multiple ways you will find out yourself. But I'm a
beer Brewer by Hobby so that caught my attention. Our podcast
title today is open source, Abreu and t. Now a few more
details, which is very different 97% of commercial code contains
open source and I actually learned this from our guest.
When looking at some of the tech trends such as containerization,
like Docker and Kubernetes, originally developed at Google
and released as open source in 2014. Big data like Apache Spark
with Hadoop or Kafka, a lot of open source is out there. Again,
as I said, Welcome to humans of DevOps podcast, I'm Avalon early
Chief Research Officer at DevOps Institute. Today, we have a very
special guest, and I'm excited to have him with us for multiple
reasons. Max Howell, CEO of T dot x, y, z. Hello, Max. Welcome
to our podcast.
Max Howell: Thank you. I'm pleased to be here.
Eveline Oehrlich: Yes, excited for you to be with us. Tell us
the first thing really I'm curious about is your journey
into an across open source.
Max Howell: Yeah, well, it's a fairly good story. So I did a
chemistry degree because I thought I wanted to be a
scientist. And in the process of doing that degree, and then
doing a year in the industry, I discovered that actually, it was
really not for me, I was working with this device that measured
surface tension for like various chemical solutions. And I
realized that if I kept working there, like in 10 years, I would
still be using that single machine and making very minor
contributions to the world. So I fell into this sort of
depressing funk. And I discovered open source.
Programming was always something that I had considered a hobby.
My dad taught me when I was six, which, you know, very young, but
I only used it to like make video games essentially, like
most kids do when they learned to program. And I never really
considered it for a career until my career was on the rocks. And
I didn't know what I was doing myself. So propelled into open
source by installing Linux. And I loved the community, I loved
how everyone was super passionate about what they were
doing, and just doing it because they wanted to do something that
changed the world or improved their workflow or just just
making cool things. And I ended up working on a few apps with
people from all over the world and it was just like nothing
else. For me. Honestly, it's, it's difficult to find it as
good as it was when I first got into it. And we were all on IRC
together. And doing that led me to getting a job in the
industry, which you know, it's it's kind of luck in many ways.
But there was a company in London that used one of the apps
I was working on and invited me to go and interview and it
wasn't really as much of an interview as just having a chat.
So I actually got myself into the industry. And a few years
later after that, you know, I kept doing open source and so I
created homebrew, which is one of the biggest open source
projects of all time. At this point, and I'm sure you'll have
some more questions about that.
Eveline Oehrlich: Yep. Yep, keep going, because I will come,
we'll come back to that.
Max Howell: So I created homebrew because at the time, we
were making all these different apps. I was at last firm in
London, and we made six different apps. One was for
Linux and Mac and Windows, we have an iPhone app, and an
Android app, and even this Blackberry app. And we we build
them all on Mac, because Mac was like the unified platform, time,
and what it really has become the platform for development
now. But this was 2008. And at that time, like, developers
hadn't really decided what platform they're using, it was
all over the place, it was still still a lot of people using
Windows for development, which is not as common nowadays. And
in the our office was an awful lot of people on Linux, and I
was one of the few who converted to Mac. Because Apple had a bad
rep. With developers. It's hard to believe now. But well, it's
easier to believe over the last few years. But there was
certainly a period where everyone in development used to
back. But that was new. And we were using it because it was the
platform that you could do Android Dev, you could do Linux
Dev, you could have a Windows VM running. So we could do all six
platforms quite easily. And like the package management solutions
were the acceptable, they weren't great. And they
certainly weren't designed for developers, their impact
managers turned out with Linux. And in many ways, they were the
thing which defined the Linux flavor. And it's still still the
case. But I kind of felt that they were designed for sis ops
and DevOps, but not development. So I wanted to build something
that was more for that. And so I got on with it. And after a few
months, I realized that it was kind of neat. So I open sourced
it. And then it took a few months before anyone noticed it.
But then when he got noticed, it took off, like amazingly
quickly, huge amounts of contribution, but a lot of
excitement, because I've managed to tap into something which
people wanted, needed, especially with this burgeoning
developer platform, which was what the Mac was becoming. And
yeah, so here we are, like, almost 14 years later. And yeah,
it's it's an enormous project. At this point. It's very hard to
meet people that haven't heard of it or used it.
Eveline Oehrlich: You know, I have watched your speak at Web
Summit in 2022. And you talked about something there called the
Nebraska problem. And And honestly, I've been in
infrastructure operations for a long time, but I'm not a
developer. So maybe that's why I've never really heard. I've
understood, I've heard of it, but I really don't understand or
didn't understand what it actually meant. So can you give
us a quick explanation of what is meant by the Nebraska
problem? And how it relates to the open source? And if
possible, any examples you have seen recently? Maybe?
Max Howell: Yeah, sure. So what we call the Nebraska problem is
the open source funding problem. And we call it that because
there's this famous XKCD comic, you'll see it trotted out
whenever there's any discussion about open source funding
issues, and represent it shows like this tower of blocks, and
it represents all modern digital infrastructure. And then near
the bottom of the towers is precarious, little pillar, and
it's holding the whole thing up. And there's an arrow pointing to
it, which says, this open source package maintained by some
someone from Nebraska, since 2003, thanklessly. Are a
suddenly experienced this with homebrew, like, it was kind of
the first time I'd really experienced it, because even
though I'd done some relatively successful open source before
that, well, you know, I did work full time on a few projects
before that, but I was I didn't have a job and didn't know what
I was going to do with myself. I was still living with my
parents. So it was to do it. With homebrew, I had to have two
full time jobs. One was paid and one which didn't. And I took
some time off here and there so I could work on homebrew full
time. But I never had enough money saved up to work on it for
a long period of time, but I couldn't abandon it because it
was well it was my easily at the time like my greatest creation.
I was extremely proud of it but also people depended on it. And
I Couldn't let them down, I just didn't see how I could. So I
work at, you know, my office job, and often was doing some
homebrew stuff to be honest. And then in the evenings and the
weekends, yeah, I didn't really have a social life and just
worked on homebrew. And that was like, wanted or needed, but it
wasn't sustainable. And in the end, I did burn out on it. And I
haven't really worked on homebrew since 2016. I passed it
to the community, which, you know, I was, I'm lucky that I
could, that suddenly other people turned up. And this is
how open source is and how it should work. Of course,
community is essential to open source. And so it was good that
I could hand it over, but I can personally keep working on it.
And we see this all over the place with open source, I think
a lot of the time the people who make successful open source are
just making a hobby project for some kind, something they need.
And then it becomes an essential piece of internet
infrastructure. And they're stuck in a situation where they
either like, abandon it. And then that's not what they want
to do. They can't afford to work on it as much as they need to.
While they try to find funding, and let's face it most most
funding doesn't doesn't work very well for open source. A lot
of the time then Nebraska projects is what we call them,
like famous one couple of years ago was locked for J which is a
logging piece of logging software for Java applications.
And there was a massive exploit found in it where you could root
servers just by typing stuff in if the stuff you talked to him
would go through the logging software in some capacity. And
no one had really heard of this package, because it was deep in
the stack. And the sounds themselves often works, right.
It's like these essential pieces of software have been built. But
over time, they've just been buried on the things that are
built on top of them. And you got really mature pieces of
software, which still need to be maintained and everyone's
forgotten exist. A lot of for Jay was a great example of that.
And they fixed the bug. And they asked that maybe they could get
some funding in future so that they could afford to spend more
time actually maintaining the software and maybe making sure
it doesn't have these security holes. And I don't think they
ever received anything, because once the bug was fixed, everyone
forgot about them again. And then more recently, there was
core J S, nine D in downloads in is since it was released, the
core of the base of every node app that exists, every you know,
everything was built on Node or uses Node, and an essential
piece of infrastructure really, but the person who maintains it
is giving up because they're just fed up with the fact that
they don't get any funding. Everyone uses this thing.
Everyone is extremely entitled about how they treat open
source. I'm afraid to say it's often true. And he's just like,
well, I can't afford it I'm giving up. And what's going to
happen, I don't know, it's not a good thing to use at the
infrastructure of and, you know, I, I've done loads of open
source. And the truth is, I really should just been doing
open source all these years like I created one homebrew with the
small amount of free time I had. People I myself shouldn't have
to make a choice between working on software that improves the
world and is beneficial to like all layers of the software
stack. Or some of the some of the jobs I did were not
particularly beneficial to the world, really. So that's the
Nebraska problem.
Narrator: Skill up it learning is the perfect online
destination to learn about DevOps and digital
transformation, anytime, anywhere. Our digital learning
platform provides you with all the resources you need to
upskill and learn about these important topics, including
expert led course videos, access to certification, prep courses,
and a community of supportive peers. Our subscription based
model makes it easy for you to gain the skills needed for
success in the modern workplace. Visit DevOps institute.com now
and explore our different learning plans.
Eveline Oehrlich: Wow, wow, quite quite involved. quite
complicated. And when you started out just said, you know,
this noble cause of contributing to the world really with what
you have done with homebrew. We'll get to T XYZ in a minute.
But I wanted to ask a couple other questions. Because of the
topic of open source, there's still some people seem to not
understand that perspective. So that's why I was very, very
interested in your thoughts and your journey there. Let's switch
a little bit to the benefits in terms of of leveraging open
source. You know, take it from either the benefits into from a
developer perspective collaborating and into the open
source community, or for the benefits of the community over
in its large meaning the world right. Talk a little bit to us
about what you see the benefits are of open for open, not open
force, but open source.
Max Howell: Yeah, open source is really interesting. I don't
think that human history is really got any other equivalent
examples. Or markets usually up here, because there's an
economic incentive there. Like I don't know if anything else like
this. But I've been thinking about this quite a lot recently.
And I remember how in the 90s, the entire software stack was
basically owned by Microsoft. And, you know, there was
numerous issues with this. But certainly, as the Internet
became a thing, it became more and more clear that Microsoft
was holding the Internet back, because Internet Explorer was
the only browser that anyone used, and they didn't want to
advance it. They didn't care, too, there are no real
incentive. And it was this kind of attitude, which led to open
source taking more and more of a hold. In the end, like now, it's
difficult to imagine a world where open source isn't the base
of every stack, like we say, 97% of all commercial software has
open source involved in it in some capacity, and like, this is
a really free percent that doesn't, I'd like to see what
those source bases and understand how they get away
with not having any open source at all. It Over time we, we
built the open source app, and we swapped off the Microsoft
stack. And it became like the point where developers realized
that they not only gained like all this functionality for free,
but also they were getting these step ups in productivity. And
proficiency is open source is usually designed to be like
these tiny little self contained pieces that you slop together,
and you build on top of each other. And then you've got all
this slight advanced functionality. But in the
process of replacing Microsoft with open source, we never
transferred all that money that makes over making while they
were trillions of dollars in. And instead, we just sort of
ended up with Freebase without real any maintenance on it,
which, you know, we're trying to solve AT T realized the the
value of open sources. People throw out ideas and the good
ones stick, there's no marketing behind it. There's no like
someone's trying to sell you something that actually they
know isn't a good fit for you. You put some open source out
there. And if it works for the developers of this world, they
adopt it? Well, this is sort of people's world, they adopt it.
And then over time, if it continues to fit, it builds
until it becomes like a mature block in the stack.
Eveline Oehrlich: So with that there are challenges as well.
What are some challenges you would say? Again, from multiple
perspectives, take it from the developer, take it from the
community or take it from the software world of vendors out
there?
Max Howell: Well, I think vendors have the highest risk
right now, right? But if you're building an app, and it has 6000
dependencies, either, there's no way you can vet that. And be
sure that all those dependencies are secure. First off, also
without any malicious insertions of any kind. It's one thing
we're going to try and fix with what we're doing at T. But I do
wonder how, you know, if I had to report to someone and say
that our app is secure, we we can guarantee that I don't know
how I'd say that with a straight face 6000 depths in there. So
there's a lot of risk there for vendors. I think developers,
most of them just enjoy open source existing because it
really does make computing more enjoyable in general, especially
now. Like 14 years ago when I made homebrew, like the amount
of people doing open sources so much smaller. And now you can't
go a day without dozens of new things being released
continuously. And some of those are great. But in most
developers feel bad that they just consume this open source
and don't give back but I don't see how they can effectively
give back right because you can try and spot through a few open
source steps here and there, but we're all using 1000s and 1000s
of open source projects. And it's just infeasible without
some layer of automation to contribute to those properly.
But open source developers are the ones that I feel for but you
know, most of them are just do it, they start off by doing side
projects, it really is, it's like, it's one of the beautiful
things about software engineering is that it's so
cheap and easy. All you need is time, that if you're gonna lay
there, and you got some time, you can, you can create some
open source. But what happens with all the open source devs,
I've talked to who are, have got successful projects is that
after a while, it's a huge amount of work. And you just
can't abandon that. You have to somehow find the time to do it.
People on myself, we've, the way I approached, it was always to
try and make it so that as much as possible, it was robust. And
then but if you did break, it would tell the user what they
can do to try and fix it themselves. And they'll make
sure the documentation was as good as I can get it. And that
would reduce the burden on me as much as I could, as much as
possible. And then, you know, if you if you're lucky, a community
turns up, mostly open source, I've done communities that turn
up, homebrew was
Eveline Oehrlich: really quite an exception. Hmm, fascinating.
Absolutely fascinating. So again, at the Web Summit, you
said, I love this vision you painted, you said, you want to
support those who are passionate on open source development, but
make sure that they get rewarded. And you actually call
it I love this to digital revolution for developers. What
do you mean by that? And then we'll get to T XYZ, because I
think there is a relationship. So tell us about the digital
revolution for the developers?
Max Howell: Well, let's face it, in the sense of web two, as we
call it, which was, you know, mid 2000s, till, kind of
recently, I'd say, a huge amount of money was made by people
building on top of open source. And it was gradual, you know,
like the way open source became so essential to what we're
building and software was gradual. And I think a lot of
these essential pieces, if they were built by hackers who were
building them for other hackers, and then when web two turned up
and figured out how to monetize all that, there wasn't any
transition of wealth. And so that's why it's a digital
revolution. These huge, mega billion dollar companies that
have never really given any substantial amount to the open
source that they depend on. It's not right, like we were okay
with it as it happened, because it was gradual. But now we like
hanging on. You know, why am I working two full time jobs when
Microsoft has billions and billions of dollars coming in
every month? Doesn't make sense. So what we're trying to build a
T is a way to successfully remunerate how open source is
funded. There's been lots of attempts, sponsorship, bounties,
things like this, but we see that as being reward favorites.
And some people make a lot of money with sponsorship. But the
truth is, there's millions of open source packages. And most
of them are Nebraska projects that nobody knows about. React
doesn't need to be funded, because Facebook, pay for it
essentially, which is another thing we want to fix instantly.
I don't believe that these huge corporations should have so much
agenda over how these open source projects that the
community needs to figure out what makes sense for open
source. Facebook could just like close down react tomorrow. I'd
want with what we're building a tee is for these developers at
Facebook and Microsoft who do work on open source or who are
just extremely talented people to be able to quit and work on
open source full time because the pay is exactly the same or
better. Even, frankly, we're building an economy on code. And
if you make a bunch of successful open source packages,
providing you're using the T protocol, you could be much
wealthier than you are currently as a software developer.
Eveline Oehrlich: Interesting. So how can developers get
involved if I am sure we have a few of those who are going ooh,
I love this. I love hearing this. How can they get involved?
Max Howell: Well, four months ago I released the tea package
manager so tea is the successor to brew you know I've stopped
working on brew in mid to 2010s. And I didn't stop thinking about
what could be better about broom. But I didn't really have
a good incentive I feel to make another brutal I'd already done
it. And I remember the tireless hours that I spent working on it
robust, defying it, filling out the package graph, answering
people's support questions, and just like trying to make sure it
was a successful and useful piece of software. But I didn't
really want to go and do that again. I think I like doing
things once usually, but But 18 months ago, while I was looking
into web three, and crypto, which basically was the first
time I've looked into this stuff, because I've never been
particularly interested in it. I thought Bitcoin was pretty
impressive. As a wonderful mystery story to it also pretty
ingenious, how it works, and how it has successfully become so
large, even though it's not controlled. But I never really
thought that there was much more that was interesting there for
me. But as I was diving into web three stuff, I started to see
the different contracts were actually pretty interesting, I
had this moment of inspiration while messing around an open sea
with NF Ts. Because NF T's allowed you to well view the
open sea digital contract anyway forced any repurchases to put
10% back to the original creator. I was like, Oh, so you
can write a digital contract that funnels money automatically
to different entities. And I realized suddenly that the
package graph of all open source was that kind of, if you put a
digital contract on it anyway, you could funnel money to all
the dependencies, all the packages. So all these Nebraska
projects could get like little bits of token from any insertion
of token near the top. And that's how it works, right? Like
people sponsor these big projects of near the top. And
it's all the ones underneath that that are failing, have are
maintained by people without any things. And they're essential
pieces of the digital infrastructure. Without them,
everything would collapse. I like everything being that
fragile is scary, like people remember that for a long time?
How it could all collapse any moment, but somehow it doesn't.
And we're just waiting for a bigger claps really. But also,
like, think about what would happen if it was all funded
correctly. People myself, like make, like occasionally big and
important projects. But we have to do it in our spare time. What
if I was working on that full time? And then, you know, we
find security holes and open SSL every two months? Well, if it
was funded correctly, so that well, either of us so could
really afford to hire people who could, you know, like kind of
like a corporation is what we're thinking things light, each of
these big projects will have their own dowels on the T
protocol. And they can operate them. Similar to a kind of
corporation, they can hire people using the token, they can
split the rewards that they're getting from the tee protocol
across everyone is contributing to open SSL. So if you open a
pull request, you could end up getting paid for that pull
request. Because the DAO is a set of smart contracts that
figure out how to channel the token based on the governance
structure that those projects have chosen. CL I came up with
the idea, like build a package manager and build it on top of
blockchain. So that open source can be remunerated, then we
managed to raise 80 million, and we've been building it since. So
I released the package manager itself four months ago, got six
sales and styles now got some really passionate users. I think
it's special. And if anyone is interested in getting involved
with what we're doing, that's the that's the place to start.
Currently, what we got is github.com/t. XYZ and you can
check it out, install it, and I think it's a better package
manager the brew it's, it's more than a package manager, really,
the ideas I had over the years for how the package manager
could be improved. I realized that it's a unique piece of
software, right? It sits underneath all other tooling.
There's so many things that you can do, which nobody's really
tried before. I don't really get it I think perhaps package
managers just aren't very sexy. Is the truth and don't many
people that are interested in working on them but for some
reason I've always enjoyed it. I love getting into that sort of
area of it.
Eveline Oehrlich: It is a brilliant idea. You are
brilliant. I am honored to talk to you. If I come up with a name
for this that I will share it with You but it is. It's
certainly a super cool and absolutely honorable, really
honorable thing you're doing. I love the idea. I'm, I'm very
grateful. Now I have one closing question. Hmm, maybe it involves
beer drinking, but what do you do for fun? What do you do for
fun if you don't do these brilliant things and manage
these fantastic things? You are You were you were talking about?
Max Howell: Yeah, well, so like, programming was always a
hobbies. Switch when I switch to doing it for career did become
kind of what I did one as well. But certainly, you know, I used
to live in the UK. I live in the States now. And I used to love
going to pubs friends. So there was some beer drinking in my
palace for sure. But I got a son now he's 10 months. And so
there's certainly less beer in my life at the moment.
Otherwise, for fun, so Blimey, I do like video games here and
there on Sunday not playing them as much as I used to. And I like
hiking or going out or camping isn't some of the things I
really enjoy.
Eveline Oehrlich: Excellent. Well, if you make it to my part
of the world, I'm happy to take you out for beer. We got some
pretty good beer in Germany. So yes, yes. Well, this has been
fantastic. Thank you so much Max. We have been talking to max
Howell CEO of T dot XYZ and brilliant thought leader in the
topic of open source Mexican, thank you so much for joining me
today on humans of DevOps podcast. Thank you very much.
Humans of DevOps podcast is produced by DevOps Institute.
Our audio production team includes Julia pape, Daniel
Newman, Schultz and Brendan Lee, shout out to my colleagues who
do a fantastic job. I'm humans of DevOps podcast, Executive
Producer, evolutionarily, if you would like to join us on a
podcast, please contact us at humans of DevOps podcast at
DevOps institute.com. I'm evolutionarily talk to you soon.
Narrator: Thanks for listening to this episode of the humans of
DevOps podcast. Don't forget to join our global community to get
access to even more great resources like this. Until next
time, remember, you are part of something bigger than yourself.
You belong