00:00 - Hello, I'm Seth Simmons, and welcome to a special episode of Opt Out. Opt Out is a show where I sit down with passionate people to learn why privacy matters to them, the tools and techniques they've found and leveraged, and where we encourage and inspire others towards personal privacy and data sovereignty. Have you been thinking about switching to Linux for your daily driver, or wondering what core boot or open source firmware are? This episode, we're sitting down with Jeremy Solaire to chat about System76, the company that makes my laptop, among others, and are behind Pop!OS, a user-friendly Linux distribution.

Welcome on to Opt Out, Jeremy. - Thanks for having me, Seth. - Yeah, I'm really glad to finally connect. I've followed you on Twitter for a while, but recently purchasing a System76 laptop, I got the Lemur Pro to be precise. Yeah, it made me really fall in love with using Linux as a daily driver again, and it's really got me hyped for open source firmware, core boot, just all the fun stuff we got going on in there, so. - That's what I'm using right now, by the way.

- Nice. I'm actually, I know it's a shame, but I'm not actually recording this episode on it, just because I have my desktop all set up. - Well, at least there's one Lemur. One Lemur in the equation. - I know, there's one Lemur. My Lemur's up next to me, but not the active recording laptop. For listeners who aren't familiar with you, or with System76, do you mind introducing yourself a little bit, and then telling us what System76 does? - Sure, I'm the principal engineer at System76, and what System76 is, is a Linux laptop, desktop, and server manufacturer.

We also produce some hardware, like the Launch Keyboard that we just released, and my role as principal engineer is to develop the Pop!OS operating system, firmware for our devices, do electrical engineering for our devices, like the Launch Keyboard, and also do driver development. - Okay, awesome. Yeah, I reached out earlier, and sent out a tweet, and you retweeted it, and we got a lot of interest with specific questions about System76, so I'm excited to kind of walk through those to learn a lot myself, and just to make sure that all those questions get covered.

So the first kind of area of focus that it seemed like people wanted to hear about was the right to repair movement in general, and that's something that System76 really seems to put forward, and then Framework is another company that's really kind of focused on right to repair. Would you mind kind of breaking down what right to repair is in general, and why you think it's an important thing? - In general, right to repair is the capability of a consumer to do repairs on a unit that may otherwise be done by the manufacturer in the current environment, and the reason why I and System76 support this, there's a lot of crossover between the concepts in right to repair and the open source concepts that led to the development of System76.

So in right to repair, it's expected that you provide the consumer with documentation about the hardware they have. It's a very similar concept in open source hardware, and part of our development for firmware for our devices, the ones that are running open firmware, was to collect a lot of this documentation from the hardware manufacturer for the motherboard, and then be able to pass that on to our customers. So on demand, the customer can ask for this schematic for the Lemur Pro, and we can send it to them, which is something that is extremely rare in the hardware world, especially for x86 laptops.

So right to repair is something we support heavily, I think because of how similar it is to open source, which really is the primary purpose of System76, the development of open source software and open source hardware that goes along with it. Yeah, I think to me, a rate to repair kind of jumps out as close as you can get to open source hardware when the hardware is not actually open source. Like it allows the consumer to better understand what's going on, better repair it, better take it apart.

It lets you in as much as possible. Exactly. I consider it a subset of open source. When you look at like the launch configurable keyboard, it's a completely open source design. We have the design of the chassis, the design of the PCB, the design of the firmware is all open source, GPL version three available on GitHub. And the idea that a consumer would want to repair their own device is very important to us. And a consumer with a launch keyboard can go look at literally any part of it and find out how to perform very common repairs.

Like if they break off a switch, they can see, okay, what was the exact part that needed to go there? Where can it be ordered? How do I put it on the board? What does it connect to? If they need to flash firmware, if they need to replace any component on the PCB, it's very, very easy to do that. And our laptops are a little bit of a step back because we don't have that same exact control over the PCB design. We do have an ODM that does the PCB design, but we are able to get schematics for those.

And from those schematics, we're able to implement open source firmware. So now we have the ability for a consumer with a laptop to connect up debugging hardware to the laptop and do firmware debugging, stuff that you would almost never see outside of a factory in China for other laptop manufacturers. Yeah, no kidding. And like you mentioned, it's not even just allowing it. It's that you'll go way beyond that and make it as easy as possible for the user.

Yeah, it's more like our primary purpose. Yeah, it's been great to see. And I've already cracked open my lemurs. First thing I did to swap out to an SSD that I already had. Great. No security torque screws, no complicated ribbon cables, nothing making it complex. It's very easy. Yeah, it's made to be worked on. And if you want the schematics, do reach out because I find it very interesting to read through them and see, you know, this is hooked up to this, this is hooked up to that.

And then you can look at the motherboard and also verify that there's no strange components there. And during firmware development, I'll often hook up logic analyzers to different parts of the motherboard. I don't think this process would be possible without having schematics available. Yeah, I'll definitely have to get that because I would love to walk through it a little bit more and see how it compares to my laptop in hand. A question that a few listeners had is there's a recent laptop that's come out this year called the Framework.

I'm not sure if you're familiar with it, but I was curious if you kind of break down maybe a little bit of how the Framework laptop compares to System76 laptops. Yeah, I'm familiar with it. And I think a big problem for small laptop vendors like ourselves and like Framework is we have to shop around for an ODM who's willing to deal with us at a relatively small volume. You're talking thousands of units, whereas a big ODM like Dell might ship hundreds of thousands of units.

So Framework, the concept for them is to enable the right to repair movement and to provide documentation about the laptop, to provide modularity in the design of the laptop. I think there is a differentiation here that System76 is concerned more about open source. And that is why we have, for example, open source firmware on most of our laptops, all of them except for the AMD laptops, which are primarily waiting for AMD engineering teams to help out with core boot development, something that's an ongoing development.

But we can get back to that later. Framework does not have open source firmware at the moment. And I think Framework offers a little more modularity with the ability to swap out ports. But when you consider what I would think would be the final modular laptop design, if I was to start from scratch, I really think it would start with an open source motherboard design done in KECAD with the source files available rather than having an ODM lock that stuff up.

Because I'm sure Framework, just like us, has difficulty getting all of the documentation from the original motherboard manufacturer. Yeah, those are the main differences when I was looking at the Framework. Kind of at the same time that I ended up purchasing the Lemur Pro was the main difference for me. It was core boot and open source firmware. Plus, I think the Linux-first approach of System76, you'll have really good integrations into Pop!OS and the firmware plays really nicely with Linux.

So that Linux-first approach is something that's very, very, very much a requirement for me, honestly, at this point. I think there's definitely something Framework is going to learn

10:00 I think they're going to provide a Linux pre-installed option at some point. I just don't see it not going that way because there's so much intersection between their customer base and the Linux community. And it seems odd to me that they're selling this right to repair device with Windows as the only option. But that's because of the split between the two communities. There is a right to repair community and an open source community, and they are distinct.

And I think they share a lot of principles. And over time, they will probably merge. But at the moment, framework and System76 are kind of examples of where those two concepts deviate. Yeah, definitely. Definitely understood there. One of those main differences that we talked about was CoreBoot. And I'd love to hear a little bit of your thoughts around what CoreBoot is and why it's an important aspect of System76 laptops. CoreBoot is an open source boot firmware.

For devices like laptops, desktops, and servers. It supports, nowadays because of Chromebooks, it supports some of the latest processors that are out there. Which has been the main reason why we started switching over our devices. So about two years ago, right before the release of the Darter Pro 6 and the Galago Pro 4, we were investigating open source firmware. And our main concern was that the speed of development for firmware support for open source firmware was far behind the SoC development.

So you would have a brand new CPU from Intel or AMD and it wouldn't be supported for six months to a year. And Google's Chromebooks were using CoreBoot. But they were generations behind in some cases. So what really was required for Intel to step in and start doing CoreBoot development ahead of release of their products. So they already have Alder Lake support in right now, for example, in CoreBoot. This is something AMD needs to catch up on and is working to catch up on.

They're working on Cezanne support. Cezanne is already available, but it is the latest generation. And I think this partnership between Google, Intel, AMD has really enabled CoreBoot to develop for the latest SoCs. So we were finally able to, in 2019, to consider CoreBoot as a replacement for the firmware that we were already using, which was AMI firmware. And the beginning of that process was to take the Darter Pro and to try to get to boot with CoreBoot.

There was an initial port that had been done that was, honestly, it didn't support quite a lot of hardware. It was unfinished for the Galago Pro 4, for the Clavo model of it. So I used that as a reference. And I used our own vendor firmware and the schematics that came from our motherboard vendor and data sheets for the embedded controller and things like that as a reference. And I was able to design, I was able to port CoreBoot to the Galago Pro and Darter Pro.

Then we released those pretty soon after. Next was the System76 EC code, which is running in the Lemur Pro you have. We noticed quite a lot of problems with CoreBoot and running it on top of the proprietary embed controller code. And something that hasn't really been done outside of Chromebooks is to look at replacing the embed controller code. Even for Libre booted ThinkPad, they still use the proprietary ThinkPad EC. So I first tried to look into porting Chrome EC, which is the project that Chromebooks use.

And I had a call with one of the developers. And they said their intention is primarily to support 32-bit embedded controllers. And we have an 8-bit embedded controller. And its architecture is based on the Intel 8051, which is a very old architecture. But because of its age and because of patent and copyright things, it's very, very popular for hardware devices to have an embedded 8051 core. So I started working completely from scratch on new embedded controller firmware targeting the Lemur Pro.

It was a very arduous process. Hundreds of man hours, if not more. And just getting it to turn any lights on without starting on fire is a difficult thing to do. But eventually, it was booting. And it was at a point where we could verify functionality compared to the proprietary embedded controller code. And now we don't even use the proprietary embedded controller firmware even as a reference. We just wipe it off. Like the first time I get a device, I'll look at the schematic.

And I'll go through the way the embedded controller is connected and port the embedded controller firmware. And that will be it. Then I flash it on. So this allowed us to kind of grab the supply chain for firmware in a way that I can only really think of Google ever doing that. And that's for Chromebooks. Purism recently with the Librem 14 was able to port System76 EC. So that leaves three variants of open source firmware out there that can actually achieve this level of integration where they have both the embedded controller and the system firmware and the operating system all designed for each other.

Purism, Google with Chromebooks, and System76. Yeah, it's an awesome holistic approach having all of that being open source, all of that being visible. Anybody able to contribute and help out to find things, fix bugs, add features. If I was to throw one dig in at the others, our primary concern with our open source firmware was to provide a standard UEFI firmware that would support all UEFI operating systems. And so our default firmware is able to boot Windows.

Our default firmware is able to boot any UEFI operating system. The same cannot be said of the default firmware of Chromebooks. They're designed to only support booting the depth charge payload and booting Chrome OS. And Purism defaults to CBIOS or HEADS as the two options. And those two do not support booting every operating system. So I really like to think of it as we wanted to make an open source personal computer in the same ecosystem. Others may have gone down paths that provide interesting advantages.

Purism using HEADS, they're able to do things with disk encryption we aren't able to do. Chromebooks have an incredibly robust boot infrastructure such that failures are nearly impossible to introduce in the boot process. There's no way to overwrite firmware and get the thing to fail without going into developer mode. So I think of the three different options out there. We are on one side where we wanted to make something that would be compatible with the rest of the PC ecosystem firmware wise.

It's UEFI firmware. It supports all of the Windows features. It supports BitLocker, for example, which was not an easy thing to do. You have to do EDK2. You have to do UEFI drivers for the TPM. And you have to do certain things that would not necessarily be required if we were only targeting Linux. But the advantage of that is our customers can have this product and run any operating system without thinking about the firmware. So if they want to think about the open firmware, they're certainly welcome to.

If they don't want to think about it, it ought to work the same as any other PC on the market. Yeah, it's been...

20:00 It's been great so far for me. I haven't had any issues. And it's great to know that it is open source and that, like you said, other companies, organizations are able to use that, hopefully contribute back upstream to it. And that's a key feature for me, for sure. And you mentioned the TPM there with BitLocker support. Is that something that is supported and used in like Pop! OS, for example? How common is it for the TPM to be a part of the kind of the security infrastructure of a system?

It's something, yeah, the TPM, by default in Pop! OS is not utilized, but there are user space tools to use it that I know there are users using it. We do have a plan to implement measured boot for Pop! OS. What is that? The way that that will work, so the TPM is able to measure boot components all the way from the firmware up to the bootloader to the operating system kernel and even the drivers of the operating system. It has a number of registers that basically store the hash of data that is fed to the TPM.

So during boot, the processor can feed it all of the data of the firmware image it's about to load. And that hash can then be compared later on. For example, if you unlock your disk, now you're in territory where only the user could have authorized the code that's about to run because the user just put in their password. Now that code can verify that the firmware components match what is expected. And that's essentially how BitLocker works. The bootloader feeds in data about itself and the kernel and then checks that the final hash measurement of the entire boot infrastructure matches what BitLocker expects before unlocking the disk.

And it's a little more complicated than that, but the effect is that you can't unlock your disk if your firmware has changed. And typically there's a recovery key that you can enter in if you want to authorize a firmware change. But what that allows is for the firmware to be validated by the operating system in the process of decrypting the disk. Heads does something very similar. The Heads firmware. Okay, awesome. Yeah, I was curious if there's an easy way for users as well to verify firmware when they get a device from System76.

If TPM doesn't do it right now, is that something that users can do using some sort of CLI tools or something? Yeah, my recommendation is to buy an external flasher. That's my recommendation. And that's useful if you're going to develop firmware or if you want to verify firmware. And the first thing you should do is build the firmware on a computer that you trust from source and then flash it using an external flasher. Even if you were getting the device straight from the manufacturer, that's still what I would recommend, even if there was boot infrastructure for making sure the firmware didn't change.

Because with the right tools, without an absolute root of trust, an attacker could feasibly do the same thing the manufacturer does and set it up and make it look like it came from the manufacturer. And the absolute root of trust, the way it works in Intel systems, is something that isn't really friendly with open firmware. It's called boot guard. And what that does is that fuses a signature for the firmware vendor into the Intel PCH so that during boot, it will verify against that signature and only that signature.

And if it doesn't match, it won't even boot. So if that was in place, you could imagine a user would not be able to compile firmware on their own. They'd have to send things to us to get us to compile them and sign them. So the best you can do right now with devices that support open firmware and leave it in a mode that allow flashing it, which most of them do, is definitely to rewrite the firmware with an external programmer. Yeah, that makes sense.

I mean, a similar approach, I think most people getting a laptop shipped to them are going to take would be overriding the OS, flashing it themselves, and making sure that, again, that's a valid image directly from the system. There's definitely people who do that. The way our disk encryption works is actually to do a reinstall in the user's hands. So we have it set up where the recovery image is the only thing on disk when a user receives our machine.

So it boots into the recovery image, which is a copy of the ISO. So it's the same thing you would download from our website. It boots straight into that. You could plug in a USB stick with the same ISO and do the same thing yourself. It just wouldn't be as streamlined. But that way, the encryption key is not known to anyone in the middle or the manufacturer even. Especially for user data. It's one thing for firmware where it's open source firmware and it doesn't really matter if it's encrypted.

It doesn't need to be encrypted. It could be better if it was signed and the signature was checked. But you can generate the same firmware blob no matter where you are on any computer that you have a compiler on. So there's no reason why you shouldn't just rewrite the whole thing. For user data though, having the encryption key in the user's hands only generated at that time is pretty important. So when I see computers that are shipped with encryption enabled beforehand, I get very inquisitive because I wonder, well, who has the keys?

If it was encrypted prior to leaving the factory, then that means that the manufacturer has the keys to your personal data. So... Yeah, it's something I've never understood when manufacturers or vendors do offer to do that. It's always something that the user should be setting up on their end. So I'm glad that you'll approach it that way. You talked a little bit about... Okay, go ahead. Yeah, and I think the process for having a TPM measured boot process would be very similar to where the installer walks you through doing that while the computer is in your hands.

So long as you don't leave the computer lying around while you're in the middle of installing it, you will watch the entire process of it going through and ensuring that the firmware is valid and setting up measurement alongside disk encryption. That's my hope. Yeah, I would love to see that feature. I think that would be... I think that would go a long ways to helping people who aren't able to actually do the firmware flashing themselves, at least give them another layer of visible security that they can verify.

They could go to our website and verify. These are the current firmware images and hash sums. And so long as there's no TPM intersection, TPM... So long as there's no man in the middle on the TPM chip itself... Yeah. That can be somewhat trusted. It's all about how much you can trust it. At the end of the day, you have a device that came from third parties. If you're not melting the silicon down yourself, can you really trust it? So yeah, that's why I have a paper wallet.

You've mentioned Purism laptops a little bit already, but I'm curious if you don't mind breaking down a little bit of the security approach of System76 laptops versus something like the Purism. I think it's the Librem 14 is their latest. I think their default firmware matches pretty well with what we're doing by default as well. And their Heads firmware is a step ahead at the cost of requiring a specialized operating system. Yeah, because you mentioned it's not UEFI compatible.

Yeah. So I hope what we are able to do is have firmware that can support the same concept for multiple operating systems. Awesome. And the last kind of hardware-related question I had for you, I know a lot of listeners are not in the U.S. and System76 are a U.S.-based company, but you do ship to other countries, right? Yes, we do. Okay. And is there any kind of keyboard localization or local suppliers to avoid import fees when doing that type of thing?

We have in the past had keyboard options, but they're pretty limited. The vast majority of our sales are still in the United States, and shipping outside can be pretty expensive, shipping outside the United States, especially now. Yeah. So we have...

30:00 have had freight freight forwarders been used for customers before but they have to set that up themselves that's where somewhere someone in the u.s would receive the the laptop and then somehow some way get it into the country that they're living in not always legitimately but you know when you look at the the tariffs that are on these laptops i'm not sure that doing it the legitimate way makes any sense for some countries um yeah that can be quite a mess for sure yeah so let's shift gears a little bit and i wanted to chat about pop os and some of the the approaches that have been taken there um sure so the the first question is just before we get into pop os itself you can use any kind of linux distribution you want on a system 76 laptop um but are there any specific considerations especially around like the firmware that users should be aware of i think since the open firmware we've done a lot of work to make sure that everything required is upstream so we have a system 76 dash acpi driver that's in the upstream linux repository that's been available since probably 5.8 that uh

handles the majority of the specialized features um like the hotkeys and uh providing sensor data so uh you should be able to take an open firmware laptop and install basically any links distribution on it so long as the kernel is is relatively new that being said we do have some support uh utilities that that make some things a little easier uh like switchable graphics is uh it can be a mess to set up in some links distributions we have a project called system 76 power that we use in pop os to provide the switchable graphics support we have a system 76 dash driver project that uh basically exists to provide workarounds for issues uh usually for older kernels so uh that that project i recommend to be installed and then a system 76 dash firmware project that provides uh automatic firmware updates and uh we do provide for the launch we provide updates through through flopped uh which is usually available on on uh most length distributions uh we don't do that for system 76 laptops yet but we are looking to for desktops we're still not able to

get schematics that we need to port firmware over um so it's kind of out of the question our our system 76 firmware utility is able to use the proprietary firmware update mechanism whatever that is for whatever uh bios vendor for desktops you know we use gigabyte or asus motherboards they're going to use ami uh usually as their bios vendor so uh definitely everything should work and if it doesn't on an open firmware laptop i would love to hear about it on other laptops make sure you have the system 76 driver installed it's usually available in some repository for the popular links distributions and if it doesn't work after that then then i would love to hear about it yeah it's good to hear that customers have that flexibility so they do love pop os great if not i can move to the yeah and we see our support team is there for whatever os you install on it even if you're going to install haiku os okay even if you're going to install a free dos um yeah we don't care what operating system you're using on your hardware just if you have a problem

um our laptops come with lifetime support so make sure to utilize it that's awesome uh so let's let's jump into pop os itself uh would you mind breaking down a little bit about what pop os is and and who the the focus of pop os is pop os started um back when ubuntu dropped unity in favor of gnome 3 um because we were really hoping to have more input into the linux distribution that we ran because from a hardware vendor perspective there's a lot of issues with the distributions that are out there right now the rolling release distributions for example will often release kernels that are newer and user space services and and hardware related items that are newer but don't have the infrastructure necessary to test for regressions in hardware support and the stable release distributions will usually have kernels and and mesa and related hardware support packages that are a little too old for brand new hardware so we get intel cpus well before the the chip shortages we would get them sometime before release before the public

release but very close to the public release and we would want to sell our hardware at exactly the time that the public release happened so um even when we provide ubuntu to users because that's one of our imaging options it comes with the hardware enablement from pop os we copy those packages over because there's just no way to ship brand new hardware with a stable release operating system and on top of that so that's the kind of the lower level on top of that we really wanted to enforce our look and feel on gnome 3 when ubuntu switched over i felt that 1804 was pretty rough and um in 1710 before that was the first time that they shipped gnome 3 it was even rougher so our first pop os release was to try and fix the things that we felt were issues but ubuntu may not have felt were issues with the gnome 3 version of ubuntu and uh from 1710 onwards we've been doing pop pop os and growing it from there um our focus mainly being on making gnome 3 as powerful as possible and now gnome 40 by adding features like tiling uh by adding more custom customizability with with cosmic and making the hardware enablement move at the speed that hardware requires while not introducing regression so every time we have a hardware enablement update we test it across

across a very wide spectrum of hardware which is something we have to do already as a hardware vendor and before we would say okay well this doesn't work let's go to ubuntu and let's go through a process that takes four weeks to get something fixed. Now we can just say, okay, we'll release a new package if it fixes the problem on PopOS, and we'll copy it over to the Ubuntu PPA. It's a way for us to stay more in line with our actual upstreams, Linux and the Mesa project and X11 and so on, and not have to wait around for distributions, other distributions to get hardware support merged.

Yeah, I think for me it makes it a much more cohesive tool. I think it gets closer to what people are used to with something like macOS or Windows laptops, where it, I know this is kind of a meme at this point, but just works as in everything works well together. You don't have to worry about drivers constantly. It's a very seamless experience overall. We definitely would hope that if you get a system with PopOS pre-installed, you can update it forever and never run into issues.

Now, is that always the case? No. It's not always the case with Windows or macOS either. Yeah. But people have definitely used the phrase, it just works, in reference to PopOS. Yeah, it's definitely been the kicker for me and something that makes it something that I'm much more willing to use daily for every single task that I do, rather than just using it for specialized things. And I think that's one of the reasons why I love Pop, is because it's something I can't recommend to people, especially if they have a System76 laptop.

But even outside of that, it seems to be a much more seamless experience, in my experience so far, at least. Yeah, it's not just for our laptops, though. The primary focus is to ship our hardware, for sure. But because we ship such a wide swath of PC-compatible hardware, it ends up enabling a lot of other devices, too. Yeah, for sure. That's definitely visible in the way it works. And then Ubuntu is the base for PopOS. Was that just kind of because y'all were using Ubuntu previously, or is there a very intentional approach there?

Ubuntu was chosen previously. The time that System76 was started, around 2006, Ubuntu was decided to be the...

40:00 the best option. And I think I would agree with that if I was there at the time. Nowadays, I do think a lot of distributions have caught up, but I still feel most at home in an Ubuntu-based environment. I don't know exactly why. I just feel like I'm not fighting the distribution so much. I don't have to worry that they're going to break something. We have our set of packages that we patch and we override. But the things that we import from Ubuntu, the patches that they apply usually are there for a good reason. We do take out some snap stuff. That's probably the biggest thing that we have to take out. But as a base, it's been really good for us. When you think about other bases, you think about Debian. Well, if you base on Debian stable, you'd be behind on hardware support even further to the point where it's difficult to even compile a new kernel because the compiler is so old. So if you look at Debian testing, it's expressly not supposed to be used as a base.

Some people use it anyways. And it also isn't that far ahead, actually. Gnome 40 wasn't in Debian testing for quite some time after release. It was in Debian experimental. And Debian experimental, if you try to base on that, well, guess what? You're going to be in broken package land for a while. It's determined that experimental will be the place where packages are pushed first. And they come into testing basically when they become mature enough that they can become an upgrade and they're not in staging anymore. So testing is something you could base off of. But it wasn't as up to date as I would have liked for a rolling release. Arch is something we're definitely considering and Fedora for bases.

Fedora though, I feel that the attitude towards proprietary software limits the spread of Fedora. When you consider the lack of the proprietary NVIDIA driver and how difficult it is to install it compared to other distributions. I think that's a major reason why Pop! OS has taken off is having a pre-packaged NVIDIA driver in a variant of the installer. So you can go to our website and download that and boot it up without having to add kernel parameters or do anything weird and install it.

And it just works. And I wish other distributions really understood that. I think Manjaro figured it out because they're starting to package the NVIDIA driver in their in their ISO. But yeah, those were the options basically. Debian, Ubuntu, Fedora, Arch. And Ubuntu just seemed like the one that was closest to what we needed to do. That we didn't have to fight the upstream as much. Though that might be changing. Maybe we could base off Arch sometime. I don't know.

Yeah, I'm gonna be completely honest. The whole reason I stumbled across Pop! OS was because my personal desktop, it's a custom built desktop PC and it has an NVIDIA GPU in it. And I was just constantly struggling with NVIDIA driver issues and saw that Pop! OS had a package that had NVIDIA drivers pre-installed, installed it and had literally zero issues. Yeah. Straight out of the gate. And that was just a break there for me. That's our hope. We test against most of the generations of NVIDIA GPUs from the past five years. And I think that helps a lot. There's still issues with NVIDIA, of course.

Yeah. The hardest thing with it is if there is an issue, you have to wait for NVIDIA because there's no way you're gonna fix it on your own. There's no source code or anything. There's a few times when you can patch up the DKMS driver, but that's really just a thin wrapper around a very big user space library that's completely closed source. But a great example of why rolling releases can sometimes backfire is the latest kernel right now does not work with the latest NVIDIA driver on G-Sync monitors. And this is a regression from the previous NVIDIA driver. But the previous NVIDIA driver, kernel crashes before the GUI can even show up on later kernels. So if you were using a new kernel and trying to update the kernel without thinking about the NVIDIA driver as well, you'd test, okay, it works fine on AMD GPUs and Intel GPUs, let's release it. And then all the NVIDIA users of your operating system are gonna start complaining, "Hey, my system doesn't boot anymore." And then you say, "Okay, well, oh, there's an NVIDIA update. Let's pull that." And then you release it. And then they say,

"Oh, hey, our G-Sync monitors don't work anymore." So I do think regression testing is incredibly important and using hardware to do that. And I feel that that's the major missing piece of most other distributions is that they don't have the manpower or testing library, hardware testing library that they need. I wish we could help out. I haven't really been contacted by anyone, but I would love to help out if we could. But Pop! OS is kind of our reference for if you are testing hardware, this is what you have to release. These are the packages.

So right now we're at the 5.13 kernel and testing the 5.14.9 kernel that was just released today, and waiting on the NVIDIA driver to be updated. And we won't release anything until everything is working. Yeah, that's great to hear. I love that approach. Is there anything in particular in store for Pop! OS that you'd love to share? Yeah, I wrote an internal memo called "Exporting Pop!" and I think this is my concept for what it should be in the future, that it should be a source of projects for other distributions and other desktop environments, not just a single stack of GNOME on top of Ubuntu. So I would really hope that we re-architect Pop! OS, in the coming, I don't know, year and a half. So that hopefully we'll have the ability to install the Pop! session on almost any desktop, on almost any distribution. And additionally install any desktop environment easily on top of Pop! OS.

Because I feel that there are so many great things about Pop! like the hardware enablement that I'm talking about. I really don't like the fact that we can't make it available beyond Pop! OS. There's plenty of people who would benefit from it, who for whatever reason they want to use DNF, they like the AUR, are unable to use the Pop! OS hardware enablement stack, or unable to use the stuff that we've done to GNOME shell. So cutting up Pop! OS into modular pieces that can be redeployed on any distribution and reused by any other desktop environment, I think will spread the utility of what we're doing to many more users. Yeah, I'd love to see that. I think there's a lot of people who don't want to use Pop! OS for whatever reason, but a lot of the tools would benefit them. So I'm glad that that's a focus and I'll definitely be keeping an eye out for that kind of modular approach to all the tools that are part of Pop! OS. And the last question around Pop! OS was just what can listeners do to help support System76 and the development of Pop! OS? And are there any particular areas or tools

that need testing or feedback? Well, we've had a lot of very enthusiastic users and that's something I'm very happy for. And they will break their systems just to help us out. So I would say stay involved on the Pop! OS GitHub and keep watching things that come in. Join the Pop! chat, chat.pop-os.org. We talk about most development things there. And just try to convince other people to install Linux. It doesn't have to be Pop! OS. The purpose of Pop! OS is not to take over every single computer, but we'd love for that to be the case. The purpose is to package up the pieces of Linux that we think are best and the open source software that we think is best in a way that we hope our customers will like. And that has ended up drawing in a lot of people who are not customers. If you want to support us financially, buy our hardware.

The launch keyboard is a great entry point too. If you're not able to purchase a more expensive computer,

50:00 You can also purchase a, I'm doing air quotes, subscription. You're not really subscribing to anything. You're just paying $12 a year, $1 a month to support Pop!OS. That still pales in comparison to our hardware sales. So definitely want to support Pop!OS financially, hardware. Buy it from System76. If you want to support us in our community, join the Pop! chat and be active on the Pop!OS GitHub. And if you have any questions, reach out to me personally.

I'm on Twitter. My DMs are open. I answer emails. Just however you can get a hold of me. I don't mind. Awesome. I just had a few kind of general questions for you that were some fun ones that listeners and some of your followers came up with. And the first one of those was, what's something you love about System76 that people may not expect? Well, I think they all expect it. The thing I love is just the people who work there and how positive they are about open source.

And this is something where you can see other companies. I'm sure if I was working at Dell and I was working on the developer edition laptops at Dell, it would not feel the same. I'm just sure of it. This is something where we're 100% in to open source. So I think customers can feel that too. I'm not really sure if there's something you don't expect. If I'm supposed to say something like that, then I would say, we use Windows for some things. You may not expect that.

We have lasers that run Windows. They're thousands and thousands of dollars, $100,000 or more machines, and they run Windows. And there's no way to replace that. It's just so sad. Yeah, it's very sad. Yeah, but hey, they cut out metal, and then we bin that metal, and we put the components in to make an actual Linux machine. So I feel like that's planting enough trees to cover up the whatever Windows greenhouse gas it releases. They're serving a good purpose.

They're building Linux machines. Yeah. The other one was, I think, a really good one that somebody on Twitter brought up, and it was, if you were given the budget to develop a new product, what gadget or appliance would you make? And it's not just another computer or keyboard. Oh, well, that's, I mean, even the keyboard is a computer. I think no matter what it would be, it would be a computer. I was thinking, though, recently, because I've been so, I have been just, I don't know if there's a word for it.

It's oppressive, okay? It's oppression from our HP printer, okay? It's oppressive is what it is. We need some open source printer. I know printers are old, okay? Printers are dumb, okay? I agree. But, you know, every now and then you've got to print something out, like a legal document, or, like, we have to file some paperwork, tax crap, or whatever. You have to print it out. You want to ship a package? You have to print out a label. So, printers are going to be here for a while.

And we just had to dump hundreds of dollars into this damn printer. Not enough that we would buy a new printer with the same amount of money, but definitely enough that I'm thinking, why does it take so, why are printers so damn expensive? All they're doing, it's ridiculous. It's insane. 3D printers are open source. Why aren't 2D printers? When are we going to get a System76 open source laptop, open source firmware? I mean, the possibilities are in us here.

Definitely that would be way more fun than a printer. And so, I'm hoping that the launch keyboard was kind of our test for laptop design. The chassis itself had some techniques that are required for laptops, like lots of rounded edges and stuff, and milling instead of cutting and bending. And the PCB had a lot of high-speed design. So, it was kind of our test into, could we do something similar to a laptop in certain aspects in-house? And we were able to do it.

So, I think now we're just moving on to, the hardest part now is the chip market. It's just really hard to get supply of components right now. Even for the launch keyboard, we're not able to find components. So, when we move on to a more advanced thing like a laptop, it could be out of stock for a while. I'm not sure when we would even get enough components to do prototyping. So, yeah, it's a difficult environment, and we thought we would be done a lot sooner than we will be.

If it wasn't for COVID and wasn't for the chip shortage, I think we'd already have prototypes and probably even have public visibility into them. Yeah, I'm definitely hopeful that that situation will improve for Y'all's sake and just for all of the people who need to buy hardware at the moment. It's definitely not a fun time to be in the need. Yeah, yeah. And then the last question I had for you was just if there were any particular tools outside of System76 or Pop! OS that you recommend privacy conscious listeners look into.

Yeah, definitely Tor. I use it a lot, but I won't tell you. I don't use it. Bitcoin, definitely. Those are pretty important projects. I use Firefox and only Firefox. I even use it on mobile, even though it's really not optimized that well for mobile. So I use it on Android. I think Firefox is a very important tool in the privacy battle. And people who use Chromium-based browsers should really, really look into what those browsers are providing because most of them are not very privacy conscious.

Chrome, of course, is not very privacy conscious. Brave, I don't think is. Edge, definitely not. So support an open source browser that comes from a third party that isn't a big monolith. Support Firefox. Yeah, Firefox is what I use regularly. Just harden it on desktop. And then I bounce back and forth on mobile, but I keep coming back to Firefox. I think I'm going to stick around there. I did have a question that came out of that with you mentioning Bitcoin.

Is there any future plans for System76 to accept Bitcoin or other cryptocurrencies for purchases? We briefly accepted Bitcoin and had to roll it back because our payment provider was not able to provide refunds using cryptocurrency. So the refund question is still a problem for us. It was when a customer pays in a cryptocurrency and the cryptocurrency changes value. And assuming that we end up spending the money that we get from the customer. We spend it on something else.

And we expect to fund refunds out of an account that has a static value. And, you know, Bitcoin goes up, Bitcoin goes down. You can say the dollar is going up and going down. I don't know. That's ridiculous. Yes, it is, but not as much. Absolutely. Currencies are changing. But when you look at how much raw goods, raw material Bitcoin can buy, it fluctuates more. And we are purchasing components using, you know, fiat currency. Yeah. So when we have to refund, we have to refund the value in fiat.

Because the only way for us to stay solvent is to store all of our reserve in fiat. Because all of our vendors for components only take fiat. So it ended up being difficult to figure out how to properly and do returns for users. In the case that they purchase in cryptocurrency, do we do the return in the fiat value? That kind of defeats the whole purpose of buying in cryptocurrency. Because now they have to accept fiat. So, yeah, it's an open question.

If there was an easier way to plumb into purchases and maybe have somebody who was able to take some of the risk for conversions between fiat and crypto, it would be easier. Yeah. The only real thing I could think of there would just be to allow resellers that do like cryptocurrency only or cryptocurrency primarily. And then it's not. It's definitely not.

1:00:00 forbidden. You get one of our laptops, there's nothing. We don't make you sign any agreements or anything like that. So yeah, if you want to purchase in bulk, you can definitely do that and be a reseller. I think the major reason why you don't see very many resellers, the cost ends up being more than they expect, even if they try to have a direct relationship with us. We can lower the price if we're selling in bulk, but not by as much as I think they would expect because again, we aren't the original source of the laptop chassis at the moment.

So our margins on the hardware are not as high, I think, as they would be once we in-house design the laptop because then we're able to cut out a lot of middlemen from the equation. A lot of companies that are making money off of basically just moving something from point A to point B. Yeah, I think that'll definitely be interesting. I'll have to keep an eye on when you do have laptops released because I think it would be really valuable to have people able to easily be proxy merchants between people who are only comfortable spending cryptocurrency and you and not making you bear that.

I think when we have our own in-house design, we would be more open to even setting up factories in other places so that there can be a direct manufacturing to the end user pipeline that doesn't have to go through the United States. The biggest cost right now is exporting from the United States and importing to another country. I feel that's the biggest cost for international users. So you kind of can solve two problems at once. Other laptop providers, you look at Dell, you look at Lenovo, they definitely have a much larger reseller market. For example, they sell to Best Buy and Best Buy has, and that's just a volume thing. If you have the volume, you can drop the cost lower for yourself. You're able to take on additional markets like putting things in brick and mortar stores.

For us, I mean, it would be like putting one laptop in one out of every 10 stores. It wouldn't make any sense. And the likelihood that our customer is going to be the one walking through that store. So yeah, we have a direct to customer model that I feel has grown out of our volume. If our volume was higher, we'd probably have more ability to have resellers and have shipping to more countries and things like that. And more keyboard options, too. Well, here's to hoping that we can keep driving those sales. I really do love what y'all are doing. I'm really grateful for System76's focus on free open source software, on making the hardware as open source and right to repair friendly. And I think there's a lot of value that y'all are bringing that I just don't see in other companies. I'm definitely very thankful for what y'all are doing and glad you're able and willing to come on and chat a little bit about System76 and PopOS, Jeremy.

Yeah, I appreciate that. Thanks for having me. Yeah, have a great day. Yeah, you too. Thanks for listening. And I hope you enjoyed this episode of Opt Out. If you did, please take a moment and subscribe to the podcast. Or if you're already subscribed, share it with one friend or family member this week. As always, you can check out the links to our guest content and contact info, as well as links to all of the tools we discussed in today's episode in the show notes or at optoutpod.com. Now get out there and opt out this week.

Let's take a quick break from this episode to chat about the sponsors of Opt Out, Cake Wallet and LocalMonero. Cake Wallet is a key tool that I use daily, as it allows me to easily and quickly use Monero for private by default payments. It's available in both iOS and Android. It is a fantastic way to get started buying and using Monero with a simple and easy to understand user experience. I regularly onboard new users to Cake Wallet and hope that it will help simplify and ease your journey into cryptocurrency.

If you're interested in purchasing Monero for the first time or helping to bring others into a parallel economy, I'd recommend you look at using LocalMonero, like I do, to buy and sell Monero while maintaining your privacy and avoiding invasive exchange surveillance. LocalMonero is entirely peer-to-peer and is an important part of opting out of the surveillance state and into a parallel economy. Thank you to both sponsors for their incredible support and partnership.

And I hope you'll take a moment after the episode to learn more in the show notes or at optoutpod.com slash sponsors. Music