R-Evolution: A Story of Rust Adoption
Live captioning by White Coat Captioning
RYAN: All right, hello, hello. Hopefully the slides will show up at some point. Almost. We are getting there. Technology! Hey, all right, there we go. All right, so we are both very happy to be here. Come on a little bit closer so people can see you. All right, we are both very happy to be here to talk to all of you today. We are going to be talking about a story of Rust adoption at the company that we work at and this has really been an evolutionary process and a revolutionary one. Evolutionary in the fact that it is taking a very, very long time, but that's to be expected, and revolutionary in the fact that this represents for us really a new opportunity and one that does not come along very often. We are a company that normally invents programming languages and, if we do, it's usually in the order of one per decade that sees the light of day and so bringing in a new programming less than is something that doesn't happen every few months, but really every ten years or so. So this is a very exciting time for us. But before we get started talking about that real quick, a little bit about ourselves. My name is Ryan, you can find me on Twitter. Say hello. I have been writing Rust for quite some time now before I joined Microsoft, before 1.0 came out, and I was the MC of the very first RustFest, so it is extremely exciting to see so many people here who are excited about Rust. Back in the day, it used to be I would mention that I liked Rust and people would go: what's Rust? And that's happening a lot less often now which is really, really cool.
SEBASTIAN: Hello Barcelona, my name is Sebastian, I'm a security engineer which means I work with vulnerabilities in software, those kind of things, I have been also writing Rust for six years now. I actually checked yesterday for how long I have been writing Rust and I found that my first contribution to the Rust compiler was in 2013 and it was a tiny batch, four lines, but it was a good way of learning Rust for me so yes, here we are. We both work for Microsoft, opposite sides of Microsoft. To be honest we don't even work in the same country. Ryan works from Germany, as a developer advocate, and I work in the Microsoft Security Response Center. So yes, not really in the same team, but we teamed up for pushing for Rust adoption in Microsoft and so we are part of a team that we like the same programming language that you might have seen in some of the blog posts that we have been publishing.
RYAN: So we are going to be talking about adoption of Rust today at Microsoft but before we begin with that, the first thing to talk about is the problem that we are actually trying to solve. Obviously we don't want to adopt a programming language just because it's cool or just because we two like to write it, as much as I wish that were possible. You don't get very far with that kind of reasoning and so we are really trying to address a problem here. This isn't really a problem that affects us as a company only; it's also a problem that we believe impacts the entire industry, the entire software industry, and because of the situation we've got ourselves into as the software industry that really impacts the entire world for better or for worse. This is, we should say, a billion dollar problem. It really impacts us in an extremely deep way and I know money is perhaps not the best way to measure this, but you can think of it not only as a problem that causes us to lose money but also causes everybody in this room, everybody in the world really, a lot of headache and pain. At the core of this problem, at the very, very core of this problem is the technology that we have been using in the software industry that has gotten us very, very far and been really, really great at a lot of things, but is really starting to show for lack of a better word its age. That technology is C and C++. C and C++ are extremely great at writing low level systems. They use very little resources on the machine. They are, in fact, really the basis on which we create our systems today but the issue with that, of course, is that they are very, very unsafe and, when they were developed, did not really have safety in mind. When we say safety, really we mean the ability to write secure and correct code with it.
SEBASTIAN: So I would like to introduce a little about what the security response centre does and it's basically an organisation in Microsoft which centralises the management of vulnerabilities. So it means that we work with external researchers, internal researchers as well, to find vulnerabilities in the Microsoft products. We work with the broad teams then to fix them and also try to avoid developers from repeating these mistakes. We also work in different mitigations to prevent degradation of the vulnerabilities and we have all this data about vulnerabilities and for the sake of the argument today this is a binary classification of the vulnerabilities that we have. I've seen that Peter mentioned this 70% before so I don't think it requires much introduction but yes, around 70% of the vulnerabilities that we patch in our software are related to memory safety. As you can see, this has been maintained in time, so the problem is not getting any better.
RYAN: So naturally the question comes up then: okay, this seems to be a problem that you have vulnerabilities but how much does this actually cost you? How much does this actually cost the industry as a whole?
SEBASTIAN: Well, a very conservative estimate, this is $150,000 per issue. You can see this is a lot of money and this is only for Microsoft. It's not accounting for the work and time that our customers have to spend rebooting machines, installing the patches and doing everything else that we have to do.
RYAN: Okay, so $150,000 per issue, but okay, if there's only one issue per year then perhaps that's not too bad. How many issues are we actually talking about?
SEBASTIAN: Only in 2018 we had 468 issues. You can see that this is a lot of money if you make the maths. Sorry, if you make the maths and types the amount of money per issue, yes, this is a lot. As we can see, the issue is not getting any better, it's actually getting worse year over year and only in 2019 we had over 470 issues and the year is not even over yet. Sorry about the mic.
RYAN: All right, so the issue is getting worse over time. That's not so good. But is it possible that the cost could be even higher than what we are talking about here?
SEBASTIAN: This cost is only for when the good people find the issues, they report it to us, then we patch them and then everyone is happy, but what if malware also finds the bug and writes for it? This has happened many, many times in history but there are a few occasions that it got out of hand. I don't know if you remember SQL Slammer back in 2003? It was a worm that was exploiting a vulnerability in the SQL server, then was infecting the machine and using that machine to infect other machines. Then we had Configure that was exploiting also, to your surprise, a buffered overflow in one of the RPC servers of Windows, infecting the machine and then also using the machine to distribute the virus to the other machines. And most recently this one. I don't know if you were affected by it or your organisation, but you probably know more than one person that was affected by it. This worm was exploiting a server of Windows, then was infecting the machine and asking for I think $300 to decrypt the device. This worm was distributed to over 200 countries only in the first week and to give an example, the national health care system of the UK was affected by this ransom. There's an estimation that it cost that organisation only $96 million. The global estimate for this ransom was around $4 billion. Yes. That's what you are seeing, it's nine zeros. Some people would estimate even over $8 billion. So as you can see, this is a lot, a lot of money.
RYAN: All right, so we have these memory safety vulnerabilities that are causing lots of headache and pain, upwards of $4 billion at a time. That's really not a good thing but how could we actually fix this issue? There have been many ideas that have occurred over the years, some better than others, and we are going to talk about a few of them. So the first bright idea that was had, and it's an idea that you often see on a particular orange website nowadays, is that we need better programmers. All right, we are done, thank you. [Laughter]
No, please stop. This does not work. We need better programmers is not an answer to this. Training does help. You can try to mitigate some issues by making people better at writing security software, so we are not saying that you should not train people, but this is absolutely by no means a way to address this issue. You can have extremely highly talented, highly trained individuals writing code and they will still make mistakes. So, no. Now, that is how of the way, that's good. What about another great idea? This one does work a lot better. That is the need for better analysis tools. It's something that has been happening over the years quite often, both static and dynamic analysis tools that allow you to look at a program and determine if it has vulnerabilities before it ever reaches anybody's machine to actually run, and this works fairly well over time. And it's something that we still want to actively invest in because no matter what happens after we leave the room today the world will still continue to have plenty of insecurity code in it. People will continue to write in insecure technologies, and so we need the ability to take a look at that code and determine if it has vulnerability inside. But these tools are not perfect and we are convinced that they can never be perfect so there's really only one idea left that we can actually use, and this is really the only way that we can address this issue fully. That's the idea that we need to make these issues impossible to introduce in the first place. Over time there's actually been a lot of active research into this area: how to make these issues impossible. There's a few things that have been introduced in software that allow them to not actually occur.
SEBASTIAN: Okay, so I think to understand the issue we have to understand the problem, to find the solution. So we can classify the vulnerabilities in two big classes of vulnerabilities. One is related to spatial memory safety. What does it mean, that when you have a pointer or a buffer, some type of memory, it usually has a size. It's not infinite. So when you try to index out of that buffer or pointer, you get what C developers call undefined behaviour, so it means that everything is wrong and it is not doing what you are expecting it to do. For fixing this issue, there is a clear way that most languages have now, that's a run time check. When you allocate memory you can embed that information in the pointer or in the buffer, so now you have a rich pointer or a fat pointer. Yes. So when you try to index that pointer now you have a run time check that you know how to do anything, and if the index is outside the bounce it will fail the memory access and nothing bad will happen.
The second type of memory safety that we can talk about is spatial memory safety. This means that, when you claim memory or when you allocate memory that memory will be valid only for some time. Once it gets freed, you shouldn't still be using that memory but sometimes in the languages you don't clear the references and then you are going to still access it. It's a way of solving that that a lot of languages use and that's using garbage collection. What does it mean? You can claim memory but you cannot free memory manually. You expect the run time to free the memory, and this is basically always leaving the memory there and then you will have this program inside the run time that will go and find the memory that is not used anymore and free it.
RYAN: But of course garbage collection does come with its downsides. There are a whole class of programmes out there where a run time like a garbage collected run time is not really acceptable so if you are writing perhaps a low level system like a database or something like that, more often than not you don't want to pay the penalty of having a garbage collector which comes with performance penalties and also with penalties for not really knowing when certain things will run, and that's just oftentimes unacceptable. But we believe that there is a technology out there that allows us to have our cake and eat it too. Spoiler alert: it's Rust! Rust is not particularly exactly perfect in this regard and we are going to talk a little bit about the caveats to this, but Rust does allow us to write performant systems programmes in a safe way, and to look into what Rust actually gets us in practice, we will take a look at this.
SEBASTIAN: So Ryan was talking before about making these issues impossible and that's kind of a hard thing, but what if we could isolate what is unsafe from not unsafe? What's unsafe from safe. When we are looking at the C++ code base we find that 100% of the code base is written in unsafe - you have no way of isolating what is safe from unsafe. For example, when we are talking about software, we have to make threat models about that software. What does it mean, that you have to make an analysis of how the software can be attacked. If we are talking about a browser, for example, for a browser the threat model is that every website can be malicious. If we are talking about a kernel, we have to start from the ground up. Each program running in user space will be also malicious so that has to protect from every program.
After you have the threat model for these two examples you come to the conclusion that almost all the code is exposed to the malicious input.
When we look at Rust, there is a clear boundary of what is safe/unsafe and this is a big advantage for security software engineers like myself that we have to audit these code bases. We made an analysis of the most used and came to the conclusion that the amount of lines that were unsafe in the top 200 grades was only 1%, and I remember I was at RustConf watching Jeremy's talk and he was talking about Rust adoption in Facebook and he was saying that the technology has to overcome, to be ten times better than the next thing that is available, and as we can see here Rust is not over that threshold but we can consider that in this aspect it's 100 times better than the next technology available.
RYAN: But of course we are at RustFest so we are kind of preaching to the choir here. Mostly everybody in this room probably already understands this. You may not have selected to use Rust or this property but you've probably heard about it before, and so let's just say for the sake of argument that we are going to use Rust. We have chosen it, we have been able to convince ourselves that this is a good thing to adopt. What does that actual adoption process look like? And this is really where it actually begins because, while Rust might present a very interesting opportunity for us as an industry, we have a whole ton of C and C++ code written out there and we can't just snap our fingers and have it written in Rust. It's going to be a very, very, very, very long time until that goes away, even if we were all 100% convinced that that's the right thing to do.
So when we talk about Rust adoption, the first thing that we should take a step back and look at is actually how do languages get adopted, and particularly how do they get adopted in existing large companies? Because if it's a start-up, it's quite easy. You can just choose to use a new technology because you haven't written any code before, but if you have a huge code base that already has C and C++ in it, what is the process actually like? The issue with this is the following: you have costs and you have benefits. I don't know if you can see it, but down there it says "benefits" but they are very hard to see. The costs are quite clear. You have the cost of introducing the code inside of your code base, of introducing the compiler inside of your build system. These are all very clear costs and are actually very easy to reason about how much it will end up costing your organisation to actually adopt it, but the benefits of introducing a new language may not be so clear. If you are introducing a new language because it has a great type system and you think that that type system will not introduce as many bugs as the current language you are using, how do you actually prove that? It's extremely difficult to actually measure the impact of something like a more advanced type system. But at the end of the day this is what makes Rust more adoptable because for Rust that picture looks more like this, where the costs are still quite clear but at least this one benefit that we have been talking about of memory safety is extremely clear. We know that Rust will get us much further along to writing memory safe code and we know that memory unsafe code leads to billions and billions and billions of dollars of damage, so it's quite easy to go to people and say: how about we stop spending billions of dollars on this stuff? They go: sounds good. Of course, we've only written "benefit" here. It's not to say that Rust doesn't have other benefits, it's just that those are harder to talk about and harder to convince others in a large organisation to bet on Rust because of them, so really when it comes to adopting it in a large organisation, it really is enough to just talk about its memory safety properties in order to convince others that it's worth adopting.
SEBASTIAN: Yes, so maybe - just maybe - we shouldn't be writing security critical software in C++ because of the things we were mentioning before, so now the bet is: what do we use for writing this security critical software? This is very Rust. Rust has - you can write performance code and having all these extra things that are related to the memory safety of the language. But we cannot just ditch everything that we did until now and start with Rust. So what we can do now, do we just keep continually using the thing that we were using before, which is C++, or we can maybe try integrating Rust with our existing code bases and technologies. This means that we have to integrate Rust with our infrastructure and that's not easy because Rust is not just a language; Rust is a set of tools that you are bringing to your infrastructure. Especially for Windows people, Rust comes with this beast, the backend that Rust uses, that is LLVM and LLVM has great support for Linux and others but Windows support is not really there and one example is, for example, LLVM didn't have a link for Windows until a few months ago. Continuing on bringing all these Rust tools to your infrastructure is that we have a lot of tools that we have to run in our pipelines to generate the binaries and to check that the binaries are okay.
An example is that we check all our - all the binaries that we ship to our customers for performance. That means analysing all the binaries at run time, checking that they are doing well and sometimes writing those binaries, it means re-arranging the basic blocks that are inside so they are more performant and more friendly, and the problem that we have with those tools is that those tools are used to analyse binaries produced by MSVC and obviously you cannot just expect that LLVM and MSVC are generating the same type of binaries. They have different naming, different symbols, they arrange different information in a different way, have different ways of dealing with code so this is a whole new set of tools that we have to adopt, and continuing on this line of tooling, we have also build systems that have been optimised for many, many years to perform well and to comply with a set of rules that we have. Just for a second imagine theoretically that you want to include a Rust component inside Windows and these challenges are that when you want to use Rust code there, you have obviously cargo which people use to build Rust binaries, and cargo is a great tool. Who can imagine using Rust without using cargo? You are losing half of the features. Cargo is a great build tool and a great package management tool. But it has to be integrated in an existing system. We cannot just let cargo manage our whole build system. Obviously for cargo there is Rust code and not Rust code. There is nothing else for cargo.
Continuing on this line, we have - we must be able to interoperate with the existing software as I was mentioning before, and the software is usually written in C and C++, and sometimes you have a clear boundary between these different DLLs maybe. One is written in C and another will be written in Rust in the future. Rust fortunately is very good at interoperating with other technologies because it supports most of the APIs supported by C. Also, another great tool that comes with Rust is bind gen. Unless you take a C header and generate a Rust binding for it, now you have all these bindings that you need to make a safe wrapper around it so maybe the idea is that in the future we can kind of guess if we can make a safe wrapper around these C headers.
The last one is crazy but there have been many technologies to make interoperability languages agnostic. COM and WinRT, these are well defined standards because C++ didn't have a well-defined API back then. And these are things that the community have been working on. We analyse most of the things available for COM and we decided to also write another that we've released this month, I think. Yes, so we are getting there with interoperability.
Also we have had problems that common people don't have. Actually, common people is a bad description, but individual developers maybe don't have or small companies, but big companies put processes in place to prevent errors from happening and it means that everything that you bring into the company has to - must comply with a set of rules and one much them is having a trusted toolchain. When you are developing with Rust, you execute in the PC and then after two minutes, depending on your Internet connection, you have Rust available to be used, but when you are bringing Rust into a system like the Windows one, you just cannot pull a binary from the Internet and run it there. Even more if these binaries connect into the Internet, are pulling even more binaries from there, so using Rust up there or pulling the Rust compiler into the system is not an option. It means that we have to build our own tools and it is a big challenge because we have to start from a trusted source code and a trusted binary and then build the trusted binary that we can use in our build system.
The following one is about - these are all examples. There are many more, but we have binary security policies. All the binaries that we ship to our customers comply with a set of rules to make them more secure, so for example all the binaries that are shipped have a few other memory expiration mitigations enabled in them. And there is something to protect the contraflow of the program so malicious input cannot hijack it and then do something that the program wants the program not to do. This mitigation wasn't available in LLVM. Now we can happily say that we made this mitigation and contributed it back to LLVM and it has been in master I think since last week and we hope we can enable it in Rust in the following months as well, so super happy about that.
Then we have also the humans challenge, we can call it like that. This is probably the most interesting to me. I'm used to dealing with machines, but not really with people. You have this trouble with convincing people to use new technology. Fortunately for us, Rust is a very adoptable technology but when you try to tell someone: hey, maybe we should use Rust because it's memory safe - the first thing they think is, "Hey, we are already have C# for that, why are you suggesting this?" Then you have to start explaining that Rust doesn't have a garbage collector and it doesn't have a run time and it's more similar to C++ than actually C#.
RYAN: The last point is you have to be able to train people who might be used to writing C and C++ for five, ten, 30, 40 years, and be able to show them how to write Rust code, that they might not be used to writing. The good news about this is that before, when we've introduced Rust to seasoned C++ programmers, they generally are able to get it rather quickly because it kind of just formalises things that they already have in their head. When people are coming from other backgrounds, it might be a little bit more difficult, but while the learning curve is quite steep, generally people get through it and once they are through that learning curve, they are quite productive, which is really nice to see. But we have been talking for a while now about the challenges that we are facing as a company and, okay, that's great, we are going to face these challenges, but there are also challenges that we as an entire Rust community have to face when Rust becomes more and more adopted.
One of the first things to talk about is the idea of governance. Really we want to, as a community, ensure that Rust is owned by the community and not by any one particular entity and as more and more companies come in and want to use Rust, there's always the temptation to start taking control, and that's something that we do not want. We want to ensure that Rust stays in as a community-owned project and directed by the community because frankly it has gotten here because of the community and the community is the best one that will be able to take it forward from here on out. Really what this means is that we want to take the community principles that we as a Rust community have established over time and really establish them in more or less a formal way, and of course there's great work going on already from the governance working group and a lot of talk happening in many directions around this, and we are really excited to see this go even further.
With that, over time Rust will continue to change. We just had async-await come out yesterday which is really exciting. The language that as it looks today, although it still compiles, looks different than it did in 1.0 and the language will continue to change over time. But we want these changes to continue to benefit the majority of people. We've already shown that we have some rather interesting and perhaps esoteric needs that maybe most people in this room won't ever need in their language, and I am sure that we will be introducing or helping to introduce some changes to the language that make the job of creating Rust software at Microsoft easier, but we want these changes to at the very least not disturb anybody else who doesn't need them, and at best make sure that they are actually helpful for everybody else and so this is extremely important. Going back to the idea of community, we want this design to be again community-driven, so just like the RFC process has happened in the past with input from everybody in the community if they are able and willing to give it, we don't want some back room community design happening and then, plop, there's changes to the language all of a sudden. We want these changes to be driven by the community as a whole.
The last point that we want to talk about is this idea of reluctant Rustaceans. Presumably everybody in this room is here voluntarily? [Laughter] At least I hope so. But in the future that might not be the case. I started writing Rust because I wanted to, not because I had to, but in the future there probably will be people who have to write Rust and they may not want to. What does that do to our community? As we introduce more people into it who are kind of reluctantly there. The real challenge with this is maintaining what I like to call the community spirit or the thing that we already collectively have as a community, the they think that in my opinion makes Rust awesome. How do we continue to have that as more and more people who are not necessarily there by choice interact with us and come into our community?
Okay, so we are at the end now and this is really where we ask you to come talk with us, come engage with us. Even if you work for a small start-up or you are just a hobbyist, maybe you just like to use Rust in your free time, all the way up to you are working at a huge company and have some of the similar problems that we have, we would love to talk with you, hear from you, work with you and really what we want to be as Microsoft is a part of this community. We are not here to take over; we are here to be a part of it and work with all of you. So thank you very much. [Applause]