Author | Cai Fangfang
Not long ago, the Moderation Team of the Rust programming language announced its collective resignation through GitHub, sparking a heated discussion about open source governance in the tech world. The attention of the Rust community has also soared, and related reports continue to swipe in the circle of friends. An insider joked about this: "I knew Rust fire, I didn't expect it to be so hot." "The last time Rust received such a high level of attention was in early 2021 when the Rust Foundation was officially announced.
On February 9, 2021, the Rust Foundation was established, with Mozilla, Amazon, Huawei, Google, and Microsoft as founding platinum members, and the five companies committed to investing no less than $1 million a year in a budget of no less than $1 million per year over two years to develop, maintain, and promote the Rust project.
Many people were surprised by the presence of Huawei among the founding members of the Rust Foundation, because they had not heard much about Huawei's investment in the Rust project (the other four founding member companies had a large number of Rust project team members). In fact, huawei has many products written in Rust, such as StratoVirt, which is a rust-based next-generation virtualization platform under Huawei's openEuler operating system, which has been developed since 2019. And if you trace the opportunity of Huawei's initial association with Rust, even as early as 2017.
Recently, InfoQ had the honor to conduct an exclusive interview with Hou Peixin, chief expert of Huawei's open source and director of the Rust Foundation, and a number of technical experts engaged in Rust development at Huawei, to explore the people and things behind Huawei's vigorous investment in the Rust community, to further understand the Rust work that Huawei is carrying out, and to think about the direction and future development of Rust's key technologies.
Interviewees:
Dr. PEIXIN HOU, Chief Expert of Open Source of Huawei, Director of Rust Foundation, Director of Linux Foundation
Professor Yijun Yu is Huawei's chief expert on trusted programming
Guillaume Gomez, Huawei Rust technical expert, Rust official dev-tools (development tools team) Member, rustdoc team leader
Dr. Amanieu d'Antras, Huawei Rust technology expert, Member of rust's official library team
Li Yuan, Huawei Rust engineer, major contributor to the Rust ARM SIMD roadmap feature
Rust is on a new journey
As a general-purpose system-level programming language, Rust has attracted a lot of developer attention in recent years due to its excellent memory security mechanism and performance advantages comparable to that of C. Since 2016, after the official release of version 1.0, Rust has been named the "Most Popular" programming language in the Stack Overflow Developer Annual Survey for six consecutive years.
In addition to being popular among developers, Rust has also received public support from many major manufacturers at home and abroad. Major vendors are using the Rust language to support a wide range of application scenarios, from websites to developer tools to video games. Facebook has been using the Rust language to build Diem Coin (the original Libra Coin); Amazon has used Rust to write code for products such as its serverless platform Lambda (which allows developers to easily run and extend development code without having to manage the underlying servers); Microsoft has publicly stated that it will explore the use of the Rust programming language as an alternative to C, C++, and other languages to improve the security posture of applications. It has also launched some experiments to rewrite Windows system components using Rust; Google has also begun using Rust in settings that place high demands on memory security and performance, including in the critical Chrome browser and Android systems.
While the Rust language itself has maintained good momentum, the Rust team experienced a wave of turmoil in 2020. Rust was founded in Mozilla, and the first very important use case was the development of the Firefox browser. In August 2020, Mozilla announced organizational changes due to the pressure of the pandemic, laying off approximately 250 employees, including active developers from the Rust project and the Rust community. The layoffs have created a lot of uncertainty for the Rust project and have caused the community to worry about Rust's future fate.
The end of this turmoil was the establishment of the Rust Foundation, which was seen by many as a bullish signal for the future of the Rust language.
The Rust community has long had the idea of promoting open governance and attracting more players, and the August 2020 layoffs at Mozilla due to the pandemic were an opportunity for members of the Rust community to accelerate the process. In the second half of 2020, Huawei learned of this news during its communication with members of the Rust community. Optimistic about the future development of the Rust language and support for the Rust community, Huawei took the initiative to join this process and applied the previous experience of participating in the governance or creation of foundations such as the Linux Foundation and CNCF. Eventually, members of the Rust community worked together to establish the Rust Foundation, and Huawei became a founding Platinum member.
Huawei has been exploring programming languages for a long time, and has also been following the development of Rust, and on Huawei's knowledge-sharing platform, the introduction and analysis of Rust can be traced back to 2013. As early as 2017, the prototype development of Huawei's microkernel operating system tried to use Rust.
According to Hou Peixin, there are a large number of scenarios in Huawei's business form that require high network security and high performance, and the problems caused by incorrect memory access in network security account for a fairly prominent proportion, and Rust is a system programming language that introduces memory access problems during programming while maintaining high performance, which is highly compatible with Huawei's target scenarios. In the past two years, the industry has paid more and more attention to memory security issues, coupled with the gradual maturity of the Rust language itself, Huawei has more and more products and platforms begun to use Rust development.
Before and after the establishment of the Rust Foundation, Silicon Valley simultaneously set off a Rust talent war. Many active developers of the original Rust team have turned to google, Microsoft, Amazon, Facebook and other technology giants, and Huawei has also recruited some of the talents of the official Rust team. The movement of Rust talent actually marks a new turning point in the development of the language community.
Hou Peixin believes that the rapid development of the Rust language is due to a large number of individual contributors, which makes the Rust community quite open and diverse. At the same time, the long-term development of an open source technology is also inseparable from the use and continuous investment of a large number of commercial companies, so some Rust developers are recruited by commercial companies and continue to invest full-time in the community, which is a very good guarantee for the sustainable development of the community and technology. While solving the technical problems encountered by commercial companies, these developers will also fall into the community's version and technical roadmap, which can increase rust's efficiency and technical resilience in large-scale commercial scenarios, and the result will be a win-win situation. In his view, the establishment of the Rust Foundation is to ensure that Rust technology and community can continue to receive support from funds, developer expansion, branding, etc., and the charter and structure of the Rust Foundation's open governance will also try to avoid the monopoly and control of a certain institution on the community, and ensure the openness and diversity of Rust.
In order to better support the many individual developers in the Rust community from all over the world with different cultural backgrounds and personalities, the Rust Foundation recently launched a community funding program to provide financial support to the maintainers of the community and related ecosystems (Huawei is also one of the first sponsors of this program), and on the other hand, it is also helping team members in the community to sort out and adjust the governance structure of the community to ensure the long-term development of Rust.
As the only Chinese technology company among the founding members of the Rust Foundation, Huawei will actively assume the responsibility of large-scale promotion and ecological construction of Rust in China, including discovering and empowering more domestic developers and application scenarios, and promoting China's transformation from users to contributors.
Huawei's technology exploration around Rust
At present, within Huawei, Rust is mainly used in embedded system development, system drivers, cloud computing, virtual storage, network transmission protocols, concurrent programming framework basic libraries and other products. At the same time, Huawei's Trusted Theory, Technology, and Engineering Lab is building software engineering capabilities based on Rust, and is working on internal capability improvement, including exploring advanced code parsing, security analysis, and other tools to solve memory security problems in existing embedded systems.
Huawei's exploration of Rust is not only the use of Rust, but also plans and leads some very important work in several major technical directions of the Rust community, including language features, core libraries, standard libraries, basic libraries, documentation, and construction toolchains. These include C-to-Rust conversion, inline assembly, cross-compilation, Parking Lot concurrent library, SIMD base library, document navigation, code polymorphism, hot patching, AOP, build optimization, clone detection, deep learning security algorithm recognition, etc. There are many features that are important work in Rust community roadmaps and RFCs, and are also being developed by Huawei's Rust technical experts.
There are both newcomers like Li Yuan and senior Rust experts like Amanieu d'Antras.
Born in 1998, Li Yuan has only been involved in the Rust community for more than a year, but he is now the main contributor to ARM SIMD, the official Roadmap feature of the Rust community. He is very optimistic about the development potential of Rust, and believes that Rust will become a new generation of memory security system programming language, which can better solve the memory leakage, buffer overflow and other security problems that cannot be solved by traditional system languages (such as C language).
In his more than a year of participation in the Rust community, Li Yuan has solved many simD feature-related problems, including the problem that some SIMD interfaces cannot be implemented due to the IR generation mechanism of the Rust compiler. The single-instruction multiple-stream (SIMD) feature is important because it is a computational acceleration technique that can significantly improve program efficiency, which is an important consideration for evaluating programs and programming languages. According to Li Yuan, the SIMD features on the x86 architecture can already be used directly through the standard library under the stable version of the compiler, and the SIMD features on the ARM architecture are about to be stabilized.
Amanieu d'Antras came into contact with Rust much earlier, around 2014, when he heavily used C++ at work, but then switched to Rust because of the complexity of C++. In his view, Rust retains the core idea of zero-cost C++ abstraction while making significant improvements in language design that are completely impossible for C++.
Amanieu d'Antras, whose current focus is on improving the usability of Rust to write low-level code, leads a Rust working group to add support for inline assembly for Rust. According to reports, the current working group is progressing smoothly, and plans to provide inline assembly in the stable version of Rust by the end of this year. In addition, Amanieu d'Antras is a member of the Rust library team, responsible for the maintenance of the Rust standard library and some related crates.
Amanieu d'Antras told InfoQ that one of the biggest challenges in Rust development was maintaining Rust's stability guarantees. Rust provides a very strong API stability guarantee, ensuring that even Rust code written in 2015 (when Rust 1.0 was released) continues to compile with the latest version of the compiler. This means that once the library team adds an API to the standard library, it has to be supported forever and cannot be changed. "This put a lot of pressure on the library team because team members need to be very careful when reviewing APIs: any mistakes we make can become permanent!"
Earlier this year, Huawei submitted a series of codes for the Rust compiler, enabling Rust compilation targets to support the ARM AArch64 32-bit big-endian variant of the ILP32 chipset, enabling communications vendors, including Huawei, to execute Rust native programs on common network hardware architectures. The code was submitted via Amanieu d'Antras to open source projects such as the LLVM compiler, the libc library, and the Rust compiler.
Rust open source contributions made by Huawei employees (existing work)
Rust is too difficult to learn, is there a solution?
With the efforts of all rust community members, including Huawei, the Rust language is becoming more and more mature and complete, which is a good aspect of Rust's development, but on the other hand, Rust's actual user growth data is not as good as other data seem.
According to the newly released TIOBE programming language ranking data released in November, Rust accounted for only 0.54%, ranking 29th. Although this proportion of data and rankings has improved slightly compared to the beginning of this year, there is still a big gap between the Go language, which ranks 18th (down five places from November last year, accounting for 1.21%).
In last year's Stack Overflow Developer Annual Survey, although respondents continued to name Rust as their favorite programming language, 93% of respondents said they had never used Rust, and their love for The Rust language was still in the "want to learn" stage. Much of the reason for this is due to Rust's long-criticized learning difficulty.
In the 2020 Rust official survey, when respondents were asked what advice they would give to increasing Rust adoption, one option many mentioned was to make Rust more difficult to learn and make Rust easier to learn. Of those, 15.8 percent said they would use Rust more if they were "less intimidating, easier to learn, or less complex."
Therefore, in addition to the work around the core technology direction of Rust mentioned above, Huawei is also making some technical preparations internally around improving the learnability of Rust.
First of all, for the cross-language code conversion migration of existing software projects, Huawei has developed a set of TXL code conversion tools to achieve the ability to migrate from C projects to Rust projects, and increases the proportion of security codes in C code based on patterns; secondly, through deep code learning, let the machine learn some simple tasks for people, such as Huawei's self-developed deep code learning Corder tool has been able to achieve automatic classification of unsafe Rust code, and the accuracy rate can reach more than 93%; and more Optimizations to Rust documentation, such as by generating interconnectable variable-type documents, make rustdoc easier to navigate. Guillaume Gomez's current focus on rustdoc optimization in the Rust community makes Rust documents faster, lighter, and more interactive. He is both the rustdoc team leader and a member of Rust's official dev-tools (development tools team).
In addition, in order to better promote Rust best practices, the work of developing Rust coding specifications is also steadily advancing. According to Yu Yijun, the Rust coding specification was created by Rust expert Zhang Handong and Huawei's internal Rust developers in the product development process. Last year, a version was completed, and this year it has been refreshed and enhanced on the original basis. The team has expanded and improved huawei's internal mainstream programming language compliance requirements and rust specifications of major industry manufacturers, and also standardized the unsafe Rust code encoding.
During this time, the team also introduced specification checking tools such as Clippy to ensure that these coding specifications not only guide programmers to code by hand, but also automatically pick out the details of the specification violations, improving rust development efficiency and quality. The team plans to roll out version 1.0 next year to be further rolled out within the company.
As a member of the Rust Foundation, Huawei hopes to contribute this coding specification to the community and build it with the community in the future. Huawei's internal version will be the same as the community version, but huawei's internal use of the specification will be more stringent, there will be some mandatory rules that do not allow violations to meet the quality requirements of system software development.
According to Huawei's statistics in project practice, on the basis of the above learningability optimization work, the existing C/C++ team only needs 2 to 4 weeks to contribute the productized Rust code. From the perspective of coding, testing, publishing, and maintenance, R&D productivity can be improved, and memory-safe and reliable code can also bring higher value to customers.
Yu Yijun said: "Soon, the problem of the learning curve should be more of an excuse for not wanting to try new things." ”
"No one language can solve all problems"
Referring to Gartner's technological maturity curve, Yu Yijun believes that the current stage of development of the Rust language should have passed the peak of hype. Mainly reflected in several aspects: the rust compiler version has launched the third major version (Rust 2021) corresponding to the stable version 1.56.1, bugs have been very few; the number of Rust libraries in the open source community is the most popular boutique, the top 100 libraries GitHub Star at least more than 3000, the top 20 libraries download at least more than 10 million; Rust toolchain technical support is very strong, most of the issues can be solved in time Compared to other system programming languages, the Rust compiler is consistent, there are not so many different implementations, and the new language features can be quickly used by the most developers, which is the advantage of experiencing the various C/C++ compilation software versions of embedded development. And all of this means that Rust is relatively mature.
Yu Yijun said that there is also a trend worth mentioning, that is, Linux Kernel and and Andriod, Fuchsia and other operating systems part of the driver code is shifting to Rust, in his opinion, if the number of use of Rust machines, Rust may soon be at the top.
Although he is very optimistic about the future development of Rust, Yu Yijun also admitted that as a language, there are still many technical problems that Rust can improve. For example, how to learn from the advantages of some other programming languages and learn from each other's strengths? How do you ensure the safe use of the language Unsafe code and prove the security of the Rust toolchain itself? These are all questions that the Rust community needs to think more about.
In addition, if Rust wants to be more widely adopted, the cost of programming language migration is a difficult hurdle that must be faced.
Rust's learning curve shouldn't be a barrier to its use in existing projects. However, the migration of any programming language has a cost, and the smooth migration of trusted code that Huawei is currently exploring aims to reduce this cost and make the migration work less difficult. Yu Yijun said that Huawei's accumulation of C/C++ code and other programming language code over the years is massive, and it is a huge project to enable various product lines to migrate some of the security-critical code to Rust.
At present, the team mainly adopts an enabling approach, through the development of better migration tools and more perfect tool support, so that the migration cost is reduced below the revenue, so as to promote a large number of developers to migrate spontaneously. At present, Huawei has piloted this migration tool in three products and achieved the expected results, the API security of the function interface has been increased to more than 95%, and the converted Rust project has correct functions and can also run on the embedded product board. Further refactoring and optimization of the converted Rust code, as well as meeting the product line's customization requirements for Rust code, are the focus of the team's next phase.
Yu Yijun stressed, "No language can solve all problems. While further promoting Rust applications, we should also see that some scenarios are better without Rust, or even better without code. In his opinion, these cross-language problem-solving capabilities, even program analysis, generation to synthesis, if Rust can do better than other programming languages, and it is easier to customize related to the application field, then its development prospects will be better.
At present, Huawei and domestic and foreign universities are jointly exploring technology, and at the same time trying multiple paths, hoping to apply the latest achievements of academia to language at the fastest speed. One of the biggest challenges is how to find language features that are appropriate for Rust's development from the needs of the community, make the methods that can solve specific problems or more academic prototype tools and technologies more versatile and practical, and make the knowledge graph about Rust more known and useful.