2021 InfoQ software industry development trend observation

Written by | Daniel Bryant, Wesley Reisz, Thomas Betts, Charles Humble, et al

Translated by | Hirakawa

Planned by | Ding Xiaoyun

As 2021 draws to a close, the co-hosts of infoQ podcasts met to discuss the key trends in the software industry that we've observed over the past year, and what they will continue to focus on in 2022. In addition to this summary, you can listen to the podcast episode or view the full transcript of the discussion.

Daniel Bryant moderated the discussion, which was attended by Wesley Reisz, Shane Hastie, Charles Humble, Srini Penchikala and Thomas Betts.

Hybrid and telecommuting

The most significant change in the way teams work in 2020 was the forced shift to telecommuting, which has now evolved into a "hybrid" office and is likely to become the standard of the future. Shane Hastie, editor-in-chief of InfoQ Culture and Methods and host of the InfoQ Culture Podcast, said: "We will never go back to a life of five days a week, three hours a day. In the software world, most people don't need to stay in the office every day to be more productive."

However, this does present a new set of challenges. In the mixed office model, team members are mostly at home and occasionally in the office, which requires finding the right balance to let the team work together. Some organizations are increasingly adopting asynchronous patterns and minimizing or even eliminating synchronous meetings, such as Zoom teleconferencing. Hastie says we need to make shifts in culture, philosophy and mindset so that these new ways of working are to be successful. In the long run, he argues, the biggest impact will be to see which trials will succeed, learn from failures and make adjustments.

We are in the midst of a wave of "big resignations" and it's clear that individuals are placing more emphasis on the well-being of themselves and their families. This has to do with the developer and employee experience, and organizations must work harder to prove that they can provide a great work environment.

The past year has been mixed in terms of diversity, inclusion and equity. There has been a reversal in some areas, most notably that women have taken on more pandemic-related responsibilities than men. Still, diversity and inclusion have improved in companies that currently employ remote employees rather than those who live near offices or can relocate.

Mixed environments have the potential to create a new divide, and companies need to make a deliberate effort to treat all employees equally, whether they're in the office or remotely. Wesley Reisz mentioned QCon Plus as a topic focused on returning to the office, with example being Netflix's dedicated approach to enabling leaders to better connect with their remote employees.

Thomas Betts presented "The Spectrum of Synchronous," which was the subject of James Stanier's talk on the topic. One extreme is that everyone is in the office and the communication is completely synchronized, and the other extreme case is that everyone is remote and all communication is asynchronous. A hybrid environment is somewhere in between, with an appropriate balance. For document-like stuff, there's nothing wrong with being asynchronous, but it loses its humanity, so it still takes some face-to-face time in order to have a healthy environment.

Another aspect of remote and hybrid work is building networks of connections and relationships. This is especially challenging for new young employees who may start their first job remotely. Companies and old employees should consciously help young employees to make connections.

Architectural trends

Microservices remain a major architectural model, and in 2021, Thomas Betts proposed Conway's Law's COVID inference that a company's ability to develop distributed systems is directly related to their ability to maintain full remote work. Conversely, if your processes rely on many people communicating in the same room, you may struggle with your transition to telecommuting and will most likely not be able to successfully adopt microservices. In addition, effective asynchronous communication between remote teams can be a signal of asynchronous software development capabilities.

In terms of tools and standards, in 2021, AsyncAPI has matured a lot, and there are many new tools and languages that support the specification. It's a few years behind the OpenAPI specification it's based on, and growth and adoption are increasing in a similar way. This will allow more people to use contracts to define boundaries for asynchronous communication, as well as to use code to help developers who integrate event-driven architectures.

Data grids are also beginning to emerge as a new trend. The idea is to apply the concept of domain-driven design to data, and data products are just an extra responsibility for the development team. In 2021, we've seen a lot of people talking about data mesh, and Betts is looking forward to seeing companies start implementing it in 2022 and sharing their success stories and the challenges they have to overcome in reality.

AI 和 ML

In 2021, Srini Penchikala observed 4 trends in AI and ML. The first is data management. "Data management includes data ingestion, data storage, data processing, data consolidation (using GraphQL or other API-based data access solutions), and then data processing and analytics." Streaming is the main trend, and data is generated very frequently, for example from IoT devices or self-driving cars.

The next two categories are infrastructure and operations, Penchikala says, and the data space, which typically lags behind in terms of innovation, is now catching up with the trends we're seeing in development and architecture. For infrastructure, companies are leveraging containers to run Spark jobs, using Kubernetes to scale or shrink machine learning solutions as needed, and implementing GPU-based solutions.

Data operations is a DevOps-based approach to data systems operations that evolves into ML Ops. A typical machine learning lifecycle requires models, training data, and test data, and then iterations are needed to find out which model works well and deploy it to production. Manual operations cannot meet the needs of business users, and ML Ops tries to automate this process and enable versioning and dynamic application of models.

Ethics, Energy and the Environment

The last classification Penchikala mentioned for data is ethics, where the industry is trying to balance responsibility and fairness when using AI, ML, and data engineering solutions. But data is only one aspect of software ethics, and the topic has only recently been taught as part of a computer science curriculum. "The systems we build can have a profound impact on people's lives, and we don't talk enough about that, we don't reason enough, it's really scary," Humble says. ”

While we see ethics in the curriculum of a computer degree, its degree cannot be compared to engineering. In engineering, certified professional engineers sign off to confirm that the design is safe. As Reisz points out, one of the challenges in teaching software ethics is that "we haven't ethically classified the software we write today." When software uses ML models to decide who gets resources, this is a more complex problem than trolleybus solutions.

The impact of software on the environment is also a growing concern, and it's only recently been discussed. Software consumes 10% of the world's electricity and has a greater impact on the environment than the aviation industry. There are some specific examples, such as cryptocurrency mining, but all ai and ML have a significant impact on people and the environment.

Humble argues that we tend to focus on what individual developers can do, but that's actually something that large cloud vendors need to focus on. If you move the code to the cloud and then require all the energy used by the cloud vendor to be green, it's going to be a great start and it's going to be more valuable than anything else. The idea of requirements shaping, which makes your code run in the most environmentally friendly way possible, is a topic in the Asim Hussain InfoQ podcast.

Reisz goes so far as to say that "sustainability and ethics are probably the biggest issues we face as software developers." He mentioned Astrid Atkinson, who started Camus Energy after leaving Google and speaking at QCon Plus. Reisz believes this is just one example of people using software skills to focus on energy and environmental issues.

Store data about people

Data about people is a subset of ethical issues about how it is stored, used, and provided. Over the past few years, gdpr, CCPA, and other similar bills have forced people to think about these issues. In the past, regulations have often focused on forcing companies to adopt a solution that may have become obsolete. The GDPR, by contrast, raises a problem that has not yet been clearly resolved, sparking more open discussion of a solution. Discussions continue, and there are many new questions that need to be answered. How do we keep individuals' privacy secure? Who should have this data? Where should the data be stored?

GitHub Copilot with coding aids

The introduction of GitHub Copilot and other AI-based coding aids has opened up the possibility of no longer needing developers in extreme cases. Reisz believes that we won't see true general AI anytime soon, but we'll see AI being able to augment human capabilities. He made an analogy with the cruise control of a car. What started out as just maintaining a set pace has now evolved to lane keeping assist and eventually to fully autonomous driving. Copilot helps with boilerplate code, and for decades developers have complained about writing such code. This allows developers to focus more on specific business needs.

According to Charles Humble, these tools only deal with the simple parts, but, "in software development, the difficult part is figuring out what to build, and none of these tools really help in this regard." "The same goes for low-code solutions, which allow business users to solve small problems quickly. But when you need to build scalable, growable software in a repeatable way, these solutions fail. In his keynote address at QCon San Francisco 2018, Grady Booch described software as art dedicated to constantly raising the level of abstraction. Reisz sees this as a higher level of abstraction.

cloud computing

Cloud computing is a $90 billion (USD) industry, and it's not going to get any smaller. Multi-cloud and hybrid cloud is an unavoidable reality, most commonly due to mergers and acquisitions. Because of this reality, we are now seeing technology solutions for hybrid cloud needs, such as GCP Anthos and AWS Outposts. Bryant believes that looking at the control plane will be interesting because companies handle governance, security, and other cross-cloud vendor interactions.

Adopting a cloud-native pattern is still critical because it better supports cloud-agnostic designs that can be deployed differently. One subset of this is cloud-native data practices, which architecturally provide an integration layer to handle common policies such as user authorization. This can help alleviate the problem of "data gravity," where everything in the system is built based on where the data is stored.

A number of tools to "pave the way" for cloud platforms are being created, two of which are the Open Policy Agent (Reisz discussed with the project's co-creators) and Crossplane (which allows you to build your own PaaS following the open application model).

Software Supply Chain, Blockchain and NFTs

Reflections on some of the high-profile cyberattacks of 2021 have sparked discussions about the software supply chain and how we should deal with dependencies. Almost all software has dependencies, whether libraries or cloud services, which raises concerns about how to understand the entire code stack they run. There are some tools that are being created, but at the moment, there is no way for us to say, "I can verify that this software is what I think it is". Sadly, this complexity can lead to more supply chain attacks, whether through injection at compile time of software, or through distribution updates.

Reisz looks forward to seeing some practical uses of blockchain and the Non-Forgeable Token (NFT) in 2022. Initially, we'll see them applied in games and metaspaces, where you'll be able to buy something truly unique or limited in number, and it will always belong to you. This will give birth to other types of software that can travel with you in different games. Blockchain can also be used for privacy use cases, such as an individual's private medical records.

While it hasn't been used that way yet, blockchain can also be used for software certification to verify the authenticity of something in the supply chain. Docker, VMware, and other companies are providing approved catalogs of components and containers that allow enterprises to validate what they're using.

Looking ahead to 2022

Bryant asked each participant what they were looking forward to seeing in the new year.

Penchikala points out that all trends, technologies and practices are interconnected. He looks forward to seeing data engineering, ARTIFICIAL intelligence, and ML solutions become more ethical and fair. Machine learning and ARTIFICIAL intelligence may not replace us, but in order for them to really make our jobs better, we need to take the time to make sure they are ethical and fair. This can be achieved by standardizing our way of verifying the ethical nature of these solutions, including by defining fairness for all parties involved.

Humble is very interested in the idea of improving the way we read and understand code. For the past 30 years, our industry has focused on simplifying coding. But as a programmer, you probably spend most of your time reading code, whether yours or someone else's, and trying to understand what it does. He has been working on the Ballerina language, which has built-in sequence diagrams that can represent concurrency models. Modifying the graph also changes the code, which gives us a better understanding of how the concurrency model works. He recently spoke to James Clark about this.

Hastie is looking for and visiting more humane workplaces. "I hope it's purely because organizations care about people, but in my view as a cynical person, they have to do it because they're losing good people." If you want to attract great people, you need to provide them with a good working environment."

Reisz observes a common theme of raising the level of abstraction of software. This enhances the software's ability to do more.

Betts would love to see a good example of an enterprise implementing data mesh, not just high-level architectural design. He thinks there's going to be some really interesting developments, and people will bring up some great stories about how they're dealing with data, how everything works, and why that's an improvement over the past. He also wanted to see if it was possible to prove his COVID inference about Conway's Law that hybrid and remote-first companies are better at writing distributed software.

Everyone has something to say about ethics and sustainability, and believes that this is the main driver of everything else in the coming years. Humble plans to seize every opportunity to raise it as an industry issue. Everyone hopes that we will see progress in this regard in the new year.

About the Author:

Daniel Bryant is Director of DevRel at Ambassador Labs, News Manager at InfoQ and Chairman of QCon London. Currently, he focuses on "DevOps" tools, cloud/container platforms, and microservices implementations. Daniel is the head of the Java Community (LJC) in London, where he has contributed to several open source projects, written for well-known technology sites such as InfoQ, O'Reilly and DZone, and regularly speaks at international conferences such as QCon, JavaOne and Devoxx.

Wesley Reisz is a platform architect at VMWare. He focuses on cloud-native practices and is interested in edge K8s (he was previously vice president of technology at an edge computing platform startup). Wes also serves as chairman of the QCon San Francisco Conference and co-hosts the InfoQ podcast. His articles and recordings reflect his interest in edge, architecture, cloud-native computing, and machine learning. Previously, Wes served as The Product Leader for all English QCon conferences worldwide, was the Lead Architect for HP Enterprise Systems, and was an Adjunct Professor at the University of Louisville for over 13 years.

Thomas Betts is the Lead Editor of InfoQ Architecture and Design, co-host of the InfoQ podcast, and Senior Chief Software Architect at Blackbaud. For more than two decades, he has focused on providing software solutions that delight customers. He has worked in a variety of industries, including retail, finance, healthcare, defense, and tourism. Thomas lives in Denver with his wife and son and enjoys hiking and exploring the beautiful state of Colorado.

Charles Humble is the editor-in-chief of Container Solutions, a cloud-native consulting firm. As a former software engineer, architect and CTO, he has worked as an executive in technology companies and, more recently, in content groups (from small start-ups to medium-sized companies, working with offshore, onshore and remote teams). From March 2014 to April 2020, he served as Editor-in-Chief of InfoQ.com. Prior to that, he was CTO of PRPi Consulting, a compensation research firm acquired by PwC in July 2012. In his spare time, he composed music as one of two members of the British environmental technology band Twofish, who released their second album in 2021.

Shane Hastie is the head of InfoQ.com's Culture & Methods Editorial Team and hosts the Weekly InfoQ Culture Podcast. For more than 30 years, Shane has been a practitioner and leader in the roles of developer, tester, trainer, project manager, and business analyst, helping teams achieve results that align with overall business goals. He has worked with organizations large and small, from individual teams to large organizations around the world. He has more than 30 years of hands-on experience at all levels of information technology and software-intensive product development. From 2011 to 2016, Shane was a Director of the Agile Alliance and a founding Chairman of the Agile Alliance of New Zealand. Shane is also an ICF-registered professional coach. "I firmly believe that organizations around the world today desperately need a human way of working and an agile way of thinking. It's important to push agile values and principles beyond the confines of software and properly incorporate them into the culture of the business – the industry we're in touches every aspect of people's lives and has a huge impact on society as a whole, and I want to be part of making sure it's both ethical and sustainable. ”

Srini Penchikala currently works as a Senior Software Architect in Austin, Texas. He is also the Lead Editor of the InfoQAI/ML/Data Engineering Community. Srini has over 22 years of experience in software architecture, design and development. He is the author of Apache Spark Big Data Processing and co-author of Spring Roo in Action published by Manning Publications. Srini has spoken at conferences such as Big Data Conference, Enterprise Data World, JavaOne, SEI Architecture Technology Conference (SATURN), IT Architect Conference (ITARC), No Fluff Just Stuff, NoSQL Now, and Project World Conference. He has also published numerous articles on software architecture, security and risk management, and NoSQL databases on InfoQ, The ServerSide, OReilly Network (ONJava), DevX Java, java.net, and JavaWorld.

https://www.infoq.com/articles/summary-podcast-2021-review/