What kind of products are needed in the era of distributed databases?

1. Distributed databases, it's time

Giving full play to hardware computing power downward and supporting upper-layer applications upward, databases are indispensable as the foundation of the IT industry.

The database is the crown jewel of basic software, which has been on a par with operating systems for decades, and is the core of every company's business system, "harvesting" corporate profits in order to "harvest" corporate profits.

Over the past few decades, centralized databases have become mainstream.

If the processing power and capacity of the database are analogous to a "horse-drawn cart", the solution of the centralized database is to make the "horse" bigger, make the stand-alone machine more powerful, and drive the storage and processing of data.

The speed, ease of use, stability, scalability, cost, etc., of the database are critical to the basic business and growth resilience of the enterprise.

The speed at which data is generated and how it is used has changed dramatically. Data is growing at a massive scale and at a high speed, and the data deployment environment is becoming increasingly complex.

The amount of data is growing explosively, the application load is increasing rapidly, and the capacity demand is increasing dramatically. But in the face of the new era, centralized databases are no longer sufficient.

First, it is difficult for the growth of "horses" to keep up with the speed of "cars", and second, centralized databases have capacity limitations, which can generally only reach the terabyte level, which cannot adapt to the rapid growth of data storage and processing needs in the era of big data.

On the other hand, the overall performance and stability of traditional centralized databases rely too much on the performance and stability of the underlying hardware itself, which is why traditional core databases often use small models and high-end storage devices, which greatly increases the construction cost of customers and the later operation and maintenance costs.

Since it is difficult to support alone, why not "everyone gathers firewood"?

Distributed database is to use many "ponies" to pull this "big car" together.

Compared to traditional centralized databases, distributed databases are more scalable. In the face of rapidly growing demand, the latter does not rely on a single machine, and can achieve simple and cheap expansion.

Distributed databases are also more resilient. The distributed database "does not put all the eggs in one basket", and when a single machine in the system has a problem, the database can still run stably. "Two cities and three centers, two data centers in the same city" provide ultra-high system availability.

Distributed relational databases use distributed technology to break through the capacity and performance bottlenecks of traditional centralized databases, simplify O&M, and help reduce reliance on high-performance hardware.

In fact, the era of distributed databases has become unstoppable.

In the past few decades, there were few Chinese manufacturers on the database table, but the emergence of big data and cloud computing has given Chinese manufacturers more cards.

In terms of business, the digital transformation of enterprises continues to deepen, and the business has higher requirements for high availability, concurrent processing capabilities, and elastic expansion of databases. Distributed databases can be scaled up and down based on demand and can handle virtually unlimited amounts of data.

In the cloud computing scenario, the distributed database manages the database resource pool, and has the characteristics of multi-mode, multi-tenancy, HTAP, elastic expansion, and high availability, which is a proper match for cloud computing and distributed application development models.

The integration of cloud and database provides rapid deployment, high scalability, high availability, portability, easy O&M, and resource isolation, reducing the repeated configuration of database parameters.

The CTO of a well-known database manufacturer once told Leifeng.com: "Compared with the traditional stand-alone version of the classic database, due to its longer accumulation time, the maturity of the independent innovation database will definitely be worse." However, in the field of cloud databases, there is no technical gap between the mainland and similar foreign products, and even has advantages in some high-concurrency scenarios. ”

In terms of policy, various policy documents have been issued at the national and industry levels to drive the development and implementation of distributed database products, and the pace of independent innovation has been moving rapidly.

Distributed database technology is also maturing. With the gradual increase in scenario applications, the iteration of database vendors and products has accelerated, and the technology maturity has made great progress compared with the previous decade.

Hundreds of database vendors in China, traditional database vendors, cloud vendors, emerging database vendors, big data vendors, database service providers, system integrators and other types of suppliers have launched products one after another, and the market is extremely hot, including more than 100 distributed database products.

The development of cloud databases has enabled independent innovation databases to achieve "lane change and overtaking".

Second, the perfect combination of database + server

Database is one of the three carriages of basic software, which is related to the core business system of the enterprise, but this field has heavy attributes and high barriers:

It requires all kinds of database professional and technical personnel, and it needs to be verified in strict and complex business scenarios with high concurrency and large data volume in order to temper the stability and performance of the product, but not every company has the verification opportunities given by the enterprise.

It takes time to run and precipitate in order to temper the maturity of the product, and not every company has enough time and money to invest in a product that will only be effective in the long run.

In other words, the threshold for database product research and development is extremely high, and it takes a lot of time, money, and energy, not based on "open source middleware + open source database" to develop a qualified market product, and manufacturers must guard against arrogance and rashness, and settle down for long-term investment.

Sharpening a sharp blade is not a day's work. There is no standard answer to how long the "long-term investment" should be, and Tencent Cloud Database has spent 17 years.

Tencent Cloud's enterprise-level distributed database TDSQL has been incubated internally since 2007, and in 2009, when the Internet entered a period of rapid social development, Tencent Cloud Database supports billing and payment services, and has carried out research and development layout on high-performance throughput, distributed horizontal expansion, and distributed KV storage on the basis of 7*24 hours of high availability and strong data consistency, helping to get rid of the dependence of business system traffic on the number of servers, and break through performance bottlenecks, data reliability assurance, and high availability. Impossible triangle".

In the era of cloud computing, Tencent Cloud's database product capabilities have been polished in a large number of actual user scenarios in the public cloud.

WeBank is the first bank in China to adopt the Internet distributed technology architecture, and it is also the first bank to not use centralized commercial databases such as Oracle for its core system. Behind it, TDSQL plays a very central role as the foundation of its distributed database.

In December 2020, Tencent Cloud's database brand was upgraded, and Tencent Cloud's original three product lines of TDSQL, TBase, and CynosDB will be upgraded to "Tencent Cloud Enterprise Distributed Database TDSQL". The newly upgraded Tencent Cloud TDSQL covers a complete database product system that integrates multiple engines, such as distributed, analytical, and cloud-native, to meet the application requirements of different scenarios.

In the application of finance, insurance, carriers and other industries, TDSQL's Oracle compatible engine was born by removing "O", with a compatibility of more than 95%, ultra-high compression ratio, greatly saving resources, helping the industry to complete database replacement and upgrading at low cost, quickly and smoothly, and accelerating the implementation of practice in traditional fields.

TDSQL has also helped the seventh national population census with a data scale of 1 billion to achieve full electronicization. Today, TDSQL services span multiple verticals such as finance, public services, and telecommunications, with more than 4,000 customers.

Inspur Information, on the other hand, is the boss in the server field.

As the world's leading server and data center solution provider, Inspur Information has strong technical strength and rich industry experience, and can provide high-performance and high-reliability hardware support for Tencent Cloud.

Inspur Information and Tencent Cloud are not just complementary technologies.

In terms of market sharing, Inspur Information has a wide range of customer bases and market channels in domestic and foreign markets, which can help Tencent Cloud expand its market coverage and increase its market share. Tencent Cloud has a wide range of application scenarios and customer resources in multiple industries such as the Internet, finance, e-commerce, and games, which can provide more application opportunities and market demand for Inspur's hardware products.

In terms of customer needs, Inspur Information has an in-depth understanding of the needs of enterprise-level customers, and can provide Tencent Cloud with customized hardware solutions that meet market demand. Tencent Cloud is able to provide customers with more complete and optimized cloud computing solutions.

Wei Jian said that in the future, high cost performance, high performance, and high reliability are the three important criteria for database solutions, and TDSQL has excellent performance.

At the same time, Inspur Information is also the largest server partner of Tencent Cloud, so it also has a natural advantage in deepening cooperation.

Based on the historical experience of friendly cooperation, the two parties further combined their respective advantageous products to achieve deep integration and optimization of software and hardware, and launched today's distributed database all-in-one products.

3. The advantages of the metabrain TDSQL database all-in-one machine

Migrating the core business to the independent and innovative distributed database is a major project, not all migrations are simple database replacements, but more about the replacement of the entire technology stack, and involve the top-down information transfer of the entire business chain.

The five core elements of distributed database selection are reliability and stability, overall cost, product features and ease of use, compatibility, and scalability.

Reliability, stability, and availability. The database is the core of the entire application system, and once it is paralyzed, it means that the application and business are interrupted.

Wei Jian told Leifeng.com that high availability is a system concept, and the high availability design of a single module/layer is not enough to provide the high availability of the system required by users.

In the financial field, its high security level, high transaction frequency, complex data environment, strict technical standards, high concurrency, and high reliability requirements are the most prominent.

Therefore, in the design of the Metabrain TDSQL all-in-one machine, they have realized the full-dimensional high availability, reliability and stability design from the architecture, software and hardware configuration, and finally to the solution, so that any node failure will not affect the user's experience and realize the failure is not perceived, such as:

In terms of architecture, Inspur Information has built a fully redundant distributed software and hardware cluster on the overall architecture to avoid a single point of failure, and any node failure will not affect the stability and availability of the business.

In terms of software and hardware mechanisms/configurations, there are mechanisms such as master, multiple slaves, sharding, replicas, and strong consistency in software. In terms of hardware, it is conducive to automatic memory fault isolation and "pre-diagnosis and healing", hardware redundant array, multi-NIC binding and other mechanisms to ensure the reliability and availability of each module.

Wei Jian said that the distributed database appliance has higher hardware requirements than the traditional centralized database, and they follow a stricter quality process and quality standards than ordinary servers from internal R&D quality, production testing, and delivery. The usual server aging time of 24 hours is sufficient, and they carried out an aging time of 48 hours for the Metabrain TDSQL all-in-one machine based on the key business scenarios of the database. Double the aging test, just to polish the better landing ability.

Public utilities and enterprise customers may have requirements for integrated intelligent O&M management and unified management of multi-system databases.

For O&M, the use of a distributed architecture will make it more difficult to operate and backup operations for the entire data cluster, and O&M personnel will also worry that after migrating to an independently innovated distributed database, the O&M skills and methods will be too different from the original ones, and the database cannot be effectively O&M.

In order to make it efficient and convenient for customers to use, Tencent Cloud and Inspur Information optimize and prefabricate software and hardware to avoid the complexity and uncontrollable risks of traditional delivery in terms of delivery management and operation experience. The unified O&M management interface of software and hardware, as well as the tools of full-white screen visualization and intelligent analysis are used to reduce the O&M difficulty of users.

"The metabrain TDSQL all-in-one machine is ready to use out of the box, leaving the complexity to ourselves, and we do a variety of joint debugging and joint testing, so that users can focus more on the application." Wei Jian said that there is a very big gap between the optimization of the database all-in-one machine, and they have made it possible for customers to use it out of the box.

The technical barriers to a real all-in-one machine are not low, it is not assembled, but a combination of software and hardware.

Software and hardware joint tuning, such as what BMC tunes, how to tune in different scenarios, software manufacturers are not familiar with, as well as the combination of parameters at the OS level and database level, is a huge workload, without years of accumulation, it is difficult to achieve the ultimate, Inspur Information and Tencent Cloud give full play to their respective advantages, and jointly do software and hardware joint tuning, so that the Metabrain TDSQL all-in-one machine can reach the optimal level.

Secondly, the software and hardware management system of the Metabrain TDSQL database all-in-one machine monitors the running status of the system in real time, and will trigger an automatic alarm once the abnormal situation of the database and hardware equipment is detected to reduce the risk of failure.

In addition, the Metabrain TDSQL all-in-one machine provides a complete backup and disaster recovery mechanism to avoid the impact of human error and natural disasters on business availability from the solution level.

Scalability is considered one of the most attractive features of distributed databases. In the past, distributed products were more used in the pan-Internet field, so they were highly scalable.

In the future, in terms of application scenarios, based on the strong R&D strength of both parties and combined with the needs of the industry, they will continue to optimize and innovate to provide users with better all-in-one exclusive products that better meet their needs. Leifeng Net, Leifeng Net, Leifeng Net