background
The quality assurance team continues to promote the construction of Zhihu's quality system, and initiates and participates in a series of work, including but not limited to:
- The production and research process is standardized
- Special tests
- Continuous integration of assembly line construction
- The Little Black Room and the crowd testing mechanism
- Closed beta and grayscale
- Online monitoring of quality issues
- Client Quality Contest
Each of these efforts generates a large amount of quality data, which can not only be used to measure the effectiveness of the QA team's work, but also to further enhance the quality awareness of other teams through the release of quality data, and better build a company-wide quality culture.
In the early days, we used scripts or doxing to get the information we needed from various internal systems, and displayed this quality data in the form of graphs and graphs on Excel. While this approach is occasionally error-prone and time-consuming, the cost is acceptable. With the rapid development of the company's business, the number of QA teams has also ushered in continuous rapid growth, and it has been divided into different groups according to business segments. At the same time, we are also facing the challenge of shortening the iteration cycle of the Zhihu client from two weeks to one week and the componentization of the client, and the collection, statistics, and display of quality data require more and more time and energy, and it is almost impossible to manually sort out complete, reliable, and visual quality data. We urgently needed a quality platform that could automatically collect, process, and display all types of data, and combined with the actual work of the entire QA team, we decided to make the highest priority to automate the collection and display of client requirements and bug cases for each release.
General design
As mentioned earlier, the initial objectives of the quality platform include the following two points:
- Test reports for each client version and each business unit are automatically generated as much as possible
- The trend of quality data changes is highly visualized
According to the initial objectives, the quality platform should include the following three modules:
- Data Collection: Automatically collect underlying data from various systems
- Data processing: The basic data is associated and combined into a test report, and some information that cannot be automatically obtained can be manually entered
- Data display: displays the trend of each quality data in the form of charts and graphs with the version
Data Collection
The main content of the test report is the requirements completed and the bug statistics found by the specified business department on the client version, so the basic data we obtain are divided into the following categories:
Version information
Version is the basic dimension of the test report, for the Zhihu client, the entire life cycle of a version is divided into 4 stages: development, testing, grayscale, and launch, and the test report needs to record the code submission and bug situation of each stage, so the boundary of each stage needs to be clarified.
At present, there are three important operations in the release process of the Zhihu client:
- When the time point of integration testing is reached, we will create a new Release branch based on the develop branch, the creation of this new branch is a sign that the version has entered the testing stage from the development stage, and the bugs found in the test will be fixed on the new branch
- Grayscale: After complete regression testing and bugfixes in the testing phase, we will release a grayscale version for our invited beta users to try
- Arraignment: After the quality of the version is qualified after multiple grayscale verifications, it will be submitted to the App Market for review, and after the arraignment operation is completed, you will receive an email from the App Market or the colleague responsible for the application marketing
In summary, the boundaries of each stage can be summarized in the following table, and the quality platform will judge the stage of a certain version based on these conditions
Business unit information
The business department is also the basic dimension of the test report, but there is no direct method to associate the requirements and bug information with the business department, and the quality platform finally finds the corresponding business department indirectly through the information of the person in charge of the requirements and bugs.
For each requirement or bug, there is a single clear development leader, after getting this person, the quality platform will request Zhihu's internal address book, so as to obtain the information of the business department to which the person in charge belongs, and the business department is recorded as the business department to which the requirement or bug belongs.
Bug information
When QA creates or updates a bug, JIRA will send all the information of the bug to the quality platform through our pre-configured webhook, and the quality platform will extract the required information from it and store it in the database.
The QA team has clear specifications for filling in bugs on JIRA, and it is necessary to accurately fill in the client version, release stage, business department and other information to which the bug belongs, so that the quality platform can automatically match the bug with the test report according to these specifications.
Request information
Zhihu uses JIRA as a requirements management tool internally, but due to the large differences in the requirements management methods of each business unit, the quality platform cannot automatically collect and match the requirements on JIRA like bug information.
So we instead collect the code commit information for each version, and since each code commit from the client already asks to be associated with the corresponding JIRA issue (including requirements and bugs), we can indirectly get the requirements information for each version.
Code commit information
Zhihu internally uses Gitlab as a code management tool, and since the Zhihu client is in the process of component refactoring, there are currently two ways to commit code changes: submit MR to the main repository and upgrade the component version number through the component management platform.
When the engineer submits code through MR, our pre-configured webhook will send a message to the quality platform when MR merges, which contains information such as the amount of MR code change, the person in charge, the merge time, the target branch, etc., through the merge time + target branch, you can locate which stage of which version this MR belongs to, and which business department this MR belongs to through MR author information. Since Gitlab supports integration with JIRA, Zhihu engineers will fill in the ID of the issue on JIRA in the MR title, and we can use this ID to associate MR with requirements or bugs on JIRA.
When an engineer submits a new component version through the component management platform, he or she will be asked to fill in the following information, and when the test is passed and the test component is upgraded, the component management platform will send this information to the quality platform with the upgrade time. The quality platform can locate which version of the component upgrade belongs to which stage through the upgrade time + main engineering branch, and which business department the component upgrade belongs to by testing the operator information.
data processing
With the above basic data, we can easily combine them into a test report for a specific version and a specific business unit, and the report content of a certain version stage in the report is shown in the following figure, where Diff refers to the component upgrade:
At the same time, the data of each stage of the same version can be accumulated to obtain the following overall data of the version:
In addition to this automatically collected data, the test report also supports manual bug filling. Click the "Add" button in the "Bug List" to add any issue of JIRA at the specified release stage by filling in jql.
Data display
In order to quickly realize the requirements and reduce maintenance costs, we decided to use an open-source BI system to display the data, and conducted research on several excellent products on the market:
Based on the above advantages and disadvantages, we finally chose MetaBase as the platform data visualization system.
It is worth mentioning that for the flexibility of configuration, we use the "native query" function provided by MetaBase (that is, to get the data in the report through sql). At the same time, when we design the database, we deliberately add some redundant information to each table, which can reduce the complexity of SQL query statements and reduce the join operation, thereby reducing the query time.
Based on the basic data collected, we can make quality data trend charts for each dimension on MetaBase
End
This article introduces the design ideas and implementation solutions of the Zhihu quality platform to solve the problem of collecting and displaying quality data of client versions. Since the Zhihu quality platform was put into use in September 2018, it has recorded the quality data of more than 50 client versions and generated more than 100 client version test reports.
In addition, after continuous iteration, the quality platform currently supports various types of reports such as weekly product quality reports, demand dimension test reports, online fault reports, etc., and further collects data such as client startup time, network request response time, and packet volume to help other technical teams formulate work plans more accurately. We'll cover these features in more detail in a future article. With the support of the quality platform, the work efficiency of the QA team has been significantly improved, and the data provided by the quality platform has also provided strong support for the construction of quality culture in various business divisions and the entire Zhihu.
Author: Zhongli Sugar-free
Source: https://zhuanlan.zhihu.com/p/68059263