background
Designing test cases in conjunction with code can effectively improve test accuracy, so we have developed Timon, a tool that can collect code coverage in real time. Timon is connected to the company's containerized build system, ZAE; Support coverage statistics for Java, Python and Golang; Ability to generate full and incremental code coverage reports; Support for merging coverage data; It can be used in scenarios such as interface testing and integration testing. Timon tools support coverage statistics for more than 90% of automation interface use cases; With QA tested with Timon accessibility, incremental code coverage averages over 80%. In the following content, the article will introduce the principles and practices of the tool.
Test Coverage Tool Selection
Our test coverage tools of choice include the Jacoco, Coverage.py, and go test commands. The detailed introduction of each tool can be viewed on the official website, and the article will not be repeated. Table 2.1 compares the three tools. Among them, Jacoco's use of on-the-fly mode instrumentation has met the needs of use. Coverage.py and go test need to kill the application process in order to get the report. go test needs to be instrumented during the compilation phase.
Table 2.1 Comparison of tools
Principles and module design
During the compilation phase, we instrumentate the code. During the deployment phase, the Timon client is deployed into a container. When you open an app, ZAE will automatically modify the app launch command to start the Timon client. The Timon tool consists of two large modules: the client and the server. The client has three sub-modules, each for each of the three languages. The main responsibilities are to upload container information, listen to ports, request configuration files, generate coverage raw data files, upload code diff information, etc. The main functions of the server include incremental code coverage calculation, configuration file generation, historical data management, and report display interface. A detailed list of modules can be found in the System Module Diagram (Figure 3.1). Here's how to implement the key features of the tool.
Figure 3.1 System Modules
1. Plugging in the compilation stage
Golang applications need to be instrumented during the compilation stage to collect coverage data. After the development adds the following configuration to the CI configuration file, the ZAE platform automatically completes the instrumentation process. The configuration file contains information such as the project source code, the object file, and the path to each service entry. After detecting the instrumentation configuration, ZAE first puts the test file provided by the Timon client into the directory where the service entry is located, and then inserts the compilation command into the generated Jenkinsfile. Projects in Python and Java languages do not need to be instrumented during the compilation phase.
2. Initialization of the deployment phase
When the Timon client is started, it uploads container information, listens on ports, and requests configuration files. Figure 3.2 illustrates the initialization process for the client.
Figure 3.2 Timon Client Initialization Process
Once the Timon client is launched, the main tasks are to listen for coverage report requests, destroy container information, and upload coverage reports. Because after the code is updated, the test joint debugging environment will automatically deploy the new version of the code and destroy the current container, so the Timon client needs to upload the coverage data before the container is destroyed. The Java Language Coverage Detection Tool uses a scheme where coverage data is uploaded every 10 minutes. However, the Go and Python coverage detection tools need to restart the application before it can be uploaded, so in order to prevent the service process from restarting frequently, we use another solution: the Timon client listens for the destruction signal sent by the system to the container, and the client uploads the coverage data after receiving the signal. In addition, when a user requests a coverage report, the Timon client will also upload the coverage report in real time.
3. Upload coverage data
The timeline diagram of a user from the time they request the current deployment version coverage report to the time they receive it is shown in Figure 3.3. Suppose there is only one application in a test environment, and the application has three containers to provide services, the Timon server waits for containers 1 and 2 to upload the original data, sends the original data collection to container 3, and finally container 3 merges the data and generates a full code coverage report in html format. If there are multiple applications in the test environment, the processes in the sequence diagram are parallel. The scheme for preserving coverage raw data is similar until the container is destroyed.
Figure 3.3 Client-server interaction sequence diagram
4. Calculate coverage data
Methods of statistical coverage include branch coverage and executable code coverage. In practice, coverage is often calculated from executable code, and the same is true for Timon. Incremental code coverage calculations require diff data from the code and the main branch and full code coverage reports. The numerator is the number of lines of executing code that have been overridden by incremental code, and non-business code files such as test files are removed. The denominator is the number of lines of code that the delta code partially executes. In order to help users find the incremental code in a timely manner, we have made some changes to the reports generated by third-party tools. For example, Java's incremental code coverage report only shows files with incremental code, and you can jump to the incremental code location by searching for "new code" directly in the browser. Coverage reports for Golang and Python languages have similar functionality.
Figure 3.4 Incremental Code Coverage Report for Java Edition
If you want to calculate the incremental code coverage of MR as a whole, you also need to combine the coverage data of multiple Commits. Suppose there are three versions of Commit A, Commit B, and Commit C coverage reports, Commit C is the last commit, and Commit C is the file with incremental code is c. Here's a look at the merge process.
- Locate the content of the code being executed in Commit A file c
- Look for the same code content in Commit C
- Indicate that the line was executed in the Commit C coverage report
- 重复以上三个过程,其他 Commit 代码与 Commit C 的代码比对
The resulting coverage report can be used to aid QA analysis of areas that are not covered due to missing test cases, uncovered test environments, or other issues such as redundant code.
User usage process
We have achieved zero-cost access, and users can turn on the "Start Coverage Detection Environment" button when they join the application in the test joint debugging environment. Users can view the report in the front-end interface of the tool, and they can also add comments in MR to get a report on the coverage data. The User Usage Flowchart (Figure 2.1) illustrates the process of analyzing test cases using the Coverage Report. If you're using automated scripts for testing, a single-version code coverage report can help with the analysis. In the case of manual testing, MR's overall incremental code coverage report can help find missing test cases because the code is constantly updated.
Figure 4.1 User Usage Flow Diagram
Figure 4.2 Front-End Request Coverage Report
Figure 4.3 MR Comment Request Coverage Report
Business Practices
Before the advent of the Timon tool, black-box testing could not guarantee coverage of all logic and anomalies; Follow up the development progress of automated test scripts, typically by comparing the percentage of use case completion; The quality evaluation of automated test cases is also mostly based on testing experience. Timon helps testers quickly find missing test scenarios, increases productivity, and pushes testers to focus more on the code itself. Timon also provides a new way to evaluate automated test cases. In addition, we have found in practice that the coverage tool can also help testers find other problems, such as developing and submitting code that is not related to requirements, and manually testing the environment to create certain exceptions. The Timon tool is a module of Zhihu's quality assurance system, which adds bricks and tiles to accurate testing. QA changed testing habits because of the use of Timon tools. We will continue to look for best practices for Timon coverage tools in the production and research system, and at the same time link with other quality assurance platforms to maximize its value.
Author: Xiao Guoguo
Source: https://zhuanlan.zhihu.com/p/102744098