Capacity building and practice of Mini Program testability

This article is compiled from the 77th issue of Meituan Technology Salon, "Quality Risk Prevention and Control and Stability Governance Practice of Meituan's 100-million-level flow system". As a terminal product ecosystem, Mini Programs occupy a very important position in the industry. This article analyzes the testing difficulties of Mini Programs based on the quality assurance requirements of Mini Programs, introduces the basic concepts of Mini Program testability, and introduces the generalized solution of Meituan's in-store R&D platform for improving the testability of Mini Programs. Finally, this article aims to provide readers with some valuable insights and enlightenment in the field of Mini Program quality assurance.

• 1. Introduction
• 2. Introduction to the testability of Mini Programs
• 2.1 Usage and Effects
• 2.2 Access mode
• 2.3 Implementation Principles
• 3. Meituan's ticket business Mini Program testing practice
• 3.1 Measurability landing
• 3.2 Summary of business practices
• 4. Summary and outlook
• Q&A

1. Introduction

A test activity can essentially be thought of as the process by which the system under test generates responses to a stimulus and comprehensively tests those responses. This process (excitation->-response-> check) involves two roles: the tester and the test subject, the tester performs the excitation and check response, which is done by the machine (program) or human; The test subject receives the stimulus and produces a response. From the point of view of this process: the excitation is controllable, the response is considerable, and it is called measurable. For example, modifying caches, network request MCOKs, page redirects, and user login status settings are all testability.

In end products that have not undergone any testability improvements, testers can only complete the most basic quality assurance through UI interaction and observation from the UI interface. However, there are all kinds of complex logic and states inside the application, and more in-depth testing requires intervention and observation of this information. For example, when performing a dotting test, you need to confirm whether the dotting information is reported properly after operating the page, which usually relies on the network proxy debugging tool to complete the verification. Similarly, in the user login test session, after the login is completed, it is necessary to check whether the login information has been correctly recorded in the cache, which requires the ability to view the cache, which reflects the requirements for testability in actual business test scenarios.

Overall, there are many complex aspects involved in constructing a target scene for testing, and it is difficult to see if it meets expectations, as shown in the figure below. Endpoint testing is a long-term challenge. To address these challenges, we build on enhanced testability throughout the testing campaign, enabling testing to examine systems at a more granular level, increasing test depth and efficiency.

As a form of terminal product, Mini Programs are "light applications" that run on host applications (such as WeChat, Kuaishou, Baidu, etc.), which have developed rapidly since their launch by WeChat in 2017. Since Mini Programs are very dependent on the host application environment, in addition to the inherent difficulties of terminal testing, there are some special influencing factors in the testing process.

From the perspective of the operation mechanism, the code logic of the Mini Program runs in the container environment provided by the host application, and it cannot directly control the host application itself and the mobile phone system, which increases the difficulty of testing and testability improvement to a certain extent.

In current practice, there are mainly the following tools and strategies for testing Mini Programs:

1. Use network proxy tools such as Charles, Fiddler, etc. to analyze and verify HTTP/HTTPS requests and responses. While these tools are suitable for packet capture and analysis, they usually can't penetrate into the internal architecture of Mini Programs, so they can't fully control or perceive the internal state of the application.
2. Automated testing tools such as Airtest, which use image processing technology, mainly focus on interface-level operations and fail to touch the logic behind the application, so they still fall under the category of "black box testing".
3. Use the Minium Mini Program testing tool provided by WeChat to perform more refined testing operations, and can carry out internal controls such as API mocking. However, the method is complex to operate and relies on WeChat developer tools, which are different from the real device environment, which may affect the accuracy of the test results.
4. Dedicated self-developed debugging panels are developed to validate program logic and test specific scenarios, but these tools are often designed to focus on specific applets and are not easy to migrate to other applications, and they often do not support automated testing processes.

In summary, although there are a variety of testing tools and methods, there is currently a lack of a comprehensive and easy-to-use testing toolset that can comprehensively improve the testability of Mini Programs.

2. Introduction to the testability of Mini Programs

A panoramic view of terminal measurability capabilities

The goal of Mini Program Metrology is to build a comprehensive set of general Mini Program testability capabilities. The system seamlessly supports both physical and simulator environments, is compatible with multiple devices and platforms, and allows different applications to be easily accessed at a low cost. It can go deep into the core and provide comprehensive and diversified observability and controllability for Mini Programs, covering key areas such as application interface, internal state, and storage. This system is designed to enable testers to deal with complex test scenarios more conveniently, significantly improving the efficiency and depth of testing.

After a long period of construction and accumulation, we have built a relatively comprehensive set of terminal testability capabilities, including Android, iOS, Mini Programs, Web and other technology stacks. Among them, due to the particularity of the structure of the system, the testability of the Mini Program will be somewhat different from that of other terminals. The testability of Mini Programs mainly includes three levels: business logic testability, application testability, and system & device testability, and each level contains multiple vertical subdivision directions, in addition to supporting the public testability capabilities of multiple technology stacks, it also provides special capabilities such as AppData, host application information can be considerably controlled, etc. The following is a few typical capabilities to illustrate the measurability, use, and effect of Mini Programs.

| 2.1 Usage and Effects

In the actual manual and automated testing work, the testability of the Mini Program can be easily used and plays an important role in multiple scenarios.

2.1.1 Manual Testing

The following will take cache management and page jump functions as examples to introduce the use and effect of Mini Programs in manual testing.

In the actual test work, it will be combined with the testability of Lyrebird, a terminal test workbench developed by Meituan's in-store R&D platform, including terminal status data management, network request proxy and mock, defect logging, custom plug-in extensions and other capabilities. It also provides a graphical user interface that serves as an entry point for using testability capabilities in manual and automated testing.

After the Mini Program is connected to the Testability SDK, it can be connected to Lyrebird through the code scanning function provided by the Measurability SDK, and then you can use the measurability to control and observe the Mini Program on the PC side through Lyrebird.

Cache management

We can verify the correctness of the business logic that relies on the cache through the cache management function, such as the form information\user information staging to the cache function, etc.

• As shown in the figure below, the cache edit box at place 1 displays all the cache information of the Mini Program on the currently selected device, and manages these caches, supporting batch addition, deletion and modification.
• The cache change event information of the target Mini Program is displayed in 2 places, including the editing of the cache on this page and the addition, deletion and modification of the cache within the Mini Program itself, which will be updated in real time with the triggering of the event.

Page jump

Page jump is a heavily used ability in the business test of Mini Programs, which can be used to jump to intermediate pages such as form pages, product detail pages, etc., and no longer need to enter the target test page step by step from the home page, reducing the pre-test preparation.

2.1.2 Automated Testing

Combine testability capabilities with Lyrebird for automated testing. For example, the page jump capability can be used to directly reach the test scene, and then the scene state is restored by using the page state data recorded through testability, and the page is rendered to obtain the data/layout display on the page, and finally the actual operation map is compared with the pre-set page benchmark map, and the rendering difference results are provided for visual DIFF testing.

This kind of "visual test" is based on the page, through deep link jump technology and a series of terminal application itself testability transformation, directly to the test scene, and through image processing technology such as long image fusion, image incremental comparison and text recognition ability for visual DIFF testing.

Testability is built with the ability to observe and control the internal state of the application, and for any test method, as long as it involves the internal state of the application, testability can play an important role. For example, in the robustness test, the destructive abnormal scene is constructed through testability, or the operation of the Mini Program under all possible usage scenarios is tested by simulating different entry methods (such as QR code, video account, search, etc.) in the functional test.

| 2.2 Access mode

The Mini Program Testability Capability SDK is encapsulated as an NPM package, which can be introduced into the source code or compiled product project of the Mini Program to achieve access to the testability without additional adaptation.

Cross-platform operation

In addition to the support for WeChat Mini Programs, the Mini Program Testability Capability SDK extends capabilities to multi-homed applications by integrating an adapter, including support for platforms such as Meituan, Alipay, Kuaishou, and Baidu. The basic library APIs of these platforms are similar to WeChat, and the adapters will adjust the code accordingly according to the characteristics of different platforms, including the basic library APIs, front-end syntax or file types, etc., to ensure compatibility and consistency on various platforms and achieve cross-platform operation.

| 2.3 Implementation Principles

The core idea of the measurability of the Mini Program is to transparently intervene in the target modules such as the JS basic library of the WeChat Mini Program and the public basic components of the business in the JavaScript Runtime of the Mini Program through JavaScript Hook, so as to achieve considerable controllability within the Mini Program. After that, a network link is established between the central control and the outside through the testability SDK, so as to realize the considerable control of the internal state and function of the Mini Program at the remote end.

JavaScript Hook介绍

JavaScript Hook基于JavaScript的动态特性,有以下方法:

Function Hook: Directly override or modify the original function:

let _originAlert = alert;  // 保存原函数
alert = function () {
  console.log('alert执行开始');
  _originAlert.apply(this, arguments); //执行原函数
  console.log('alert执行结束');
}           

Object Property Hook: Use Object defineProperty to define new or directly modify the properties of an object, such as modifying the Getter/Setter method, to control the process of obtaining/setting an object.

Object.defineProperty(document, 'cookie', {
  set: function(val) {  // 控制cookie的设置流程
    console.log('获得cookie: ', val);
    currentCookie = val;
    return val;
  },
  get: function() {  // 控制cookie的获取流程
    return null;
  }
});           

Prototype Chain Hook: Modify the data on the prototype chain, such as String and Date.

let _originalGetTime = Date.prototype.getTime;  // 保存原型链原方法
Date.prototype.getTime = function() {
  console.log('getTime has been called'); 
  return originalGetTime.apply(this, arguments); //执行原方法
};           

Proxy Object: Creates a proxy mode to replace the original object, and can redefine basic object operations such as getting, setting, and defining properties.

// 创建Proxy有两个参数：
// target：要代理的原始对象
// handler：定义哪些操作将被拦截以及如何重新定义被拦截操作的对象
let handler = {
  get: function(target, prop) {
    console.log(`获取 ${prop}`);
    return target[prop];
  },
  set: function(target, prop, val) {
    console.log(`设置 ${prop} 值为 ${val}`);
    target[prop] = val;
    return true;
  }
};

let proxy = new Proxy(window, handler);

proxy.test = 'test';     // 输出: Setting test to test
console.log(proxy.test); // 输出: Getting test
                         //      test   
                                    

Static Hook: When a Mini Program is built, the JavaScript source code of a Mini Program is directly modified in a specific file.

The other methods will not be expanded into here.

The testability SDK can be broadly divided into four layers:

• Communication Layer: Communicates with the outside and is responsible for the bidirectional flow of instructions and data to the remote end (e.g., Lyrebird).
• Instruction distribution layer: parses the parameter instructions received by the communication layer, and calls the functional layer modules that control the relevant state of the Mini Program in turn.
• Functional layer: Implement the business logic of the specific functions of the Mini Program with considerable and controllable functions, including UI, network request, storage, application status and other modules, and realize functions such as request proxy and modification, switching login state, or controlling cache testability.
• Hook layer: Transparent intervention in the state and methods of the actual logical module. Since the internal state/data of the Mini Program application is associated with the developer's code, the Hook layer intercepts and intervenes in the three types of functional modules of the host application: basic library, public components, and business-specific logic through JavaScript Hook, making its state/data considerable and controllable, and providing an implementation basis for the functional layer. Generally, the hook layer needs to be loaded before the service code to ensure the effectiveness of interception.
• The underlying library of the host application. The generality transformation intervenes in the system-level interfaces provided by the Mini Program container, such as network requests, geographic information, etc.
• Public components. Component-level general, such as Meituan's public login component, after its transformation, the Mini Programs that access the login component can use the corresponding measurability capabilities, such as the ability to switch login states/simulate logout.
• Business-specific logic. The unique business logic of a Mini Program is transformed through the API provided by the testability SDK and then integrated with customization capabilities in the form of plug-ins.

The following will take the considerable and controllable network request as an example to introduce the implementation principle of the testability of the Mini Program.

Network Request Proxy

When an external person wants to control the Mini Program to set up a network proxy, the overall process is as follows:

1. The external (human/machine) first passes the instruction containing the setting of the Mini Program request proxy through HTTP/WebSocket, as shown in the figure, please forward the intercepted Mini Program sent to the 127.0.0.1:1234 proxy server;
2. The testability SDK receives the corresponding instructions at the communication layer. Pass it to the instruction distribution layer. In the instruction distribution layer, the instruction is parsed after receiving it, and the instruction execution is arranged according to the predetermined rules to determine the execution sequence.
3. The instruction distribution layer calls the functional layer to set up the network proxy in the orchestration order and passes in the open state and proxy server address parameters, and the functional layer controls the interception of the request API by the hook layer by modifying these two variables, so as to change the state of the request proxy.
4. The request method of the hook layer to intercept wx objects in the basic library of WeChat, as shown in the code shown in the following figure, is divided into the following processes:

a. Save the reference to the original wx.request method (3 lines) and set the wx object to a writable state (4-8 lines) via Object.defineProperty;

b. A new way to modify wx.request to a Hook. The input of the new method is the same as that of the original wx.request, including the request header, request address, response body, etc., so these parameters can be modified (12 lines), such as replacing the request domain name, adding the request header, and modifying the response body data.

c. Finally executed using the original method with the modified parameters (13 lines).

The hook layer controls whether the applet is proxied and the proxy server address (lines 19-22) through two variables, mockStatus and mockUrl. If a proxy is set up, the request address will be modified according to the proxy server protocol so that the request is proxied.

3. Meituan's ticket business Mini Program testing practice

Among the many businesses that have applied the testability capabilities of Mini Programs in stores, Meituan's ticket business has participated in the construction of Mini Program testability since 2021, and at present, in the ticket quality assurance work, the testability-related capabilities are deeply applied in various types of testing activities such as new demand testing, regression testing, and online inspection.

| 3.1 Measurability landing

The following is a specific example of a new requirements test for the ticket business to illustrate how testability can be implemented in a testing activity.

Background of the requirements

After the user enters the order filling page from the product details page, and after selecting the date, quantity, or filling in the information such as the player, in order to reduce the user's operation, the information filled in before entering the order filling page again needs to remain unchanged.

Operation path division

The process needs to go through the following steps: go to the fill in form page —> open the price calendar pop-up layer, select the corresponding date —> add quantity —> fill in or select the player —> click back to exit the fill in page —> enter the fill in page again to check its current status. We choose to improve the testability of the cache, relying on instruction data-driven + internal method calls to achieve the same UI operation effect, ensure the stability of testing in such scenarios and improve the execution efficiency.

Technical implementation

The whole is achieved through caching. When entering the fill-in page, the cache on the applet will first be read and rendered; When selecting the date, number and number of players, the relevant information is temporarily stored; When you exit the fill page, write this staged data to the cache.

Test analysis

Since entering the single-filling page requires reading the cache for rendering, the test process should first verify from the UI to determine whether the date, quantity, and player of the second entry are consistent with the status selected when entering the previous entry. Secondly, it should also be verified from the data, that is, there is an action of "reading" the cache when entering the page of filling in the form; When exiting the fill page, the staged data needs to be written to the cache, so you should verify that the data is written to the cache correctly and that the correct values are in the cache.

Measurability capability is practiced

• Verify the correctness of "write" through considerable verification. For Write, verify that the cached write action is correct and that the data written to the cache is correct. The appreciability transformation of the cache can report the "write" action, the current value of the "write" and the specific information of the current cache, so that you can automatically verify whether the cache value has changed correctly after the current operation, so as to complete the verification of the cache "write".

• The correctness of the "read" is verified through a controllable check. For "read", first verify that the UI can be displayed correctly, and secondly, verify the cached "read" action from the data. Because the test cache must go through multiple steps such as selecting the date, selecting the quantity, selecting the player, and returning to the exit page. The test path is cumbersome, therefore, after the controllability of the cache is transformed, the corresponding configuration instructions (as introduced in section 2.2) are passed to control the data in the cache, directly reach the page and state under test, and compare the currently running page and page benchmark map through automated tests to determine whether it is rendered correctly, so as to complete the verification of the cache "read" from the data and UI respectively.

As shown in the figure below, the ticket business has implemented a variety of measurability capabilities in the Mini Program test, including controlling page jumping, requesting proxies, controlling logins, log reporting, privacy governance, front-end and back-end environments, recording and playback, automatic interaction control, etc., all of which have been implemented accordingly and play a very important role in ticket testing activities.

| 3.2 Summary of business practices

The ticket business has made the test coverage more comprehensive with the help of testability improvements, and 30%+ of the current test scenarios rely on testability capabilities to build. There are hundreds of automated test cases on the ticket channels of Meituan Mini Programs and Dianping Mini Programs, as well as on the ticket independent Mini Program, with page coverage reaching 100% and scene coverage reaching 80%+. These test cases will trigger automatic execution at various stages such as ticket new demand testing and regression testing, and have assisted in discovering hundreds of effective problems.

4. Summary and outlook

Meituan's core local commercial/in-store R&D platform has systematically built the testability of Mini Programs since 2021, and is now integrated into the in-store terminal testing tool chain and quality assurance system. In the future, we will continue to focus on the stability of basic testability capabilities and focus on testability capabilities with more business features.

Q&A

Q: If there is a bug in the proxy logic, will it have a big impact?

A: The proxy logic itself is very simple, and there is little chance of error. When hooks are performed, there are exception monitoring capabilities and corresponding back-up policies to minimize the impact on the actual use of services even if there is a problem.

Q: Does the testability SDK need to transform the business code?

A: No, the testability SDK is transparent to business applications.

Q: What is the relationship between the Lyrebird project and the Mini Program testability SDK?

A: Lyrebird and Mini Program testability are two separate projects. The Mini Program Testability SDK is implemented in the form of an NPM package, and the NPM package can be installed in the Mini Program to make the Mini Program testable. Lyrebird can connect with the communication interface of the Mini Program Testability SDK, and then users can use the Mini Program testability capabilities through the Lyrebird Mini Program Testability page.

Q: For the testability capacity building and practice of Mini Programs, I would like to ask, if we want to use your testing tools, do we need to do any adaptations?

A: There is no need for additional adaptation, the final presentation will be in the form of an NPM package, and we can access our testability capabilities and control it when installed in the product.

Q: Will the production environment be connected to the observable SDK? If how much impact does access have on performance?

A: First of all, the impact on its performance, we actually intercepted the API or some state data of the basic library in the Mini Program, which will have a certain impact on the performance, but at present, this impact scope is relatively small for the business and is acceptable. The production environment will not introduce testability SDKs, so there will be no impact on online quality.

Q: Is there a scenario where the testability of the Mini Program is not suitable for use?

The testability of the Mini Program is mainly aimed at the front-end manual and automation scenarios of the Mini Program to improve the ability, it has a set of general extensibility framework, which can be expanded according to business needs at a low cost, however, there are specific circumstances where its applicability is limited: first, due to the constraints of the operating environment, the testability of the Mini Program cannot be supported for the measurability requirements of the host application such as WeChat or Alipay. In addition, the testability of Mini Programs focuses on terminal testing, so it is not suitable for those scenarios that require back-end service link verification, and needs to be used with targeted tools.

Author: In-store R&D platform

Source-WeChat public account: Meituan technical team

Source: https://mp.weixin.qq.com/s/NeCQUwfyrKz5VKE_E-hg_g