background
In order to make full use of the equipment maintained by the cloud test platform and improve the utilization rate of idle devices, automated testing is carried out to replace partial regression testing, repetitive testing, and multi-device compatibility testing, and meet the following types of automated testing requirements:
Random tests (monkey, random operation instructions):
On the basis of multi-device execution, a series of operations such as installation, startup, override installation, monkey test/random instructions, and uninstallation are completed.
Traversal tests (deep traversal, intelligent exploration):
On the basis of the monkey test/random instructions, the execution steps are modified, the random operation is replaced by the deep traversal of the internal pages of the app, and the algorithm strategy is designed to carry out intelligent exploration operations on the tested app to a certain extent.
UI automated testing (appium):
On the basis of the support of the automated test framework, it provides the ability to run multi-device scripts, and test developers only need to write test scripts that meet the rules, and then they can select multiple devices for testing on the cloud test platform and obtain test results.
At the same time as the above automated testing, the test report needs to reflect the following contents:
- Real-time generation of test reports and test result data
- Obtain device logs in real time and provide corresponding analysis results
- Controlled, automated testing process
- Take a screenshot or record a video of the device page during the execution process
- Data analysis of device types in test results
In the above background description, several test types are introduced, the first two have a relatively simple design process and do not require the participation of external personnel, and the third test type is more complex in design, and the subsequent article also focuses on the third test type.
Automate the execution framework design
Automation framework
at
Chen Kangkang: Zhihu Mobile Cloud Test Platform Practice (2) - Agent Design and Implementation 32 Agree · 5 Review articles
In the research and comparison of automation frameworks and the use of major cloud test platforms, the automated testing framework Appium, which has certain advantages in all aspects, was selected as the execution control layer of automated testing, and the local launch of Appium Hub received script execution requests, which will not be repeated here.
Scripting languages and execution frameworks
The cloud test platform is developed by Java + kotlin, and the client control is based on Java implementation, here it is natural to choose Java as the scripting language, and the subsequent scripts and process descriptions are also based on the Java language implementation, but it is not mandatory here in the selection of scripting languages, and Ruby, Python, PHP, JavaScript and C# can also be selected, but the subsequent implementation needs to be compatible with the platform one more step.
In terms of script execution, no third-party running frameworks such as junit and testng are used, mainly to maintain the control of script running and the interaction of running data during the running process, the following is the script running implementation scheme:
1. The platform provides a certain range of scripting capabilities
It mainly refers to the scripting of the running process, and how to provide common methods such as screenshots, step logs, checkpoints, etc., for Java, some public methods can be abstracted and put into the parent object of the script, and the scripting ability is given to the script itself through inheritance, and Python can also unify a standard class library and use it by introduction.
2. The script is dynamically compiled by the agent at runtime, and the script object is instantiated by reflection
Runtime processing scripts need to distinguish between dynamic and non-dynamic languages, or take Java and Python as examples, because there is no borrowing of third-party testing frameworks, triggering script running needs to be compiled for Java, that is, dynamic compilation mentioned in the title, and then run by reflection instantiation objects, there are two requirements, firstly, the script writing needs to be within the package limited by the cloud test platform, and secondly, the method of script running and inheritance needs to conform to the agreed rules. For Python, the script content is written to memory in the form of IO, and then reflected in the form of strings, imported modules, and then searched for contract function execution by modules.
3. Use the reflection instance object to run the script, and call the methods and scripts in the instance to interact with data and strong control
After instantiating the script, you need to inject the required running data into the instance before running, for example: appium's appiumDriver, and at the same time, you can call the agreed methods in the instantiated object at any time to control the script run, such as obtaining execution steps, logs, pictures, passing parameters, and controlling the interaction of the script to pause, run, and stop, which is why some third-party frameworks are not used to trigger the test.
Through the above process, the basic process from use case to execution is basically realized, as shown in the figure:
In other words, as long as a scripts that conform to the specification are provided, the generality of the framework can be used to run the scripts on any Appium-supported device, which also strips the most critical part of automated testing, that is, scripting, and provides more possibilities for feature richness for use cases such as script management and data template combination.
Process design
The following is the complete business execution solution of script automation testing:
The automated test UI code is hosted on the company's git server in the form of a git project, the script is written on the basis of the project, and the debugging script is merged into the git project after it is passed, and each script will have a unique address on gitlab, through which the specified script can be obtained, and then the script is submitted to run the test through the interface/form, and the quality report is obtained after the completion of the run.
It is worth mentioning that in the process of automated testing, script data such as public methods and selenium / appium public page objects will become larger and larger with the deepening of automated testing, so the extraction process of public methods in the diagram is required, which brings the benefit that the script writing efficiency will be faster and faster due to the extraction and reuse of scripts.
The following is a script execution process diagram, including the process of script execution and the collection of results:
Script execution data, execution of interaction design schemes:
For example, the driver of the appium needs to be used when the script is executed, and the driver is determined by the device parameters of the platform, and the platform dynamically generates the driver during the runtime and then injects the driver into the script through the protocol class, and controls it through the stop, pause, wait, and destroy methods of the protocol class during the runtime, and obtains the running result through the protocol class after the operation is completed. When the script needs some specific functions, it can also introduce interface methods through the protocol class, and then run the platform to dynamically inject the implementation class through the interface proxy.
In the process of implementation, two difficulties were found, mainly as follows:
-
Script compilation issues caused by class and third-party package reference management
In the process of automated test script writing, it is inevitable to use some introduction packages, such as some convenient tool class use packages, etc., for JAVA scripts, update classes, package introduction, etc. need to be recompiled and deployed before they can be run and used, and the test platform is not likely to recompile and deploy the platform because of the changes in script classes and packages, and most of the writing of scripts is the change of class introduction, and the proportion of package introduction changes is very small, so Java dynamic compilation technology is used to solve the dynamic introduction of classes on the change of classes. Third-party package introduction updates are introduced in advance through the release of platform projects, and these package upgrades are completed.
-
Automated update process for protocol class and package updates
In the middle of the cloud platform and the script project, the data is exchanged through the protocol class, and the defined protocol class and package need to be redeployed by the cloud platform according to the above scheme, and in practice, it is found that the capacity building and expansion of the script need to be realized through the modification of the protocol class, which determines that the protocol class will change frequently, so the agent project manually verifies the protocol class version on the server before dynamic compilation, and if a new version is found, the new version of the protocol class is downloaded jar, which is replaced with the protocol class version parameter of -classpath during dynamic compilation, so that the content of the specified protocol class can be dynamically updated and automatically deployed.
Script design
As mentioned above, "For Java, some common methods can be abstracted and put into the parent object of the script, and the scripting ability can be given to the script itself through inheritance", so the script design is divided into two modules:
Protocol parent object implementation
As shown in the figure, the parent object provides script capabilities such as test driver androidDriver, log logs, checkPoint checkpoints, and img screenshots
Written with example objects
As shown in the figure, the actual script inherits the capabilities of the parent object and can complete the automated test task of writing relevant page test logic and business logic
Script debugging and running
Platform script debugging can be performed in the following two ways:
Debugging locally
Use the main method yourself, or use the unit testing framework to debug the script to pass
The body is then modified to an automation script in the prescribed format, modifying the appiumDriver ingestion method, adding logging, screenshots, checkpoints, and so on, as appropriate
Remote debugging
For remote debugging, you need to apply for a device on the cloud test platform, as shown in the following figure:
Click the button in the diagram to paste the script into the run, as shown below:
The test report is displayed
The report is mainly divided into the following parts
Test basic information
It mainly displays the basic information of the app and some information related to the test, the overview shows the passing ratio of all checkpoints of the script, the data statistics of the device, pass and failure of the script run, and the operation overview shows the process and results of each script running on each device
Checkpoint overview
The core results of the script run mainly display all the test results in the automated test process
Screenshots/Steps/Exceptions
After the script is run, the detailed data of the script use case is obtained through reflection, and the steps, pictures, and inspection results of each run are displayed in detail through "Screenshots/Steps/Exceptions".
Performance charts, process videos
At the same time, during the script run, the Appium framework function will also be used to realize the network and uplink statistics of the device.
Get Performance Data - Appiumappium.io/docs/en/commands/device/performance-data/get-performance-data/
and
Start Screen Recording - Appiumappium.io/docs/en/commands/device/recording-screen/start-recording-screen/
, and then render the performance data to generate charts and provide a video playback of the script execution process
Author: Chen Kangkang
Source: https://zhuanlan.zhihu.com/p/84323222