Dynamic template automated test exploration

background

What is dynamism ?

Dynamic refers to the cross-platform dynamic template engine technology solution, which can use the same set of templates to render the UI on both ends to meet business needs.

Dynamic Template Engine (hereinafter referred to as GaiaX) is a set of lightweight pure native dynamic card cross-terminal solutions developed by Alibaba's Youku technical team.

GaiaX was incubated to solve the bottleneck of R&D performance of multi-terminal card-based UI components.

A template refers to the abstraction of the page UI, which is finally implemented as a template file, including layout, style, data, and events. When any client fully realizes the parsing and rendering of this template file, the UI can be drawn, so as to achieve cross-end. Example:

What are the features of the dynamic solution?

• Visualization tool template building:
• Provide two supporting R&D tools, Sketch plug-in and template visualization construction Studio, which can quickly complete template export, construction, editing and other work through visual solutions, and cooperate with the code scanning preview tool to quickly complete template rehearsal and visual acceptance.
• FlexBox Layout:
• Based on StretchKit, FlexBox is implemented as the layout of the engine, and the rust language is used at the bottom to ensure the high performance and cross-platform characteristics of the overall solution.
• Cross-end solutions:
• Ensure the consistency of multi-terminal technology experience, realize a set of templates and multi-terminal rendering, and comprehensively improve R&D efficiency.
• Pure Native rendering to ensure the client experience:
• Ensure the success rate of client-side rendering, and improve the scrolling frame rate of the page through view frame equality optimization.

Why do you need to be dynamic?

• The development process reduces costs and increases efficiency, and ideally double-end work can be realized by single-end development to improve the demand throughput rate.
• The low-code development platform can support products or operations, and complete the development and launch of lightweight requirements under certain guidance.
• The pace of the online process is flexible, and ideally there is no need to follow the client to release the version, which can be launched quickly and reduce the cost of trial and error.
• Ensure consistent product experience and technical implementation at both ends.
• Pure native rendering to ensure a client-side experience.
• Online UI issues can be dynamically fixed and launched in a timely manner to reduce the scope of impact.

Dynamic application scenarios?

Dynamics are more suitable for the scenarios of card-based UI components, and there are many scenarios in the form of cards in the Zhihu client, such as: hot lists, ideas, and short containers.

Of course, at the level of business use, we are still exploring and excavating.

Dynamic technology system

The dynamic template technology system is centered on cross-terminal dynamic SDK, which reduces the technical cost of core work such as docking, development, and debugging in the design stage, R&D stage, and joint debugging stage, so as to truly improve R&D efficiency.

At present, the entire technical solution includes the thrush cross-platform animation solution, the intelligent design platform GaiaSketch, the cross-terminal template dynamic solution GaiaX, the dynamic template visualization and code building platform GaiaStudio, and the auxiliary debugging tool Quick Preview FastPreview.

The dynamic template technology solution connects the client R&D link in series, and allows developers to efficiently complete the shortest path from the original design draft to the runnable code of the component through a complete tool-based support system.

As shown in the figure below, the dynamic hierarchical design architecture is as follows:

The dynamic basic capability is a general capability provided by the dynamic middle layer + SDK capability layer + template management, including the following functions: download templates, load templates, and render templates.

Dynamically provide the underlying basic capabilities; → the business side develops business templates→ uploads and delivers templates through the template management platform.

As shown in the figure below, the dynamic hierarchical design architecture is as follows:

Quality assurance program

First of all, from the project testing process, we expect to have a complete and clear process, and we need to reach a consensus with the product and R&D.

The overall testing process includes:

Note: Marked in red requires special attention

• Request a testing resource
• When it comes to the requirements of dynamic underlying capability changes, it is necessary to know the QA before the project starts, and it is necessary to go through the production and research requirements.
• Demand KO:
• If it is a purely technical change, you can combine the requirements KO and the technical KO into one, and make sure that the technical document is clear and easy to understand.
• Invitation to Judges:
• Invite relevant QAs to participate in the requirements review and all technical RFC reviews on the scope of changes: Be sure to be specific to the page features, page basics, and feature entrances to avoid invalid full regression testing.
• Make a plan:
• Formulate a test plan and clarify the schedule, test scope, test data, test environment, test method, etc. The test scope is divided according to the responsibilities of each business module.
• Design Use Cases:
• Responsible for the overall design and review of test cases, coordinate each QA team to complete the use case design and review of their respective modules, output smoke test cases and deliver them to the R&D team.
• Sync Progress:
• Regularly synchronize the development progress with the relevant QA, and if there is a risk of delay, it needs to be fed back to the QA leader of the relevant business department in time and the schedule should be redetermined.
• Feedback:
• If there is any shortage of resources during the testing process, communicate with the QA leader of the business line in time, and if it cannot be solved, feedback to your QA leader in time, and the R&D teacher needs to fix the problem as soon as possible after raising the problem.
• Publish the report:
• After the test is completed, the test data is summarized and the test report is published.
• Summary and review:
• Promote the summary and review of each business team.
• Organizational examples:
• Organize regression test cases and identify the QA team responsible for regressing the feature in subsequent releases.

Automated testing scenarios

From the dynamic architecture layering, it can be divided into three layers:

• Business layer: e.g. short containers.
• Middle layer: Provide templates for the business layer and interact with module services for module data.
• Module management: Download and load templates.

In practice, if there is an upgrade or code change in the dynamic middle layer and the underlying SDK, you need to look at the test effect at the business layer, such as: short containers and hot lists, but there are several problems in this test:

• To upgrade the underlying capabilities, you need to go through all business scenario tests
• In the business scenario, you need to configure a whitelist and understand the business
• Lots of functional regression testing
• Backend service changes cannot be tested
• How template layer expressions are tested

Based on the above test pain points, we hope to introduce automated testing methods into dynamic business, which can be mainly divided into three types:

• Service layer: Through automatic interface testing, you can ensure the correctness of processes and fields at the interface layer.
• Middle layer: Implement automatic testing of the UI layer through Debug UI to ensure the quality of the UI layout, title, and copywriting in the middle layer.
• Template layer: Ensures the quality of expression testing through unit testing.

UI automated testing

The purpose is to conduct targeted tests based on Debug UI pages for the middle layer and the underlying SDK, and the purpose of the Debug UI is to:

In the process of dynamic implementation of different business scenarios, R&D and testing students have more and more strong requirements for the basic capabilities of developing and testing dynamic templates.

In order to improve the ecological construction of the client-side dynamic project, this requirement is proposed, and a convenient debug page is provided for R&D and testing in the future, which is convenient for development and testing templates.

• External: Support the implementation of dynamic capability requirements of the business side, reduce the R&D cost of the business side, and reduce the resistance to dynamic implementation.
• Internally: Improve the dynamic capability ecosystem, reduce the cost of R&D and testing students, and develop and test dynamic basic capabilities without relying on business scenarios.

The Debug UI is implemented as follows: You can use the MR package and integration package to jump to the debug center.

If the card is successfully rendered (image, text, layout, etc.), it means that the dynamic underlying rendering capability is normal.

Test Methods & Scenario Design

The focus of a single piece of data is tested

• The card loads normally
• The card can be clicked to jump

Multiple data test focuses

• The card loads normally
• The list card is swipetable
• The card can be clicked to jump

The test process is as follows:

The above solution will be implemented based on the UI automation script, and the test results of each execution will be uploaded to the test platform for archiving, including: test steps, test screenshots, and image comparison screenshots

Image Diff (Similarity) Test

1) Related concepts:

Image diff mainly solves the similarity between the image of this test and the baseline test image, if the data is the same, the similarity between the two should be 100%

There are two ways to compare image similarity:

1. The first type: intuitive pixel difference comparison

It is to carry out the gray difference between the two pictures, and this comparison is suitable for the situation that the two pictures are very close, or even derived from the same picture, but only partially trimmed, so that you can compare which parts have been modified.

2. The second way: use a professional algorithm to evaluate the similarity of two pictures

This kind of function is often applied to major websites for image search, and there are fewer opportunities for individuals to directly contact such algorithms, and if only the similarity value of two images is calculated, the application scenarios for ordinary users are limited.

There are two principles of professional algorithms:

Part of the algorithm is to define a similarity evaluation metric, and the calculated size of the value represents the size of the similarity. The process roughly includes: size processing, grayscale processing, calculation of pixel mean, hash calculation, distance calculation/structural similarity calculation.

There is also a type of calculation method based on neural network algorithms, which requires training samples to learn, simulating the process of human beings learning to recognize objects, and finally forming a network for the similarity recognition of specific pictures after continuous learning and feedback.

How to compare the similarity of images

The comparison method is to choose the comparison method with the baseline image, because the screen size of each machine is not the same as the factor, you can build the baseline image of each device in the project project.

Enforce the strategy

Create a scheduled task in Jenkins and execute it during the integration testing phase every week.

Click Alluer Report to view the screenshots and baseline charts included in the report, and the test results will be sent to WeCom.

Automated testing of expressions

The single test on Android is only for the expression parsing logic, and will not involve other logic of the ZHGaiaX SDK. The Android single test is based on the AndroidTest environment, relies on the Android platform, and the current test cases are written in Kotlin.

The coverage of single tests on iOS is relatively wide, not only for testing the expression parsing logic, but also for testing other logic of the ZHGaiaX SDK, such as template loading, parsing, view creation, etc.

iOS Unit Test Process:

• 使用 Xcode 打开 GaiaX/GaiaXiOS/GaiaXiOS.xcworkspace 工程
• Select Scheme:GaiaXiOSTests, then press and hold the run button and select "Test", and you can run a single test

• 修改或增加测试用例:修在 GaiaX/GaiaXiOS/GaiaXiOSTests/TestCase 下的文件
• Tests for expression parsing: GXExpressionTest.mm
• Test to load the theme: GXTemplateTest.mm
• Rendering engine interface testing: GXEgnineTest.mm
• View single-test coverage
• Configure options

• Run tests and see coverage

Access to automated testing processes:

The purpose of automated testing is to automatically trigger tests when code changes, so as to detect problems earlier. So, let's take the manual testing above and turn it into an automated test.

Automated testing mainly relies on Jenkins services and third-party plug-ins to complete full-link automated testing.

Jenkins

• Jenkins is a leader in open source CI&CD software, with over 1,000 plugins to build, deploy, and automate for any project.
• Jenkins can be run in a variety of ways, either through a system package, Docker, or through a stand-alone Java program.
• Jenkins relies on the Java environment and requires jdk to be installed.

slather

slather is a tool for generating test coverage reports for Xcode projects and hooking them up to CI.

Slather Address: https://github.com/SlatherOrg/slather

Installation:

sudo gem install slather           

How to use the command:

slather coverage -s --scheme YourXcodeSchemeName path/to/project.xcodeproj           

ocunit2junit

OCUnit2JUnit is a script that converts the output of OCUnit or Kiwi into a format used by JUnit. The main purpose is to be able to parse the output of Objective-C (OCUnit) test cases on Java-based build servers such as Jenkins.

github address: https://github.com/ciryon/OCUnit2JUnit/

Installation Method:

sudo gem install ocunit2junit           

HTML Publisher

The HTML Publisher plug-in can be used to publish build-generated HTML reports to jobs and build pages. It is designed to work with the Freestyle project as well as the Jenkins Pipeline.

You can install it directly in the local Jenkins plugin management path: Jenkins -- System Management -- Plugin Management, and search for HTMLPublisher to install it directly.

Integration into Jenkins

1. Create a new job, take the name here ios_unit_test, pay attention to the name of this job should not be repeated.

2. Configure the address and code branch of Gitlab in source code management.

3、增加构建步骤，选择 Execute shell 脚本

The configuration script is as follows:

#!/usr/bin/env bash
 
source ~/.bash_profile
 
#pod可能失败的全局参数设置
 
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8
export LC_ALL=en_US.UTF-8
 
pod install
 
xcodebuild test -workspace xxxx.xcworkspace -scheme xxxxiOSTests -destination 'platform=iOS Simulator,name=iPhone 11 Pro' -enableCodeCoverage YES 2>&1 | ocunit2junit
 
# 清除上次运行生成的文件
 
rm -rf html
 
# 可视化查看覆盖率
# --html：创建 html 静态页面，并默认创建 html 文件夹；可以通过 --output-directory 指定存放路径
# --show：打开浏览器展示 html 的内容
 
# 创建 xml 报告，默认 xml 报告保存在当前目录 test-reports 文件夹
 
slather coverage --scheme xxxxiOSTests --workspace GaiaXiOS.xcworkspace xxxxiOS.xcodeproj
 
# 创建 html 报告，默认 html 报告保存路径为前目录 html 文件夹
 
slather coverage --html --scheme xxxxiOSTests --workspace xxxxiOS.xcworkspace xxxxiOS.xcodeproj           

项目包含 xcworkspace 要指定 -workspace UnitTest.xcworkspace。 -scheme 为当前测试的 scheme,如果不知道,可以在xcworkspace 目录下执行命令 xcodebuild -list ,可以看到所有的 scheme。

By default, slather is generated as an XML report, in the test-reports folder of the current directory. Slather generates an HTML report when --html is added, in the html folder of the current directory.

4. Configure post-build operations

If you configure the test report display in the place where you will operate after the build, you can see the coverage report results through the Jenkins address.

HTML directory to archive是测试报告存储的文件路径，Index page 是测试报告名称。

Once the execution is complete, click HTMLReport to view the test report.

The coverage report for a single execution is as follows.

summary

The above is the practical experience of dynamic template automated test exploration, which can be seen that after the birth of new technologies and new frameworks, it is a quality assurance challenge for test engineers.

How to make the test efficiency more efficient and the test quality more reliable is the responsibility of the test engineer. Automation can be introduced into a project to improve test efficiency, reduce test costs, and enhance test reliability and consistency.

Of course, automated testing is not a panacea. It is not suitable for all test scenarios. Some testing tasks require manual testing because of the need for human judgment and perceptual experience.

Automated testing requires a certain amount of time, including the purchase and training of test tools, test scripting and maintenance, and test execution and monitoring. However, it is worthwhile to invest in automated testing as it will bring advantages and returns to the project.

Author: Nobuki

Source: https://zhuanlan.zhihu.com/p/612566879