Introduction: Efficient use of Java build tools | Maven. As we all know, the most mainstream Java build tool is Maven/Gradle/Bazel, for each tool, I will start from the common scenario problems in daily work, such as dependency management, build acceleration, flexible development, efficient migration, etc., and introduce how to use these three tools efficiently and flexibly.
Hello everyone, I am Hu Xiaoyu, currently in the cloud effect is mainly responsible for The Flow pipeline orchestration, task scheduling and execution engine related work.
As a CRUD expert with many years of experience in the development of Java development and testing toolchains, he has used all the mainstream Java construction tools and has precipitated a set of methods for how to use Java construction tools efficiently. As we all know, the most mainstream Java build tool is Maven/Gradle/Bazel, for each tool, I will start from the common scenario problems in daily work, such as dependency management, build acceleration, flexible development, efficient migration, etc., and introduce how to use these three tools efficiently and flexibly.
The past and present lives of Java build tools
In ancient times, Java construction was using make, and there were many awkward and inconvenient places to write makefile for Java construction.
Immediately after the birth of Apache Ant, Ant can flexibly define processes such as cleaning up compilation test packaging, but there are still many inconveniences due to the lack of dependency management functions and the need to write complex xml.
Then came the birth of Apache Maven, a dependency management and build automation tool that follows rules that are greater than conventions. Although xml is also required, it is easier to manage for complex projects, with a standardized engineering structure and clear dependency management. In addition, since Maven is essentially a plugin execution framework and also provides a certain degree of openness, we can create a certain flexibility for building the composition through Maven's plugin development.
However, due to the use of conventions greater than configuration, a certain degree of flexibility is lost, and due to the use of XML to manage the construction process and dependencies, with the expansion of the project, configuration management will still bring a lot of complexity, in this context, the collection of Ant and Maven's respective advantages Gradle was born.
Gradle is also a tool that integrates dependency management and build automation. First of all, he no longer uses XML but instead uses Groovy-based DSL to describe tasks to string together the entire build process, while also supporting plugins to provide convention-based builds similar to Maven.com. In addition to its many advantages in build dependency management, Gradle also has an advantage in build speed, providing powerful caching and incremental build capabilities.
In addition to the above Java build tools, Google opened a powerful but difficult to get started distributed build tool Bazel in 2015, with the characteristics of multi-language, cross-platform, reliable incremental build, which can be multiplied on build speed because it only recompiles files that need to be recompiled. Bazel also offers distributed remote builds and remote build caches to help speed up builds.
At present, there are relatively few people in the industry who use Ant, mainly using Maven, Gradle and Bazel, how to really play their greatest role based on the characteristics of these three tools, is the problem that this series of articles will help you solve. Let's start with Maven.
Use Maven gracefully and efficiently
When we are maintaining a Maven project, focusing on the following three issues can help us use Maven better.
● How to gracefully manage dependencies
● How to speed up our build testing process
● How to extend our own plugins
Elegant dependency management
In dependency management, there are the following practical principles that can help us implement dependency management in different scenarios gracefully and efficiently.
Use the endowment Management in the parent module to configure dependencies
Use dependencies in submodules, use dependencies
● Use profiles for multi-environment management
Take, for example, a standard spring-boot multi-module Maven project that I maintain in my day-to-day development.
The dependencies between the various modules within the project are as follows, which is usually the structure of the standard spring-boot restful api multi-module project.
Convenient dependency upgrades
Usually we encounter the following problems when relying on upgrades:
● Multiple dependency association upgrades
● Multiple modules need to be upgraded together
In the pom .xml of the parent module, we have configured the basic spring-boot dependency, and also configured the logback dependency required for log output, and it can be seen that we follow the following principles:
(1) Configure the endowmentManagement in the pom of the parent module of all submodules to manage the dependent versions in a unified manner. Configure dependencies directly in submodules, no longer entangled in specific versions, and avoid potential dependency version conflicts.
(2) Configure the same dependencies of groupId together, such as groupId as org.springframework.boot, we configure them together.
(3) Configure the groupId to be the same, but need a set of artifactIds that rely on the common function, configure it together, and extract the version number into variables at the same time, so as to facilitate the upgrade of the version common to the subsequent set of functions. For example, the version that spring-boot depends on is extracted as spring-boot.version.
In the pom .xml of the submodule build-engine-api, since the version of the spring-boot-related dependency dependent in the dependencymanagement is configured in the parent pom, in the pom of the submodule, it is only necessary to declare the dependency directly in the dependencies, ensuring the consistency of the dependent version.
Reasonable scope of dependency
Maven dependencies have a definition of scope, compile/provieded/runtime/test/system/import, in principle, only configure the scope of dependencies according to the actual situation, and at the necessary stage, only the necessary dependencies are introduced.
90% of Java programmers should have used org.projectlombok: lombok to simplify our code by converting annotations into Java implementations during compilation. Therefore, the dependent scope is provided, which is required at compile time, but needs to be excluded when the final product is built.
When your code needs to use jdbc to connect to a mysql database, we often want to code for the standard JDBC abstraction, rather than using the MySQL driver implementation by mistake. At this time, the dependent scope needs to be set to runtime. This means that we cannot use the dependency at compile time, which is included in the final product and can be found under the classpath when the program is finally executed.
In the submodule dao, we have a scenario where sql is tested and an in-memory database h2 needs to be introduced.
Therefore, we set the scope of h2 to test so that we can use it when testing compiles and executes, while avoiding it from appearing in the final product.
For more information about the use of scope, please refer to the official help documentation.
Multi-environment support
As a simple example, when our service is deployed in the public cloud, we use a MySQL version of 8.0 on the cloud, and when we want to deploy to a private cloud, the user provides a version of MySQL with version 5.7 for self-operation. Therefore, we use different versions of mysql in different environments: mysql-connector-java.
Similarly, we are often faced with the same set of code during the actual development of a project. When deployed in multiple sets of environments, there are some dependencies that are inconsistent.
For more uses of profiles, you can refer to the official help documentation
Dependency error correction
If you've used the endowment Management in the parent pom to lock in a dependency version, there's a high probability that you'll rarely encounter a dependency conflict.
But when you still unexpectedly see the Two exceptions of NoSuchMethodError and ClassNotFoundException, there are two ways to quickly correct your mistakes.
(1) Find conflicting dependencies through dependency analysis
(2) By adding stdout code to find the conflicting class from which dependency it was actually found
Use the corresponding version information in the specific path to find the corresponding version and correct it.
Of course, this method can also correct some dependencies that are incorrectly loaded into the classpath, and conflicts caused by non-project dependency configurations.
The test build process is accelerated
As a developer, we always want our project to be performed quickly and steadily no matter what the situation, so in the use of Maven, we need to follow the following principles.
● Reuse the cache as much as possible
● Build or test in parallel as much as possible
Rely on download acceleration
Typically, depending on the configuration in the Maven configuration file ${user.home}/.m2/settings.xml, it is cached at ${user.home}/.m2/repository/.
Usually in the construction process, dependent downloads tend to become a time-consuming part, but through some simple settings, we can effectively reduce dependent downloads and updates.
● Optimize updatePolicy settings
updatePolicy specifies how often an update is attempted. Maven compares the timestamp of the local POM (stored in the maven-metadata file in the repository) to the remote. Options include: always, daily( default), interval: X (where X is an integer in minutes), never.
● Use offline builds
In addition to that, if a cache already exists in the build environment, maven's offline mode can be used to build, avoiding dependencies or downloading updates for plugins.
Intuitively, information such as Downloading will not appear in the log.
The build process is accelerated
By default, the Maven build process doesn't take full advantage of the full capabilities of your hardware, he'll sequentially build each module of your maven project. At this time, if parallel builds can be used, there will be an opportunity to improve the build speed.
The above are two commands built in parallel, and the corresponding commands can be selected according to the actual CPU situation. But if you find that the build time hasn't been reduced, then there may be similar dependencies between your maven modules, which are just a simple pass-through.
Then parallel construction is not suitable for you, if your inter-module dependencies have the possibility of parallelism, then use the above commands to build, in order to make parallel construction work.
The testing process is accelerated
When we try to speed up the part of the maven engineering test case, then we have to mention a plugin, maven-surefire-plugin.
When you're executing the mvn test, the surefire plugin is working by default. If we want to use the ability to parallelize in our tests, we can configure it as follows.
However, it is important to note that improper use of parallelism for testing may have side effects. For example, when parallel is configured for metads, but for some reason there is a sequence requirement between the execution of test cases, there will be a failure of the use case because the use case method is executed in parallel, so it also forces us, if you want to get faster test speed, the writing of the case also needs to be independent and efficient.
For more information on the use of the surefire plugin, please refer to this document.
Maven plugin development
Maven is essentially a plugin execution framework, and all execution processes are done independently by one plugin. For the core plugins of maven, please refer to this document.
In addition to these plugins that maven-install-plugin/mvn-surefire-plugin/mvn-deploy-plugin, there are also some plugins provided by the third party, single-test coverage plugin mvn-jacoco-plugin, swagger-maven-plugin that generates API documents, and so on.
In the process of daily work, I encountered such a problem: there was an obvious problem with sql being released to the pre-release environment, and because the pre-release and production were using the same db instance, due to the performance problems of sql, it affected the online.
In addition to avoiding similar problems through the necessary code review access, it is simpler that we can implement a plugin for SQL scanning in the code by ourselves, so that the code directly fails in CI, and automatically avoids the occurrence of such problems. So we developed a maven plugin, the use of the method and effect is as follows:
Introduce the plugin com.aliyun.yunxiao:mybatis-sql-scan that we developed and deployed into the project.
Execute the following command, or other command that contains the validation stage execution.
We will see in the log the following information about the execution of the plugin
When scanning for defects, build fails and the corresponding message appears in the log:
In the GlobalLockMapper .java this file, we have a full table scan of the sql statement that may be risky,
Also build fails.
Next, I will start with how to develop this maven plugin for abnormal sql scanning to help you understand the process of plugin development.
1. Create a project
The generated sample project is as follows,
Where MyMojo .java defines the entry implementation of the plugin,
Also in the root pom .xml can be seen,
● packaging为“maven-plugin”。
● In the dependency configuration, it relies on some basic two-party libraries developed by plug-ins.
● Under the plugin node, we rely on maven-plugin-plugin to help us complete the construction of the plugin.
2. Mojo implementation
Before starting to implement our Mojo, we need to do the following analysis:
● Which lifetime of the plugin is executed in maven
● What entry parameters are required for the plug-in to execute
● How to exit after the plug-in execution is completed
Since the plugin we want to implement is to do mybatis annotation scans such as @Update/@Select, to determine whether there are abnormal sqls, such as whether there is a full table scan of sql, whether there is a full table update of sql, etc., for this scenario,
● Because you need to scan a specific source code, you need to know where the project source code is located and which files are scanned
● When the plug-in scans for an exception, it can only report an error, and does not need to produce any report
● You want to trigger a scan when you subsequently execute mvn validate
So the expected plugin is like this,
so
● @Mojo (name = "check") defines the gold
● @Parameter
○ @Parameter (defaultValue = "${project}", readonly = true) parameters bind the root directory of the project, and project.getCompileSourceRoots() to get the root path to the source code
○ We define mapperFiles to be responsible for scanning wildcards of which files, and excludeFiles for which files are excluded
● execute()
○ With the above foundation, in the execute method we can implement the corresponding logic, when the scan results in the abnormal SQL, throw a MojoFailureException exception, the plugin will fail to terminate.
Above, we have completed the development of the basic capabilities of a plugin.
3. Packaging and uploading of plug-ins
After the plugin development is completed, we can configure the distributionManagement, and then perform mvn deploy to complete the construction and release of the plugin.
I hope that through my introduction, I can help you better use maven, the next article we talk about Gradle, welcome to continue to pay attention to us.
301 Moved Permanently
This article is the original content of Alibaba Cloud and may not be reproduced without permission.