Magic change xxl-job, say goodbye to manual configuration tasks completely!

XL-Job is a very good task scheduling middleware, which has the advantages of being lightweight, simple to use, and supporting distribution, which makes it widely used in our projects and solves many scheduling problems of scheduled tasks.

We all know that in the process of use, you need to go to the task dispatch center page of xxl-job first, configure the executor and specific task job, this process is good if the number of scheduled tasks in the project is not much, if there are many tasks, it is still quite laborious.

Suppose there are hundreds of such scheduled tasks in the project, then each task needs to go through the process of binding the jobHander backend interface and filling in cron expressions...

I just want to ask, who can not be confused if you fill in too much?

So out of the motivation of functional optimization (lazy), I came up with an idea the other day, is there any way to say goodbye to the xxl-job management page, so that I no longer need to go to the page to manually register executors and tasks, so that they can be automatically registered to the dispatch center.

analysis

Analysis, in fact, what we need to do is very simple, just actively register the executor and each jobHandler to the dispatch center when the project starts, the process is as follows:

Some friends may want to ask, when I create an executor on the page, isn't there an option called automatic registration, why do we have to add a new executor here?

In fact, there is a misunderstanding here, the automatic registration here refers to the automatic registration of the configured machine address to the address list of this executor according to the xxl.job.executor.appname configured in the project. However, if you have not manually created an actuator before, it will not automatically add a new actuator to the dispatch center.

Now that we have an idea, let's go straight to it, first go to github and pull a copy of the source code of xxl-job down:

https://github.com/xuxueli/xxl-job

After the whole project is imported into the idea, let's take a look at the structure:

Combined with the documentation and code, let's sort out what each module does:

• xxl-job-admin: task dispatch center, which can access the management page after starting, register executors and tasks, and call tasks
• xxl-job-core: public dependencies, which are the dependencies to be introduced when xxl-job is used in the project
• xxl-job-executor-samples: execution examples, including the springboot version and the version that does not use the framework

In order to figure out which interfaces are called by the executor and jobHandler to register and query, let's take a look at a request from the page:

Okay, so you can locate the /jobgroup/save interface in the xxl-job-admin module, and then you can easily find the source code location:

According to this idea, the following key interfaces can be found:

• /jobgroup/pageList: a conditional query of the executor list
• /jobgroup/save: adds an executor
• /jobinfo/pageList: a conditional query of the task list
• /jobinfo/add: adds a task

But if you call these interfaces directly, you will find that it will jump to the login page of xxl-job-admin:

In fact, if you think about it, it is also clear that for security reasons, it is impossible for the interface of the dispatch center to allow naked tuning. If you look back at the request on the page just now, you can see that it adds a cookie called XXL_JOB_LOGIN_IDENTITY to the headers:

As for this cookie, it is returned when the /login interface of the dispatch center is called through the username and password, and the returned response can be obtained directly. As long as you save it and carry it with you every time you request it in the future, you will be able to access other interfaces normally.

At this point, the 5 interfaces we need are basically ready, and then we are ready to start the formal transformation work.

remodelling

The purpose of our transformation is to implement a starter, and in the future, as long as this starter is introduced, the executor and jobHandler can be automatically registered, and the key dependencies to be introduced are the following two:

<dependency>
    <groupId>com.xuxueli</groupId>
    <artifactId>xxl-job-core</artifactId>
    <version>2.3.0</version>
</dependency>

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-autoconfigure</artifactId>
</dependency>
           

1. Interface call

Before calling the API of the dispatch center, we first put the XxlJobInfo and XxlJobGroup classes in the xxl-job-admin module into our starter project to receive the results of the API call.

Login API

Create a JobLoginService, obtain cookies through the login API before calling the service API, and cache the cookie to the local map after obtaining the cookie.

private final Map<String,String> loginCookie=new HashMap<>();

public void login() {
    String url=adminAddresses+"/login";
    HttpResponse response = HttpRequest.post(url)
            .form("userName",username)
            .form("password",password)
            .execute();
    List<HttpCookie> cookies = response.getCookies();
    Optional<HttpCookie> cookieOpt = cookies.stream()
            .filter(cookie -> cookie.getName().equals("XXL_JOB_LOGIN_IDENTITY")).findFirst();
    if (!cookieOpt.isPresent())
        throw new RuntimeException("get xxl-job cookie error!");

    String value = cookieOpt.get().getValue();
    loginCookie.put("XXL_JOB_LOGIN_IDENTITY",value);
}
           

Other interfaces fetch cookies directly from the cache when called, and call the /login interface if they do not exist in the cache, allowing up to 3 retries to avoid this process failing.

public String getCookie() {
    for (int i = 0; i < 3; i++) {
        String cookieStr = loginCookie.get("XXL_JOB_LOGIN_IDENTITY");
        if (cookieStr !=null) {
            return "XXL_JOB_LOGIN_IDENTITY="+cookieStr;
        }
        login();
    }
    throw new RuntimeException("get xxl-job cookie error!");
}
           

Actuator interface

Create a JobGroupService and query the list of executors based on appName and executor title.

public List<XxlJobGroup> getJobGroup() {
    String url=adminAddresses+"/jobgroup/pageList";
    HttpResponse response = HttpRequest.post(url)
            .form("appname", appName)
            .form("title", title)
            .cookie(jobLoginService.getCookie())
            .execute();

    String body = response.body();
    JSONArray array = JSONUtil.parse(body).getByPath("data", JSONArray.class);
    List<XxlJobGroup> list = array.stream()
            .map(o -> JSONUtil.toBean((JSONObject) o, XxlJobGroup.class))
            .collect(Collectors.toList());
    return list;
}
           

We need to judge whether the current executor has been registered to the dispatch center according to the appName and title in the configuration file, and skip it if it has been registered, and the /jobgroup/pageList interface is a fuzzy query interface, so in the result list of the query list, an exact match is needed.

public boolean preciselyCheck() {
    List<XxlJobGroup> jobGroup = getJobGroup();
    Optional<XxlJobGroup> has = jobGroup.stream()
            .filter(xxlJobGroup -> xxlJobGroup.getAppname().equals(appName)
                    && xxlJobGroup.getTitle().equals(title))
            .findAny();
    return has.isPresent();
}
           

Register a new executor to Mission Dispatch:

public boolean autoRegisterGroup() {
    String url=adminAddresses+"/jobgroup/save";
    HttpResponse response = HttpRequest.post(url)
            .form("appname", appName)
            .form("title", title)
            .cookie(jobLoginService.getCookie())
            .execute();
    Object code = JSONUtil.parse(response.body()).getByPath("code");
    return code.equals(200);
}
           

Task interface

Create a JobInfoService to query the task list based on the executor id and jobHandler name, which is also a fuzzy query like the above:

public List<XxlJobInfo> getJobInfo(Integer jobGroupId,String executorHandler) {
    String url=adminAddresses+"/jobinfo/pageList";
    HttpResponse response = HttpRequest.post(url)
            .form("jobGroup", jobGroupId)
            .form("executorHandler", executorHandler)
            .form("triggerStatus", -1)
            .cookie(jobLoginService.getCookie())
            .execute();

    String body = response.body();
    JSONArray array = JSONUtil.parse(body).getByPath("data", JSONArray.class);
    List<XxlJobInfo> list = array.stream()
            .map(o -> JSONUtil.toBean((JSONObject) o, XxlJobInfo.class))
            .collect(Collectors.toList());

    return list;
}
           

Registering a new task will eventually return the ID of the new task created:

public Integer addJobInfo(XxlJobInfo xxlJobInfo) {
    String url=adminAddresses+"/jobinfo/add";
    Map<String, Object> paramMap = BeanUtil.beanToMap(xxlJobInfo);
    HttpResponse response = HttpRequest.post(url)
            .form(paramMap)
            .cookie(jobLoginService.getCookie())
            .execute();

    JSON json = JSONUtil.parse(response.body());
    Object code = json.getByPath("code");
    if (code.equals(200)){
        return Convert.toInt(json.getByPath("content"));
    }
    throw new RuntimeException("add jobInfo error!");
}
           

2. Create a new annotation

When creating a task, in addition to the executor and jobHandler, the required fields include the task description, the person in charge, the Cron expression, the scheduling type, and the running mode. Here, we default to CRON as the scheduling type, BEAN as the running mode, and the information of the other three fields needs to be specified by the user.

Therefore, we need to create a new annotation @XxlRegister to use with the native @XxlJob annotation, and fill in the information in these fields:

@Target({ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
public @interface XxlRegister {
    String cron();
    String jobDesc() default "default jobDesc";
    String author() default "default Author";
    int triggerStatus() default 0;
}
           

Finally, an additional triggerStatus property is added to indicate the default scheduling status of the task, with 0 being the stop state and 1 being the running state.

3. Automatic registration of the core

After the basic preparations are completed, the following is the core code for automatic registration of the executor and jobHandler. The core class implements the ApplicationListener interface, which starts to execute the auto-registration logic after receiving the ApplicationReadyEvent event.

@Component
public class XxlJobAutoRegister implements ApplicationListener<ApplicationReadyEvent>, 
        ApplicationContextAware {
    private static final Log log =LogFactory.get();
    private ApplicationContext applicationContext;
    @Autowired
    private JobGroupService jobGroupService;
    @Autowired
    private JobInfoService jobInfoService;

    @Override
    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
        this.applicationContext=applicationContext;
    }

    @Override
    public void onApplicationEvent(ApplicationReadyEvent event) {
        addJobGroup();//注册执行器
        addJobInfo();//注册任务
    }
}
           

The code of auto-registration of executors is very simple, according to the exact match of appName and title in the configuration file, check whether the executor has been registered in the dispatch center, skip if it exists, and register a new one if it does not exist:

private void addJobGroup() {
    if (jobGroupService.preciselyCheck())
        return;

    if(jobGroupService.autoRegisterGroup())
        log.info("auto register xxl-job group success!");
}
           

The logic for auto-registration tasks is a bit more complex and needs to be done:

• Get all the beans in the spring container through the applicationContext, and then get all the methods with @XxlJob annotations added to those beans
• Check the method obtained above to see if we have added our custom @XxlRegister annotations, and if not, skip it and do not register it automatically
• For methods that have both @XxlJob and @XxlRegister, check whether the executor id and jobHandler have been registered with the dispatch center, and skip them if they already exist
• If the jobHandler meets the annotation conditions and has not been registered, the API is called to register with the dispatch center

The specific code is as follows:

private void addJobInfo() {
    List<XxlJobGroup> jobGroups = jobGroupService.getJobGroup();
    XxlJobGroup xxlJobGroup = jobGroups.get(0);

    String[] beanDefinitionNames = applicationContext.getBeanNamesForType(Object.class, false, true);
    for (String beanDefinitionName : beanDefinitionNames) {
        Object bean = applicationContext.getBean(beanDefinitionName);

        Map<Method, XxlJob> annotatedMethods  = MethodIntrospector.selectMethods(bean.getClass(),
                new MethodIntrospector.MetadataLookup<XxlJob>() {
                    @Override
                    public XxlJob inspect(Method method) {
                        return AnnotatedElementUtils.findMergedAnnotation(method, XxlJob.class);
                    }
                });
        for (Map.Entry<Method, XxlJob> methodXxlJobEntry : annotatedMethods.entrySet()) {
            Method executeMethod = methodXxlJobEntry.getKey();
            XxlJob xxlJob = methodXxlJobEntry.getValue();

            //自动注册
            if (executeMethod.isAnnotationPresent(XxlRegister.class)) {
                XxlRegister xxlRegister = executeMethod.getAnnotation(XxlRegister.class);
                List<XxlJobInfo> jobInfo = jobInfoService.getJobInfo(xxlJobGroup.getId(), xxlJob.value());
                if (!jobInfo.isEmpty()){
                    //因为是模糊查询，需要再判断一次
                    Optional<XxlJobInfo> first = jobInfo.stream()
                            .filter(xxlJobInfo -> xxlJobInfo.getExecutorHandler().equals(xxlJob.value()))
                            .findFirst();
                    if (first.isPresent())
                        continue;
                }

                XxlJobInfo xxlJobInfo = createXxlJobInfo(xxlJobGroup, xxlJob, xxlRegister);
                Integer jobInfoId = jobInfoService.addJobInfo(xxlJobInfo);
            }
        }
    }
}
           

4. Automatic assembly

Create a configuration class that scans beans:

@Configuration
@ComponentScan(basePackages = "com.xxl.job.plus.executor")
public class XxlJobPlusConfig {
}
           

Add it to the META-INF/spring.factories file:

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
  com.xxl.job.plus.executor.config.XxlJobPlusConfig
           

At this point, the starter is written, and you can use Maven to publish the jar package to the local or private server:

mvn clean install/deploy
           

Test

Create a new springboot project and bring in the package we made above:

<dependency>
    <groupId>com.cn.hydra</groupId>
    <artifactId>xxljob-autoregister-spring-boot-starter</artifactId>
    <version>0.0.1</version>
</dependency>
           

To configure the information of xxl-job in application.properties, the first thing is the original configuration content:

xxl.job.admin.addresses=http://127.0.0.1:8080/xxl-job-admin
xxl.job.accessToken=default_token
xxl.job.executor.appname=xxl-job-executor-test
xxl.job.executor.address=
xxl.job.executor.ip=127.0.0.1
xxl.job.executor.port=9999
xxl.job.executor.logpath=/data/applogs/xxl-job/jobhandler
xxl.job.executor.logretentiondays=30
           

In addition, we would like to add the new configuration content required by our own starter:

# admin用户名
xxl.job.admin.username=admin
# admin 密码
xxl.job.admin.password=123456
# 执行器名称
xxl.job.executor.title=test-title
           

Once you're done, configure the XxlJobSpringExecutor in your code, and then add the native @XxlJob annotation and our custom @XxlRegister annotation to the test interface:

@XxlJob(value = "testJob")
@XxlRegister(cron = "0 0 0 * * ? *",
        author = "hydra",
        jobDesc = "测试job")
public void testJob(){
    System.out.println("#公众号：码农参上");
}


@XxlJob(value = "testJob222")
@XxlRegister(cron = "59 1-2 0 * * ?",
        triggerStatus = 1)
public void testJob2(){
    System.out.println("#作者：Hydra");
}

@XxlJob(value = "testJob444")
@XxlRegister(cron = "59 59 23 * * ?")
public void testJob4(){
    System.out.println("hello xxl job");
}
           

Start the project, and you can see that the executor is automatically registered successfully:

If you open the task management page of the dispatch center, you can see that the tasks with two annotations have been automatically registered:

Manually executing the task from the page for testing, you can execute successfully:

At this point, even if the writing and testing process of starter is basically completed, after it is introduced in the project, it will save more time to fish and learn in the future~

At last

The complete code of the project has been passed to my github, if you need it, you can download it yourself, and you are welcome to support it with a star~

https://github.com/trunks2008/xxl-job-auto-register

Source: https://mp.weixin.qq.com/s/Zzv1ZzBUBnH2BrJIxra40g