When developers perform data processing tasks, they often encounter errors such as the non-existence of a field or the non-existence of a table due to the modification or deletion of a field by the business system, resulting in task failure.
In versions earlier than FDL 4.1.5, because scheduled tasks could not automatically obtain changes in the structure of the source table, changes to the business system could not be transmitted to the data warehouse in a timely manner, and developers had to spend a lot of time manually maintaining them, and even had to be called up in the middle of the night to fix bugs. And in the process of frequently changing the database structure, it is easy for human error to occur.
Solution - DDL synchronization of scheduled tasks
The DDL synchronization function of scheduled tasks provided by FineDataLink solves the preceding problems at one time. When IT personnel develop scheduled tasks, FDL can prompt the DDL changes of the source data table and notify the specified users. At the same time, you can directly process the fields of the target table in the task and update the structure of the target table.
1. Monitoring of changes in the structure of the source table
Developers can go to the "Data Synchronization" node, check the "Source Table Structure Change Monitoring" option, and the node will automatically obtain the source table structure each time it is executed, and compare it with the table structure configured by the current node to determine the source table structure change.
Users can set "Result Notification" in "Task Control" to automatically notify the specified user after the change, and the user can manually adjust the task.
2. Update field mapping
If the data source is SQLServer, Oracle, PostgreSQL, or MySQL, FDL prompts the user to update the field mapping when the schema of the source table changes.
3. Update the target table
When the data destination is set to "Existing Table", if the source table field has changed compared with the previous one, the developer can click "Update Destination Table" to adjust the target table. In Update Destination Table, you can select an existing target table field or manually enter a new field, and then generate the corresponding DDL statement and execute it.
By automatically monitoring and synchronizing changes in the schema of the source table, FDL can notify relevant personnel of the changes in real time, and support automatic or manual updating of the schema of the target table, thus effectively reducing errors and data inconsistencies caused by human operations.
This automated DDL synchronization process not only frees developers from tedious database structure maintenance, saves a lot of valuable time, but also significantly improves work efficiency and data processing reliability. Whether it is facing the challenge of frequent changes in business systems or the need to ensure the consistency of data flows, FDL's DDL synchronization function of scheduled tasks has become an indispensable tool for developers.
With FineDataLink, data processing is more efficient, development is easier, and business is smoother. Here is the link to the use of FineDataLink attached below, interested friends can click the link to download it for free! https://s.fanruan.com/aufoa ETL/ELT|Data Fusion|Data Cleaning-FineDataLink Data Integration Platform