datax从DB2抽取到OceanBase

DataX实现DB2到OceanBase

DataX实现DB2到OceanBase

一：环境准备

①、Datax 处理

Datax源码进行打包时，确保有配置rdbmsreader和rdbmswriter ,确保包里有 db2jcc4.jar ，打包完成后上传服务器进行环境配置，能够执行 Python datax.py test.json

②、json 配置

test.json

{
"job": {
        "setting": {
            "speed": {
                "channel": 2,
            },
            }
        },
        "content": [
            {"reader": {
                "name": "rdbmsreader",
                "parameter": {
                    "connection": [{
                        "jdbcUrl": ["jdbc:db2://10.10.8.1:50000/test"],
                        "table": [
                            "table_test"
                        ]
                    }],
                    "username": "user_test",
                    "password": "passwd_test",
                    "table": "table_test",
                    "column":["*"]}
                },
                "writer": {
                    "name": "oceanbasev10writer",
                    "parameter": {
                        "writeMode": "replace",
                        "username": "test",
                        "password": "PassTest",
                        "writerThreadCount": 5,
                        "column": ["*"
                        ],
                        "connection": [
                           {
                                "jdbcUrl": "||_dsc_ob10_dsc_||集群:租户||_dsc_ob10_dsc_||jdbc:mysql://10.10.250.1:2883/testdb?useUnicode=true&characterEncoding=utf-8",
                                "table": [
                                    "table_test"
                                ]
                            }
                        ]
                    }
                }
            }
        ]
    }
}
           

二：执行导入数据

python datax.py test.json
           

A、截取一段输出

2021-09-16 17:16:11.942 [job-0] INFO  JobContainer - 
	 [total cpu info] => 
		averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
		-1.00%                         | -1.00%                         | -1.00%
                        

	 [total gc info] => 
		 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
		 PS MarkSweep         | 14                 | 14                 | 14                 | 10.713s            | 10.713s            | 10.713s            
		 PS Scavenge          | 142                | 142                | 142                | 6.004s             | 6.004s             | 6.004s             

2021-09-16 17:16:11.942 [job-0] INFO  JobContainer - PerfTrace not enable!
2021-09-16 17:16:11.942 [job-0] INFO  StandAloneJobContainerCommunicator - Total 103080 records, 89593314 bytes | Speed 1.71MB/s, 2061 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 18.757s |  All Task WaitReaderTime 14.881s | Percentage 100.00%
2021-09-16 17:16:11.943 [job-0] INFO  JobContainer - 
任务启动时刻                    : 2021-09-16 17:15:20
任务结束时刻                    : 2021-09-16 17:16:11
任务总计耗时                    :                 51s
任务平均流量                    :            1.71MB/s
记录写入速度                    :           2061rec/s
读出记录总数                    :              103080
读写失败总数                    :                   0
           

B、一般问题

1：确认端口和IP通的

2：确认配置的JDBC格式正确

3：确认用户名密码正确

4：确认集群、租户、数据库名正确

5：确认表结构是一致的，或者配置的字段和目标数据库表字段一致

多测试，可以

参考OceanBase 官方文档：https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.0/use-datax-to-full-migration-data-to-oceanbase

参考Datax文档：https://gitee.com/mirrors/DataX/blob/master/rdbmsreader/doc/rdbmsreader.md