datax從DB2抽取到OceanBase

DataX實作DB2到OceanBase

DataX實作DB2到OceanBase

一：環境準備

①、Datax 處理

Datax源碼進行打包時，確定有配置rdbmsreader和rdbmswriter ,確定包裡有 db2jcc4.jar ，打包完成後上傳伺服器進行環境配置，能夠執行 Python datax.py test.json

②、json 配置

test.json

{
"job": {
        "setting": {
            "speed": {
                "channel": 2,
            },
            }
        },
        "content": [
            {"reader": {
                "name": "rdbmsreader",
                "parameter": {
                    "connection": [{
                        "jdbcUrl": ["jdbc:db2://10.10.8.1:50000/test"],
                        "table": [
                            "table_test"
                        ]
                    }],
                    "username": "user_test",
                    "password": "passwd_test",
                    "table": "table_test",
                    "column":["*"]}
                },
                "writer": {
                    "name": "oceanbasev10writer",
                    "parameter": {
                        "writeMode": "replace",
                        "username": "test",
                        "password": "PassTest",
                        "writerThreadCount": 5,
                        "column": ["*"
                        ],
                        "connection": [
                           {
                                "jdbcUrl": "||_dsc_ob10_dsc_||叢集:租戶||_dsc_ob10_dsc_||jdbc:mysql://10.10.250.1:2883/testdb?useUnicode=true&characterEncoding=utf-8",
                                "table": [
                                    "table_test"
                                ]
                            }
                        ]
                    }
                }
            }
        ]
    }
}
           

二：執行導入資料

python datax.py test.json
           

A、截取一段輸出

2021-09-16 17:16:11.942 [job-0] INFO  JobContainer - 
	 [total cpu info] => 
		averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
		-1.00%                         | -1.00%                         | -1.00%
                        

	 [total gc info] => 
		 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
		 PS MarkSweep         | 14                 | 14                 | 14                 | 10.713s            | 10.713s            | 10.713s            
		 PS Scavenge          | 142                | 142                | 142                | 6.004s             | 6.004s             | 6.004s             

2021-09-16 17:16:11.942 [job-0] INFO  JobContainer - PerfTrace not enable!
2021-09-16 17:16:11.942 [job-0] INFO  StandAloneJobContainerCommunicator - Total 103080 records, 89593314 bytes | Speed 1.71MB/s, 2061 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 18.757s |  All Task WaitReaderTime 14.881s | Percentage 100.00%
2021-09-16 17:16:11.943 [job-0] INFO  JobContainer - 
任務啟動時刻                    : 2021-09-16 17:15:20
任務結束時刻                    : 2021-09-16 17:16:11
任務總計耗時                    :                 51s
任務平均流量                    :            1.71MB/s
記錄寫入速度                    :           2061rec/s
讀出記錄總數                    :              103080
讀寫失敗總數                    :                   0
           

B、一般問題

1：确認端口和IP通的

2：确認配置的JDBC格式正确

3：确認使用者名密碼正确

4：确認叢集、租戶、資料庫名正确

5：确認表結構是一緻的，或者配置的字段和目标資料庫表字段一緻

多測試，可以

參考OceanBase 官方文檔：https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.0/use-datax-to-full-migration-data-to-oceanbase

參考Datax文檔：https://gitee.com/mirrors/DataX/blob/master/rdbmsreader/doc/rdbmsreader.md