OB君:好消息!「 OceanBase 2.2 版本 」正式上線官網啦!(點選閱讀原文即可直接下載下傳)OceanBase 2.2版本是成功支撐2019年天貓雙11大促的穩定版本,同時也是用于TPC-C測試且榮登TPC-C性能榜首的版本。我們将在接下來的時間裡為大家持續推出 “OceanBase 2.2 手把手系列” ,将手把手帶大家一起體驗OceanBase 2.2的強大功能。歡迎持續關注!
引言
OceanBase 2.2版本近期已認證官網提供下載下傳 (
https://oceanbase.alipay.com/download/resource),2.2支援Oracle租戶。OceanBase在2019年10月2日榮膺國際事務委員會(TPC)審計釋出的TPC-C基準測試榜首,用的就是Oracle租戶。TPC-C測試使用了207多台阿裡雲高配ECS伺服器,是因為TPC-C标準對應用、資料庫等規範非常細緻嚴格。一般來說普通企業或個人很難有那樣的條件去測試。
BenchmarkSQL是開源的TPC-C測試程式,它弱化了TPC-C的關鍵标準(資料分布和應用執行行為方面),使得用幾台伺服器就可以跑TPC-C成為可能。當然這個結果不能跟官方TPC-C的結果相比較。不過,使用BenchmarkSQL來比較不同的資料庫的事務處理能力還是有一定參考意義的,尤其是相比Sysbench而言。
OceanBase測試租戶準備
1.sys租戶參數修改
BenchmarkSQL會加載大量資料,短時間内對OceanBase記憶體消耗速度會很快,是以需要針對記憶體當機合并和限流參數做一些調優。
在sys租戶執行:
ALTER SYSTEM SET enable_merge_by_turn=FALSE;
ALTER SYSTEM set minor_freeze_times=100;
ALTER SYSTEM set freeze_trigger_percentage=70;
ALTER SYSTEM set writing_throttling_trigger_percentage=70 tenant='obbmsql';
ALTER SYSTEM set writing_throttling_maximum_duration='10m' tenant='obbmsql';
show parameters where name in ('minor_freeze_times','freeze_trigger_percentage');
注意:業務租戶限流參數的修改是在sys租戶裡,需要指定相應的租戶名。然後檢視确認需要到業務租戶裡。
在業務租戶執行:
SHOW parameters WHERE name IN ('writing_throttling_trigger_percentage','writing_throttling_maximum_duration');
2. 業務租戶參數修改
OceanBase跟Oracle/MySQL相比,會有個預設SQL逾時和事務逾時機制。這個可能會導緻後面檢視修改資料的SQL報錯。是以先修改一下這些參數。
set global recyclebin=off;
set global ob_query_timeout=1000000000;
set global ob_trx_idle_timeout=1200000000;
set global ob_trx_timeout=1000000000;
此外,需要為bmsql準備一個單獨的schema(即使用者)。
drop user tpcc cascade;
create user tpcc identified by 123456;
grant all privileges on tpcc.* to tpcc with grant option ;
grant create, drop on *.* to tpcc;
至此測試租戶使用者名為:tpcc@obbmsql#obdemo 或 obdemo:obbmsql:tpcc。
3. OBProxy配置修改
OBProxy是OceanBase的通路代理,其内部一些參數也可能影響性能。如下面的壓縮參數對CPU有一定消耗,測試時可以關閉。
$ obclient -h127.1 -uroot@sys#obdemo -P2883 -p123456 -c -A oceanbase
alter proxyconfig set enable_compression_protocol=False;
show proxyconfig like 'enable_compression_protocol';
該參數修改後,需要重新開機obproxy程序
[admin@xxx /home/admin]
$kill -9 `pidof obproxy`
[[email protected] /home/admin]
$cd /opt/taobao/install/obproxy
[admin@xxx /opt/taobao/install/obproxy]
$bin/obproxy
bin/obproxy
BenchmarkSQL準備
BenchmarkSQL 官方下載下傳位址是:
https://sourceforge.net/projects/benchmarksql/,下載下傳後請參考 HOW-TO-RUN.txt 中說明先編譯安裝BenchmarkSQL。然後按下面建議修改部分腳本增加對OceanBase支援。也可以直接下載下傳我編譯修改好的BenchmarkSQL,位址是:
https://github.com/obpilot/benchmarksql-5.0。
1. 準備OceanBase驅動檔案
BenchmarkSQL是通過jdbc連接配接各個資料庫的。此次OceanBase的測試租戶是Oracle類型,是以需要借用 lib/oracle 目錄,然後把相關jar包一并放入其中。其中 oceanbase-client-*.jar 是OceanBase提供的,其他jar包可以從網際網路擷取。
[admin@xxx /home/admin/benchmarksql-5.0]$ll lib/oracle/
total 3728
-rwxr-xr-x 1 admin admin 52988 Jul 12 2019 commons-cli-1.3.1.jar
-rwxr-xr-x 1 admin admin 245274 Jul 12 2019 commons-lang-2.3.jar
-rwxr-xr-x 1 admin admin 2256213 Jul 12 2019 guava-18.0.jar
-rwxr-xr-x 1 admin admin 54495 Jul 12 2019 json-20160810.jar
-rwxr-xr-x 1 admin admin 1121698 Dec 3 15:04 oceanbase-client-1.0.8.jar
-rwxr-xr-x 1 admin admin 174 Jul 12 2019 README.txt
-rwxr-xr-x 1 admin admin 76997 Jul 12 2019 toolkit-common-logging-1.10.jar
2. 準備OB配置檔案
$cat props.ob
db=oracle
driver=com.alipay.oceanbase.obproxy.mysql.jdbc.Driver
conn=jdbc:oceanbase://127.0.0.1:2883/tpcc?useUnicode=true&characterEncoding=utf-8
user=tpcc@obbmsql#obdemo
password=123456
warehouses=10
loadWorkers=10
//fileLocation=/home/t4/tmp
terminals=10
//To run specified transactions per terminal- runMins must equal zero
runTxnsPerTerminal=0
//To run for specified minutes- runTxnsPerTerminal must equal zero
runMins=10
//Number of total transactions per minute
limitTxnsPerMin=0
//Set to true to run in 4.x compatible mode. Set to false to use the
//entire configured database evenly.
terminalWarehouseFixed=true
//The following five values must add up to 100
newOrderWeight=45
paymentWeight=43
orderStatusWeight=4
deliveryWeight=4
stockLevelWeight=4
// Directory name to create for collecting detailed result data.
// Comment this out to suppress.
resultDirectory=my_result_%tY-%tm-%td_%tH%tM%tS
osCollectorScript=./misc/os_collector_linux.py
osCollectorInterval=1
//osCollectorSSHAddr=user@dbhost
//osCollectorDevices=net_eth0 blk_sda
注意:
- 倉庫數(warehouses)決定了資料量。正式的壓測倉庫數一般在10000以上。
- loadworkers數決定了資料加載的性能。如果OceanBase租戶資源很小(尤其是記憶體資源),那加載速度也不要太快;否則容易把租戶記憶體打爆。
- 并發數(terminals)是後期做TPC-C測試的用戶端并發數。這個每次測試都可以調整,以友善觀察不同壓力下的性能。
- 壓測時間(runMin)是每次測試時間,越長測試結果越好且穩定。因為有時候資料通路有個預熱過程,效果會展現在記憶體命中率上。
3. 建立BenchmarkSQL相關表
1)建表腳本
該SQL腳本不需要直接執行。
create table bmsql_config (
cfg_name varchar2(30) primary key,
cfg_value varchar2(50)
);
create tablegroup tpcc_group partition by hash partitions 12;
create table bmsql_warehouse (
w_id integer not null,
w_ytd decimal(12,2),
w_tax decimal(4,4),
w_name varchar2(10),
w_street_1 varchar2(20),
w_street_2 varchar2(20),
w_city varchar2(20),
w_state char(2),
w_zip char(9),
primary key(w_id)
)tablegroup='tpcc_group' partition by hash(w_id) partitions 12;
create table bmsql_district (
d_w_id integer not null,
d_id integer not null,
d_ytd decimal(12,2),
d_tax decimal(4,4),
d_next_o_id integer,
d_name varchar2(10),
d_street_1 varchar2(20),
d_street_2 varchar2(20),
d_city varchar2(20),
d_state char(2),
d_zip char(9),
PRIMARY KEY (d_w_id, d_id)
)tablegroup='tpcc_group' partition by hash(d_w_id) partitions 12;
create table bmsql_customer (
c_w_id integer not null,
c_d_id integer not null,
c_id integer not null,
c_discount decimal(4,4),
c_credit char(2),
c_last varchar2(16),
c_first varchar2(16),
c_credit_lim decimal(12,2),
c_balance decimal(12,2),
c_ytd_payment decimal(12,2),
c_payment_cnt integer,
c_delivery_cnt integer,
c_street_1 varchar2(20),
c_street_2 varchar2(20),
c_city varchar2(20),
c_state char(2),
c_zip char(9),
c_phone char(16),
c_since timestamp,
c_middle char(2),
c_data varchar2(500),
PRIMARY KEY (c_w_id, c_d_id, c_id)
)tablegroup='tpcc_group' use_bloom_filter=true compress partition by hash(c_w_id) partitions 12;
create sequence bmsql_hist_id_seq;
create table bmsql_history (
hist_id integer,
h_c_id integer,
h_c_d_id integer,
h_c_w_id integer,
h_d_id integer,
h_w_id integer,
h_date timestamp,
h_amount decimal(6,2),
h_data varchar2(24)
)tablegroup='tpcc_group' use_bloom_filter=true compress partition by hash(h_w_id) partitions 12;
create table bmsql_new_order (
no_w_id integer not null ,
no_d_id integer not null,
no_o_id integer not null,
PRIMARY KEY (no_w_id, no_d_id, no_o_id)
)tablegroup='tpcc_group' use_bloom_filter=true compress partition by hash(no_w_id) partitions 12;
create table bmsql_oorder (
o_w_id integer not null,
o_d_id integer not null,
o_id integer not null,
o_c_id integer,
o_carrier_id integer,
o_ol_cnt integer,
o_all_local integer,
o_entry_d timestamp,
PRIMARY KEY (o_w_id, o_d_id, o_id)
)tablegroup='tpcc_group' use_bloom_filter=true compress partition by hash(o_w_id) partitions 12;
create table bmsql_order_line (
ol_w_id integer not null,
ol_d_id integer not null,
ol_o_id integer not null,
ol_number integer not null,
ol_i_id integer not null,
ol_delivery_d timestamp,
ol_amount decimal(6,2),
ol_supply_w_id integer,
ol_quantity integer,
ol_dist_info char(24),
PRIMARY KEY (ol_w_id, ol_d_id, ol_o_id, ol_number)
)tablegroup='tpcc_group' use_bloom_filter=true compress partition by hash(ol_w_id) partitions 12;
create table bmsql_item (
i_id integer not null,
i_name varchar2(24),
i_price decimal(5,2),
i_data varchar2(50),
i_im_id integer,
PRIMARY KEY (i_id)
)use_bloom_filter=true compress locality='F,R{all_server}@zone1, F,R{all_server}@zone2, F,R{all_server}@zone3' primary_zone='zone1' duplicate_scope='cluster';
create table bmsql_stock (
s_w_id integer not null,
s_i_id integer not null,
s_quantity integer,
s_ytd integer,
s_order_cnt integer,
s_remote_cnt integer,
s_data varchar2(50),
s_dist_01 char(24),
s_dist_02 char(24),
s_dist_03 char(24),
s_dist_04 char(24),
s_dist_05 char(24),
s_dist_06 char(24),
s_dist_07 char(24),
s_dist_08 char(24),
s_dist_09 char(24),
s_dist_10 char(24),
PRIMARY KEY (s_w_id, s_i_id)
)tablegroup='tpcc_group' use_bloom_filter=true compress partition by hash(s_w_id) partitions 12;
- 建表語句中的分區數目可以根據實際情況調整,跟叢集節點數有關。如果叢集是3台(1-1-1),建議是6個或6的倍數;如果叢集是6台(2-2-2),建議是12個或12的倍數;如果叢集是9台(3-3-3),建議是36個或36的倍數。這樣是友善後期彈性伸縮測試的時候能盡可能保證每個節點上的分區數均衡。
- 上面bmsql_item使用了【複制表】功能,在租戶的所有節點上都會有一個副本。當然主副本始終隻有一個。有關【複制表】功能介紹請參考《OceanBase事務引擎特性和應用實踐分享》。
- 建表語句不包含非主鍵索引,是為了後面加載資料性能更快。
2)建表
./runSQL.sh props.ob ./sql.oceanbase/tableCreates.sql
建表後,可以檢視主副本分布
SELECT t1.tenant_id,t1.tenant_name,t2.database_name,t3.table_id,t3.table_Name,t3.tablegroup_id,t3.part_num,t4.partition_Id,t4.zone,t4.svr_ip,t4.role, round(t4.data_size/1024/1024) data_size_mb
from `gv$tenant` t1
join `gv$database` t2 on (t1.tenant_id = t2.tenant_id)
join gv$table t3 on (t2.tenant_id = t3.tenant_id and t2.database_id = t3.database_id and t3.index_type = 0)
left join `__all_virtual_meta_table` t4 on (t2.tenant_id = t4.tenant_id and ( t3.table_id = t4.table_id or t3.tablegroup_id = t4.table_id ) and t4.role in (1))
where t1.tenant_id = 1001
order by t3.tablegroup_id, t4.partition_Id, t3.table_name ;
4. 加載資料
1)開始加載資料
./runLoader.sh props.ob
2)觀察資料加載性能
為了對資料寫入速度進行觀察,可以在sys租戶下反複執行下面SQL,主要是觀察增量記憶體增速和增量記憶體總量,以及是否接近總增量記憶體限制。
SELECT tenant_id, ip, round(active/1024/1024) active_mb, round(total/1024/1024) total_mb, round(freeze_trigger/1024/1024) freeze_trg_mb, round(mem_limit/1024/1024) mem_limit_mb
, freeze_cnt , round((active/freeze_trigger),2) freeze_pct, round(total/mem_limit, 2) mem_usage
FROM `gv$memstore`
WHERE tenant_id IN (1001)
ORDER BY tenant_id, ip;
當然,觀察資料加載另外一個方法就是使用監控。OCP的監控或者dooba腳本監控。
python dooba.py -h 127.1 -uroot@sys#obdemo -P2883 -p123456
dooba 進去後,預設是sys租戶。按字母小寫'c',選擇業務租戶。按數字'1'檢視幫助,數字'2'檢視租戶總覽,數字'3'檢視租戶的機器性能資訊,按TAB切換目前焦點,按字母小寫'd' 删除目前TAB,按字母大寫R 恢複所有TAB。總覽裡的NET TAB沒有意義可以删除以節省螢幕空間。
5. 建索引
索引很少,就2條。由于相關表是分區表,可以建全局索引或者本地索引。我們建本地索引。
$cat ./sql.oceanbase/indexCreates.sql
create index bmsql_customer_idx1
on bmsql_customer (c_w_id, c_d_id, c_last, c_first) local;
create index bmsql_oorder_idx1
on bmsql_oorder (o_w_id, o_d_id, o_carrier_id, o_id) local;
開始建索引。OceanBase建索引很快就會傳回,索引建構是異步的。
./runSQL.sh props.ob ./sql.oceanbase/indexCreates.sql
6. 資料校驗
檢查一下各個表的資料量
obclient> select /*+ parallel(16) read_consistency(weak) */ count(*) from TPCC.BMSQL_CONFIG;
*+ parallel(16) read_consistency(weak) */ count(*) from TPCC.BMSQL_STOCK;+----------+
| COUNT(*) |
+----------+
| 4 |
+----------+
1 row in set (0.06 sec)
obclient> select /*+ parallel(16) read_consistency(weak) */ count(*) from TPCC.BMSQL_WAREHOUSE;
+----------+
| COUNT(*) |
+----------+
| 10 |
+----------+
1 row in set (0.06 sec)
obclient> select /*+ parallel(16) read_consistency(weak) */ count(*) from TPCC.BMSQL_DISTRICT;
+----------+
| COUNT(*) |
+----------+
| 100 |
+----------+
1 row in set (0.06 sec)
obclient> select /*+ parallel(16) read_consistency(weak) */ count(*) from TPCC.BMSQL_CUSTOMER;
+----------+
| COUNT(*) |
+----------+
| 300000 |
+----------+
1 row in set (0.34 sec)
obclient> select /*+ parallel(16) read_consistency(weak) */ count(*) from TPCC.BMSQL_HISTORY;
+----------+
| COUNT(*) |
+----------+
| 300000 |
+----------+
1 row in set (0.10 sec)
obclient> select /*+ parallel(16) read_consistency(weak) */ count(*) from TPCC.BMSQL_NEW_ORDER;
+----------+
| COUNT(*) |
+----------+
| 90000 |
+----------+
1 row in set (0.07 sec)
obclient> select /*+ parallel(16) read_consistency(weak) */ count(*) from TPCC.BMSQL_OORDER;
+----------+
| COUNT(*) |
+----------+
| 300000 |
+----------+
1 row in set (0.11 sec)
obclient> select /*+ parallel(16) read_consistency(weak) */ count(*) from TPCC.BMSQL_ORDER_LINE;
+----------+
| COUNT(*) |
+----------+
| 3001782 |
+----------+
1 row in set (0.27 sec)
obclient> select /*+ parallel(16) read_consistency(weak) */ count(*) from TPCC.BMSQL_ITEM;
+----------+
| COUNT(*) |
+----------+
| 100000 |
+----------+
1 row in set (0.08 sec)
obclient> select /*+ parallel(16) read_consistency(weak) */ count(*) from TPCC.BMSQL_STOCK;
+----------+
| COUNT(*) |
+----------+
| 1000000 |
+----------+
1 row in set (0.63 sec)
為了避免産生的資料不符合規範(如中間報錯導緻有事務失敗),運作下面校驗腳本
#!/usr/bin/sh
cc1="
SELECT /*+ no_use_px parallel(8) */ * FROM(
SELECT w.w_id, w.w_ytd, d.sum_d_ytd
FROM bmsql_warehouse w,
(SELECT /*+ no_use_px parallel(8) */ d_w_id, sum(d_ytd) sum_d_ytd FROM bmsql_district GROUP BY d_w_id) d
WHERE w.w_id= d.d_w_id
) x
WHERE w_ytd != sum_d_ytd;
"
cc2="
SELECT /*+ no_use_px parallel(8) */ * FROM(
SELECT d.d_w_id, d.d_id, d.d_next_o_id, o.max_o_id, no.max_no_o_id
FROM bmsql_district d,
(SELECT /*+ no_use_px parallel(8) */ o_w_id, o_d_id, MAX(o_id) max_o_id FROM bmsql_oorder GROUP BY o_w_id, o_d_id) o,
(SELECT /*+ no_use_px parallel(8) */ no_w_id, no_d_id, MAX(no_o_id) max_no_o_id FROM bmsql_new_order GROUP BY no_w_id, no_d_id) no
WHERE d.d_w_id= o.o_w_id AND d.d_w_id= no.no_w_id AND d.d_id= o.o_d_id AND d.d_id= no.no_d_id
) x
WHERE d_next_o_id - 1!= max_o_id OR d_next_o_id - 1!= max_no_o_id;
"
cc3="
SELECT /*+ no_use_px paratLel(8) */ * FROM(
SELECT /*+ no_use_px parallel(8) */ no_w_id, no_d_id, MAX(no_o_id) max_no_o_id, MIN(no_o_id) min_no_o_id, COUNT(*) count_no
FROM bmsql_new_order
GROUP BY no_w_id, no_d_Id
) x
WHERE max_no_o_id - min_no_o_id+ 1!= count_no;
"
cc4="
SELECT /*+ no_use_px parallel(8) */ * FROM (
SELECT o.o_w_id, o.o_d_id, o.sum_o_ol_cnt, ol.count_ol
FROM (SELECT /*+ no_use_px parallel(8) */ o_w_id, o_d_id, SUM(o_ol_cnt) sum_o_ol_cnt FROM bmsql_oorder GROUP BY o_w_id, o_d_id) o,
(SELECT /*+ no_use_px parallel(8) */ ol_w_id, ol_d_id, COUNT(*) count_ol FROM bmsql_order_line GROUP BY ol_w_id, ol_d_id) ol
WHERE o.o_w_id = ol.ol_w_id AND o.o_d_id = ol.ol_d_id
) x
WHERE sum_o_ol_cnt != count_ol;
"
cc5="
SELECT /*+ no_use_px parallel(8) */ * FROM (
SELECT o.o_w_id, o.o_d_id, o.o_id, o.o_carrier_id, no.count_no
FROM bmsql_oorder o,
(SELECT /*+ no_use_px parallels) */ no_w_id, no_d_id, no_o_id, COUNT(*) count_no FROM bmsql_new_order GROUP BY no_w_id, no_d_id, no_o_id) no
WHERE o.o_w_id = no.no_w_id AND o.o_d_id = no.no_d_id AND o.o_id = no.no_o_id
) x
WHERE (o_carrier_id IS NULL AND count_no = 0) OR (o_carrier_id IS NOT NULL AND count_no != 0);
"
cc6="
SELECT /*+ no_use_px parallel(8) */ * FROM (
SELECT o.o_w_id, o.o_d_id, o.o_id, o.o_ol_cnt, ol.count_ol
FROM bmsql_oorder o,
(SELECT /*+ no_use_px parallel(8) */ ol_w_id, ol_d_id, ol_o_id, COUNT(*) count_ol FROM bmsql_order_line GROUP BY ol_w_id, ol_d_id, ol_o_id) ol
WHERE o.o_w_id = ol.ol_w_id AND o.o_d_id = ol.ol_d_id AND o.o_id = ol.ol_o_id
) x
WHERE o_ol_cnt != count_ol;
"
cc7="
SELECT /*+ no_use_px parallel(8) */ * FROM (
SELECT /*+ no_use_px parallel(8) */ * FROM (
SELECT o.o_w_id, o.o_d_id, o.o_id, o.o_ol_cnt, ol.count_ol
FROM bmsql_oorder o,
(SELECT /*+ no_use_px parallel(8) */ ol_w_id, ol_d_id, ol_o_id, COUNT(*) count_ol FROM bmsql_order_line GROUP BY ol_w_id, ol_d_id, ol_o_id) ol
WHERE o.o_w_id = ol.ol_w_id AND o.o_d_id = ol.ol_d_id AND o.o_id = ol.ol_o_id
) x
WHERE o_ol_cnt != count_ol;
"
cc7="
SELECT /*+ no_use_px parallel(8) */ * FROM (
SELECT /*+ no_use_px parallel(8) */ ol.ol_w_id, ol.ol_d_id, ol.ol_o_id, ol.ol_delivery_d, o.o_carrier_id
FROM bmsql_order_line ol, bmsql_oorder o
WHERE ol.ol_w_id = o.o_w_id AND
ol.ol_d_id = o.o_d_id AND
ol.ol_o_id = o.o_id
) x
WHERE (ol_delivery_d IS NULL AND o_carrier_id IS NOT NULL) OR
(ol_delivery_d IS NOT NULL AND o_carrier_id IS NULL);
"
cc8="
SELECT /*+ no_use_px parallel(8) */ * FROM (
SELECT w.w_id, w.w_ytd, h.sum_h_amount
FROM bmsql_warehouse w,
(SELECT /*+ no_use_px parallel(8) */ h_w_id, SUM(h_amount) sum_h_amount FROM bmsql_history GROUP BY h_w_id) h
WHERE w.w_id = h.h_w_id) x
WHERE w_ytd != sum_h_amount;
"
cc9="
SELECT /*+ no_use_px parallel(8) */ * FROM (
SELECT d.d_w_id, d.d_id, d.d_ytd, h.sum_h_amount
FROM bmsql_district d,
(SELECT /*+ no_use_px parallel(8) */ h_w_id, h_d_id, SUM(h_amount) sum_h_amount FROM bmsql_history GROUP BY h_w_id, h_d_id) h
WHERE d.d_w_id = h.h_w_id AND d.d_id = h.h_d_id
) x
WHERE d_ytd != sum_h_amount;
"
cc_list="$cc1|$cc2|$cc3|$cc4|$cc5|$cc6|$cc7|$cc8|$cc9"
oldIFS=$IFS
IFS="|"
counter=0
for sql in $cc_list
do
let counter++
echo `date '+%F %X'`" cc$counter start"
obclient -Dtpcc -h127.1 -P2883 -utpcc@obbmsql#obdemo -p123456 -A -c -e "$sql"
#echo $?
if [[ $? -ne 0 ]];then
IFS=$oldIFS
echo `date '+%F %X'`" cc$counter failed"
exit 1
fi
echo `date '+%F %X'`" cc$counter finished"
done
IFS=$oldIFS
BenchmarkSQL TPC-C場景分析
1. E-R模型
2. 場景SQL
TPC-C 系統需要處理的交易有以下五種:
場景的比例是在資料庫配置檔案中定義的。這裡是預設值。對于前四種類型的交易,要求響應時間在 5 秒以内;對于庫存狀況的查詢交易,要求響應時間在 20 秒以内。
這五種交易作用在圖 1 所示的九張表上,事務操作類型包括更新,插入,删除和取消操作。
下面是我事先通過OceanBase的全量SQL審計抓出的TPCC的事務SQL(做過去重,但可能不全)。
1)場景1:New-Order
SELECT d_tax, d_next_o_id FROM bmsql_district WHERE d_w_id = 778 AND d_id = 5 FOR UPDATE;
SELECT c_discount, c_last, c_credit, w_tax FROM bmsql_customer JOIN bmsql_warehouse ON (w_id = c_w_id) WHERE c_w_id = 778 AND c_d_id = 5 AND c_id = 2699;
UPDATE bmsql_district SET d_next_o_id = d_next_o_id + 1 WHERE d_w_id = 778 AND d_id = 5 ;
INSERT INTO bmsql_oorder ( o_id, o_d_id, o_w_id, o_c_id, o_entry_d, o_ol_cnt, o_all_local) VALUES (5686, 5, 778, 2699, timestamp '2020-01-04 13:49:34.137', 8, 1);
INSERT INTO bmsql_new_order ( no_o_id, no_d_id, no_w_id) VALUES (5686, 5, 778);
SELECT i_price, i_name, i_data FROM bmsql_item WHERE i_id = 7752 ; -- 循環8次
SELECT s_quantity, s_data, s_dist_01, s_dist_02, s_dist_03, s_dist_04, s_dist_05, s_dist_06, s_dist_07, s_dist_08, s_dist_09, s_dist_10 FROM bmsql_stock WHERE s_w_id = 778 AND s_i_id = 7752 FOR UPDATE; -- 循環8次
SHOW VARIABLES WHERE Variable_name = 'tx_read_only';
UPDATE bmsql_stock SET s_quantity = 47, s_ytd = s_ytd + 8, s_order_cnt = s_order_cnt + 1, s_remote_cnt = s_remote_cnt + 0 WHERE s_w_id = 778 AND s_i_id = 7752; -- 循環8次
SHOW VARIABLES WHERE Variable_name = 'tx_read_only';
INSERT INTO bmsql_order_line ( ol_o_id, ol_d_id, ol_w_id, ol_number, ol_i_id, ol_supply_w_id, ol_quantity, ol_amount, ol_dist_info) VALUES (5686, 5, 778, 1, 7752, 778, 8, 589.36, 'lYvcNHkOvt3iNoBb5W29umGO'); -- 循環8次
COMMIT;
2)場景2:New-Order
SELECT c_id FROM bmsql_customer WHERE c_w_id = 778 AND c_d_id = 2 AND c_last = 'PRICALLYPRES' ORDER BY c_first;
SELECT c_first, c_middle, c_last, c_balance FROM bmsql_customer WHERE c_w_id = 778 AND c_d_id = 2 AND c_id = 2694;
SELECT o_id, o_entry_d, o_carrier_id FROM bmsql_oorder WHERE o_w_id = 778 AND o_d_id = 2 AND o_c_id = 2694 AND o_id = ( SELECT max(o_id) FROM bmsql_oorder WHERE o_w_id = 778 AND o_d_id = 2 AND o_c_id = 2694 );
SELECT ol_i_id, ol_supply_w_id, ol_quantity, ol_amount, ol_delivery_d FROM bmsql_order_line WHERE ol_w_id = 778 AND ol_d_id = 2 AND ol_o_id = 4494 ORDER BY ol_w_id, ol_d_id, ol_o_id, ol_number;
ROLLBACK;
3)場景3:Payment
UPDATE bmsql_district SET d_ytd = d_ytd + 4806.11 WHERE d_w_id = 778 AND d_id = 10;
SELECT d_name, d_street_1, d_street_2, d_city, d_state, d_zip FROM bmsql_district WHERE d_w_id = 778 AND d_id = 10;
UPDATE bmsql_warehouse SET w_ytd = w_ytd + 4806.11 WHERE w_id = 778;
SELECT w_name, w_street_1, w_street_2, w_city, w_state, w_zip FROM bmsql_warehouse WHERE w_id = 778 ;
SELECT c_id FROM bmsql_customer WHERE c_w_id = 778 AND c_d_id = 10 AND c_last = 'ESEBAROUGHT' ORDER BY c_first;
SELECT c_first, c_middle, c_last, c_street_1, c_street_2, c_city, c_state, c_zip, c_phone, c_since, c_credit, c_credit_lim, c_discount, c_balance FROM bmsql_customer WHERE c_w_id = 778 AND c_d_id = 10 AND c_id = 502 FOR UPDATE;
UPDATE bmsql_customer SET c_balance = c_balance - 4806.11, c_ytd_payment = c_ytd_payment + 4806.11, c_payment_cnt = c_payment_cnt + 1 WHERE c_w_id = 778 AND c_d_id = 10 AND c_id = 502;
INSERT INTO bmsql_history ( h_c_id, h_c_d_id, h_c_w_id, h_d_id, h_w_id, h_date, h_amount, h_data) VALUES (502, 10, 778, 10, 778, timestamp '2020-01-04 13:49:34.148', 4806.11, 'HfYovpM6 b6aJtf2Xk6');
COMMIT;
4)場景4:
SELECT count(*) AS low_stock FROM ( SELECT s_w_id, s_i_id, s_quantity FROM bmsql_stock WHERE s_w_id = 778 AND s_quantity < 10 AND s_i_id IN ( SELECT ol_i_id FROM bmsql_district JOIN bmsql_order_line ON ol_w_id = d_w_id AND ol_d_id = d_id AND ol_o_id >= d_next_o_id - 20 AND ol_o_id < d_next_o_id WHERE d_w_id = 778 AND d_id = 1 ) );
ROLLBACK;
5)場景5:
SELECT no_o_id FROM bmsql_new_order WHERE no_w_id = 778 AND no_d_id = 1 ORDER BY no_o_id ASC;
DELETE FROM bmsql_new_order WHERE no_w_id = 778 AND no_d_id = 1 AND no_o_id = 4488;
UPDATE bmsql_oorder SET o_carrier_id = 2 WHERE o_w_id = 778 AND o_d_id = 1 AND o_id = 4488;
SELECT o_c_id FROM bmsql_oorder WHERE o_w_id = 778 AND o_d_id = 1 AND o_id = 4488;
UPDATE bmsql_order_line SET ol_delivery_d = timestamp '2020-01-04 13:49:34.181' WHERE ol_w_id = 778 AND ol_d_id = 1 AND ol_o_id = 4488;
SELECT sum(ol_amount) AS sum_ol_amount FROM bmsql_order_line WHERE ol_w_id = 778 AND ol_d_id = 1 AND ol_o_id = 4488;
UPDATE bmsql_customer SET c_balance = c_balance + 3733.14, c_delivery_cnt = c_delivery_cnt + 1 WHERE c_w_id = 778 AND c_d_id = 1 AND c_id = 1260;
<---循環8次--->
commit
注意:可能還有事務SQL沒有找到。
3. TPC-C輸出名額
TPC-C 的測試結果主要有兩個名額:
- 流量名額(tpmC):描述了系統在執行 Payment,Order-Status,Delivery,Stock-level 這四種交易的同時,每分鐘可以處理的 New-Order交易的數量。流量名額值越大越好。tpm 是 transactions per minute 的簡稱;C 指 TPC 中的 C 基準程式。它的定義是每分鐘内系統處理的新訂單個數。要注意的是,在處理新訂單的同時,系統還要按圖 1 的要求處理其 它 4 類事務 請求。從圖 1 可以看出,新訂單請求不可能超出全部事務請求的 45%,是以,當一個系統的性能為 1000tpmC 時,它每分鐘實際處理的請求數是 2000 多個。
- 成本效益(Price/tpmC):測試系統價格與流量名額的比值。成本效益越小越好。
運作BenchmarkSQL TPC-C測試
1. OceanBase記憶體當機與合并
前面加載了大量資料,OceanBase的增量都在記憶體中,需要做一次major freeze以釋放增量記憶體。這個事件分兩步。一是當機操作,這個很快。二是合并操作,這個跟增量資料量有關,通常要幾分鐘或者幾十分鐘。
每次重複測試的時候都建議做一次major freeze事件以釋放記憶體,弊端就是随後測試中記憶體資料通路又需要一個預熱過程。
1)觀察記憶體增量使用情況
select tenant_id, ip, round(active/1024/1024) active_mb, round(total/1024/1024) total_mb, round(freeze_trigger/1024/1024) freeze_trg_mb, round(mem_limit/1024/1024) mem_limit_mb, freeze_cnt, round(total/mem_limit,2) total_pct
from `gv$memstore` where tenant_id>1001 order by tenant_id;
2)發起記憶體major freeze事件
ALTER SYSTEM major freeze;
3) 觀察合并進度
觀察合并事件
SELECT DATE_FORMAT(gmt_create, '%b%d %H:%i:%s') gmt_create_ , module, event, name1, value1, name2, value2, rs_svr_ip
FROM __all_rootservice_event_history
WHERE 1 = 1 AND module IN ('daily_merge')
ORDER BY gmt_create DESC
LIMIT 100;
觀察合并進度
select ZONE,svr_ip,major_version,ss_store_count ss_sc, merged_ss_store_count merged_ss_sc, modified_ss_store_count modified_ss_sc, date_format(merge_start_time, "%h:%i:%s") merge_st, date_format(merge_finish_time,"%h:%i:%s") merge_ft, merge_process
from `__all_virtual_partition_sstable_image_info` s
order by major_version, zone, svr_ip ;
2. 跑TPC-C測試
1)運作測試程式
$./runBenchmark.sh props.ob
2)性能監控
這個監控界面重點關注QPS/TPS、以及相應的RT、增量記憶體的增量和總量占比等。此外還能看出測試過程中還是有不少實體讀IO。
這個監控界面裡的重點看各個節點的QPS和TPS分布,以及遠端SQL的數量占總QPS的比例(SRC/SLC)。TPC-C業務定義會有約1%的遠端倉庫交易事務,在OceanBase裡這個交易又有一定機率是分布式事務。
3)TPC-C報告
運作結束後會生成結果。
從圖上看,tpmC結果是48204。這個業務租戶總資源是20C25G*3。倉庫數隻有10倉太少了,如果機器好一點,并做10000倉,這個結果應該會更高。
運作同時還生成了一個檔案夾
$ll my_result_2020-01-13_175531/
total 16
drwxrwxr-x 2 admin admin 4096 Jan 13 17:55 data
-rw-rw-r-- 1 admin admin 5130 Jan 13 18:10 report.html
-rwxr-xr-x 1 admin admin 1050 Jan 13 17:55 run.properties
以上就是通過BenchmarkSQL跑TPC-C測試程式的完整過程,感興趣的同學也可以按照上述步驟體驗。有更多回報歡迎在文章評論區留言。
立即申請免費體驗OceanBase 2.2版本
「 OceanBase 2.2 版本 」正式上線官網啦!OceanBase 2.2版本是成功支撐2019年天貓雙11大促的穩定版本,同時也是用于TPC-C測試且榮登TPC-C性能榜首的版本。OceanBase 2.2版本除了在螞蟻金服和網商銀行廣泛使用外,目前也在部分金融機構中使用。
想要立即體驗「OceanBase 2.2版本」?
免費擷取連結:
如果你在安裝和使用的過程中遇到問題且希望跟OceanBase一線專家進行技術交流,我們為開發者使用者提供了兩種管道:
加入OceanBase技術交流釘釘群,打開釘釘搜尋群号:21949783(備注:OB 2.2) 即可加入
我們非常重視來自每一位開發者使用者的體驗和心得,希望能夠獲得你們的寶貴回報。