寫在前面：最近因為手頭上的事情比較多，關于模型融合并沒有太深入的研究，這裡僅介紹下模型融合大緻的内容。
1 模型融合目标
- 對于多種調參完成的模型進行模型融合。
- 完成對于多種模型的融合，送出融合結果并打卡。
2 内容介紹
2.1 stacking（參考連結:https://blog.csdn.net/maqunfi/article/details/82220115）
Stacking的思想是一種有層次的融合模型，比如我們将用不同特征訓練出來的三個GBDT模型進行融合時，我們會将三個GBDT作為基層模型，在其上在訓練一個次學習器（通常為線性模型LR）,用于組織利用基學習器的答案，也就是将基層模型的答案作為輸入，讓次學習器學習組織給基層模型的答案配置設定權重。
假設我們有三個基模型M1,M2,M3和一個元模型M4, 有訓練集train和測試集test，我們進行下面的操作(為了便于了解，我這裡還給出了代碼的形式)：
用訓練集train訓練基模型M1(M1.fit(train))， 然後分别在train和test上做預測， 得到P1(M1.predict(train))和T1(M1.predict(test)
用訓練集train訓練基模型M2(M2.fit(train))， 然後分别在train和test上做預測， 得到P2(M2.predict(train))和T2(M2.predict(test)
用訓練集train訓練基模型M3(M3.fit(train))， 然後分别在train和test上做預測， 得到P3(M3.predict(train))和T3(M3.predict(test)
Stacking本質上就是這麼直接的思路，但是直接這樣有時對于如果訓練集和測試集分布不那麼一緻的情況下是有一點問題的，其問題在于用初始模型訓練的标簽再利用真實标簽進行再訓練，毫無疑問會導緻一定的模型過拟合訓練集，這樣或許模型在測試集上的泛化能力或者說效果會有一定的下降，是以現在的問題變成了如何降低再訓練的過拟合性，這裡我們一般有兩種方法。
- 次級模型盡量選擇簡單的線性模型
- 利用K折交叉驗證
2.2 代碼示例
（1）簡單權重平均，結果直接融合
## 生成一些簡單的樣本資料，test_prei 代表第i個模型的預測值
test_pre1 = [1.2, 3.2, 2.1, 6.2]
test_pre2 = [0.9, 3.1, 2.0, 5.9]
test_pre3 = [1.1, 2.9, 2.2, 6.0]
# y_test_true 代表第模型的真實值
y_test_true = [1, 3, 2, 6]
import numpy as np
import pandas as pd
## 定義結果的權重平均函數
def Weighted_method(test_pre1,test_pre2,test_pre3,w=[1/3,1/3,1/3]):
Weighted_result = w[0]*pd.Series(test_pre1)+w[1]*pd.Series(test_pre2)+w[2]*pd.Series(test_pre3)
return Weighted_result
from sklearn import metrics
# 各模型的預測結果計算MAE
print('Pred1 MAE:',metrics.mean_absolute_error(y_test_true, test_pre1))
print('Pred2 MAE:',metrics.mean_absolute_error(y_test_true, test_pre2))
print('Pred3 MAE:',metrics.mean_absolute_error(y_test_true, test_pre3))
預測的MAE結果：
Pred1 MAE: 0.175
Pred2 MAE: 0.075
Pred3 MAE: 0.1
如果帶有權重計算：
## 根據權重計算MAE
w = [0.3,0.4,0.3] # 定義比重權值
Weighted_pre = Weighted_method(test_pre1,test_pre2,test_pre3,w)
print('Weighted_pre MAE:',metrics.mean_absolute_error(y_test_true, Weighted_pre))
Weighted_pre MAE: 0.0575
（2） stacking融合(回歸)：
from sklearn import linear_model
def Stacking_method(train_reg1,train_reg2,train_reg3,y_train_true,test_pre1,test_pre2,test_pre3,model_L2= linear_model.LinearRegression()):
model_L2.fit(pd.concat([pd.Series(train_reg1),pd.Series(train_reg2),pd.Series(train_reg3)],axis=1).values,y_train_true)
Stacking_result = model_L2.predict(pd.concat([pd.Series(test_pre1),pd.Series(test_pre2),pd.Series(test_pre3)],axis=1).values)
return Stacking_result
## 生成一些簡單的樣本資料，test_prei 代表第i個模型的預測值
train_reg1 = [3.2, 8.2, 9.1, 5.2]
train_reg2 = [2.9, 8.1, 9.0, 4.9]
train_reg3 = [3.1, 7.9, 9.2, 5.0]
# y_test_true 代表第模型的真實值
y_train_true = [3, 8, 9, 5]
test_pre1 = [1.2, 3.2, 2.1, 6.2]
test_pre2 = [0.9, 3.1, 2.0, 5.9]
test_pre3 = [1.1, 2.9, 2.2, 6.0]
# y_test_true 代表第模型的真實值
y_test_true = [1, 3, 2, 6]
model_L2= linear_model.LinearRegression()
Stacking_pre = Stacking_method(train_reg1,train_reg2,train_reg3,y_train_true,
test_pre1,test_pre2,test_pre3,model_L2)
print('Stacking_pre MAE:',metrics.mean_absolute_error(y_test_true, Stacking_pre))
Stacking_pre MAE: 0.0421348314607
可以發現模型結果相對于之前有進一步的提升，這是我們需要注意的一點是，對于第二層Stacking的模型不宜選取的過于複雜，這樣會導緻模型在訓練集上過拟合，進而使得在測試集上并不能達到很好的效果。
2.2 分類模型融合：
對于分類，同樣可以使用融合方法，比如簡單投票，Stacking...
1）Voting投票機制：
Voting即投票機制，分為軟投票和硬投票兩種，其原理采用少數服從多數的思想。
from sklearn.datasets import make_blobs
from sklearn import datasets
from sklearn.tree import DecisionTreeClassifier
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import VotingClassifier
from xgboost import XGBClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_moons
from sklearn.metrics import accuracy_score,roc_auc_score
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import StratifiedKFold
'''
硬投票：對多個模型直接進行投票，不區分模型結果的相對重要度，最終投票數最多的類為最終被預測的類。
'''
iris = datasets.load_iris()
x=iris.data
y=iris.target
x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.3)
clf1 = XGBClassifier(learning_rate=0.1, n_estimators=150, max_depth=3, min_child_weight=2, subsample=0.7,
colsample_bytree=0.6, objective='binary:logistic')
clf2 = RandomForestClassifier(n_estimators=50, max_depth=1, min_samples_split=4,
min_samples_leaf=63,oob_score=True)
clf3 = SVC(C=0.1)
# 硬投票
eclf = VotingClassifier(estimators=[('xgb', clf1), ('rf', clf2), ('svc', clf3)], voting='hard')
for clf, label in zip([clf1, clf2, clf3, eclf], ['XGBBoosting', 'Random Forest', 'SVM', 'Ensemble']):
scores = cross_val_score(clf, x, y, cv=5, scoring='accuracy')
print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))
硬投票的結果：
Accuracy: 0.97 (+/- 0.02) [XGBBoosting]
Accuracy: 0.33 (+/- 0.00) [Random Forest]
Accuracy: 0.95 (+/- 0.03) [SVM]
Accuracy: 0.94 (+/- 0.04) [Ensemble]
'''
軟投票：和硬投票原理相同，增加了設定權重的功能，可以為不同模型設定不同權重，進而差別模型不同的重要度。
'''
x=iris.data
y=iris.target
x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.3)
clf1 = XGBClassifier(learning_rate=0.1, n_estimators=150, max_depth=3, min_child_weight=2, subsample=0.8,
colsample_bytree=0.8, objective='binary:logistic')
clf2 = RandomForestClassifier(n_estimators=50, max_depth=1, min_samples_split=4,
min_samples_leaf=63,oob_score=True)
clf3 = SVC(C=0.1, probability=True)
# 軟投票
eclf = VotingClassifier(estimators=[('xgb', clf1), ('rf', clf2), ('svc', clf3)], voting='soft', weights=[2, 1, 1])
clf1.fit(x_train, y_train)
for clf, label in zip([clf1, clf2, clf3, eclf], ['XGBBoosting', 'Random Forest', 'SVM', 'Ensemble']):
scores = cross_val_score(clf, x, y, cv=5, scoring='accuracy')
print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))
軟投票的結果為：
Accuracy: 0.96 (+/- 0.02) [XGBBoosting]
Accuracy: 0.33 (+/- 0.00) [Random Forest]
Accuracy: 0.95 (+/- 0.03) [SVM]
Accuracy: 0.96 (+/- 0.02) [Ensemble]
2）分類的Stacking\Blending融合：
stacking是一種分層模型內建架構。
以兩層為例，第一層由多個基學習器組成，其輸入為原始訓練集，第二層的模型則是以第一層基學習器的輸出作為訓練集進行再訓練，進而得到完整的stacking模型, stacking兩層模型都使用了全部的訓練資料。''' 5-Fold Stacking ''' from sklearn.ensemble import RandomForestClassifier from sklearn.ensemble import ExtraTreesClassifier,GradientBoostingClassifier import pandas as pd #建立訓練的資料集 data_0 = iris.data data = data_0[:100,:] target_0 = iris.target target = target_0[:100] #模型融合中使用到的各個單模型 clfs = [LogisticRegression(solver='lbfgs'), RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'), ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='gini'), ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'), GradientBoostingClassifier(learning_rate=0.05, subsample=0.5, max_depth=6, n_estimators=5)] #切分一部分資料作為測試集 X, X_predict, y, y_predict = train_test_split(data, target, test_size=0.3, random_state=2020) dataset_blend_train = np.zeros((X.shape[0], len(clfs))) dataset_blend_test = np.zeros((X_predict.shape[0], len(clfs))) #5折stacking n_splits = 5 skf = StratifiedKFold(n_splits) skf = skf.split(X, y) for j, clf in enumerate(clfs): #依次訓練各個單模型 dataset_blend_test_j = np.zeros((X_predict.shape[0], 5)) for i, (train, test) in enumerate(skf): #5-Fold交叉訓練，使用第i個部分作為預測，剩餘的部分來訓練模型，獲得其預測的輸出作為第i部分的新特征。 X_train, y_train, X_test, y_test = X[train], y[train], X[test], y[test] clf.fit(X_train, y_train) y_submission = clf.predict_proba(X_test)[:, 1] dataset_blend_train[test, j] = y_submission dataset_blend_test_j[:, i] = clf.predict_proba(X_predict)[:, 1] #對于測試集，直接用這k個模型的預測值均值作為新的特征。 dataset_blend_test[:, j] = dataset_blend_test_j.mean(1) print("val auc Score: %f" % roc_auc_score(y_predict, dataset_blend_test[:, j])) clf = LogisticRegression(solver='lbfgs') clf.fit(dataset_blend_train, y) y_submission = clf.predict_proba(dataset_blend_test)[:, 1] print("Val auc Score of Stacking: %f" % (roc_auc_score(y_predict, y_submission)))
預測結果為：
val auc Score: 1.000000
val auc Score: 0.500000
val auc Score: 0.500000
val auc Score: 0.500000
val auc Score: 0.500000
Val auc Score of Stacking: 1.000000
Blending，其實和Stacking是一種類似的多層模型融合的形式
其主要思路是把原始的訓練集先分成兩部分，比如70%的資料作為新的訓練集，剩下30%的資料作為測試集。
在第一層，我們在這70%的資料上訓練多個模型，然後去預測那30%資料的label，同時也預測test集的label。
在第二層，我們就直接用這30%資料在第一層預測的結果做為新特征繼續訓練，然後用test集第一層預測的label做特征，用第二層訓練的模型做進一步預測
其優點在于：
- 1.比stacking簡單（因為不用進行k次的交叉驗證來獲得stacker feature）
- 2.避開了一個資訊洩露問題：generlizers和stacker使用了不一樣的資料集
缺點在于：
- 1.使用了很少的資料（第二階段的blender隻使用training set10%的量）
- 2.blender可能會過拟合
- 3.stacking使用多次的交叉驗證會比較穩健 '''
'''
Blending
'''
#建立訓練的資料集
#建立訓練的資料集
data_0 = iris.data
data = data_0[:100,:]
target_0 = iris.target
target = target_0[:100]
#模型融合中使用到的各個單模型
clfs = [LogisticRegression(solver='lbfgs'),
RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),
RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),
ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),
#ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),
GradientBoostingClassifier(learning_rate=0.05, subsample=0.5, max_depth=6, n_estimators=5)]
#切分一部分資料作為測試集
X, X_predict, y, y_predict = train_test_split(data, target, test_size=0.3, random_state=2020)
#切分訓練資料集為d1,d2兩部分
X_d1, X_d2, y_d1, y_d2 = train_test_split(X, y, test_size=0.5, random_state=2020)
dataset_d1 = np.zeros((X_d2.shape[0], len(clfs)))
dataset_d2 = np.zeros((X_predict.shape[0], len(clfs)))
for j, clf in enumerate(clfs):
#依次訓練各個單模型
clf.fit(X_d1, y_d1)
y_submission = clf.predict_proba(X_d2)[:, 1]
dataset_d1[:, j] = y_submission
#對于測試集，直接用這k個模型的預測值作為新的特征。
dataset_d2[:, j] = clf.predict_proba(X_predict)[:, 1]
print("val auc Score: %f" % roc_auc_score(y_predict, dataset_d2[:, j]))
#融合使用的模型
clf = GradientBoostingClassifier(learning_rate=0.02, subsample=0.5, max_depth=6, n_estimators=30)
clf.fit(dataset_d1, y_d2)
y_submission = clf.predict_proba(dataset_d2)[:, 1]
print("Val auc Score of Blending: %f" % (roc_auc_score(y_predict, y_submission)))
準确率得分：
val auc Score: 1.000000
val auc Score: 1.000000
val auc Score: 1.000000
val auc Score: 1.000000
val auc Score: 1.000000
Val auc Score of Blending: 1.000000
3）分類的Stacking融合(利用mlxtend)：
!pip install mlxtend
import warnings
warnings.filterwarnings('ignore')
import itertools
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
from sklearn import datasets
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier
from mlxtend.classifier import StackingClassifier
from sklearn.model_selection import cross_val_score
from mlxtend.plotting import plot_learning_curves
from mlxtend.plotting import plot_decision_regions
# 以python自帶的鸢尾花資料集為例
iris = datasets.load_iris()
X, y = iris.data[:, 1:3], iris.target
clf1 = KNeighborsClassifier(n_neighbors=1)
clf2 = RandomForestClassifier(random_state=1)
clf3 = GaussianNB()
lr = LogisticRegression()
sclf = StackingClassifier(classifiers=[clf1, clf2, clf3],
meta_classifier=lr)
label = ['KNN', 'Random Forest', 'Naive Bayes', 'Stacking Classifier']
clf_list = [clf1, clf2, clf3, sclf]
fig = plt.figure(figsize=(10,8))
gs = gridspec.GridSpec(2, 2)
grid = itertools.product([0,1],repeat=2)
clf_cv_mean = []
clf_cv_std = []
for clf, label, grd in zip(clf_list, label, grid):
scores = cross_val_score(clf, X, y, cv=3, scoring='accuracy')
print("Accuracy: %.2f (+/- %.2f) [%s]" %(scores.mean(), scores.std(), label))
clf_cv_mean.append(scores.mean())
clf_cv_std.append(scores.std())
clf.fit(X, y)
ax = plt.subplot(gs[grd[0], grd[1]])
fig = plot_decision_regions(X=X, y=y, clf=clf)
plt.title(label)
plt.show()
可以發現 基模型 用 'KNN', 'Random Forest', 'Naive Bayes' 然後再這基礎上 次級模型加一個 'LogisticRegression'，模型測試效果有着很好的提升。
3 本賽題示例
（1）EDA分析+特征工程
import pandas as pd
import numpy as np
import warnings
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
warnings.filterwarnings('ignore')
%matplotlib inline
import itertools
import matplotlib.gridspec as gridspec
from sklearn import datasets
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier
# from mlxtend.classifier import StackingClassifier
from sklearn.model_selection import cross_val_score, train_test_split
# from mlxtend.plotting import plot_learning_curves
# from mlxtend.plotting import plot_decision_regions
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import train_test_split
from sklearn import linear_model
from sklearn import preprocessing
from sklearn.svm import SVR
from sklearn.decomposition import PCA,FastICA,FactorAnalysis,SparsePCA
import lightgbm as lgb
import xgboost as xgb
from sklearn.model_selection import GridSearchCV,cross_val_score
from sklearn.ensemble import RandomForestRegressor,GradientBoostingRegressor
from sklearn.metrics import mean_squared_error, mean_absolute_error
## 資料讀取
Train_data = pd.read_csv('datalab/231784/used_car_train_20200313.csv', sep=' ')
TestA_data = pd.read_csv('datalab/231784/used_car_testA_20200313.csv', sep=' ')
print(Train_data.shape)
print(TestA_data.shape)
numerical_cols = Train_data.select_dtypes(exclude = 'object').columns
print(numerical_cols)
feature_cols = [col for col in numerical_cols if col not in ['SaleID','name','regDate','price']]
X_data = Train_data[feature_cols]
Y_data = Train_data['price']
X_test = TestA_data[feature_cols]
print('X train shape:',X_data.shape)
print('X test shape:',X_test.shape)
def Sta_inf(data):
print('_min',np.min(data))
print('_max:',np.max(data))
print('_mean',np.mean(data))
print('_ptp',np.ptp(data))
print('_std',np.std(data))
print('_var',np.var(data))
print('Sta of label:')
Sta_inf(Y_data)
X_data = X_data.fillna(-1)
X_test = X_test.fillna(-1)
def build_model_lr(x_train,y_train):
reg_model = linear_model.LinearRegression()
reg_model.fit(x_train,y_train)
return reg_model
def build_model_ridge(x_train,y_train):
reg_model = linear_model.Ridge(alpha=0.8)#alphas=range(1,100,5)
reg_model.fit(x_train,y_train)
return reg_model
def build_model_lasso(x_train,y_train):
reg_model = linear_model.LassoCV()
reg_model.fit(x_train,y_train)
return reg_model
def build_model_gbdt(x_train,y_train):
estimator =GradientBoostingRegressor(loss='ls',subsample= 0.85,max_depth= 5,n_estimators = 100)
param_grid = {
'learning_rate': [0.05,0.08,0.1,0.2],
}
gbdt = GridSearchCV(estimator, param_grid,cv=3)
gbdt.fit(x_train,y_train)
print(gbdt.best_params_)
# print(gbdt.best_estimator_ )
return gbdt
def build_model_xgb(x_train,y_train):
model = xgb.XGBRegressor(n_estimators=120, learning_rate=0.08, gamma=0, subsample=0.8,\
colsample_bytree=0.9, max_depth=5) #, objective ='reg:squarederror'
model.fit(x_train, y_train)
return model
def build_model_lgb(x_train,y_train):
estimator = lgb.LGBMRegressor(num_leaves=63,n_estimators = 100)
param_grid = {
'learning_rate': [0.01, 0.05, 0.1],
}
gbm = GridSearchCV(estimator, param_grid)
gbm.fit(x_train, y_train)
return gbm
（2）XGBoost的五折交叉回歸驗證實作
## xgb
xgr = xgb.XGBRegressor(n_estimators=120, learning_rate=0.1, subsample=0.8,\
colsample_bytree=0.9, max_depth=7) # ,objective ='reg:squarederror'
scores_train = []
scores = []
## 5折交叉驗證方式
sk=StratifiedKFold(n_splits=5,shuffle=True,random_state=0)
for train_ind,val_ind in sk.split(X_data,Y_data):
train_x=X_data.iloc[train_ind].values
train_y=Y_data.iloc[train_ind]
val_x=X_data.iloc[val_ind].values
val_y=Y_data.iloc[val_ind]
xgr.fit(train_x,train_y)
pred_train_xgb=xgr.predict(train_x)
pred_xgb=xgr.predict(val_x)
score_train = mean_absolute_error(train_y,pred_train_xgb)
scores_train.append(score_train)
score = mean_absolute_error(val_y,pred_xgb)
scores.append(score)
print('Train mae:',np.mean(score_train))
print('Val mae',np.mean(scores))
訓練接和驗證集的結果為：
Train mae: 558.212360169
Val mae 693.120168439
（3）劃分資料集，并用多種方法訓練和預測
## Split data with val
x_train,x_val,y_train,y_val = train_test_split(X_data,Y_data,test_size=0.3)
## Train and Predict
print('Predict LR...')
model_lr = build_model_lr(x_train,y_train)
val_lr = model_lr.predict(x_val)
subA_lr = model_lr.predict(X_test)
print('Predict Ridge...')
model_ridge = build_model_ridge(x_train,y_train)
val_ridge = model_ridge.predict(x_val)
subA_ridge = model_ridge.predict(X_test)
print('Predict Lasso...')
model_lasso = build_model_lasso(x_train,y_train)
val_lasso = model_lasso.predict(x_val)
subA_lasso = model_lasso.predict(X_test)
print('Predict GBDT...')
model_gbdt = build_model_gbdt(x_train,y_train)
val_gbdt = model_gbdt.predict(x_val)
subA_gbdt = model_gbdt.predict(X_test)
4 一般比賽中效果最為顯著的兩種方法
print('predict XGB...')
model_xgb = build_model_xgb(x_train,y_train)
val_xgb = model_xgb.predict(x_val)
subA_xgb = model_xgb.predict(X_test)
print('predict lgb...')
model_lgb = build_model_lgb(x_train,y_train)
val_lgb = model_lgb.predict(x_val)
subA_lgb = model_lgb.predict(X_test)
（1）權重融合
def Weighted_method(test_pre1,test_pre2,test_pre3,w=[1/3,1/3,1/3]):
Weighted_result = w[0]*pd.Series(test_pre1)+w[1]*pd.Series(test_pre2)+w[2]*pd.Series(test_pre3)
return Weighted_result
## Init the Weight
w = [0.3,0.4,0.3]
## 測試驗證集準确度
val_pre = Weighted_method(val_lgb,val_xgb,val_gbdt,w)
MAE_Weighted = mean_absolute_error(y_val,val_pre)
print('MAE of Weighted of val:',MAE_Weighted)
## 預測資料部分
subA = Weighted_method(subA_lgb,subA_xgb,subA_gbdt,w)
print('Sta inf:')
Sta_inf(subA)
## 生成送出檔案
sub = pd.DataFrame()
sub['SaleID'] = X_test.index
sub['price'] = subA
sub.to_csv('./sub_Weighted.csv',index=False)
## 與簡單的LR（線性回歸）進行對比
val_lr_pred = model_lr.predict(x_val)
MAE_lr = mean_absolute_error(y_val,val_lr_pred)
print('MAE of lr:',MAE_lr)
MAE結果為：
MAE of lr: 2597.45638384
（2） Stacking融合
## Starking
## 第一層
train_lgb_pred = model_lgb.predict(x_train)
train_xgb_pred = model_xgb.predict(x_train)
train_gbdt_pred = model_gbdt.predict(x_train)
Strak_X_train = pd.DataFrame()
Strak_X_train['Method_1'] = train_lgb_pred
Strak_X_train['Method_2'] = train_xgb_pred
Strak_X_train['Method_3'] = train_gbdt_pred
Strak_X_val = pd.DataFrame()
Strak_X_val['Method_1'] = val_lgb
Strak_X_val['Method_2'] = val_xgb
Strak_X_val['Method_3'] = val_gbdt
Strak_X_test = pd.DataFrame()
Strak_X_test['Method_1'] = subA_lgb
Strak_X_test['Method_2'] = subA_xgb
Strak_X_test['Method_3'] = subA_gbdt
## level2-method
model_lr_Stacking = build_model_lr(Strak_X_train,y_train)
## 訓練集
train_pre_Stacking = model_lr_Stacking.predict(Strak_X_train)
print('MAE of Stacking-LR:',mean_absolute_error(y_train,train_pre_Stacking))
## 驗證集
val_pre_Stacking = model_lr_Stacking.predict(Strak_X_val)
print('MAE of Stacking-LR:',mean_absolute_error(y_val,val_pre_Stacking))
## 預測集
print('Predict Stacking-LR...')
subA_Stacking = model_lr_Stacking.predict(Strak_X_test)
subA_Stacking[subA_Stacking<10]=10 ## 去除過小的預測值
sub = pd.DataFrame()
sub['SaleID'] = X_test.index
sub['price'] = subA_Stacking
sub.to_csv('./sub_Stacking.csv',index=False)
MAE結果為：
MAE of Stacking-LR: 628.399441036
MAE of Stacking-LR: 707.673951794
5 經驗總結
比賽的融合這個問題，個人的看法來說其實涉及多個層面，也是提分和提升模型魯棒性的一種重要方法：
1）結果層面的融合，這種是最常見的融合方法，其可行的融合方法也有很多，比如根據結果的得分進行權重融合，還可以做Log，exp處理等。在做結果融合的時候，有一個很重要的條件是模型結果的得分要比較近似，然後結果的差異要比較大，這樣的結果融合往往有比較好的效果提升。
2）特征層面的融合，這個層面其實感覺不叫融合，準确說可以叫分割，很多時候如果我們用同種模型訓練，可以把特征進行切分給不同的模型，然後在後面進行模型或者結果融合有時也能産生比較好的效果。
3）模型層面的融合，模型層面的融合可能就涉及模型的堆疊和設計，比如加Staking層，部分模型的結果作為特征輸入等，這些就需要多實驗和思考了，基于模型層面的融合最好不同模型類型要有一定的差異，用同種模型不同的參數的收益一般是比較小的。
參考：
1. https://tianchi.aliyun.com/competition/entrance/231784/forum
![LabelImg的安裝與使用（Anaconda環境）Labellmg的安裝[圖]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)