近年来，在社会和行为科学(SBS)研究成果的可重复性和稳健性验证方面投入了大量精力，其中大部分涉及资源密集型的复制项目。本文研究了基于特征集的机器学习方法对SBS论文再现性的预测。我们提出了一个框架，从学术工作中提取五种类型的特征，可用于支持评估已发表的研究主张的再现性。文献计量特征、地点特征和作者特征从公共api收集或使用带有定制解析器的开源机器学习库提取。统计特征，如p值，是通过识别主体文本中的模式来提取的。语义特征，如资金信息，可以从公共APIs中获得，或者使用自然语言处理模型提取。我们分析了个体特征之间的两两相关，以及它们对于预测一组人类评估的地面真相标签的重要性。在此过程中，我们确定了9个顶级特征的子集，这些特征在预测语料库中SBS论文的再现性方面扮演着相对重要的角色。通过比较10个基于不同特征集训练的监督预测分类器的性能，验证了结果。
原文题目：Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models
原文：In recent years, significant effort has been invested verifying the reproducibility and robustness of research claims in social and behavioral sciences (SBS), much of which has involved resource-intensive replication projects. In this paper, we investigate prediction of the reproducibility of SBS papers using machine learning methods based on a set of features. We propose a framework that extracts five types of features from scholarly work that can be used to support assessments of reproducibility of published research claims. Bibliometric features, venue features, and author features are collected from public APIs or extracted using open source machine learning libraries with customized parsers. Statistical features, such as p-values, are extracted by recognizing patterns in the body text. Semantic features, such as funding information, are obtained from public APIs or are extracted using natural language processing models. We analyze pairwise correlations between individual features and their importance for predicting a set of human-assessed ground truth labels. In doing so, we identify a subset of 9 top features that play relatively more important roles in predicting the reproducibility of SBS papers in our corpus. Results are verified by comparing performances of 10 supervised predictive classifiers trained on different sets of features.
使用监督学习模型预测社会和行为科学论文的再现性.pdf