Characterizing Scientific Reporting in Security Literature: An analysis of ACM CCS and IEEE S&P Paper

Morgan Burcham, Mehran Al-Zyoud, Jeffrey Carver, Mohammed Alsaleh, Hongyi Du, F. Gilani, Jin Jiang, Akond Rahman, Ozgur Kafali, Ehab Al-Shaer, and Laurie Williams in 5th Annual Hot Topics in the Science of Security (HoTSoS) Symposium, 2017 Pre-print

Scientific advancement is fueled by solid fundamental research, followed by replication, meta-analysis, and theory building. To support such advancement, researchers and government agencies have been working towards a ‘science of security’. As in other sciences, security science requires high-quality fundamental research addressing important problems and reporting approaches that capture the information necessary for replication, meta-analysis, and theory building. The goal of this paper is to aid security researchers in establishing a baseline of the state of scientific reporting in security through an analysis of indicators of scientific research as reported in top security conferences, specifically the 2015 ACM CCS and 2016 IEEE S&P proceedings. To conduct this analysis, we employed a series of rubrics to analyze the completeness of information reported in papers relative to the type of evaluation used (e.g. empirical study, proof, discussion). Our findings indicated some important information is often missing from papers, including explicit documentation of research objectives and the threats to validity. Our findings show a relatively small number of replications reported in the literature. We hope that this initial analysis will serve as a baseline against which we can measure the advancement of the science of security.