Quality Assurance for Infrastructure Orchestrators: Emerging Results from Ansible

Yue Zhang, Muktadir Rahman, Fan Wu, and Akond Rahman in 2nd International Workshop on the Foundations of Infrastructure Specification and Testing, co-located with the 20th IEEE International Conference on Software Architecture (ICSA 2023), 2023 Pre-print

Infrastructure as code (IaC) is the practice of automatically managing computing infrastructure at scale. Despite yielding multiple benefits for organizations, the practice of IaC is susceptible to quality concerns, which can lead to large-scale consequences. While researchers have studied quality concerns in IaC manifests, quality aspects of infrastructure orchestrators, i.e., tools that implement the practice of IaC, remain an under-explored area. A systematic investigation of defects in infrastructure orchestrators can help foster further research in the domain of IaC. From our empirical study with 22,445 commits mined from the Ansible infrastructure orchestrator we observe (i) a defect density of 17.9 per KLOC, (ii) 12 categories of Ansible components for which defects appear, and (iii) the `Module’ component to include more defects than the other 11 components. Based on our empirical study, we provide recommendations for researchers to conduct future research to enhance the quality of infrastructure orchestrators.