Ansible module for the SIMPLE Grid Framework

Description

The Worldwide LHC Computing Grid (WLCG) unites resources from over 160 computing centres and research institutes spread across the world and the number is expected to grow in the coming years. However, provisioning resources (compute, network, storage) at new sites to support WLCG workloads is still no straightforward task and often requires significant assistance from WLCG experts. Recently, the WLCG community has initiated steps towards reducing such overheads, for example, through the use of prefab Docker containers or OpenStack VM images, along with the adoption of popular tools like Puppet for configuration. In 2017, the SIMPLE Grid project was initiated to construct shared community repositories providing such building blocks. These repositories are governed by a single SIMPLE Framework Specification Document which describes a modular way to define site components such as Batch Systems, Compute Elements, Worker Nodes, Networks etc

The SIMPLE Grid project is an extension of the SIMPLE Framework that combines popular configuration management technologies such as Puppet/Ansible and container orchestration technologies such as Docker Swarm/Kubernetes to allow deployment of complex computing clusters using a single site level configuration file. The various components of the framework and their functions are described in the SIMPLE specification document. The central configuration manager is a core component of the framework that controls the execution pipelines for all of the other components of the framework. A Puppet based implementation of the central configuration manager that complies with the specification document has been developed by the team at CERN. Efforts to develop a parallel implementation in Ansible were started during Google Summer of Code 2018 and this year we plan to develop it further over the summer.

Task ideas and expected results

Through this project, in broad terms, the student participant can expect to extend and improve the Central Configuration Manager implemented in Ansible during Google Summer of Code 2018. The functions of the Central Configuration Manager are described in the SIMPLE framework’s specification document. They have already been implemented in the simple_grid_puppet_module available on Github. Using these resources, the student can expect to work towards the following deliverables:

Create roles for Config_Master (CM) and Lightweight_Component (LC) nodes: The framework abstracts the computing infrastructure into the above-mentioned node types.
Implement Ansible playbooks for Installation stage : Ansible tasks related to configuring Ansible, downloading the Ansible module for the SIMPLE Grid framework
Implement Ansible playbooks for Configuration stage: Ansible tasks to configure the SIMPLE Grid Ansible module. Compile the site level configuration file via the SIMPLE Grid YAML compiler and validate its contents using the Simple Grid Validation Engine
Implement Ansible playbooks for Pre-Deployment stage: Set up a container orchestrator. In case of Docker Swarm, set up an attachable overlay network and a DNS file. Copy the output of the YAML compiler to all the Lightweight Component nodes and copy lifecycle callbacks for component repositories to appropriate nodes based on the execution_id generated by the compiler.
Implement Ansible playbooks for Deployment stage: Configure Firewall, CVMFS on lightweight component nodes, execute lifecycle hooks and lifecycle events of component repositories and start the Docker containers for each component repository.
Implement logging infrastructure for each stage.
Implement tracking of various deployment stages.

Requirements

Familiarity and prior experience with Ansible.
Programming experience in Python is a plus.
Linux string fundamentals.
Git/Agile development methodology.

Mentors

Mayank Sharma (Primary mentor)
Maarten Litmaath (Co-mentor)

Links:

Corresponding Project

WLCG

Participating Organizations

CERN