As part of a recent project, I worked on an Ansible playbook to deploy LEMP stack and Drupal/Wordpress CMS in multiple Ubuntu servers.
The Problem – handling project-specific variables in Ansible
Multiple CMS and different distributions require some small tweaks and customisations to my Ansible playbook. By expanding my Ansible playbook to support those customisations, I increase its complexity and introduce potential issues for other CMS or distributions – which requires a lot of testing. It also increases the maintainability effort of my Ansible playbook over the time.
My setup
- I created a custom, generic Ansible playbook to deploy CMS files into production and UAT environments. The playbook pulls a specified Git tag from the repository, deploys the files into the destination folder and executes pre-deployment actions, such as database backup, and post-deployment actions, such as filesystem hardening, database synchronisation from Production to UAT and much more.
- A Jenkins server is executing my generic Ansible playbook to deploy multiple CMSs and custom distributions of Drupal and WordPress.
The Solution – use external YAML file with variable definitions
The solution was inspired by modern continuous integration systems such as CircleCI, TravisCI, Bitbucket pipelines and others. The beauty of using variables, defined in a single YAML file added to the project is hard to overestimate.
In my case, moving custom variables into the project scope reduced playbook complexity and minimised the risk of an error. I have a solid Ansible playbook and all my custom variables are now pulled from the project itself, coming from a YAML file.
Deploying CMS code with Ansible
My playbook is using a set of tasks to pull the specified Git reference from the repository and deploy it into the destination remote server. It works well until the point, where ongoing customisations and a lot of conditions were necessary to account for various distributions and their configuration variations. For example, running Drupal or WordPress multisite, or maybe running those in a subdirectory and many other examples required custom conditions additions.
In my configuration, I used Ansible to deploy project files into one of the available environments, but more granular customisations were required for some projects. Instead of adding complexity to my playbook and extending it with additional condition checks, I decided to add support to a project-specific YAML file. I was hoping that custom variables and their values could be added to the project that we deploy to the server, offloading the Ansible playbook itself. For a simple example, variables for my Drupal website domain name, website name, web root and few other variables are defined in a YAML file, within the project itself.
My idea was possible with the Ansible’s fetch module. The idea is that the Ansible playbook (my deployment scripts) was pulling a YAML file with a predefined filename from the project and used variables defined in that file. This way I could add flexibility to the deployment scripts that are specific to the project being deployed, instead of piling up those as conditions in my Ansible playbook.
To support project-specific YAML file, I extended my custom Ansible playbook with the following tasks:
1. Load a YAML file from a remote project.
This task loads the YAML file with local overrides from the remote server’s filesystem, where the project files are deployed from a Git repository source. This task had to be added in my Ansible playbook immediately after the Git pull task.
- name: Load remote YAML file to alter the local configuration.
fetch:
src: "/absolute-path/filename.yml"
dest: "tmp/{{ server_hostname }}/"
flat: yes
fail_on_missing: no
In the example above:
- The flat option simplifies the destination filename path and avoids adding the full absolute path from the source server.
- The fail_on_missing option allows avoiding stopping Ansible playbook run if the YAML file does not exist. Instead, I am using the step 2 to register the file exists condition and reuse it later on.
Note, that the filename.yml file should contain valid YAML syntax, or the playbook execution fail.
2. Optional check for the file
This optional check for the file that we fetched in step 1. Since a Jenkins server runs my Ansible playbooks, I had to add three extra lines (lines 3-5) to avoid permission-related errors.
- name: Check if the local overrides file exists.
local_action: stat path="tmp/{{ server_hostname }}/filename.yml"
sudo: no
become: yes
become_user: jenkins
register: local_overrides_file
In the example above
- The local_action was required instead of the action, to instruct my Jenkins (Ansible runner) to check the file in the local filesystem, rather than in the remote server.
- The local_overrides_file variable registers the local_overrides_file.stat.exists condition as true if the filename.yml file was found in the local_action path (tmp/{{ server_hostname }}/filename.yml), or false otherwise.
3. Loading the variables defined in our overrides YAML file into our playbook
Loading the YAML file only if, after fetching it in the step 1, it exists in my Jenkins server at the expected location:
- name: Loads ansible filename.yml file
include_vars:
file: "tmp/{{ server_hostname }}/filename.yml"
when: local_overrides_file.stat.exists
- In this step, we load the variables, defined in the filename.yml file as the playbook variables.
- It is important to note, that those variables, defined in the filename.yml file, should not be used before this step in my Ansible playbook.
After the step 3, the remaining part of my Ansible playbook could use the new or overridden variables defined in the filename.yml file, which is added to my project’s Git repository. This simplifies my Ansible playbook and allows to define additional variables, execute additional, project specific tasks via a generic deployment playbook.
4. Clean up
To clean up our tasks, we should remove our filename.yml file with the file Ansible module. Missing this step may result in an unpredicted behaviour during future playbook runs.
- name: Remove filename.yml file.
file:
path: "tmp/{{ server_hostname }}/filename.yml"
state: absent
when: local_overrides_file.stat.exists
Credits
Thanks, Jeff Geerling and his excellent book Ansible for DevOps, which inspired me in finding the way around the problem I faced.