Deploying lighttpd, your flask-apps, gunicorn and supervisor with Ansible on CentOS

Deploying your first Python-app can be tedious if you want to do everything by yourself. There are many great tutorials that help you set up all the necessary programs you’ll need to host your webapp.

And trust me, it’s not easy: You’ll have to use a webserver that proxies the requests from your browser to a WSGI server. This in turn handles the communication with your app. You’ll also have to serve the static media like images, which will be handled again by your webserver. Then you’ll have to make sure that your WSGI server remains online (or comes back up after failure).

Of course you can just use a SaaS-platform like Heroku to avoid all these tasks, but where’s the fun in THAT?

Table of contents

Preface

Turns out, setting everything up yourself numerous times isn’t fun either. That’s why I wrote an Ansible-role that automatically does all these steps for me.

It install and configures:

• lighttpd, the webserver
• pip, to install the gunicorn, WSGI server
• virtualenvwrapper, so you don’t install the required packages into your system
• supervisor, to manage gunicorn
• Git, to clone the app from Github

Right now this Ansible role only works on RedHat-derivates. It shouldn’t be a problem to adopt the playbook to run on Ubuntu or other operating systems, though. Pull requests welcome!

For the impatient ones, here’s the Ansible Galaxy link and here’s Github-link. Read on if you want to know how to use it and what the role does in detail.

I assume that you’ve already installed Ansible and are somewhat familar with it. If not, read trough the Ansible Introduction or watch the quickstart video.

Installation

Download the role with the ansible-galaxy command and switch into the newly created directory (assuming you installed Ansible into the default directory):

ansible-galaxy install zufallsheld.lighttpd-gunicorn-flask
cd /etc/ansible/roles/zufallsheld.lighttpd-gunicorn-flask/

Configuring variables

Before we look into the tasks-file to see what the role does, let’s edit the variables required to successfully run the role. You’ll find the variables in the file vars/main.yml.

• Leave the lgf_vhosts_dir as it is, if you don’t know what you’re doing and don’t want to mess with the configuration afterwards.
• Change the lgf_supervisor_password to something other than 123.

The next part is an Ansible-list, containing the variables for your app:

• The lgf_servername must be the name of your application.
• The next three variables describe the path of the virtual environment, your application and the media-directory from where your pictures or videos are served. You have to use full paths here!
• The following three variables describe where the WSGI-server and the webserver listen on for connections. For the lgf_gunicorn_port use a port > 1024, as these are reserved ports.
• The lgf_host should be the IP-address of the server you’re deploying the app on.
• lfg_port is the port the webserver listens on (default is 80).

Finally, here is an example:

# lighttpd settings
lgf_vhosts_dir: "/etc/lighttpd/vhosts.d"

# supervisor settings
lgf_supervisor_password: "123"
lgf_supervisor_socket: "/tmp/supervisor.sock"

lgf_app:
  - lgf_server_name: "APP1"
    lgf_virtualenv_dir: "/var/www/html/venv_APP1"
    lgf_app_dir: "/var/www/html/APP1"
    lgf_media_dir: "/var/www/html/APP1/media"
    lgf_gunicorn_port: "12345"
    lgf_host: "192.168.100.25"
    lgf_port: "80"
    lgf_repo: "https://github.com/USER/APP1"

If you want to deploy another app to the same host, start another list under the one you already created, but be sure to change all the variables (except for the host) to something different:

lgf_app:
  - lgf_server_name: "APP1"
    lgf_virtualenv_dir: "/var/www/html/venv_APP1"
    lgf_app_dir: "/var/www/html/APP1"
    lgf_media_dir: "/var/www/html/APP1/media"
    lgf_gunicorn_port: "12345"
    lgf_host: "192.168.100.25"
    lgf_port: "80"
    lgf_repo: "https://github.com/USER/APP1"

  - lgf_server_name: "APP2"
    lgf_virtualenv_dir: "/var/www/html/venv_APP2"
    lgf_app_dir: "/var/www/html/APP2"
    lgf_media_dir: "/var/www/html/APP2/media"
    lgf_gunicorn_port: "12346"
    lgf_host: "192.168.100.25"
    lgf_port: "8080"
    lgf_repo: "https://github.com/USER/APP2"

The tasks

Now let’s take a look at what exactly this role does. All steps are in the tasks/main.yml-file. The first task is to import the epel-repository that contains supervisor, pip and other packages we need.

- name: Create the repository for EPEL
  yum: name="epel-release" state=installed

The next task is actually install all the required packages. In this step not only the packages are installed, most of the required directories, configurations and users are automatically generated.

- name: Install all needed packages
  yum: pkg="{{item}}" state=present
  with_items:
    - lighttpd
    - lighttpd-fastcgi
    - supervisor
    - git
    - python-pip
    - python-virtualenvwrapper

After everything is installed, its time to clone the app from Github. In this task, the variables for the repository and the destination you defined earlier are used:

- name: clone the github-repository containing the python code
  git: dest="{{item.lgf_app_dir}}" repo="{{item.lgf_repo}}" accept_hostkey=yes
  with_items:
    - "{{lgf_app}}"

After changing the permissions of the newly created repo, it’s time to install all the requirements (hopefully) located in the requirements.txt of the application. Ansible has its own pip-module to install the requirements. This module not only installs requirements, it can also install them inside a virtualenv and prior to this, create the virtual environment! It was never easier to deploy a new virtual environment filled with all the needed applications!

- name: install requirements for app
  pip: requirements="{{item.lgf_app_dir}}/requirements.txt" virtualenv="{{item.lgf_virtualenv_dir}}" virtualenv_command=virtualenv
  with_items:
    - "{{lgf_app}}"

Most likely, a WSGI-server is not part of the requirements-file of your application. That’s why the Ansible-role installs it separatly. Again, the pip-module is used:

- name: install gunicorn in virtualenv
  pip: name=gunicorn virtualenv="{{item.lgf_virtualenv_dir}}"
  with_items:
    - "{{lgf_app}}"

Next step is the configuration of lighttpd. The following step creates the vhost-directory we just included:

- name: create vhost-dir
  file: dest="{{lgf_vhosts_dir}}" state=directory owner=root group=root mode=0644

Then, the vhost-template configuration is copied to the host into the vhost-directory. The template uses many variables we defined earlier, like the host and app-name and the media-dir. No need to change the template, though!

- name: copy vhost-configuration
  template: src="vhost.conf.j2" dest={{lgf_vhosts_dir}}/vhost.{{item.lgf_server_name}}.conf
  with_items:
    - "{{lgf_app}}"

Here the lighttpd.conf is edited to include the vhosts-configuration. This task looks for a line beginning with “include_shell”. If it finds it, nothing happens. If the line is not present, it gets inserted.

- name: activate loading of vhosts
  lineinfile: dest="/etc/lighttpd/lighttpd.conf" line='include_shell "cat /etc/lighttpd/vhosts.d/*.conf"'

In the next step, the modules.conf is copied to the host.

- name: copy modified modules.conf in place
  copy: src="modules.conf" dest="/etc/lighttpd/modules.conf"

Next item for configuration is the supervisor. Again, a template is used to create the supervisor-configurationfile. There’s a loop in it so for every app defined in the variables, a section in the supervisor.conf is created. Here’s the loop:

{% for item in lgf_app %}
[program:{{item.lgf_server_name}}]
command={{item.lgf_virtualenv_dir}}/bin/gunicorn -b 127.0.0.1:{{item.lgf_gunicorn_port}} --error-logfile /var/log/lighttpd/{{item.lgf_server_name}}.log -u lighttpd -g lighttpd {{item.lgf_server_name}}:app
process_name={{item.lgf_server_name}}
directory={{item.lgf_app_dir}}
environment=PYTHONPATH={{item.lgf_virtualenv_dir}}/bin
{% endfor %}

And here’s the task that creates the configuration from the template and copies it to the host:

- name: copy supervisor config into place
  template: src="supervisor.conf.j2" dest="/etc/supervisord.conf" owner=root group=---------------root mode=0640

The second last step restarts supervisor and lighttpd to make sure, the new configuration is properly loaded. In the last step, your app is started through supervisor.

If everything went right, you should now be able to access your Flask-app on the IP-address you defined.

Troubleshooting

If not, there are some logfiles you can look into. For convenience, the above setup places all relevant logfiles into /var/log/lighttpd. There is:

• supervisord.log where the supervisor logs,
• error.log and access.log where lighttpd logs and
• YOUR_APP.log, a logfile named after your application, where its error are logged.