vim, git, aws and other three-letter words

Software Development Notes

Deploy websites to Scaleway with Docker

Overview

The purpose of this setup is:

  • Set up multiple web apps with different dependencies on the same server
  • Link all apps to the same MySQL server
  • Manage uploaded files for web apps in a single place (so it is easy to back them up)
  • Automatically deploy and update apps on the remote server
  • Run the same setup locally, so the development environment is very close to production
  • Set up backups for MySQL databases and for uploaded files

In this case, I deploy to Scaleway, but the same approach can be used for almost any cloud service.

Cloud Server

First, you need a cloud server with SSH access and one of the operating systems supported by Docker Machine. I used Scaleway's VC1S server with Ubuntu 14.04. You also need to install Docker Engine and Docker Machine locally.

Docker Machine - set up Docker remotely

The next step is to set up Docker on the server. This is done with the docker-machine create command:

SSH_HOST=111.222.333.44
SSH_USER=root
SSH_KEY=~/.ssh/id_rsa_scaleway

docker-machine create --driver generic \
 --generic-ip-address $SSH_HOST \
 --generic-ssh-user $SSH_USER \
 --generic-ssh-key $SSH_KEY \
 scaleway

To make it work, it is necessary to open port 2375 on the server. On Scaleway, it is done via the security group configuration. After changing the security group, it is necessary to reboot the server (stop / run via Archive option or Hard reboot).

Here I have used the generic Docker Machine driver; there are also specific drivers for popular cloud providers: AWS, DigitalOcean, etc. Note: Scaleway has a Docker Machine plugin; with it, you can do even more and automatically launch the new instance during the setup.

Check the full setup script here. On Scaleway, I also had to create loopback devices because the Docker setup failed with [graphdriver] prior storage driver \"devicemapper\" failed: loopback attach failed.

If something goes wrong during the setup, run docker-machine rm scaleway, fix the problem and run the setup again.

Deployment setup

This project is responsible for the deployment of the web applications to the remote host.

Projects layout

There are several web applications which I want to deploy, each packaged into a Docker container.

On the filesystem, they reside in the same ~/web folder along with the web-deploy project which acts as glue and sets everything up together, along with the MySQL container (used by all projects) and HAProxy (listens to port 80 and forwards requests to individual containers):

in ~/web $
projecta.com/
   Dockerfile
   config/
   db/
   www/
projectb.com/
   Dockerfile
   config/
   db/
   www/
projectc.com/
   Dockerfile
   app/
   config/
   db/
   ...
web-deploy/

The typical Dockerfile for the PHP application looks like this:

FROM php:5.6-apache

RUN apt-get update && \
  apt-get install -y msmtp wget && \
  apt-get clean

RUN apt-get update && apt-get install -y \
        libfreetype6-dev \
        libjpeg62-turbo-dev \
        libmcrypt-dev \
        libpng12-dev \
    && docker-php-ext-install -j$(nproc) iconv mcrypt \
    && docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ \
    && docker-php-ext-install -j$(nproc) gd pdo pdo_mysql mysql mysqli

COPY config/msmtprc /etc/msmtprc

RUN echo 'sendmail_path = "/usr/bin/msmtp -t -i"' > /usr/local/etc/php/conf.d/mail.ini
RUN a2enmod expires && a2enmod rewrite

COPY www/ /var/www/html/

VOLUME /var/www/html/files

This container is based on the official PHP image. To make the PHP mail function work, I also set up msmtp and configure PHP to use it. The example of the msmtp configuration file is here.

On Scaleway, SMTP ports are disabled by default. To make emails work, it is necessary to configure the security group (switch "Block SMTP" from On to Off). After changing the security group, the server should be rebooted (stop / run via Archive option or Hard reboot).

Here is also an example of the Ruby on Rails application (Redmine) Dockerfile:

# re-use the image which we already use
FROM php:5.6-apache
MAINTAINER serebrov@gmail.com

RUN DEBIAN_FRONTEND=noninteractive apt-get update \
    && apt-get install -y ruby libruby \
    libapache2-mod-passenger ruby-dev \
    libmysqlclient-dev libmagickcore-dev libmagickwand-dev \
    build-essential \
    apache2 ruby-passenger \
    && gem install bundler

WORKDIR /var/www/redmine/
COPY ./ /var/www/redmine/
COPY config_prod/*.yml /var/www/redmine/config/

RUN bundle install

COPY config_prod/redmine.conf /etc/apache2/sites-available
RUN chown -R www-data:www-data /var/www/redmine
RUN a2enmod passenger
RUN a2ensite redmine

VOLUME /var/www/redmine/files

This container is based on the same official PHP image as the PHP applications, just to reuse the already downloaded layers. The Ruby application runs under Apache with mod_passenger.

The web deploy project

The web-deploy project is the glue to build and start the containers for all web projects, link them to MySQL if necessary and set up HAProxy to forward requests to each sub-project.

in ~/web $
projecta.com/
projectb.com/
projectc.com/
   ...
web-deploy/
  deploy.sh
  init-docker.sh
  init-db-files.sh
  mysql-cli.sh
  utils.sh
  /haproxy
    Dockerfile
    build.sh
    haproxy.cfg
    mycompany.pem
  /mysql
    dumps/
      load-dumps.sh
      projecta.sql
      projectb.sql
      projectc.sql
    Dockerfile
    build.sh
    init-db.sh
  /build
    projecta.com.sh
    projectb.com.sh
    projectc.com.sh

Top-level scripts include:

The deploy.sh script uses docker-machine to build and run the containers on the remote server. There are several modes it can be used in:

  • ./deploy.sh: deploy (or update) all projects locally
  • ./deploy.sh local projecta.com: deploy only projecta.com locally
  • ./deploy.sh production: deploy all projects on the remote server
  • ./deploy.sh production projecta.com: deploy only projecta.com on the remote server

The script looks like this:

#!/usr/bin/env bash
set -e

SCRIPT_PATH=`dirname $0`
ENVIRONMENT="local"
if [[ "$1" != "" ]]; then
    ENVIRONMENT=$1
else
    echo Environment name is not specified, using 'local'.
fi

if [[ "$2" != "" ]]; then
    # deploy only the specified project
    # for example ./deploy.sh production projecta.com
    # will run ./build/projecta.com.sh
    PROJECTS=$SCRIPT_PATH/build/$2.sh
else
    # deploy all projects
    PROJECTS=$SCRIPT_PATH/build/*.sh
fi

echo "Deploy $PROJECTS"
read -p "Do you want to continue? " -n 1 -r
echo    # (optional) move to a new line
if [[ ! $REPLY =~ ^[Yy]$ ]]
then
    exit 1
fi

# add mysql and haproxy, always update them
PROJECTS="$SCRIPT_PATH/mysql/build.sh "$PROJECTS" $SCRIPT_PATH/haproxy/build.sh"

if [[ "$ENVIRONMENT" == "production" ]]; then
  eval "$(docker-machine env scaleway)"
  docker-machine ip scaleway
  set +a
  export | grep DOCKER
fi

for project in $PROJECTS
do
    echo 'Project: ' $project
    $project $ENVIRONMENT
done

if [[ "$ENVIRONMENT" == "production" ]]; then
  set +a
  export | grep DOCKER
fi

docker ps

Internally, the deploy.sh script goes over the scripts in the build sub-folder and executes them to create or update Docker containers.

The build script looks like this:

#!/usr/bin/env bash
set -e

SCRIPT_PATH=`dirname $0`
APP_NAME=projecta
CONTAINER_NAME=$APP_NAME-docker
PORT=8091
source $SCRIPT_PATH/../utils.sh
SRC=$SCRIPT_PATH/../../projecta.com

# if we are running locally, the container will use sources from the host filesystem
# (we create the volume pointing to /html)
# for production, we will copy the sources into the container
ENVIRONMENT="local"
if [[ "$1" != "" ]]; then
    ENVIRONMENT=$1
fi
ENV_DOCKER_OPTS=""
if [[ "$ENVIRONMENT" == "local" ]]; then
  ENV_DOCKER_OPTS="-v $(realpath $SRC)/html:/var/www/html"
fi
echo "Environment: $ENVIRONMENT"

# go to the source folder, where Dockerfile is
cd $SRC
# remove the image if it already exists
docker_rm_app_image $APP_NAME
# rebuild the image
docker build -t $CONTAINER_NAME .
# start the container
docker run -d -p $PORT:80 -v /var/web/projecta.com/files:/var/www/html/data/upload --name $APP_NAME --link web-mysql:mysql --restart always $ENV_DOCKER_OPTS $CONTAINER_NAME
# initially files belong to the root user of the host OS
# make them available to container's www-data user
docker exec $APP_NAME sh -c 'chown -R www-data:www-data /var/www/html/files'

A few interesting things happen here:

  • This project will use port 8091 on the host machine (can be accessed as localhost:8091); this port will also be mentioned in the HAProxy config to redirect requests to this app based on the requested domain name
  • For local deployment, the container will use source files from the host machine folder, so we can edit them directly without having to log in to the container; this is very convenient for local development
  • The web application files will be stored in the volume, which is available on the host machine at /var/web/projecta.com/files
  • The problem with permissions to the shared volume is solved by running chown from within the container (Apache runs as www-data, and after creation the files folder will belong to the root user)
  • This build script is used both for local and remote deployment; the remote part is handled by Docker Machine, which allows running Docker commands against the remote host (this is handled in the deploy.sh script)

Note: the docker_rm_app_image function from the build script is defined in the utils.sh script.

The web-deploy project also includes the setup for the HAProxy and MySQL containers.

HAProxy setup

This container (it resides in web-deploy/haproxy) runs HAProxy and serves as the main entry point on the server. Requests to port 80 are reverse-proxied to other containers. The Dockerfile is simple:

FROM haproxy:1.5
COPY ./haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
# this is needed to set up SSL for one of the projects
COPY ./mycompany.pem /etc/ssl/certs/

The configuration file (web-deploy/haproxy/haproxy.cfg) looks like this:

global
    log /dev/log local2
    maxconn     2048
    tune.ssl.default-dh-param 2048
    #debug

defaults
    log     global
    option  dontlognull
    mode http
    option forwardfor
    option http-server-close
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    stats enable
    stats auth haadimn:ZyXuuXYZ
    stats uri /haproxyStats

frontend http-in
    bind *:80

    # Define hosts based on domain names
    acl host_projecta hdr(host) -i projecta.com
    acl host_projecta hdr(host) -i www.projecta.com
    # this name is used for local testing
    acl host_projecta hdr(host) -i www.projecta.local
    acl host_projectb hdr(host) -i projectb.com
    acl host_projectb hdr(host) -i www.projectb.com
    # handle sub-domains (use hdr_end here, not hdr)
    acl host_projectc hdr_end(host) -i projectc.com

    # redirect projectd to https
    redirect scheme https if host_projectd !{ ssl_fc }

    ## figure out backend to use based on domainname
    use_backend projecta if host_projecta
    use_backend projectb if host_projectb
    use_backend projectc if host_projectc

frontend https-in
    bind *:443 ssl crt /etc/ssl/certs/mycompany.pem

    # Define hosts based on domain names
    acl host_projectd hdr(host) -i projectd.com

    use_backend projectd if host_projectd

backend projecta
    balance roundrobin
    server srv_pawnshop-soft-ru 127.0.0.1:8091

backend projectb
    balance roundrobin
    server srv_pawnshop-soft-kz 127.0.0.1:8092

backend projectc
    balance roundrobin
    server srv_lombard 127.0.0.1:8093

backend projectd
    balance roundrobin
    server srv_redmine 127.0.0.1:8100

And the build script (web-deploy/haproxy/build.sh) is this:

#!/usr/bin/env bash

export | grep DOCKER
docker-machine ip scaleway

SCRIPT_PATH=`dirname $0`
APP_NAME=web-haproxy
CONTAINER_NAME=web-haproxy-docker
source $SCRIPT_PATH/../utils.sh

pushd $SCRIPT_PATH
docker_rm_app_image $APP_NAME
docker build -t $CONTAINER_NAME .
docker run -d --restart always --net=host -p 80:80 -p 443:443 -v /dev/log:/dev/log --name $APP_NAME $CONTAINER_NAME
popd

The HAProxy container is launched with the --net=host option, so it can directly access all the ports exposed by other containers.

The HAProxy also handles SSL (HTTPS) connections. For one of the projects, it redirects HTTP to HTTPS. See the config file and this post for more information.

The HAProxy stats are available via http://hostname.com/haproxyStats. Login and password are set in the config file (haproxy.cfg, see the stats auth line in the defaults section).

Some hints to debug problems with HAProxy setup:

  • Uncomment debug in the config file
  • Check docker logs web-haproxy
  • Check the system log (/var/log/syslog); with the configuration above, the HAProxy container will log to the host's system log (see https://github.com/dockerfile/haproxy/issues/3)

MySQL setup

The MySQL container (it is under the web-deploy/mysql folder) also has a simple Dockerfile:

FROM mysql:5.6
ENV MYSQL_ROOT_PASSWORD=myrootpassword

VOLUME /var/www/mysql

The build script looks like this:

#!/usr/bin/env bash

SCRIPT_PATH=`dirname $0`
APP_NAME=web-mysql
CONTAINER_NAME=$APP_NAME-docker
source $SCRIPT_PATH/../utils.sh

pushd $SCRIPT_PATH
docker_rm_app_image $APP_NAME
docker build -t $CONTAINER_NAME .

docker run -d -v /var/web/mysql-data:/var/lib/mysql --restart always --name $APP_NAME $CONTAINER_NAME
popd

The data folder is mapped to /var/web/mysql-data on the host machine. The app containers which need the database are linked to this container.

Useful Docker Commands

Some useful commands to view and manage Docker containers:

  • Information about Docker: docker info
  • List running containers: docker ps
  • Information about a specific container: docker inspect {appname}
  • View application logs: docker logs {appname}
  • Restart an application: docker restart {appname}
  • Run shell inside the container: docker exec -it {appname} bash # replace {appname} with name from docker ps
  • Run the command inside the container (in this case, back up MySQL databases): docker exec web-mysql sh -c 'exec mysqldump --all-databases -uroot -p"$MYSQL_ROOT_PASSWORD"' | gzip -9 > all-databases.sql.gz

Cron in Docker and Backups on Scaleway

There are a few options to run cron with Docker:

Here I have chosen the first option: use cron on the host machine. First, the host machine is Ubuntu 14.04, so it already has cron. Second, everything runs on the same machine and I have no plans to scale out this setup, so it was the easiest option.

The web-deploy/init-db-files.sh script contains code to set up cron; here is the related part:

docker-machine ssh scaleway apt-get install -y postfix mutt s3cmd
docker-machine scp -r ./config/web-cron scaleway:/etc/cron.d/
docker-machine scp -r ./config/.s3cfg scaleway:/root

The postfix and mutt are needed for cron to send local emails about jobs. You can check these emails by ssh'ing to the server and running mutt. The s3cmd is used to back up databases and files from the shared storage to S3. The .s3cfg contains credentials for s3cmd; it can be generated by running s3cmd --configure.

Backups are very important, because at the moment Scaleway does not provide highly reliable hardware (no RAID disks, see this thread).

And to make a snapshot, you need to archive the instance (it takes a few minutes), make the snapshot and launch the instance again, so there is a noticeable downtime period. So I used cron to back up files and databases to Amazon's S3.

Note: Scaleway has storage which is compatible with S3, called SIS, but at the moment it is not available (at least in my account). When I try to create a bucket from the command line, it returns S3 error: 400 (TooManyBuckets): You have attempted to create more buckets than allowed. And the note in the Scaleway UI states "We're currently scaling up our SIS infrastructure to handle the load generated by all our new users. All new bucket creation are currently forbidden to preserve the service quality for current SIS users."

The cron file to perform backups looks like this:

# In case of problems, cron sends a local email, which can be checked with mutt
# (it requires apt-get install postfix mutt)
# Also check cron messages in /var/log/syslog
# update geolite database
# Note: docker exec should not have -t (tty) option, there is no tty, -i is also not needed
0 1 * * Sun root docker exec projecta /var/www/service/cron-update-geoip-db.sh
# backup all databases to s3
0 5 * * *   root docker exec web-mysql sh -c 'exec mysqldump --all-databases -uroot -p"$MYSQL_ROOT_PASSWORD"' | gzip -9 > all-databases.sql.gz && s3cmd put all-databases.sql.gz s3://myweb-backup/$(date +\%Y-\%m-\%d-\%H-\%M-\%S)-all-databases.sql.gz && rm all-databases.sql.gz
0 6 * * *   root s3cmd sync /var/web s3://myweb-backup/storage/

The command to back up MySQL databases does a few things:

  • Run mysqldump inside the MySQL container via docker exec
  • Pipe the result to gzip to archive it
  • Upload the archive to S3, adding a timestamp to the file name

And since all application containers map their files folders under the host's /var/web, I simply sync this folder to my S3 bucket.

Conclusion

As we can see, Docker can be successfully used not only to isolate application dependencies, but, along with Docker Machine, also to deploy and update applications on the remote server.

I don't recommend using the approach above for serious production setups, but it can be useful for small / personal projects.

Update: now that it exists, I think it would be easier to use Docker swarm mode to build a system like the one described above.

Tags: docker