AWS: Drone CI setup on EC2
Drone CI is a Continuous Integration platform. It uses Docker containers to run tests for your application hosted on GitHub.
It is not complex to set up the automatic testing for your application and run Drone CI on an EC2 instance using Elastic Beanstalk. It is not even necessary to have a dedicated EC2 instance for the CI system; for example, I run it on the staging server.
Drone CI setup
First, you'll need to create a Drone configuration file, .drone.yml,
which looks like this:
image: serebrov/centos-python2.7-java
env:
- GOPATH=/var/cache/drone
script:
- pip install -r requirements.txt
- ./scripts/ci_test.sh
notify:
email:
recipients:
- mymail@gmail.com
As you can see, the setup is really simple. I have a custom Docker image
which installs all the application requirements. Then Drone runs
pip install -r requirements.txt to install Python app dependencies and
runs the ci_test.sh shell script to launch testing. This script looks like
this:
#!/bin/bash
# set -e
SCRIPT_PATH=`dirname $0`
PYTHON=python
if which python27; then
PYTHON=python27
fi
cd $SCRIPT_PATH/..
$PYTHON -m unittest discover -s tests
RESULT_MOCK=$?
pushd $SCRIPT_PATH/../tests/dynamodb-local
java -Djava.library.path=./DynamoDBLocal_lib -jar ./DynamoDBLocal.jar -inMemory -sharedDb &
PID=$!
popd
echo "Started local dynamodb: $PID"
USE_MOCK= $PYTHON -m unittest discover -s tests
RESULT_LOCALDB=$?
kill -9 $PID
exit $(($RESULT_MOCK+$RESULT_LOCALDB))
My application uses Amazon DynamoDB, and I run tests twice - first using a simple in-memory DynamoDB mock and one more time using local DynamoDB.
The mock is used during development, and tests run very fast (a few seconds). Local DynamoDB is much slower, and we need a few minutes to run tests, but we can catch some specific errors related to the database usage.
Elastic Beanstalk setup
The Elastic Beanstalk setup allows us to automatically install Drone CI when a new EC2 instance is launched.
Here it is good to have a single-instance Elastic Beanstalk environment, and the scripts below will install Drone on the EC2 instance.
First, the EB config (.ebextensions/app.config):
container_commands:
004-start-container-commands:
command: logger "Start deploy script" -t "DEPLOY"
005-command:
command: chmod +x .ebextensions/deploy.sh
006-deploy:
command: .ebextensions/deploy.sh 2>&1 | /usr/bin/logger -t "DEPLOY" ; test ${PIPESTATUS[0]} -eq 0
010-start-container-commands:
command: logger "Start container commands" -t "DEPLOY"
190-clearcaches:
command: (echo 'flush_all' | nc localhost 11211) 2>&1 |/usr/bin/logger -t "DEPLOY"
200-end-container-commands:
command: logger "End container commands" -t "DEPLOY"
packages:
yum:
gcc: []
gcc-c++: []
docker: []
python-devel: []
atlas-sse3-devel: []
lapack-devel: []
services:
sysvinit:
docker:
enabled: true
ensureRunning: true
It will install the packages I need (including Docker) and run the
deploy.sh script which does the Drone installation:
#!/usr/bin/env bash
set -e
SCRIPT_PATH=`dirname $0`
if [[ $APP_ENV != "staging" ]]; then
echo 'Not a staging env, exit without installation'
exit 0
fi
copy_ext $SCRIPT_PATH/files/droned.conf /etc/init/droned.conf 0755 root root
echo "Start drone check/install"
if which /usr/local/bin/drone; then
echo "Found drone, skip install"
copy_ext $SCRIPT_PATH/files/drone.toml /etc/drone/drone.toml 0755 root root
if (stop droned); then
echo 'Stopped drone'
fi
start droned
else
echo "Installing drone"
wget downloads.drone.io/master/drone.rpm
yum -y -q install drone.rpm
# Create a user and group 'droned', -M == no home dir
# Note: better to use a separate user, but there was a permission error:
# Post http:///var/run/docker.sock/v1.9/build?q=1&rm=1&t=drone-4dcf1ea3fb: dial unix /var/run/docker.sock: permission denied
# so for now use sudo; it would be good to use the droned user (see upstart script)
# probably this should help: https://web.archive.org/web/20150707183331/https://docs.docker.com/installation/ubuntulinux/#create-a-docker-group
adduser -M --user-group droned
passwd -l droned
chown -R droned:droned /var/lib/drone
# # to prevent 'sudo: sorry, you must have a tty to run sudo'
sed -ie 's/Defaults.*requiretty.*/# Defaults requiretyy/gI' /etc/sudoers
copy_ext $SCRIPT_PATH/files/drone.toml /etc/drone/drone.toml 0755 root root
start droned
fi
This script uses utils.sh; you can find it in this post.
There are two more configuration files used in the setup process.
The first is droned.conf, an upstart script.
You may not need it if your system has systemd, but the Amazon Linux I used on my instance didn't have it and Drone raised this error:
which: no systemctl in (/sbin:/bin:/usr/sbin:/usr/bin:/usr/X11R6/bin) Apr 14 11:55:44 ip-172-31-1-79 DEPLOY: Couldn't find systemd to control Drone, cannot proceed. Apr 14 11:55:44 ip-172-31-1-79 DEPLOY: Open an issue and tell us about your system. added init script
So I added a custom upstart script to fix this:
#!upstart
description "Droned upstart job"
start on startup
stop on shutdown
script
# custom http server settings
# export DRONE_SERVER_PORT=""
# export DRONE_SERVER_SSL_KEY=""
# export DRONE_SERVER_SSL_CERT=""
# session settings
# export DRONE_SESSION_SECRET=""
# export DRONE_SESSION_EXPIRES=""
# custom database settings
# export DRONE_DATABASE_DRIVER=""
# export DRONE_DATABASE_DATASOURCE=""
# github configuration
# export DRONE_GITHUB_CLIENT=""
# export DRONE_GITHUB_SECRET=""
# export DRONE_GITHUB_OPEN=false
# email configuration
# export DRONE_SMTP_HOST=""
# export DRONE_SMTP_PORT=""
# export DRONE_SMTP_FROM=""
# export DRONE_SMTP_USER=""
# export DRONE_SMTP_PASS=""
# worker nodes
# these are optional. If not specified, Drone will add
# two worker nodes that connect to $DOCKER_HOST
# export DRONE_WORKER_NODES="tcp://0.0.0.0:2375,tcp://0.0.0.0:2375"
echo $$ > /var/run/droned.pid
# exec sudo -u droned \
# DRONE_SERVER_PORT=$DRONE_SERVER_PORT \
# DRONE_GIHUB_CLIENT=$DRONE_GIHUB_CLIENT \
# /usr/local/bin/droned >> /var/log/droned.log 2>&1
# exec sudo -u droned \
exec sudo \
/usr/local/bin/droned --config=/etc/drone/drone.toml >> /var/log/droned.log 2>&1
end script
pre-start script
echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" >> /var/log/droned.log
end script
pre-stop script
rm /var/run/droned.pid
echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Stopping" >> /var/log/droned.log
end script
Drone parameters can be changed via environment variables in the upstart
script, or you can attach these environment variables to the instance in
the Elastic Beanstalk configuration (in the AWS UI). Or you can use a
special drone.toml configuration file:
# Drone configuration
[server]
port=":8080"
#####################################################################
# SSL configuration
#
# [server.ssl]
# key=""
# cert=""
#####################################################################
# Assets configuration
#
# [server.assets]
# folder=""
# [session]
# secret=""
# expires=""
#####################################################################
# Database configuration, by default using SQLite3.
# You can also use postgres and mysql. See the documentation
# for more details.
[database]
driver="sqlite3"
datasource="/var/lib/drone/drone.sqlite"
[github]
client="xxxxxxxxxxxxxxxxxxxx"
secret="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
# orgs=[]
# open=true
#####################################################################
# SMTP configuration for Drone. This is required if you plan
# to send email notifications for build statuses.
#
[smtp]
host="email-smtp.us-east-1.amazonaws.com"
port="25"
from="myemail@gmail.com"
user="XXXXXXXXXXXXXXXXXXXX"
pass="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
# [worker]
# nodes=[
# "unix:///var/run/docker.sock",
# "unix:///var/run/docker.sock"
# ]
[worker]
nodes=[
"unix:///var/run/docker.sock"
]
To integrate Drone CI with GitHub, you will need to create an application and generate client and secret tokens; see details here.
That's all. Now, if you launch an environment with this configuration,
you will have Drone CI running. Access it via a URL like
your_eb_environment_url.elasticbeanstalk.com:8080.
Setup Drone users
In the configuration above, we didn't set the option to open new user registration to Drone, so random people will not see your repositories and tests. Instead, you will need to explicitly invite all the people who need access.
First, log in with GitHub as the repository owner. This user will become
the Drone CI admin and will be able to invite other users. Now select
Users in the Drone menu and invite more users (using GitHub accounts).
If you have problems with user setup, check this issue.
Persist Drone CI users using a custom AMI
We set up Drone CI to use a local SQLite database; it is easy, but the problem is that when your instance is terminated and launched again, you will lose all user accounts.
So when the setup is done, it is a good idea to make an image of the running instance and then set it as the machine image for the Elastic Beanstalk environment:
- Do the following once your Drone is up and running and you added all the users
- Open the EC2 instances page in the AWS UI, select the instance with Drone and
select the action
Image -> Create imagefor it - Open the
AMIspage and wait until the image is created, get the AMI ID (like ami-d2faa5b0) - Go to the Elastic Beanstalk environment, Configuration, Instances and set
Custom AMI ID
Now, if your instance is re-created, all the settings will stay. The drawback is that you'll need to repeat the process once you add more users. An alternative is to use an external RDS instance with MySQL as the Drone database.
How to run a build locally
To run the build manually, on your local machine, use the following command (you also need to install Docker and Drone locally):
sudo drone build
You can also run the build manually on the EC2 server if you SSH to it and run:
# for some reason it doesn't work without the full path under sudo:
$ sudo /usr/local/bin/drone build
Custom Docker image
For your Drone environment, you can choose one of the existing Docker images; check Docker Hub.
But a custom image with additional setup steps can speed up the build process, because you don't need to wait for dependencies to install on each build.
I use the automated build feature of Docker Hub, which means that I have a GitHub repository with a Dockerfile and description.
Every time I push a new Dockerfile version to this repository, the image is rebuilt and becomes available on Docker Hub.
Note: if you modify a Dockerfile and have Drone already running, then it
will not see the change. By default, the image is fetched only once. To fix
this, add the :latest tag to the image name and trigger a build; Drone will
update the image. See this pull request for details.
I didn't try it, but probably this issue can also be fixed in one of the following ways:
- rebuild the Elastic Beanstalk environment, so everything will be re-built from scratch
- use Docker directly to pull the new version of the image
I use a t2.small instance to run Drone in a single-instance Elastic
Beanstalk environment. But take into account that a small EC2 instance
may not be enough to build or rebuild the image. If Docker fails with
the can not allocate memory error (check in /var/log/docker), then it is
possible to use a larger instance initially and then switch to a smaller
one:
- assume that you have a running environment with a t2.small instance
- change the instance type to t2.medium in the environment configuration
- save, wait until the new instance is launched
- trigger a build by pushing some change, it will take longer than usual to initialize the new container image
- make sure the build finished successfully
- make an instance image (from the EC2 console)
- in the EB environment settings, change the type back to t2.small and point it to the new image
- save and wait until it launches, check if the build passes on the small instance