What is Jenkins

Jenkins (http://jenkins-ci.org) is almost the standard for CI (continuous integration) now. It has a big community with lots of plugins for nice features, in order to learn those things, it will be good to practice it by setting the environment.

Also if you want to introduce new features of jenkins to others, you want to have a demo environment for them quickly.

How this can be done easily? Docker is my favorite to achieve this.

In this blog series, I use some examples to describe how to achieve this step by step.

Demo jenkins small feature – AnsiColor plugin

Jenkins AnsiColor plugin is one of my favorite small plugin, it can turn console log looks better.

So I want to setup a demo environment for everyone can try it locally without installation, since mostly we want to try before deployment.

Result

Let’s see result first since you may just interesting this feature

docker run –p 8080:8080 –t larrycai/jenkins-demo1

The jenkins is started in console, then open the browser to access 8080 port.

Looks great, one craft sample job is there and Jenkins is latest LTS version 1.580.1

Click Craft job and run it, then check the console, great, it shows the color in console

Then back to see how it is configured

Now the demo is completed

How it works

Here is the Dockerfile, see source

FROM ubuntu:trusty

MAINTAINER Larry Cai <larry.caiyu@gmail.com>

ENV REFRESHED_AT 2014-11-03

RUN apt-get update  && apt-get install -qqy curl openjdk-6-jdk

ENV JENKINS_HOME /opt/jenkins/data
ENV JENKINS_MIRROR http://mirrors.jenkins-ci.org

# install jenkins.war and plugins

RUN mkdir -p $JENKINS_HOME/plugins $JENKINS_HOME/jobs/craft
RUN curl -sf -o /opt/jenkins/jenkins.war -L $JENKINS_MIRROR/war-stable/latest/jenkins.war

RUN for plugin in chucknorris greenballs scm-api git-client ansicolor description-setter \
    envinject job-exporter git ws-cleanup ;\
    do curl -sf -o $JENKINS_HOME/plugins/${plugin}.hpi \
       -L $JENKINS_MIRROR/plugins/${plugin}/latest/${plugin}.hpi ; done

# ADD sample job craft

ADD craft-config.xml $JENKINS_HOME/jobs/craft/config.xml

# start script

ADD ./start.sh /usr/local/bin/start.sh
RUN chmod +x /usr/local/bin/start.sh

EXPOSE 8080

CMD [ "/usr/local/bin/start.sh" ]

Started with installing openjdk/curl package and setting related environment which is needed when jenkins starts.

Jenkins app (.war) can be found in http://jenkins-ci.org/, you can choose latest version or LTS (Long Term Support) stable version, here I choose LTS version http://mirrors.jenkins-ci.org/war-stable/latest/jenkins.war

All the plugins can be found in mirror sites: http://mirrors.jenkins-ci.org/, you need to find the Plugin Id for your plugin like ansicolor, which can be mapped to http://mirrors.jenkins-ci.org/plugins/ansicolor/latest/ansicolor.hpi

In jenkins, job’s configuration is saved as config.xml, you can prepare in advance and put it under $JENKINS_HOME/jobs/craft in docker image.

<?xml version='1.0' encoding='UTF-8'?>
<project>
  <actions/>
  <description></description>
  <keepDependencies>false</keepDependencies>
  <properties/>
  <scm class="hudson.scm.NullSCM"/>
  <canRoam>true</canRoam>
  <disabled>false</disabled>
  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
  <triggers/>
  <concurrentBuild>false</concurrentBuild>
  <builders>
    <hudson.tasks.Shell>
      <command>#!/bin/bash
env
echo -e "\e[1;31;42m Using docker to demo is awful, v5 \e[0m"
echo see more in http://misc.flogisoft.com/bash/tip_colors_and_formatting
</command>
    </hudson.tasks.Shell>
  </builders>
  <publishers/>
  <buildWrappers>
    <hudson.plugins.ansicolor.AnsiColorBuildWrapper plugin="ansicolor@0.4.0">
       <colorMapName>xterm</colorMapName>
    </hudson.plugins.ansicolor.AnsiColorBuildWrapper>
  </buildWrappers>
</project>

The most easy way is to get the file from running jenkins directly (append config.xml in your job URL)

And in the end, add small start.sh which will start your jenkins during startup.

exec java -jar /opt/jenkins/jenkins.war

Then you can build your docker image

docker build –t larrycai/jenkins-demo1 .

How to share it public

You can put your project into github or bitbucket and run your build in http://hub.docker.com , then others can simple run docker command to run it (you can search for guideline)

Summary

In this blog, we demoed how to dockerize your jenkins application with sample job, which is well prepared for the configuration. It will be easily for your audience to know your demo feature.

Now you can pack your nice jenkins feature into the docker.

In next blog, I will show how to organize the JENKINS home better

Docker can help us a lot.

What is Fig ?

Fig http://fig.sh is a simple orchestration tool for managing multiple docker containers, now the company is acquired by docker.

Simple you write configuration in fig.yml file (source from https://github.com/larrycai/codingwithme-ansible/blob/master/fig.yml like

ansible:
  image: dockerfile/ansible
  volumes: 
   - /home/docker/codingwithme-ansible:/data
  links: 
   - haproxy
   - web1
   - web2
   - database
haproxy:
  image: larrycai/ubuntu-sshd
web1:
  image: larrycai/ubuntu-sshd
web2:
  image: larrycai/ubuntu-sshd
database:
  image: larrycai/ubuntu-sshd

Then execute fig run ansible, it will launch the docker container as you defined: sample above is to start 5 docker containers and enter into ansible container for testing.

Learn fig more in http://blog.docker.com/2014/08/getting-started-with-orchestration-using-fig

Problem for running environment

Before using the fig, you mostly need to install it, while if you check http://www.fig.sh/install.html, you will notice it only support OS X (with boot2docker 1.3) and other Linux system, you can’t install it in Windows (boot2docker env).

Also if you start to play with CoreOS, you will find it is difficult to install it as well, in official document, it recommends to use fleet (coreos tool) instead to manage docker service. Then you need to run fig2coreos, it is not so sweat for small things.

How can we solve it ?

TLDR;

docker run larrycai/fig

Using docker in docker for fig

The solution is not difficult actually, we can simple to have a fig container to run docker inside, surely it needs docker inside docker as well.

See Dockerfile

## docker run -v /var/run/docker.sock:/docker.sock -v <figapp>:/app larrycai/fig

FROM ubuntu:latest
MAINTAINER Larry Cai "larry.caiyu@gmail.com"
ENV REFREST_AT 20141015

RUN apt-get update && apt-get install -y curl make

RUN \
    curl -L https://get.docker.io/builds/Linux/x86_64/docker-latest -o /usr/local/bin/docker  && \
    chmod +x /usr/local/bin/docker && \
    
    # see http://www.fig.sh/install.html 
    curl -L https://github.com/docker/fig/releases/download/1.0.0/fig-`uname -s`-`uname -m`  -o /usr/local/bin/fig && \
    chmod +x /usr/local/bin/fig 
    
ENV DOCKER_HOST unix:///docker.sock

WORKDIR /app

# set initial command

ENTRYPOINT ["/usr/local/bin/fig"]
CMD ["-v"]

It will install latest docker inside, and also follow the fig installation to install 1.0.0 version as well

How to use it ?

As normally docker container, run it directly to see the fig help (which is default)

docker@boot2docker:~$ docker run larrycai/fig
Punctual, lightweight development environments using Docker.

Usage:
  fig [options] [COMMAND] [ARGS...]
  fig -h|--help
....

Now if I clone my codingwithme-ansible sample code locally into /home/docker (see fig.yml sample above), then I can run

docker@boot2docker:~/codingwithme-ansible$ docker run -it \
 -v /var/run/docker.sock:/docker.sock \
 -v /home/docker/codingwithme-ansible:/app \
 larrycai/fig run ansible

Then it will start web stack containers (haproxy/web/database) at once ( download needed docker image for first time) and run into ansible container

-v /var/run/docker.sock:/docker.sock is used to pass the docker daemon socket into docker container so docker inside can communicate outside to control docker.
-v /home/docker/codingwithme-ansible:/app is to share the host folder inside.

Remind

Just remind one thing, since those docker commands are run in host machine, please specify the absolute path in volumes, don’t use .

  volumes: 
   - /home/docker/codingwithme-ansible:/data

This is the only limitation for using fig docker container, I guess.

Summary

Now using fig docker image, you don’t need to install fig manually, this is core value for docker. And it works in Windows (boot2docker 1.3+)/CoreOS as well.

Enjoy.

Introduction

Docker is the rising star recently in cloud industry, docker’s ecosystem grows even more fast compare to 2 years ago of openstack’s.

Docker is an open platform for distributed applications for developers and sysadmins. It is based on existing container technology, and make it is much easier to use.

Please visit https://www.docker.com/whatisdocker/ for more information.

In this blog, I share three use cases on how to use docker to improve development efficiency based on my current experience

Use case 1: Product Building Service

Some of our products have different OS and different release, it demands several physical HW or VM (mostly in IT-Hub) to host different build platform, there are some issues around this

1. Difficult to maintain those servers, we have to request IT engineer to install the packages and sometimes patches, in worse case, some designer may request additional package to install packages (surely process is problem as well). It is not well version controlled.
2. Developer team (use windows) has to find the way to clone the build server locally to make sure the code can be compiled in the same way with real build server, sometimes it builds failure in CI though it is ok locally. it has consistency problem
3. Due to the life-cycle of the product, we have to maintain them and keep them alive with cooperating with IT dept., it is the waste.

By using docker, we move our build service into docker image, And consolidate different build server into one powerful HW/VM machine, they can share the load.

Cost is not the big gain for those small machines, Instead, we make our build environment more professional

• The build server is now version controlled, each build server is docker image, generated from Dockerfile, which is the plain text to describe how the server comes. They are put into git control system, all the changes are visible to everyone.
• We are able to run the docker instance for different release in seconds: just checkout the Dockerfile and rebuild the image and launch it.
• Developers now can run those build service locally, which make sure if it is passed locally, it works in CI as well Using docker, build environment is well maintained. see my previous blog Create the docker images for product development for more detail.

Use case 2: Local cloud testing environment

Recently I am studying how to use ansible to do image installation in cloud (openstack) environment, there are two things bother me during development:

1. Remote Openstack environment is in another network, I need to sync my code base from windows to target machines to test, it wastes time.
2. It demands me to start/stop VM frequently, the start-up of VM lasts several minutes mostly, it is kind of waste of time since some testing are none VM related like installation packages, while in order to get clean environment, I have to restart them. So I switch for most of development time using docker.

Boot2docker provides the docker environment in windows, it can share the folder with my Window folder, so I use normal text editor to write the codes, and testing it inside docker container.

Also I created several docker images to simulate the real VM node, and using fig to start those nodes in seconds. It can provide almost the same environment as openstack environment, and it is much much fast.

I only use openstack when I have to deal with cloud VM and integration test in the end. I have to say I love it.

Use case 3: Test more locally with different scenario

My working product is combined with several nodes with different deployment. By using some simulators around it, we can do end-2-end testing.

While testing is costly, several issues:

1. Since too many deployment cases exists and fail-over and load balancer kind of stuff are difficult to setup the environment, it is mostly tested in later phase, which cause lots of trouble and waste.
2. competence is low due to less chance to practice, mostly they are developed and tested with experienced engineers only.
3. since it is difficult to get environment ready, it causes the difference setup and view for designers and tester.

By using docker, we managed to establish those nodes in docker images. Inside, we provide testing environment (legacy, complex) as well, cool. Professional tester quickly like it.

Now all the developers (including traditional testers) can do most of the work locally. They may build the java codes in eclipse, “deploy the package” (actually shared with docker instance) into docker environment, doing testing.

Since the environment is easy to be established, therefore everyone can experience different setup in seconds, they learn that quickly by practice

It is just the beginning, and designers give a strong feedback, “Wooh, it can be done like this”.

Summary

The unique features of docker brings new trend of software development, it will be definitely to contribution to better software development.

The use case above is just the beginning of the usage of docker, I will soon drive the DevOps concept inside software development based on CI.

Introduction

"CodingWithMe - Learning by Coding" is a coding workshop, it aims to learn one technology in 90 minutes by practice.

Normally CodingWithMe runs several different sessions every month to let participants to choose, see the real poster below

I initiated this in 2009 inside R&D department in the company, and so far it has 46 times, it is quite successful.

In detail

Each CodingWithMe session includes practical coding exercises (recommend to >50% time on coding). Topics could be directly connected to the current work, and mostly it introduces new things.

Take a look at one example Learn REST API with Python in 90 minutes

Typically the first exercise is "hello world" sample, it shall be able to executed in 10-20 minutes, less talking more coding. And everyone shall follow it.

After that, the teacher will give more explanation, then with exercise 2

Last exercise could be little challenge.

As I stated above, mostly it is organized with several sessions to let developers to choose since different people had different flavors.

Benefit

The area of engineering skills and SW craftsmanship is truly important and lessons learned from this initiative are e.g:

• Improved the awareness of the new technology
• Boost innovation in efficiency way
• for their coding skills (among other things)

This initiative was started by a number of developers based on the insight that growth of coding skills is a key area of importance and they then organized this community by themselves.

So running this inside community or company can get more recognition.

Others

Why 90 minutes

60 minutes are too short to prepare environment and 3 exercise, 90 minutes is just ok

And I used to have this workshop in the afternoon, it can be finished in 2 time slots: 13:00-14:30 and 14:30-16:00

How many people inside one CodingWithMe session

6-10 is the right scale, since the teacher needs to take care the progress, too many people are not easily to be covered.

Less than 6 people is little waste

What's the big difference compare to self-learning

Some people learn it and forge the good exercises, just bring your laptop inside and learn on topic one month.

Reference

• Real CodingWithMe example 1: Learn REST API with Python in 90 minutes
• Real CodingWithMe example 2: Learn MultiMarkdown to book