Refactoring a function name across several patches with git rebase

git rebase is one of my favorite git commands. It allows to update a set of local patches against another git branch and also to rework, trough the -i flag some previous patches.

The problem I had to deal with was quite simple, rename a function called notProperPythonCode to proper_python that was defined in the first patch and be sure that all other patches are using the correct name. The –exec flag allows to run a custom script after each patch is applied, so that I could run sed to process the Python files and replace the old function name with the new one. The process is quite simple, except that such changes would trigger a lot of merge conflicts, trivial to solve but quite annoying.
Fortunately git rebase allows to choose what merge strategy must be adopted for solving conflicts, the theirs strategy in case of a conflict, will take the previous version of the patch and silently use it. That is fine for this simple substitution case, where we process each file ending by *.py in the repository.

In the end, the final command looked like this:


git rebase -X theirs -i \
  --exec "sh -c \"git ls-files *.py \
  | xargs sed --follow-symlinks -i -e \
   's|notProperPythonCode|proper_python|g'\" \
  && git commit --amend -a -C HEAD" origin/master

System containers for Atomic

The main reason behind system containers was the inability to run Flannel in a Docker container as Flannel is required by Docker itself. CoreOS solved this chicken and egg problem by using another instance of Docker (called early-docker) that is used to setup only Etcd and Flannel.

Differently, Atomic system containers will be managed by runc and systemd.

The container images, even though being served through the Docker v2 registry, are slighty different than a regular Docker container in order to be used by Atomic. The installer expects that some files are present in the container rootfs under /exports, the OCI spec file for running the Runc container and the unit file for Systemd. Both these files are templates and some values are replaced by the installer.
The communication with the Docker registry is done through Skopeo, that is used internally by the atomic cli tool.

The demo shows the current status of the project, these features are not still part of any release and most likely there will be more changes before they will be released.

I wanted to give a try to asciinema, this is the result:

The images used in the demo are maintained here: https://github.com/giuseppe/atomic-oci-containers/

More details on the current design can be found here: https://github.com/projectatomic/atomic/issues/298.

ostree-docker-builder

rpm-ostree, used together with OStree, is a powerful tool to generate immutable images for .rpm based systems, why not to use it for generating Docker images as well?

rpm-ostree already supports the generation of a Docker container tree, that can be feed to Docker almost as it is; ostree-docker-builder instead is a new tool to make this task simpler.

The following JSON description is enough to create an Emacs container using rpm-ostree based on Fedora-22.


{  
    "ref": "fedora-atomic/f22/x86_64/emacs",  
    "repos": ["fedora-22"],  
    "container": true,  
    "packages": ["emacs"]  
}

It references the fedora-22 repo. Be sure that in the same directory as the .json file there is a .repo file which contains the definition for fedora-22fedora-22.repo that looks like:


[fedora-22]  
name=Fedora 22 $basearch  
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-22&arch=$basearch  
enabled=0  
gpgcheck=0  
metadata_expire=1d

These two files are enough to generate an OStree commit, assuming the first file is called emacs.json, and that repo is a valid OStree repository as:


sudo rpm-ostree --repo=repo compose tree emacs.json

At this point, once we get a commit for the fedora-atomic/f22/x86_64/emacs branch we can use ostree-docker-builder to create the Docker image. The code for the program is on github at: https://github.com/giuseppe/ostree-docker-builder.


sudo ostree-docker-builder --repo=repo -c emacs fedora-atomic/f22/x86_64/emacs --entrypoint=/usr/bin/emacs-24.5

ostree-docker-builder accepts some arguments that change how the Dockerfile, which is provided to build the Docker image, is generated.

In the example above we use –entrypoint to set the ENTRYPOINT in the Dockerfile, more information can be found in the Docker documentation: https://docs.docker.com/reference/builder/

If everything works as expected, the image should be ready after that command and we can run it as:


sudo docker run --rm -ti emacs

Repeating the same command twice won’t have any effect, unless –force is specified, if there is no new OStree commit available, ostree-docker-builder stores this information in the image itself using a Docker LABEL.

Tagging

Another feature is the automatic tagging of images, when –tag is specified, the built image will be tagged as the name provided as argument to –tag and automatically pushed to the configured Docker registry.

Advantages of ostree-docker-builder

There are mainly two advantages in using ostree-docker-builder instead of a Dockerfile:

  • The same tool to generate both the OS image and the containers
  • Use OStree to track what files were changed, added or removed. If there are no differences then no image is created

Special thanks to Colin Walters for his suggestions while experimenting ostree-docker-builder and how to take advantage of the OStree checksum.

Summer of Code 2015 for wget

coming as a surprise, this year we have got 4 students to work full-time during the summer on wget. More than all the students who have ever worked for wget before during a Summer of Code!

The accepted projects cover different areas: security, testing, new protocols and some speed-up optimizations. Our hope is that we will be able to use the new pieces as soon as possible, this is why we ask students to keep their code always rebased on top of the current wget development version.

Improve Wget’s security
The project aims at adding HSTS support in wget and enhance FTP security through FTPS.
Speed up Wget’s Download Mechanism
Support two performance enhancements: conditional GET requests and TCP Fast Open.
HTTP/2 support
Basic HTTP/2 support on top of Nghttp2.
FTP Server Test Suite
Augment the tests suite with FTP tests.

Create a QCOW2 image for Fedora 22 Atomic

This tutorial shows how to create a QCOW2 image that can directly imported via virt-install to test out Fedora 22 Atomic starting from a custom OStree repo.

To create the image, we are going to use both rpm-ostree and rpm-ostree-toolbox. Ensure they are installed as well as Docker, libvirtd and Vagrant-libvirt.

The first phase consists in generating the OStree repo that is going to be used by the image. We can use directly the files from the fedora-atomic project as:


git clone --branch=f22 https://git.fedorahosted.org/git/fedora-atomic.git
ostree --repo=repo init --mode=archive-z2
rpm-ostree-toolbox treecompose -c fedora-atomic/config.ini --ostreerepo repo # Creates a new repo

At the end of this process, we have a new OStree repository which contains the tree of a Fedora 22 Cloud.

The second phase is more tricky and requires some manual customizations. Also it requires Docker, Vagrant-libvirt and libvirtd.

To use the repository that we have created in the first phase, we need to spawn an OStree HTTP daemon that will serve the files.

We do it by running:


cd repo
ostree -d trivial-httpd -p - # It will print the TCP port it is listening on

As the comment above says, OStree will print to stout the port where the server is listening on. Take note of it as we will need it later.

We are almost ready to create the QCOW2 image. For the unattended installation of the operating system, we need the fedora-cloud-atomic.ks file from the spin-kickstarts.git project.


git clone --branch=f22 https://git.fedorahosted.org/git/spin-kickstarts.git
cp spin-kickstarts/fedora-cloud-atomic.ks .

At this point, modify ./fedora-cloud-atomic.ks to point to our OStree repository.

This is how I modified the file to point to the OStree repo accessible at http://192.168.125.225:37375/. Use the correct settings for your machine, and the port used to serve the OStree repository that we noted before.


--- spin-kickstarts/fedora-cloud-atomic.ks	2015-04-17 15:41:17.124330230 +0200
+++ fedora-cloud-atomic.ks	2015-04-20 00:52:12.990728422 +0200
@@ -33,14 +33,14 @@
 logvol / --size=3000 --fstype="xfs" --name=root --vgname=atomicos
 
 # Equivalent of %include fedora-repo.ks
-ostreesetup --nogpg --osname=fedora-atomic --remote=fedora-atomic --url=http://kojipkgs.fedoraproject.org/mash/atomic/22/ --ref=fedora-atomic/f22/x86_64/docker-host
+ostreesetup --nogpg --osname=fedora-atomic --remote=fedora-atomic --url=http://192.168.125.225:37375/ --ref=fedora-atomic/f22/x86_64/docker-host
 
 reboot
 
 %post --erroronfail
 # See https://github.com/projectatomic/rpm-ostree/issues/42
 ostree remote delete fedora-atomic
-ostree remote add --set=gpg-verify=false fedora-atomic 'http://dl.fedoraproject.org/pub/fedora/linux/atomic/22/'
+ostree remote add --set=gpg-verify=false fedora-atomic 'http://192.168.125.225:37375/'
 
 # older versions of livecd-tools do not follow "rootpw --lock" line above
 # https://bugzilla.redhat.com/show_bug.cgi?id=964299

Now we are really ready to generate the image:


rpm-ostree-toolbox imagefactory -c fedora-atomic/config.ini -o output -i kvm -k fedora-cloud-atomic.ks --tdl fedora-atomic/fedora-atomic-22.tdl --ostreerepo repo

If everything goes as expected, the image file will be under output/images.


ls output/images
fedora-atomic-f22.qcow2.gz  SHA256SUMS

At this point it can be imported through virt-install as (atomic0cidata.iso is a CD iso which contains the cloud-init initialization data):


gunzip output/images/fedora-atomic-f22.qcow2.gz
virt-install --name f22-cloud --ram 2048 --import --disk path=output/images/fedora-atomic-f22.qcow2 --os-type=fedora-21 --graphics spice --disk path=atomic0cidata.iso,device=cdrom

This command will create a new VM named f22-cloud with 2G of RAM using the QCOW2 image we’ve generated.

Have fun!

How to deploy a WordPress Docker container using docker-compose

These are the steps to setup the current website in a Docker container:



wget -O- https://github.com/docker/compose/releases/download/1.2.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose

mkdir wordpress
cd wordpress

Then create a file fig.yml which contains:


db:
  image: mysql:5.5
  environment:
    MYSQL_ROOT_PASSWORD: "A VERY STRONG PASSWORD"
web:
  image: wordpress:latest
  ports:
    - "80:80"
  links:
    - db:mysql
EOF

This description takes advantage of a Docker feature to bind together two or more containers: Docker links. We use it to make the WordPress container depends on another container with MySQL 5.5.

The docker-compose up command will read fig.yml, download the needed data and deploy the two containers.


/usr/local/bin/docker-compose up

Et voilà, the port 80 of the host which runs the container will be forwarded to the port 80 of the WordPress container.