Written by We currently have two different Linux build machine configurations. One does our 32bit desktop and Android builds, the other does our 64bit desktop builds. This isn’t very efficient because we have to segregate these two types of machine in their own pools. If we have a burst of mobile builds, we end up with a pool of overloaded 32bit machines while our 64bit machines sit idle. We also have one build environment in which all linux based builds need to be done. With boot2gecko and other projects coming online, we need to ensure that we are able to scale to serve these projects while allowing us to manage our build slaves in a sustainable way.
I have been working on a proof of concept for adapting Fedora’s Mock utility to build Firefox, Firefox for Mobile and other projects like boot2gecko in their own sandboxed build environment. Mock is what the Fedora project uses to build every single RPM included in their distribution. Mock separates our build host from our build environment, allows us to scale build hosts any way we desire, allows us to be more efficient and agile, improves security, improves the developer workflow and enables community engagement.
Mock’s Architecture Simplified
When Mock is used, yum and rpm populate the basic sandbox. Each time a command is run, Mock changes the root directory (chroot) to this sandboxed build environment. Mock then drops privileges as requested and runs the command in the sandbox. To speed up the creation of these sandboxes, Mock instructs yum to locally cache the rpm files it downloads. Mock also creates a tarball containing the sandbox after installing the base packages to further speed up sandbox creation. In all, it takes Mock approximately one minute on my local Fusion VM to initialize a build environment once the caches are warmed.
Build Environment Isolation
The host operating system is responsible for providing the kernel, the tools to drive Mock and Mock itself. Because each build environment is built on demand, the libraries that the host operating system provides aren’t important or used. This means that we can take security patches on the build host OS without risking our build environments. We are also able to bring up new fast hardware which may not support our build environment’s OS natively. Mock uses the chroot system call. This call has been around for quite some time and its abilities and flaws are well known. Chroot works on virtual machines which means we can use Mock even if we decide to scale to virtualized hardware.
Efficiency and Agility
Having no build dependencies installed on the build hosts, other than Mock itself, allows us to use the same configuration on our test and build machines. The test machines could even benefit from having Mock by allowing us to run tests against Gnome 2 and Gnome 3 on the same pool of slaves.
A common scenario that I’ve dealt with having to update a library which we link to Firefox. Lets say this we are using libfoo version 1 in Firefox 10. To support a developer landing a change to Firefox 11 that requires libfoo version 2, we currently need to figure out if Firefox 10 can maintain binary compatibility when built with libfoo version 2 or figure out how to install version 2 in a separate location and have only Firefox 11 and newer use libfoo version 2. Using Mock, we would create a new build environment that only has libfoo version 2. Because libfoo version 1 is unavailable to Firefox 11 builds and libfoo version 2 is unavailable to Firefox 10, we don’t risk depending on a new version of libfoo in Firefox 10 and we don’t risk silently falling back to using libfoo version 1 on Firefox 11.
Mobile moves quickly, often needing a completely unique build environment. When we split each mobile project out to its own Mock build environment, we are able to set up new environments significantly quicker because we are able to limit the scope of changes and no longer risk breaking Firefox by changing anything related to mobile. We can also allow mobile projects to use much newer Fedora software in their build environment while still running the more stable CentOS on our build hosts. This means we enable mobile developers to use modern software while avoiding rebuilding a slew of packages in an alternate location.
Security
I understand that chroots are not a perfect sandboxing tool. I think we need to make sure that we don’t make perfect the enemy of good. We currently have no real sandboxing on our build slaves. Mock will enable us to run all developer submitted commands in an unprivileged chroot that is rebuilt for every build. Mock allows us to keep our system software updated on the build host to lessen our exposure to security vulnerabilities.
A special case in our infrastructure is Try. Each Try build is currently able to modify the system in ways that we might not want it to. As a result, we currently have a completely separate pool of machines which run our Try builds. Mock allows us to sandbox all developer submitted code into a disposable sandbox. Because all of the developer submitted code is limited to the sandbox, we can start looking at merging the Try and production pools. The failure mode here is that the developer is able to exploit a kernel bug to gain root permissions in their sandbox. Once a user has root in the sandbox they would be able to chroot back out of the sandbox. This will be important when we discuss merging our Try and production pools.
Correctness
Each build environment will install the exact set of packages required for the build. When a new dependency is required, we will explicitly add it to the list of dependencies. This means we won’t be surprised when Firefox starts depending on another library or tool. We are also able to easily move actively developed branches of Firefox forward without taking risky changes to the build environment of shipping versions of Firefox. This is increasingly important with the ESR (Extended Support Release)/ release. Using Mock, we can continue to build in the exact same environment we use today for the rest of the ESR release even though we are building a 10 month newer version of Firefox with different dependencies on the same build host. Having a single pool of slaves that can do all of our Linux based builds without having to figure out alternate install locations for each dependency is a major scalability win in my books.
Developer workflow improvements
Upgrading a library on our build slaves is currently a pain for developers and releng alike. Setting up a local copy of our Linux build machines is a pain. Mock makes this easier by allowing developers to use their existing machines to exactly replicate our official build environments. Setting up a copy of our build environment on their machine would becomes trivially easy. Developers could test their code changes in an exact match of our production build environment. Furthermore, with a tool like mozharness, a developer would be able to test an entire build using the exact same bits we use in production with the exact same commands and flow control.
Mock allows us to engage the community more effectively. Community members are able to work on our build machine configuration without impediment and would have access to the exact tools and packages that we use to build Firefox and Mobile. When a community member finds or fixes a bug in our configuration or wants to upgrade a dependency, they are able to do so and submit patches for review. This is a lot easier than our current process of signing out a slave for them, adding them to our vpn, letting them do their work followed by removing them from our vpn, reimaging the slave and setting it up again to be in production. Project like SeaMonkey will be able to match official Firefox build environments with a lot less work required.
Conclusion
Mock is a very useful tool. Using it enables us to handle developer requests quicker, engage the community, improve developer workflow, improve security and reduce risk of breaking shipping versions of Firefox when moving development versions forward. There are some changes that I’d like to make to the proof of concept before calling mock_mozilla complete, mainly how the driver script (mock_mozilla.py) parses the command line arguments and how build environment locking works.
If you’d like to take a look at the code for yourself, please feel free to check it out on github! https://github.com/jhford/mock_mozilla.
Demonstrations
Before you start running code in a Mock sandbox, you need to intialize it. Initializing cleans out the old contents and sets up the base packages specified in the sandbox config file.
~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --init INFO: mock_mozilla.py version 1.1.17 starting... INFO: State Changed: init plugins INFO: selinux disabled INFO: State Changed: start INFO: State Changed: lock buildroot INFO: State Changed: clean INFO: State Changed: unlock buildroot INFO: State Changed: init INFO: State Changed: lock buildroot INFO: Mock Version: 1.1.17 INFO: Mock Version: 1.1.17 INFO: Mock Version: 1.1.17 INFO: calling preinit hooks INFO: enabled root cache INFO: State Changed: unpacking root cache INFO: enabled yum cache INFO: State Changed: cleaning yum metadata INFO: enabled ccache INFO: State Changed: running yum INFO: State Changed: unlock buildroot INFO: State Changed: end
Mock uses yum to install packages into the sandbox, but that doesn’t mean that we are limited to using yum and rpm to populate tools.
~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --shell "zip --version"
INFO: mock_mozilla.py version 1.1.17 starting...
INFO: State Changed: init plugins
INFO: selinux disabled
INFO: State Changed: start
INFO: State Changed: lock buildroot
INFO: State Changed: shell
/bin/sh: zip: command not found
INFO: State Changed: unlock buildroot
~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --install zip
INFO: mock_mozilla.py version 1.1.17 starting...
INFO: State Changed: init plugins
INFO: selinux disabled
INFO: State Changed: start
INFO: Mock Version: 1.1.17
INFO: Mock Version: 1.1.17
INFO: Mock Version: 1.1.17
INFO: State Changed: lock buildroot
INFO: installing package(s): zip
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
zip x86_64 3.0-3.fc15 fedora 251 k
Transaction Summary
================================================================================
Install 1 Package(s)
Total size: 251 k
Installed size: 770 k
Installed:
zip.x86_64 0:3.0-3.fc15
INFO: State Changed: unlock buildroot
INFO: State Changed: end
~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --shell "zip --version"
INFO: mock_mozilla.py version 1.1.17 starting...
INFO: State Changed: init plugins
INFO: selinux disabled
INFO: State Changed: start
INFO: State Changed: lock buildroot
INFO: State Changed: shell
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon. Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.
Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip,
as of above date; see http://www.info-zip.org/ for other sites.
Compiled with gcc 4.6.0 20110205 (Red Hat 4.6.0-0.6) for Unix (Linux ELF) on Feb 8 2011.
Zip special compilation options:
USE_EF_UT_TIME (store Universal Time)
SYMLINK_SUPPORT (symbolic links supported)
LARGE_FILE_SUPPORT (can read and write large files on file system)
ZIP64_SUPPORT (use Zip64 to store large files in archives)
UNICODE_SUPPORT (store and read UTF-8 Unicode paths)
STORE_UNIX_UIDs_GIDs (store UID/GID sizes/values using new extra field)
UIDGID_NOT_16BIT (old Unix 16-bit UID/GID extra field not used)
[encryption, version 2.91 of 05 Jan 2007] (modified for Zip 3)
Encryption notice:
The encryption code of this program is not copyrighted and is
put in the public domain. It was originally written in Europe
and, to the best of our knowledge, can be freely distributed
in both source and object forms from any country, including
the USA under License Exception TSU of the U.S. Export
Administration Regulations (section 740.13(e)) of 6 June 2002.
Zip environment options:
ZIP: [none]
ZIPOPT: [none]
INFO: State Changed: unlock buildroot
Mock allows us to run developer submitted code as an unprivileged user to isolate their code and files from the rest of the machine.
~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --shell "whoami" INFO: mock_mozilla.py version 1.1.17 starting... INFO: State Changed: init plugins INFO: selinux disabled INFO: State Changed: start INFO: State Changed: lock buildroot INFO: State Changed: shell root INFO: State Changed: unlock buildroot ~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --unpriv --shell "whoami" INFO: mock_mozilla.py version 1.1.17 starting... INFO: State Changed: init plugins INFO: selinux disabled INFO: State Changed: start INFO: State Changed: lock buildroot INFO: State Changed: shell mock_mozilla INFO: State Changed: unlock buildroot
Putting all of the above together, I have written a proof of concept script and recorded a screencast of it building Firefox in a Mock sandbox. I’ve cut the middle bit out because this takes a while on my Macbook-based VM.
Mock is definitely a nice tool. I just wish that I could find a Mock equivalent for Debian-based distros and a debootstrap equivalent for RPM-based distros so I could do “one guy in a basement” release testing and package building for Debian, Ubuntu, Fedora, and OpenSuSE using just my Gentoo box, some unit tests, and some scripting.
Sadly, if they exist, they’re not turning up in the Google searches I’ve thought to try.
(debootstrap is included in the Portage tree and makes setting up a base debian chroot as simple as
sudo debootstrap squeeze /chroots/debian_squeeze.)Stephan, I’ve been thinking about it and I am pretty sure that it would be possible to add apt/deb support to mock. I too would love to be able to test Debian derived operating systems using this tool and is something I’d like to investigate.
There is http://people.redhat.com/~rjones/febootstrap/ , but aiui, its a tool for creating appliances instead of general purpose environments.
This is a really great post. I hadn’t considered that we could use Mock on the test slaves at all, let alone that it would let us unify the host OS for build & test machines.
Oh, and I agree that we shouldn’t think of Mock as a great security tool, but I’d rather have an up-to-date host OS and builds running in a chroot than an ancient host with non-chrooted builds.
For people running OSes that can’t just use your Mock setup, it seems like you could at least now give developers access only to a Mock chroot on a shared “developer punching bag” mini-pool with multiple Mocks lying around.
Merging the try builders with the production builders doesn’t sound so nice from a security point of view — you’re merging L1 access jobs with L3 access jobs. Have you considered partitioning pools by the access level of the pusher? Then production jobs and most try jobs could share a large pool, and the subset of try jobs pushed by L1 committers would share a small pool. (To avoid having two classes of performance, it would be nice if the buildbot scheduling doohickey could dynamically adjust the pool sizes. Except you’d probably want to wipe the machines when transitions L1 -> L3. If that isn’t too horribly expensive?)
I’m ignoring L2 because it’s kind of pointless. Though it might make sense to split out production builds (beta/aurora/nightly) from mozilla-central and friends.
Do you have instructions for how I could try this out? I’m on Fedora 14. I cloned your github thing, I have mock installed already, I installed fedora-packager. I did ./autogen.sh, make rpm, and installed the resulting rpm. I ran sh poc.sh, but it keeps wanting me to authenticate as root, and it’s looking for /builds/targets/mozilla-f15-x86_64//builds//mozilla-central/.mozconfig. Do you have complete instructions anywhere, or should I just keep plowing through?
Thanks!
I haven’t confirmed, but I think that any linux with chroot(2), yum, rpm and mock_mozilla should be able to run mock. As for non-Linux OSes, you’d need to install something like CentOS.
The goal of mock is to have the system setup in a way that there is absolutely nothing that is shared between the build environment and the host. Because the environment is rebuilt per-build, nothing would be shared outside of the sandbox.
I don’t have a full set of instructions for setting up mock_mozilla because its very early stage. I’ll spin up a VM to check that the build process works properly. I do know that I run the configure stuff, but use the test.sh script inside of the repo for building and installing. A lot of this is hacky code I am using to make development faster. Please file issues in github with any issues you have, and a solution if you have it! If you don’t do github, I’m in #build and i sometimes read blog comments.
Thanks for your interest in testing. I
It looks like I was just about there; I can now run poc.sh and it builds the browser. There’s something messy about the initial setup and authorization, but I think once I was in the mock_mozilla group (can’t remember if one of your scripts did that or if I did it manually), it all worked!
> The goal of mock is to have the system setup in a way that there is absolutely nothing that is shared between the build environment and the host. Because the environment is rebuilt per-build, nothing would be shared outside of the sandbox.
I assume this was in response to my comment on splitting up L1 and L3 access instead of production vs Try access? Sharing nothing is great for repeatable builds, but chroot still seems to me (IMHO) to be pretty weak as a security boundary. I understand the appeal of having a single pool of Linux slaves, but breaking out of a chroot just isn’t that hard, and I’d like the barrier to getting L1 access to be as low as possible. If the damage an L1 committer can do is limited to a separate pool of physical machines, then we can hand out those rights more freely. (Or at least separate VMs, which are far better for security than chroot jails, but still aren’t really meant for that purpose.)