We currently have two different Linux build machine configurations. One does our 32bit desktop and Android builds, the other does our 64bit desktop builds. This isn’t very efficient because we have to segregate these two types of machine in their own pools. If we have a burst of mobile builds, we end up with a pool of overloaded 32bit machines while our 64bit machines sit idle. We also have one build environment in which all linux based builds need to be done. With boot2gecko and other projects coming online, we need to ensure that we are able to scale to serve these projects while allowing us to manage our build slaves in a sustainable way.
I have been working on a proof of concept for adapting Fedora’s Mock utility to build Firefox, Firefox for Mobile and other projects like boot2gecko in their own sandboxed build environment. Mock is what the Fedora project uses to build every single RPM included in their distribution. Mock separates our build host from our build environment, allows us to scale build hosts any way we desire, allows us to be more efficient and agile, improves security, improves the developer workflow and enables community engagement.
Mock’s Architecture Simplified
When Mock is used, yum and rpm populate the basic sandbox. Each time a command is run, Mock changes the root directory (chroot) to this sandboxed build environment. Mock then drops privileges as requested and runs the command in the sandbox. To speed up the creation of these sandboxes, Mock instructs yum to locally cache the rpm files it downloads. Mock also creates a tarball containing the sandbox after installing the base packages to further speed up sandbox creation. In all, it takes Mock approximately one minute on my local Fusion VM to initialize a build environment once the caches are warmed.
Build Environment Isolation
The host operating system is responsible for providing the kernel, the tools to drive Mock and Mock itself. Because each build environment is built on demand, the libraries that the host operating system provides aren’t important or used. This means that we can take security patches on the build host OS without risking our build environments. We are also able to bring up new fast hardware which may not support our build environment’s OS natively. Mock uses the chroot system call. This call has been around for quite some time and its abilities and flaws are well known. Chroot works on virtual machines which means we can use Mock even if we decide to scale to virtualized hardware.
Efficiency and Agility
Having no build dependencies installed on the build hosts, other than Mock itself, allows us to use the same configuration on our test and build machines. The test machines could even benefit from having Mock by allowing us to run tests against Gnome 2 and Gnome 3 on the same pool of slaves.
A common scenario that I’ve dealt with having to update a library which we link to Firefox. Lets say this we are using libfoo version 1 in Firefox 10. To support a developer landing a change to Firefox 11 that requires libfoo version 2, we currently need to figure out if Firefox 10 can maintain binary compatibility when built with libfoo version 2 or figure out how to install version 2 in a separate location and have only Firefox 11 and newer use libfoo version 2. Using Mock, we would create a new build environment that only has libfoo version 2. Because libfoo version 1 is unavailable to Firefox 11 builds and libfoo version 2 is unavailable to Firefox 10, we don’t risk depending on a new version of libfoo in Firefox 10 and we don’t risk silently falling back to using libfoo version 1 on Firefox 11.
Mobile moves quickly, often needing a completely unique build environment. When we split each mobile project out to its own Mock build environment, we are able to set up new environments significantly quicker because we are able to limit the scope of changes and no longer risk breaking Firefox by changing anything related to mobile. We can also allow mobile projects to use much newer Fedora software in their build environment while still running the more stable CentOS on our build hosts. This means we enable mobile developers to use modern software while avoiding rebuilding a slew of packages in an alternate location.
I understand that chroots are not a perfect sandboxing tool. I think we need to make sure that we don’t make perfect the enemy of good. We currently have no real sandboxing on our build slaves. Mock will enable us to run all developer submitted commands in an unprivileged chroot that is rebuilt for every build. Mock allows us to keep our system software updated on the build host to lessen our exposure to security vulnerabilities.
A special case in our infrastructure is Try. Each Try build is currently able to modify the system in ways that we might not want it to. As a result, we currently have a completely separate pool of machines which run our Try builds. Mock allows us to sandbox all developer submitted code into a disposable sandbox. Because all of the developer submitted code is limited to the sandbox, we can start looking at merging the Try and production pools. The failure mode here is that the developer is able to exploit a kernel bug to gain root permissions in their sandbox. Once a user has root in the sandbox they would be able to chroot back out of the sandbox. This will be important when we discuss merging our Try and production pools.
Each build environment will install the exact set of packages required for the build. When a new dependency is required, we will explicitly add it to the list of dependencies. This means we won’t be surprised when Firefox starts depending on another library or tool. We are also able to easily move actively developed branches of Firefox forward without taking risky changes to the build environment of shipping versions of Firefox. This is increasingly important with the ESR (Extended Support Release)/ release. Using Mock, we can continue to build in the exact same environment we use today for the rest of the ESR release even though we are building a 10 month newer version of Firefox with different dependencies on the same build host. Having a single pool of slaves that can do all of our Linux based builds without having to figure out alternate install locations for each dependency is a major scalability win in my books.
Developer workflow improvements
Upgrading a library on our build slaves is currently a pain for developers and releng alike. Setting up a local copy of our Linux build machines is a pain. Mock makes this easier by allowing developers to use their existing machines to exactly replicate our official build environments. Setting up a copy of our build environment on their machine would becomes trivially easy. Developers could test their code changes in an exact match of our production build environment. Furthermore, with a tool like mozharness, a developer would be able to test an entire build using the exact same bits we use in production with the exact same commands and flow control.
Mock allows us to engage the community more effectively. Community members are able to work on our build machine configuration without impediment and would have access to the exact tools and packages that we use to build Firefox and Mobile. When a community member finds or fixes a bug in our configuration or wants to upgrade a dependency, they are able to do so and submit patches for review. This is a lot easier than our current process of signing out a slave for them, adding them to our vpn, letting them do their work followed by removing them from our vpn, reimaging the slave and setting it up again to be in production. Project like SeaMonkey will be able to match official Firefox build environments with a lot less work required.
Mock is a very useful tool. Using it enables us to handle developer requests quicker, engage the community, improve developer workflow, improve security and reduce risk of breaking shipping versions of Firefox when moving development versions forward. There are some changes that I’d like to make to the proof of concept before calling mock_mozilla complete, mainly how the driver script (mock_mozilla.py) parses the command line arguments and how build environment locking works.
If you’d like to take a look at the code for yourself, please feel free to check it out on github! https://github.com/jhford/mock_mozilla.
Before you start running code in a Mock sandbox, you need to intialize it. Initializing cleans out the old contents and sets up the base packages specified in the sandbox config file.
~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --init INFO: mock_mozilla.py version 1.1.17 starting... INFO: State Changed: init plugins INFO: selinux disabled INFO: State Changed: start INFO: State Changed: lock buildroot INFO: State Changed: clean INFO: State Changed: unlock buildroot INFO: State Changed: init INFO: State Changed: lock buildroot INFO: Mock Version: 1.1.17 INFO: Mock Version: 1.1.17 INFO: Mock Version: 1.1.17 INFO: calling preinit hooks INFO: enabled root cache INFO: State Changed: unpacking root cache INFO: enabled yum cache INFO: State Changed: cleaning yum metadata INFO: enabled ccache INFO: State Changed: running yum INFO: State Changed: unlock buildroot INFO: State Changed: end
Mock uses yum to install packages into the sandbox, but that doesn’t mean that we are limited to using yum and rpm to populate tools.
~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --shell "zip --version" INFO: mock_mozilla.py version 1.1.17 starting... INFO: State Changed: init plugins INFO: selinux disabled INFO: State Changed: start INFO: State Changed: lock buildroot INFO: State Changed: shell /bin/sh: zip: command not found INFO: State Changed: unlock buildroot ~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --install zip INFO: mock_mozilla.py version 1.1.17 starting... INFO: State Changed: init plugins INFO: selinux disabled INFO: State Changed: start INFO: Mock Version: 1.1.17 INFO: Mock Version: 1.1.17 INFO: Mock Version: 1.1.17 INFO: State Changed: lock buildroot INFO: installing package(s): zip ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: zip x86_64 3.0-3.fc15 fedora 251 k Transaction Summary ================================================================================ Install 1 Package(s) Total size: 251 k Installed size: 770 k Installed: zip.x86_64 0:3.0-3.fc15 INFO: State Changed: unlock buildroot INFO: State Changed: end ~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --shell "zip --version" INFO: mock_mozilla.py version 1.1.17 starting... INFO: State Changed: init plugins INFO: selinux disabled INFO: State Changed: start INFO: State Changed: lock buildroot INFO: State Changed: shell Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license. This is Zip 3.0 (July 5th 2008), by Info-ZIP. Currently maintained by E. Gordon. Please send bug reports to the authors using the web page at www.info-zip.org; see README for details. Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip, as of above date; see http://www.info-zip.org/ for other sites. Compiled with gcc 4.6.0 20110205 (Red Hat 4.6.0-0.6) for Unix (Linux ELF) on Feb 8 2011. Zip special compilation options: USE_EF_UT_TIME (store Universal Time) SYMLINK_SUPPORT (symbolic links supported) LARGE_FILE_SUPPORT (can read and write large files on file system) ZIP64_SUPPORT (use Zip64 to store large files in archives) UNICODE_SUPPORT (store and read UTF-8 Unicode paths) STORE_UNIX_UIDs_GIDs (store UID/GID sizes/values using new extra field) UIDGID_NOT_16BIT (old Unix 16-bit UID/GID extra field not used) [encryption, version 2.91 of 05 Jan 2007] (modified for Zip 3) Encryption notice: The encryption code of this program is not copyrighted and is put in the public domain. It was originally written in Europe and, to the best of our knowledge, can be freely distributed in both source and object forms from any country, including the USA under License Exception TSU of the U.S. Export Administration Regulations (section 740.13(e)) of 6 June 2002. Zip environment options: ZIP: [none] ZIPOPT: [none] INFO: State Changed: unlock buildroot
Mock allows us to run developer submitted code as an unprivileged user to isolate their code and files from the rest of the machine.
~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --shell "whoami" INFO: mock_mozilla.py version 1.1.17 starting... INFO: State Changed: init plugins INFO: selinux disabled INFO: State Changed: start INFO: State Changed: lock buildroot INFO: State Changed: shell root INFO: State Changed: unlock buildroot ~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --unpriv --shell "whoami" INFO: mock_mozilla.py version 1.1.17 starting... INFO: State Changed: init plugins INFO: selinux disabled INFO: State Changed: start INFO: State Changed: lock buildroot INFO: State Changed: shell mock_mozilla INFO: State Changed: unlock buildroot
Putting all of the above together, I have written a proof of concept script and recorded a screencast of it building Firefox in a Mock sandbox. I’ve cut the middle bit out because this takes a while on my Macbook-based VM.