This means you no longer have to download Gaia sources and build it yourself to test a site on B2G or check out our progress. Building Gaia is easy if you already have a development environment, but setting up a development environment can be pretty painful, especially on Windows. As of last week, our nightly builds now include a fully built Gaia profile and a small wrapper program to allow launching the Firefox OS simulator with one click. This is especially useful on Mac, where launching the B2G directly in a terminal instead of through Finder causes input to be redirected to the terminal.

These builds end up here: http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/latest-mozilla-central/

On Windows and Linux, downloading the appropriate package, extracting it then running “b2g/b2g” will launch the simulator. On Mac, you’ll need to mount the DMG then copy the B2G.app bundle to a writable location. The App bundle needs to be on a writable location so the profile can be modified by Gecko.

Because we use a wrapper program to launch B2G, we’ve moved the actual ‘b2g’ and ‘b2g.exe’ files to ‘b2g-bin’ and ‘b2g-bin.exe’, respectively. If you need to launch b2g directly, please use the new b2g-bin name. To see where the output of the B2G process goes on Mac, launch the ‘Console’ application

Note: there is a crash in libxul.dll on Windows that is affecting startup.

Integrating with Buildbot automation

Mock integrates with Buildbot through the MockInit, MockInstall, MockCommand and MockProperty classes. The first two commands are used to configure and create the environment while the third and fourth are used to execute commands inside of a build environment.

MockInit is a basic wrapper for calling mock --init -r target_name. It is provided to encapsulate the logic for this relatively simple operation. Internally, initializing an environment means making sure that the environment is completely clean and has been recreated based on the contents of the root cache and the base packages specified in the mock config file.

MockInstall is also another basic wrapper. MockInstall internally calls mock -r target_name --install package1 package2 packageN. This step will install into the specified target environment the packages needed and any the packages requested depend on. This requires all build tools and dependencies be packaged and added to a Yum repository. Doing this is highly desirable regardless of whether or not Mock is being used.

MockCommand is derived from the standard ShellCommand as intended to operate like the base class. When in non-Mock mode (i.e. use_mock is set to False), this command does nothing to the configuration options and passes all calls straight to the base ShellCommand. When in Mock mode, the command runs its magic method to modify the command line to properly pass the requested buildbot environment through to the Mock call as well as passing the correctly modified command line given the working directory requested (using build environment paths) and the target that should be used.

MockProperty is to SetProperty what MockCommand is to ShellCommand with the important distinction that MockProperty inherits singly from MockCommand and duplicates logic from SetProperty.

In the current implementation, Mock has /builds/ bind mounted into the build environment. This means that all paths under /builds/ have the exact same paths in the build environment and on the build host. If the build process is substantially reworked to use a scripting framework other than Buildbot, it might make sense to stop making the /builds/ directory available inside of the build root. All commands which are “build system” commands are run under a MockCommand with use_mock set to true. Currently, there are no uploads or X11 dependent jobs run under mock. I have a blog post that details X11 access and uploads could be fixed by bind-mounting the host’s ~/.ssh directory into the build environment.

RPM and Yum infrastructure

Because all dependencies are fetched and installed through Yum+RPM on each build, it is necessary to package all tools as RPMs. How RPMs are built and Yum repositories are created is outside the scope of this document. The current setup is for Mozilla to internally mirror the host OS and the build environment OS as well as a custom repo for each distribution. Ideally, the mirrored copy of the repositories would be completely unaltered. I haven’t experimented with overriding core packages yet, but we should be fine to put newer packages that override OS packages in the custom Mozilla repository. As for building RPMs, I’ve been using spec files in my repository to build the rpms.

This repository has a directory for each distributions for which custom packages are generated, currently only ‘centos6′. Inside the distribution directory are directories for each package. There are no directories for each architecture, since rpm has facilities for specifying which architectures are valid for a given package. Each package directory contains small patches and source files needed to build the package, a tooltool manifest for larger files and a symlink to a file called “actions.sh”. This is a symlink to a script in a hidden directory at the root of the repository. In this script, there are commands for the following actions:

• fetch — use tooltool to fetch the sources
• store — add the specified file to the tooltool manifest
• build — perform a simple rpmbuild on the given spec file
• mock_build — perform a mock build of the package using upstream’s mock package

It is the responsibility of the package author to add sources, specs and manifests to the repository and upload the files to the tooltool resource host. The mock_build command can be invoked with the syntax ./actions.sh mock_build fedora-16-x86_64 x86_64 my_package.spec. The SRPMs created should be included in the list of rpm files included in Yum repository creation, since they are the canonical source of what the binary packages were built from.

Creating a new target

In the case that a new build environment configuration is needed, I’d recommend starting with an existing mock_mozilla configuration. The values that you’re likely going to be working with are the root name, arch list, base package list, bind mount and repository list. Deploying the cfg file is required and should be done through something like puppet.

Development

A release of mock_mozilla can be generated by running

git clone git://github.com/jhford/mock_mozilla.git
cd mock_mozilla
./autogen.sh
make srpm # this is to get the tarball, don't care about the srpm created here
mock --buildsrpm --spec mock_mozilla.spec --sources . --resultdir srpm --target x86_64
mock -r epel-6-x86_64 mock_mozilla-1.0.1-1.el6.src.rpm --resultdir x86_64 --target x86_64

The code to mock_mozilla is on github.

Tooltool basics

Tooltool is a client side program written in Python that uses a file manifest in concert with HTTP servers to materialize large binary payloads for use in a job.  The manifests are JSON files which list details of individual files.  Each file is represented in the JSON by a dictionary with the keys “filename”, “digest”, “size” and “algorithm”. An example is located here.  The current JSONEncoder and JSONDecoder derived classes only understand how to work with these keys. Making the JSON encoder and decoder work with an extensible version of the manifests should not be difficult.

Tooltool fetch API

When Tooltool needs to download a file from the file server it does so by creating a URL.  In the current implementation, there is a single base url that is used to construct the URLs that are to be fetched.  A good change to make would be to convert this single URL to a list of possible base urls.  The address of the file to fetch is generated using a string concatenation of the base URL, a slash, the hashing algorightm’s name, another slash and the full hash value represented in hexadecimal. Given a base url of “http://files.r.us:8080/tooltool” and a file that has a SHA512 hash value of 0123456789abcdef, Tooltool will try to fetch  “http://files.r.us:8080/tooltool/sha512/0123456789abcdef”.

There is no server component for Tooltool yet.  As a result, the file server is currently implemented as a simple directory on an HTTP host that has the correct directory structure for responding to requests.  I’ve started working on a server side component but it isn’t finished. The server side component would allow for easy uploads by developers, easy listing of contents on the server and a way to store files in a nicer way on the server’s file system.

Using Tooltool

Tooltool has four commands presently: list, validate, add and fetch.  There are global options and command arguments.  All terminal interactions after the option parser finishes is done through the Python logging API.  The default is to print logging.INFO and higher messages.  Currently, the following global options exist:

• -q/--quiet tells Tooltool to print only logging.ERROR and higher messages
• -v/--verbose specifies to print logging.INFO and higher
• -m/--manifest <file> instructs Tooltool to reference a manifest file located at the specified path
• -d/--algorithm <algorithm> instructs Tooltool to use the specified algorithm
• -o/--overwrite tells Tooltool to overwrite a local file if the filename matches the manifest but the hash value is different to the manifest
• --url specifies the base url to be used for remote operations

Listing and Validating

The two most basic commands list a manifest and validate the local files against the manifest.  The list command lists out all of the files in the manifest as well as whether they are present and/or valid.  The return code from listing is zero unless there was an error in listing the files.  Absent or invalid files will still result in an exit code of zero if there was no error in the listing process.  The validate command is used to check if all the files in the manifest are present and valid.  The exit code for validating is zero if all files in the manifest are present and their hash matches the manifest.  It is non-zero if any file is missing locally or the file does not have the same hash as the manifest.

Listing and validating a Tooltool manifest. Note the exit codes

Fetching

The fetch command is used to materialize files locally.  Before fetching a file from a remote host, Tooltool will validate local files which match the filename specified in the manifest.  The default behaviour when a local file matches the filename but not the hash value is to exit with a non-zero exit code.  If –o or –overwrite is specified on the command line Tooltool will overwrite the local file without confirmation with the remote file.  The local file will be truncated as soon as Tooltool attempts to start writing the remote file locally.

Fetching a file that exists locally with contents differing from manifest. Note the difference in behaviour when using --overwrite

Adding

Files are added to manifests with the add command. This command looks for a manifest using either the default name of manifest.tt or by the value specified by the -m/–manifest command line option. If the manifest does not exist already on the file system, a new one will be created to store the first file given. An error message will be displayed if a file is added a second time, regardless of whether or not the local contents are the same as what is in the manifest. There is currently no logic to remove a file from a manfiest or overwrite a manifest entry.

Future direction and contributions

Tooltool is a generic lookaside cache. My intention is to keep it as general as possible by not including logic to deal with payloads. We currently include a bootstrap script in the b2g manifests that understands how to take the rest of the payload and set up a working toolchain. Using a bootstrap script means that Tooltool can be tool agnostic while still allowing complex operations on the fetched tools. A standardized bootstrap script name and interface that is called by Tooltool might make sense. I’d also like to finish the server-side component that has an interface to upload files with and a method for storing files in a less obtuse method than just mirroring the API paths. A command for removing files from a manifest would be helpful, as would the ability to update a manifest with the contents of the directory as they exist right now. Another really cool feature would be the ability to configure a system wide cache directory where files are downloaded to. Once the server component is working, I’d like to add a way for servers to automatically sync their file stores when they are asked for a file that exists on another server but not locally.

Tooltool is GPLv2 and the source is available on github. The best way to contribute code is to send a pull request on github. Things are more likely to be merged if they pass the whole test suite (make check), have tests, improve Tooltool and don’t make Tooltool overly specific.

Caymans at Dealers:

• Sonnen (Mill Valley)
• Rector (Burlingame)
• Michael Stead Porsche (Walnut Creek)
• Stevens Creek
• Carlsen (Redwood City)
• Monterey
• Livermore
• Fremont
• Michael (Fresno)
• Bakersfield

Yahoo Autos for Cayman + Cayman S
Edmunds for Cayman + Cayman S

If you find any issues with either of these toolchains, please let us know as soon as possible by filing a mozilla.org::Release Engineering bug and CCing me (:jhford)

To echo my message to a couple news groups, these are some things you might like to know:

1. you need http://hg.mozilla.org/mozilla-central/rev/b687cabf75a1 merged in to your branch to get the biggest build time improvement
2. pushing mozilla-aurora, mozilla-beta, mozilla-esr10 or mozilla-1.9.2 code to try will now attempt a build on 10.7 hardware.  Doing this is not suggested because your build may fail to build at all and it definitely will not match what we currently build on mozilla-aurora and older.
3. These builds are able to run on 10.5
4. we are using gcc-4.2 from xcode 4.1 running on OS X 10.7.2 right now and building against the 10.6sdk
5. the machines are macmini5,3 (quad-i7 2.0GHz, 8GB ram, 2x500GB 7200RPM in Raid0, exclusively intel graphics)
6. shark builds don’t work, bug 723301

I don’t know much about epub, but I do know that it is supported on the iphone and has better text layout than US letter sized PDF files on the device. Python docs are generated using a tool called Sphinx and it turns out that they have an epub builder. Important to note is that this epub builder is experimental.

To generate my epub file, I followed the directions in the Python-2.7.2/Doc/README.txt file, using a builder name of epub instead of one of the listed ones. The result was that I ran these commands:

wget http://www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz
tar zxf Python-2.7.2.tgz
cd Python-2.7.2/Doc
virtualenv doc-tools && source doc-tools/bin/activate
pip install sphinx jinja2 pygments
python tools/sphinx-build.py -bepub . build/epub

Doing this resulted in a large number of html files and an epub file, build/epub/Python.ebub. I opened the file with itunes open -a iTunes build/epub/Python.epub and synced my iphone.

So far my experience is that ibooks is very slow when working with this file. The file is twice as large as the complete works of Shakespeare and has a lot more markup! Most of the things I’d expect to be links are links and the indexes seem to be working just fine. Ibooks (the iphone epub reader) even shows the correct table of contents. Code samples are legible, but are a nested panning area, which is awkward when the same gesture flips to the next page.

If you are feeling lazy, here is a copy.

These builds also represent a huge step forward in making our infrastructure more responsive to developer requests.  We are decoupling our build host OS from our build environment using a fork of Fedora’s Mock tool.  We are checking in a tool manifest that empowers community contributed toolchains to be deployed with tooltool.  We are also using a new set of puppet manifests and a bunch of brand new HP build machines that are pretty fast.  These machines are set up using kickstart instead of a reference image, which means that when we want to rev our hardware, we aren’t tied to a particular model of hardware or even OS version.

These changes represent the direction we are moving in terms of our linux build environment.  A longer term goal is to move our Firefox desktop and mobile builds to mock build slaves for the same reasons we are doing b2g with mock.

Because of the shear number of brand new systems in play here there might be a few teething pains.  I’ve been running the full pool of 42 build machines on my staging master for 2 weeks with the code that landed today and for the last 3 weeks while in development and nothing has inexplicably failed.  Please don’t hesitate to file a bug if you see a problem with this new setup but make sure you CC me on it.

I’ll be writing up a more detailed blog post about what I did and the new tools written as time permits.  I am now focusing my attention towards getting our new quad-core Mac Mini machines into production, which will cut our nightly and release mac build times down from 4+ hours to 1:20.

I tried specifying the hostname in my DISPLAY environment variable but got the following error:

bash-4.2# DISPLAY=localhost:0 xeyes
Error: Can't open display: localhost:0

I did a bit of searching and found that a lot of people suggested that I run xhost +. I found that had no effect on what I was trying. Even more searching and I found out that Fedora and Centos have been invoking X11 with the --nolisten tcp option for some time. This is a great default, but not for what I want to do.

In my search, I found that there were many, many ways with which to specify how X11 is launched. All of these are part of the GDM configuration. GDM is the login system in Gnome. Over the years, there have been many substantial rewrites of GDM, each one seemingly completely changing where the options for the X server reside.

In the end, I found that the relevant file on CentOS 6.2 was /etc/gdm/gdm.schemas. The relevant passage is

<schema>
      <key>security/DisallowTCP</key>
      <signature>b</signature>
      <default>true</default>
</schema>

I switched the false to true and retried running my command, mock_mozilla -r test-f16-b --shell "/usr/bin/env DISPLAY=localhost:0 glxgears" which resulted in a working GLXGears!

The keen observer will notice that this screenshot has a fairly low frame rate. Even though this number is low, the build host gets approximately half this framerate outside of the chroot. It looks like the VMware 3D drivers are slower than pure software. Since glxgears doesn’t actually tell you whether or not you have acceleration, I ran glxinfo loking for the ‘direct rendering’ line.

libGL: OpenDriver: trying /usr/lib/dri/swrast_dri.so
libGL error: dlopen /usr/lib/dri/swrast_dri.so failed (/usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory)
libGL error: unable to load driver: swrast_dri.so
libGL: OpenDriver: trying /usr/lib/dri/swrastg_dri.so
libGL error: dlopen /usr/lib/dri/swrastg_dri.so failed (/usr/lib/dri/swrastg_dri.so: cannot open shared object file: No such file or directory)
libGL error: unable to load driver: swrastg_dri.so
libGL error: reverting to indirect rendering
direct rendering: No (If you want to find out why, try setting LIBGL_DEBUG=verbose)

Looks like the user land library component of the driver isn’t present in the build environment. I hit this problem with a massive hammer and installed all mesa related packages with mock_mozilla -r test-f16-b --install "*mesa*" and now I get direct rendering: Yes and framerates similar to what the build host is showing (~730). I compared the output of glxinfo with a fedora 16 vm on the same VMware host and noticed that I was getting very similar output. I’d want to double check this on real hardware, but it looks like I am getting the same level of 3D acceleration in the build environment as on the build host!