Freeze the Linux Production Image Build (Yocto)

Why

Every software or hardware engineer releases things as part of their daily routine, and what that release looks like depends on the domain. For software, a release is typically a binary with the corresponding source tagged under the same version in Git. If you want to reproduce that exact build, you check out the tag and build it again. Straightforward.

An Embedded Linux image is a different story. It’s a single file made up of many components: the bootloader, kernel, root file system, third-party frameworks, and other partitions tailored to the product’s requirements. Building that image depends on a build system, and which one you use depends on the project.

Some reference board designs have strong documentation for buildroot, so teams build on top of that. But Yocto is the most widely used Embedded Linux build system today, thanks to its flexibility, community support, documentation, and vendor backing.

This article focuses on freezing a Yocto-based build so it no longer depends on external repositories. If you’re using a different build system, let us know and we’ll cover freezing that one too.

What

To freeze a Yocto-based build, it helps to understand Yocto’s basic architecture. Yocto uses Poky as its base distribution, which bundles a set of metadata, tools, the OpenEmbedded core, and BitBake.

There are three things that need to move under your own control:

Metadata. Yocto’s output is generated from metadata stored in meta-layers, so the final image depends entirely on these layers. They need to live somewhere you control.

Downloads. These are the external repositories used during the build. Almost every Yocto recipe produces a package, and an Embedded Linux image is essentially a collection of those packages built from other recipes. Recipes take inputs too, whether local, remote, or both, and it’s the remote ones (a tarball, a Git repository, a single file) that matter here. Before compiling, a recipe fetches these resources and stores them in the downloads folder. That folder has to be under your control, or the build still depends on the outside world.

Special repositories. Not every project has these, but some builds clone extra repositories before building and configure a path to them, using the clone as a local resource. If your build has repositories like this, they need to move too.

How

Here’s a worked example using an FX30 LoRa Gateway build from Sierra Wireless, migrating its repositories to Bitbucket. Bitbucket is just one option: GitHub, GitLab, and others work the same way.

The workspace folder structure looks like this:

poky
meta-openembedded
meta-gplv2
meta-swi
meta-swi-extras
meta-columbia-x
meta-user

poky is the core of the build: base metadata, required tools, and the OpenEmbedded core. It’s normally cloned from a third-party repository and should be migrated to your own Bitbucket (or GitHub).

meta-user is a meta layer added on top, generally used to customize the image to your requirements. In this example it’s already in a private repository.

All the other meta-* layers are metadata maintained by third parties (vendors, community-funded projects, etc.), and those need migrating too.

Migrating metadata

The steps below cover one meta layer; repeat for each one.

# Move into the layer's directory
$ cd Ing-Yocto-4/poky

# Check the existing remote
$ git remote -v
legato-github  https://github.com/legatoproject/poky (fetch)
legato-github  https://github.com/legatoproject/poky (push)

# Clone a bare mirror to a temp folder
$ cd /tmp
$ git clone --bare https://github.com/legatoproject/poky

Create an empty repository in your own cloud (no README, no .gitignore), then push the mirror into it:

$ cd poky.git
$ git push --mirror <your-repository-URL>

# Clean up
$ cd ..
$ rm poky.git

Migrating the downloads folder

The downloads folder is created on the first build, at whatever path is set in local.conf (default: build/downloads). You can change it with:

DL_DIR ?= "Path to your downloads folder"

To make downloaded Git repositories mirror as tarballs, also set:

BB_GENERATE_MIRROR_TARBALLS = "1"

After the first build completes with this configuration, the downloads folder will contain tarballs alongside git2/svn subfolders (where repositories are fetched before being compressed). Those subfolders are redundant once the tarballs exist, and just take up space, so remove them before freezing:

$ rm -rf ./downloads/git2
$ rm -rf ./downloads/svn

Then push the downloads folder the same way, whether to Bitbucket, FTP, Google Drive, or any storage you control (the exact steps below assume Git, but the idea holds for other storage too):

# In the downloads folder
$ git init
$ git remote set-url origin <your-repository-URL>
$ git branch "build_0101"

# Add your compressed files (e.g. by extension, or with -A)
$ git add <compressed files, e.g. *.tar.gz>
$ git push origin build_0101

If the push fails on size limits, try splitting the files across multiple commits.

Migrating special repositories

In this example project, a few directories are used as local repositories and dependencies:

kernel
legato
lk
mdm9x28

The process is the same as for meta layers: create the repository in your version control platform and push each local directory to it. If a directory isn’t already a Git repository, initialize it first (git init, add, commit) before pushing.

Rebuilding the image from private sources

Once everything above is migrated, clone it all into your working directory and adjust the build configuration to stop it from reaching out to third-party repositories.

Point the download path at your cloned downloads repository (making sure you’re on the correct branch and revision):

DL_DIR ?= "Path to your downloads folder"

Disable network access entirely:

BB_NO_NETWORK = "1"

With the network disabled, any recipe using SRC_REV = "${AUTOREV}" will error out, since it can no longer check the latest revision from a remote server. Run the build once to find which packages trigger this, then pin each one to the exact revision you used, either directly in the recipe or in local.conf:

SRCREV_pn-<your-package-name> = "<your commit ID>"

Source your environment and build as usual. From this point on, you can reproduce the same image indefinitely with no dependency on external repositories. One caveat: Yocto has version dependencies on the Ubuntu build host, so make a note of which Ubuntu version you built with.

Repo tool and manifest.xml

Pulling each repository manually doesn’t scale once the repository count grows: it becomes slow and error-prone. That’s what the repo tool and manifest.xml files are for, letting you track many repositories and their revisions declaratively. We’ll cover that in a follow-up article.