[Buildroot] How to organize build into multiple target filesystems?

[Buildroot] How to organize build into multiple target filesystems?

[David Wuertele]

I would like my target to have a small initramfs, and a large-ish /usr
filesystem mounted at runtime.  The initramfs will be populated with some
of my
packages, the usr fs will be populated with the rest.  I don't want the
initramfs to contain anything under usr except for the /usr mountpoint
directory.

In general, I'm looking for a way to divert my package outputs into an
arbitrary number of filesystems, which I then package in various ways,
including but not limited to bundling into a kernel initramfs.

Is there a way to specify such an organization in buildroot?

Thanks,
Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20170221/2c473a67/attachment.html>

[Buildroot] How to organize build into multiple target filesystems?

[Arnout Vandecappelle]

 Hi Dave,

 A bit late to answer this, but perhaps still relevant.

On 21-02-17 20:08, David Wuertele wrote:
> I would like my target to have a small initramfs, and a large-ish /usr
> filesystem mounted at runtime.  The initramfs will be populated with some of my
> packages, the usr fs will be populated with the rest.  I don't want the
> initramfs to contain anything under usr except for the /usr mountpoint
> directory.
> 
> In general, I'm looking for a way to divert my package outputs into an
> arbitrary number of filesystems, which I then package in various ways,
> including but not limited to bundling into a kernel initramfs.
> 
> Is there a way to specify such an organization in buildroot?

 Not directly. The Buildroot Way is to keep things simple, preferably without
blocking real use cases. For your use case, you need specific treatment in a
fakeroot script.

 Buildroot will still build a monolithic filesystem, and your fakeroot script
can extract parts that need special treatment. For example, you can make a
tarball of $TARGET_DIR/usr, then remove the /usr tree, or remove the part that
you don't need. You can also use $BUILD_DIR/packages-file-list.txt to find out
which file comes from which package, to do this on a per-package basis.

 You will also need to add the necessary scripts (or systemd units) in a rootfs
overlay to stitch things back together.

 I'm adding Thomas DS in Cc, he described a somewhat similar setup in the last
BR developer meeting.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] How to organize build into multiple target filesystems?

[Thomas De Schampheleire]

Hi Dave,

On Mon, Mar 6, 2017 at 12:52 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
>  Hi Dave,
>
>  A bit late to answer this, but perhaps still relevant.
>
> On 21-02-17 20:08, David Wuertele wrote:
>> I would like my target to have a small initramfs, and a large-ish /usr
>> filesystem mounted at runtime.  The initramfs will be populated with some of my
>> packages, the usr fs will be populated with the rest.  I don't want the
>> initramfs to contain anything under usr except for the /usr mountpoint
>> directory.
>>
>> In general, I'm looking for a way to divert my package outputs into an
>> arbitrary number of filesystems, which I then package in various ways,
>> including but not limited to bundling into a kernel initramfs.
>>
>> Is there a way to specify such an organization in buildroot?
>
>  Not directly. The Buildroot Way is to keep things simple, preferably without
> blocking real use cases. For your use case, you need specific treatment in a
> fakeroot script.
>
>  Buildroot will still build a monolithic filesystem, and your fakeroot script
> can extract parts that need special treatment. For example, you can make a
> tarball of $TARGET_DIR/usr, then remove the /usr tree, or remove the part that
> you don't need. You can also use $BUILD_DIR/packages-file-list.txt to find out
> which file comes from which package, to do this on a per-package basis.
>
>  You will also need to add the necessary scripts (or systemd units) in a rootfs
> overlay to stitch things back together.
>
>  I'm adding Thomas DS in Cc, he described a somewhat similar setup in the last
> BR developer meeting.

Sorry for the late reply.
We are creating some opkg packages and thus extracting these files
from the rootfs.
The script I created for this is below. I guess it can be split in
two: the core part and the opkg creation, as some people may just want
tar files or something else. Feedback welcome.

diff --git a/support/scripts/create-pkgs b/support/scripts/create-pkgs
new file mode 100755
index 0000000..8e512b2
--- /dev/null
+++ b/support/scripts/create-pkgs
@@ -0,0 +1,212 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2016 Thomas De Schampheleire
<thomas.de_schampheleire@nokia.com>
+
+import argparse
+import collections
+import csv
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import pkgutil
+
+class Package(object):
+
+    @staticmethod
+    def parse_file_list():
+        """Create a dictionary pkg -> list of files installed by pkg"""
+
+        files = dict()
+        f = open(os.path.join(Package.outputdir, "build",
"packages-file-list.txt"))
+
+        # Example contents of packages-file-list.txt:
+        #     jansson,./usr/lib/libjansson.so.4.7.0
+        #     jansson,./usr/include/jansson.h
+        #     busybox,./usr/share/udhcpc/default.script
+        #     busybox,./etc/init.d/S01logging
+
+        for row in csv.reader(f):
+            fpath = row[1]
+
+            # remove the initial './' in each file path
+            fpath = fpath.strip()[2:]
+
+            # skip all files we know would be removed by target-finalize
+            # see Buildroot Makefile
+            if fpath.startswith(('usr/include/', 'usr/share/aclocal/',
+                                 'usr/lib/pkgconfig/', 'usr/share/pkgconfig/',
+                                 'usr/lib/cmake/', 'usr/share/cmake/',
+                                 'usr/man/', 'usr/share/man/',
+                                 'usr/info/', 'usr/share/info/',
+                                 'usr/doc/', 'usr/share/doc/',
+                                 'usr/share/gtk-doc/')):
+                continue
+            if fpath.endswith(('.cmake', '.a', '.la')):
+                continue
+
+            if not row[0] in files:
+                files[row[0]] = []
+            files[row[0]] += [fpath]
+        f.close()
+
+        return files
+
+    @staticmethod
+    def setup(outputdir, arch, requested_pkgs):
+        Package.outputdir = os.path.abspath(outputdir)
+        Package.arch = arch
+        Package.requested_pkgs = requested_pkgs
+        Package.files = Package.parse_file_list()
+        Package.versions = pkgutil.get_version(Package.files.keys(),
+                                                  verbose=False)
+        Package.depends = pkgutil.get_depends(Package.files.keys(),
+                                                  verbose=False)
+        Package.pkgdir = os.path.join(Package.outputdir, 'pkgs')
+        try:
+            os.makedirs(Package.pkgdir)
+        except OSError:
+            pass
+
+    def __init__(self, pkg):
+        if pkg not in Package.files:
+            raise Exception("Error: '%s' is not built and can thus
not be packaged" % pkg)
+
+        self.pkg = pkg
+        self.version = Package.versions[self.pkg]
+
+    def isolate_files(self, tempdir):
+        for f in Package.files[self.pkg]:
+            src = os.path.join(Package.outputdir, "target", f)
+            if not os.path.exists(src):
+                print('%s: WARNING: file %s does not exist, pkg will
likely be corrupt' % (self.pkg, f))
+                continue
+
+            dst = os.path.join(tempdir, os.path.dirname(f))
+            if not os.path.isdir(dst):
+                os.makedirs(dst)
+
+            if os.path.islink(src):
+                linkto = os.readlink(src)
+                linkname = os.path.basename(src)
+                dst = os.path.join(dst,linkname)
+                os.symlink(linkto, dst)
+            else:
+                shutil.copy2(src, dst)
+
+
+    def remove_files(self):
+        for f in Package.files[self.pkg]:
+            src = os.path.join(Package.outputdir, "target", f)
+            if not os.path.lexists(src):
+                continue
+            os.remove(src)
+
+    def write_control(self, tempdir, **kwargs):
+        """Write a CONTROL/control file according to the opkg file format"""
+
+        controldir = os.path.join(tempdir, 'CONTROL')
+        os.makedirs(controldir)
+
+        control = ''
+        for key,value in kwargs.items():
+            control += "%s: %s\n" % (key.title(), value)
+
+        f = open(os.path.join(controldir, 'control'), 'wb')
+        f.write(control)
+        f.close()
+
+        # Print out to the console for inspection
+        print('\n' + control)
+
+    def get_opkg_depends(self):
+        """Return the dependencies that need to be declared in the opkg"""
+        deps =
set(Package.depends[self.pkg]).intersection(set(Package.requested_pkgs))
+        if deps is not None:
+            return ','.join(deps)
+        else:
+            return ''
+
+    def create(self):
+        tempdir = tempfile.mkdtemp()
+
+        self.isolate_files(tempdir)
+
+        self.write_control(tempdir, **{
+                'package': self.pkg,
+                'version': self.version,
+                'description': self.pkg,
+                'architecture': Package.arch,
+                'maintainer': 'unknown',
+                'section': 'unknown',
+                'priority': 'optional',
+                'depends': self.get_opkg_depends(),
+                'source': 'unknown',
+                })
+
+        # generate opkg
+        p = subprocess.Popen([
+                 os.path.join(Package.outputdir, 'host/usr/bin/opkg-build'),
+                 tempdir,
+                 Package.pkgdir], stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
+        out, err = p.communicate()
+        # ignore stdout, only print errors
+        print(err)
+        if p.returncode != 0:
+            print('opkg-build returned with error code %d,
exiting...' % p.returncode)
+            sys.exit(p.returncode)
+
+        shutil.rmtree(tempdir)
+
+        self.remove_files()
+
+    @property
+    def filename(self):
+        return '%s_%s_%s.ipk' % (self.pkg, self.version, Package.arch)
+
+    @property
+    def path(self):
+        return os.path.join(Package.pkgdir, self.filename)
+
+    def is_outdated(self):
+        """Check whether pkg exists and is up-to-date with the target files"""
+        if not os.path.exists(self.path):
+            return True
+
+        stamp = os.path.join(Package.outputdir, 'build', '%s-%s' %
(self.pkg, self.version), '.stamp_target_installed')
+
+        if os.stat(self.path).st_mtime < os.stat(stamp).st_mtime:
+            return True
+
+        return False
+
+def main():
+
+    parser = argparse.ArgumentParser(description='Create binary packages')
+
+    parser.add_argument("--outputdir", '-d', metavar="OUTPUTDIR",
required=True,
+                        help="Buildroot output directory")
+    parser.add_argument("--arch", "-a", required=True,
+                        help="Architecture (free-format)")
+    parser.add_argument('pkgnames', nargs='+',
+                        help="List of packages to create a pkg for")
+    args = parser.parse_args()
+
+    print('Creating packages for: %s' % ', '.join(args.pkgnames))
+
+    Package.setup(args.outputdir, args.arch, args.pkgnames)
+
+    for pkgname in args.pkgnames:
+        pkg = Package(pkgname)
+        if pkg.is_outdated():
+            print('%s: creating package...' % pkgname)
+            pkg.create()
+            print('%s: created %s' % (pkgname, pkg.path))
+        else:
+            print('%s: package already exists and is up-to-date: %s'
% (pkgname, pkg.path))
+
+    print("\nNOTE: to force recreation of a package, run 'make
foo-reinstall' before calling this script.")
+
+if __name__ == '__main__':
+    main()


I call this script from a post-build script, wrapping it in fakeroot
and providing architecture and list of packages to package.
The post-build script looks like this:

# Expose only the selected key/value pair. This is safer than getopt_simple
# because it avoids overwriting variables of the calling script unexpectedly.
# Usage: getopt_simple_onevar <key> <cmdline>
getopt_simple_onevar()
{
    local key=$1
    shift

    until [ -z "$1" ]
    do
        parameter=${1%%=*}     # Extract name.
        value=${1##*=}         # Extract value.
        if [ "$parameter" = "$key" ]; then
            eval $parameter=\"$value\"
        fi
        shift                  # Drop $1 and shift $2 to $1
    done
}

# Parse passed arguments (POST_SCRIPT_ARGS)
getopt_simple_onevar ARCH "$@"
getopt_simple_onevar PKGLIST "$@"

# Replace commas back to spaces. The reverse conversion was done to accommodate
# getopt_simple_onevar who uses spaces as word-separator
PKGLIST=${PKGLIST//,/ }

if [ -z "$PKGLIST" ]; then
    echo "WARNING: create-pkgs feature enabled but package list empty"
else
    $HOST_DIR/usr/bin/fakeroot -- support/scripts/create-pkgs
--outputdir output/ --arch "$ARCH" $PKGLIST
fi



And via the config we set BR2_POST_SCRIPT_ARGS to something like
"ARCH=$(ARCH) PKGLIST=foo,bar"

Best regards,
Thomas

[Buildroot] How to organize build into multiple target filesystems?

[Thomas De Schampheleire]

On Tue, Mar 21, 2017 at 9:34 PM, Thomas De Schampheleire
<patrickdepinguin+buildroot@gmail.com> wrote:
> Hi Dave,
>
> On Mon, Mar 6, 2017 at 12:52 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
>>  Hi Dave,
>>
>>  A bit late to answer this, but perhaps still relevant.
>>
>> On 21-02-17 20:08, David Wuertele wrote:
>>> I would like my target to have a small initramfs, and a large-ish /usr
>>> filesystem mounted at runtime.  The initramfs will be populated with some of my
>>> packages, the usr fs will be populated with the rest.  I don't want the
>>> initramfs to contain anything under usr except for the /usr mountpoint
>>> directory.
>>>
>>> In general, I'm looking for a way to divert my package outputs into an
>>> arbitrary number of filesystems, which I then package in various ways,
>>> including but not limited to bundling into a kernel initramfs.
>>>
>>> Is there a way to specify such an organization in buildroot?
>>
>>  Not directly. The Buildroot Way is to keep things simple, preferably without
>> blocking real use cases. For your use case, you need specific treatment in a
>> fakeroot script.
>>
>>  Buildroot will still build a monolithic filesystem, and your fakeroot script
>> can extract parts that need special treatment. For example, you can make a
>> tarball of $TARGET_DIR/usr, then remove the /usr tree, or remove the part that
>> you don't need. You can also use $BUILD_DIR/packages-file-list.txt to find out
>> which file comes from which package, to do this on a per-package basis.
>>
>>  You will also need to add the necessary scripts (or systemd units) in a rootfs
>> overlay to stitch things back together.
>>
>>  I'm adding Thomas DS in Cc, he described a somewhat similar setup in the last
>> BR developer meeting.
>
> Sorry for the late reply.
> We are creating some opkg packages and thus extracting these files
> from the rootfs.
> The script I created for this is below. I guess it can be split in
> two: the core part and the opkg creation, as some people may just want
> tar files or something else. Feedback welcome.
>
> diff --git a/support/scripts/create-pkgs b/support/scripts/create-pkgs
> new file mode 100755
> index 0000000..8e512b2
> --- /dev/null
> +++ b/support/scripts/create-pkgs
> @@ -0,0 +1,212 @@
> +#!/usr/bin/env python
> +
> +# Copyright (C) 2016 Thomas De Schampheleire
> <thomas.de_schampheleire@nokia.com>
> +
> +import argparse
> +import collections
> +import csv
> +import os
> +import shutil
> +import subprocess
> +import sys
> +import tempfile
> +import pkgutil

On latest master this needs to become 'import brpkgutil' and
references below to pkgutil need to be changed too.

I also forgot to mention that this change depends on the following
queued patches:

http://patchwork.ozlabs.org/patch/724235/
http://patchwork.ozlabs.org/patch/724236/

/Thomas