[PATCH v3] scripts/package: snap-pkg target

[PATCH v3] scripts/package: snap-pkg target

[Paolo Pisati]

Following in footsteps of other targets like 'deb-pkg, 'rpm-pkg' and 'tar-pkg',
this patch adds a 'snap-pkg' target for the creation of a Linux kernel snap
package using the kbuild infrastructure.

A snap, in its general form, is a self contained, sandboxed, universal package
and it is intended to work across multiple distributions and/or devices. A snap
package is distributed as a single compressed squashfs filesystem.

A kernel snap is a snap package carrying the Linux kernel, kernel modules,
accessory files (DTBs, System.map, etc) and a manifesto file.  The purpose of a
kernel snap is to carry the Linux kernel during the creation of a system image,
eg. Ubuntu Core, and its subsequent upgrades.

For more information on snap packages: https://snapcraft.io/docs/

Signed-off-by: Paolo Pisati <paolo.pisati@canonical.com>
---
Changes since v2:
 - use a tarball to instead of cloning the src tree via git
 - set 'kernel-with-firmware: false' to accomodate for
   'firmware_install' removal

Changes since v1:
 - slightly improve the messaging
---
 .gitignore                         |  5 +++++
 scripts/package/Makefile           | 16 ++++++++++++++++
 scripts/package/snapcraft.template | 14 ++++++++++++++
 3 files changed, 35 insertions(+)
 create mode 100644 scripts/package/snapcraft.template

diff --git a/.gitignore b/.gitignore
index f6050b8..705e099 100644
--- a/.gitignore
+++ b/.gitignore
@@ -66,6 +66,11 @@ modules.builtin
 /debian/
 
 #
+# Snap directory (make snap-pkg)
+#
+/snap/
+
+#
 # tar directory (make tar*-pkg)
 #
 /tar-install/
diff --git a/scripts/package/Makefile b/scripts/package/Makefile
index 9ed96aef..5996300 100644
--- a/scripts/package/Makefile
+++ b/scripts/package/Makefile
@@ -95,6 +95,21 @@ bindeb-pkg: FORCE
 
 clean-dirs += $(objtree)/debian/
 
+# snap-pkg
+# ---------------------------------------------------------------------------
+snap-pkg: FORCE
+	rm -rf $(objtree)/snap
+	mkdir $(objtree)/snap
+	$(MAKE) clean
+	$(call cmd,src_tar,$(KERNELPATH))
+	sed "s@KERNELRELEASE@$(KERNELRELEASE)@; \
+		s@SRCTREE@$(shell realpath $(KERNELPATH).tar.gz)@" \
+		$(srctree)/scripts/package/snapcraft.template > \
+		$(objtree)/snap/snapcraft.yaml
+	cd $(objtree)/snap && \
+	snapcraft --target-arch=$(UTS_MACHINE)
+
+clean-dirs += $(objtree)/snap/
 
 # tarball targets
 # ---------------------------------------------------------------------------
@@ -139,6 +154,7 @@ help: FORCE
 	@echo '  binrpm-pkg          - Build only the binary kernel RPM package'
 	@echo '  deb-pkg             - Build both source and binary deb kernel packages'
 	@echo '  bindeb-pkg          - Build only the binary kernel deb package'
+	@echo '  snap-pkg            - Build only the binary kernel snap package (will connect to external hosts)'
 	@echo '  tar-pkg             - Build the kernel as an uncompressed tarball'
 	@echo '  targz-pkg           - Build the kernel as a gzip compressed tarball'
 	@echo '  tarbz2-pkg          - Build the kernel as a bzip2 compressed tarball'
diff --git a/scripts/package/snapcraft.template b/scripts/package/snapcraft.template
new file mode 100644
index 0000000..626d278e
--- /dev/null
+++ b/scripts/package/snapcraft.template
@@ -0,0 +1,14 @@
+name: kernel
+version: KERNELRELEASE
+summary: Linux kernel
+description: The upstream Linux kernel
+grade: stable
+confinement: strict
+type: kernel
+
+parts:
+  kernel:
+    plugin: kernel
+    source: SRCTREE
+    source-type: tar
+    kernel-with-firmware: false
-- 
2.7.4

Re: [PATCH v3] scripts/package: snap-pkg target

[Jim Davis]

On Mon, Nov 27, 2017 at 4:07 AM, Paolo Pisati
<paolo.pisati@canonical.com> wrote:
> Following in footsteps of other targets like 'deb-pkg, 'rpm-pkg' and 'tar-pkg',
> this patch adds a 'snap-pkg' target for the creation of a Linux kernel snap
> package using the kbuild infrastructure.

With this patch applied to the 4.14 kernel source, and running make
snap-pkg on a Ubuntu 16.04 VM with the latest 16.04 kernel config
file, I ran into

Traceback (most recent call last):
  File "/usr/lib/python3.5/shutil.py", line 538, in move
    os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory:
'/home/jim/linux/snap/parts/kernel/install/lib/firmware' ->
'/home/jim/linux/snap/parts/kernel/install/firmware'

What did I miss?

-- 
Jim

Re: [PATCH v3] scripts/package: snap-pkg target

[Paolo Pisati]

On Mon, Nov 27, 2017 at 7:33 PM, Jim Davis <jim.epost@gmail.com> wrote:
>
> With this patch applied to the 4.14 kernel source, and running make
> snap-pkg on a Ubuntu 16.04 VM with the latest 16.04 kernel config
> file, I ran into
>
> Traceback (most recent call last):
>   File "/usr/lib/python3.5/shutil.py", line 538, in move
>     os.rename(src, real_dst)
> FileNotFoundError: [Errno 2] No such file or directory:
> '/home/jim/linux/snap/parts/kernel/install/lib/firmware' ->
> '/home/jim/linux/snap/parts/kernel/install/firmware'
>
> What did I miss?

You need a recent version of snapcraft, i landed a fix for that last week.

$ snap install --classic --edge snapcraft

or refresh it, in case you already installed via snaps:

$ snap refresh --edge snapcraft

and then 'make snap-pkg' again.
Beware of $PATH in case you have the snap and deb version installed.
-- 
bye,
p.

Re: [PATCH v3] scripts/package: snap-pkg target

[Jim Davis]

On Tue, Nov 28, 2017 at 10:14 AM, Paolo Pisati
<paolo.pisati@canonical.com> wrote:
> On Mon, Nov 27, 2017 at 7:33 PM, Jim Davis <jim.epost@gmail.com> wrote:
>>
>> With this patch applied to the 4.14 kernel source, and running make
>> snap-pkg on a Ubuntu 16.04 VM with the latest 16.04 kernel config
>> file, I ran into
>>
>> Traceback (most recent call last):
>>   File "/usr/lib/python3.5/shutil.py", line 538, in move
>>     os.rename(src, real_dst)
>> FileNotFoundError: [Errno 2] No such file or directory:
>> '/home/jim/linux/snap/parts/kernel/install/lib/firmware' ->
>> '/home/jim/linux/snap/parts/kernel/install/firmware'
>>
>> What did I miss?
>
> You need a recent version of snapcraft, i landed a fix for that last week.
>
> $ snap install --classic --edge snapcraft
>
> or refresh it, in case you already installed via snaps:
>
> $ snap refresh --edge snapcraft
>
> and then 'make snap-pkg' again.
> Beware of $PATH in case you have the snap and deb version installed.

Ok, after updating and setting ~/snapcraft/bin/snapcraft ahead of
/usr/bin in my $PATH, the build completed.   Oddly both
/usr/bin/snapcraft --version and ~/snapcraft/bin/snapcraft --version
return 2.34, though the build failed with /usr/bin/snapcraft.

-- 
Jim

Re: [PATCH v3] scripts/package: snap-pkg target

[Masahiro Yamada]

2017-11-29 4:35 GMT+09:00 Jim Davis <jim.epost@gmail.com>:
> On Tue, Nov 28, 2017 at 10:14 AM, Paolo Pisati
> <paolo.pisati@canonical.com> wrote:
>> On Mon, Nov 27, 2017 at 7:33 PM, Jim Davis <jim.epost@gmail.com> wrote:
>>>
>>> With this patch applied to the 4.14 kernel source, and running make
>>> snap-pkg on a Ubuntu 16.04 VM with the latest 16.04 kernel config
>>> file, I ran into
>>>
>>> Traceback (most recent call last):
>>>   File "/usr/lib/python3.5/shutil.py", line 538, in move
>>>     os.rename(src, real_dst)
>>> FileNotFoundError: [Errno 2] No such file or directory:
>>> '/home/jim/linux/snap/parts/kernel/install/lib/firmware' ->
>>> '/home/jim/linux/snap/parts/kernel/install/firmware'
>>>
>>> What did I miss?
>>
>> You need a recent version of snapcraft, i landed a fix for that last week.
>>
>> $ snap install --classic --edge snapcraft
>>
>> or refresh it, in case you already installed via snaps:
>>
>> $ snap refresh --edge snapcraft
>>
>> and then 'make snap-pkg' again.
>> Beware of $PATH in case you have the snap and deb version installed.
>
> Ok, after updating and setting ~/snapcraft/bin/snapcraft ahead of
> /usr/bin in my $PATH, the build completed.   Oddly both
> /usr/bin/snapcraft --version and ~/snapcraft/bin/snapcraft --version
> return 2.34, though the build failed with /usr/bin/snapcraft.
>
> --
> Jim
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Worked for me too,
after updating snapcraft.


Is it really impossible to check the snapcraft version?
What is the minimum version?   2.35 ?



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v3] scripts/package: snap-pkg target

[Paolo Pisati]

On Wed, Nov 29, 2017 at 8:33 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
>
> Worked for me too,
> after updating snapcraft.
>
>
> Is it really impossible to check the snapcraft version?
> What is the minimum version?   2.35 ?

The minimum version is 2.35+ - what is about to become 2.36 anytime soon now.
-- 
bye,
p.

Re: [PATCH v3] scripts/package: snap-pkg target

[Jim Davis]

On Wed, Nov 29, 2017 at 10:05 AM, Paolo Pisati
<paolo.pisati@canonical.com> wrote:
> On Wed, Nov 29, 2017 at 8:33 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>>
>> Worked for me too,
>> after updating snapcraft.
>>
>>
>> Is it really impossible to check the snapcraft version?
>> What is the minimum version?   2.35 ?
>
> The minimum version is 2.35+ - what is about to become 2.36 anytime soon now.

If I run /snap/bin/snapcraft --version now on that Ubuntu 16.04 VM I
am getting 2.36+git2.ae61453.

I noticed that if I tried to make snap-pkg with the O=/some/dir option
the tar step failed but the rest of the build continued, and seems to
have finished successfully.  Should the snap-pkg target stop after a
tar failure?

-- 
Jim

Re: [PATCH v3] scripts/package: snap-pkg target

[Paolo Pisati]

On Fri, Dec 1, 2017 at 12:15 AM, Jim Davis <jim.epost@gmail.com> wrote:
>
> If I run /snap/bin/snapcraft --version now on that Ubuntu 16.04 VM I
> am getting 2.36+git2.ae61453.

Yep, the edge channel follows the bleeding edge:

$ snap info snapcraft
...
channels:
  stable:    2.35                (794) 53MB classic
  candidate: ↑
  beta:      2.36.1              (876) 53MB classic
  edge:      2.36.1+git6.3524514 (895) 53MB classic

you better switch to beta (and then stable when 2.36.1 gets promoted there):

$ snap refresh --beta snapcraft

> I noticed that if I tried to make snap-pkg with the O=/some/dir option
> the tar step failed but the rest of the build continued, and seems to
> have finished successfully.  Should the snap-pkg target stop after a
> tar failure?

That is weird, how did you do that?

My experience with the O=... option (that mimicks the other rpm-pkg
and deb-pkg targets) is that, you first create the dir, copy there the
.config and it builds fine:

~/linux $ mke defconfig
...
~/linux $ mkdir /tmp/foobar
~/linux $ mv .config /tmp/foobar
~/linux $ git clean -ffdx
~/linux $ make O=/tmp/foobar snap-pkg
...
Snapped kernel_4.14.0+_amd64.snap
~/linux $ ls -la /tmp/foobar/snap/
-rw-r--r-- 1 flag flag 14114816 dic  5 12:41 kernel_4.14.0+_amd64.snap
drwxrwxr-x 3 flag flag     4096 dic  5 12:38 parts
drwxrwxr-x 4 flag flag     4096 dic  5 12:41 prime
drwxrwxr-x 3 flag flag     4096 dic  5 12:38 snap
-rw-rw-r-- 1 flag flag      274 dic  5 12:38 snapcraft.yaml
drwxrwxr-x 3 flag flag     4096 dic  5 12:41 stage




-- 
bye,
p.

Re: [PATCH v3] scripts/package: snap-pkg target

[Jim Davis]

On Tue, Dec 5, 2017 at 4:43 AM, Paolo Pisati <paolo.pisati@canonical.com> wrote:

>> I noticed that if I tried to make snap-pkg with the O=/some/dir option
>> the tar step failed but the rest of the build continued, and seems to
>> have finished successfully.  Should the snap-pkg target stop after a
>> tar failure?
>
> That is weird, how did you do that?

make O=/some/dir defconfig (or your-config-of-choice)
make O=/some/dir snap-pkg

The snap-pkg target is calling the src_tar command, which bails when
$(objtree) isn't $(srctree).

Is $(objtree) = $(srctree) a necessary restriction with snap builds?
It's a restriction for rpm builds for some reason or another, but I
thought that was just a peculiarity of the rpm build process.  If the
snap's tarball could be built without using src_tar, like the buildtar
script does, then Bob's your uncle.

Though if the source tree is an essential part of the snap build then
perhaps the build should just exit if the tarball can't be generated
for whatever reason, including users doing weird things :)

-- 
Jim

Re: [PATCH v3] scripts/package: snap-pkg target

[Masahiro Yamada]

2017-12-06 6:23 GMT+09:00 Jim Davis <jim.epost@gmail.com>:
> On Tue, Dec 5, 2017 at 4:43 AM, Paolo Pisati <paolo.pisati@canonical.com> wrote:
>
>>> I noticed that if I tried to make snap-pkg with the O=/some/dir option
>>> the tar step failed but the rest of the build continued, and seems to
>>> have finished successfully.  Should the snap-pkg target stop after a
>>> tar failure?
>>
>> That is weird, how did you do that?
>
> make O=/some/dir defconfig (or your-config-of-choice)
> make O=/some/dir snap-pkg
>
> The snap-pkg target is calling the src_tar command, which bails when
> $(objtree) isn't $(srctree).
>
> Is $(objtree) = $(srctree) a necessary restriction with snap builds?
> It's a restriction for rpm builds for some reason or another,

Rather, restriction for our build system.

Working directory is changed at the top Makefile if O= is given.
This option is propagated to sub-make.


> but I
> thought that was just a peculiarity of the rpm build process.  If the
> snap's tarball could be built without using src_tar, like the buildtar
> script does, then Bob's your uncle.

Probably, you must reset O=, KBUILD_OUTPUT, KBUILD_SRC, etc.
before calling rpmbuild, snapcraft, etc.

This is not a peculiarity of rpmbuild or snapcraft.

I guess things are not so simple as they look at a glance.



> Though if the source tree is an essential part of the snap build then
> perhaps the build should just exit if the tarball can't be generated
> for whatever reason, including users doing weird things :)
>

I agree, but this is not a problem specific to snap-pkg,
so Paolo's work will not be blocked.

I sent this patch:
https://patchwork.kernel.org/patch/10095377/



Just one question to Paolo:

  If some change like firmware removal happens in the future,
  patching for snapcraft is necessary?
    (i.e. users are required to upgrade snapcraft from --edge ?)

I guess the answer is yes, because "plugin: kernel" needs to know
the kernel-build system internal.

-- 
Best Regards
Masahiro Yamada

Re: [PATCH v3] scripts/package: snap-pkg target

[Paolo Pisati]

On Wed, Dec 6, 2017 at 11:36 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Just one question to Paolo:
>
>   If some change like firmware removal happens in the future,
>   patching for snapcraft is necessary?
>     (i.e. users are required to upgrade snapcraft from --edge ?)
>
> I guess the answer is yes, because "plugin: kernel" needs to know
> the kernel-build system internal.

Well, in this case, i had to fix the 'kernel-with-firmware' option to
skip the 'firmware_install' step since
we took it by default - let's say that in the future the
'modules_install' target is removed or change name,
then i'll have to patch snapcraft.

Wrt updating from -edge, i told you to take it from there because snap
packages land there more often and more quickly
compared to the canonicaldeb  archive, so from a developer perspective
it's easier&quicker to iterate over it, but sooner
or later (e.g. once snapcraft 2.36.1 reaches the snap -stable
channel), it will be copied to the deb archive too -
IOW, you would get using apt-get only too.
-- 
bye,
p.

Re: [PATCH v3] scripts/package: snap-pkg target

[Masahiro Yamada]

2017-12-07 3:14 GMT+09:00 Paolo Pisati <paolo.pisati@canonical.com>:
> On Wed, Dec 6, 2017 at 11:36 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> Just one question to Paolo:
>>
>>   If some change like firmware removal happens in the future,
>>   patching for snapcraft is necessary?
>>     (i.e. users are required to upgrade snapcraft from --edge ?)
>>
>> I guess the answer is yes, because "plugin: kernel" needs to know
>> the kernel-build system internal.
>
> Well, in this case, i had to fix the 'kernel-with-firmware' option to
> skip the 'firmware_install' step since
> we took it by default - let's say that in the future the
> 'modules_install' target is removed or change name,
> then i'll have to patch snapcraft.
>
> Wrt updating from -edge, i told you to take it from there because snap
> packages land there more often and more quickly
> compared to the canonicaldeb  archive, so from a developer perspective
> it's easier&quicker to iterate over it, but sooner
> or later (e.g. once snapcraft 2.36.1 reaches the snap -stable
> channel), it will be copied to the deb archive too -
> IOW, you would get using apt-get only too.
> --


I will pick up this for v4.16-rc1.
It sounds like this target is a bit fragile, but let's see.
I hope a breakage will not happen...


-- 
Best Regards
Masahiro Yamada

Re: [PATCH v3] scripts/package: snap-pkg target

[Masahiro Yamada]

2017-12-08 0:31 GMT+09:00 Masahiro Yamada <yamada.masahiro@socionext.com>:
> 2017-12-07 3:14 GMT+09:00 Paolo Pisati <paolo.pisati@canonical.com>:
>> On Wed, Dec 6, 2017 at 11:36 AM, Masahiro Yamada
>> <yamada.masahiro@socionext.com> wrote:
>>> Just one question to Paolo:
>>>
>>>   If some change like firmware removal happens in the future,
>>>   patching for snapcraft is necessary?
>>>     (i.e. users are required to upgrade snapcraft from --edge ?)
>>>
>>> I guess the answer is yes, because "plugin: kernel" needs to know
>>> the kernel-build system internal.
>>
>> Well, in this case, i had to fix the 'kernel-with-firmware' option to
>> skip the 'firmware_install' step since
>> we took it by default - let's say that in the future the
>> 'modules_install' target is removed or change name,
>> then i'll have to patch snapcraft.
>>
>> Wrt updating from -edge, i told you to take it from there because snap
>> packages land there more often and more quickly
>> compared to the canonicaldeb  archive, so from a developer perspective
>> it's easier&quicker to iterate over it, but sooner
>> or later (e.g. once snapcraft 2.36.1 reaches the snap -stable
>> channel), it will be copied to the deb archive too -
>> IOW, you would get using apt-get only too.
>> --
>
>
> I will pick up this for v4.16-rc1.
> It sounds like this target is a bit fragile, but let's see.
> I hope a breakage will not happen...
>


Applied to linux-kbuild/misc.  Thanks!

-- 
Best Regards
Masahiro Yamada

Re: [PATCH v3] scripts/package: snap-pkg target

[Riku Voipio]

On 29 November 2017 at 19:05, Paolo Pisati <paolo.pisati@canonical.com> wrote:
> On Wed, Nov 29, 2017 at 8:33 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>>
>> Worked for me too,
>> after updating snapcraft.
>>
>>
>> Is it really impossible to check the snapcraft version?
>> What is the minimum version?   2.35 ?
>
> The minimum version is 2.35+ - what is about to become 2.36 anytime soon now.

Since you are already patching snapcraft for this, why not embed the
whole support in snapcraft? Eg something like snapcraft --kernel
/path/to/kernel

The key advantage you'll get is being able to support old kernel versions too.

Riku

Re: [PATCH v3] scripts/package: snap-pkg target

[Paolo Pisati]

On Fri, Dec 1, 2017 at 1:03 PM, Riku Voipio <riku.voipio@linaro.org> wrote:
>
> Since you are already patching snapcraft for this, why not embed the
> whole support in snapcraft? Eg something like snapcraft --kernel
> /path/to/kernel
>
> The key advantage you'll get is being able to support old kernel versions too.

There's a lot more you can do inside a snapcraft.yaml (bundle external
files, override .config options, run pre or post scripts, etc) but by
embedding everything inside the kernel/kbuild plugins
you would loose all these options, and people would loose visibility
of these opportunities.

Besides, snapcraft requires a snapcraft.yaml by design, and i can't
remove that (such a change would never be accepted).
-- 
bye,
p.

Re: [PATCH v3] scripts/package: snap-pkg target

[Paolo Pisati]

Weird, are you sure you are checking the snap version?

$ /snap/bin/snapcraft --version
snapcraft, version 2.35+git26.0474d85

$ snap info snapcraft
name:      snapcraft
summary:   easily create snaps
publisher: canonical
contact:   snappy-canonical-storeaccount@canonical.com
description: |
  Snapcraft aims to make upstream developers' lives easier and as such is not a
  single toolset, but instead is a collection of tools that enable the natural
  workflow of an upstream to be extended with a simple release step into Snappy
  enabled system.
snap-id: vMTKRaLjnOJQetI78HjntT37VuoyssFE
commands:
  - snapcraft
tracking:    edge
installed:   2.35+git26.0474d85 (864) 53MB classic
refreshed:   2017-11-29 09:54:04 +0100 CET
channels:
  stable:    2.35               (794) 53MB classic
  candidate: ↑
  beta:      ↑
  edge:      2.35+git26.0474d85 (864) 53MB classic

On Tue, Nov 28, 2017 at 8:35 PM, Jim Davis <jim.epost@gmail.com> wrote:
> On Tue, Nov 28, 2017 at 10:14 AM, Paolo Pisati
> <paolo.pisati@canonical.com> wrote:
>> On Mon, Nov 27, 2017 at 7:33 PM, Jim Davis <jim.epost@gmail.com> wrote:
>>>
>>> With this patch applied to the 4.14 kernel source, and running make
>>> snap-pkg on a Ubuntu 16.04 VM with the latest 16.04 kernel config
>>> file, I ran into
>>>
>>> Traceback (most recent call last):
>>>   File "/usr/lib/python3.5/shutil.py", line 538, in move
>>>     os.rename(src, real_dst)
>>> FileNotFoundError: [Errno 2] No such file or directory:
>>> '/home/jim/linux/snap/parts/kernel/install/lib/firmware' ->
>>> '/home/jim/linux/snap/parts/kernel/install/firmware'
>>>
>>> What did I miss?
>>
>> You need a recent version of snapcraft, i landed a fix for that last week.
>>
>> $ snap install --classic --edge snapcraft
>>
>> or refresh it, in case you already installed via snaps:
>>
>> $ snap refresh --edge snapcraft
>>
>> and then 'make snap-pkg' again.
>> Beware of $PATH in case you have the snap and deb version installed.
>
> Ok, after updating and setting ~/snapcraft/bin/snapcraft ahead of
> /usr/bin in my $PATH, the build completed.   Oddly both
> /usr/bin/snapcraft --version and ~/snapcraft/bin/snapcraft --version
> return 2.34, though the build failed with /usr/bin/snapcraft.
>
> --
> Jim



-- 
bye,
p.