WireGuard Archive on lore.kernel.org
 help / color / Atom feed
From: Neal Gompa <ngompa13@gmail.com>
To: "Jason A. Donenfeld" <Jason@zx2c4.com>
Cc: unit193@ubuntu.com, Daniel Kahn Gillmor <dkg@fifthhorseman.net>,
	 Andy Whitcroft <apw@canonical.com>,
	Ubuntu Kernel Team <kernel-team@lists.ubuntu.com>,
	 Martin Hauke <mardnh@gmx.de>, Joe Doss <joe@solidadmin.com>,
	 WireGuard mailing list <wireguard@lists.zx2c4.com>,
	Carl George <carl@redhat.com>
Subject: Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs
Date: Sat, 23 May 2020 03:07:04 -0400
Message-ID: <CAEg-Je8HRbwymZqxZJ8ZyGPk3yAG_R9bg7doeO4SdJ_h6Ni-9g@mail.gmail.com> (raw)
In-Reply-To: <CAHmME9oX+8HoK_wAoc5oR9rWenYz4ki8EK+NebQEV2ALZDSrdg@mail.gmail.com>

On Fri, May 22, 2020 at 4:05 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
>
> Hi distro kernel maintainers,
>
> I currently have build.wireguard.com churning away at every new kernel
> release between 3.10 and 5.5 for each and every wireguard-linux-compat
> commit, making sure that we don't break anything. It's a pretty CPU
> intensive, but it means we don't break mainline kernels ever when
> releasing. Scroll down to the bottom to see what I mean.
>
> This is not the case, however, with distro kernels, which differ
> notoriously from mainline in surprising ways. There's been some
> breakage, and unless we do something about it, I imagine there will
> continue to be much breakage. Because so many users use Linux via your
> kernels, it seems imperative that we support these well. And with
> distros like Ubuntu now supporting WireGuard directly through
> wireguard-linux-compat, it seems ever more important that we minimize
> breakage so as to not create release delays downstream of us.
>
> At the moment, wireguard-linux-compat supports:
> - Debian oldoldstable (8), oldstable (9), stable (10), testing (11), sid
> - Ubuntu 14.04, 16.04, 18.04, 19.04, 19.10, 20.04
> - OpenSUSE 42, 15, 15.2 (leap)
> - RHEL 7.8, 8.2
> - CentOS 8.1
>
> The logic here is that we support the latest single kernel for each
> major supported release of these distros. For example, on Debian
> stable, we support 4.19.0-9 but not 4.19.0-8 anymore.
>
> I'd like to put these on build.wireguard.com to mitigate breakage. In
> order to make this happen, I'll need two things:
> - A URL I can scape that will give me the latest kernel versions for
> relevant distros.
> - A URL I can construct using a selected version to download a boring
> kernel source tarball.
> I would prefer to not involve git, if possible, and for these URLs to
> point to the sources for actual kernels that are shipping as the
> standard latest-kernel for each of the above releases.
>
> If you can provide those pieces of information for my CI scripts, it
> seems likely that we can drastically reduce breakage for your
> distribution frankenkernels. If you cannot provide that, but other
> distros can, I will probably naturally prioritize support for those
> other distros that make maintenance easier, simply as a matter of
> habit rather than something intentional.
>

For CentOS kernels, you can query the kernel builds that have been
made for CentOS 7 and CentOS 8 by requesting the list of tags from the
kernel package[1].

The Pagure API[2] provides a straightforward way to do this, even with
curl and jq:

$ curl --silent --header "Content-Type: application/json" \
https://git.centos.org/api/0/rpms/kernel/git/tags | jq '.["tags"] '

This will return a JSON list of tags. For CentOS 7, you'd want the
tags beginning with "imports/c7/". For CentOS 8, you'd want the tags
beginning with "imports/c8/". The string after the prefix is the NVR
(name-version-release) of the package, and is the main part of the
Source RPM file name. The highest NVR in each tag namespace is the
latest package. There are various ways to do the comparison, but the
simplest one would be to sort them with the highest number on top, as
the numbers are guaranteed to numerically advance. I'm not good with
jq, so I couldn't figure out how to do it as a one-liner.

Sources are shipped in the form of Source RPMs and pushed to the
CentOS Vault[3][4]. Sources are archived at each CentOS point release,
meaning that you need to be aware of the CentOS point release where
the kernel was shipped.

So for example, for CentOS 7.8, you'd be looking at the following
locations for source packages:
* http://vault.centos.org/7.8.2003/os/Source/SPackages/
* http://vault.centos.org/7.8.2003/updates/Source/SPackages/

CentOS 8.1 is a bit simpler, with only the one location:
http://vault.centos.org/8.1.1911/BaseOS/Source/SPackages/

With the information from the API and knowing the location of where
the packages are, you can fetch a kernel source package like so:

$ wget http://vault.centos.org/8.1.1911/BaseOS/Source/SPackages/${kernel_version_release}.src.rpm

All the stuff in the source RPM is enough to rebuild the kernel. If
you just want to grab the tarball inside, then you can grab it by
unpacking the Source RPM and grabbing the tarball inside.


[1]: https://git.centos.org/rpms/kernel
[2]: https://pagure.io/api
[3]: http://vault.centos.org/7.8.2003/
[4]: http://vault.centos.org/8.1.1911/



--
真実はいつも一つ!/ Always, there's only one truth!

  parent reply index

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-22  8:04 Jason A. Donenfeld
2020-05-22  8:44 ` Jason A. Donenfeld
2020-05-23  5:59   ` Jason A. Donenfeld
2020-05-22 12:43 ` Georg Faerber
2020-05-23 12:31   ` Georg Faerber
2020-05-23  7:07 ` Neal Gompa [this message]
2020-05-23  7:21   ` Jason A. Donenfeld
2020-05-23 11:49     ` Neal Gompa
2020-05-24  0:01       ` Jason A. Donenfeld
2020-05-24  0:20         ` Neal Gompa
2020-05-24  1:10           ` Jason A. Donenfeld
2020-05-24  1:41             ` Jason A. Donenfeld

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAEg-Je8HRbwymZqxZJ8ZyGPk3yAG_R9bg7doeO4SdJ_h6Ni-9g@mail.gmail.com \
    --to=ngompa13@gmail.com \
    --cc=Jason@zx2c4.com \
    --cc=apw@canonical.com \
    --cc=carl@redhat.com \
    --cc=dkg@fifthhorseman.net \
    --cc=joe@solidadmin.com \
    --cc=kernel-team@lists.ubuntu.com \
    --cc=mardnh@gmx.de \
    --cc=unit193@ubuntu.com \
    --cc=wireguard@lists.zx2c4.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

WireGuard Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/wireguard/0 wireguard/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 wireguard wireguard/ https://lore.kernel.org/wireguard \
		wireguard@lists.zx2c4.com
	public-inbox-index wireguard

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/com.zx2c4.lists.wireguard


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git