* [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"
@ 2019-09-04 0:52 Cleber Rosa
2019-09-04 5:18 ` Yash Mankad
` (3 more replies)
0 siblings, 4 replies; 13+ messages in thread
From: Cleber Rosa @ 2019-09-04 0:52 UTC (permalink / raw)
To: Peter Maydell, qemu-devel
Cc: Yash Mankad, Fam Zheng, Eduardo Habkost,
Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
Alex Bennée
The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
from tests/acceptance/linux_initrd.py, is currently failing to fetch
the "vmlinuz" file. The reason for the failure is that the Fedora
project retires older versions from the "dl.fedoraproject.org" URL,
and keeps them in "archives.fedoraproject.org". As an added note,
that test uses a Fedora 28 image, because of the specific Linux kernel
version requirements of the test.
For the sake of stability, let's use URLs from the archived and
supposedely ever stable URLs. The good news is that the currently
supported versions are also hosted on the later. This change limits
itself to change the URLs, while keeping the fetched files the same
(as can be evidenced by the unchanged hashes).
Documentation and the "vm tests" fedora definition were also updated.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
qemu-doc.texi | 6 +++---
tests/acceptance/boot_linux_console.py | 25 +++++++++++++++----------
tests/acceptance/linux_initrd.py | 5 +++--
tests/vm/fedora | 2 +-
4 files changed, 22 insertions(+), 16 deletions(-)
diff --git a/qemu-doc.texi b/qemu-doc.texi
index 577d1e8376..37795f86fb 100644
--- a/qemu-doc.texi
+++ b/qemu-doc.texi
@@ -440,15 +440,15 @@ of <protocol>.
Example: boot from a remote Fedora 20 live ISO image
@example
-qemu-system-x86_64 --drive media=cdrom,file=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
+qemu-system-x86_64 --drive media=cdrom,file=https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
-qemu-system-x86_64 --drive media=cdrom,file.driver=http,file.url=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
+qemu-system-x86_64 --drive media=cdrom,file.driver=http,file.url=http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
@end example
Example: boot from a remote Fedora 20 cloud image using a local overlay for
writes, copy-on-read, and a readahead of 64k
@example
-qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",, "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
+qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, "file.url":"http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",, "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
qemu-system-x86_64 -drive file=/tmp/Fedora-x86_64-20-20131211.1-sda.qcow2,copy-on-read=on
@end example
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 2504ef0150..8a9a314ab4 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -76,8 +76,9 @@ class BootLinuxConsole(Test):
:avocado: tags=arch:x86_64
:avocado: tags=machine:pc
"""
- kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
- 'releases/29/Everything/x86_64/os/images/pxeboot/vmlinuz')
+ kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
+ '/linux/releases/29/Everything/x86_64/os/images/pxeboot'
+ '/vmlinuz')
kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
@@ -250,8 +251,9 @@ class BootLinuxConsole(Test):
:avocado: tags=arch:aarch64
:avocado: tags=machine:virt
"""
- kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
- 'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
+ kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
+ '/linux/releases/29/Everything/aarch64/os/images/pxeboot'
+ '/vmlinuz')
kernel_hash = '8c73e469fc6ea06a58dc83a628fc695b693b8493'
kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
@@ -271,8 +273,9 @@ class BootLinuxConsole(Test):
:avocado: tags=arch:arm
:avocado: tags=machine:virt
"""
- kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
- 'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
+ kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
+ '/linux/releases/29/Everything/armhfp/os/images/pxeboot'
+ '/vmlinuz')
kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4'
kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
@@ -318,8 +321,9 @@ class BootLinuxConsole(Test):
:avocado: tags=arch:s390x
:avocado: tags=machine:s390_ccw_virtio
"""
- kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/'
- 'releases/29/Everything/s390x/os/images/kernel.img')
+ kernel_url = ('https://archives.fedoraproject.org/pub/archive'
+ '/fedora-secondary/releases/29/Everything/s390x/os/images'
+ '/kernel.img')
kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313'
kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
@@ -360,8 +364,9 @@ class BootLinuxConsole(Test):
:avocado: tags=arch:ppc64
:avocado: tags=machine:pseries
"""
- kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/'
- 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz')
+ kernel_url = ('https://archives.fedoraproject.org/pub/archive'
+ '/fedora-secondary/releases/29/Everything/ppc64le/os'
+ '/ppc/ppc64/vmlinuz')
kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
diff --git a/tests/acceptance/linux_initrd.py b/tests/acceptance/linux_initrd.py
index 23be5a63aa..c61d9826a4 100644
--- a/tests/acceptance/linux_initrd.py
+++ b/tests/acceptance/linux_initrd.py
@@ -54,8 +54,9 @@ class LinuxInitrd(Test):
QEMU has supported up to 4 GiB initrd for recent kernel
Expect guest can reach 'Unpacking initramfs...'
"""
- kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
- 'Everything/x86_64/os/images/pxeboot/vmlinuz')
+ kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
+ '/linux/releases/28/Everything/x86_64/os/images/pxeboot/'
+ 'vmlinuz')
kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
max_size = 2 * (1024 ** 3) + 1
diff --git a/tests/vm/fedora b/tests/vm/fedora
index e8fa5bf0d2..7fec1479fb 100755
--- a/tests/vm/fedora
+++ b/tests/vm/fedora
@@ -23,7 +23,7 @@ class FedoraVM(basevm.BaseVM):
name = "fedora"
arch = "x86_64"
- base = "http://dl.fedoraproject.org/pub/fedora/linux/releases/30/"
+ base = "https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/"
link = base + "Server/x86_64/iso/Fedora-Server-netinst-x86_64-30-1.2.iso"
repo = base + "Server/x86_64/os/"
full = base + "Everything/x86_64/os/"
--
2.21.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"
2019-09-04 0:52 [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org" Cleber Rosa
@ 2019-09-04 5:18 ` Yash Mankad
2019-09-04 12:36 ` Wainer dos Santos Moschetta
` (2 subsequent siblings)
3 siblings, 0 replies; 13+ messages in thread
From: Yash Mankad @ 2019-09-04 5:18 UTC (permalink / raw)
To: Cleber Rosa, Peter Maydell, qemu-devel
Cc: Fam Zheng, Eduardo Habkost, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Willian Rampazzo, Alex Bennée
On 9/3/19 8:52 PM, Cleber Rosa wrote:
> The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
> from tests/acceptance/linux_initrd.py, is currently failing to fetch
> the "vmlinuz" file. The reason for the failure is that the Fedora
> project retires older versions from the "dl.fedoraproject.org" URL,
> and keeps them in "archives.fedoraproject.org". As an added note,
> that test uses a Fedora 28 image, because of the specific Linux kernel
> version requirements of the test.
>
> For the sake of stability, let's use URLs from the archived and
> supposedely ever stable URLs. The good news is that the currently
> supported versions are also hosted on the later. This change limits
> itself to change the URLs, while keeping the fetched files the same
> (as can be evidenced by the unchanged hashes).
>
> Documentation and the "vm tests" fedora definition were also updated.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
> qemu-doc.texi | 6 +++---
> tests/acceptance/boot_linux_console.py | 25 +++++++++++++++----------
> tests/acceptance/linux_initrd.py | 5 +++--
> tests/vm/fedora | 2 +-
> 4 files changed, 22 insertions(+), 16 deletions(-)
>
> diff --git a/qemu-doc.texi b/qemu-doc.texi
> index 577d1e8376..37795f86fb 100644
> --- a/qemu-doc.texi
> +++ b/qemu-doc.texi
> @@ -440,15 +440,15 @@ of <protocol>.
>
> Example: boot from a remote Fedora 20 live ISO image
> @example
> -qemu-system-x86_64 --drive media=cdrom,file=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> +qemu-system-x86_64 --drive media=cdrom,file=https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
>
> -qemu-system-x86_64 --drive media=cdrom,file.driver=http,file.url=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> +qemu-system-x86_64 --drive media=cdrom,file.driver=http,file.url=http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> @end example
>
> Example: boot from a remote Fedora 20 cloud image using a local overlay for
> writes, copy-on-read, and a readahead of 64k
> @example
> -qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",, "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
> +qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, "file.url":"http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",, "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
>
> qemu-system-x86_64 -drive file=/tmp/Fedora-x86_64-20-20131211.1-sda.qcow2,copy-on-read=on
> @end example
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 2504ef0150..8a9a314ab4 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -76,8 +76,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:x86_64
> :avocado: tags=machine:pc
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
> - 'releases/29/Everything/x86_64/os/images/pxeboot/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> + '/linux/releases/29/Everything/x86_64/os/images/pxeboot'
> + '/vmlinuz')
> kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> @@ -250,8 +251,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:aarch64
> :avocado: tags=machine:virt
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
> - 'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> + '/linux/releases/29/Everything/aarch64/os/images/pxeboot'
> + '/vmlinuz')
> kernel_hash = '8c73e469fc6ea06a58dc83a628fc695b693b8493'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> @@ -271,8 +273,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:arm
> :avocado: tags=machine:virt
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
> - 'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> + '/linux/releases/29/Everything/armhfp/os/images/pxeboot'
> + '/vmlinuz')
> kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> @@ -318,8 +321,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:s390x
> :avocado: tags=machine:s390_ccw_virtio
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/'
> - 'releases/29/Everything/s390x/os/images/kernel.img')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive'
> + '/fedora-secondary/releases/29/Everything/s390x/os/images'
> + '/kernel.img')
> kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> @@ -360,8 +364,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:ppc64
> :avocado: tags=machine:pseries
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/'
> - 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive'
> + '/fedora-secondary/releases/29/Everything/ppc64le/os'
> + '/ppc/ppc64/vmlinuz')
> kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> diff --git a/tests/acceptance/linux_initrd.py b/tests/acceptance/linux_initrd.py
> index 23be5a63aa..c61d9826a4 100644
> --- a/tests/acceptance/linux_initrd.py
> +++ b/tests/acceptance/linux_initrd.py
> @@ -54,8 +54,9 @@ class LinuxInitrd(Test):
> QEMU has supported up to 4 GiB initrd for recent kernel
> Expect guest can reach 'Unpacking initramfs...'
> """
> - kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
> - 'Everything/x86_64/os/images/pxeboot/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> + '/linux/releases/28/Everything/x86_64/os/images/pxeboot/'
> + 'vmlinuz')
> kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> max_size = 2 * (1024 ** 3) + 1
> diff --git a/tests/vm/fedora b/tests/vm/fedora
> index e8fa5bf0d2..7fec1479fb 100755
> --- a/tests/vm/fedora
> +++ b/tests/vm/fedora
> @@ -23,7 +23,7 @@ class FedoraVM(basevm.BaseVM):
> name = "fedora"
> arch = "x86_64"
>
> - base = "http://dl.fedoraproject.org/pub/fedora/linux/releases/30/"
> + base = "https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/"
> link = base + "Server/x86_64/iso/Fedora-Server-netinst-x86_64-30-1.2.iso"
> repo = base + "Server/x86_64/os/"
> full = base + "Everything/x86_64/os/"
Thanks for the fix Cleber.
Reviewed-by: Yash Mankad <ymankad@redhat.com>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"
2019-09-04 0:52 [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org" Cleber Rosa
2019-09-04 5:18 ` Yash Mankad
@ 2019-09-04 12:36 ` Wainer dos Santos Moschetta
2019-09-04 13:05 ` Philippe Mathieu-Daudé
2019-09-10 8:31 ` Peter Maydell
2019-09-10 8:41 ` Daniel P. Berrangé
3 siblings, 1 reply; 13+ messages in thread
From: Wainer dos Santos Moschetta @ 2019-09-04 12:36 UTC (permalink / raw)
To: Cleber Rosa, Peter Maydell, qemu-devel
Cc: Yash Mankad, Fam Zheng, Eduardo Habkost, Alex Bennée,
Willian Rampazzo, Philippe Mathieu-Daudé
Hi Cleber,
On 09/03/2019 09:52 PM, Cleber Rosa wrote:
> The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
> from tests/acceptance/linux_initrd.py, is currently failing to fetch
> the "vmlinuz" file. The reason for the failure is that the Fedora
> project retires older versions from the "dl.fedoraproject.org" URL,
> and keeps them in "archives.fedoraproject.org". As an added note,
> that test uses a Fedora 28 image, because of the specific Linux kernel
> version requirements of the test.
>
> For the sake of stability, let's use URLs from the archived and
> supposedely ever stable URLs. The good news is that the currently
> supported versions are also hosted on the later. This change limits
> itself to change the URLs, while keeping the fetched files the same
> (as can be evidenced by the unchanged hashes).
>
> Documentation and the "vm tests" fedora definition were also updated.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
> qemu-doc.texi | 6 +++---
> tests/acceptance/boot_linux_console.py | 25 +++++++++++++++----------
> tests/acceptance/linux_initrd.py | 5 +++--
> tests/vm/fedora | 2 +-
> 4 files changed, 22 insertions(+), 16 deletions(-)
>
> diff --git a/qemu-doc.texi b/qemu-doc.texi
> index 577d1e8376..37795f86fb 100644
> --- a/qemu-doc.texi
> +++ b/qemu-doc.texi
> @@ -440,15 +440,15 @@ of <protocol>.
>
> Example: boot from a remote Fedora 20 live ISO image
> @example
> -qemu-system-x86_64 --drive media=cdrom,file=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> +qemu-system-x86_64 --drive media=cdrom,file=https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
>
> -qemu-system-x86_64 --drive media=cdrom,file.driver=http,file.url=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> +qemu-system-x86_64 --drive media=cdrom,file.driver=http,file.url=http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> @end example
>
> Example: boot from a remote Fedora 20 cloud image using a local overlay for
> writes, copy-on-read, and a readahead of 64k
> @example
> -qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",, "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
> +qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, "file.url":"http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",, "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
>
> qemu-system-x86_64 -drive file=/tmp/Fedora-x86_64-20-20131211.1-sda.qcow2,copy-on-read=on
> @end example
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 2504ef0150..8a9a314ab4 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -76,8 +76,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:x86_64
> :avocado: tags=machine:pc
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
> - 'releases/29/Everything/x86_64/os/images/pxeboot/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> + '/linux/releases/29/Everything/x86_64/os/images/pxeboot'
> + '/vmlinuz')
> kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> @@ -250,8 +251,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:aarch64
> :avocado: tags=machine:virt
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
> - 'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> + '/linux/releases/29/Everything/aarch64/os/images/pxeboot'
> + '/vmlinuz')
> kernel_hash = '8c73e469fc6ea06a58dc83a628fc695b693b8493'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> @@ -271,8 +273,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:arm
> :avocado: tags=machine:virt
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
> - 'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> + '/linux/releases/29/Everything/armhfp/os/images/pxeboot'
> + '/vmlinuz')
> kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> @@ -318,8 +321,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:s390x
> :avocado: tags=machine:s390_ccw_virtio
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/'
> - 'releases/29/Everything/s390x/os/images/kernel.img')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive'
> + '/fedora-secondary/releases/29/Everything/s390x/os/images'
> + '/kernel.img')
> kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> @@ -360,8 +364,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:ppc64
> :avocado: tags=machine:pseries
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/'
> - 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive'
> + '/fedora-secondary/releases/29/Everything/ppc64le/os'
> + '/ppc/ppc64/vmlinuz')
> kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> diff --git a/tests/acceptance/linux_initrd.py b/tests/acceptance/linux_initrd.py
> index 23be5a63aa..c61d9826a4 100644
> --- a/tests/acceptance/linux_initrd.py
> +++ b/tests/acceptance/linux_initrd.py
> @@ -54,8 +54,9 @@ class LinuxInitrd(Test):
> QEMU has supported up to 4 GiB initrd for recent kernel
> Expect guest can reach 'Unpacking initramfs...'
> """
> - kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
> - 'Everything/x86_64/os/images/pxeboot/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> + '/linux/releases/28/Everything/x86_64/os/images/pxeboot/'
> + 'vmlinuz')
> kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> max_size = 2 * (1024 ** 3) + 1
> diff --git a/tests/vm/fedora b/tests/vm/fedora
> index e8fa5bf0d2..7fec1479fb 100755
> --- a/tests/vm/fedora
> +++ b/tests/vm/fedora
> @@ -23,7 +23,7 @@ class FedoraVM(basevm.BaseVM):
> name = "fedora"
> arch = "x86_64"
>
> - base = "http://dl.fedoraproject.org/pub/fedora/linux/releases/30/"
> + base = "https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/"
> link = base + "Server/x86_64/iso/Fedora-Server-netinst-x86_64-30-1.2.iso"
> repo = base + "Server/x86_64/os/"
> full = base + "Everything/x86_64/os/"
Currently it has just a few test cases that boot a Linux kernel. At this
point maintain the urls to fetch the files across a few source file is
manageable but it can get complex as it increases in number of tests.
While ago I started to work on the introduction of a "kernel provider"
mechanism to replace this "fetch a kernel" boilerplate. Unfortunately I
didn't time to complete its implementation, now If you think it is
valuable then I can manage to resume the work. Find the draft
implementation here:
https://github.com/wainersm/qemu/tree/acceptance_boot_linux
An alternate approach would be to extend Avocado by mimic the
avocado.utils.vmimage module [1], but rather have kernel providers.
[1]
https://avocado-framework.readthedocs.io/en/71.0/api/utils/avocado.utils.html#module-avocado.utils.vmimage
Anyway, the changes proposed on this patch looks good to me.
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Thanks,
Wainer
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"
2019-09-04 12:36 ` Wainer dos Santos Moschetta
@ 2019-09-04 13:05 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 13+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-09-04 13:05 UTC (permalink / raw)
To: Wainer dos Santos Moschetta, Cleber Rosa, Peter Maydell, qemu-devel
Cc: Yash Mankad, Fam Zheng, Alex Bennée, Eduardo Habkost,
Willian Rampazzo
On 9/4/19 2:36 PM, Wainer dos Santos Moschetta wrote:
> Hi Cleber,
>
> On 09/03/2019 09:52 PM, Cleber Rosa wrote:
>> The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
>> from tests/acceptance/linux_initrd.py, is currently failing to fetch
>> the "vmlinuz" file. The reason for the failure is that the Fedora
>> project retires older versions from the "dl.fedoraproject.org" URL,
>> and keeps them in "archives.fedoraproject.org". As an added note,
>> that test uses a Fedora 28 image, because of the specific Linux kernel
>> version requirements of the test.
>>
>> For the sake of stability, let's use URLs from the archived and
>> supposedely ever stable URLs. The good news is that the currently
>> supported versions are also hosted on the later. This change limits
>> itself to change the URLs, while keeping the fetched files the same
>> (as can be evidenced by the unchanged hashes).
>>
>> Documentation and the "vm tests" fedora definition were also updated.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>> qemu-doc.texi | 6 +++---
>> tests/acceptance/boot_linux_console.py | 25 +++++++++++++++----------
>> tests/acceptance/linux_initrd.py | 5 +++--
>> tests/vm/fedora | 2 +-
>> 4 files changed, 22 insertions(+), 16 deletions(-)
>>
>> diff --git a/qemu-doc.texi b/qemu-doc.texi
>> index 577d1e8376..37795f86fb 100644
>> --- a/qemu-doc.texi
>> +++ b/qemu-doc.texi
>> @@ -440,15 +440,15 @@ of <protocol>.
>> Example: boot from a remote Fedora 20 live ISO image
>> @example
>> -qemu-system-x86_64 --drive
>> media=cdrom,file=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
>>
>> +qemu-system-x86_64 --drive
>> media=cdrom,file=https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
>>
>> -qemu-system-x86_64 --drive
>> media=cdrom,file.driver=http,file.url=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
>>
>> +qemu-system-x86_64 --drive
>> media=cdrom,file.driver=http,file.url=http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
>>
>> @end example
>> Example: boot from a remote Fedora 20 cloud image using a local
>> overlay for
>> writes, copy-on-read, and a readahead of 64k
>> @example
>> -qemu-img create -f qcow2 -o
>> backing_file='json:@{"file.driver":"http",,
>> "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",,
>> "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
>> +qemu-img create -f qcow2 -o
>> backing_file='json:@{"file.driver":"http",,
>> "file.url":"http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",,
>> "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
>> qemu-system-x86_64 -drive
>> file=/tmp/Fedora-x86_64-20-20131211.1-sda.qcow2,copy-on-read=on
>> @end example
>> diff --git a/tests/acceptance/boot_linux_console.py
>> b/tests/acceptance/boot_linux_console.py
>> index 2504ef0150..8a9a314ab4 100644
>> --- a/tests/acceptance/boot_linux_console.py
>> +++ b/tests/acceptance/boot_linux_console.py
>> @@ -76,8 +76,9 @@ class BootLinuxConsole(Test):
>> :avocado: tags=arch:x86_64
>> :avocado: tags=machine:pc
>> """
>> - kernel_url =
>> ('https://download.fedoraproject.org/pub/fedora/linux/'
>> -
>> 'releases/29/Everything/x86_64/os/images/pxeboot/vmlinuz')
>> + kernel_url =
>> ('https://archives.fedoraproject.org/pub/archive/fedora'
>> +
>> '/linux/releases/29/Everything/x86_64/os/images/pxeboot'
>> + '/vmlinuz')
>> kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
>> kernel_path = self.fetch_asset(kernel_url,
>> asset_hash=kernel_hash)
>> @@ -250,8 +251,9 @@ class BootLinuxConsole(Test):
>> :avocado: tags=arch:aarch64
>> :avocado: tags=machine:virt
>> """
>> - kernel_url =
>> ('https://download.fedoraproject.org/pub/fedora/linux/'
>> -
>> 'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
>> + kernel_url =
>> ('https://archives.fedoraproject.org/pub/archive/fedora'
>> +
>> '/linux/releases/29/Everything/aarch64/os/images/pxeboot'
>> + '/vmlinuz')
>> kernel_hash = '8c73e469fc6ea06a58dc83a628fc695b693b8493'
>> kernel_path = self.fetch_asset(kernel_url,
>> asset_hash=kernel_hash)
>> @@ -271,8 +273,9 @@ class BootLinuxConsole(Test):
>> :avocado: tags=arch:arm
>> :avocado: tags=machine:virt
>> """
>> - kernel_url =
>> ('https://download.fedoraproject.org/pub/fedora/linux/'
>> -
>> 'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
>> + kernel_url =
>> ('https://archives.fedoraproject.org/pub/archive/fedora'
>> +
>> '/linux/releases/29/Everything/armhfp/os/images/pxeboot'
>> + '/vmlinuz')
>> kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4'
>> kernel_path = self.fetch_asset(kernel_url,
>> asset_hash=kernel_hash)
>> @@ -318,8 +321,9 @@ class BootLinuxConsole(Test):
>> :avocado: tags=arch:s390x
>> :avocado: tags=machine:s390_ccw_virtio
>> """
>> - kernel_url =
>> ('https://download.fedoraproject.org/pub/fedora-secondary/'
>> -
>> 'releases/29/Everything/s390x/os/images/kernel.img')
>> + kernel_url = ('https://archives.fedoraproject.org/pub/archive'
>> +
>> '/fedora-secondary/releases/29/Everything/s390x/os/images'
>> + '/kernel.img')
>> kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313'
>> kernel_path = self.fetch_asset(kernel_url,
>> asset_hash=kernel_hash)
>> @@ -360,8 +364,9 @@ class BootLinuxConsole(Test):
>> :avocado: tags=arch:ppc64
>> :avocado: tags=machine:pseries
>> """
>> - kernel_url =
>> ('https://download.fedoraproject.org/pub/fedora-secondary/'
>> -
>> 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz')
>> + kernel_url = ('https://archives.fedoraproject.org/pub/archive'
>> +
>> '/fedora-secondary/releases/29/Everything/ppc64le/os'
>> + '/ppc/ppc64/vmlinuz')
>> kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
>> kernel_path = self.fetch_asset(kernel_url,
>> asset_hash=kernel_hash)
>> diff --git a/tests/acceptance/linux_initrd.py
>> b/tests/acceptance/linux_initrd.py
>> index 23be5a63aa..c61d9826a4 100644
>> --- a/tests/acceptance/linux_initrd.py
>> +++ b/tests/acceptance/linux_initrd.py
>> @@ -54,8 +54,9 @@ class LinuxInitrd(Test):
>> QEMU has supported up to 4 GiB initrd for recent kernel
>> Expect guest can reach 'Unpacking initramfs...'
>> """
>> - kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
>> - 'Everything/x86_64/os/images/pxeboot/vmlinuz')
>> + kernel_url =
>> ('https://archives.fedoraproject.org/pub/archive/fedora'
>> +
>> '/linux/releases/28/Everything/x86_64/os/images/pxeboot/'
>> + 'vmlinuz')
>> kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
>> kernel_path = self.fetch_asset(kernel_url,
>> asset_hash=kernel_hash)
>> max_size = 2 * (1024 ** 3) + 1
>> diff --git a/tests/vm/fedora b/tests/vm/fedora
>> index e8fa5bf0d2..7fec1479fb 100755
>> --- a/tests/vm/fedora
>> +++ b/tests/vm/fedora
>> @@ -23,7 +23,7 @@ class FedoraVM(basevm.BaseVM):
>> name = "fedora"
>> arch = "x86_64"
>> - base = "http://dl.fedoraproject.org/pub/fedora/linux/releases/30/"
>> + base =
>> "https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/"
>>
>> link = base +
>> "Server/x86_64/iso/Fedora-Server-netinst-x86_64-30-1.2.iso"
>> repo = base + "Server/x86_64/os/"
>> full = base + "Everything/x86_64/os/"
>
> Currently it has just a few test cases that boot a Linux kernel. At this
> point maintain the urls to fetch the files across a few source file is
> manageable but it can get complex as it increases in number of tests.
> While ago I started to work on the introduction of a "kernel provider"
> mechanism to replace this "fetch a kernel" boilerplate. Unfortunately I
> didn't time to complete its implementation, now If you think it is
> valuable then I can manage to resume the work. Find the draft
> implementation here:
> https://github.com/wainersm/qemu/tree/acceptance_boot_linux
I remember and liked it, maybe you should start with a post asking the
community if the effort is worthwhile and would be useful, and what
combinations (distrib{name/version} kernel{version} arch
profile{debug,lpae,...} ...) are expected.
>
> An alternate approach would be to extend Avocado by mimic the
> avocado.utils.vmimage module [1], but rather have kernel providers.
>
> [1]
> https://avocado-framework.readthedocs.io/en/71.0/api/utils/avocado.utils.html#module-avocado.utils.vmimage
>
>
> Anyway, the changes proposed on this patch looks good to me.
>
> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Ditto:
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"
2019-09-04 0:52 [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org" Cleber Rosa
2019-09-04 5:18 ` Yash Mankad
2019-09-04 12:36 ` Wainer dos Santos Moschetta
@ 2019-09-10 8:31 ` Peter Maydell
2019-09-10 8:38 ` Peter Maydell
2019-09-10 12:30 ` Cleber Rosa
2019-09-10 8:41 ` Daniel P. Berrangé
3 siblings, 2 replies; 13+ messages in thread
From: Peter Maydell @ 2019-09-10 8:31 UTC (permalink / raw)
To: Cleber Rosa
Cc: Yash Mankad, Fam Zheng, Eduardo Habkost,
Philippe Mathieu-Daudé,
QEMU Developers, Wainer dos Santos Moschetta, Willian Rampazzo,
Alex Bennée
On Wed, 4 Sep 2019 at 01:52, Cleber Rosa <crosa@redhat.com> wrote:
>
> The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
> from tests/acceptance/linux_initrd.py, is currently failing to fetch
> the "vmlinuz" file. The reason for the failure is that the Fedora
> project retires older versions from the "dl.fedoraproject.org" URL,
> and keeps them in "archives.fedoraproject.org". As an added note,
> that test uses a Fedora 28 image, because of the specific Linux kernel
> version requirements of the test.
>
> For the sake of stability, let's use URLs from the archived and
> supposedely ever stable URLs. The good news is that the currently
> supported versions are also hosted on the later. This change limits
> itself to change the URLs, while keeping the fetched files the same
> (as can be evidenced by the unchanged hashes).
>
> Documentation and the "vm tests" fedora definition were also updated.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
I was planning to apply this to master to fix the Travis builds,
but it doesn't apply -- conflicts in qemu-doc.texi. Could you respin?
thanks
-- PMM
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"
2019-09-10 8:31 ` Peter Maydell
@ 2019-09-10 8:38 ` Peter Maydell
2019-09-10 12:44 ` Cleber Rosa
2019-09-10 12:30 ` Cleber Rosa
1 sibling, 1 reply; 13+ messages in thread
From: Peter Maydell @ 2019-09-10 8:38 UTC (permalink / raw)
To: Cleber Rosa
Cc: Yash Mankad, Fam Zheng, Eduardo Habkost,
Philippe Mathieu-Daudé,
QEMU Developers, Wainer dos Santos Moschetta, Willian Rampazzo,
Alex Bennée
On Tue, 10 Sep 2019 at 09:31, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Wed, 4 Sep 2019 at 01:52, Cleber Rosa <crosa@redhat.com> wrote:
> >
> > The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
> > from tests/acceptance/linux_initrd.py, is currently failing to fetch
> > the "vmlinuz" file. The reason for the failure is that the Fedora
> > project retires older versions from the "dl.fedoraproject.org" URL,
> > and keeps them in "archives.fedoraproject.org". As an added note,
> > that test uses a Fedora 28 image, because of the specific Linux kernel
> > version requirements of the test.
> >
> > For the sake of stability, let's use URLs from the archived and
> > supposedely ever stable URLs. The good news is that the currently
> > supported versions are also hosted on the later. This change limits
> > itself to change the URLs, while keeping the fetched files the same
> > (as can be evidenced by the unchanged hashes).
> >
> > Documentation and the "vm tests" fedora definition were also updated.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
>
> I was planning to apply this to master to fix the Travis builds,
> but it doesn't apply -- conflicts in qemu-doc.texi. Could you respin?
Ah, never mind -- Alex tells me he's already picked this patch up
in a pending pull request and has done the rebase and fixup there.
thanks
-- PMM
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"
2019-09-04 0:52 [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org" Cleber Rosa
` (2 preceding siblings ...)
2019-09-10 8:31 ` Peter Maydell
@ 2019-09-10 8:41 ` Daniel P. Berrangé
2019-09-10 12:29 ` Cleber Rosa
3 siblings, 1 reply; 13+ messages in thread
From: Daniel P. Berrangé @ 2019-09-10 8:41 UTC (permalink / raw)
To: Cleber Rosa
Cc: Yash Mankad, Peter Maydell, Eduardo Habkost, Alex Bennée,
qemu-devel, Wainer dos Santos Moschetta, Willian Rampazzo,
Fam Zheng, Philippe Mathieu-Daudé
On Tue, Sep 03, 2019 at 08:52:18PM -0400, Cleber Rosa wrote:
> The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
> from tests/acceptance/linux_initrd.py, is currently failing to fetch
> the "vmlinuz" file. The reason for the failure is that the Fedora
> project retires older versions from the "dl.fedoraproject.org" URL,
> and keeps them in "archives.fedoraproject.org". As an added note,
> that test uses a Fedora 28 image, because of the specific Linux kernel
> version requirements of the test.
>
> For the sake of stability, let's use URLs from the archived and
> supposedely ever stable URLs. The good news is that the currently
> supported versions are also hosted on the later. This change limits
> itself to change the URLs, while keeping the fetched files the same
> (as can be evidenced by the unchanged hashes).
The download.fedoraproject.org site we're (mostly) currently using
is serviced by the Fedora mirrors which is very desirable as it
spreads the load.
The archive.fedoraproject.org site is the master Fedora hosting
server(s). dl.fedoraproject.org is the same master hosting service
that is intended for use by the mirror sites to sync their content
from. Projects really shouldn't use either of these URLs for getting
any content which is available via the mirror service as it places
uncessary load on the Fedora master servers.
> Documentation and the "vm tests" fedora definition were also updated.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
> qemu-doc.texi | 6 +++---
> tests/acceptance/boot_linux_console.py | 25 +++++++++++++++----------
> tests/acceptance/linux_initrd.py | 5 +++--
> tests/vm/fedora | 2 +-
> 4 files changed, 22 insertions(+), 16 deletions(-)
>
> diff --git a/qemu-doc.texi b/qemu-doc.texi
> index 577d1e8376..37795f86fb 100644
> --- a/qemu-doc.texi
> +++ b/qemu-doc.texi
> @@ -440,15 +440,15 @@ of <protocol>.
>
> Example: boot from a remote Fedora 20 live ISO image
> @example
> -qemu-system-x86_64 --drive media=cdrom,file=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> +qemu-system-x86_64 --drive media=cdrom,file=https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
>
> -qemu-system-x86_64 --drive media=cdrom,file.driver=http,file.url=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> +qemu-system-x86_64 --drive media=cdrom,file.driver=http,file.url=http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> @end example
>
> Example: boot from a remote Fedora 20 cloud image using a local overlay for
> writes, copy-on-read, and a readahead of 64k
> @example
> -qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",, "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
> +qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, "file.url":"http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",, "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
>
> qemu-system-x86_64 -drive file=/tmp/Fedora-x86_64-20-20131211.1-sda.qcow2,copy-on-read=on
> @end example
Nothing quite says "cutting edge" like examples pointing to 5+ years old
software. Or to put it another way, we should update this to point to a
modern OS release. ie Fedora 30.
These should also be changed to download instead of dl.
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 2504ef0150..8a9a314ab4 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -76,8 +76,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:x86_64
> :avocado: tags=machine:pc
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
> - 'releases/29/Everything/x86_64/os/images/pxeboot/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> + '/linux/releases/29/Everything/x86_64/os/images/pxeboot'
> + '/vmlinuz')
> kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> @@ -250,8 +251,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:aarch64
> :avocado: tags=machine:virt
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
> - 'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> + '/linux/releases/29/Everything/aarch64/os/images/pxeboot'
> + '/vmlinuz')
> kernel_hash = '8c73e469fc6ea06a58dc83a628fc695b693b8493'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> @@ -271,8 +273,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:arm
> :avocado: tags=machine:virt
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
> - 'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> + '/linux/releases/29/Everything/armhfp/os/images/pxeboot'
> + '/vmlinuz')
> kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> @@ -318,8 +321,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:s390x
> :avocado: tags=machine:s390_ccw_virtio
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/'
> - 'releases/29/Everything/s390x/os/images/kernel.img')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive'
> + '/fedora-secondary/releases/29/Everything/s390x/os/images'
> + '/kernel.img')
> kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> @@ -360,8 +364,9 @@ class BootLinuxConsole(Test):
> :avocado: tags=arch:ppc64
> :avocado: tags=machine:pseries
> """
> - kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/'
> - 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive'
> + '/fedora-secondary/releases/29/Everything/ppc64le/os'
> + '/ppc/ppc64/vmlinuz')
> kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
Don't change these URLs. We can update to Fedora 30 though if you want
to.
>
> diff --git a/tests/acceptance/linux_initrd.py b/tests/acceptance/linux_initrd.py
> index 23be5a63aa..c61d9826a4 100644
> --- a/tests/acceptance/linux_initrd.py
> +++ b/tests/acceptance/linux_initrd.py
> @@ -54,8 +54,9 @@ class LinuxInitrd(Test):
> QEMU has supported up to 4 GiB initrd for recent kernel
> Expect guest can reach 'Unpacking initramfs...'
> """
> - kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
> - 'Everything/x86_64/os/images/pxeboot/vmlinuz')
> + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> + '/linux/releases/28/Everything/x86_64/os/images/pxeboot/'
> + 'vmlinuz')
> kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
> kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> max_size = 2 * (1024 ** 3) + 1
> diff --git a/tests/vm/fedora b/tests/vm/fedora
> index e8fa5bf0d2..7fec1479fb 100755
> --- a/tests/vm/fedora
> +++ b/tests/vm/fedora
> @@ -23,7 +23,7 @@ class FedoraVM(basevm.BaseVM):
> name = "fedora"
> arch = "x86_64"
>
> - base = "http://dl.fedoraproject.org/pub/fedora/linux/releases/30/"
> + base = "https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/"
> link = base + "Server/x86_64/iso/Fedora-Server-netinst-x86_64-30-1.2.iso"
> repo = base + "Server/x86_64/os/"
> full = base + "Everything/x86_64/os/"
This should be changed to download instead of dl.
> --
> 2.21.0
>
>
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"
2019-09-10 8:41 ` Daniel P. Berrangé
@ 2019-09-10 12:29 ` Cleber Rosa
2019-09-10 19:43 ` Eduardo Habkost
0 siblings, 1 reply; 13+ messages in thread
From: Cleber Rosa @ 2019-09-10 12:29 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Yash Mankad, Peter Maydell, Eduardo Habkost, Alex Bennée,
qemu-devel, Wainer dos Santos Moschetta, Willian Rampazzo,
Fam Zheng, Philippe Mathieu-Daudé
On Tue, Sep 10, 2019 at 09:41:20AM +0100, Daniel P. Berrangé wrote:
> On Tue, Sep 03, 2019 at 08:52:18PM -0400, Cleber Rosa wrote:
> > The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
> > from tests/acceptance/linux_initrd.py, is currently failing to fetch
> > the "vmlinuz" file. The reason for the failure is that the Fedora
> > project retires older versions from the "dl.fedoraproject.org" URL,
> > and keeps them in "archives.fedoraproject.org". As an added note,
> > that test uses a Fedora 28 image, because of the specific Linux kernel
> > version requirements of the test.
> >
> > For the sake of stability, let's use URLs from the archived and
> > supposedely ever stable URLs. The good news is that the currently
> > supported versions are also hosted on the later. This change limits
> > itself to change the URLs, while keeping the fetched files the same
> > (as can be evidenced by the unchanged hashes).
>
> The download.fedoraproject.org site we're (mostly) currently using
> is serviced by the Fedora mirrors which is very desirable as it
> spreads the load.
>
> The archive.fedoraproject.org site is the master Fedora hosting
> server(s). dl.fedoraproject.org is the same master hosting service
> that is intended for use by the mirror sites to sync their content
> from. Projects really shouldn't use either of these URLs for getting
> any content which is available via the mirror service as it places
> uncessary load on the Fedora master servers.
>
> > Documentation and the "vm tests" fedora definition were also updated.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> > qemu-doc.texi | 6 +++---
> > tests/acceptance/boot_linux_console.py | 25 +++++++++++++++----------
> > tests/acceptance/linux_initrd.py | 5 +++--
> > tests/vm/fedora | 2 +-
> > 4 files changed, 22 insertions(+), 16 deletions(-)
> >
> > diff --git a/qemu-doc.texi b/qemu-doc.texi
> > index 577d1e8376..37795f86fb 100644
> > --- a/qemu-doc.texi
> > +++ b/qemu-doc.texi
> > @@ -440,15 +440,15 @@ of <protocol>.
> >
> > Example: boot from a remote Fedora 20 live ISO image
> > @example
> > -qemu-system-x86_64 --drive media=cdrom,file=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> > +qemu-system-x86_64 --drive media=cdrom,file=https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> >
> > -qemu-system-x86_64 --drive media=cdrom,file.driver=http,file.url=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> > +qemu-system-x86_64 --drive media=cdrom,file.driver=http,file.url=http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> > @end example
> >
> > Example: boot from a remote Fedora 20 cloud image using a local overlay for
> > writes, copy-on-read, and a readahead of 64k
> > @example
> > -qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",, "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
> > +qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, "file.url":"http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",, "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
> >
> > qemu-system-x86_64 -drive file=/tmp/Fedora-x86_64-20-20131211.1-sda.qcow2,copy-on-read=on
> > @end example
>
> Nothing quite says "cutting edge" like examples pointing to 5+ years old
> software. Or to put it another way, we should update this to point to a
> modern OS release. ie Fedora 30.
>
Hi Daniel,
You're right, we should, and that's an obvious next step. I tried to
keep this patch within the context of fixing the test regressions, and
making the overall experience stable.
> These should also be changed to download instead of dl.
>
I disagree, specially here, and believe that we shouldn't use the
"archive" site. Having documentation that is accurate and whose
examples can be reproduced without caveats is a nice usability
touch.
I understand your worries about the eventual load on the Fedora
server. While I'm certainly not qualified to speak for the Fedora
project, it'd be natural that the project would eventually cope
with the additional load, if that's actually significant.
>
> > diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> > index 2504ef0150..8a9a314ab4 100644
> > --- a/tests/acceptance/boot_linux_console.py
> > +++ b/tests/acceptance/boot_linux_console.py
> > @@ -76,8 +76,9 @@ class BootLinuxConsole(Test):
> > :avocado: tags=arch:x86_64
> > :avocado: tags=machine:pc
> > """
> > - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
> > - 'releases/29/Everything/x86_64/os/images/pxeboot/vmlinuz')
> > + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> > + '/linux/releases/29/Everything/x86_64/os/images/pxeboot'
> > + '/vmlinuz')
> > kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
> > kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >
> > @@ -250,8 +251,9 @@ class BootLinuxConsole(Test):
> > :avocado: tags=arch:aarch64
> > :avocado: tags=machine:virt
> > """
> > - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
> > - 'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
> > + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> > + '/linux/releases/29/Everything/aarch64/os/images/pxeboot'
> > + '/vmlinuz')
> > kernel_hash = '8c73e469fc6ea06a58dc83a628fc695b693b8493'
> > kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >
> > @@ -271,8 +273,9 @@ class BootLinuxConsole(Test):
> > :avocado: tags=arch:arm
> > :avocado: tags=machine:virt
> > """
> > - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
> > - 'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
> > + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> > + '/linux/releases/29/Everything/armhfp/os/images/pxeboot'
> > + '/vmlinuz')
> > kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4'
> > kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >
> > @@ -318,8 +321,9 @@ class BootLinuxConsole(Test):
> > :avocado: tags=arch:s390x
> > :avocado: tags=machine:s390_ccw_virtio
> > """
> > - kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/'
> > - 'releases/29/Everything/s390x/os/images/kernel.img')
> > + kernel_url = ('https://archives.fedoraproject.org/pub/archive'
> > + '/fedora-secondary/releases/29/Everything/s390x/os/images'
> > + '/kernel.img')
> > kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313'
> > kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >
> > @@ -360,8 +364,9 @@ class BootLinuxConsole(Test):
> > :avocado: tags=arch:ppc64
> > :avocado: tags=machine:pseries
> > """
> > - kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/'
> > - 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz')
> > + kernel_url = ('https://archives.fedoraproject.org/pub/archive'
> > + '/fedora-secondary/releases/29/Everything/ppc64le/os'
> > + '/ppc/ppc64/vmlinuz')
> > kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
> > kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
> Don't change these URLs. We can update to Fedora 30 though if you want
> to.
>
I believe your suggestion to not change the URLs is based solely on your
perspective on the load on the "archives" server, right?
You should know that there's a number of counter points. One is that
these tests are used on environments, which are maintained for a much
longer time than the lifespan of the currently maintained Fedora
versions. We really need to make it stable and reliable, and IMO it
should be done upstream, for the benefit of all.
Best,
- Cleber.
> >
> > diff --git a/tests/acceptance/linux_initrd.py b/tests/acceptance/linux_initrd.py
> > index 23be5a63aa..c61d9826a4 100644
> > --- a/tests/acceptance/linux_initrd.py
> > +++ b/tests/acceptance/linux_initrd.py
> > @@ -54,8 +54,9 @@ class LinuxInitrd(Test):
> > QEMU has supported up to 4 GiB initrd for recent kernel
> > Expect guest can reach 'Unpacking initramfs...'
> > """
> > - kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
> > - 'Everything/x86_64/os/images/pxeboot/vmlinuz')
> > + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
> > + '/linux/releases/28/Everything/x86_64/os/images/pxeboot/'
> > + 'vmlinuz')
> > kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
> > kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> > max_size = 2 * (1024 ** 3) + 1
> > diff --git a/tests/vm/fedora b/tests/vm/fedora
> > index e8fa5bf0d2..7fec1479fb 100755
> > --- a/tests/vm/fedora
> > +++ b/tests/vm/fedora
> > @@ -23,7 +23,7 @@ class FedoraVM(basevm.BaseVM):
> > name = "fedora"
> > arch = "x86_64"
> >
> > - base = "http://dl.fedoraproject.org/pub/fedora/linux/releases/30/"
> > + base = "https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/"
> > link = base + "Server/x86_64/iso/Fedora-Server-netinst-x86_64-30-1.2.iso"
> > repo = base + "Server/x86_64/os/"
> > full = base + "Everything/x86_64/os/"
>
> This should be changed to download instead of dl.
>
> > --
> > 2.21.0
> >
> >
>
> Regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"
2019-09-10 8:31 ` Peter Maydell
2019-09-10 8:38 ` Peter Maydell
@ 2019-09-10 12:30 ` Cleber Rosa
1 sibling, 0 replies; 13+ messages in thread
From: Cleber Rosa @ 2019-09-10 12:30 UTC (permalink / raw)
To: Peter Maydell
Cc: Yash Mankad, Fam Zheng, Eduardo Habkost, Alex Bennée,
QEMU Developers, Wainer dos Santos Moschetta, Willian Rampazzo,
Philippe Mathieu-Daudé
On Tue, Sep 10, 2019 at 09:31:30AM +0100, Peter Maydell wrote:
> On Wed, 4 Sep 2019 at 01:52, Cleber Rosa <crosa@redhat.com> wrote:
> >
> > The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
> > from tests/acceptance/linux_initrd.py, is currently failing to fetch
> > the "vmlinuz" file. The reason for the failure is that the Fedora
> > project retires older versions from the "dl.fedoraproject.org" URL,
> > and keeps them in "archives.fedoraproject.org". As an added note,
> > that test uses a Fedora 28 image, because of the specific Linux kernel
> > version requirements of the test.
> >
> > For the sake of stability, let's use URLs from the archived and
> > supposedely ever stable URLs. The good news is that the currently
> > supported versions are also hosted on the later. This change limits
> > itself to change the URLs, while keeping the fetched files the same
> > (as can be evidenced by the unchanged hashes).
> >
> > Documentation and the "vm tests" fedora definition were also updated.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
>
> I was planning to apply this to master to fix the Travis builds,
> but it doesn't apply -- conflicts in qemu-doc.texi. Could you respin?
>
> thanks
> -- PMM
>
Sure!
- Cleber.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"
2019-09-10 8:38 ` Peter Maydell
@ 2019-09-10 12:44 ` Cleber Rosa
0 siblings, 0 replies; 13+ messages in thread
From: Cleber Rosa @ 2019-09-10 12:44 UTC (permalink / raw)
To: Peter Maydell
Cc: Yash Mankad, Fam Zheng, Eduardo Habkost,
Philippe Mathieu-Daudé,
QEMU Developers, Wainer dos Santos Moschetta, Willian Rampazzo,
Alex Bennée
On Tue, Sep 10, 2019 at 09:38:55AM +0100, Peter Maydell wrote:
> On Tue, 10 Sep 2019 at 09:31, Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > On Wed, 4 Sep 2019 at 01:52, Cleber Rosa <crosa@redhat.com> wrote:
> > >
> > > The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
> > > from tests/acceptance/linux_initrd.py, is currently failing to fetch
> > > the "vmlinuz" file. The reason for the failure is that the Fedora
> > > project retires older versions from the "dl.fedoraproject.org" URL,
> > > and keeps them in "archives.fedoraproject.org". As an added note,
> > > that test uses a Fedora 28 image, because of the specific Linux kernel
> > > version requirements of the test.
> > >
> > > For the sake of stability, let's use URLs from the archived and
> > > supposedely ever stable URLs. The good news is that the currently
> > > supported versions are also hosted on the later. This change limits
> > > itself to change the URLs, while keeping the fetched files the same
> > > (as can be evidenced by the unchanged hashes).
> > >
> > > Documentation and the "vm tests" fedora definition were also updated.
> > >
> > > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > > ---
> >
> > I was planning to apply this to master to fix the Travis builds,
> > but it doesn't apply -- conflicts in qemu-doc.texi. Could you respin?
>
> Ah, never mind -- Alex tells me he's already picked this patch up
> in a pending pull request and has done the rebase and fixup there.
>
> thanks
> -- PMM
Oh, OK, good!
Thanks!
- Cleber.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"
2019-09-10 12:29 ` Cleber Rosa
@ 2019-09-10 19:43 ` Eduardo Habkost
2019-09-10 20:16 ` Cleber Rosa
0 siblings, 1 reply; 13+ messages in thread
From: Eduardo Habkost @ 2019-09-10 19:43 UTC (permalink / raw)
To: Cleber Rosa
Cc: Yash Mankad, Peter Maydell, Daniel P. Berrangé,
Alex Bennée, qemu-devel, Wainer dos Santos Moschetta,
Willian Rampazzo, Fam Zheng, Philippe Mathieu-Daudé
On Tue, Sep 10, 2019 at 08:29:24AM -0400, Cleber Rosa wrote:
> On Tue, Sep 10, 2019 at 09:41:20AM +0100, Daniel P. Berrangé wrote:
> > On Tue, Sep 03, 2019 at 08:52:18PM -0400, Cleber Rosa wrote:
> > > The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
> > > from tests/acceptance/linux_initrd.py, is currently failing to fetch
> > > the "vmlinuz" file. The reason for the failure is that the Fedora
> > > project retires older versions from the "dl.fedoraproject.org" URL,
> > > and keeps them in "archives.fedoraproject.org". As an added note,
> > > that test uses a Fedora 28 image, because of the specific Linux kernel
> > > version requirements of the test.
> > >
> > > For the sake of stability, let's use URLs from the archived and
> > > supposedely ever stable URLs. The good news is that the currently
> > > supported versions are also hosted on the later. This change limits
> > > itself to change the URLs, while keeping the fetched files the same
> > > (as can be evidenced by the unchanged hashes).
> >
> > The download.fedoraproject.org site we're (mostly) currently using
> > is serviced by the Fedora mirrors which is very desirable as it
> > spreads the load.
> >
> > The archive.fedoraproject.org site is the master Fedora hosting
> > server(s). dl.fedoraproject.org is the same master hosting service
> > that is intended for use by the mirror sites to sync their content
> > from. Projects really shouldn't use either of these URLs for getting
> > any content which is available via the mirror service as it places
> > uncessary load on the Fedora master servers.
> >
[...]
> > > @@ -360,8 +364,9 @@ class BootLinuxConsole(Test):
> > > :avocado: tags=arch:ppc64
> > > :avocado: tags=machine:pseries
> > > """
> > > - kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/'
> > > - 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz')
> > > + kernel_url = ('https://archives.fedoraproject.org/pub/archive'
> > > + '/fedora-secondary/releases/29/Everything/ppc64le/os'
> > > + '/ppc/ppc64/vmlinuz')
> > > kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
> > > kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >
> > Don't change these URLs. We can update to Fedora 30 though if you want
> > to.
> >
>
> I believe your suggestion to not change the URLs is based solely on your
> perspective on the load on the "archives" server, right?
>
> You should know that there's a number of counter points. One is that
> these tests are used on environments, which are maintained for a much
> longer time than the lifespan of the currently maintained Fedora
> versions. We really need to make it stable and reliable, and IMO it
> should be done upstream, for the benefit of all.
We're trying to offload the costs of long term hosting for our
test cases to a third party (Fedora Project), but we need to know
if that's really acceptable usage of archives.fedoraproject.org.
--
Eduardo
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"
2019-09-10 19:43 ` Eduardo Habkost
@ 2019-09-10 20:16 ` Cleber Rosa
2019-09-11 15:22 ` Daniel P. Berrangé
0 siblings, 1 reply; 13+ messages in thread
From: Cleber Rosa @ 2019-09-10 20:16 UTC (permalink / raw)
To: Eduardo Habkost
Cc: Yash Mankad, Peter Maydell, Daniel P. Berrangé,
Alex Bennée, qemu-devel, Wainer dos Santos Moschetta,
Willian Rampazzo, Fam Zheng, Philippe Mathieu-Daudé
On Tue, Sep 10, 2019 at 04:43:00PM -0300, Eduardo Habkost wrote:
> On Tue, Sep 10, 2019 at 08:29:24AM -0400, Cleber Rosa wrote:
> > On Tue, Sep 10, 2019 at 09:41:20AM +0100, Daniel P. Berrangé wrote:
> > > On Tue, Sep 03, 2019 at 08:52:18PM -0400, Cleber Rosa wrote:
> > > > The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
> > > > from tests/acceptance/linux_initrd.py, is currently failing to fetch
> > > > the "vmlinuz" file. The reason for the failure is that the Fedora
> > > > project retires older versions from the "dl.fedoraproject.org" URL,
> > > > and keeps them in "archives.fedoraproject.org". As an added note,
> > > > that test uses a Fedora 28 image, because of the specific Linux kernel
> > > > version requirements of the test.
> > > >
> > > > For the sake of stability, let's use URLs from the archived and
> > > > supposedely ever stable URLs. The good news is that the currently
> > > > supported versions are also hosted on the later. This change limits
> > > > itself to change the URLs, while keeping the fetched files the same
> > > > (as can be evidenced by the unchanged hashes).
> > >
> > > The download.fedoraproject.org site we're (mostly) currently using
> > > is serviced by the Fedora mirrors which is very desirable as it
> > > spreads the load.
> > >
> > > The archive.fedoraproject.org site is the master Fedora hosting
> > > server(s). dl.fedoraproject.org is the same master hosting service
> > > that is intended for use by the mirror sites to sync their content
> > > from. Projects really shouldn't use either of these URLs for getting
> > > any content which is available via the mirror service as it places
> > > uncessary load on the Fedora master servers.
> > >
> [...]
> > > > @@ -360,8 +364,9 @@ class BootLinuxConsole(Test):
> > > > :avocado: tags=arch:ppc64
> > > > :avocado: tags=machine:pseries
> > > > """
> > > > - kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/'
> > > > - 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz')
> > > > + kernel_url = ('https://archives.fedoraproject.org/pub/archive'
> > > > + '/fedora-secondary/releases/29/Everything/ppc64le/os'
> > > > + '/ppc/ppc64/vmlinuz')
> > > > kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
> > > > kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> > >
> > > Don't change these URLs. We can update to Fedora 30 though if you want
> > > to.
> > >
> >
> > I believe your suggestion to not change the URLs is based solely on your
> > perspective on the load on the "archives" server, right?
> >
> > You should know that there's a number of counter points. One is that
> > these tests are used on environments, which are maintained for a much
> > longer time than the lifespan of the currently maintained Fedora
> > versions. We really need to make it stable and reliable, and IMO it
> > should be done upstream, for the benefit of all.
>
> We're trying to offload the costs of long term hosting for our
> test cases to a third party (Fedora Project), but we need to know
> if that's really acceptable usage of archives.fedoraproject.org.
>
> --
> Eduardo
I'd argue that we're boosting archives.fedoraproject.org morale. As a
public Internet service, it begs to better be used! :)
Now, more seriously, I agree that we should coordinate with them. I
have the feeling that it shouldn't be a problem of resources per se,
given that we're redirecting traffic. But, it may be a matter of
letting them better allocate these resources.
Daniel,
You seem to have some much better than average knowledge about the
mirror structure. Does that mean you know who to contact about this?
Thanks,
- Cleber.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"
2019-09-10 20:16 ` Cleber Rosa
@ 2019-09-11 15:22 ` Daniel P. Berrangé
0 siblings, 0 replies; 13+ messages in thread
From: Daniel P. Berrangé @ 2019-09-11 15:22 UTC (permalink / raw)
To: Cleber Rosa
Cc: Yash Mankad, Peter Maydell, Eduardo Habkost, Alex Bennée,
qemu-devel, Wainer dos Santos Moschetta, Willian Rampazzo,
Fam Zheng, Philippe Mathieu-Daudé
On Tue, Sep 10, 2019 at 04:16:58PM -0400, Cleber Rosa wrote:
> On Tue, Sep 10, 2019 at 04:43:00PM -0300, Eduardo Habkost wrote:
> > On Tue, Sep 10, 2019 at 08:29:24AM -0400, Cleber Rosa wrote:
> > > On Tue, Sep 10, 2019 at 09:41:20AM +0100, Daniel P. Berrangé wrote:
> > > > On Tue, Sep 03, 2019 at 08:52:18PM -0400, Cleber Rosa wrote:
> > > > > The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
> > > > > from tests/acceptance/linux_initrd.py, is currently failing to fetch
> > > > > the "vmlinuz" file. The reason for the failure is that the Fedora
> > > > > project retires older versions from the "dl.fedoraproject.org" URL,
> > > > > and keeps them in "archives.fedoraproject.org". As an added note,
> > > > > that test uses a Fedora 28 image, because of the specific Linux kernel
> > > > > version requirements of the test.
> > > > >
> > > > > For the sake of stability, let's use URLs from the archived and
> > > > > supposedely ever stable URLs. The good news is that the currently
> > > > > supported versions are also hosted on the later. This change limits
> > > > > itself to change the URLs, while keeping the fetched files the same
> > > > > (as can be evidenced by the unchanged hashes).
> > > >
> > > > The download.fedoraproject.org site we're (mostly) currently using
> > > > is serviced by the Fedora mirrors which is very desirable as it
> > > > spreads the load.
> > > >
> > > > The archive.fedoraproject.org site is the master Fedora hosting
> > > > server(s). dl.fedoraproject.org is the same master hosting service
> > > > that is intended for use by the mirror sites to sync their content
> > > > from. Projects really shouldn't use either of these URLs for getting
> > > > any content which is available via the mirror service as it places
> > > > uncessary load on the Fedora master servers.
> > > >
> > [...]
> > > > > @@ -360,8 +364,9 @@ class BootLinuxConsole(Test):
> > > > > :avocado: tags=arch:ppc64
> > > > > :avocado: tags=machine:pseries
> > > > > """
> > > > > - kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/'
> > > > > - 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz')
> > > > > + kernel_url = ('https://archives.fedoraproject.org/pub/archive'
> > > > > + '/fedora-secondary/releases/29/Everything/ppc64le/os'
> > > > > + '/ppc/ppc64/vmlinuz')
> > > > > kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
> > > > > kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> > > >
> > > > Don't change these URLs. We can update to Fedora 30 though if you want
> > > > to.
> > > >
> > >
> > > I believe your suggestion to not change the URLs is based solely on your
> > > perspective on the load on the "archives" server, right?
> > >
> > > You should know that there's a number of counter points. One is that
> > > these tests are used on environments, which are maintained for a much
> > > longer time than the lifespan of the currently maintained Fedora
> > > versions. We really need to make it stable and reliable, and IMO it
> > > should be done upstream, for the benefit of all.
> >
> > We're trying to offload the costs of long term hosting for our
> > test cases to a third party (Fedora Project), but we need to know
> > if that's really acceptable usage of archives.fedoraproject.org.
> >
> > --
> > Eduardo
>
> I'd argue that we're boosting archives.fedoraproject.org morale. As a
> public Internet service, it begs to better be used! :)
Not really, in general we want users to keep updated with latest
supported Fedora releases and not use EOL versions from archive.
The critical reason for archives.fp.org to exist is license,
beyond that its largely just a historical record, but we digress
off topic here...
> Now, more seriously, I agree that we should coordinate with them. I
> have the feeling that it shouldn't be a problem of resources per se,
> given that we're redirecting traffic. But, it may be a matter of
> letting them better allocate these resources.
>
> Daniel,
>
> You seem to have some much better than average knowledge about the
> mirror structure. Does that mean you know who to contact about this?
I spoke with some folks I know who didn't have an answer on the impact
or policy around using archive.fp.org, but did point out that there's
a way to avoid breaking
The Fedora mirror manager software provides a URL to query what
sites currently host a given version
eg to find Fedora 16 for x86_64 fetch this
https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-16&arch=x86_64
whereupon it reports something like this:
# repo = fedora-16 arch = x86_64 country = DK country = DE country = UA country = global
https://mirrors.dotsrc.org/fedora-buffet/archive/fedora/linux/releases/16/Everything/x86_64/os/
https://fedora-archive.ip-connect.info/fedora/linux/releases/16/Everything/x86_64/os/
https://ftp-stud.hs-esslingen.de/pub/Mirrors/archive.fedoraproject.org/fedora/linux/releases/16/Everything/x86_64/os/
https://fedora-archive.ip-connect.vn.ua/fedora/linux/releases/16/Everything/x86_64/os/
http://ftp.cuhk.edu.hk/pub/linux/fedora-archive/fedora/linux/releases/16/Everything/x86_64/os/
http://fedora-archive.mirror.liquidtelecom.com/archive/fedora/linux/releases/16/Everything/x86_64/os/
http://mirrors.kernel.org/fedora-buffet/archive/fedora/linux/releases/16/Everything/x86_64/os/
http://mirror.math.princeton.edu/pub/fedora-archive/fedora/linux/releases/16/Everything/x86_64/os/
https://pubmirror2.math.uh.edu/fedora-buffet/archive/fedora/linux/releases/16/Everything/x86_64/os/
https://pubmirror1.math.uh.edu/fedora-buffet/archive/fedora/linux/releases/16/Everything/x86_64/os/
https://dl.fedoraproject.org/pub/archive/fedora/linux/releases/16/Everything/x86_64/os/
any one of those returned URLs should serve the content we want. It is
interesting to see that there's several mirrors still carrying ancient
software, despite 'download.fp.org' returning 404
Admittedly this extra redirection is a bit more tedious for the test
suite, but we could hide that behind a common helper function.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2019-09-11 15:42 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-04 0:52 [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org" Cleber Rosa
2019-09-04 5:18 ` Yash Mankad
2019-09-04 12:36 ` Wainer dos Santos Moschetta
2019-09-04 13:05 ` Philippe Mathieu-Daudé
2019-09-10 8:31 ` Peter Maydell
2019-09-10 8:38 ` Peter Maydell
2019-09-10 12:44 ` Cleber Rosa
2019-09-10 12:30 ` Cleber Rosa
2019-09-10 8:41 ` Daniel P. Berrangé
2019-09-10 12:29 ` Cleber Rosa
2019-09-10 19:43 ` Eduardo Habkost
2019-09-10 20:16 ` Cleber Rosa
2019-09-11 15:22 ` Daniel P. Berrangé
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.