[PATCH v2 0/2] Allwinner H3 fixes for EMAC and acceptance tests

[PATCH v2 0/2] Allwinner H3 fixes for EMAC and acceptance tests

[Niek Linnenbank]

The following are maintenance patches for the Allwinner H3. The first patch
is a proposal to relocate the binary artifacts of the acceptance tests away
from the apt.armbian.com domain. In the past we had problems with artifacts being
removed, and now the recently added Armbian 20.08.1 image has been removed as well:

$ wget https://dl.armbian.com/orangepipc/archive/Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz
Connecting to dl.armbian.com (dl.armbian.com)|2605:7900:20::5|:443... connected.
...
HTTP request sent, awaiting response... 404 Not Found
2021-02-11 22:34:45 ERROR 404: Not Found.

I've now added the artifacts to github as a short term solution:
  https://github.com/nieklinnenbank/QemuArtifacts

The second patch is a fix for the EMAC that is used by the Allwinner H3 / Orange Pi PC machine.

ChangeLog:

v2:
 - added Reviewed-By tags
 - changed URL for artifacts to github.com/nieklinnenbank/QemuArtifacts

Kind regards,

Niek

Niek Linnenbank (2):
  tests/acceptance: replace unstable apt.armbian.com URLs for
    orangepi-pc, cubieboard
  hw/net/allwinner-sun8i-emac: traverse transmit queue using TX_CUR_DESC
    register value

 hw/net/allwinner-sun8i-emac.c          | 58 ++++++++++++++------------
 tests/acceptance/boot_linux_console.py | 49 +++++++++-------------
 tests/acceptance/replay_kernel.py      |  6 +--
 3 files changed, 53 insertions(+), 60 deletions(-)

-- 
2.25.1

[PATCH v2 1/2] tests/acceptance: replace unstable apt.armbian.com URLs for orangepi-pc, cubieboard

[Niek Linnenbank]

Currently the automated acceptance tests for the Orange Pi PC and cubieboard
machines are disabled by default. The tests for both machines require artifacts
that are stored on the apt.armbian.com domain. Unfortunately, some of these artifacts
have been removed from apt.armbian.com and it is uncertain whether more will be removed.

This commit moves the artifacts previously stored on apt.armbian.com to github
and retrieves them using the path: '/<machine>/<artifact>'.

Signed-off-by: Niek Linnenbank <nieklinnenbank@gmail.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 49 ++++++++++----------------
 tests/acceptance/replay_kernel.py      |  6 ++--
 2 files changed, 21 insertions(+), 34 deletions(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index eb01286799..0f8a824c61 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -507,15 +507,13 @@ def test_arm_exynos4210_initrd(self):
         self.wait_for_console_pattern('Boot successful.')
         # TODO user command, for now the uart is stuck
 
-    @skipUnless(os.getenv('ARMBIAN_ARTIFACTS_CACHED'),
-                'Test artifacts fetched from unreliable apt.armbian.com')
     def test_arm_cubieboard_initrd(self):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:cubieboard
         """
-        deb_url = ('https://apt.armbian.com/pool/main/l/'
-                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
+        deb_url = ('https://github.com/nieklinnenbank/QemuArtifacts/raw/'
+                   'master/cubieboard/linux-image-dev-sunxi_5.75_armhf.deb')
         deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
         kernel_path = self.extract_from_deb(deb_path,
@@ -549,15 +547,13 @@ def test_arm_cubieboard_initrd(self):
                                                 'system-control@1c00000')
         # cubieboard's reboot is not functioning; omit reboot test.
 
-    @skipUnless(os.getenv('ARMBIAN_ARTIFACTS_CACHED'),
-                'Test artifacts fetched from unreliable apt.armbian.com')
     def test_arm_cubieboard_sata(self):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:cubieboard
         """
-        deb_url = ('https://apt.armbian.com/pool/main/l/'
-                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
+        deb_url = ('https://github.com/nieklinnenbank/QemuArtifacts/raw/'
+                   'master/cubieboard/linux-image-dev-sunxi_5.75_armhf.deb')
         deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
         kernel_path = self.extract_from_deb(deb_path,
@@ -678,15 +674,13 @@ def test_arm_quanta_gsj_initrd(self):
         self.wait_for_console_pattern(
                 'Give root password for system maintenance')
 
-    @skipUnless(os.getenv('ARMBIAN_ARTIFACTS_CACHED'),
-                'Test artifacts fetched from unreliable apt.armbian.com')
     def test_arm_orangepi(self):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:orangepi-pc
         """
-        deb_url = ('https://apt.armbian.com/pool/main/l/'
-                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
+        deb_url = ('https://github.com/nieklinnenbank/QemuArtifacts/raw/'
+                   'master/orangepi-pc/linux-image-dev-sunxi_5.75_armhf.deb')
         deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
         kernel_path = self.extract_from_deb(deb_path,
@@ -705,15 +699,13 @@ def test_arm_orangepi(self):
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
 
-    @skipUnless(os.getenv('ARMBIAN_ARTIFACTS_CACHED'),
-                'Test artifacts fetched from unreliable apt.armbian.com')
     def test_arm_orangepi_initrd(self):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:orangepi-pc
         """
-        deb_url = ('https://apt.armbian.com/pool/main/l/'
-                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
+        deb_url = ('https://github.com/nieklinnenbank/QemuArtifacts/raw/'
+                   'master/orangepi-pc/linux-image-dev-sunxi_5.75_armhf.deb')
         deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
         kernel_path = self.extract_from_deb(deb_path,
@@ -749,24 +741,23 @@ def test_arm_orangepi_initrd(self):
         # Wait for VM to shut down gracefully
         self.vm.wait()
 
-    @skipUnless(os.getenv('ARMBIAN_ARTIFACTS_CACHED'),
-                'Test artifacts fetched from unreliable apt.armbian.com')
     def test_arm_orangepi_sd(self):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:orangepi-pc
         :avocado: tags=device:sd
         """
-        deb_url = ('https://apt.armbian.com/pool/main/l/'
-                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
+        deb_url = ('https://github.com/nieklinnenbank/QemuArtifacts/raw/'
+                   'master/orangepi-pc/linux-image-dev-sunxi_5.75_armhf.deb')
         deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
         kernel_path = self.extract_from_deb(deb_path,
                                             '/boot/vmlinuz-4.20.7-sunxi')
         dtb_path = '/usr/lib/linux-image-dev-sunxi/sun8i-h3-orangepi-pc.dtb'
         dtb_path = self.extract_from_deb(deb_path, dtb_path)
-        rootfs_url = ('http://storage.kernelci.org/images/rootfs/buildroot/'
-                      'kci-2019.02/armel/base/rootfs.ext2.xz')
+        # Rootfs is based on buildroot 2019.02 from kernelci.org
+        rootfs_url = ('https://github.com/nieklinnenbank/QemuArtifacts/raw/'
+                      'master/orangepi-pc/rootfs.ext2.xz')
         rootfs_hash = '692510cb625efda31640d1de0a8d60e26040f061'
         rootfs_path_xz = self.fetch_asset(rootfs_url, asset_hash=rootfs_hash)
         rootfs_path = os.path.join(self.workdir, 'rootfs.cpio')
@@ -828,8 +819,6 @@ def do_test_arm_orangepi_uboot_armbian(self, image_path):
                                       'to <orangepipc>')
         self.wait_for_console_pattern('Starting Load Kernel Modules...')
 
-    @skipUnless(os.getenv('ARMBIAN_ARTIFACTS_CACHED'),
-                'Test artifacts fetched from unreliable apt.armbian.com')
     @skipUnless(os.getenv('AVOCADO_ALLOW_LARGE_STORAGE'), 'storage limited')
     @skipUnless(P7ZIP_AVAILABLE, '7z not installed')
     def test_arm_orangepi_bionic_19_11(self):
@@ -840,7 +829,8 @@ def test_arm_orangepi_bionic_19_11(self):
         """
 
         # This test download a 196MB compressed image and expand it to 1GB
-        image_url = ('https://dl.armbian.com/orangepipc/archive/'
+        image_url = ('https://github.com/nieklinnenbank/QemuArtifacts/raw/'
+                     'master/orangepi-pc/'
                      'Armbian_19.11.3_Orangepipc_bionic_current_5.3.9.7z')
         image_hash = '196a8ffb72b0123d92cea4a070894813d305c71e'
         image_path_7z = self.fetch_asset(image_url, asset_hash=image_hash)
@@ -851,8 +841,6 @@ def test_arm_orangepi_bionic_19_11(self):
 
         self.do_test_arm_orangepi_uboot_armbian(image_path)
 
-    @skipUnless(os.getenv('ARMBIAN_ARTIFACTS_CACHED'),
-                'Test artifacts fetched from unreliable apt.armbian.com')
     @skipUnless(os.getenv('AVOCADO_ALLOW_LARGE_STORAGE'), 'storage limited')
     def test_arm_orangepi_bionic_20_08(self):
         """
@@ -865,7 +853,8 @@ def test_arm_orangepi_bionic_20_08(self):
         # to 1036 MiB, but the underlying filesystem is 1552 MiB...
         # As we expand it to 2 GiB we are safe.
 
-        image_url = ('https://dl.armbian.com/orangepipc/archive/'
+        image_url = ('https://github.com/nieklinnenbank/QemuArtifacts/raw/'
+                     'master/orangepi-pc/'
                      'Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz')
         image_hash = ('b4d6775f5673486329e45a0586bf06b6'
                       'dbe792199fd182ac6b9c7bb6c7d3e6dd')
@@ -884,8 +873,8 @@ def test_arm_orangepi_uboot_netbsd9(self):
         :avocado: tags=device:sd
         """
         # This test download a 304MB compressed image and expand it to 2GB
-        deb_url = ('http://snapshot.debian.org/archive/debian/'
-                   '20200108T145233Z/pool/main/u/u-boot/'
+        deb_url = ('https://github.com/nieklinnenbank/QemuArtifacts/raw/'
+                   'master/orangepi-pc/'
                    'u-boot-sunxi_2020.01%2Bdfsg-1_armhf.deb')
         deb_hash = 'f67f404a80753ca3d1258f13e38f2b060e13db99'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index c1cb862468..ca5670fa76 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -177,15 +177,13 @@ def test_arm_virt(self):
         self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1)
 
     @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
-    @skipUnless(os.getenv('ARMBIAN_ARTIFACTS_CACHED'),
-                'Test artifacts fetched from unreliable apt.armbian.com')
     def test_arm_cubieboard_initrd(self):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:cubieboard
         """
-        deb_url = ('https://apt.armbian.com/pool/main/l/'
-                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
+        deb_url = ('https://github.com/nieklinnenbank/QemuArtifacts/raw/'
+                   'master/cubieboard/linux-image-dev-sunxi_5.75_armhf.deb')
         deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
         kernel_path = self.extract_from_deb(deb_path,
-- 
2.25.1

[PATCH v2 2/2] hw/net/allwinner-sun8i-emac: traverse transmit queue using TX_CUR_DESC register value

[Niek Linnenbank]

Currently the emulated EMAC for sun8i always traverses the transmit queue
from the head when transferring packets. It searches for a list of consecutive
descriptors whichs are flagged as ready for processing and transmits their payloads
accordingly. The controller stops processing once it finds a descriptor that is not
marked ready.

While the above behaviour works in most situations, it is not the same as the actual
EMAC in hardware. Actual hardware uses the TX_CUR_DESC register value to keep track
of the last position in the transmit queue and continues processing from that position
when software triggers the start of DMA processing. The currently emulated behaviour can
lead to packet loss on transmit when software fills the transmit queue with ready
descriptors that overlap the tail of the circular list.

This commit modifies the emulated EMAC for sun8i such that it processes
the transmit queue using the TX_CUR_DESC register in the same way as hardware.

Signed-off-by: Niek Linnenbank <nieklinnenbank@gmail.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 hw/net/allwinner-sun8i-emac.c | 58 +++++++++++++++++++----------------
 1 file changed, 32 insertions(+), 26 deletions(-)

diff --git a/hw/net/allwinner-sun8i-emac.c b/hw/net/allwinner-sun8i-emac.c
index 042768922c..e586c147e5 100644
--- a/hw/net/allwinner-sun8i-emac.c
+++ b/hw/net/allwinner-sun8i-emac.c
@@ -339,35 +339,40 @@ static void allwinner_sun8i_emac_update_irq(AwSun8iEmacState *s)
     qemu_set_irq(s->irq, (s->int_sta & s->int_en) != 0);
 }
 
-static uint32_t allwinner_sun8i_emac_next_desc(AwSun8iEmacState *s,
-                                               FrameDescriptor *desc,
-                                               size_t min_size)
+static bool allwinner_sun8i_emac_desc_owned(FrameDescriptor *desc,
+                                            size_t min_buf_size)
 {
-    uint32_t paddr = desc->next;
+    return (desc->status & DESC_STATUS_CTL) && (min_buf_size == 0 ||
+           (desc->status2 & DESC_STATUS2_BUF_SIZE_MASK) >= min_buf_size);
+}
 
-    dma_memory_read(&s->dma_as, paddr, desc, sizeof(*desc));
+static void allwinner_sun8i_emac_get_desc(AwSun8iEmacState *s,
+                                          FrameDescriptor *desc,
+                                          uint32_t phys_addr)
+{
+    dma_memory_read(&s->dma_as, phys_addr, desc, sizeof(*desc));
+}
 
-    if ((desc->status & DESC_STATUS_CTL) &&
-        (desc->status2 & DESC_STATUS2_BUF_SIZE_MASK) >= min_size) {
-        return paddr;
-    } else {
-        return 0;
-    }
+static uint32_t allwinner_sun8i_emac_next_desc(AwSun8iEmacState *s,
+                                               FrameDescriptor *desc)
+{
+    const uint32_t nxt = desc->next;
+    allwinner_sun8i_emac_get_desc(s, desc, nxt);
+    return nxt;
 }
 
-static uint32_t allwinner_sun8i_emac_get_desc(AwSun8iEmacState *s,
-                                              FrameDescriptor *desc,
-                                              uint32_t start_addr,
-                                              size_t min_size)
+static uint32_t allwinner_sun8i_emac_find_desc(AwSun8iEmacState *s,
+                                               FrameDescriptor *desc,
+                                               uint32_t start_addr,
+                                               size_t min_size)
 {
     uint32_t desc_addr = start_addr;
 
     /* Note that the list is a cycle. Last entry points back to the head. */
     while (desc_addr != 0) {
-        dma_memory_read(&s->dma_as, desc_addr, desc, sizeof(*desc));
+        allwinner_sun8i_emac_get_desc(s, desc, desc_addr);
 
-        if ((desc->status & DESC_STATUS_CTL) &&
-            (desc->status2 & DESC_STATUS2_BUF_SIZE_MASK) >= min_size) {
+        if (allwinner_sun8i_emac_desc_owned(desc, min_size)) {
             return desc_addr;
         } else if (desc->next == start_addr) {
             break;
@@ -383,14 +388,14 @@ static uint32_t allwinner_sun8i_emac_rx_desc(AwSun8iEmacState *s,
                                              FrameDescriptor *desc,
                                              size_t min_size)
 {
-    return allwinner_sun8i_emac_get_desc(s, desc, s->rx_desc_curr, min_size);
+    return allwinner_sun8i_emac_find_desc(s, desc, s->rx_desc_curr, min_size);
 }
 
 static uint32_t allwinner_sun8i_emac_tx_desc(AwSun8iEmacState *s,
-                                             FrameDescriptor *desc,
-                                             size_t min_size)
+                                             FrameDescriptor *desc)
 {
-    return allwinner_sun8i_emac_get_desc(s, desc, s->tx_desc_head, min_size);
+    allwinner_sun8i_emac_get_desc(s, desc, s->tx_desc_curr);
+    return s->tx_desc_curr;
 }
 
 static void allwinner_sun8i_emac_flush_desc(AwSun8iEmacState *s,
@@ -470,7 +475,8 @@ static ssize_t allwinner_sun8i_emac_receive(NetClientState *nc,
         bytes_left -= desc_bytes;
 
         /* Move to the next descriptor */
-        s->rx_desc_curr = allwinner_sun8i_emac_next_desc(s, &desc, 64);
+        s->rx_desc_curr = allwinner_sun8i_emac_find_desc(s, &desc, desc.next,
+                                                         AW_SUN8I_EMAC_MIN_PKT_SZ);
         if (!s->rx_desc_curr) {
             /* Not enough buffer space available */
             s->int_sta |= INT_STA_RX_BUF_UA;
@@ -495,10 +501,10 @@ static void allwinner_sun8i_emac_transmit(AwSun8iEmacState *s)
     size_t transmitted = 0;
     static uint8_t packet_buf[2048];
 
-    s->tx_desc_curr = allwinner_sun8i_emac_tx_desc(s, &desc, 0);
+    s->tx_desc_curr = allwinner_sun8i_emac_tx_desc(s, &desc);
 
     /* Read all transmit descriptors */
-    while (s->tx_desc_curr != 0) {
+    while (allwinner_sun8i_emac_desc_owned(&desc, 0)) {
 
         /* Read from physical memory into packet buffer */
         bytes = desc.status2 & DESC_STATUS2_BUF_SIZE_MASK;
@@ -524,7 +530,7 @@ static void allwinner_sun8i_emac_transmit(AwSun8iEmacState *s)
             packet_bytes = 0;
             transmitted++;
         }
-        s->tx_desc_curr = allwinner_sun8i_emac_next_desc(s, &desc, 0);
+        s->tx_desc_curr = allwinner_sun8i_emac_next_desc(s, &desc);
     }
 
     /* Raise transmit completed interrupt */
-- 
2.25.1

Re: [PATCH v2 1/2] tests/acceptance: replace unstable apt.armbian.com URLs for orangepi-pc, cubieboard

[Philippe Mathieu-Daudé]

Hi Niek,

On 2/23/21 11:53 PM, Niek Linnenbank wrote:
> Currently the automated acceptance tests for the Orange Pi PC and cubieboard
> machines are disabled by default. The tests for both machines require artifacts
> that are stored on the apt.armbian.com domain. Unfortunately, some of these artifacts
> have been removed from apt.armbian.com and it is uncertain whether more will be removed.
> 
> This commit moves the artifacts previously stored on apt.armbian.com to github
> and retrieves them using the path: '/<machine>/<artifact>'.
> 
> Signed-off-by: Niek Linnenbank <nieklinnenbank@gmail.com>
> Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>

> Tested-by: Cleber Rosa <crosa@redhat.com>

Did Cleber test this new version?

> ---
>  tests/acceptance/boot_linux_console.py | 49 ++++++++++----------------
>  tests/acceptance/replay_kernel.py      |  6 ++--
>  2 files changed, 21 insertions(+), 34 deletions(-)
> 
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index eb01286799..0f8a824c61 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -507,15 +507,13 @@ def test_arm_exynos4210_initrd(self):
>          self.wait_for_console_pattern('Boot successful.')
>          # TODO user command, for now the uart is stuck
>  
> -    @skipUnless(os.getenv('ARMBIAN_ARTIFACTS_CACHED'),
> -                'Test artifacts fetched from unreliable apt.armbian.com')
>      def test_arm_cubieboard_initrd(self):
>          """
>          :avocado: tags=arch:arm
>          :avocado: tags=machine:cubieboard
>          """
> -        deb_url = ('https://apt.armbian.com/pool/main/l/'
> -                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
> +        deb_url = ('https://github.com/nieklinnenbank/QemuArtifacts/raw/'
> +                   'master/cubieboard/linux-image-dev-sunxi_5.75_armhf.deb')

Can you replace 'master/' by 'ed9829b/' please (other occurences).

With it:
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

(using both master / ed9829b):
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

>          deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
>          deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
>          kernel_path = self.extract_from_deb(deb_path,
> @@ -549,15 +547,13 @@ def test_arm_cubieboard_initrd(self):
>                                                  'system-control@1c00000')
>          # cubieboard's reboot is not functioning; omit reboot test.
>  
> -    @skipUnless(os.getenv('ARMBIAN_ARTIFACTS_CACHED'),
> -                'Test artifacts fetched from unreliable apt.armbian.com')
>      def test_arm_cubieboard_sata(self):
>          """
>          :avocado: tags=arch:arm
>          :avocado: tags=machine:cubieboard
>          """
> -        deb_url = ('https://apt.armbian.com/pool/main/l/'
> -                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
> +        deb_url = ('https://github.com/nieklinnenbank/QemuArtifacts/raw/'
> +                   'master/cubieboard/linux-image-dev-sunxi_5.75_armhf.deb')
...

Re: [PATCH v2 1/2] tests/acceptance: replace unstable apt.armbian.com URLs for orangepi-pc, cubieboard

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

On Wed, Feb 24, 2021 at 10:12:10AM +0100, Philippe Mathieu-Daudé wrote:
> Hi Niek,
> 
> On 2/23/21 11:53 PM, Niek Linnenbank wrote:
> > Currently the automated acceptance tests for the Orange Pi PC and cubieboard
> > machines are disabled by default. The tests for both machines require artifacts
> > that are stored on the apt.armbian.com domain. Unfortunately, some of these artifacts
> > have been removed from apt.armbian.com and it is uncertain whether more will be removed.
> > 
> > This commit moves the artifacts previously stored on apt.armbian.com to github
> > and retrieves them using the path: '/<machine>/<artifact>'.
> > 
> > Signed-off-by: Niek Linnenbank <nieklinnenbank@gmail.com>
> > Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> > Reviewed-by: Cleber Rosa <crosa@redhat.com>
> 
> > Tested-by: Cleber Rosa <crosa@redhat.com>
> 
> Did Cleber test this new version?
>

Nope, and I'm having issues with those URLs.  For instance:

   $ curl -L https://github.com/nieklinnenbank/QemuArtifacts/raw/master/cubieboard/linux-image-dev-sunxi_5.75_armhf.deb
   version https://git-lfs.github.com/spec/v1
   oid sha256:a4b765c851de76592f55023b1ff4104f7fd29bf90937e6054e0a64fdda56380b
   size 20331524

Looks like it has to do with GitHub's behavior wrt quota.

Regards,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/2] tests/acceptance: replace unstable apt.armbian.com URLs for orangepi-pc, cubieboard

[Niek Linnenbank]

[-- Attachment #1: Type: text/plain, Size: 3756 bytes --]

Hi Philippe, Cleber,

On Wed, Feb 24, 2021 at 8:14 PM Cleber Rosa <crosa@redhat.com> wrote:

> On Wed, Feb 24, 2021 at 10:12:10AM +0100, Philippe Mathieu-Daudé wrote:
> > Hi Niek,
> >
> > On 2/23/21 11:53 PM, Niek Linnenbank wrote:
> > > Currently the automated acceptance tests for the Orange Pi PC and
> cubieboard
> > > machines are disabled by default. The tests for both machines require
> artifacts
> > > that are stored on the apt.armbian.com domain. Unfortunately, some of
> these artifacts
> > > have been removed from apt.armbian.com and it is uncertain whether
> more will be removed.
> > >
> > > This commit moves the artifacts previously stored on apt.armbian.com
> to github
> > > and retrieves them using the path: '/<machine>/<artifact>'.
> > >
> > > Signed-off-by: Niek Linnenbank <nieklinnenbank@gmail.com>
> > > Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> > > Reviewed-by: Cleber Rosa <crosa@redhat.com>
> >
> > > Tested-by: Cleber Rosa <crosa@redhat.com>
> >
> > Did Cleber test this new version?
> >
>

You're right, it was the previous version (v1) that Cleber tested using my
own machine URL's.

I was actually not sure whether I should or should not have added the
Tested-by/Reviewed-by tags in such scenario.
The content had to be changed due to the outcome of our discussion but also
I thought I don't want to silently drop
the tags since Cleber invested his time into it too.

What should I do here, next time?



>
> Nope, and I'm having issues with those URLs.  For instance:
>
>    $ curl -L
> https://github.com/nieklinnenbank/QemuArtifacts/raw/master/cubieboard/linux-image-dev-sunxi_5.75_armhf.deb
>    version https://git-lfs.github.com/spec/v1
>    oid
> sha256:a4b765c851de76592f55023b1ff4104f7fd29bf90937e6054e0a64fdda56380b
>    size 20331524
>
> Looks like it has to do with GitHub's behavior wrt quota.
>

Indeed. Just this morning I received an e-mail from github with the
following text:

"[GitHub] Git LFS disabled for nieklinnenbank

Git LFS has been disabled on your personal account nieklinnenbank because
you’ve exceeded your data plan by at least 150%.
Please purchase additional data packs to cover your bandwidth and storage
usage:

  https://github.com/account/billing/data/upgrade

Current usage as of 24 Feb 2021 09:49AM UTC:

  Bandwidth: 1.55 GB / 1 GB (155%)
  Storage: 0.48 GB / 1 GB (48%)"

I wasn't aware of it but it appears that Github has these quota's for the
Large File Storage (LFS). I uploaded the files in the git LFS
because single files are also limited to 100MiB each on the regular Git
repositories.

With those strict limits, in my opinion Github isn't really a solution
since the bandwidth limit will be reached very quickly. At least for the
LFS part that is. I don't know yet if there is any limit for regular access.

My current ideas:
  - we can try to splitup the larger files into sizes < 100MiB in order to
use github regular storage. and then download each part to combine into the
final image.
    im not really in favour of this but it can work, if github doesnt have
any other limit/quota. the cost is that we have to add more complexity to
the acceptance test code.
  - we can try to just update the URLs to armbian that are working now
(with the risk of breaking again in the near future). Ive also found this
link, which may be more stable:
     https://archive.armbian.com/orangepipc/archive/
  - or use the server that im hosting - and i don't mind to add the license
files on it if needed (should be GPLv2 right?)

I'd be interested to hear your opinion and suggestions.

Kind regards,
Niek


>
> Regards,
> - Cleber.
>


-- 
Niek Linnenbank

[-- Attachment #2: Type: text/html, Size: 5851 bytes --]

Re: [PATCH v2 1/2] tests/acceptance: replace unstable apt.armbian.com URLs for orangepi-pc, cubieboard

[Philippe Mathieu-Daudé]

+Thomas/Daniel/Alex/Peter/Paolo/Stefan/Markus

On 2/24/21 9:02 PM, Niek Linnenbank wrote:
> Hi Philippe, Cleber,
> 
> On Wed, Feb 24, 2021 at 8:14 PM Cleber Rosa <crosa@redhat.com
> <mailto:crosa@redhat.com>> wrote:
> 
>     On Wed, Feb 24, 2021 at 10:12:10AM +0100, Philippe Mathieu-Daudé wrote:
>     > Hi Niek,
>     >
>     > On 2/23/21 11:53 PM, Niek Linnenbank wrote:
>     > > Currently the automated acceptance tests for the Orange Pi PC
>     and cubieboard
>     > > machines are disabled by default. The tests for both machines
>     require artifacts
>     > > that are stored on the apt.armbian.com <http://apt.armbian.com>
>     domain. Unfortunately, some of these artifacts
>     > > have been removed from apt.armbian.com <http://apt.armbian.com>
>     and it is uncertain whether more will be removed.
>     > >
>     > > This commit moves the artifacts previously stored on
>     apt.armbian.com <http://apt.armbian.com> to github
>     > > and retrieves them using the path: '/<machine>/<artifact>'.
>     > >
>     > > Signed-off-by: Niek Linnenbank <nieklinnenbank@gmail.com
>     <mailto:nieklinnenbank@gmail.com>>
>     > > Reviewed-by: Willian Rampazzo <willianr@redhat.com
>     <mailto:willianr@redhat.com>>
>     > > Reviewed-by: Cleber Rosa <crosa@redhat.com
>     <mailto:crosa@redhat.com>>
...

>     Nope, and I'm having issues with those URLs.  For instance:
> 
>        $ curl -L
>     https://github.com/nieklinnenbank/QemuArtifacts/raw/master/cubieboard/linux-image-dev-sunxi_5.75_armhf.deb
>     <https://github.com/nieklinnenbank/QemuArtifacts/raw/master/cubieboard/linux-image-dev-sunxi_5.75_armhf.deb>
>        version https://git-lfs.github.com/spec/v1
>     <https://git-lfs.github.com/spec/v1>
>        oid
>     sha256:a4b765c851de76592f55023b1ff4104f7fd29bf90937e6054e0a64fdda56380b
>        size 20331524
> 
>     Looks like it has to do with GitHub's behavior wrt quota.
> 
> 
> Indeed. Just this morning I received an e-mail from github with the
> following text:
> 
> "[GitHub] Git LFS disabled for nieklinnenbank
> 
> Git LFS has been disabled on your personal account nieklinnenbank
> because you’ve exceeded your data plan by at least 150%.
> Please purchase additional data packs to cover your bandwidth and
> storage usage:
> 
>   https://github.com/account/billing/data/upgrade
> <https://github.com/account/billing/data/upgrade>
> 
> Current usage as of 24 Feb 2021 09:49AM UTC:
> 
>   Bandwidth: 1.55 GB / 1 GB (155%)
>   Storage: 0.48 GB / 1 GB (48%)"
>  
> I wasn't aware of it but it appears that Github has these quota's for
> the Large File Storage (LFS). I uploaded the files in the git LFS
> because single files are also limited to 100MiB each on the regular Git
> repositories.
> 
> With those strict limits, in my opinion Github isn't really a solution
> since the bandwidth limit will be reached very quickly. At least for the
> LFS part that is. I don't know yet if there is any limit for regular access.
> 
> My current ideas:
>   - we can try to splitup the larger files into sizes < 100MiB in order
> to use github regular storage. and then download each part to combine
> into the final image.
>     im not really in favour of this but it can work, if github doesnt
> have any other limit/quota. the cost is that we have to add more
> complexity to the acceptance test code.
>   - we can try to just update the URLs to armbian that are working now
> (with the risk of breaking again in the near future). Ive also found
> this link, which may be more stable:
>      https://archive.armbian.com/orangepipc/archive/
> <https://archive.armbian.com/orangepipc/archive/>
>   - or use the server that im hosting - and i don't mind to add the
> license files on it if needed (should be GPLv2 right?)
> 
> I'd be interested to hear your opinion and suggestions.
> 
> Kind regards,
> Niek

Some of the unpractical options I can think of...:

- do not contribute tests using binary blob
- do not allow test image >100 MiB
- contribute tests with sha1 of (big) image but say "if you want
  the test image contact me off-list" then when the contributor
  stop responding we remove the test
- have anyone setup its servers with tests source and images,
  without committing anything to the repository. Interested
  maintainers/testers are on their own.
- testing done behind the scene

TBH I'm a bit hopeless.

Regards,

Phil.

Re: [PATCH v2 1/2] tests/acceptance: replace unstable apt.armbian.com URLs for orangepi-pc, cubieboard

[Thomas Huth]

On 25/02/2021 00.10, Philippe Mathieu-Daudé wrote:
> +Thomas/Daniel/Alex/Peter/Paolo/Stefan/Markus
> 
> On 2/24/21 9:02 PM, Niek Linnenbank wrote:
>> Hi Philippe, Cleber,
[...]
>> Indeed. Just this morning I received an e-mail from github with the
>> following text:
>>
>> "[GitHub] Git LFS disabled for nieklinnenbank
>>
>> Git LFS has been disabled on your personal account nieklinnenbank
>> because you’ve exceeded your data plan by at least 150%.
>> Please purchase additional data packs to cover your bandwidth and
>> storage usage:
>>
>>    https://github.com/account/billing/data/upgrade
>>
>> Current usage as of 24 Feb 2021 09:49AM UTC:
>>
>>    Bandwidth: 1.55 GB / 1 GB (155%)
>>    Storage: 0.48 GB / 1 GB (48%)"
>>   
>> I wasn't aware of it but it appears that Github has these quota's for
>> the Large File Storage (LFS). I uploaded the files in the git LFS
>> because single files are also limited to 100MiB each on the regular Git
>> repositories.
>>
>> With those strict limits, in my opinion Github isn't really a solution
>> since the bandwidth limit will be reached very quickly. At least for the
>> LFS part that is. I don't know yet if there is any limit for regular access.
>>
>> My current ideas:
>>    - we can try to splitup the larger files into sizes < 100MiB in order
>> to use github regular storage. and then download each part to combine
>> into the final image.
>>      im not really in favour of this but it can work, if github doesnt
>> have any other limit/quota. the cost is that we have to add more
>> complexity to the acceptance test code.

Well, if you want to go down that road (which I don't really like), you 
could also host the binaries on gitlab instead, where our CI is running, so 
the binaries would be hosted in the same network as the CI.

>>    - we can try to just update the URLs to armbian that are working now
>> (with the risk of breaking again in the near future). Ive also found
>> this link, which may be more stable:
>>       https://archive.armbian.com/orangepipc/archive/

I'd give this a try! If we then later have to discover that the links are 
still not stable, we can still reconsider something else.

>>    - or use the server that im hosting - and i don't mind to add the
>> license files on it if needed (should be GPLv2 right?)

You know that you also have to be ready to provide all the source codes to 
the binaries that you host to adhere the conditions of the GPL? That should 
be doable if you originally downloaded them along with the binaries, but 
otherwise finding the sources to such binaries can be hard...

>> I'd be interested to hear your opinion and suggestions.
>>
>> Kind regards,
>> Niek
> 
> Some of the unpractical options I can think of...:
> 
> - do not contribute tests using binary blob

That would be a huge step backward to the times when we did not have the 
"acceptance" tests yet.

> - do not allow test image >100 MiB

Test images should not be too huge anyway, but I don't think we should 
introduce such an artificial limit.

> - contribute tests with sha1 of (big) image but say "if you want
>    the test image contact me off-list" then when the contributor
>    stop responding we remove the test

That does not really scale. And how do you add such a test to the CI? That'd 
mean that everybody has to contribute gitlab runners? I don't think that's 
feasible either.

> - have anyone setup its servers with tests source and images,
>    without committing anything to the repository. Interested
>    maintainers/testers are on their own.
> - testing done behind the scene

That's maybe an option for these tests where the binaries cannot be found in 
the internet anymore.

> TBH I'm a bit hopeless.

C'mon, most of the "acceptance" test are just working fine (from the 
"download" perspective), it's just some few tests that are troubled. I'd say 
let's give it another try with archive.armbian.com and if that does not 
work, we can still consider to simply remove those troubled tests.

  Thomas

Re: [PATCH v2 1/2] tests/acceptance: replace unstable apt.armbian.com URLs for orangepi-pc, cubieboard

[Daniel P. Berrangé]

On Wed, Feb 24, 2021 at 09:02:51PM +0100, Niek Linnenbank wrote:
> Hi Philippe, Cleber,
> 
> On Wed, Feb 24, 2021 at 8:14 PM Cleber Rosa <crosa@redhat.com> wrote:
> 
> > On Wed, Feb 24, 2021 at 10:12:10AM +0100, Philippe Mathieu-Daudé wrote:
> > > Hi Niek,
> > >
> > > On 2/23/21 11:53 PM, Niek Linnenbank wrote:
> > > > Currently the automated acceptance tests for the Orange Pi PC and
> > cubieboard
> > > > machines are disabled by default. The tests for both machines require
> > artifacts
> > > > that are stored on the apt.armbian.com domain. Unfortunately, some of
> > these artifacts
> > > > have been removed from apt.armbian.com and it is uncertain whether
> > more will be removed.
> > > >
> > > > This commit moves the artifacts previously stored on apt.armbian.com
> > to github
> > > > and retrieves them using the path: '/<machine>/<artifact>'.
> > > >
> > > > Signed-off-by: Niek Linnenbank <nieklinnenbank@gmail.com>
> > > > Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> > > > Reviewed-by: Cleber Rosa <crosa@redhat.com>
> > >
> > > > Tested-by: Cleber Rosa <crosa@redhat.com>
> > >
> > > Did Cleber test this new version?
> > >
> >
> 
> You're right, it was the previous version (v1) that Cleber tested using my
> own machine URL's.
> 
> I was actually not sure whether I should or should not have added the
> Tested-by/Reviewed-by tags in such scenario.
> The content had to be changed due to the outcome of our discussion but also
> I thought I don't want to silently drop
> the tags since Cleber invested his time into it too.
> 
> What should I do here, next time?
> 
> 
> 
> >
> > Nope, and I'm having issues with those URLs.  For instance:
> >
> >    $ curl -L
> > https://github.com/nieklinnenbank/QemuArtifacts/raw/master/cubieboard/linux-image-dev-sunxi_5.75_armhf.deb
> >    version https://git-lfs.github.com/spec/v1
> >    oid
> > sha256:a4b765c851de76592f55023b1ff4104f7fd29bf90937e6054e0a64fdda56380b
> >    size 20331524
> >
> > Looks like it has to do with GitHub's behavior wrt quota.
> >
> 
> Indeed. Just this morning I received an e-mail from github with the
> following text:
> 
> "[GitHub] Git LFS disabled for nieklinnenbank
> 
> Git LFS has been disabled on your personal account nieklinnenbank because
> you’ve exceeded your data plan by at least 150%.
> Please purchase additional data packs to cover your bandwidth and storage
> usage:
> 
>   https://github.com/account/billing/data/upgrade
> 
> Current usage as of 24 Feb 2021 09:49AM UTC:
> 
>   Bandwidth: 1.55 GB / 1 GB (155%)
>   Storage: 0.48 GB / 1 GB (48%)"
> 
> I wasn't aware of it but it appears that Github has these quota's for the
> Large File Storage (LFS). I uploaded the files in the git LFS
> because single files are also limited to 100MiB each on the regular Git
> repositories.
> 
> With those strict limits, in my opinion Github isn't really a solution
> since the bandwidth limit will be reached very quickly. At least for the
> LFS part that is. I don't know yet if there is any limit for regular access.
> 
> My current ideas:

>   - we can try to just update the URLs to armbian that are working now
> (with the risk of breaking again in the near future). Ive also found this
> link, which may be more stable:
>      https://archive.armbian.com/orangepipc/archive/

Just do this, as it is the simplest option that gets things working. We
have already spent far too long talking about the problem instead of
just fixing the URLs.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 1/2] tests/acceptance: replace unstable apt.armbian.com URLs for orangepi-pc, cubieboard

[Niek Linnenbank]

[-- Attachment #1: Type: text/plain, Size: 4312 bytes --]

Hi Philippe, Thomas, Daniel,

Thank you all. Then I'll just prepare a new version of the patch that uses
currently working archive.armbian.com links.

Kind regards,
Niek

On Thu, Feb 25, 2021 at 10:46 AM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Wed, Feb 24, 2021 at 09:02:51PM +0100, Niek Linnenbank wrote:
> > Hi Philippe, Cleber,
> >
> > On Wed, Feb 24, 2021 at 8:14 PM Cleber Rosa <crosa@redhat.com> wrote:
> >
> > > On Wed, Feb 24, 2021 at 10:12:10AM +0100, Philippe Mathieu-Daudé wrote:
> > > > Hi Niek,
> > > >
> > > > On 2/23/21 11:53 PM, Niek Linnenbank wrote:
> > > > > Currently the automated acceptance tests for the Orange Pi PC and
> > > cubieboard
> > > > > machines are disabled by default. The tests for both machines
> require
> > > artifacts
> > > > > that are stored on the apt.armbian.com domain. Unfortunately,
> some of
> > > these artifacts
> > > > > have been removed from apt.armbian.com and it is uncertain whether
> > > more will be removed.
> > > > >
> > > > > This commit moves the artifacts previously stored on
> apt.armbian.com
> > > to github
> > > > > and retrieves them using the path: '/<machine>/<artifact>'.
> > > > >
> > > > > Signed-off-by: Niek Linnenbank <nieklinnenbank@gmail.com>
> > > > > Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> > > > > Reviewed-by: Cleber Rosa <crosa@redhat.com>
> > > >
> > > > > Tested-by: Cleber Rosa <crosa@redhat.com>
> > > >
> > > > Did Cleber test this new version?
> > > >
> > >
> >
> > You're right, it was the previous version (v1) that Cleber tested using
> my
> > own machine URL's.
> >
> > I was actually not sure whether I should or should not have added the
> > Tested-by/Reviewed-by tags in such scenario.
> > The content had to be changed due to the outcome of our discussion but
> also
> > I thought I don't want to silently drop
> > the tags since Cleber invested his time into it too.
> >
> > What should I do here, next time?
> >
> >
> >
> > >
> > > Nope, and I'm having issues with those URLs.  For instance:
> > >
> > >    $ curl -L
> > >
> https://github.com/nieklinnenbank/QemuArtifacts/raw/master/cubieboard/linux-image-dev-sunxi_5.75_armhf.deb
> > >    version https://git-lfs.github.com/spec/v1
> > >    oid
> > > sha256:a4b765c851de76592f55023b1ff4104f7fd29bf90937e6054e0a64fdda56380b
> > >    size 20331524
> > >
> > > Looks like it has to do with GitHub's behavior wrt quota.
> > >
> >
> > Indeed. Just this morning I received an e-mail from github with the
> > following text:
> >
> > "[GitHub] Git LFS disabled for nieklinnenbank
> >
> > Git LFS has been disabled on your personal account nieklinnenbank because
> > you’ve exceeded your data plan by at least 150%.
> > Please purchase additional data packs to cover your bandwidth and storage
> > usage:
> >
> >   https://github.com/account/billing/data/upgrade
> >
> > Current usage as of 24 Feb 2021 09:49AM UTC:
> >
> >   Bandwidth: 1.55 GB / 1 GB (155%)
> >   Storage: 0.48 GB / 1 GB (48%)"
> >
> > I wasn't aware of it but it appears that Github has these quota's for the
> > Large File Storage (LFS). I uploaded the files in the git LFS
> > because single files are also limited to 100MiB each on the regular Git
> > repositories.
> >
> > With those strict limits, in my opinion Github isn't really a solution
> > since the bandwidth limit will be reached very quickly. At least for the
> > LFS part that is. I don't know yet if there is any limit for regular
> access.
> >
> > My current ideas:
>
> >   - we can try to just update the URLs to armbian that are working now
> > (with the risk of breaking again in the near future). Ive also found this
> > link, which may be more stable:
> >      https://archive.armbian.com/orangepipc/archive/
>
> Just do this, as it is the simplest option that gets things working. We
> have already spent far too long talking about the problem instead of
> just fixing the URLs.
>
>
> Regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>

-- 
Niek Linnenbank

[-- Attachment #2: Type: text/html, Size: 6994 bytes --]