[Bug 1914117] [NEW] Short files returned via FTP on Qemu with various architectures and OSes

[Bug 1914117] [NEW] Short files returned via FTP on Qemu with various architectures and OSes

[Chris Pinnock]

Public bug reported:


Qemu 5.2 on Mac OS X Big Sur.

I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
Still getting the same problem,.

On the following architectures: 
arm64, amd64 and sometimes i386 running NetBSD host OS; 
i386 running OpenBSD host OS:

I have seen a consistent problem with FTP returning short files. The
file will be a couple of bytes too short. I do not believe this is a
problem with the OS. Downloading the perl source code from CPAN does not
work properly, nor does downloading bind from isc. I've tried this on
different architectures as above.

(Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
gut feel is there is something not right on the Mac OS version of Qemu
or a bug in 5.2 - obviously in the network layer somewhere. If you have
anything you want me to try, please let me know - happy to help get a
resolution.)

** Affects: qemu
     Importance: Undecided
         Status: New


** Tags: arm

** Tags added: arm

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  New

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Thomas Huth]

Please provide more information: How did you compile QEMU? Which version
did you exactly use? And most important: How do you *run* QEMU? System
emulation? User mode? What kind of FTP are you doing??

** Changed in: qemu
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  Incomplete

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

Re: [Bug 1914117] Short files returned via FTP on Qemu with various architectures and OSes

[Chris Pinnock]

Apologies.


Host OS is Big Sur Mac OS X latest - with Xcode latest. Qemu is 5.2 - tar ball directly from the website.

- Compile Qemu on Mac OS/Big Sur - completely stock build :  install Ninja, mkdir build  && cd build && ../configure && make && make install
- But also the issue is with the binary in home-brew (e.g. brew install Qemu) - both methods get me to the same problem.

* Installed NetBSD/amd64 or i386 or OpenBSD/i386. 
Qemu-image create -f raw image 10G
qmu-system-ARCH -m 256M -hda image -cdrom “netbsd.iso”  -boot d -net user  -net nic

(For i386 & amd64 I tend to add -nographic for the installer)

* Run the image:
Qmu-system-ARCH -m 256M -hda $IMAGE -net user -net nic

Also NetBSD/arm64 has the issue using their image.
qemu-system-aarch64 -M virt -cpu cortex-a53 -smp 4 -m 4g \
      -drive if=none,file=netbsd-disk-arm64.img,id=hd0 -device virtio-blk-device,drive=hd0 \
      -netdev type=user,id=net0 -device virtio-net-device,netdev=net0,mac=00:11:22:33:44:55 \
      -bios QEMU_EFI.fd -nographic

* The issue seems to be downloading large files. 
In the host OS two files that seem to tickle the bug often are:

* ftp -a http://cpan.pair.com/src/5.0/perl-5.32.1.tar.xz
On NetBSD this file seems to be one byte shorter than it should be. On arm64 is was several bytes shorter.

* ftp -a ftp://ftp.isc.org/isc/bind9/9.16.11/bind-9.16.11.tar.xz
Also seems to tickle the bug


I saw this while trying to use pkgsrc on NetBSD. Saw this on Amd64, i386
and arm64. Tried OpenBSD to rule out NetBSD as the problem. OpenBSD/i386
sees the same issue (ftp returns short read and file is a couple of
bytes smaller).

The screenshot is from amd64 - a fresh boot this morning running on a
fairly idle host.

Kind regards
Chris

> On 2 Feb 2021, at 05:24, Thomas Huth <1914117@bugs.launchpad.net> wrote:
> 
> Please provide more information: How did you compile QEMU? Which version
> did you exactly use? And most important: How do you *run* QEMU? System
> emulation? User mode? What kind of FTP are you doing??
> 
> ** Changed in: qemu
>       Status: New => Incomplete
> 
> -- 
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1914117
> 
> Title:
>  Short files returned via FTP on Qemu with various architectures and
>  OSes
> 
> Status in QEMU:
>  Incomplete
> 
> Bug description:
> 
>  Qemu 5.2 on Mac OS X Big Sur.
> 
>  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
>  Still getting the same problem,.
> 
>  On the following architectures: 
>  arm64, amd64 and sometimes i386 running NetBSD host OS; 
>  i386 running OpenBSD host OS:
> 
>  I have seen a consistent problem with FTP returning short files. The
>  file will be a couple of bytes too short. I do not believe this is a
>  problem with the OS. Downloading the perl source code from CPAN does
>  not work properly, nor does downloading bind from isc. I've tried this
>  on different architectures as above.
> 
>  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
>  gut feel is there is something not right on the Mac OS version of Qemu
>  or a bug in 5.2 - obviously in the network layer somewhere. If you
>  have anything you want me to try, please let me know - happy to help
>  get a resolution.)
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions


** Attachment added: "PastedGraphic-1.tiff"
   https://bugs.launchpad.net/bugs/1914117/+attachment/5459003/+files/PastedGraphic-1.tiff

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  Incomplete

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Chris Pinnock]

Apologies.


Host OS is Big Sur Mac OS X latest - with Xcode latest. Qemu is 5.2 - tar ball directly from the website.

- Compile Qemu on Mac OS/Big Sur - completely stock build :  install Ninja, mkdir build  && cd build && ../configure && make && make install
- But also the issue is with the binary in home-brew (e.g. brew install Qemu) - both methods get me to the same problem.

* Installed NetBSD/amd64 or i386 or OpenBSD/i386. 
Qemu-image create -f raw image 10G
qmu-system-ARCH -m 256M -hda image -cdrom “netbsd.iso”  -boot d -net user  -net nic

(For i386 & amd64 I tend to add -nographic for the installer)

* Run the image:
Qmu-system-ARCH -m 256M -hda $IMAGE -net user -net nic

Also NetBSD/arm64 has the issue using their image.
qemu-system-aarch64 -M virt -cpu cortex-a53 -smp 4 -m 4g \
      -drive if=none,file=netbsd-disk-arm64.img,id=hd0 -device virtio-blk-device,drive=hd0 \
      -netdev type=user,id=net0 -device virtio-net-device,netdev=net0,mac=00:11:22:33:44:55 \
      -bios QEMU_EFI.fd -nographic

* The issue seems to be downloading large files. 
In the host OS two files that seem to tickle the bug often are:

* ftp -a http://cpan.pair.com/src/5.0/perl-5.32.1.tar.xz
On NetBSD this file seems to be one byte shorter than it should be. On arm64 is was several bytes shorter.

* ftp -a ftp://ftp.isc.org/isc/bind9/9.16.11/bind-9.16.11.tar.xz
Also seems to tickle the bug

I saw this while trying to use pkgsrc on NetBSD. Saw this on Amd64, i386
and arm64. Tried OpenBSD to rule out NetBSD as the problem. OpenBSD/i386
sees the same issue (ftp returns short read and file is a couple of
bytes smaller).

The screenshot is from amd64 - a fresh boot this morning running on a
fairly idle host.

Kind regards
Chris

** Attachment added: "Screenshot 2021-02-02 at 06.56.22.png"
   https://bugs.launchpad.net/qemu/+bug/1914117/+attachment/5459002/+files/Screenshot%202021-02-02%20at%2006.56.22.png

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  Incomplete

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Chris Pinnock]

Some more info.

This evening I've tried some more things.

Qemu 5.2/Mac OS X Catalina (Qemu from home-brew)

Host OS - OpenBSD/i386
1. Booted with

2. Booted with
qemu-system-i386 -m 256M -hda openbsd-disk-i386.img -netdev user,id=mynet0 -device virtio-net,netdev=mynet0 

With both ftp'ed ftp://ftp.isc.org/isc/bind9/9.16.11/bind-9.16.11.tar.xz
Both were short and did not match the find at ISC.

See attached. SHA1 should be 1bfb5725c85fd9dffe868d8e826a1f8c0de509cc


** Changed in: qemu
       Status: Incomplete => New

** Attachment added: "Screenshot 2021-02-02 at 21.24.05.png"
   https://bugs.launchpad.net/qemu/+bug/1914117/+attachment/5459182/+files/Screenshot%202021-02-02%20at%2021.24.05.png

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  New

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Chris Pinnock]

First boot in previous comment was with:
qemu-system-i386 -m 256M -hda openbsd-disk-i386.img -net user -net nic

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  New

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Chris Pinnock]

I've spent some more time on this.
I've tcpdump'ed the connection whilst doing the download (via both HTTP & FTP).

In the last data packet, the last byte that is missing on the filesystem
is in the packet, but the packet has the urgent bit set with the urgent
pointer the same as the length of the packet.

I'm not sure but this might cause the client app to discard part of the packet?
Unclear.

Also I've build Qemu 4.2.1 on MacOS X/Big Sur - I'm seeing the same issue on FreeBSD/amd64.
This bug might be related:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237441


** Bug watch added: bugs.freebsd.org/bugzilla/ #237441
   https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237441

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  New

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Chris Pinnock]

The more I look at this, the more I think it may be a macOS bug
underneath.

I've tested OpenBSD as a guest on a Debian AWS instance running 4.2.1 - all is fine.
I've tested OpenBSD as a guest on a FreeBSD AWS instance running whatever is in ports and all is fine.

Also others are having trouble:
https://twitter.com/astr0baby/status/1354952352713887754
Mac OS on M1 silicon with Free and NetBSD as guest OS.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  New

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Chris Pinnock]

This is NOT a fix but we can get working FTPs again with this patch -
narrowing into where the problem is. Looks like the behaviour of this
code is different on macOS to other OSes.

--- slirp.c.orig	2021-02-08 21:05:20.000000000 +0000
+++ slirp.c	2021-02-10 11:00:00.000000000 +0000
@@ -621,18 +621,7 @@
              * This will soread as well, so no need to
              * test for SLIRP_POLL_IN below if this succeeds
              */
-            if (revents & SLIRP_POLL_PRI) {
-               ret = sorecvoob(so);
-               if (ret < 0) {
-                   /* Socket error might have resulted in the socket being
-                     * removed, do not try to do anything more with it. */
-                    continue;
-                }
-            }
-            /*
-             * Check sockets for reading
-             */
-            else if (revents & 
+            if (revents & 
                      (SLIRP_POLL_IN | SLIRP_POLL_HUP | SLIRP_POLL_ERR)) {
                 /*
                  * Check for incoming connections

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  New

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Chris Pinnock]

ok - one of my friends has written a test program. we will provide a
writeup tomorrow, but basically towards the end of a stream both HUP &
PRI are getting set on a poll call (on Mac) which means the code above
would be invoked - on other platforms these aren't see. Better
explanation & more details to follow tomorrow.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  New

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Chris Pinnock]

Writeup as promised.

Symptom: 
--------
Qemu on Mac OS X - both Catalina and Big Sur.
The issue occurs in both 5.2 and 4.2* branches of Qemu.

Applications such as ftp that read large amounts of data from the network 
may ignore valid data due to the Urgent flag being set on packets in the 
stream.

- Install a Unix VM (e.g. NetBSD, OpenBSD, etc) on Qemu using Mac OS X.
- Try to FTP a large file, such as 
		ftp://ftp.isc.org/isc/bind9/9.16.11/bind-9.16.11.tar.xz
  and you will be one byte short (not just this file, it's just an ex).

Synopsis: 
---------
- On inspection, the urgent flag is being set on the last packet of data
- As a result data is missing and is not received by the client app
  because it is considered out of band.
- poll() on Mac OS X has different behaviour to other Unices.
- towards the end of a stream, PRI and HUP are sent (whereas on FreeBSD
  and others it is not)
- as a result of PRI, the slirp library used in Qemu for the user 
  network interface adds an urgent bit to the relevant  packets

To see the different behaviour, we setup a server to serve a large file
and wrote a client to receive it, using poll() and dumping information about the flags.

Here is FreeBSD - the IN flag is set throughout.

ec2-user@freebsd:~/polltest $ ./a.out -w -P lXXX.net
Resolving lXXX.net: trying XXX.XXX.XXX.XXX... OK
FD 3 ready: POLLIN
Read 1024 byte(s)
FD 3 ready: POLLIN
Read 1024 total byte(s)
[snipped]

FD 3 ready: POLLIN
Read 102400 total byte(s)
ec2-user@freebsd:~/polltest $

Here is Mac OS X (Big Sur). You can see at the end of the stream,
both PRI & HUP are set.

Resolving lXXX.net: trying XXX.XXX.XXX.XXX .. OK
FD 5 ready: POLLIN 
Read 1024 byte(s)
[Snipped]

FD 5 ready: POLLIN 
Read 416 byte(s)
FD 5 ready: POLLIN POLLPRI POLLHUP 
Hangup on FD 5
Read 160 byte(s)
FD 5 ready: POLLIN POLLPRI POLLHUP 
Hangup on FD 5
Read 102400 total byte(s)

Towards a fix:
--------------
The following patch removes the symptom simply by ignoring these flags.
This is not necessarily the final answer, but we have run with this patch
for a couple of days and haven't seen any negative behaviour.

diff -ru qemu-5.2.0/slirp/src/slirp.c qemu-5.2.0-wrk/slirp/src/slirp.c
--- qemu-5.2.0/slirp/src/slirp.c	2021-02-10 11:02:07.000000000 +0000
+++ qemu-5.2.0-wrk/slirp/src/slirp.c	2021-02-10 13:07:17.000000000 +0000
@@ -23,7 +23,7 @@
  * THE SOFTWARE.
  */
 #include "slirp.h"
-
+#define IGNOREPOLLPRI
 
 #ifndef _WIN32
 #include <net/if.h>
@@ -621,6 +621,8 @@
              * This will soread as well, so no need to
              * test for SLIRP_POLL_IN below if this succeeds
              */
+
+#ifndef IGNOREPOLLPRI
             if (revents & SLIRP_POLL_PRI) {
                ret = sorecvoob(so);
                if (ret < 0) {
@@ -633,6 +635,9 @@
              * Check sockets for reading
              */
             else if (revents & 
+#else
+            if (revents & 
+#endif
                      (SLIRP_POLL_IN | SLIRP_POLL_HUP | SLIRP_POLL_ERR)) {
                 /*
                  * Check for incoming connections


Adam Chappell figured most of this out (because a. he is (mostly) cleverer than me, b. he didn't sell his copy of Stevens UNIX Network Programming like I did in the 00s).

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  New

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Thomas Huth]

Maybe related:
https://bugs.launchpad.net/qemu/+bug/1916344
(and https://gitlab.freedesktop.org/slirp/libslirp/-/issues/35 )

** Bug watch added: gitlab.freedesktop.org/slirp/libslirp/-/issues #35
   https://gitlab.freedesktop.org/slirp/libslirp/-/issues/35

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  New

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Chris Pinnock]

libslirp now has a workaround for this in slirp.c.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  New

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Thomas Huth]

Could we close this ticket now if there is a workaround in libslirp now?

** Changed in: qemu
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  Incomplete

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

Re: [Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Chris Pinnock]

If it’s included in qemu when one downloads the sources I’m happy.

Sent from my iPhone

> On 15 May 2021, at 11:55, Thomas Huth <1914117@bugs.launchpad.net> wrote:
> 
> Could we close this ticket now if there is a workaround in libslirp now?
> 
> ** Changed in: qemu
>       Status: New => Incomplete
> 
> -- 
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1914117
> 
> Title:
>  Short files returned via FTP on Qemu with various architectures and
>  OSes
> 
> Status in QEMU:
>  Incomplete
> 
> Bug description:
> 
>  Qemu 5.2 on Mac OS X Big Sur.
> 
>  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
>  Still getting the same problem,.
> 
>  On the following architectures: 
>  arm64, amd64 and sometimes i386 running NetBSD host OS; 
>  i386 running OpenBSD host OS:
> 
>  I have seen a consistent problem with FTP returning short files. The
>  file will be a couple of bytes too short. I do not believe this is a
>  problem with the OS. Downloading the perl source code from CPAN does
>  not work properly, nor does downloading bind from isc. I've tried this
>  on different architectures as above.
> 
>  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
>  gut feel is there is something not right on the Mac OS version of Qemu
>  or a bug in 5.2 - obviously in the network layer somewhere. If you
>  have anything you want me to try, please let me know - happy to help
>  get a resolution.)
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  Incomplete

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Thomas Huth]

** Changed in: qemu
       Status: Incomplete => In Progress

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  In Progress

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Thomas Huth]

slirp has been updated for QEMU 6.1-rc2, so this should be fixed in the
latest 6.1 release candidate. If you've got some spare minutes, could
you please check whether it's working for you now in 6.1-rc4 ?

** Changed in: qemu
       Status: In Progress => Fix Committed

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  Fix Committed

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

[Bug 1914117] Re: Short files returned via FTP on Qemu with various architectures and OSes

[Thomas Huth]

** Changed in: qemu
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  Fix Released

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions

Re: [Bug 1914117] Short files returned via FTP on Qemu with various architectures and OSes

[Chris Pinnock]

I tested Qemu 6.1 (MacOS using brew to install) with guest OS NetBSD/i386. The bind distribution file downloaded fine by FTP.
Libslurp has a workaround for MacOS and it looks like its gone in.
I think this one can be closed.
Sorry for the delay
Kind regards
Chris


> On 25 Aug 2021, at 08:18, Thomas Huth <1914117@bugs.launchpad.net> wrote:
> 
> ** Changed in: qemu
>       Status: Fix Committed => Fix Released
> 
> -- 
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1914117
> 
> Title:
>  Short files returned via FTP on Qemu with various architectures and
>  OSes
> 
> Status in QEMU:
>  Fix Released
> 
> Bug description:
> 
>  Qemu 5.2 on Mac OS X Big Sur.
> 
>  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
>  Still getting the same problem,.
> 
>  On the following architectures: 
>  arm64, amd64 and sometimes i386 running NetBSD host OS; 
>  i386 running OpenBSD host OS:
> 
>  I have seen a consistent problem with FTP returning short files. The
>  file will be a couple of bytes too short. I do not believe this is a
>  problem with the OS. Downloading the perl source code from CPAN does
>  not work properly, nor does downloading bind from isc. I've tried this
>  on different architectures as above.
> 
>  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
>  gut feel is there is something not right on the Mac OS version of Qemu
>  or a bug in 5.2 - obviously in the network layer somewhere. If you
>  have anything you want me to try, please let me know - happy to help
>  get a resolution.)
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions
>

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1914117

Title:
  Short files returned via FTP on Qemu with various architectures and
  OSes

Status in QEMU:
  Fix Released

Bug description:
  
  Qemu 5.2 on Mac OS X Big Sur.

  I originally thought that it might be caused by the home-brew version of Qemu, but this evening I have removed the brew edition and compiled from scratch (using Ninja & Xcode compiler). 
  Still getting the same problem,.

  On the following architectures: 
  arm64, amd64 and sometimes i386 running NetBSD host OS; 
  i386 running OpenBSD host OS:

  I have seen a consistent problem with FTP returning short files. The
  file will be a couple of bytes too short. I do not believe this is a
  problem with the OS. Downloading the perl source code from CPAN does
  not work properly, nor does downloading bind from isc. I've tried this
  on different architectures as above.

  (Qemu 4.2 on Ubuntu/x86_64 with NetBSD/i386 seems to function fine. My
  gut feel is there is something not right on the Mac OS version of Qemu
  or a bug in 5.2 - obviously in the network layer somewhere. If you
  have anything you want me to try, please let me know - happy to help
  get a resolution.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1914117/+subscriptions