regressions.lists.linux.dev archive mirror
 help / color / mirror / Atom feed
* 9p: MPTCP tests regressions due to new 9p features in v6.4
@ 2023-06-06 14:30 Matthieu Baerts
       [not found] ` <CAFkjPTnX0=-GK8sFzd4S+V2+cA8E-FAqYHNndZui2Sh_MvoHPw@mail.gmail.com>
  0 siblings, 1 reply; 10+ messages in thread
From: Matthieu Baerts @ 2023-06-06 14:30 UTC (permalink / raw)
  To: Eric Van Hensbergen; +Cc: v9fs, MPTCP Upstream, regressions

Hi Eric and other 9p devs,

TL;DR: it looks like there is a (small?) problem with the new 9p
features you recently sent and it causes MPTCP tests to be unstable. It
is tracked there: https://github.com/multipath-tcp/mptcp_net-next/issues/400


First, thank you very much for maintaining this very useful FS!

For the MPTCP subsystem, we are running various tests. Many are ran by a
public CI [1] in a VM by using QEmu with 9p thanks to Virtme [2].
Virtme, its dependences and a script to compile the kernel and run the
tests are "packaged" in a Docker container which eases the deployment
and the reproduction of issues using the same environment [3].


Since v6.4-rc1, we noticed that our public CI was reporting various
instabilities, mainly with Packetdrill tests. Packetdrill [4] allows us
to easily call some system calls, craft and "inject" some network
packets and verify that the kernel generates the expected ones. It was
difficult to reproduce the issue but after investigations, it looks like
since v6.4-rc1, Packetdrill is slower to craft and inject packets,
causing the kernel to retransmit packets (retransmission timeout) that
are not expected and tests are then marked as failed. For more details,
a ticket [5] has been opened in our public bugs tracker.

I ran a 'git bisect' and found out that it seems to be caused by the new
9p features for 6.4:

  8e15605be8ba ("Merge tag '9p-6.4-for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/ericvh/v9fs")


If I revert these commits on top of our "export" branch, I can no longer
reproduce the bug:

  $ git log --oneline --reverse v6.3-rc1..21e26d5e54ab
  d9bc0d11e33b fs/9p: Consolidate file operations and add readahead and
writeback
  740b8bf87322 fs/9p: Remove unnecessary superblock flags
  8142db4f2792 fs/9p: allow disable of xattr support on mount
  46c30cb8f539 9p: Add additional debug flags and open modes
  6deffc8924b5 fs/9p: Add new mount modes
  1543b4c5071c fs/9p: remove writeback fid and fix per-file modes
  4eb3117888a9 fs/9p: Rework cache modes and add new options to
Documentation
  21e26d5e54ab fs/9p: Fix bit operation logic error

'git bisect' seems to suggest the issue is due to the first commit
d9bc0d11e33b ("fs/9p: Consolidate file operations and add readahead and
writeback").

It is not clear why these modifications are causing such issues because
the Packetdrill tests are not doing intensive read and no write on the
disk (only stdout). Not so much info has to be read from the disk: one
small Python script (255 lines [6]) launches multiple threads, each
executing the same bash script (22 lines [7]) launching the same
"packetdrill" binary (1.1 MB) in a newly created and dedicated netns.
Then each test reads a different test script of ~30 lines, e.g. [8]. I
think most of the time the problem is seen at the end of a test. From
what I see [9], Packetdrill tries to read the whole test script. This is
confirmed by quickly looking at the output of strace available on the
ticket [5].


It is unclear to me what else I can do and share with you to help fixing
this "regression". I can quite easily reproduce the issue on my side and
provide more info if needed.

Just in case you wish to reproduce it on your side using our docker
container [3], it is easy:

  $ cd [kernel source code]
  $ cat <<'EOF' > .virtme-exec-run
for z in $(seq 15); do
  run_packetdrill_one mptcp/dss || true
  grep -q "[l]ive packet payload:" "${OUTPUT_VIRTME}" && break
  rm "${RESULTS_DIR}"/*.tap; done
EOF
  $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-debug

This will compile the kernel in ".virtme" dir with the required .config.
The VM will start and run a category of packetdrill tests 15 times. On
my side, I was able to reproduce the issue with a busy machine and
packetdrill had to run ~5 times. To stress my machine, I ran stress-ng
in parallel (outside the VM) when the tests were being executed (after
the compilation not to slow down everything), e.g. with

  nproc2=$(nproc); nproc2=$((nproc2 * 2))
  stress-ng --cpu "${nproc2}" --io "${nproc2}" --vm "${nproc2}" \
            --vm-bytes 1G --timeout 60m


I hope you don't mind if I cc the regression ML: this is probably not an
important regression but I would not have guessed the issues we had when
running network tests were due to 9p, this report might help others with
similar issues :)

#regzbot introduced: d9bc0d11e33b


Cheers,
Matt

[1] https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export-net
[2] https://github.com/amluto/virtme
[3] https://github.com/multipath-tcp/mptcp-upstream-virtme-docker
[4] https://github.com/google/packetdrill/
[5] https://github.com/multipath-tcp/mptcp_net-next/issues/400
[6]
https://github.com/multipath-tcp/packetdrill/blob/mptcp-net-next/gtests/net/packetdrill/run_all.py
[7]
https://github.com/multipath-tcp/packetdrill/blob/mptcp-net-next/gtests/net/packetdrill/in_netns.sh
[8]
https://github.com/multipath-tcp/packetdrill/blob/mptcp-net-next/gtests/net/mptcp/dss/mpc_with_data_client.pkt
[9]
https://github.com/google/packetdrill/blob/master/gtests/net/packetdrill/parser.y#L168
-- 
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: 9p: MPTCP tests regressions due to new 9p features in v6.4
       [not found] ` <CAFkjPTnX0=-GK8sFzd4S+V2+cA8E-FAqYHNndZui2Sh_MvoHPw@mail.gmail.com>
@ 2023-06-06 15:08   ` Matthieu Baerts
  2023-06-06 18:47     ` evanhensbergen
  0 siblings, 1 reply; 10+ messages in thread
From: Matthieu Baerts @ 2023-06-06 15:08 UTC (permalink / raw)
  To: Eric Van Hensbergen; +Cc: MPTCP Upstream, regressions, v9fs

Hi Eric,

On 06/06/2023 16:49, Eric Van Hensbergen wrote:
> thanks for the detailed report - I’ll have a look.

Thank you for the quick reply!

No hurry, we can also revert these patches in our tree for the time being.

> Do you know off hand
> what cache options for 9p are in use?

Sorry, I'm not sure what to look at.

In the VM, I see that this mount command is used:

  /bin/mount -n -t 9p -o ro,version=9p2000.L,trans=virtio,access=any \
             virtme.guesttools /run/virtme/guesttools

In virtme source, I can see these commands should be executed:

  /bin/mount -n -t 9p -o ro,version=9p2000.L,trans=virtio,access=any \
             /dev/root /newroot/
  mount -t 9p -o version=9p2000.L,trans=virtio,access=any \
        "virtme.initmount${tag:16}" "${!tag}"

From the VM, I can see:

> # mount | grep 9p
> /dev/root on / type 9p (ro,relatime,access=any,trans=virtio)
> virtme.guesttools on /run/virtme/guesttools type 9p (ro,relatime,access=any,trans=virtio)
> virtme.initmount0 on /mptcp_net-next type 9p (rw,relatime,access=any,trans=virtio)

And the VM is started using:

> /usr/bin/qemu-system-x86_64 -fsdev local,id=virtfs1,path=/,security_model=none,readonly=on,multidevs=remap -device virtio-9p-pci,fsdev=virtfs1,mount_tag=/dev/root -fsdev local,id=virtfs5,path=/opt/virtme/virtme/guest,security_model=none,readonly=on,multidevs=remap -device virtio-9p-pci,fsdev=virtfs5,mount_tag=virtme.guesttools -fsdev local,id=virtfs9,path=.,security_model=none,multidevs=remap -device virtio-9p-pci,fsdev=virtfs9,mount_tag=virtme.initmount0 -machine accel=kvm:tcg -watchdog i6300esb -cpu host -parallel none -net none -echr 1 -serial none -chardev stdio,id=console,signal=off,mux=on -serial chardev:console -mon chardev=console -vga none -display none -m 2048M -smp 2 -device virtio-net-pci,netdev=n0 -netdev user,id=n0 -kernel .virtme/build/arch/x86/boot/bzImage -append 'virtme_link_mods=.virtme/build/.virtme_mods/lib/modules/0.0.0 virtme_initmount0=/mptcp_net-next earlyprintk=serial,ttyS0,115200 console=ttyS0 psmouse.proto=exps "virtme_stty_con=rows 59 cols 132 iutf8" TERM=xterm virtme.dhcp virtme_chdir=/mptcp_net-next rootfstype=9p rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect ro mitigations=off init=/bin/sh -- -c "mount -t tmpfs run /run;mkdir -p /run/virtme/guesttools;/bin/mount -n -t 9p -o ro,version=9p2000.L,trans=virtio,access=any virtme.guesttools /run/virtme/guesttools;exec /run/virtme/guesttools/virtme-init"'

I don't see anything about the cache and nothing in the dmesg. If you
think this info is in the dmesg, the logs are publicly available there:

https://cirrus-ci.com/task/5569063884161024?logs=test#L5626


> Since things are slower, my first
> suspect is the don't cache qid.version == 0 change.  That can be
> overridden with a mount option but perhaps we should invert the default
> as it is new behavior from a linux perspective.

Can I easily change the default behaviour on the kernel side to try
(without modifying the tools launching the mount commands)?

Cheers,
Matt
-- 
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: 9p: MPTCP tests regressions due to new 9p features in v6.4
  2023-06-06 15:08   ` Matthieu Baerts
@ 2023-06-06 18:47     ` evanhensbergen
  2023-06-13 14:56       ` Linux regression tracking (Thorsten Leemhuis)
  0 siblings, 1 reply; 10+ messages in thread
From: evanhensbergen @ 2023-06-06 18:47 UTC (permalink / raw)
  To: Matthieu Baerts; +Cc: Eric Van Hensbergen, MPTCP Upstream, regressions, v9fs

Interesting - so looks like caches aren’t being used so curious that there is any impact at all from mostly cache-related patches!
Okay thanks, I’ll dig in and see what I can find.

      -Eric


> On Jun 6, 2023, at 10:08 AM, Matthieu Baerts <matthieu.baerts@tessares.net> wrote:
> 
> Hi Eric,
> 
> On 06/06/2023 16:49, Eric Van Hensbergen wrote:
>> thanks for the detailed report - I’ll have a look.
> 
> Thank you for the quick reply!
> 
> No hurry, we can also revert these patches in our tree for the time being.
> 
>> Do you know off hand
>> what cache options for 9p are in use?
> 
> Sorry, I'm not sure what to look at.
> 
> In the VM, I see that this mount command is used:
> 
>  /bin/mount -n -t 9p -o ro,version=9p2000.L,trans=virtio,access=any \
>             virtme.guesttools /run/virtme/guesttools
> 
> In virtme source, I can see these commands should be executed:
> 
>  /bin/mount -n -t 9p -o ro,version=9p2000.L,trans=virtio,access=any \
>             /dev/root /newroot/
>  mount -t 9p -o version=9p2000.L,trans=virtio,access=any \
>        "virtme.initmount${tag:16}" "${!tag}"
> 
> From the VM, I can see:
> 
>> # mount | grep 9p
>> /dev/root on / type 9p (ro,relatime,access=any,trans=virtio)
>> virtme.guesttools on /run/virtme/guesttools type 9p (ro,relatime,access=any,trans=virtio)
>> virtme.initmount0 on /mptcp_net-next type 9p (rw,relatime,access=any,trans=virtio)
> 
> And the VM is started using:
> 
>> /usr/bin/qemu-system-x86_64 -fsdev local,id=virtfs1,path=/,security_model=none,readonly=on,multidevs=remap -device virtio-9p-pci,fsdev=virtfs1,mount_tag=/dev/root -fsdev local,id=virtfs5,path=/opt/virtme/virtme/guest,security_model=none,readonly=on,multidevs=remap -device virtio-9p-pci,fsdev=virtfs5,mount_tag=virtme.guesttools -fsdev local,id=virtfs9,path=.,security_model=none,multidevs=remap -device virtio-9p-pci,fsdev=virtfs9,mount_tag=virtme.initmount0 -machine accel=kvm:tcg -watchdog i6300esb -cpu host -parallel none -net none -echr 1 -serial none -chardev stdio,id=console,signal=off,mux=on -serial chardev:console -mon chardev=console -vga none -display none -m 2048M -smp 2 -device virtio-net-pci,netdev=n0 -netdev user,id=n0 -kernel .virtme/build/arch/x86/boot/bzImage -append 'virtme_link_mods=.virtme/build/.virtme_mods/lib/modules/0.0.0 virtme_initmount0=/mptcp_net-next earlyprintk=serial,ttyS0,115200 console=ttyS0 psmouse.proto=exps "virtme_stty_con=rows 59 cols 132 iutf8" TERM=xterm virtme.dhcp virtme_chdir=/mptcp_net-next rootfstype=9p rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect ro mitigations=off init=/bin/sh -- -c "mount -t tmpfs run /run;mkdir -p /run/virtme/guesttools;/bin/mount -n -t 9p -o ro,version=9p2000.L,trans=virtio,access=any virtme.guesttools /run/virtme/guesttools;exec /run/virtme/guesttools/virtme-init"'
> 
> I don't see anything about the cache and nothing in the dmesg. If you
> think this info is in the dmesg, the logs are publicly available there:
> 
> https://cirrus-ci.com/task/5569063884161024?logs=test#L5626
> 
> 
>> Since things are slower, my first
>> suspect is the don't cache qid.version == 0 change.  That can be
>> overridden with a mount option but perhaps we should invert the default
>> as it is new behavior from a linux perspective.
> 
> Can I easily change the default behaviour on the kernel side to try
> (without modifying the tools launching the mount commands)?
> 
> Cheers,
> Matt
> -- 
> Tessares | Belgium | Hybrid Access Solutions
> www.tessares.net
> 


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: 9p: MPTCP tests regressions due to new 9p features in v6.4
  2023-06-06 18:47     ` evanhensbergen
@ 2023-06-13 14:56       ` Linux regression tracking (Thorsten Leemhuis)
  2023-06-13 16:07         ` Eric Van Hensbergen
  0 siblings, 1 reply; 10+ messages in thread
From: Linux regression tracking (Thorsten Leemhuis) @ 2023-06-13 14:56 UTC (permalink / raw)
  To: evanhensbergen, Matthieu Baerts
  Cc: Eric Van Hensbergen, MPTCP Upstream, regressions, v9fs

On 06.06.23 20:47, evanhensbergen@icloud.com wrote:
> Interesting - so looks like caches aren’t being used so curious that there is any impact at all from mostly cache-related patches!
> Okay thanks, I’ll dig in and see what I can find.

Gentle reminder that this apparently made no progress for a week now. Or
was there progress and I just missed it? Then apologies in advance.

I'm asking, as it afaics would be nice to have this resolved in mainline
before the next -rc. That would be ideal to avoid last minute changes
after the last rc.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

#regzbot poke

>> On Jun 6, 2023, at 10:08 AM, Matthieu Baerts <matthieu.baerts@tessares.net> wrote:
>>
>> Hi Eric,
>>
>> On 06/06/2023 16:49, Eric Van Hensbergen wrote:
>>> thanks for the detailed report - I’ll have a look.
>>
>> Thank you for the quick reply!
>>
>> No hurry, we can also revert these patches in our tree for the time being.
>>
>>> Do you know off hand
>>> what cache options for 9p are in use?
>>
>> Sorry, I'm not sure what to look at.
>>
>> In the VM, I see that this mount command is used:
>>
>>  /bin/mount -n -t 9p -o ro,version=9p2000.L,trans=virtio,access=any \
>>             virtme.guesttools /run/virtme/guesttools
>>
>> In virtme source, I can see these commands should be executed:
>>
>>  /bin/mount -n -t 9p -o ro,version=9p2000.L,trans=virtio,access=any \
>>             /dev/root /newroot/
>>  mount -t 9p -o version=9p2000.L,trans=virtio,access=any \
>>        "virtme.initmount${tag:16}" "${!tag}"
>>
>> From the VM, I can see:
>>
>>> # mount | grep 9p
>>> /dev/root on / type 9p (ro,relatime,access=any,trans=virtio)
>>> virtme.guesttools on /run/virtme/guesttools type 9p (ro,relatime,access=any,trans=virtio)
>>> virtme.initmount0 on /mptcp_net-next type 9p (rw,relatime,access=any,trans=virtio)
>>
>> And the VM is started using:
>>
>>> /usr/bin/qemu-system-x86_64 -fsdev local,id=virtfs1,path=/,security_model=none,readonly=on,multidevs=remap -device virtio-9p-pci,fsdev=virtfs1,mount_tag=/dev/root -fsdev local,id=virtfs5,path=/opt/virtme/virtme/guest,security_model=none,readonly=on,multidevs=remap -device virtio-9p-pci,fsdev=virtfs5,mount_tag=virtme.guesttools -fsdev local,id=virtfs9,path=.,security_model=none,multidevs=remap -device virtio-9p-pci,fsdev=virtfs9,mount_tag=virtme.initmount0 -machine accel=kvm:tcg -watchdog i6300esb -cpu host -parallel none -net none -echr 1 -serial none -chardev stdio,id=console,signal=off,mux=on -serial chardev:console -mon chardev=console -vga none -display none -m 2048M -smp 2 -device virtio-net-pci,netdev=n0 -netdev user,id=n0 -kernel .virtme/build/arch/x86/boot/bzImage -append 'virtme_link_mods=.virtme/build/.virtme_mods/lib/modules/0.0.0 virtme_initmount0=/mptcp_net-next earlyprintk=serial,ttyS0,115200 console=ttyS0 psmouse.proto=exps "virtme_stty_con=rows 59 cols 132 iutf8" TERM=xterm virtme.dhcp virtme_chdir=/mptcp_net-next rootfstype=9p rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect ro mitigations=off init=/bin/sh -- -c "mount -t tmpfs run /run;mkdir -p /run/virtme/guesttools;/bin/mount -n -t 9p -o ro,version=9p2000.L,trans=virtio,access=any virtme.guesttools /run/virtme/guesttools;exec /run/virtme/guesttools/virtme-init"'
>>
>> I don't see anything about the cache and nothing in the dmesg. If you
>> think this info is in the dmesg, the logs are publicly available there:
>>
>> https://cirrus-ci.com/task/5569063884161024?logs=test#L5626
>>
>>
>>> Since things are slower, my first
>>> suspect is the don't cache qid.version == 0 change.  That can be
>>> overridden with a mount option but perhaps we should invert the default
>>> as it is new behavior from a linux perspective.
>>
>> Can I easily change the default behaviour on the kernel side to try
>> (without modifying the tools launching the mount commands)?
>>
>> Cheers,
>> Matt
>> -- 
>> Tessares | Belgium | Hybrid Access Solutions
>> www.tessares.net
>>
> 
> 
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: 9p: MPTCP tests regressions due to new 9p features in v6.4
  2023-06-13 14:56       ` Linux regression tracking (Thorsten Leemhuis)
@ 2023-06-13 16:07         ` Eric Van Hensbergen
  2023-06-13 16:27           ` Matthieu Baerts
  2023-08-07 15:42           ` Matthieu Baerts
  0 siblings, 2 replies; 10+ messages in thread
From: Eric Van Hensbergen @ 2023-06-13 16:07 UTC (permalink / raw)
  To: Linux regressions mailing list
  Cc: evanhensbergen, Matthieu Baerts, MPTCP Upstream, v9fs

Understood, this on top of my 9p stack, but day job has been getting
in the way lately so I need to unstack there first.  Will do my best
to reproduce and understand as quickly as I can -- but given its a
performance regression it is likely going to be tricky to reproduce
and understand, so I'm not sure how confident I am that we'll be able
to close it out prior to the final -rc.

     -eric


     -eric

On Tue, Jun 13, 2023 at 9:57 AM Linux regression tracking (Thorsten
Leemhuis) <regressions@leemhuis.info> wrote:
>
> On 06.06.23 20:47, evanhensbergen@icloud.com wrote:
> > Interesting - so looks like caches aren’t being used so curious that there is any impact at all from mostly cache-related patches!
> > Okay thanks, I’ll dig in and see what I can find.
>
> Gentle reminder that this apparently made no progress for a week now. Or
> was there progress and I just missed it? Then apologies in advance.
>
> I'm asking, as it afaics would be nice to have this resolved in mainline
> before the next -rc. That would be ideal to avoid last minute changes
> after the last rc.
>
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> If I did something stupid, please tell me, as explained on that page.
>
> #regzbot poke
>
> >> On Jun 6, 2023, at 10:08 AM, Matthieu Baerts <matthieu.baerts@tessares.net> wrote:
> >>
> >> Hi Eric,
> >>
> >> On 06/06/2023 16:49, Eric Van Hensbergen wrote:
> >>> thanks for the detailed report - I’ll have a look.
> >>
> >> Thank you for the quick reply!
> >>
> >> No hurry, we can also revert these patches in our tree for the time being.
> >>
> >>> Do you know off hand
> >>> what cache options for 9p are in use?
> >>
> >> Sorry, I'm not sure what to look at.
> >>
> >> In the VM, I see that this mount command is used:
> >>
> >>  /bin/mount -n -t 9p -o ro,version=9p2000.L,trans=virtio,access=any \
> >>             virtme.guesttools /run/virtme/guesttools
> >>
> >> In virtme source, I can see these commands should be executed:
> >>
> >>  /bin/mount -n -t 9p -o ro,version=9p2000.L,trans=virtio,access=any \
> >>             /dev/root /newroot/
> >>  mount -t 9p -o version=9p2000.L,trans=virtio,access=any \
> >>        "virtme.initmount${tag:16}" "${!tag}"
> >>
> >> From the VM, I can see:
> >>
> >>> # mount | grep 9p
> >>> /dev/root on / type 9p (ro,relatime,access=any,trans=virtio)
> >>> virtme.guesttools on /run/virtme/guesttools type 9p (ro,relatime,access=any,trans=virtio)
> >>> virtme.initmount0 on /mptcp_net-next type 9p (rw,relatime,access=any,trans=virtio)
> >>
> >> And the VM is started using:
> >>
> >>> /usr/bin/qemu-system-x86_64 -fsdev local,id=virtfs1,path=/,security_model=none,readonly=on,multidevs=remap -device virtio-9p-pci,fsdev=virtfs1,mount_tag=/dev/root -fsdev local,id=virtfs5,path=/opt/virtme/virtme/guest,security_model=none,readonly=on,multidevs=remap -device virtio-9p-pci,fsdev=virtfs5,mount_tag=virtme.guesttools -fsdev local,id=virtfs9,path=.,security_model=none,multidevs=remap -device virtio-9p-pci,fsdev=virtfs9,mount_tag=virtme.initmount0 -machine accel=kvm:tcg -watchdog i6300esb -cpu host -parallel none -net none -echr 1 -serial none -chardev stdio,id=console,signal=off,mux=on -serial chardev:console -mon chardev=console -vga none -display none -m 2048M -smp 2 -device virtio-net-pci,netdev=n0 -netdev user,id=n0 -kernel .virtme/build/arch/x86/boot/bzImage -append 'virtme_link_mods=.virtme/build/.virtme_mods/lib/modules/0.0.0 virtme_initmount0=/mptcp_net-next earlyprintk=serial,ttyS0,115200 console=ttyS0 psmouse.proto=exps "virtme_stty_con=rows 59 cols 132 iutf8" TERM=xterm virtme.dhcp virtme_chdir=/mptcp_net-next rootfstype=9p rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect ro mitigations=off init=/bin/sh -- -c "mount -t tmpfs run /run;mkdir -p /run/virtme/guesttools;/bin/mount -n -t 9p -o ro,version=9p2000.L,trans=virtio,access=any virtme.guesttools /run/virtme/guesttools;exec /run/virtme/guesttools/virtme-init"'
> >>
> >> I don't see anything about the cache and nothing in the dmesg. If you
> >> think this info is in the dmesg, the logs are publicly available there:
> >>
> >> https://cirrus-ci.com/task/5569063884161024?logs=test#L5626
> >>
> >>
> >>> Since things are slower, my first
> >>> suspect is the don't cache qid.version == 0 change.  That can be
> >>> overridden with a mount option but perhaps we should invert the default
> >>> as it is new behavior from a linux perspective.
> >>
> >> Can I easily change the default behaviour on the kernel side to try
> >> (without modifying the tools launching the mount commands)?
> >>
> >> Cheers,
> >> Matt
> >> --
> >> Tessares | Belgium | Hybrid Access Solutions
> >> www.tessares.net
> >>
> >
> >
> >

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: 9p: MPTCP tests regressions due to new 9p features in v6.4
  2023-06-13 16:07         ` Eric Van Hensbergen
@ 2023-06-13 16:27           ` Matthieu Baerts
  2023-06-22  7:53             ` Linux regression tracking (Thorsten Leemhuis)
  2023-08-07 15:42           ` Matthieu Baerts
  1 sibling, 1 reply; 10+ messages in thread
From: Matthieu Baerts @ 2023-06-13 16:27 UTC (permalink / raw)
  To: Eric Van Hensbergen, Linux regressions mailing list
  Cc: evanhensbergen, MPTCP Upstream, v9fs

Hi Eric,

On 13/06/2023 18:07, Eric Van Hensbergen wrote:
> Understood, this on top of my 9p stack, but day job has been getting
> in the way lately so I need to unstack there first.  Will do my best
> to reproduce and understand as quickly as I can -- but given its a
> performance regression it is likely going to be tricky to reproduce
> and understand, so I'm not sure how confident I am that we'll be able
> to close it out prior to the final -rc.

Thank you for your message, I understand, no issue.

On our side with MPTCP, there is no hurry as there is a workaround: the
revert of the series you sent for v6.4. So as long as it is tracked and
fixed at some points, we are fine :)

(but I don't know what's the impact for other people of course)

Cheers,
Matt
-- 
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: 9p: MPTCP tests regressions due to new 9p features in v6.4
  2023-06-13 16:27           ` Matthieu Baerts
@ 2023-06-22  7:53             ` Linux regression tracking (Thorsten Leemhuis)
  2023-06-29 14:29               ` Linux regression tracking #update (Thorsten Leemhuis)
  0 siblings, 1 reply; 10+ messages in thread
From: Linux regression tracking (Thorsten Leemhuis) @ 2023-06-22  7:53 UTC (permalink / raw)
  To: Matthieu Baerts, Eric Van Hensbergen, Linux regressions mailing list
  Cc: evanhensbergen, MPTCP Upstream, v9fs

On 13.06.23 18:27, Matthieu Baerts wrote:
> On 13/06/2023 18:07, Eric Van Hensbergen wrote:
>> Understood, this on top of my 9p stack, but day job has been getting
>> in the way lately so I need to unstack there first.  Will do my best
>> to reproduce and understand as quickly as I can -- but given its a
>> performance regression it is likely going to be tricky to reproduce
>> and understand, so I'm not sure how confident I am that we'll be able
>> to close it out prior to the final -rc.
> 
> Thank you for your message, I understand, no issue.
> [...]

As Linus will likely release 6.4 on this or the following Sunday a quick
status inquiry so I can brief him appropriately: is there any hope the
regression will be resolved any time soon? Doesn't look like it from
this thread, but maybe I missed something.

> (but I don't know what's the impact for other people of course)

Yeah, it's likely not that urgent, if the problem is just showing up in
some CI tests. But is that what we assume for now? Eric?

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

#regzbot poke

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: 9p: MPTCP tests regressions due to new 9p features in v6.4
  2023-06-22  7:53             ` Linux regression tracking (Thorsten Leemhuis)
@ 2023-06-29 14:29               ` Linux regression tracking #update (Thorsten Leemhuis)
  0 siblings, 0 replies; 10+ messages in thread
From: Linux regression tracking #update (Thorsten Leemhuis) @ 2023-06-29 14:29 UTC (permalink / raw)
  To: Linux regressions mailing list; +Cc: MPTCP Upstream, v9fs

[TLDR: This mail in primarily relevant for Linux kernel regression
tracking. See link in footer if these mails annoy you.]

On 22.06.23 09:53, Linux regression tracking (Thorsten Leemhuis) wrote:
> On 13.06.23 18:27, Matthieu Baerts wrote:
>> On 13/06/2023 18:07, Eric Van Hensbergen wrote:
>>> Understood, this on top of my 9p stack, but day job has been getting
>>> in the way lately so I need to unstack there first.  Will do my best
>>> to reproduce and understand as quickly as I can -- but given its a
>>> performance regression it is likely going to be tricky to reproduce
>>> and understand, so I'm not sure how confident I am that we'll be able
>>> to close it out prior to the final -rc.
>>
>> Thank you for your message, I understand, no issue.
>> [...]
> 
> As Linus will likely release 6.4 on this or the following Sunday a quick
> status inquiry so I can brief him appropriately: is there any hope the
> regression will be resolved any time soon? Doesn't look like it from
> this thread, but maybe I missed something.
> 
>> (but I don't know what's the impact for other people of course)
> 
> Yeah, it's likely not that urgent, if the problem is just showing up in
> some CI tests. But is that what we assume for now? Eric?

6.4 is out and this is likely to take a while to get fixed given recent
and today's replies.

To make sure I don't see this all the time while dealing with more
pressing issues I'm putting it on the backburner:

#regzbot backburner: not urgent, for now seems something that only shows
up in one CI (and is hard to reproduce)
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: 9p: MPTCP tests regressions due to new 9p features in v6.4
  2023-06-13 16:07         ` Eric Van Hensbergen
  2023-06-13 16:27           ` Matthieu Baerts
@ 2023-08-07 15:42           ` Matthieu Baerts
  2023-08-07 15:56             ` Eric Van Hensbergen
  1 sibling, 1 reply; 10+ messages in thread
From: Matthieu Baerts @ 2023-08-07 15:42 UTC (permalink / raw)
  To: Eric Van Hensbergen, Linux regressions mailing list
  Cc: evanhensbergen, MPTCP Upstream, v9fs

Hi Eric,

On 13/06/2023 18:07, Eric Van Hensbergen wrote:
> Understood, this on top of my 9p stack, but day job has been getting
> in the way lately so I need to unstack there first.  Will do my best
> to reproduce and understand as quickly as I can -- but given its a
> performance regression it is likely going to be tricky to reproduce
> and understand, so I'm not sure how confident I am that we'll be able
> to close it out prior to the final -rc.

Good news, it looks like the bug has been fixed!

I recently noticed some new patches fixing some issues in 9p in Linus
tree because they were conflicting with the revert of 9p's v6.4 PR I did
in June to stop having random issues in our tests. I then tried
reproducing the issue we had on top of Linus' tree and I was not able
too! (but I was still able to reproduce them just before the recent fixes)

After a quick git bisect, it seems the bug is fixed thanks to
350cd9b95975 ("fs/9p: remove unnecessary invalidate_inode_pages2").

So we can also tell regzbot to stop tracking this:

#regzbot fix: 350cd9b95975

Cheers,
Matt
-- 
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: 9p: MPTCP tests regressions due to new 9p features in v6.4
  2023-08-07 15:42           ` Matthieu Baerts
@ 2023-08-07 15:56             ` Eric Van Hensbergen
  0 siblings, 0 replies; 10+ messages in thread
From: Eric Van Hensbergen @ 2023-08-07 15:56 UTC (permalink / raw)
  To: Matthieu Baerts
  Cc: Linux regressions mailing list, evanhensbergen, MPTCP Upstream, v9fs

That's good news because I was having a horrible time reproducing it.
I'm glad we removed that extra bit, we were thinking it was benign,
but I guess it turns out it wasn't!

     -eric

On Mon, Aug 7, 2023 at 10:42 AM Matthieu Baerts
<matthieu.baerts@tessares.net> wrote:
>
> Hi Eric,
>
> On 13/06/2023 18:07, Eric Van Hensbergen wrote:
> > Understood, this on top of my 9p stack, but day job has been getting
> > in the way lately so I need to unstack there first.  Will do my best
> > to reproduce and understand as quickly as I can -- but given its a
> > performance regression it is likely going to be tricky to reproduce
> > and understand, so I'm not sure how confident I am that we'll be able
> > to close it out prior to the final -rc.
>
> Good news, it looks like the bug has been fixed!
>
> I recently noticed some new patches fixing some issues in 9p in Linus
> tree because they were conflicting with the revert of 9p's v6.4 PR I did
> in June to stop having random issues in our tests. I then tried
> reproducing the issue we had on top of Linus' tree and I was not able
> too! (but I was still able to reproduce them just before the recent fixes)
>
> After a quick git bisect, it seems the bug is fixed thanks to
> 350cd9b95975 ("fs/9p: remove unnecessary invalidate_inode_pages2").
>
> So we can also tell regzbot to stop tracking this:
>
> #regzbot fix: 350cd9b95975
>
> Cheers,
> Matt
> --
> Tessares | Belgium | Hybrid Access Solutions
> www.tessares.net

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2023-08-07 15:57 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-06-06 14:30 9p: MPTCP tests regressions due to new 9p features in v6.4 Matthieu Baerts
     [not found] ` <CAFkjPTnX0=-GK8sFzd4S+V2+cA8E-FAqYHNndZui2Sh_MvoHPw@mail.gmail.com>
2023-06-06 15:08   ` Matthieu Baerts
2023-06-06 18:47     ` evanhensbergen
2023-06-13 14:56       ` Linux regression tracking (Thorsten Leemhuis)
2023-06-13 16:07         ` Eric Van Hensbergen
2023-06-13 16:27           ` Matthieu Baerts
2023-06-22  7:53             ` Linux regression tracking (Thorsten Leemhuis)
2023-06-29 14:29               ` Linux regression tracking #update (Thorsten Leemhuis)
2023-08-07 15:42           ` Matthieu Baerts
2023-08-07 15:56             ` Eric Van Hensbergen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).