Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

[-- Attachment #1: Type: text/plain, Size: 1367 bytes --]

Hi,

Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
KConfig value for READ_PLUS") as one leading to NULL pointer exception
when mounting NFS root on NFSv4 client:

[   25.739003] systemd[1]: Set hostname to <odroidhc1>.
[   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
argument
[   26.199478] 8<--- cut here ---
[   26.201366] Unable to handle kernel NULL pointer dereference at
virtual address 00000004
...
[   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
[   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
[   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304

Full OOPS attached. Full log available here:
https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0

Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
not the cause, but rather making it default caused the regression.

I did not make the bisect yet which commit introduced it, if every
config includes NFS_V4_2_READ_PLUS.


Some details about platform:

1. Arch ARM Linux
2. exynos_defconfig
3. Odroid HC1 board with ARMv7, octa-core (Cortex-A7+A15), Exynos5422 SoC
4. systemd, boot up with static IP set in kernel command line
5. No swap
6. Kernel, DTB and initramfs are downloaded with TFTP
7. NFS root (NFS client) mounted from a NFSv4 server


Best regards,
Krzysztof

[-- Attachment #2: log.txt --]
[-- Type: text/plain, Size: 5913 bytes --]

25.628075] systemd[1]: systemd 242.29-1-arch running in system mode. (+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
[   25.656202] systemd[1]: Detected architecture arm.
[   25.664007] random: crng init done

Welcome to Arch Linux ARM!

[   25.739003] systemd[1]: Set hostname to <odroidhc1>.
[   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument
[   26.199478] 8<--- cut here ---
[   26.201366] Unable to handle kernel NULL pointer dereference at virtual address 00000004
[   26.209389] [00000004] *pgd=00000000
[   26.212962] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[   26.218091] Modules linked in:
[   26.221120] CPU: 4 PID: 62 Comm: kworker/u16:1 Not tainted 6.1.0-03250-ga044dab5e6e5 #92
[   26.229175] Hardware name: Samsung Exynos (Flattened Device Tree)
[   26.235241] Workqueue: rpciod rpc_async_schedule
[   26.239828] PC is at mmiocpy+0x4c/0x334
[   26.243639] LR is at xdr_inline_decode+0xec/0x16c
[   26.248320] pc : [<c0b938ec>]    lr : [<c0b5f740>]    psr: 200a0113
[   26.254561] sp : f0a71d98  ip : 00000000  fp : f0a71da8
[   26.259759] r10: 00003000  r9 : 00000009  r8 : c1d1dc18
[   26.264956] r7 : 00000ffc  r6 : f0a71e24  r5 : 00000004  r4 : f0a71e60
[   26.271457] r3 : 00000000  r2 : 00000f7c  r1 : 00000004  r0 : f0a71e24
[   26.277956] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   26.285064] Control: 10c5387d  Table: 4407006a  DAC: 00000051
[   26.290787] Register r0 information: 2-page vmalloc region starting at 0xf0a70000 allocated at kernel_clone+0x58/0x3fc
[   26.301446] Register r1 information: non-paged memory
[   26.306464] Register r2 information: non-paged memory
[   26.311490] Register r3 information: NULL pointer
[   26.316169] Register r4 information: 2-page vmalloc region starting at 0xf0a70000 allocated at kernel_clone+0x58/0x3fc
[   26.326838] Register r5 information: non-paged memory
[   26.331856] Register r6 information: 2-page vmalloc region starting at 0xf0a70000 allocated at kernel_clone+0x58/0x3fc
[   26.342522] Register r7 information: non-paged memory
[   26.347541] Register r8 information: slab kmalloc-256 start c1d1dc00 pointer offset 24 size 256
[   26.356211] Register r9 information: non-paged memory
[   26.361232] Register r10 information: non-paged memory
[   26.366345] Register r11 information: 2-page vmalloc region starting at 0xf0a70000 allocated at kernel_clone+0x58/0x3fc
[   26.377098] Register r12 information: NULL pointer
[   26.381856] Process kworker/u16:1 (pid: 62, stack limit = 0x89ca8077)
[   26.388271] Stack: (0xf0a71d98 to 0xf0a72000)
[   26.392600] 1d80:                                                       00000004 f0a71e24
[   26.400766] 1da0: c1d1dc18 00000009 f0a71e24 f0a71e60 c1d1dc00 c0b5f740 00000001 f0a71e60
[   26.408898] 1dc0: c1d91100 00000009 c1d1dc18 c04870f8 c1e527d8 c1171584 00000000 00000000
[   26.417043] 1de0: 00000003 00000000 00000000 00000000 0000001a 00000004 c26faa1c 00000000
[   26.425189] 1e00: c26faa94 00000000 c17f1498 c3321600 00000000 00003000 00000000 00003000
[   26.433335] 1e20: 00003000 c25da000 c1d90f58 00000000 f0a71e60 2b750d84 00000000 c1d90f58
[   26.441479] 1e40: c40a4000 c1360180 c1208f10 c40a4030 c1e52000 c0b340b0 c1d90f74 c0b342b4
[   26.449626] 1e60: 00001000 c40a4030 00001000 00000000 f0a71e24 00000010 c332160c 00002c24
[   26.457771] 1e80: c40a4000 2b750d84 c1208f10 c1d90f58 c1e52000 c1208f10 c0b340b0 c1360160
[   26.465916] 1ea0: c0b4e8f4 c0b4f060 00000000 00000001 04248160 c1357d57 c1171584 00000000
[   26.474062] 1ec0: 00000001 2b750d84 00000002 c1e52000 04208160 c1c08e00 c2752900 c1e52000
[   26.482208] 1ee0: c1358980 c1208f10 c2752905 c0b4f828 c1d90f7c c1cffd00 c1c08e00 c0148140
[   26.490353] 1f00: 00000001 00000000 c014809c c1e52000 00002dfd 00000000 c1e52000 c1c08e00
[   26.498498] 1f20: c19dbb60 c17f1498 00000000 c0f5ffec 00000000 2b750d84 c1c08e00 c1cffd00
[   26.506644] 1f40: c1c08e00 c1cffd18 c1c08e3c c1205d40 00000088 c1e52000 c1c08e00 c0148a60
[   26.514789] 1f60: c1e52000 c1357e40 c1e70f80 c1f2b0c0 c1e52000 c0148a0c c1cffd00 c1e70f80
[   26.522935] 1f80: f088dea4 00000000 00000000 c0151108 c1f2b0c0 c0151018 00000000 00000000
[   26.531081] 1fa0: 00000000 00000000 00000000 c0100108 00000000 00000000 00000000 00000000
[   26.539226] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   26.547371] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
[   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
[   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
[   26.573107]  call_decode from __rpc_execute+0xf8/0x8a4
[   26.578218]  __rpc_execute from rpc_async_schedule+0x1c/0x34
[   26.583848]  rpc_async_schedule from process_one_work+0x294/0x78c
[   26.589916]  process_one_work from worker_thread+0x54/0x518
[   26.595459]  worker_thread from kthread+0xf0/0x124
[   26.600226]  kthread from ret_from_fork+0x14/0x2c
[   26.604903] Exception stack(0xf0a71fb0 to 0xf0a71ff8)
[   26.609931] 1fa0:                                     00000000 00000000 00000000 00000000
[   26.618082] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   26.626226] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[   26.632814] Code: ba000002 f5d1f03c f5d1f05c f5d1f07c (e8b15378) 
[   26.638976] ---[ end trace 0000000000000000 ]---
[   39.512299] vdd_ldo12: disabling
[   39.516426] vdd_ldo16: disabling
[   39.522488] vdd_ldo24: disabling
[   39.526582] vdd_ldo26: disabling
[   39.544786] vdd_vmem: disabling

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Linux kernel regression tracking (#adding)]

[TLDR: I'm adding this report to the list of tracked Linux kernel
regressions; the text you find below is based on a few templates
paragraphs you might have encountered already in similar form.
See link in footer if these mails annoy you.]

On 07.01.23 16:44, Krzysztof Kozlowski wrote:
> Hi,
> 
> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
> KConfig value for READ_PLUS") as one leading to NULL pointer exception
> when mounting NFS root on NFSv4 client:
> 
> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
> argument
> [   26.199478] 8<--- cut here ---
> [   26.201366] Unable to handle kernel NULL pointer dereference at
> virtual address 00000004
> ...
> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
> 
> Full OOPS attached. Full log available here:
> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
> 
> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
> not the cause, but rather making it default caused the regression.
> 
> I did not make the bisect yet which commit introduced it, if every
> config includes NFS_V4_2_READ_PLUS.
> 
> 
> Some details about platform:
> 
> 1. Arch ARM Linux
> 2. exynos_defconfig
> 3. Odroid HC1 board with ARMv7, octa-core (Cortex-A7+A15), Exynos5422 SoC
> 4. systemd, boot up with static IP set in kernel command line
> 5. No swap
> 6. Kernel, DTB and initramfs are downloaded with TFTP
> 7. NFS root (NFS client) mounted from a NFSv4 server

Thanks for the report. To be sure the issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
tracking bot:

#regzbot ^introduced 7fd461c47
#regzbot title nfs: NULL pointer dereference since NFS_V4_2_READ_PLUS is
enabled by default
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply and tell me -- ideally
while also telling regzbot about it, as explained by the page listed in
the footer of this mail.

Developers: When fixing the issue, remember to add 'Link:' tags pointing
to the report (the parent of this mail). See page linked in footer for
details.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
> Hi,
> 
> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
> KConfig value for READ_PLUS") as one leading to NULL pointer exception
> when mounting NFS root on NFSv4 client:
> 
> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
> argument
> [   26.199478] 8<--- cut here ---
> [   26.201366] Unable to handle kernel NULL pointer dereference at
> virtual address 00000004
> ...
> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
> 
> Full OOPS attached. Full log available here:
> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
> 
> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
> not the cause, but rather making it default caused the regression.
> 
> I did not make the bisect yet which commit introduced it, if every
> config includes NFS_V4_2_READ_PLUS.

When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")

commit d3b00a802c845a6021148ce2e669b5a0b5729959
Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
Date:   Thu Jul 21 14:21:34 2022 -0400

    NFS: Replace the READ_PLUS decoding code

    We now take a 2-step process that allows us to place data and hole
    segments directly at their final position in the xdr_stream without
    needing to do a bunch of redundant copies to expand holes. Due to the
    variable lengths of each segment, the xdr metadata might cross page
    boundaries which I account for by setting a small scratch buffer so
    xdr_inline_decode() won't fail.

    Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
    Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>

With a trace:
[   25.898462] systemd[1]: Set hostname to <odroidhc1>.
[   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
argument
[   25.986237] random: crng init done
[   26.264564] 8<--- cut here ---
[   26.266823] Unable to handle kernel NULL pointer dereference at
virtual address 00000fe8
...
[   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
[   26.603222]  call_decode from __rpc_execute+0xd0/0x890
[   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
[   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
[   26.620030]  process_one_work from worker_thread+0x54/0x518
[   26.625570]  worker_thread from kthread+0xf4/0x128
[   26.630336]  kthread from ret_from_fork+0x14/0x2c


Best regards,
Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Trond Myklebust]

Hi Krzysztof,

> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
> 
> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
> 
> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
>> Hi,
>> 
>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>> when mounting NFS root on NFSv4 client:
>> 
>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>> argument
>> [   26.199478] 8<--- cut here ---
>> [   26.201366] Unable to handle kernel NULL pointer dereference at
>> virtual address 00000004
>> ...
>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>> 
>> Full OOPS attached. Full log available here:
>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
>> 
>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
>> not the cause, but rather making it default caused the regression.
>> 
>> I did not make the bisect yet which commit introduced it, if every
>> config includes NFS_V4_2_READ_PLUS.
> 
> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
> 
> commit d3b00a802c845a6021148ce2e669b5a0b5729959
> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
> Date:   Thu Jul 21 14:21:34 2022 -0400
> 
>    NFS: Replace the READ_PLUS decoding code
> 
>    We now take a 2-step process that allows us to place data and hole
>    segments directly at their final position in the xdr_stream without
>    needing to do a bunch of redundant copies to expand holes. Due to the
>    variable lengths of each segment, the xdr metadata might cross page
>    boundaries which I account for by setting a small scratch buffer so
>    xdr_inline_decode() won't fail.
> 
>    Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
>    Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> 
> With a trace:
> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
> argument
> [   25.986237] random: crng init done
> [   26.264564] 8<--- cut here ---
> [   26.266823] Unable to handle kernel NULL pointer dereference at
> virtual address 00000fe8
> ...
> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
> [   26.620030]  process_one_work from worker_thread+0x54/0x518
> [   26.625570]  worker_thread from kthread+0xf4/0x128
> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
> 

Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?

Thanks
  Trond

_________________________________
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

On 08/01/2023 18:09, Trond Myklebust wrote:
> Hi Krzysztof,
> 
>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>
>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
>>
>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
>>> Hi,
>>>
>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>>> when mounting NFS root on NFSv4 client:
>>>
>>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
>>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>> argument
>>> [   26.199478] 8<--- cut here ---
>>> [   26.201366] Unable to handle kernel NULL pointer dereference at
>>> virtual address 00000004
>>> ...
>>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
>>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
>>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>
>>> Full OOPS attached. Full log available here:
>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
>>>
>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
>>> not the cause, but rather making it default caused the regression.
>>>
>>> I did not make the bisect yet which commit introduced it, if every
>>> config includes NFS_V4_2_READ_PLUS.
>>
>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
>>
>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
>> Date:   Thu Jul 21 14:21:34 2022 -0400
>>
>>    NFS: Replace the READ_PLUS decoding code
>>
>>    We now take a 2-step process that allows us to place data and hole
>>    segments directly at their final position in the xdr_stream without
>>    needing to do a bunch of redundant copies to expand holes. Due to the
>>    variable lengths of each segment, the xdr metadata might cross page
>>    boundaries which I account for by setting a small scratch buffer so
>>    xdr_inline_decode() won't fail.
>>
>>    Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>    Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
>>
>> With a trace:
>> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
>> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>> argument
>> [   25.986237] random: crng init done
>> [   26.264564] 8<--- cut here ---
>> [   26.266823] Unable to handle kernel NULL pointer dereference at
>> virtual address 00000fe8
>> ...
>> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
>> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
>> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
>> [   26.620030]  process_one_work from worker_thread+0x54/0x518
>> [   26.625570]  worker_thread from kthread+0xf4/0x128
>> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
>>
> 
> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?

I would say that buggy server should not cause NULL pointer dereferences
on the client. Otherwise this is a perfect recipe for a rogue server in
the network to start crashing clients and running exploits... Imagine a
compromised machine (through some other means) in a local company
network running now a server with NFS share "HR salary data" or "HR
planned layoffs", where unsuspected people in that network access it
leading to exploit of NFS code on their side...

Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH

Which points that it is not latest stable, so anyway I need to update.

Best regards,
Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

On 09/01/2023 09:14, Krzysztof Kozlowski wrote:
> On 08/01/2023 18:09, Trond Myklebust wrote:
>> Hi Krzysztof,
>>
>>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>>
>>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
>>>
>>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
>>>> Hi,
>>>>
>>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>>>> when mounting NFS root on NFSv4 client:
>>>>
>>>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
>>>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>> argument
>>>> [   26.199478] 8<--- cut here ---
>>>> [   26.201366] Unable to handle kernel NULL pointer dereference at
>>>> virtual address 00000004
>>>> ...
>>>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
>>>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
>>>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>
>>>> Full OOPS attached. Full log available here:
>>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
>>>>
>>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
>>>> not the cause, but rather making it default caused the regression.
>>>>
>>>> I did not make the bisect yet which commit introduced it, if every
>>>> config includes NFS_V4_2_READ_PLUS.
>>>
>>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
>>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
>>>
>>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
>>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>> Date:   Thu Jul 21 14:21:34 2022 -0400
>>>
>>>    NFS: Replace the READ_PLUS decoding code
>>>
>>>    We now take a 2-step process that allows us to place data and hole
>>>    segments directly at their final position in the xdr_stream without
>>>    needing to do a bunch of redundant copies to expand holes. Due to the
>>>    variable lengths of each segment, the xdr metadata might cross page
>>>    boundaries which I account for by setting a small scratch buffer so
>>>    xdr_inline_decode() won't fail.
>>>
>>>    Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>    Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
>>>
>>> With a trace:
>>> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
>>> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>> argument
>>> [   25.986237] random: crng init done
>>> [   26.264564] 8<--- cut here ---
>>> [   26.266823] Unable to handle kernel NULL pointer dereference at
>>> virtual address 00000fe8
>>> ...
>>> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
>>> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
>>> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
>>> [   26.620030]  process_one_work from worker_thread+0x54/0x518
>>> [   26.625570]  worker_thread from kthread+0xf4/0x128
>>> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
>>>
>>
>> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?
> 
> I would say that buggy server should not cause NULL pointer dereferences
> on the client. Otherwise this is a perfect recipe for a rogue server in
> the network to start crashing clients and running exploits... Imagine a
> compromised machine (through some other means) in a local company
> network running now a server with NFS share "HR salary data" or "HR
> planned layoffs", where unsuspected people in that network access it
> leading to exploit of NFS code on their side...
> 
> Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH
> 
> Which points that it is not latest stable, so anyway I need to update.

I updated the kernel to 5.15.84-3-rpi-ARCH which is pretty close to
latest stable and I can reproduce the issue. Therefore:
1. It is visible on two stable (one new, one old) kernels on the server,
2. Buggy or rogue server should not cause NULL pointer on remote devices...

Best regards,
Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Trond Myklebust]

> On Jan 9, 2023, at 03:42, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
> 
> On 09/01/2023 09:14, Krzysztof Kozlowski wrote:
>> On 08/01/2023 18:09, Trond Myklebust wrote:
>>> Hi Krzysztof,
>>> 
>>>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>>> 
>>>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
>>>> 
>>>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
>>>>> Hi,
>>>>> 
>>>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>>>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>>>>> when mounting NFS root on NFSv4 client:
>>>>> 
>>>>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
>>>>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>> argument
>>>>> [   26.199478] 8<--- cut here ---
>>>>> [   26.201366] Unable to handle kernel NULL pointer dereference at
>>>>> virtual address 00000004
>>>>> ...
>>>>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
>>>>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
>>>>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>> 
>>>>> Full OOPS attached. Full log available here:
>>>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
>>>>> 
>>>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
>>>>> not the cause, but rather making it default caused the regression.
>>>>> 
>>>>> I did not make the bisect yet which commit introduced it, if every
>>>>> config includes NFS_V4_2_READ_PLUS.
>>>> 
>>>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
>>>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
>>>> 
>>>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
>>>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>> Date:   Thu Jul 21 14:21:34 2022 -0400
>>>> 
>>>>   NFS: Replace the READ_PLUS decoding code
>>>> 
>>>>   We now take a 2-step process that allows us to place data and hole
>>>>   segments directly at their final position in the xdr_stream without
>>>>   needing to do a bunch of redundant copies to expand holes. Due to the
>>>>   variable lengths of each segment, the xdr metadata might cross page
>>>>   boundaries which I account for by setting a small scratch buffer so
>>>>   xdr_inline_decode() won't fail.
>>>> 
>>>>   Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>   Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
>>>> 
>>>> With a trace:
>>>> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
>>>> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>> argument
>>>> [   25.986237] random: crng init done
>>>> [   26.264564] 8<--- cut here ---
>>>> [   26.266823] Unable to handle kernel NULL pointer dereference at
>>>> virtual address 00000fe8
>>>> ...
>>>> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
>>>> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
>>>> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
>>>> [   26.620030]  process_one_work from worker_thread+0x54/0x518
>>>> [   26.625570]  worker_thread from kthread+0xf4/0x128
>>>> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
>>>> 
>>> 
>>> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?
>> 
>> I would say that buggy server should not cause NULL pointer dereferences
>> on the client. Otherwise this is a perfect recipe for a rogue server in
>> the network to start crashing clients and running exploits... Imagine a
>> compromised machine (through some other means) in a local company
>> network running now a server with NFS share "HR salary data" or "HR
>> planned layoffs", where unsuspected people in that network access it
>> leading to exploit of NFS code on their side...
>> 
>> Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH
>> 
>> Which points that it is not latest stable, so anyway I need to update.
> 
> I updated the kernel to 5.15.84-3-rpi-ARCH which is pretty close to
> latest stable and I can reproduce the issue. Therefore:
> 1. It is visible on two stable (one new, one old) kernels on the server,
> 2. Buggy or rogue server should not cause NULL pointer on remote devices...
> 

The nfsd READ_PLUS code is borked up and until 6.2-rc1. I thought we had a server option to disable that code, but it seems as if that is not the case.
Chuck + Anna, can we please send a stable patch to rip out that code altogether from all the older versions? If we want to enable READ_PLUS by default on the client, then we cannot do so if the majority of NFSv4.2 servers out there are running a borked implementation.

I do agree that we cannot allow buggy servers to cause memory corruption in the client code, so I propose that we revert the Kconfig default setting change again until both the client code and the legacy servers have been fixed. Anna?

Thanks
  Trond

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Chuck Lever III]

> On Jan 9, 2023, at 9:44 AM, Trond Myklebust <trondmy@hammerspace.com> wrote:
> 
> 
> 
>> On Jan 9, 2023, at 03:42, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>> 
>> On 09/01/2023 09:14, Krzysztof Kozlowski wrote:
>>> On 08/01/2023 18:09, Trond Myklebust wrote:
>>>> Hi Krzysztof,
>>>> 
>>>>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>>>> 
>>>>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
>>>>> 
>>>>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>>>>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>>>>>> when mounting NFS root on NFSv4 client:
>>>>>> 
>>>>>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
>>>>>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>>> argument
>>>>>> [   26.199478] 8<--- cut here ---
>>>>>> [   26.201366] Unable to handle kernel NULL pointer dereference at
>>>>>> virtual address 00000004
>>>>>> ...
>>>>>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
>>>>>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
>>>>>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>>> 
>>>>>> Full OOPS attached. Full log available here:
>>>>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
>>>>>> 
>>>>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
>>>>>> not the cause, but rather making it default caused the regression.
>>>>>> 
>>>>>> I did not make the bisect yet which commit introduced it, if every
>>>>>> config includes NFS_V4_2_READ_PLUS.
>>>>> 
>>>>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
>>>>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
>>>>> 
>>>>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
>>>>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>> Date:   Thu Jul 21 14:21:34 2022 -0400
>>>>> 
>>>>>  NFS: Replace the READ_PLUS decoding code
>>>>> 
>>>>>  We now take a 2-step process that allows us to place data and hole
>>>>>  segments directly at their final position in the xdr_stream without
>>>>>  needing to do a bunch of redundant copies to expand holes. Due to the
>>>>>  variable lengths of each segment, the xdr metadata might cross page
>>>>>  boundaries which I account for by setting a small scratch buffer so
>>>>>  xdr_inline_decode() won't fail.
>>>>> 
>>>>>  Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>>  Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
>>>>> 
>>>>> With a trace:
>>>>> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
>>>>> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>> argument
>>>>> [   25.986237] random: crng init done
>>>>> [   26.264564] 8<--- cut here ---
>>>>> [   26.266823] Unable to handle kernel NULL pointer dereference at
>>>>> virtual address 00000fe8
>>>>> ...
>>>>> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
>>>>> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
>>>>> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
>>>>> [   26.620030]  process_one_work from worker_thread+0x54/0x518
>>>>> [   26.625570]  worker_thread from kthread+0xf4/0x128
>>>>> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
>>>>> 
>>>> 
>>>> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?
>>> 
>>> I would say that buggy server should not cause NULL pointer dereferences
>>> on the client. Otherwise this is a perfect recipe for a rogue server in
>>> the network to start crashing clients and running exploits... Imagine a
>>> compromised machine (through some other means) in a local company
>>> network running now a server with NFS share "HR salary data" or "HR
>>> planned layoffs", where unsuspected people in that network access it
>>> leading to exploit of NFS code on their side...
>>> 
>>> Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH
>>> 
>>> Which points that it is not latest stable, so anyway I need to update.
>> 
>> I updated the kernel to 5.15.84-3-rpi-ARCH which is pretty close to
>> latest stable and I can reproduce the issue. Therefore:
>> 1. It is visible on two stable (one new, one old) kernels on the server,
>> 2. Buggy or rogue server should not cause NULL pointer on remote devices...
>> 
> 
> The nfsd READ_PLUS code is borked up and until 6.2-rc1. I thought we had a server option to disable that code, but it seems as if that is not the case.
> Chuck + Anna, can we please send a stable patch to rip out that code altogether from all the older versions? If we want to enable READ_PLUS by default on the client, then we cannot do so if the majority of NFSv4.2 servers out there are running a borked implementation.
> 
> I do agree that we cannot allow buggy

or malicious, or non-Linux,


> servers to cause memory corruption in the client code, so I propose that we revert the Kconfig default setting change again until both the client code and the legacy servers have been fixed.

I stand ready to receive and apply server-side fixes, as you suggested.

However IMO it would be most responsible to go straight for a client code fix. The Kconfig setting is a red herring (as delicious as that might sound). Any thoughts about how difficult that fix might be?


--
Chuck Lever

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Anna Schumaker]

On Mon, Jan 9, 2023 at 10:12 AM Chuck Lever III <chuck.lever@oracle.com> wrote:
>
>
>
> > On Jan 9, 2023, at 9:44 AM, Trond Myklebust <trondmy@hammerspace.com> wrote:
> >
> >
> >
> >> On Jan 9, 2023, at 03:42, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
> >>
> >> On 09/01/2023 09:14, Krzysztof Kozlowski wrote:
> >>> On 08/01/2023 18:09, Trond Myklebust wrote:
> >>>> Hi Krzysztof,
> >>>>
> >>>>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
> >>>>>
> >>>>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
> >>>>>
> >>>>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
> >>>>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
> >>>>>> when mounting NFS root on NFSv4 client:
> >>>>>>
> >>>>>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
> >>>>>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
> >>>>>> argument
> >>>>>> [   26.199478] 8<--- cut here ---
> >>>>>> [   26.201366] Unable to handle kernel NULL pointer dereference at
> >>>>>> virtual address 00000004
> >>>>>> ...
> >>>>>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
> >>>>>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
> >>>>>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
> >>>>>>
> >>>>>> Full OOPS attached. Full log available here:
> >>>>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
> >>>>>>
> >>>>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
> >>>>>> not the cause, but rather making it default caused the regression.
> >>>>>>
> >>>>>> I did not make the bisect yet which commit introduced it, if every
> >>>>>> config includes NFS_V4_2_READ_PLUS.
> >>>>>
> >>>>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
> >>>>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
> >>>>>
> >>>>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
> >>>>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
> >>>>> Date:   Thu Jul 21 14:21:34 2022 -0400
> >>>>>
> >>>>>  NFS: Replace the READ_PLUS decoding code
> >>>>>
> >>>>>  We now take a 2-step process that allows us to place data and hole
> >>>>>  segments directly at their final position in the xdr_stream without
> >>>>>  needing to do a bunch of redundant copies to expand holes. Due to the
> >>>>>  variable lengths of each segment, the xdr metadata might cross page
> >>>>>  boundaries which I account for by setting a small scratch buffer so
> >>>>>  xdr_inline_decode() won't fail.
> >>>>>
> >>>>>  Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
> >>>>>  Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> >>>>>
> >>>>> With a trace:
> >>>>> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
> >>>>> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
> >>>>> argument
> >>>>> [   25.986237] random: crng init done
> >>>>> [   26.264564] 8<--- cut here ---
> >>>>> [   26.266823] Unable to handle kernel NULL pointer dereference at
> >>>>> virtual address 00000fe8
> >>>>> ...
> >>>>> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
> >>>>> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
> >>>>> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
> >>>>> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
> >>>>> [   26.620030]  process_one_work from worker_thread+0x54/0x518
> >>>>> [   26.625570]  worker_thread from kthread+0xf4/0x128
> >>>>> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
> >>>>>
> >>>>
> >>>> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?
> >>>
> >>> I would say that buggy server should not cause NULL pointer dereferences
> >>> on the client. Otherwise this is a perfect recipe for a rogue server in
> >>> the network to start crashing clients and running exploits... Imagine a
> >>> compromised machine (through some other means) in a local company
> >>> network running now a server with NFS share "HR salary data" or "HR
> >>> planned layoffs", where unsuspected people in that network access it
> >>> leading to exploit of NFS code on their side...
> >>>
> >>> Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH
> >>>
> >>> Which points that it is not latest stable, so anyway I need to update.
> >>
> >> I updated the kernel to 5.15.84-3-rpi-ARCH which is pretty close to
> >> latest stable and I can reproduce the issue. Therefore:
> >> 1. It is visible on two stable (one new, one old) kernels on the server,
> >> 2. Buggy or rogue server should not cause NULL pointer on remote devices...
> >>
> >
> > The nfsd READ_PLUS code is borked up and until 6.2-rc1. I thought we had a server option to disable that code, but it seems as if that is not the case.
> > Chuck + Anna, can we please send a stable patch to rip out that code altogether from all the older versions? If we want to enable READ_PLUS by default on the client, then we cannot do so if the majority of NFSv4.2 servers out there are running a borked implementation.
> >
> > I do agree that we cannot allow buggy
>
> or malicious, or non-Linux,
>
>
> > servers to cause memory corruption in the client code, so I propose that we revert the Kconfig default setting change again until both the client code and the legacy servers have been fixed.
>
> I stand ready to receive and apply server-side fixes, as you suggested.
>
> However IMO it would be most responsible to go straight for a client code fix. The Kconfig setting is a red herring (as delicious as that might sound). Any thoughts about how difficult that fix might be?

I'm wondering about how hard the fix might be as well, since it could
be a legitimate bug or some error checking that I've overlooked. I've
gotten as far as installing a 5.15 server in my testing setup, so
we'll see if I'm able to reproduce the crash this morning.

Anna

>
>
> --
> Chuck Lever
>
>
>

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Trond Myklebust]

> On Jan 9, 2023, at 10:07, Chuck Lever III <chuck.lever@oracle.com> wrote:
> 
> 
> 
>> On Jan 9, 2023, at 9:44 AM, Trond Myklebust <trondmy@hammerspace.com> wrote:
>> 
>> 
>> 
>>> On Jan 9, 2023, at 03:42, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>> 
>>> On 09/01/2023 09:14, Krzysztof Kozlowski wrote:
>>>> On 08/01/2023 18:09, Trond Myklebust wrote:
>>>>> Hi Krzysztof,
>>>>> 
>>>>>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>>>>> 
>>>>>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
>>>>>> 
>>>>>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>>>>>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>>>>>>> when mounting NFS root on NFSv4 client:
>>>>>>> 
>>>>>>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
>>>>>>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>>>> argument
>>>>>>> [   26.199478] 8<--- cut here ---
>>>>>>> [   26.201366] Unable to handle kernel NULL pointer dereference at
>>>>>>> virtual address 00000004
>>>>>>> ...
>>>>>>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
>>>>>>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
>>>>>>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>>>> 
>>>>>>> Full OOPS attached. Full log available here:
>>>>>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
>>>>>>> 
>>>>>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
>>>>>>> not the cause, but rather making it default caused the regression.
>>>>>>> 
>>>>>>> I did not make the bisect yet which commit introduced it, if every
>>>>>>> config includes NFS_V4_2_READ_PLUS.
>>>>>> 
>>>>>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
>>>>>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
>>>>>> 
>>>>>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
>>>>>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>>> Date:   Thu Jul 21 14:21:34 2022 -0400
>>>>>> 
>>>>>> NFS: Replace the READ_PLUS decoding code
>>>>>> 
>>>>>> We now take a 2-step process that allows us to place data and hole
>>>>>> segments directly at their final position in the xdr_stream without
>>>>>> needing to do a bunch of redundant copies to expand holes. Due to the
>>>>>> variable lengths of each segment, the xdr metadata might cross page
>>>>>> boundaries which I account for by setting a small scratch buffer so
>>>>>> xdr_inline_decode() won't fail.
>>>>>> 
>>>>>> Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>>> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
>>>>>> 
>>>>>> With a trace:
>>>>>> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
>>>>>> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>>> argument
>>>>>> [   25.986237] random: crng init done
>>>>>> [   26.264564] 8<--- cut here ---
>>>>>> [   26.266823] Unable to handle kernel NULL pointer dereference at
>>>>>> virtual address 00000fe8
>>>>>> ...
>>>>>> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>>> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
>>>>>> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
>>>>>> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
>>>>>> [   26.620030]  process_one_work from worker_thread+0x54/0x518
>>>>>> [   26.625570]  worker_thread from kthread+0xf4/0x128
>>>>>> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
>>>>>> 
>>>>> 
>>>>> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?
>>>> 
>>>> I would say that buggy server should not cause NULL pointer dereferences
>>>> on the client. Otherwise this is a perfect recipe for a rogue server in
>>>> the network to start crashing clients and running exploits... Imagine a
>>>> compromised machine (through some other means) in a local company
>>>> network running now a server with NFS share "HR salary data" or "HR
>>>> planned layoffs", where unsuspected people in that network access it
>>>> leading to exploit of NFS code on their side...
>>>> 
>>>> Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH
>>>> 
>>>> Which points that it is not latest stable, so anyway I need to update.
>>> 
>>> I updated the kernel to 5.15.84-3-rpi-ARCH which is pretty close to
>>> latest stable and I can reproduce the issue. Therefore:
>>> 1. It is visible on two stable (one new, one old) kernels on the server,
>>> 2. Buggy or rogue server should not cause NULL pointer on remote devices...
>>> 
>> 
>> The nfsd READ_PLUS code is borked up and until 6.2-rc1. I thought we had a server option to disable that code, but it seems as if that is not the case.
>> Chuck + Anna, can we please send a stable patch to rip out that code altogether from all the older versions? If we want to enable READ_PLUS by default on the client, then we cannot do so if the majority of NFSv4.2 servers out there are running a borked implementation.
>> 
>> I do agree that we cannot allow buggy
> 
> or malicious, or non-Linux,
> 
> 
>> servers to cause memory corruption in the client code, so I propose that we revert the Kconfig default setting change again until both the client code and the legacy servers have been fixed.
> 
> I stand ready to receive and apply server-side fixes, as you suggested.
> 
> However IMO it would be most responsible to go straight for a client code fix. The Kconfig setting is a red herring (as delicious as that might sound). Any thoughts about how difficult that fix might be?
> 

A client fix is necessary, but not sufficient. The older server READ_PLUS code is slower than ordinary read for several filesystems, since it relies upon the (often poor) performance of lseek(SEEK_HOLE). Leaving legacy servers as is, would therefore still cause regressions for clients that default to trying READ_PLUS, even if the code is fixed to not Oops.

As for the client fix, I’m leaving that to Anna, since she owns the READ_PLUS feature. I stand ready to revert the Kconfig setting if these fixes cannot be made in the 6.2 time frame.

_________________________________
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Chuck Lever III]

> On Jan 9, 2023, at 10:38 AM, Trond Myklebust <trondmy@hammerspace.com> wrote:
> 
> 
> 
>> On Jan 9, 2023, at 10:07, Chuck Lever III <chuck.lever@oracle.com> wrote:
>> 
>> 
>> 
>>> On Jan 9, 2023, at 9:44 AM, Trond Myklebust <trondmy@hammerspace.com> wrote:
>>> 
>>> 
>>> 
>>>> On Jan 9, 2023, at 03:42, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>>> 
>>>> On 09/01/2023 09:14, Krzysztof Kozlowski wrote:
>>>>> On 08/01/2023 18:09, Trond Myklebust wrote:
>>>>>> Hi Krzysztof,
>>>>>> 
>>>>>>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>>>>>> 
>>>>>>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
>>>>>>> 
>>>>>>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>>>>>>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>>>>>>>> when mounting NFS root on NFSv4 client:
>>>>>>>> 
>>>>>>>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
>>>>>>>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>>>>> argument
>>>>>>>> [   26.199478] 8<--- cut here ---
>>>>>>>> [   26.201366] Unable to handle kernel NULL pointer dereference at
>>>>>>>> virtual address 00000004
>>>>>>>> ...
>>>>>>>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
>>>>>>>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
>>>>>>>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>>>>> 
>>>>>>>> Full OOPS attached. Full log available here:
>>>>>>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
>>>>>>>> 
>>>>>>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
>>>>>>>> not the cause, but rather making it default caused the regression.
>>>>>>>> 
>>>>>>>> I did not make the bisect yet which commit introduced it, if every
>>>>>>>> config includes NFS_V4_2_READ_PLUS.
>>>>>>> 
>>>>>>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
>>>>>>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
>>>>>>> 
>>>>>>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
>>>>>>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>>>> Date:   Thu Jul 21 14:21:34 2022 -0400
>>>>>>> 
>>>>>>> NFS: Replace the READ_PLUS decoding code
>>>>>>> 
>>>>>>> We now take a 2-step process that allows us to place data and hole
>>>>>>> segments directly at their final position in the xdr_stream without
>>>>>>> needing to do a bunch of redundant copies to expand holes. Due to the
>>>>>>> variable lengths of each segment, the xdr metadata might cross page
>>>>>>> boundaries which I account for by setting a small scratch buffer so
>>>>>>> xdr_inline_decode() won't fail.
>>>>>>> 
>>>>>>> Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>>>> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
>>>>>>> 
>>>>>>> With a trace:
>>>>>>> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
>>>>>>> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>>>> argument
>>>>>>> [   25.986237] random: crng init done
>>>>>>> [   26.264564] 8<--- cut here ---
>>>>>>> [   26.266823] Unable to handle kernel NULL pointer dereference at
>>>>>>> virtual address 00000fe8
>>>>>>> ...
>>>>>>> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>>>> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
>>>>>>> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
>>>>>>> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
>>>>>>> [   26.620030]  process_one_work from worker_thread+0x54/0x518
>>>>>>> [   26.625570]  worker_thread from kthread+0xf4/0x128
>>>>>>> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
>>>>>>> 
>>>>>> 
>>>>>> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?
>>>>> 
>>>>> I would say that buggy server should not cause NULL pointer dereferences
>>>>> on the client. Otherwise this is a perfect recipe for a rogue server in
>>>>> the network to start crashing clients and running exploits... Imagine a
>>>>> compromised machine (through some other means) in a local company
>>>>> network running now a server with NFS share "HR salary data" or "HR
>>>>> planned layoffs", where unsuspected people in that network access it
>>>>> leading to exploit of NFS code on their side...
>>>>> 
>>>>> Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH
>>>>> 
>>>>> Which points that it is not latest stable, so anyway I need to update.
>>>> 
>>>> I updated the kernel to 5.15.84-3-rpi-ARCH which is pretty close to
>>>> latest stable and I can reproduce the issue. Therefore:
>>>> 1. It is visible on two stable (one new, one old) kernels on the server,
>>>> 2. Buggy or rogue server should not cause NULL pointer on remote devices...
>>>> 
>>> 
>>> The nfsd READ_PLUS code is borked up and until 6.2-rc1. I thought we had a server option to disable that code, but it seems as if that is not the case.
>>> Chuck + Anna, can we please send a stable patch to rip out that code altogether from all the older versions? If we want to enable READ_PLUS by default on the client, then we cannot do so if the majority of NFSv4.2 servers out there are running a borked implementation.
>>> 
>>> I do agree that we cannot allow buggy
>> 
>> or malicious, or non-Linux,
>> 
>> 
>>> servers to cause memory corruption in the client code, so I propose that we revert the Kconfig default setting change again until both the client code and the legacy servers have been fixed.
>> 
>> I stand ready to receive and apply server-side fixes, as you suggested.
>> 
>> However IMO it would be most responsible to go straight for a client code fix. The Kconfig setting is a red herring (as delicious as that might sound). Any thoughts about how difficult that fix might be?
>> 
> 
> A client fix is necessary, but not sufficient.

No argument from me. There is a problem on both sides.


> The older server READ_PLUS code is slower than ordinary read for several filesystems, since it relies upon the (often poor) performance of lseek(SEEK_HOLE). Leaving legacy servers as is, would therefore still cause regressions for clients that default to trying READ_PLUS, even if the code is fixed to not Oops.

Backporting eeadcb757945 ("NFSD: Simplify READ_PLUS") to stable kernels would address both concerns.


--
Chuck Lever

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Chuck Lever III]

> On Jan 9, 2023, at 12:11 PM, Chuck Lever III <chuck.lever@oracle.com> wrote:
> 
> 
> 
>> On Jan 9, 2023, at 10:38 AM, Trond Myklebust <trondmy@hammerspace.com> wrote:
>> 
>> 
>> 
>>> On Jan 9, 2023, at 10:07, Chuck Lever III <chuck.lever@oracle.com> wrote:
>>> 
>>> 
>>> 
>>>> On Jan 9, 2023, at 9:44 AM, Trond Myklebust <trondmy@hammerspace.com> wrote:
>>>> 
>>>> 
>>>> 
>>>>> On Jan 9, 2023, at 03:42, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>>>> 
>>>>> On 09/01/2023 09:14, Krzysztof Kozlowski wrote:
>>>>>> On 08/01/2023 18:09, Trond Myklebust wrote:
>>>>>>> Hi Krzysztof,
>>>>>>> 
>>>>>>>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>>>>>>> 
>>>>>>>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
>>>>>>>> 
>>>>>>>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>>>>>>>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>>>>>>>>> when mounting NFS root on NFSv4 client:
>>>>>>>>> 
>>>>>>>>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
>>>>>>>>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>>>>>> argument
>>>>>>>>> [   26.199478] 8<--- cut here ---
>>>>>>>>> [   26.201366] Unable to handle kernel NULL pointer dereference at
>>>>>>>>> virtual address 00000004
>>>>>>>>> ...
>>>>>>>>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
>>>>>>>>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
>>>>>>>>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>>>>>> 
>>>>>>>>> Full OOPS attached. Full log available here:
>>>>>>>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
>>>>>>>>> 
>>>>>>>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
>>>>>>>>> not the cause, but rather making it default caused the regression.
>>>>>>>>> 
>>>>>>>>> I did not make the bisect yet which commit introduced it, if every
>>>>>>>>> config includes NFS_V4_2_READ_PLUS.
>>>>>>>> 
>>>>>>>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
>>>>>>>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
>>>>>>>> 
>>>>>>>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
>>>>>>>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>>>>> Date:   Thu Jul 21 14:21:34 2022 -0400
>>>>>>>> 
>>>>>>>> NFS: Replace the READ_PLUS decoding code
>>>>>>>> 
>>>>>>>> We now take a 2-step process that allows us to place data and hole
>>>>>>>> segments directly at their final position in the xdr_stream without
>>>>>>>> needing to do a bunch of redundant copies to expand holes. Due to the
>>>>>>>> variable lengths of each segment, the xdr metadata might cross page
>>>>>>>> boundaries which I account for by setting a small scratch buffer so
>>>>>>>> xdr_inline_decode() won't fail.
>>>>>>>> 
>>>>>>>> Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>>>>> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
>>>>>>>> 
>>>>>>>> With a trace:
>>>>>>>> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
>>>>>>>> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>>>>> argument
>>>>>>>> [   25.986237] random: crng init done
>>>>>>>> [   26.264564] 8<--- cut here ---
>>>>>>>> [   26.266823] Unable to handle kernel NULL pointer dereference at
>>>>>>>> virtual address 00000fe8
>>>>>>>> ...
>>>>>>>> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>>>>> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
>>>>>>>> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
>>>>>>>> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
>>>>>>>> [   26.620030]  process_one_work from worker_thread+0x54/0x518
>>>>>>>> [   26.625570]  worker_thread from kthread+0xf4/0x128
>>>>>>>> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
>>>>>>>> 
>>>>>>> 
>>>>>>> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?
>>>>>> 
>>>>>> I would say that buggy server should not cause NULL pointer dereferences
>>>>>> on the client. Otherwise this is a perfect recipe for a rogue server in
>>>>>> the network to start crashing clients and running exploits... Imagine a
>>>>>> compromised machine (through some other means) in a local company
>>>>>> network running now a server with NFS share "HR salary data" or "HR
>>>>>> planned layoffs", where unsuspected people in that network access it
>>>>>> leading to exploit of NFS code on their side...
>>>>>> 
>>>>>> Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH
>>>>>> 
>>>>>> Which points that it is not latest stable, so anyway I need to update.
>>>>> 
>>>>> I updated the kernel to 5.15.84-3-rpi-ARCH which is pretty close to
>>>>> latest stable and I can reproduce the issue. Therefore:
>>>>> 1. It is visible on two stable (one new, one old) kernels on the server,
>>>>> 2. Buggy or rogue server should not cause NULL pointer on remote devices...
>>>>> 
>>>> 
>>>> The nfsd READ_PLUS code is borked up and until 6.2-rc1. I thought we had a server option to disable that code, but it seems as if that is not the case.
>>>> Chuck + Anna, can we please send a stable patch to rip out that code altogether from all the older versions? If we want to enable READ_PLUS by default on the client, then we cannot do so if the majority of NFSv4.2 servers out there are running a borked implementation.
>>>> 
>>>> I do agree that we cannot allow buggy
>>> 
>>> or malicious, or non-Linux,
>>> 
>>> 
>>>> servers to cause memory corruption in the client code, so I propose that we revert the Kconfig default setting change again until both the client code and the legacy servers have been fixed.
>>> 
>>> I stand ready to receive and apply server-side fixes, as you suggested.
>>> 
>>> However IMO it would be most responsible to go straight for a client code fix. The Kconfig setting is a red herring (as delicious as that might sound). Any thoughts about how difficult that fix might be?
>>> 
>> 
>> A client fix is necessary, but not sufficient.
> 
> No argument from me. There is a problem on both sides.
> 
> 
>> The older server READ_PLUS code is slower than ordinary read for several filesystems, since it relies upon the (often poor) performance of lseek(SEEK_HOLE). Leaving legacy servers as is, would therefore still cause regressions for clients that default to trying READ_PLUS, even if the code is fixed to not Oops.
> 
> Backporting eeadcb757945 ("NFSD: Simplify READ_PLUS") to stable kernels would address both concerns.

And by "both concerns" I mean it addresses the possible performance
regression and the server's malformed RPC reply.

The client code still needs to be fixed.

--
Chuck Lever

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Chuck Lever III]

> On Jan 9, 2023, at 10:26 AM, Anna Schumaker <schumaker.anna@gmail.com> wrote:
> 
> On Mon, Jan 9, 2023 at 10:12 AM Chuck Lever III <chuck.lever@oracle.com> wrote:
>> 
>> 
>> 
>>> On Jan 9, 2023, at 9:44 AM, Trond Myklebust <trondmy@hammerspace.com> wrote:
>>> 
>>> 
>>> 
>>>> On Jan 9, 2023, at 03:42, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>>> 
>>>> On 09/01/2023 09:14, Krzysztof Kozlowski wrote:
>>>>> On 08/01/2023 18:09, Trond Myklebust wrote:
>>>>>> Hi Krzysztof,
>>>>>> 
>>>>>>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>>>>>> 
>>>>>>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
>>>>>>> 
>>>>>>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>>>>>>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>>>>>>>> when mounting NFS root on NFSv4 client:
>>>>>>>> 
>>>>>>>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
>>>>>>>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>>>>> argument
>>>>>>>> [   26.199478] 8<--- cut here ---
>>>>>>>> [   26.201366] Unable to handle kernel NULL pointer dereference at
>>>>>>>> virtual address 00000004
>>>>>>>> ...
>>>>>>>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
>>>>>>>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
>>>>>>>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>>>>> 
>>>>>>>> Full OOPS attached. Full log available here:
>>>>>>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
>>>>>>>> 
>>>>>>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
>>>>>>>> not the cause, but rather making it default caused the regression.
>>>>>>>> 
>>>>>>>> I did not make the bisect yet which commit introduced it, if every
>>>>>>>> config includes NFS_V4_2_READ_PLUS.
>>>>>>> 
>>>>>>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
>>>>>>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
>>>>>>> 
>>>>>>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
>>>>>>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>>>> Date:   Thu Jul 21 14:21:34 2022 -0400
>>>>>>> 
>>>>>>> NFS: Replace the READ_PLUS decoding code
>>>>>>> 
>>>>>>> We now take a 2-step process that allows us to place data and hole
>>>>>>> segments directly at their final position in the xdr_stream without
>>>>>>> needing to do a bunch of redundant copies to expand holes. Due to the
>>>>>>> variable lengths of each segment, the xdr metadata might cross page
>>>>>>> boundaries which I account for by setting a small scratch buffer so
>>>>>>> xdr_inline_decode() won't fail.
>>>>>>> 
>>>>>>> Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>>>> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
>>>>>>> 
>>>>>>> With a trace:
>>>>>>> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
>>>>>>> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>>>> argument
>>>>>>> [   25.986237] random: crng init done
>>>>>>> [   26.264564] 8<--- cut here ---
>>>>>>> [   26.266823] Unable to handle kernel NULL pointer dereference at
>>>>>>> virtual address 00000fe8
>>>>>>> ...
>>>>>>> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>>>> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
>>>>>>> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
>>>>>>> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
>>>>>>> [   26.620030]  process_one_work from worker_thread+0x54/0x518
>>>>>>> [   26.625570]  worker_thread from kthread+0xf4/0x128
>>>>>>> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
>>>>>>> 
>>>>>> 
>>>>>> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?
>>>>> 
>>>>> I would say that buggy server should not cause NULL pointer dereferences
>>>>> on the client. Otherwise this is a perfect recipe for a rogue server in
>>>>> the network to start crashing clients and running exploits... Imagine a
>>>>> compromised machine (through some other means) in a local company
>>>>> network running now a server with NFS share "HR salary data" or "HR
>>>>> planned layoffs", where unsuspected people in that network access it
>>>>> leading to exploit of NFS code on their side...
>>>>> 
>>>>> Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH
>>>>> 
>>>>> Which points that it is not latest stable, so anyway I need to update.
>>>> 
>>>> I updated the kernel to 5.15.84-3-rpi-ARCH which is pretty close to
>>>> latest stable and I can reproduce the issue. Therefore:
>>>> 1. It is visible on two stable (one new, one old) kernels on the server,
>>>> 2. Buggy or rogue server should not cause NULL pointer on remote devices...
>>>> 
>>> 
>>> The nfsd READ_PLUS code is borked up and until 6.2-rc1. I thought we had a server option to disable that code, but it seems as if that is not the case.
>>> Chuck + Anna, can we please send a stable patch to rip out that code altogether from all the older versions? If we want to enable READ_PLUS by default on the client, then we cannot do so if the majority of NFSv4.2 servers out there are running a borked implementation.
>>> 
>>> I do agree that we cannot allow buggy
>> 
>> or malicious, or non-Linux,
>> 
>> 
>>> servers to cause memory corruption in the client code, so I propose that we revert the Kconfig default setting change again until both the client code and the legacy servers have been fixed.
>> 
>> I stand ready to receive and apply server-side fixes, as you suggested.
>> 
>> However IMO it would be most responsible to go straight for a client code fix. The Kconfig setting is a red herring (as delicious as that might sound). Any thoughts about how difficult that fix might be?
> 
> I'm wondering about how hard the fix might be as well, since it could
> be a legitimate bug or some error checking that I've overlooked. I've
> gotten as far as installing a 5.15 server in my testing setup, so
> we'll see if I'm able to reproduce the crash this morning.

All current long-term stable kernels have 72d78717c6d0 ("nfsd: Fixes for nfsd4_encode_read_plus_data()"), and so does v5.10.92. I'm not aware of
any other problem with READ_PLUS except for performance. Let me know (or
file a bug) if I'm missing something!

For the possible performance regression, backporting eeadcb757945
("NFSD: Simplify READ_PLUS") is the proposed stable fix.

* v6.1.3 - eeadcb757945 applies cleanly
* v6.0.18 - eeadcb757945 applies with offsets
* v5.15.86 - eeadcb757945 needs 1e37d0e5bda4, then applies with offsets
* v5.10.162 - would need a hand-written version of eeadcb757945
* v5.4.228 - does not implement READ_PLUS
* v4.19.269 - does not implement READ_PLUS
* v4.14.302 - does not implement READ_PLUS
* v4.9.337 - EOL

If someone volunteers to build and test, I would not object to
a request to stable@ to apply eeadcb757945 to 5.15, 6.0, and 6.1.


--
Chuck Lever

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Anna Schumaker]

Hi Krzysztof,

On Mon, Jan 9, 2023 at 3:46 AM Krzysztof Kozlowski
<krzysztof.kozlowski@linaro.org> wrote:
>
> On 09/01/2023 09:14, Krzysztof Kozlowski wrote:
> > On 08/01/2023 18:09, Trond Myklebust wrote:
> >> Hi Krzysztof,
> >>
> >>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
> >>>
> >>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
> >>>
> >>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
> >>>> Hi,
> >>>>
> >>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
> >>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
> >>>> when mounting NFS root on NFSv4 client:
> >>>>
> >>>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
> >>>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
> >>>> argument
> >>>> [   26.199478] 8<--- cut here ---
> >>>> [   26.201366] Unable to handle kernel NULL pointer dereference at
> >>>> virtual address 00000004
> >>>> ...
> >>>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
> >>>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
> >>>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
> >>>>
> >>>> Full OOPS attached. Full log available here:
> >>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
> >>>>
> >>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
> >>>> not the cause, but rather making it default caused the regression.
> >>>>
> >>>> I did not make the bisect yet which commit introduced it, if every
> >>>> config includes NFS_V4_2_READ_PLUS.
> >>>
> >>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
> >>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
> >>>
> >>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
> >>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
> >>> Date:   Thu Jul 21 14:21:34 2022 -0400
> >>>
> >>>    NFS: Replace the READ_PLUS decoding code
> >>>
> >>>    We now take a 2-step process that allows us to place data and hole
> >>>    segments directly at their final position in the xdr_stream without
> >>>    needing to do a bunch of redundant copies to expand holes. Due to the
> >>>    variable lengths of each segment, the xdr metadata might cross page
> >>>    boundaries which I account for by setting a small scratch buffer so
> >>>    xdr_inline_decode() won't fail.
> >>>
> >>>    Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
> >>>    Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> >>>
> >>> With a trace:
> >>> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
> >>> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
> >>> argument
> >>> [   25.986237] random: crng init done
> >>> [   26.264564] 8<--- cut here ---
> >>> [   26.266823] Unable to handle kernel NULL pointer dereference at
> >>> virtual address 00000fe8
> >>> ...
> >>> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
> >>> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
> >>> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
> >>> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
> >>> [   26.620030]  process_one_work from worker_thread+0x54/0x518
> >>> [   26.625570]  worker_thread from kthread+0xf4/0x128
> >>> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
> >>>
> >>
> >> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?
> >
> > I would say that buggy server should not cause NULL pointer dereferences
> > on the client. Otherwise this is a perfect recipe for a rogue server in
> > the network to start crashing clients and running exploits... Imagine a
> > compromised machine (through some other means) in a local company
> > network running now a server with NFS share "HR salary data" or "HR
> > planned layoffs", where unsuspected people in that network access it
> > leading to exploit of NFS code on their side...
> >
> > Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH
> >
> > Which points that it is not latest stable, so anyway I need to update.
>
> I updated the kernel to 5.15.84-3-rpi-ARCH which is pretty close to
> latest stable and I can reproduce the issue. Therefore:
> 1. It is visible on two stable (one new, one old) kernels on the server,
> 2. Buggy or rogue server should not cause NULL pointer on remote devices...

I'm still working on this issue, but I haven't been able to reproduce
it with my setup at all yet. I was hoping I could ask you a couple of
questions?

- Are both client and server run on a Raspberry Pi 3?
- Is there a specific workload that triggers the bug, or is it just
during boot when using nfsroot?
- Would it be possible to get a Wireshark network trace of the crash
(you'll have to run this on the server due to nfsroot).
- Can you share your export options from /etc/exports and the mount
options that the client uses?

Thanks,
Anna

>
> Best regards,
> Krzysztof
>

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

On 22/01/2023 23:25, Anna Schumaker wrote:
> Hi Krzysztof,
> 
> On Mon, Jan 9, 2023 at 3:46 AM Krzysztof Kozlowski
> <krzysztof.kozlowski@linaro.org> wrote:
>>
>> On 09/01/2023 09:14, Krzysztof Kozlowski wrote:
>>> On 08/01/2023 18:09, Trond Myklebust wrote:
>>>> Hi Krzysztof,
>>>>
>>>>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>>>>
>>>>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
>>>>>
>>>>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>>>>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>>>>>> when mounting NFS root on NFSv4 client:
>>>>>>
>>>>>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
>>>>>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>>> argument
>>>>>> [   26.199478] 8<--- cut here ---
>>>>>> [   26.201366] Unable to handle kernel NULL pointer dereference at
>>>>>> virtual address 00000004
>>>>>> ...
>>>>>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
>>>>>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
>>>>>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>>>
>>>>>> Full OOPS attached. Full log available here:
>>>>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
>>>>>>
>>>>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
>>>>>> not the cause, but rather making it default caused the regression.
>>>>>>
>>>>>> I did not make the bisect yet which commit introduced it, if every
>>>>>> config includes NFS_V4_2_READ_PLUS.
>>>>>
>>>>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
>>>>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
>>>>>
>>>>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
>>>>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>> Date:   Thu Jul 21 14:21:34 2022 -0400
>>>>>
>>>>>    NFS: Replace the READ_PLUS decoding code
>>>>>
>>>>>    We now take a 2-step process that allows us to place data and hole
>>>>>    segments directly at their final position in the xdr_stream without
>>>>>    needing to do a bunch of redundant copies to expand holes. Due to the
>>>>>    variable lengths of each segment, the xdr metadata might cross page
>>>>>    boundaries which I account for by setting a small scratch buffer so
>>>>>    xdr_inline_decode() won't fail.
>>>>>
>>>>>    Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>>    Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
>>>>>
>>>>> With a trace:
>>>>> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
>>>>> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>> argument
>>>>> [   25.986237] random: crng init done
>>>>> [   26.264564] 8<--- cut here ---
>>>>> [   26.266823] Unable to handle kernel NULL pointer dereference at
>>>>> virtual address 00000fe8
>>>>> ...
>>>>> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
>>>>> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
>>>>> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
>>>>> [   26.620030]  process_one_work from worker_thread+0x54/0x518
>>>>> [   26.625570]  worker_thread from kthread+0xf4/0x128
>>>>> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
>>>>>
>>>>
>>>> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?
>>>
>>> I would say that buggy server should not cause NULL pointer dereferences
>>> on the client. Otherwise this is a perfect recipe for a rogue server in
>>> the network to start crashing clients and running exploits... Imagine a
>>> compromised machine (through some other means) in a local company
>>> network running now a server with NFS share "HR salary data" or "HR
>>> planned layoffs", where unsuspected people in that network access it
>>> leading to exploit of NFS code on their side...
>>>
>>> Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH
>>>
>>> Which points that it is not latest stable, so anyway I need to update.
>>
>> I updated the kernel to 5.15.84-3-rpi-ARCH which is pretty close to
>> latest stable and I can reproduce the issue. Therefore:
>> 1. It is visible on two stable (one new, one old) kernels on the server,
>> 2. Buggy or rogue server should not cause NULL pointer on remote devices...
> 
> I'm still working on this issue, but I haven't been able to reproduce
> it with my setup at all yet. I was hoping I could ask you a couple of
> questions?
> 
> - Are both client and server run on a Raspberry Pi 3?

No, server is Rpi3, client is Odroid HC1.

> - Is there a specific workload that triggers the bug, or is it just
> during boot when using nfsroot?

No specific workload - mounting system with nfsroot. You have there full
logs as it is reproducible every time.

> - Would it be possible to get a Wireshark network trace of the crash
> (you'll have to run this on the server due to nfsroot).

I'll check.

> - Can you share your export options from /etc/exports and the mount
> options that the client uses?

exports are the same since like 5 years or more:

/srv/nfs/odroidhc1	192.168.1.0/24(rw,async,no_root_squash,no_subtree_check)
/srv/nfs/odroidhc1-home
192.168.1.0/24(rw,async,no_root_squash,no_subtree_check)


> 
> Thanks,
> Anna
> 
>>
>> Best regards,
>> Krzysztof
>>

Best regards,
Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Anna Schumaker]

Hi Krzysztof,

On Mon, Jan 23, 2023 at 2:58 AM Krzysztof Kozlowski
<krzysztof.kozlowski@linaro.org> wrote:
>
> On 22/01/2023 23:25, Anna Schumaker wrote:
> > Hi Krzysztof,
> >
> > On Mon, Jan 9, 2023 at 3:46 AM Krzysztof Kozlowski
> > <krzysztof.kozlowski@linaro.org> wrote:
> >>
> >> On 09/01/2023 09:14, Krzysztof Kozlowski wrote:
> >>> On 08/01/2023 18:09, Trond Myklebust wrote:
> >>>> Hi Krzysztof,
> >>>>
> >>>>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
> >>>>>
> >>>>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
> >>>>>
> >>>>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
> >>>>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
> >>>>>> when mounting NFS root on NFSv4 client:
> >>>>>>
> >>>>>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
> >>>>>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
> >>>>>> argument
> >>>>>> [   26.199478] 8<--- cut here ---
> >>>>>> [   26.201366] Unable to handle kernel NULL pointer dereference at
> >>>>>> virtual address 00000004
> >>>>>> ...
> >>>>>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
> >>>>>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
> >>>>>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
> >>>>>>
> >>>>>> Full OOPS attached. Full log available here:
> >>>>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
> >>>>>>
> >>>>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
> >>>>>> not the cause, but rather making it default caused the regression.
> >>>>>>
> >>>>>> I did not make the bisect yet which commit introduced it, if every
> >>>>>> config includes NFS_V4_2_READ_PLUS.
> >>>>>
> >>>>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
> >>>>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
> >>>>>
> >>>>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
> >>>>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
> >>>>> Date:   Thu Jul 21 14:21:34 2022 -0400
> >>>>>
> >>>>>    NFS: Replace the READ_PLUS decoding code
> >>>>>
> >>>>>    We now take a 2-step process that allows us to place data and hole
> >>>>>    segments directly at their final position in the xdr_stream without
> >>>>>    needing to do a bunch of redundant copies to expand holes. Due to the
> >>>>>    variable lengths of each segment, the xdr metadata might cross page
> >>>>>    boundaries which I account for by setting a small scratch buffer so
> >>>>>    xdr_inline_decode() won't fail.
> >>>>>
> >>>>>    Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
> >>>>>    Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> >>>>>
> >>>>> With a trace:
> >>>>> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
> >>>>> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
> >>>>> argument
> >>>>> [   25.986237] random: crng init done
> >>>>> [   26.264564] 8<--- cut here ---
> >>>>> [   26.266823] Unable to handle kernel NULL pointer dereference at
> >>>>> virtual address 00000fe8
> >>>>> ...
> >>>>> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
> >>>>> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
> >>>>> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
> >>>>> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
> >>>>> [   26.620030]  process_one_work from worker_thread+0x54/0x518
> >>>>> [   26.625570]  worker_thread from kthread+0xf4/0x128
> >>>>> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
> >>>>>
> >>>>
> >>>> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?
> >>>
> >>> I would say that buggy server should not cause NULL pointer dereferences
> >>> on the client. Otherwise this is a perfect recipe for a rogue server in
> >>> the network to start crashing clients and running exploits... Imagine a
> >>> compromised machine (through some other means) in a local company
> >>> network running now a server with NFS share "HR salary data" or "HR
> >>> planned layoffs", where unsuspected people in that network access it
> >>> leading to exploit of NFS code on their side...
> >>>
> >>> Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH
> >>>
> >>> Which points that it is not latest stable, so anyway I need to update.
> >>
> >> I updated the kernel to 5.15.84-3-rpi-ARCH which is pretty close to
> >> latest stable and I can reproduce the issue. Therefore:
> >> 1. It is visible on two stable (one new, one old) kernels on the server,
> >> 2. Buggy or rogue server should not cause NULL pointer on remote devices...
> >
> > I'm still working on this issue, but I haven't been able to reproduce
> > it with my setup at all yet. I was hoping I could ask you a couple of
> > questions?
> >
> > - Are both client and server run on a Raspberry Pi 3?
>
> No, server is Rpi3, client is Odroid HC1.
>
> > - Is there a specific workload that triggers the bug, or is it just
> > during boot when using nfsroot?
>
> No specific workload - mounting system with nfsroot. You have there full
> logs as it is reproducible every time.
>
> > - Would it be possible to get a Wireshark network trace of the crash
> > (you'll have to run this on the server due to nfsroot).
>
> I'll check.

Any luck getting the wireshark trace? I don't have access to the
Odroid HC1 board, so all my attempts at reproducing the problem have
been with qemu & libvirt, which doesn't seem to be hitting this issue.

I was also wondering if it would be possible to turn on KASAN in your
kernel, which should give us a little more info?

Thanks,
Anna

>
> > - Can you share your export options from /etc/exports and the mount
> > options that the client uses?
>
> exports are the same since like 5 years or more:
>
> /srv/nfs/odroidhc1      192.168.1.0/24(rw,async,no_root_squash,no_subtree_check)
> /srv/nfs/odroidhc1-home
> 192.168.1.0/24(rw,async,no_root_squash,no_subtree_check)
>
>
> >
> > Thanks,
> > Anna
> >
> >>
> >> Best regards,
> >> Krzysztof
> >>
>
> Best regards,
> Krzysztof
>

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

On 09/02/2023 19:22, Anna Schumaker wrote:
> Hi Krzysztof,
> 
> On Mon, Jan 23, 2023 at 2:58 AM Krzysztof Kozlowski
> <krzysztof.kozlowski@linaro.org> wrote:
>>
>> On 22/01/2023 23:25, Anna Schumaker wrote:
>>> Hi Krzysztof,
>>>
>>> On Mon, Jan 9, 2023 at 3:46 AM Krzysztof Kozlowski
>>> <krzysztof.kozlowski@linaro.org> wrote:
>>>>
>>>> On 09/01/2023 09:14, Krzysztof Kozlowski wrote:
>>>>> On 08/01/2023 18:09, Trond Myklebust wrote:
>>>>>> Hi Krzysztof,
>>>>>>
>>>>>>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>>>>>>
>>>>>>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
>>>>>>>
>>>>>>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>>>>>>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>>>>>>>> when mounting NFS root on NFSv4 client:
>>>>>>>>
>>>>>>>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
>>>>>>>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>>>>> argument
>>>>>>>> [   26.199478] 8<--- cut here ---
>>>>>>>> [   26.201366] Unable to handle kernel NULL pointer dereference at
>>>>>>>> virtual address 00000004
>>>>>>>> ...
>>>>>>>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
>>>>>>>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
>>>>>>>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>>>>>
>>>>>>>> Full OOPS attached. Full log available here:
>>>>>>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
>>>>>>>>
>>>>>>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
>>>>>>>> not the cause, but rather making it default caused the regression.
>>>>>>>>
>>>>>>>> I did not make the bisect yet which commit introduced it, if every
>>>>>>>> config includes NFS_V4_2_READ_PLUS.
>>>>>>>
>>>>>>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
>>>>>>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
>>>>>>>
>>>>>>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
>>>>>>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>>>> Date:   Thu Jul 21 14:21:34 2022 -0400
>>>>>>>
>>>>>>>    NFS: Replace the READ_PLUS decoding code
>>>>>>>
>>>>>>>    We now take a 2-step process that allows us to place data and hole
>>>>>>>    segments directly at their final position in the xdr_stream without
>>>>>>>    needing to do a bunch of redundant copies to expand holes. Due to the
>>>>>>>    variable lengths of each segment, the xdr metadata might cross page
>>>>>>>    boundaries which I account for by setting a small scratch buffer so
>>>>>>>    xdr_inline_decode() won't fail.
>>>>>>>
>>>>>>>    Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>>>>    Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
>>>>>>>
>>>>>>> With a trace:
>>>>>>> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
>>>>>>> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>>>> argument
>>>>>>> [   25.986237] random: crng init done
>>>>>>> [   26.264564] 8<--- cut here ---
>>>>>>> [   26.266823] Unable to handle kernel NULL pointer dereference at
>>>>>>> virtual address 00000fe8
>>>>>>> ...
>>>>>>> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>>>> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
>>>>>>> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
>>>>>>> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
>>>>>>> [   26.620030]  process_one_work from worker_thread+0x54/0x518
>>>>>>> [   26.625570]  worker_thread from kthread+0xf4/0x128
>>>>>>> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
>>>>>>>
>>>>>>
>>>>>> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?
>>>>>
>>>>> I would say that buggy server should not cause NULL pointer dereferences
>>>>> on the client. Otherwise this is a perfect recipe for a rogue server in
>>>>> the network to start crashing clients and running exploits... Imagine a
>>>>> compromised machine (through some other means) in a local company
>>>>> network running now a server with NFS share "HR salary data" or "HR
>>>>> planned layoffs", where unsuspected people in that network access it
>>>>> leading to exploit of NFS code on their side...
>>>>>
>>>>> Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH
>>>>>
>>>>> Which points that it is not latest stable, so anyway I need to update.
>>>>
>>>> I updated the kernel to 5.15.84-3-rpi-ARCH which is pretty close to
>>>> latest stable and I can reproduce the issue. Therefore:
>>>> 1. It is visible on two stable (one new, one old) kernels on the server,
>>>> 2. Buggy or rogue server should not cause NULL pointer on remote devices...
>>>
>>> I'm still working on this issue, but I haven't been able to reproduce
>>> it with my setup at all yet. I was hoping I could ask you a couple of
>>> questions?
>>>
>>> - Are both client and server run on a Raspberry Pi 3?
>>
>> No, server is Rpi3, client is Odroid HC1.
>>
>>> - Is there a specific workload that triggers the bug, or is it just
>>> during boot when using nfsroot?
>>
>> No specific workload - mounting system with nfsroot. You have there full
>> logs as it is reproducible every time.
>>
>>> - Would it be possible to get a Wireshark network trace of the crash
>>> (you'll have to run this on the server due to nfsroot).
>>
>> I'll check.
> 
> Any luck getting the wireshark trace? I don't have access to the
> Odroid HC1 board, so all my attempts at reproducing the problem have
> been with qemu & libvirt, which doesn't seem to be hitting this issue.

I'll send pcap dump off list. Failure is in similar place:
[   23.876714] systemd[1]: Hostname set to <odroidhc1>.
[   24.061568] systemd[1]: bpf-lsm: BPF LSM hook not enabled in the
kernel, BPF LSM not supported
[   24.078774] memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL,
pid=1 'systemd'
[   24.220531] 8<--- cut here ---
[   24.222333] Unable to handle kernel NULL pointer dereference at
virtual address 00000004 when read


> 
> I was also wondering if it would be possible to turn on KASAN in your
> kernel, which should give us a little more info?

I'll try with KASAN.

Best regards,
Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

On 10/02/2023 09:41, Krzysztof Kozlowski wrote:
> 
> 
>>
>> I was also wondering if it would be possible to turn on KASAN in your
>> kernel, which should give us a little more info?
> 
> I'll try with KASAN.

Not much from the KASAN, except that kernel
continues to boot and runs:

[   44.722846] vdd_vmem: disabling
[   44.793465] systemd[1]: Hostname set to <odroidhc1>.
[   45.357929] systemd[1]: bpf-lsm: BPF LSM hook not enabled in the kernel, BPF LSM not supported
[   45.980476] ==================================================================
[   45.986372] BUG: KASAN: null-ptr-deref in xdr_inline_decode+0x140/0x200
[   45.992929] Read of size 4092 at addr 00000004 by task kworker/u16:3/71
[   45.999513] 
[   46.000940] CPU: 6 PID: 71 Comm: kworker/u16:3 Not tainted 6.2.0-rc7-00018-g0983f6bf2bfc #222
[   46.009504] Hardware name: Samsung Exynos (Flattened Device Tree)
[   46.015542] Workqueue: rpciod rpc_async_schedule
[   46.020123]  unwind_backtrace from show_stack+0x10/0x14
[   46.025323]  show_stack from dump_stack_lvl+0x58/0x70
[   46.030301]  dump_stack_lvl from kasan_report+0xa8/0xe0
[   46.035501]  kasan_report from kasan_check_range+0x94/0x1a0
[   46.041048]  kasan_check_range from memcpy+0x28/0x68
[   46.045985]  memcpy from xdr_inline_decode+0x140/0x200
[   46.051098]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x298/0x5b8
[   46.057602]  nfs4_xdr_dec_read_plus from call_decode+0x39c/0x530
[   46.063581]  call_decode from __rpc_execute+0x1f4/0xc5c
[   46.068776]  __rpc_execute from rpc_async_schedule+0x2c/0x4c
[   46.074411]  rpc_async_schedule from process_one_work+0x51c/0xc44
[   46.080478]  process_one_work from worker_thread+0x9c/0x7c0
[   46.086022]  worker_thread from kthread+0x16c/0x1b8
[   46.090872]  kthread from ret_from_fork+0x14/0x2c
[   46.095550] Exception stack(0xf0ba3fb0 to 0xf0ba3ff8)
[   46.100580] 3fa0:                                     00000000 00000000 00000000 00000000
[   46.108740] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   46.116885] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[   46.123461] ==================================================================
[   46.130774] Disabling lock debugging due to kernel taint
[   50.848579] systemd[1]: Queued start job for default target Graphical Interface.
[   50.877068] systemd[1]: Created slice Slice /system/getty.


Decoded stacktrace is (this is master branch):

[   46.020123] unwind_backtrace from show_stack (arch/arm/kernel/traps.c:258) 
[   46.025323] show_stack from dump_stack_lvl (lib/dump_stack.c:107 (discriminator 4)) 
[   46.030301] dump_stack_lvl from kasan_report (mm/kasan/report.c:184 mm/kasan/report.c:519) 
[   46.035501] kasan_report from kasan_check_range (mm/kasan/generic.c:173 mm/kasan/generic.c:189) 
[   46.041048] kasan_check_range from memcpy (mm/kasan/shadow.c:65) 
[   46.045985] memcpy from xdr_inline_decode (net/sunrpc/xdr.c:1419 net/sunrpc/xdr.c:1454) 
[   46.051098] xdr_inline_decode from nfs4_xdr_dec_read_plus (fs/nfs/nfs42xdr.c:1063 fs/nfs/nfs42xdr.c:1149 fs/nfs/nfs42xdr.c:1360 fs/nfs/nfs42xdr.c:1343) 
[   46.057602] nfs4_xdr_dec_read_plus from call_decode (net/sunrpc/clnt.c:2595) 
[   46.063581] call_decode from __rpc_execute (include/asm-generic/bitops/generic-non-atomic.h:128 net/sunrpc/sched.c:954) 
[   46.068776] __rpc_execute from rpc_async_schedule (include/linux/sched/mm.h:336 net/sunrpc/sched.c:1035) 
[   46.074411] rpc_async_schedule from process_one_work (kernel/workqueue.c:2294) 
[   46.080478] process_one_work from worker_thread (include/linux/list.h:292 kernel/workqueue.c:2437) 
[   46.086022] worker_thread from kthread (kernel/kthread.c:376) 
[   46.090872] kthread from ret_from_fork (arch/arm/kernel/entry-common.S:149) 


Best regards,
Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Anna Schumaker]

Hi Krzysztof,

On Fri, Feb 10, 2023 at 3:53 AM Krzysztof Kozlowski
<krzysztof.kozlowski@linaro.org> wrote:
>
> On 10/02/2023 09:41, Krzysztof Kozlowski wrote:
> >
> >
> >>
> >> I was also wondering if it would be possible to turn on KASAN in your
> >> kernel, which should give us a little more info?
> >
> > I'll try with KASAN.
>
> Not much from the KASAN, except that kernel
> continues to boot and runs:
>
> [   44.722846] vdd_vmem: disabling
> [   44.793465] systemd[1]: Hostname set to <odroidhc1>.
> [   45.357929] systemd[1]: bpf-lsm: BPF LSM hook not enabled in the kernel, BPF LSM not supported
> [   45.980476] ==================================================================
> [   45.986372] BUG: KASAN: null-ptr-deref in xdr_inline_decode+0x140/0x200
> [   45.992929] Read of size 4092 at addr 00000004 by task kworker/u16:3/71
> [   45.999513]
> [   46.000940] CPU: 6 PID: 71 Comm: kworker/u16:3 Not tainted 6.2.0-rc7-00018-g0983f6bf2bfc #222
> [   46.009504] Hardware name: Samsung Exynos (Flattened Device Tree)
> [   46.015542] Workqueue: rpciod rpc_async_schedule
> [   46.020123]  unwind_backtrace from show_stack+0x10/0x14
> [   46.025323]  show_stack from dump_stack_lvl+0x58/0x70
> [   46.030301]  dump_stack_lvl from kasan_report+0xa8/0xe0
> [   46.035501]  kasan_report from kasan_check_range+0x94/0x1a0
> [   46.041048]  kasan_check_range from memcpy+0x28/0x68
> [   46.045985]  memcpy from xdr_inline_decode+0x140/0x200
> [   46.051098]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x298/0x5b8
> [   46.057602]  nfs4_xdr_dec_read_plus from call_decode+0x39c/0x530
> [   46.063581]  call_decode from __rpc_execute+0x1f4/0xc5c
> [   46.068776]  __rpc_execute from rpc_async_schedule+0x2c/0x4c
> [   46.074411]  rpc_async_schedule from process_one_work+0x51c/0xc44
> [   46.080478]  process_one_work from worker_thread+0x9c/0x7c0
> [   46.086022]  worker_thread from kthread+0x16c/0x1b8
> [   46.090872]  kthread from ret_from_fork+0x14/0x2c
> [   46.095550] Exception stack(0xf0ba3fb0 to 0xf0ba3ff8)
> [   46.100580] 3fa0:                                     00000000 00000000 00000000 00000000
> [   46.108740] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> [   46.116885] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
> [   46.123461] ==================================================================
> [   46.130774] Disabling lock debugging due to kernel taint
> [   50.848579] systemd[1]: Queued start job for default target Graphical Interface.
> [   50.877068] systemd[1]: Created slice Slice /system/getty.
>
>
> Decoded stacktrace is (this is master branch):
>
> [   46.020123] unwind_backtrace from show_stack (arch/arm/kernel/traps.c:258)
> [   46.025323] show_stack from dump_stack_lvl (lib/dump_stack.c:107 (discriminator 4))
> [   46.030301] dump_stack_lvl from kasan_report (mm/kasan/report.c:184 mm/kasan/report.c:519)
> [   46.035501] kasan_report from kasan_check_range (mm/kasan/generic.c:173 mm/kasan/generic.c:189)
> [   46.041048] kasan_check_range from memcpy (mm/kasan/shadow.c:65)
> [   46.045985] memcpy from xdr_inline_decode (net/sunrpc/xdr.c:1419 net/sunrpc/xdr.c:1454)

Actually, this part is really useful. net/sunrpc/xdr.c:1419 points to
the memcpy in xdr_copy_to_scratch(), which has me wondering if I'm
incorrectly setting up the xdr scratch buffer that READ_PLUS uses for
decoding. Can you try this patch and let me know if it helps?

From ac2d6c501dbcdb306480edaee625b5496f1fb4f5 Mon Sep 17 00:00:00 2001
From: Anna Schumaker <Anna.Schumaker@Netapp.com>
Date: Fri, 10 Feb 2023 15:50:22 -0500
Subject: [PATCH] NFSv4.2: Rework scratch handling for READ_PLUS

Instead of using a tiny scratch buffer, we should use a full scratch
page to match how other NFSv4 operations handle scratch data.

Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
---
 fs/nfs/nfs42xdr.c       |  4 ++--
 fs/nfs/nfs4proc.c       | 14 ++++++++++----
 include/linux/nfs_xdr.h |  1 +
 3 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index d80ee88ca996..702567d5b1db 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -1122,7 +1122,6 @@ static int decode_read_plus(struct xdr_stream
*xdr, struct nfs_pgio_res *res)
     uint32_t segments;
     struct read_plus_segment *segs;
     int status, i;
-    char scratch_buf[16];
     __be32 *p;

     status = decode_op_hdr(xdr, OP_READ_PLUS);
@@ -1143,7 +1142,6 @@ static int decode_read_plus(struct xdr_stream
*xdr, struct nfs_pgio_res *res)
     if (!segs)
         return -ENOMEM;

-    xdr_set_scratch_buffer(xdr, &scratch_buf, sizeof(scratch_buf));
     status = -EIO;
     for (i = 0; i < segments; i++) {
         status = decode_read_plus_segment(xdr, &segs[i]);
@@ -1348,6 +1346,8 @@ static int nfs4_xdr_dec_read_plus(struct rpc_rqst *rqstp,
     struct compound_hdr hdr;
     int status;

+    xdr_set_scratch_page(xdr, res->scratch);
+
     status = decode_compound_hdr(xdr, &hdr);
     if (status)
         goto out;
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 40d749f29ed3..5c589d0bd9e9 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -5440,6 +5440,8 @@ static bool nfs4_read_plus_not_supported(struct
rpc_task *task,

 static int nfs4_read_done(struct rpc_task *task, struct nfs_pgio_header *hdr)
 {
+    if (hdr->res.scratch)
+        __free_page(hdr->res.scratch);
     if (!nfs4_sequence_done(task, &hdr->res.seq_res))
         return -EAGAIN;
     if (nfs4_read_stateid_changed(task, &hdr->args))
@@ -5453,12 +5455,16 @@ static int nfs4_read_done(struct rpc_task
*task, struct nfs_pgio_header *hdr)
 }

 #if defined CONFIG_NFS_V4_2 && defined CONFIG_NFS_V4_2_READ_PLUS
-static void nfs42_read_plus_support(struct nfs_pgio_header *hdr,
+static bool nfs42_read_plus_support(struct nfs_pgio_header *hdr,
                     struct rpc_message *msg)
 {
     /* Note: We don't use READ_PLUS with pNFS yet */
-    if (nfs_server_capable(hdr->inode, NFS_CAP_READ_PLUS) && !hdr->ds_clp)
+    if (nfs_server_capable(hdr->inode, NFS_CAP_READ_PLUS) && !hdr->ds_clp) {
         msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ_PLUS];
+        hdr->res.scratch = alloc_page(GFP_KERNEL);
+        return hdr->res.scratch != NULL;
+    }
+    return false;
 }
 #else
 static void nfs42_read_plus_support(struct nfs_pgio_header *hdr,
@@ -5473,8 +5479,8 @@ static void nfs4_proc_read_setup(struct
nfs_pgio_header *hdr,
     hdr->timestamp   = jiffies;
     if (!hdr->pgio_done_cb)
         hdr->pgio_done_cb = nfs4_read_done_cb;
-    msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ];
-    nfs42_read_plus_support(hdr, msg);
+    if (!nfs42_read_plus_support(hdr, msg))
+        msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ];
     nfs4_init_sequence(&hdr->args.seq_args, &hdr->res.seq_res, 0, 0);
 }

diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index e86cf6642d21..6d821aaf0b1a 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -670,6 +670,7 @@ struct nfs_pgio_res {
         struct {
             unsigned int        replen;        /* used by read */
             int            eof;        /* used by read */
+            struct page        *scratch;    /* used by read */
         };
         struct {
             struct nfs_writeverf *    verf;        /* used by write */
-- 
2.39.1

Thanks,
Anna

> [   46.051098] xdr_inline_decode from nfs4_xdr_dec_read_plus (fs/nfs/nfs42xdr.c:1063 fs/nfs/nfs42xdr.c:1149 fs/nfs/nfs42xdr.c:1360 fs/nfs/nfs42xdr.c:1343)
> [   46.057602] nfs4_xdr_dec_read_plus from call_decode (net/sunrpc/clnt.c:2595)
> [   46.063581] call_decode from __rpc_execute (include/asm-generic/bitops/generic-non-atomic.h:128 net/sunrpc/sched.c:954)
> [   46.068776] __rpc_execute from rpc_async_schedule (include/linux/sched/mm.h:336 net/sunrpc/sched.c:1035)
> [   46.074411] rpc_async_schedule from process_one_work (kernel/workqueue.c:2294)
> [   46.080478] process_one_work from worker_thread (include/linux/list.h:292 kernel/workqueue.c:2437)
> [   46.086022] worker_thread from kthread (kernel/kthread.c:376)
> [   46.090872] kthread from ret_from_fork (arch/arm/kernel/entry-common.S:149)
>
>
> Best regards,
> Krzysztof
>

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

On 10/02/2023 21:55, Anna Schumaker wrote:
> Hi Krzysztof,
> 
> On Fri, Feb 10, 2023 at 3:53 AM Krzysztof Kozlowski
> <krzysztof.kozlowski@linaro.org> wrote:
>>
>> On 10/02/2023 09:41, Krzysztof Kozlowski wrote:
>>>
>>>
>>>>
>>>> I was also wondering if it would be possible to turn on KASAN in your
>>>> kernel, which should give us a little more info?
>>>
>>> I'll try with KASAN.
>>
>> Not much from the KASAN, except that kernel
>> continues to boot and runs:
>>
>> [   44.722846] vdd_vmem: disabling
>> [   44.793465] systemd[1]: Hostname set to <odroidhc1>.
>> [   45.357929] systemd[1]: bpf-lsm: BPF LSM hook not enabled in the kernel, BPF LSM not supported
>> [   45.980476] ==================================================================
>> [   45.986372] BUG: KASAN: null-ptr-deref in xdr_inline_decode+0x140/0x200
>> [   45.992929] Read of size 4092 at addr 00000004 by task kworker/u16:3/71
>> [   45.999513]
>> [   46.000940] CPU: 6 PID: 71 Comm: kworker/u16:3 Not tainted 6.2.0-rc7-00018-g0983f6bf2bfc #222
>> [   46.009504] Hardware name: Samsung Exynos (Flattened Device Tree)
>> [   46.015542] Workqueue: rpciod rpc_async_schedule
>> [   46.020123]  unwind_backtrace from show_stack+0x10/0x14
>> [   46.025323]  show_stack from dump_stack_lvl+0x58/0x70
>> [   46.030301]  dump_stack_lvl from kasan_report+0xa8/0xe0
>> [   46.035501]  kasan_report from kasan_check_range+0x94/0x1a0
>> [   46.041048]  kasan_check_range from memcpy+0x28/0x68
>> [   46.045985]  memcpy from xdr_inline_decode+0x140/0x200
>> [   46.051098]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x298/0x5b8
>> [   46.057602]  nfs4_xdr_dec_read_plus from call_decode+0x39c/0x530
>> [   46.063581]  call_decode from __rpc_execute+0x1f4/0xc5c
>> [   46.068776]  __rpc_execute from rpc_async_schedule+0x2c/0x4c
>> [   46.074411]  rpc_async_schedule from process_one_work+0x51c/0xc44
>> [   46.080478]  process_one_work from worker_thread+0x9c/0x7c0
>> [   46.086022]  worker_thread from kthread+0x16c/0x1b8
>> [   46.090872]  kthread from ret_from_fork+0x14/0x2c
>> [   46.095550] Exception stack(0xf0ba3fb0 to 0xf0ba3ff8)
>> [   46.100580] 3fa0:                                     00000000 00000000 00000000 00000000
>> [   46.108740] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
>> [   46.116885] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
>> [   46.123461] ==================================================================
>> [   46.130774] Disabling lock debugging due to kernel taint
>> [   50.848579] systemd[1]: Queued start job for default target Graphical Interface.
>> [   50.877068] systemd[1]: Created slice Slice /system/getty.
>>
>>
>> Decoded stacktrace is (this is master branch):
>>
>> [   46.020123] unwind_backtrace from show_stack (arch/arm/kernel/traps.c:258)
>> [   46.025323] show_stack from dump_stack_lvl (lib/dump_stack.c:107 (discriminator 4))
>> [   46.030301] dump_stack_lvl from kasan_report (mm/kasan/report.c:184 mm/kasan/report.c:519)
>> [   46.035501] kasan_report from kasan_check_range (mm/kasan/generic.c:173 mm/kasan/generic.c:189)
>> [   46.041048] kasan_check_range from memcpy (mm/kasan/shadow.c:65)
>> [   46.045985] memcpy from xdr_inline_decode (net/sunrpc/xdr.c:1419 net/sunrpc/xdr.c:1454)
> 
> Actually, this part is really useful. net/sunrpc/xdr.c:1419 points to
> the memcpy in xdr_copy_to_scratch(), which has me wondering if I'm
> incorrectly setting up the xdr scratch buffer that READ_PLUS uses for
> decoding. Can you try this patch and let me know if it helps?
> 
> From ac2d6c501dbcdb306480edaee625b5496f1fb4f5 Mon Sep 17 00:00:00 2001
> From: Anna Schumaker <Anna.Schumaker@Netapp.com>
> Date: Fri, 10 Feb 2023 15:50:22 -0500
> Subject: [PATCH] NFSv4.2: Rework scratch handling for READ_PLUS
> 

Patch is corrupted - maybe mail program reformatted it when sending:

Applying: NFSv4.2: Rework scratch handling for READ_PLUS
error: corrupt patch at line 12
Patch failed at 0001 NFSv4.2: Rework scratch handling for READ_PLUS


Best regards,
Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Anna Schumaker]

[-- Attachment #1: Type: text/plain, Size: 4464 bytes --]

On Sat, Feb 11, 2023 at 6:23 AM Krzysztof Kozlowski
<krzysztof.kozlowski@linaro.org> wrote:
>
> On 10/02/2023 21:55, Anna Schumaker wrote:
> > Hi Krzysztof,
> >
> > On Fri, Feb 10, 2023 at 3:53 AM Krzysztof Kozlowski
> > <krzysztof.kozlowski@linaro.org> wrote:
> >>
> >> On 10/02/2023 09:41, Krzysztof Kozlowski wrote:
> >>>
> >>>
> >>>>
> >>>> I was also wondering if it would be possible to turn on KASAN in your
> >>>> kernel, which should give us a little more info?
> >>>
> >>> I'll try with KASAN.
> >>
> >> Not much from the KASAN, except that kernel
> >> continues to boot and runs:
> >>
> >> [   44.722846] vdd_vmem: disabling
> >> [   44.793465] systemd[1]: Hostname set to <odroidhc1>.
> >> [   45.357929] systemd[1]: bpf-lsm: BPF LSM hook not enabled in the kernel, BPF LSM not supported
> >> [   45.980476] ==================================================================
> >> [   45.986372] BUG: KASAN: null-ptr-deref in xdr_inline_decode+0x140/0x200
> >> [   45.992929] Read of size 4092 at addr 00000004 by task kworker/u16:3/71
> >> [   45.999513]
> >> [   46.000940] CPU: 6 PID: 71 Comm: kworker/u16:3 Not tainted 6.2.0-rc7-00018-g0983f6bf2bfc #222
> >> [   46.009504] Hardware name: Samsung Exynos (Flattened Device Tree)
> >> [   46.015542] Workqueue: rpciod rpc_async_schedule
> >> [   46.020123]  unwind_backtrace from show_stack+0x10/0x14
> >> [   46.025323]  show_stack from dump_stack_lvl+0x58/0x70
> >> [   46.030301]  dump_stack_lvl from kasan_report+0xa8/0xe0
> >> [   46.035501]  kasan_report from kasan_check_range+0x94/0x1a0
> >> [   46.041048]  kasan_check_range from memcpy+0x28/0x68
> >> [   46.045985]  memcpy from xdr_inline_decode+0x140/0x200
> >> [   46.051098]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x298/0x5b8
> >> [   46.057602]  nfs4_xdr_dec_read_plus from call_decode+0x39c/0x530
> >> [   46.063581]  call_decode from __rpc_execute+0x1f4/0xc5c
> >> [   46.068776]  __rpc_execute from rpc_async_schedule+0x2c/0x4c
> >> [   46.074411]  rpc_async_schedule from process_one_work+0x51c/0xc44
> >> [   46.080478]  process_one_work from worker_thread+0x9c/0x7c0
> >> [   46.086022]  worker_thread from kthread+0x16c/0x1b8
> >> [   46.090872]  kthread from ret_from_fork+0x14/0x2c
> >> [   46.095550] Exception stack(0xf0ba3fb0 to 0xf0ba3ff8)
> >> [   46.100580] 3fa0:                                     00000000 00000000 00000000 00000000
> >> [   46.108740] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> >> [   46.116885] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
> >> [   46.123461] ==================================================================
> >> [   46.130774] Disabling lock debugging due to kernel taint
> >> [   50.848579] systemd[1]: Queued start job for default target Graphical Interface.
> >> [   50.877068] systemd[1]: Created slice Slice /system/getty.
> >>
> >>
> >> Decoded stacktrace is (this is master branch):
> >>
> >> [   46.020123] unwind_backtrace from show_stack (arch/arm/kernel/traps.c:258)
> >> [   46.025323] show_stack from dump_stack_lvl (lib/dump_stack.c:107 (discriminator 4))
> >> [   46.030301] dump_stack_lvl from kasan_report (mm/kasan/report.c:184 mm/kasan/report.c:519)
> >> [   46.035501] kasan_report from kasan_check_range (mm/kasan/generic.c:173 mm/kasan/generic.c:189)
> >> [   46.041048] kasan_check_range from memcpy (mm/kasan/shadow.c:65)
> >> [   46.045985] memcpy from xdr_inline_decode (net/sunrpc/xdr.c:1419 net/sunrpc/xdr.c:1454)
> >
> > Actually, this part is really useful. net/sunrpc/xdr.c:1419 points to
> > the memcpy in xdr_copy_to_scratch(), which has me wondering if I'm
> > incorrectly setting up the xdr scratch buffer that READ_PLUS uses for
> > decoding. Can you try this patch and let me know if it helps?
> >
> > From ac2d6c501dbcdb306480edaee625b5496f1fb4f5 Mon Sep 17 00:00:00 2001
> > From: Anna Schumaker <Anna.Schumaker@Netapp.com>
> > Date: Fri, 10 Feb 2023 15:50:22 -0500
> > Subject: [PATCH] NFSv4.2: Rework scratch handling for READ_PLUS
> >
>
> Patch is corrupted - maybe mail program reformatted it when sending:
>
> Applying: NFSv4.2: Rework scratch handling for READ_PLUS
> error: corrupt patch at line 12
> Patch failed at 0001 NFSv4.2: Rework scratch handling for READ_PLUS

That's weird. I wasn't expecting gmail to reformat the patch but I
guess it did. I've added it as an attachment so that shouldn't happen
again.

Sorry about that!
Anna

>
>
> Best regards,
> Krzysztof
>

[-- Attachment #2: 0001-NFSv4.2-Rework-scratch-handling-for-READ_PLUS.patch --]
[-- Type: text/x-patch, Size: 3664 bytes --]

From ac2d6c501dbcdb306480edaee625b5496f1fb4f5 Mon Sep 17 00:00:00 2001
From: Anna Schumaker <Anna.Schumaker@Netapp.com>
Date: Fri, 10 Feb 2023 15:50:22 -0500
Subject: [PATCH] NFSv4.2: Rework scratch handling for READ_PLUS

Instead of using a tiny scratch buffer, we should use a full scratch
page to match how other NFSv4 operations handle scratch data.

Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
---
 fs/nfs/nfs42xdr.c       |  4 ++--
 fs/nfs/nfs4proc.c       | 14 ++++++++++----
 include/linux/nfs_xdr.h |  1 +
 3 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index d80ee88ca996..702567d5b1db 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -1122,7 +1122,6 @@ static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res *res)
 	uint32_t segments;
 	struct read_plus_segment *segs;
 	int status, i;
-	char scratch_buf[16];
 	__be32 *p;
 
 	status = decode_op_hdr(xdr, OP_READ_PLUS);
@@ -1143,7 +1142,6 @@ static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res *res)
 	if (!segs)
 		return -ENOMEM;
 
-	xdr_set_scratch_buffer(xdr, &scratch_buf, sizeof(scratch_buf));
 	status = -EIO;
 	for (i = 0; i < segments; i++) {
 		status = decode_read_plus_segment(xdr, &segs[i]);
@@ -1348,6 +1346,8 @@ static int nfs4_xdr_dec_read_plus(struct rpc_rqst *rqstp,
 	struct compound_hdr hdr;
 	int status;
 
+	xdr_set_scratch_page(xdr, res->scratch);
+
 	status = decode_compound_hdr(xdr, &hdr);
 	if (status)
 		goto out;
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 40d749f29ed3..5c589d0bd9e9 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -5440,6 +5440,8 @@ static bool nfs4_read_plus_not_supported(struct rpc_task *task,
 
 static int nfs4_read_done(struct rpc_task *task, struct nfs_pgio_header *hdr)
 {
+	if (hdr->res.scratch)
+		__free_page(hdr->res.scratch);
 	if (!nfs4_sequence_done(task, &hdr->res.seq_res))
 		return -EAGAIN;
 	if (nfs4_read_stateid_changed(task, &hdr->args))
@@ -5453,12 +5455,16 @@ static int nfs4_read_done(struct rpc_task *task, struct nfs_pgio_header *hdr)
 }
 
 #if defined CONFIG_NFS_V4_2 && defined CONFIG_NFS_V4_2_READ_PLUS
-static void nfs42_read_plus_support(struct nfs_pgio_header *hdr,
+static bool nfs42_read_plus_support(struct nfs_pgio_header *hdr,
 				    struct rpc_message *msg)
 {
 	/* Note: We don't use READ_PLUS with pNFS yet */
-	if (nfs_server_capable(hdr->inode, NFS_CAP_READ_PLUS) && !hdr->ds_clp)
+	if (nfs_server_capable(hdr->inode, NFS_CAP_READ_PLUS) && !hdr->ds_clp) {
 		msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ_PLUS];
+		hdr->res.scratch = alloc_page(GFP_KERNEL);
+		return hdr->res.scratch != NULL;
+	}
+	return false;
 }
 #else
 static void nfs42_read_plus_support(struct nfs_pgio_header *hdr,
@@ -5473,8 +5479,8 @@ static void nfs4_proc_read_setup(struct nfs_pgio_header *hdr,
 	hdr->timestamp   = jiffies;
 	if (!hdr->pgio_done_cb)
 		hdr->pgio_done_cb = nfs4_read_done_cb;
-	msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ];
-	nfs42_read_plus_support(hdr, msg);
+	if (!nfs42_read_plus_support(hdr, msg))
+		msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ];
 	nfs4_init_sequence(&hdr->args.seq_args, &hdr->res.seq_res, 0, 0);
 }
 
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index e86cf6642d21..6d821aaf0b1a 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -670,6 +670,7 @@ struct nfs_pgio_res {
 		struct {
 			unsigned int		replen;		/* used by read */
 			int			eof;		/* used by read */
+			struct page		*scratch;	/* used by read */
 		};
 		struct {
 			struct nfs_writeverf *	verf;		/* used by write */
-- 
2.39.1

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

On 12/02/2023 15:05, Anna Schumaker wrote:
>>> From ac2d6c501dbcdb306480edaee625b5496f1fb4f5 Mon Sep 17 00:00:00 2001
>>> From: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>> Date: Fri, 10 Feb 2023 15:50:22 -0500
>>> Subject: [PATCH] NFSv4.2: Rework scratch handling for READ_PLUS
>>>
>>
>> Patch is corrupted - maybe mail program reformatted it when sending:
>>
>> Applying: NFSv4.2: Rework scratch handling for READ_PLUS
>> error: corrupt patch at line 12
>> Patch failed at 0001 NFSv4.2: Rework scratch handling for READ_PLUS
> 
> That's weird. I wasn't expecting gmail to reformat the patch but I
> guess it did. I've added it as an attachment so that shouldn't happen
> again.

Still null ptr (built on 420b2d4 with your patch):

[  144.690844] mmiocpy from xdr_inline_decode (net/sunrpc/xdr.c:1419 net/sunrpc/xdr.c:1454) 
[  144.695950] xdr_inline_decode from nfs4_xdr_dec_read_plus (fs/nfs/nfs42xdr.c:1063 fs/nfs/nfs42xdr.c:1147 fs/nfs/nfs42xdr.c:1360 fs/nfs/nfs42xdr.c:1341) 
[  144.702452] nfs4_xdr_dec_read_plus from call_decode (net/sunrpc/clnt.c:2595) 
[  144.708429] call_decode from __rpc_execute (include/asm-generic/bitops/generic-non-atomic.h:128 net/sunrpc/sched.c:954) 
[  144.713538] __rpc_execute from rpc_async_schedule (include/linux/sched/mm.h:336 net/sunrpc/sched.c:1035) 
[  144.719170] rpc_async_schedule from process_one_work (include/linux/jump_label.h:260 include/linux/jump_label.h:270 include/trace/events/workqueue.h:108 kernel/workqueue.c:2294) 
[  144.725238] process_one_work from worker_thread (include/linux/list.h:292 kernel/workqueue.c:2437) 
[  144.730782] worker_thread from kthread (kernel/kthread.c:378) 
[  144.735547] kthread from ret_from_fork (arch/arm/kernel/entry-common.S:149)



Best regards,
Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Olga Kornievskaia]

On Tue, Feb 14, 2023 at 6:08 AM Krzysztof Kozlowski
<krzysztof.kozlowski@linaro.org> wrote:
>
> On 12/02/2023 15:05, Anna Schumaker wrote:
> >>> From ac2d6c501dbcdb306480edaee625b5496f1fb4f5 Mon Sep 17 00:00:00 2001
> >>> From: Anna Schumaker <Anna.Schumaker@Netapp.com>
> >>> Date: Fri, 10 Feb 2023 15:50:22 -0500
> >>> Subject: [PATCH] NFSv4.2: Rework scratch handling for READ_PLUS
> >>>
> >>
> >> Patch is corrupted - maybe mail program reformatted it when sending:
> >>
> >> Applying: NFSv4.2: Rework scratch handling for READ_PLUS
> >> error: corrupt patch at line 12
> >> Patch failed at 0001 NFSv4.2: Rework scratch handling for READ_PLUS
> >
> > That's weird. I wasn't expecting gmail to reformat the patch but I
> > guess it did. I've added it as an attachment so that shouldn't happen
> > again.
>
> Still null ptr (built on 420b2d4 with your patch):
>
> [  144.690844] mmiocpy from xdr_inline_decode (net/sunrpc/xdr.c:1419 net/sunrpc/xdr.c:1454)
> [  144.695950] xdr_inline_decode from nfs4_xdr_dec_read_plus (fs/nfs/nfs42xdr.c:1063 fs/nfs/nfs42xdr.c:1147 fs/nfs/nfs42xdr.c:1360 fs/nfs/nfs42xdr.c:1341)
> [  144.702452] nfs4_xdr_dec_read_plus from call_decode (net/sunrpc/clnt.c:2595)
> [  144.708429] call_decode from __rpc_execute (include/asm-generic/bitops/generic-non-atomic.h:128 net/sunrpc/sched.c:954)
> [  144.713538] __rpc_execute from rpc_async_schedule (include/linux/sched/mm.h:336 net/sunrpc/sched.c:1035)
> [  144.719170] rpc_async_schedule from process_one_work (include/linux/jump_label.h:260 include/linux/jump_label.h:270 include/trace/events/workqueue.h:108 kernel/workqueue.c:2294)
> [  144.725238] process_one_work from worker_thread (include/linux/list.h:292 kernel/workqueue.c:2437)
> [  144.730782] worker_thread from kthread (kernel/kthread.c:378)
> [  144.735547] kthread from ret_from_fork (arch/arm/kernel/entry-common.S:149)

My 2cents...

From what I can tell read_plus only calls xdr_inline_decode() for
"numbers" (eof, #segs, type, offset, length) and we always expect that
__xdr_inline_decode() would return a a non-null "p". But if
__xdr_inline_decode() returned null, the code would call
xdr_copy_to_scratch() which would ultimately call the memcpy().
xdr_copy_to_scrach() expects the scratch buffer to be setup. However,
as I said, for the decode of numbers we don't set up the scratch
space. Which then leads to this oops. How, the reason the
__xdr_inline_decode() would return a null pointer if it ran out it's
provided xdr space which was provided #decode_read_plus_maxsz.

#define NFS42_READ_PLUS_DATA_SEGMENT_SIZE \
                                        (1 /* data_content4 */ + \
                                         2 /* data_info4.di_offset */ + \
                                         1 /* data_info4.di_length */)
#define decode_read_plus_maxsz          (op_decode_hdr_maxsz + \
                                         1 /* rpr_eof */ + \
                                         1 /* rpr_contents count */ + \
                                         NFS42_READ_PLUS_DATA_SEGMENT_SIZE)

while a data segment needs (2) + (1), a hole segment needs to be (2) +
(2) (as both offset and lengths are longs.

while a "correct" maxsz is important for page alignment for reads, it
might means we are not providing enough space for when there are hole
segments? It seems weird that for the spec we have hole length and
data length of different types (long and int).

>
>
>
> Best regards,
> Krzysztof
>

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Linux regression tracking #update (Thorsten Leemhuis)]

[TLDR: This mail in primarily relevant for Linux regression tracking. A
change or fix related to the regression discussed in this thread was
posted or applied, but it did not use a Link: tag to point to the
report, as Linus and the documentation call for. Things happen, no
worries -- but now the regression tracking bot needs to be told manually
about the fix. See link in footer if these mails annoy you.]

On 08.01.23 09:52, Linux kernel regression tracking (#adding) wrote:
> On 07.01.23 16:44, Krzysztof Kozlowski wrote:
>>
>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>> when mounting NFS root on NFSv4 client:
> [...]
> Thanks for the report. To be sure the issue doesn't fall through the
> cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
> tracking bot:
> 
> #regzbot ^introduced 7fd461c47
> #regzbot title nfs: NULL pointer dereference since NFS_V4_2_READ_PLUS is
> enabled by default
> #regzbot ignore-activity

#regzbot fix: 896e090eefedeb8a715ea19938a2791c32679

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

#regzbot ignore-activity

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

On 18/02/2023 05:42, Linux regression tracking #update (Thorsten
Leemhuis) wrote:
> [TLDR: This mail in primarily relevant for Linux regression tracking. A
> change or fix related to the regression discussed in this thread was
> posted or applied, but it did not use a Link: tag to point to the
> report, as Linus and the documentation call for. Things happen, no
> worries -- but now the regression tracking bot needs to be told manually
> about the fix. See link in footer if these mails annoy you.]
> 
> On 08.01.23 09:52, Linux kernel regression tracking (#adding) wrote:
>> On 07.01.23 16:44, Krzysztof Kozlowski wrote:
>>>
>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>>> when mounting NFS root on NFSv4 client:
>> [...]
>> Thanks for the report. To be sure the issue doesn't fall through the
>> cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
>> tracking bot:
>>
>> #regzbot ^introduced 7fd461c47
>> #regzbot title nfs: NULL pointer dereference since NFS_V4_2_READ_PLUS is
>> enabled by default
>> #regzbot ignore-activity
> 
> #regzbot fix: 896e090eefedeb8a715ea19938a2791c32679

I see it was posted and merged as "Revert "NFSv4.2: Change the default
KConfig value for READ_PLUS"". It's nice to give credits to people who
report bugs with "Reported-by" tag.

Best regards,
Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Linux regression tracking (Thorsten Leemhuis)]

On 18.02.23 16:09, Krzysztof Kozlowski wrote:
> On 18/02/2023 05:42, Linux regression tracking #update (Thorsten
> Leemhuis) wrote:
>> [TLDR: This mail in primarily relevant for Linux regression tracking. A
>> change or fix related to the regression discussed in this thread was
>> posted or applied, but it did not use a Link: tag to point to the
>> report, as Linus and the documentation call for. Things happen, no
>> worries -- but now the regression tracking bot needs to be told manually
>> about the fix. See link in footer if these mails annoy you.]
>>
>> On 08.01.23 09:52, Linux kernel regression tracking (#adding) wrote:
>>> On 07.01.23 16:44, Krzysztof Kozlowski wrote:
>>>>
>>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>>>> when mounting NFS root on NFSv4 client:
>>> [...]
>>> Thanks for the report. To be sure the issue doesn't fall through the
>>> cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
>>> tracking bot:
>>>
>>> #regzbot ^introduced 7fd461c47
>>> #regzbot title nfs: NULL pointer dereference since NFS_V4_2_READ_PLUS is
>>> enabled by default
>>> #regzbot ignore-activity
>>
>> #regzbot fix: 896e090eefedeb8a715ea19938a2791c32679
> 
> I see it was posted and merged as "Revert "NFSv4.2: Change the default
> KConfig value for READ_PLUS"". It's nice to give credits to people who
> report bugs with "Reported-by" tag.

Yup. And a "Link:" with the url to the report is missing as well (Linus
wants those, automatic regression tracking needs those [IOW: if they are
missing it causes me trouble, that's why I care], and the docs explain
this as well).

That's why I asked for those two tags, but I didn't even get a reply:

https://lore.kernel.org/all/8d26e819-a3a5-7ae1-bb9e-56bacfa7f65b@leemhuis.info/

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Anna Schumaker]

[-- Attachment #1: Type: text/plain, Size: 2370 bytes --]

Hi Krzysztof,

On Tue, Feb 14, 2023 at 6:02 AM Krzysztof Kozlowski
<krzysztof.kozlowski@linaro.org> wrote:
>
> On 12/02/2023 15:05, Anna Schumaker wrote:
> >>> From ac2d6c501dbcdb306480edaee625b5496f1fb4f5 Mon Sep 17 00:00:00 2001
> >>> From: Anna Schumaker <Anna.Schumaker@Netapp.com>
> >>> Date: Fri, 10 Feb 2023 15:50:22 -0500
> >>> Subject: [PATCH] NFSv4.2: Rework scratch handling for READ_PLUS
> >>>
> >>
> >> Patch is corrupted - maybe mail program reformatted it when sending:
> >>
> >> Applying: NFSv4.2: Rework scratch handling for READ_PLUS
> >> error: corrupt patch at line 12
> >> Patch failed at 0001 NFSv4.2: Rework scratch handling for READ_PLUS
> >
> > That's weird. I wasn't expecting gmail to reformat the patch but I
> > guess it did. I've added it as an attachment so that shouldn't happen
> > again.
>
> Still null ptr (built on 420b2d4 with your patch):

We're through the merge window and at rc1 now, so I can spend more
time scratching my head over your bug again. We've come up with a
patch (attached) that adds a bunch of printks to show us what the
kernel thinks is going on. Do you mind trying it out and letting us
know what gets printed out? You'll need to make sure
CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.

Thanks,
Anna

>
> [  144.690844] mmiocpy from xdr_inline_decode (net/sunrpc/xdr.c:1419 net/sunrpc/xdr.c:1454)
> [  144.695950] xdr_inline_decode from nfs4_xdr_dec_read_plus (fs/nfs/nfs42xdr.c:1063 fs/nfs/nfs42xdr.c:1147 fs/nfs/nfs42xdr.c:1360 fs/nfs/nfs42xdr.c:1341)
> [  144.702452] nfs4_xdr_dec_read_plus from call_decode (net/sunrpc/clnt.c:2595)
> [  144.708429] call_decode from __rpc_execute (include/asm-generic/bitops/generic-non-atomic.h:128 net/sunrpc/sched.c:954)
> [  144.713538] __rpc_execute from rpc_async_schedule (include/linux/sched/mm.h:336 net/sunrpc/sched.c:1035)
> [  144.719170] rpc_async_schedule from process_one_work (include/linux/jump_label.h:260 include/linux/jump_label.h:270 include/trace/events/workqueue.h:108 kernel/workqueue.c:2294)
> [  144.725238] process_one_work from worker_thread (include/linux/list.h:292 kernel/workqueue.c:2437)
> [  144.730782] worker_thread from kthread (kernel/kthread.c:378)
> [  144.735547] kthread from ret_from_fork (arch/arm/kernel/entry-common.S:149)
>
>
>
> Best regards,
> Krzysztof
>

[-- Attachment #2: aglo-read_plus.patch --]
[-- Type: text/x-patch, Size: 2860 bytes --]

diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index d80ee88ca996..dee5ecf0785a 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -346,6 +346,7 @@ static void encode_read_plus(struct xdr_stream *xdr,
 			     const struct nfs_pgio_args *args,
 			     struct compound_hdr *hdr)
 {
+	printk("AGLO: %s setting up decode buffer size=%d\n", __func__, decode_read_plus_maxsz);
 	encode_op_hdr(xdr, OP_READ_PLUS, decode_read_plus_maxsz, hdr);
 	encode_nfs4_stateid(xdr, &args->stateid);
 	encode_uint64(xdr, args->offset);
@@ -1059,6 +1060,7 @@ static int decode_read_plus_segment(struct xdr_stream *xdr,
 {
 	__be32 *p;
 
+	printk("AGLO: %s start seg=%p\n", __func__, seg);
 	p = xdr_inline_decode(xdr, 4);
 	if (!p)
 		return -EIO;
@@ -1082,6 +1084,7 @@ static int decode_read_plus_segment(struct xdr_stream *xdr,
 		xdr_decode_hyper(p, &seg->hole.length);
 	} else
 		return -EINVAL;
+	printk("AGLO: %s end seg=%p xdr->nwords=%d\n", __func__, seg, xdr->nwords);
 	return 0;
 }
 
@@ -1125,6 +1128,7 @@ static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res *res)
 	char scratch_buf[16];
 	__be32 *p;
 
+	printk("AGLO: %s START\n", __func__);
 	status = decode_op_hdr(xdr, OP_READ_PLUS);
 	if (status)
 		return status;
@@ -1158,6 +1162,7 @@ static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res *res)
 
 out:
 	kfree(segs);
+	printk("AGLO: %s END\n", __func__);
 	return status;
 }
 
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 0b0b9f1eed46..3449ea836d30 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1337,6 +1337,7 @@ void rpc_prepare_reply_pages(struct rpc_rqst *req, struct page **pages,
 {
 	hdrsize += RPC_REPHDRSIZE + req->rq_cred->cr_auth->au_ralign;
 
+	printk("AGLO: %s hdrsize %d (<<2 %d) HDRSIZE %d auligh %d\n", __func__, hdrsize, hdrsize << 2, RPC_REPHDRSIZE, req->rq_cred->cr_auth->au_ralign);
 	xdr_inline_pages(&req->rq_rcv_buf, hdrsize << 2, pages, base, len);
 	trace_rpc_xdr_reply_pages(req->rq_task, &req->rq_rcv_buf);
 }
diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
index f7767bf22406..e1bb9fcbaad6 100644
--- a/net/sunrpc/xdr.c
+++ b/net/sunrpc/xdr.c
@@ -1397,6 +1397,7 @@ static __be32 * __xdr_inline_decode(struct xdr_stream *xdr, size_t nbytes)
 	__be32 *p = xdr->p;
 	__be32 *q = p + nwords;
 
+	printk("AGLO: %s nwords=%d xdr->nwords=%d q=%p xdr->end=%p p=%p (%d %d)\n", __func__, nwords, xdr->nwords, q, xdr->end, p, q > xdr->end, q < p);
 	if (unlikely(nwords > xdr->nwords || q > xdr->end || q < p))
 		return NULL;
 	xdr->p = q;
@@ -1410,6 +1411,7 @@ static __be32 *xdr_copy_to_scratch(struct xdr_stream *xdr, size_t nbytes)
 	char *cpdest = xdr->scratch.iov_base;
 	size_t cplen = (char *)xdr->end - (char *)xdr->p;
 
+	printk("AGLO: %s here\n", __func__);
 	if (nbytes > xdr->scratch.iov_len)
 		goto out_overflow;
 	p = __xdr_inline_decode(xdr, cplen);

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Olga Kornievskaia]

On Mon, Mar 6, 2023 at 12:12 PM Anna Schumaker <schumaker.anna@gmail.com> wrote:
>
> Hi Krzysztof,
>
> On Tue, Feb 14, 2023 at 6:02 AM Krzysztof Kozlowski
> <krzysztof.kozlowski@linaro.org> wrote:
> >
> > On 12/02/2023 15:05, Anna Schumaker wrote:
> > >>> From ac2d6c501dbcdb306480edaee625b5496f1fb4f5 Mon Sep 17 00:00:00 2001
> > >>> From: Anna Schumaker <Anna.Schumaker@Netapp.com>
> > >>> Date: Fri, 10 Feb 2023 15:50:22 -0500
> > >>> Subject: [PATCH] NFSv4.2: Rework scratch handling for READ_PLUS
> > >>>
> > >>
> > >> Patch is corrupted - maybe mail program reformatted it when sending:
> > >>
> > >> Applying: NFSv4.2: Rework scratch handling for READ_PLUS
> > >> error: corrupt patch at line 12
> > >> Patch failed at 0001 NFSv4.2: Rework scratch handling for READ_PLUS
> > >
> > > That's weird. I wasn't expecting gmail to reformat the patch but I
> > > guess it did. I've added it as an attachment so that shouldn't happen
> > > again.
> >
> > Still null ptr (built on 420b2d4 with your patch):
>
> We're through the merge window and at rc1 now, so I can spend more
> time scratching my head over your bug again. We've come up with a
> patch (attached) that adds a bunch of printks to show us what the
> kernel thinks is going on. Do you mind trying it out and letting us
> know what gets printed out? You'll need to make sure
> CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.

Hi Krzystof,

Since you are the only one hitting the problem, could you be so kind
as to help with getting this resolved.

Thank you.

>
> Thanks,
> Anna
>
> >
> > [  144.690844] mmiocpy from xdr_inline_decode (net/sunrpc/xdr.c:1419 net/sunrpc/xdr.c:1454)
> > [  144.695950] xdr_inline_decode from nfs4_xdr_dec_read_plus (fs/nfs/nfs42xdr.c:1063 fs/nfs/nfs42xdr.c:1147 fs/nfs/nfs42xdr.c:1360 fs/nfs/nfs42xdr.c:1341)
> > [  144.702452] nfs4_xdr_dec_read_plus from call_decode (net/sunrpc/clnt.c:2595)
> > [  144.708429] call_decode from __rpc_execute (include/asm-generic/bitops/generic-non-atomic.h:128 net/sunrpc/sched.c:954)
> > [  144.713538] __rpc_execute from rpc_async_schedule (include/linux/sched/mm.h:336 net/sunrpc/sched.c:1035)
> > [  144.719170] rpc_async_schedule from process_one_work (include/linux/jump_label.h:260 include/linux/jump_label.h:270 include/trace/events/workqueue.h:108 kernel/workqueue.c:2294)
> > [  144.725238] process_one_work from worker_thread (include/linux/list.h:292 kernel/workqueue.c:2437)
> > [  144.730782] worker_thread from kthread (kernel/kthread.c:378)
> > [  144.735547] kthread from ret_from_fork (arch/arm/kernel/entry-common.S:149)
> >
> >
> >
> > Best regards,
> > Krzysztof
> >

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

On 04/04/2023 03:01, Olga Kornievskaia wrote:
> On Mon, Mar 6, 2023 at 12:12 PM Anna Schumaker <schumaker.anna@gmail.com> wrote:
>>
>> Hi Krzysztof,
>>
>> On Tue, Feb 14, 2023 at 6:02 AM Krzysztof Kozlowski
>> <krzysztof.kozlowski@linaro.org> wrote:
>>>
>>> On 12/02/2023 15:05, Anna Schumaker wrote:
>>>>>> From ac2d6c501dbcdb306480edaee625b5496f1fb4f5 Mon Sep 17 00:00:00 2001
>>>>>> From: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>>> Date: Fri, 10 Feb 2023 15:50:22 -0500
>>>>>> Subject: [PATCH] NFSv4.2: Rework scratch handling for READ_PLUS
>>>>>>
>>>>>
>>>>> Patch is corrupted - maybe mail program reformatted it when sending:
>>>>>
>>>>> Applying: NFSv4.2: Rework scratch handling for READ_PLUS
>>>>> error: corrupt patch at line 12
>>>>> Patch failed at 0001 NFSv4.2: Rework scratch handling for READ_PLUS
>>>>
>>>> That's weird. I wasn't expecting gmail to reformat the patch but I
>>>> guess it did. I've added it as an attachment so that shouldn't happen
>>>> again.
>>>
>>> Still null ptr (built on 420b2d4 with your patch):
>>
>> We're through the merge window and at rc1 now, so I can spend more
>> time scratching my head over your bug again. We've come up with a
>> patch (attached) that adds a bunch of printks to show us what the
>> kernel thinks is going on. Do you mind trying it out and letting us
>> know what gets printed out? You'll need to make sure
>> CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.
> 
> Hi Krzystof,
> 
> Since you are the only one hitting the problem, could you be so kind
> as to help with getting this resolved.
> 

My setup is not special. It is regular ARMv7 board using a regular ARMv8
NFS server. Anyway my farm is since some time offline due to relocation
and will be online in June, so I cannot test anything.

Best regards,
Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

On 06/03/2023 18:09, Anna Schumaker wrote:
> Hi Krzysztof,
> 
> On Tue, Feb 14, 2023 at 6:02 AM Krzysztof Kozlowski
> <krzysztof.kozlowski@linaro.org> wrote:
>>
>> On 12/02/2023 15:05, Anna Schumaker wrote:
>>>>> From ac2d6c501dbcdb306480edaee625b5496f1fb4f5 Mon Sep 17 00:00:00 2001
>>>>> From: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>> Date: Fri, 10 Feb 2023 15:50:22 -0500
>>>>> Subject: [PATCH] NFSv4.2: Rework scratch handling for READ_PLUS
>>>>>
>>>>
>>>> Patch is corrupted - maybe mail program reformatted it when sending:
>>>>
>>>> Applying: NFSv4.2: Rework scratch handling for READ_PLUS
>>>> error: corrupt patch at line 12
>>>> Patch failed at 0001 NFSv4.2: Rework scratch handling for READ_PLUS
>>>
>>> That's weird. I wasn't expecting gmail to reformat the patch but I
>>> guess it did. I've added it as an attachment so that shouldn't happen
>>> again.
>>
>> Still null ptr (built on 420b2d4 with your patch):
> 
> We're through the merge window and at rc1 now, so I can spend more
> time scratching my head over your bug again. We've come up with a
> patch (attached) that adds a bunch of printks to show us what the
> kernel thinks is going on. Do you mind trying it out and letting us
> know what gets printed out? You'll need to make sure
> CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.

The patch does not apply. I tried: v6.4-rc1, v6.4-rc5, next-20230609.

Best regards,
Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Anna Schumaker]

[-- Attachment #1: Type: text/plain, Size: 1985 bytes --]

Hi Krzysztof,

On Sat, Jun 10, 2023 at 6:15 AM Krzysztof Kozlowski
<krzysztof.kozlowski@linaro.org> wrote:
>
> On 06/03/2023 18:09, Anna Schumaker wrote:
> > Hi Krzysztof,
> >
> > On Tue, Feb 14, 2023 at 6:02 AM Krzysztof Kozlowski
> > <krzysztof.kozlowski@linaro.org> wrote:
> >>
> >> On 12/02/2023 15:05, Anna Schumaker wrote:
> >>>>> From ac2d6c501dbcdb306480edaee625b5496f1fb4f5 Mon Sep 17 00:00:00 2001
> >>>>> From: Anna Schumaker <Anna.Schumaker@Netapp.com>
> >>>>> Date: Fri, 10 Feb 2023 15:50:22 -0500
> >>>>> Subject: [PATCH] NFSv4.2: Rework scratch handling for READ_PLUS
> >>>>>
> >>>>
> >>>> Patch is corrupted - maybe mail program reformatted it when sending:
> >>>>
> >>>> Applying: NFSv4.2: Rework scratch handling for READ_PLUS
> >>>> error: corrupt patch at line 12
> >>>> Patch failed at 0001 NFSv4.2: Rework scratch handling for READ_PLUS
> >>>
> >>> That's weird. I wasn't expecting gmail to reformat the patch but I
> >>> guess it did. I've added it as an attachment so that shouldn't happen
> >>> again.
> >>
> >> Still null ptr (built on 420b2d4 with your patch):
> >
> > We're through the merge window and at rc1 now, so I can spend more
> > time scratching my head over your bug again. We've come up with a
> > patch (attached) that adds a bunch of printks to show us what the
> > kernel thinks is going on. Do you mind trying it out and letting us
> > know what gets printed out? You'll need to make sure
> > CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.
>
> The patch does not apply. I tried: v6.4-rc1, v6.4-rc5, next-20230609.

Can you try the attached patch on top of my 3-patch series from the
other day, and let me know what gets printed out? It adds a bunch of
printk()s at strategic points to print out what is going on with the
xdr scratch buffer since it's suddenly a bad memory address after
working for a bit on your machine.

Thanks,
Anna

>
> Best regards,
> Krzysztof
>

[-- Attachment #2: 0001-NFS-Add-debugging-printk-s-to-trace-the-xdr-scratch-.patch --]
[-- Type: text/x-patch, Size: 2716 bytes --]

From 985248165e53a67e7cb6a18ec2813aba8b26da33 Mon Sep 17 00:00:00 2001
From: Anna Schumaker <Anna.Schumaker@Netapp.com>
Date: Wed, 14 Jun 2023 16:49:37 -0400
Subject: [RFC] NFS: Add debugging printk()s to trace the xdr->scratch buffer

I'm trying to figure out at what point the xdr->scratch buffer is
allocated, freed, set, and reset to figure out why READ_PLUS suddenly
thinks it's a NULL pointer with length 16.

Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
---
 fs/nfs/nfs42xdr.c | 7 +++++++
 fs/nfs/read.c     | 8 +++++++-
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index 20aa5e746497..6f63c816b0a5 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -1351,21 +1351,28 @@ static int nfs4_xdr_dec_read_plus(struct rpc_rqst *rqstp,
 	struct compound_hdr hdr;
 	int status;
 
+	printk(KERN_INFO "%s(hdr=%px, scratch=%px)\n", __func__,
+		container_of(res, struct nfs_pgio_header, res), res->scratch);
 	xdr_set_scratch_buffer(xdr, res->scratch, READ_PLUS_SCRATCH_SIZE);
 
+	printk(KERN_INFO "    buf = {%px, %zd}\n", xdr->scratch.iov_base, xdr->scratch.iov_len);
 	status = decode_compound_hdr(xdr, &hdr);
 	if (status)
 		goto out;
+	printk(KERN_INFO "    buf = {%px, %zd}\n", xdr->scratch.iov_base, xdr->scratch.iov_len);
 	status = decode_sequence(xdr, &res->seq_res, rqstp);
 	if (status)
 		goto out;
+	printk(KERN_INFO "    buf = {%px, %zd}\n", xdr->scratch.iov_base, xdr->scratch.iov_len);
 	status = decode_putfh(xdr);
 	if (status)
 		goto out;
+	printk(KERN_INFO "    buf = {%px, %zd}\n", xdr->scratch.iov_base, xdr->scratch.iov_len);
 	status = decode_read_plus(xdr, res);
 	if (!status)
 		status = res->count;
 out:
+	printk(KERN_INFO "    buf = {%px, %zd}\n", xdr->scratch.iov_base, xdr->scratch.iov_len);
 	return status;
 }
 
diff --git a/fs/nfs/read.c b/fs/nfs/read.c
index 7dc21a48e3e7..7b93316a52de 100644
--- a/fs/nfs/read.c
+++ b/fs/nfs/read.c
@@ -47,8 +47,11 @@ static struct nfs_pgio_header *nfs_readhdr_alloc(void)
 
 static void nfs_readhdr_free(struct nfs_pgio_header *rhdr)
 {
-	if (rhdr->res.scratch != NULL)
+	if (rhdr->res.scratch != NULL) {
+		printk(KERN_INFO "%s(hdr=%px, scratch=%px)\n",
+			__func__, rhdr, rhdr->res.scratch);
 		kfree(rhdr->res.scratch);
+	}
 	kmem_cache_free(nfs_rdata_cachep, rhdr);
 }
 
@@ -114,6 +117,9 @@ bool nfs_read_alloc_scratch(struct nfs_pgio_header *hdr, size_t size)
 {
 	WARN_ON(hdr->res.scratch != NULL);
 	hdr->res.scratch = kmalloc(size, GFP_KERNEL);
+	printk(KERN_INFO "\n");
+	printk(KERN_INFO "%s(hdr=%px, size=%zd) = %px\n",
+		__func__, hdr, size, hdr->res.scratch);
 	return hdr->res.scratch != NULL;
 }
 EXPORT_SYMBOL_GPL(nfs_read_alloc_scratch);
-- 
2.41.0

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]

On 14/06/2023 22:55, Anna Schumaker wrote:
>>>> Still null ptr (built on 420b2d4 with your patch):
>>>
>>> We're through the merge window and at rc1 now, so I can spend more
>>> time scratching my head over your bug again. We've come up with a
>>> patch (attached) that adds a bunch of printks to show us what the
>>> kernel thinks is going on. Do you mind trying it out and letting us
>>> know what gets printed out? You'll need to make sure
>>> CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.
>>
>> The patch does not apply. I tried: v6.4-rc1, v6.4-rc5, next-20230609.
> 
> Can you try the attached patch on top of my 3-patch series from the
> other day, and let me know what gets printed out? It adds a bunch of
> printk()s at strategic points to print out what is going on with the
> xdr scratch buffer since it's suddenly a bad memory address after
> working for a bit on your machine.
> 

Here you have entire log - attached (113 kB, I hope goes past mailing
lists/spam filters).

Best regards,
Krzysztof

[-- Attachment #2: oops-nfs-read-plus-20230615.txt --]
[-- Type: text/plain, Size: 116050 bytes --]

Kernel image @ 0x40008000 [ 0x000000 - 0x7b4788 ]
## Loading init Ramdisk from Legacy Image at 45000000 ...
   Image Name:   U-Boot Odroid ARMv7 ramdisk
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    8076595 Bytes = 7.7 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 44000000
   Booting using the fdt blob at 0x44000000
   Loading Ramdisk to 4f84c000, end 4ffffd33 ... OK
   Loading Device Tree to 4f83a000, end 4f84be41 ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x100
[    0.000000] Linux version 6.4.0-rc6-next-20230615-00004-g8e95ecb4298d (krzk@krzk-bin) (arm-linux-gnueabi-gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #83 SMP PREEMPT Thu Jun 15 10:47:34 CEST 2023
[    0.000000] CPU: ARMv7 Processor [410fc073] revision 3 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: Hardkernel Odroid HC1
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] cma: Reserved 96 MiB at 0xb8800000
[    0.000000] Samsung CPU ID: 0xe5422001
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000040000000-0x000000006fffffff]
[    0.000000]   HighMem  [mem 0x0000000070000000-0x00000000be9fffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000040000000-0x00000000be9fffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x00000000be9fffff]
[    0.000000] Running under secure firmware.
[    0.000000] percpu: Embedded 17 pages/cpu s38964 r8192 d22476 u69632
[    0.000000] Kernel command line: console=tty1 console=ttySAC2,115200n8 ip=192.168.1.12:192.168.1.10:192.168.1.1:255.255.255.0::eth0:none nfsrootdebug root=/dev/nfs nfsroot=192.168.1.10:/srv/nfs/odroidhc1,vers=4,nolock rootwait rw no_console_suspend 
[    0.000000] Unknown kernel command line parameters "nfsrootdebug", will be passed to user space.
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 517120
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 1923984K/2074624K available (12288K kernel code, 1528K rwdata, 4228K rodata, 1024K init, 6604K bss, 52336K reserved, 98304K cma-reserved, 1189888K highmem)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] trace event string verifier disabled
[    0.000000] Running RCU self tests
[    0.000000] Running RCU synchronous self tests
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU lockdep checking is enabled.
[    0.000000] 	Trampoline variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] Running RCU synchronous self tests
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] GIC: Using split EOI/Deactivate mode
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] Switching to timer-based delay loop, resolution 41ns
[    0.000000] clocksource: mct-frc: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[    0.000002] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
[    0.000063] genirq: irq_chip COMBINER did not update eff. affinity mask of irq 57
[    0.001891] arch_timer: cp15 timer(s) running at 24.00MHz (phys).
[    0.001927] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x588fe9dc0, max_idle_ns: 440795202592 ns
[    0.001958] sched_clock: 56 bits at 24MHz, resolution 41ns, wraps every 4398046511097ns
[    0.001981] Ignoring duplicate/late registration of read_current_timer delay
[    0.003422] Console: colour dummy device 80x30
[    0.003593] printk: console [tty1] enabled
[    0.011039] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[    0.011231] ... MAX_LOCKDEP_SUBCLASSES:  8
[    0.011340] ... MAX_LOCK_DEPTH:          48
[    0.011450] ... MAX_LOCKDEP_KEYS:        8192
[    0.011562] ... CLASSHASH_SIZE:          4096
[    0.011674] ... MAX_LOCKDEP_ENTRIES:     32768
[    0.011787] ... MAX_LOCKDEP_CHAINS:      65536
[    0.011901] ... CHAINHASH_SIZE:          32768
[    0.012016]  memory used by lock dependency info: 4125 kB
[    0.012147]  memory used for stack traces: 2112 kB
[    0.012267]  per task-struct memory footprint: 1536 bytes
[    0.012480] Calibrating delay loop (skipped), value calculated using timer frequency.. 48.00 BogoMIPS (lpj=240000)
[    0.012732] pid_max: default: 32768 minimum: 301
[    0.013643] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[    0.013849] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[    0.018285] CPU: Testing write buffer coherency: ok
[    0.018671] Running RCU synchronous self tests
[    0.018810] Running RCU synchronous self tests
[    0.021269] CPU0: thread -1, cpu 0, socket 1, mpidr 80000100
[    0.026367] RCU Tasks: Setting shift to 3 and lim to 1 rcu_task_cb_adjust=1.
[    0.027009] Running RCU-tasks wait API self tests
[    0.130396] Setting up static identity map for 0x40100000 - 0x40100060
[    0.131615] ARM CCI driver probed
[    0.132455] Exynos MCPM support installed
[    0.140366] rcu: Hierarchical SRCU implementation.
[    0.140502] rcu: 	Max phase no-delay instances is 1000.
[    0.146785] smp: Bringing up secondary CPUs ...
[    0.151107] CPU1: thread -1, cpu 1, socket 1, mpidr 80000101
[    0.156008] CPU2: thread -1, cpu 2, socket 1, mpidr 80000102
[    0.159993] CPU3: thread -1, cpu 3, socket 1, mpidr 80000103
[    0.164005] CPU4: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.164029] CPU4: detected I-Cache line size mismatch, workaround enabled
[    0.164041] CPU4: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    0.164616] CPU4: Spectre BHB: enabling loop workaround for all CPUs
[    0.168543] CPU5: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.168562] CPU5: detected I-Cache line size mismatch, workaround enabled
[    0.168573] CPU5: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    0.172673] CPU6: thread -1, cpu 2, socket 0, mpidr 80000002
[    0.172693] CPU6: detected I-Cache line size mismatch, workaround enabled
[    0.172705] CPU6: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    0.176793] CPU7: thread -1, cpu 3, socket 0, mpidr 80000003
[    0.176812] CPU7: detected I-Cache line size mismatch, workaround enabled
[    0.176824] CPU7: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    0.178119] smp: Brought up 1 node, 8 CPUs
[    0.178286] SMP: Total of 8 processors activated (384.00 BogoMIPS).
[    0.178448] CPU: All CPU(s) started in HYP mode.
[    0.178572] CPU: Virtualization extensions available.
[    0.182904] devtmpfs: initialized
[    0.250315] Callback from call_rcu_tasks() invoked.
[    0.270259] VFP support v0.3: implementor 41 architecture 4 part 30 variant f rev 0
[    0.271644] Running RCU synchronous self tests
[    0.271812] Running RCU synchronous self tests
[    0.273250] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.273445] futex hash table entries: 2048 (order: 5, 131072 bytes, linear)
[    0.276089] pinctrl core: initialized pinctrl subsystem
[    0.281581] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.286494] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.297000] thermal_sys: Registered thermal governor 'step_wise'
[    0.297887] cpuidle: using governor menu
[    0.299348] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
[    0.299554] hw-breakpoint: maximum watchpoint size is 8 bytes.
[    0.378142] gpio gpiochip0: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.379838] gpio gpiochip1: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.381274] gpio gpiochip2: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.382299] gpio gpiochip3: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.383305] gpio gpiochip4: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.389512] gpio gpiochip5: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.390510] gpio gpiochip6: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.391477] gpio gpiochip7: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.392465] gpio gpiochip8: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.393515] gpio gpiochip9: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.394593] gpio gpiochip10: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.395559] gpio gpiochip11: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.396539] gpio gpiochip12: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.397505] gpio gpiochip13: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.398473] gpio gpiochip14: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.399458] gpio gpiochip15: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.400480] gpio gpiochip16: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.401464] gpio gpiochip17: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.407136] gpio gpiochip18: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.408120] gpio gpiochip19: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.409143] gpio gpiochip20: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.410134] gpio gpiochip21: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.411140] gpio gpiochip22: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.412143] gpio gpiochip23: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.413111] gpio gpiochip24: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.414073] gpio gpiochip25: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.420188] gpio gpiochip26: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.421170] gpio gpiochip27: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.422177] gpio gpiochip28: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.423172] gpio gpiochip29: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.424213] gpio gpiochip30: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.425177] gpio gpiochip31: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.426136] gpio gpiochip32: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.427103] gpio gpiochip33: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.428094] gpio gpiochip34: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.433462] gpio gpiochip35: Static allocation of GPIO base is deprecated, use dynamic allocation.
[    0.449122] EXYNOS5420 PMU initialized
[    0.700863] iommu: Default domain type: Translated
[    0.700963] iommu: DMA domain TLB invalidation policy: strict mode
[    0.703305] SCSI subsystem initialized
[    0.704300] usbcore: registered new interface driver usbfs
[    0.704579] usbcore: registered new interface driver hub
[    0.704857] usbcore: registered new device driver usb
[    0.707173] pps_core: LinuxPPS API ver. 1 registered
[    0.707267] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.707457] PTP clock support registered
[    0.709223] Advanced Linux Sound Architecture Driver Initialized.
[    0.712096] nfc: nfc_init: NFC Core ver 0.1
[    0.712415] NET: Registered PF_NFC protocol family
[    0.714102] clocksource: Switched to clocksource mct-frc
[    0.758835] NET: Registered PF_INET protocol family
[    0.759565] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.762662] tcp_listen_portaddr_hash hash table entries: 512 (order: 2, 20480 bytes, linear)
[    0.763225] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.763376] TCP established hash table entries: 8192 (order: 3, 32768 bytes, linear)
[    0.763835] TCP bind hash table entries: 8192 (order: 7, 655360 bytes, linear)
[    0.767737] TCP: Hash tables configured (established 8192 bind 8192)
[    0.768242] UDP hash table entries: 512 (order: 3, 49152 bytes, linear)
[    0.768660] UDP-Lite hash table entries: 512 (order: 3, 49152 bytes, linear)
[    0.769470] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.771692] RPC: Registered named UNIX socket transport module.
[    0.771857] RPC: Registered udp transport module.
[    0.771949] RPC: Registered tcp transport module.
[    0.772038] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.774188] Trying to unpack rootfs image as initramfs...
[    0.794609] hw perfevents: enabled with armv7_cortex_a7 PMU driver, 5 counters available
[    0.797997] hw perfevents: enabled with armv7_cortex_a15 PMU driver, 7 counters available
[    0.806372] Initialise system trusted keyrings
[    0.807372] workingset: timestamp_bits=30 max_order=19 bucket_order=0
[    0.809377] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.810688] NFS: Registering the id_resolver key type
[    0.810954] Key type id_resolver registered
[    0.811094] Key type id_legacy registered
[    0.811254] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.811529] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.811788] romfs: ROMFS MTD (C) 2007 Red Hat, Inc.
[    0.812745] Key type asymmetric registered
[    0.812897] Asymmetric key parser 'x509' registered
[    0.813376] bounce: pool size: 64 pages
[    0.813587] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
[    0.813769] io scheduler mq-deadline registered
[    0.813860] io scheduler kyber registered
[    0.814120] io scheduler bfq registered
[    0.816437] samsung-pinctrl 13400000.pinctrl: Failed to create device link (0x180) with soc
[    0.823580] samsung-usb2-phy 12130000.phy: supply vbus not found, using dummy regulator
[    0.826898] exynos5_usb3drd_phy 12100000.phy: supply vbus not found, using dummy regulator
[    0.827618] exynos5_usb3drd_phy 12100000.phy: supply vbus-boost not found, using dummy regulator
[    0.829243] exynos5_usb3drd_phy 12500000.phy: supply vbus not found, using dummy regulator
[    0.829906] exynos5_usb3drd_phy 12500000.phy: supply vbus-boost not found, using dummy regulator
[    0.851948] dma-pl330 3880000.dma-controller: Loaded driver for PL330 DMAC-241330
[    0.852092] dma-pl330 3880000.dma-controller: 	DBUFF-4x8bytes Num_Chans-6 Num_Peri-16 Num_Events-6
[    0.864503] dma-pl330 121a0000.dma-controller: Loaded driver for PL330 DMAC-241330
[    0.864646] dma-pl330 121a0000.dma-controller: 	DBUFF-32x4bytes Num_Chans-8 Num_Peri-32 Num_Events-32
[    0.874275] dma-pl330 3880000.dma-controller: PM domain MAU will not be powered off
[    0.877902] dma-pl330 121b0000.dma-controller: Loaded driver for PL330 DMAC-241330
[    0.878041] dma-pl330 121b0000.dma-controller: 	DBUFF-32x4bytes Num_Chans-8 Num_Peri-32 Num_Events-32
[    0.882192] dma-pl330 10800000.dma-controller: Loaded driver for PL330 DMAC-241330
[    0.882331] dma-pl330 10800000.dma-controller: 	DBUFF-64x8bytes Num_Chans-8 Num_Peri-1 Num_Events-32
[    1.169030] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    1.182982] 12c00000.serial: ttySAC0 at MMIO 0x12c00000 (irq = 122, base_baud = 0) is a S3C6400/10
[    1.186914] 12c10000.serial: ttySAC1 at MMIO 0x12c10000 (irq = 123, base_baud = 0) is a S3C6400/10
[    1.190323] 12c20000.serial: ttySAC2 at MMIO 0x12c20000 (irq = 124, base_baud = 0) is a S3C6400/10
[    1.191034] printk: console [ttySAC2] enabled
[    1.464180] Freeing initrd memory: 7888K
[    1.470536] 12c30000.serial: ttySAC3 at MMIO 0x12c30000 (irq = 125, base_baud = 0) is a S3C6400/10
[    2.635151] random: crng init done
[    2.635720] exynos-trng 10830600.rng: Exynos True Random Number Generator.
[    2.704852] brd: module loaded
[    2.743322] loop: module loaded
[    2.753815] usbcore: registered new device driver r8152-cfgselector
[    2.758900] usbcore: registered new interface driver r8152
[    2.764336] usbcore: registered new interface driver asix
[    2.769643] usbcore: registered new interface driver ax88179_178a
[    2.775794] usbcore: registered new interface driver cdc_ether
[    2.781520] usbcore: registered new interface driver smsc75xx
[    2.787302] usbcore: registered new interface driver smsc95xx
[    2.792957] usbcore: registered new interface driver net1080
[    2.798652] usbcore: registered new interface driver cdc_subset
[    2.804544] usbcore: registered new interface driver zaurus
[    2.810048] usbcore: registered new interface driver cdc_ncm
[    2.815721] usbcore: registered new interface driver r8153_ecm
[    2.827664] phy phy-12130000.phy.6: phy_power_on was called before phy_init
[    2.833354] exynos-ehci 12110000.usb: EHCI Host Controller
[    2.839261] exynos-ehci 12110000.usb: new USB bus registered, assigned bus number 1
[    2.847595] exynos-ehci 12110000.usb: irq 126, io mem 0x12110000
[    2.874200] exynos-ehci 12110000.usb: USB 2.0 started, EHCI 1.00
[    2.880076] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.04
[    2.887207] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.894339] usb usb1: Product: EHCI Host Controller
[    2.899098] usb usb1: Manufacturer: Linux 6.4.0-rc6-next-20230615-00004-g8e95ecb4298d ehci_hcd
[    2.907784] usb usb1: SerialNumber: 12110000.usb
[    2.916343] hub 1-0:1.0: USB hub found
[    2.918912] hub 1-0:1.0: 3 ports detected
[    2.928484] phy phy-12130000.phy.6: phy_power_on was called before phy_init
[    2.934154] exynos-ohci 12120000.usb: USB Host Controller
[    2.939488] exynos-ohci 12120000.usb: new USB bus registered, assigned bus number 2
[    2.947598] exynos-ohci 12120000.usb: irq 126, io mem 0x12120000
[    3.018788] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.04
[    3.025907] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.032891] usb usb2: Product: USB Host Controller
[    3.037685] usb usb2: Manufacturer: Linux 6.4.0-rc6-next-20230615-00004-g8e95ecb4298d ohci_hcd
[    3.046309] usb usb2: SerialNumber: 12120000.usb
[    3.053025] hub 2-0:1.0: USB hub found
[    3.055586] hub 2-0:1.0: 3 ports detected
[    3.064522] usbcore: registered new interface driver uas
[    3.068604] usbcore: registered new interface driver usb-storage
[    3.076570] UDC core: g_ether: couldn't find an available UDC
[    3.086860] i2c_dev: i2c /dev entries driver
[    3.118324] s5m-rtc s2mps14-rtc: registered as rtc0
[    3.125693] s5m-rtc s2mps14-rtc: setting system clock to 2000-01-01T00:00:32 UTC (946684832)
[    3.134158] s2mps11-clk s2mps11-clk: DMA mask not set
[    3.172125] vddq_mmc2: Bringing 3300000uV into 2800000-2800000uV
[    3.178268] exynos-tmu 100a0000.tmu: More trip points than supported by this TMU.
[    3.184475] exynos-tmu 100a0000.tmu: 3 trip points should be configured in polling mode.
[    3.196471] s3c2410-wdt 101d0000.watchdog: watchdog inactive, reset disabled, irq disabled
[    3.205518] device-mapper: ioctl: 4.48.0-ioctl (2023-03-01) initialised: dm-devel@redhat.com
[    3.211444] vdd_sd: Bringing 3300000uV into 2800000-2800000uV
[    3.218697] sdhci: Secure Digital Host Controller Interface driver
[    3.224510] sdhci: Copyright(c) Pierre Ossman
[    3.229563] Synopsys Designware Multimedia Card Interface Driver
[    3.237764] dwmmc_exynos 12220000.mmc: IDMAC supports 32-bit address mode.
[    3.244056] dwmmc_exynos 12220000.mmc: Using internal DMA controller.
[    3.249699] dwmmc_exynos 12220000.mmc: Version ID is 250a
[    3.251764] s5p-secss 10830000.sss: s5p-sss driver registered
[    3.255594] dwmmc_exynos 12220000.mmc: DW MMC controller at irq 136,64 bit host data width,64 deep fifo
[    3.267986] usbcore: registered new interface driver usbhid
[    3.275643] usbhid: USB HID core driver
[    3.298358] mmc_host mmc2: Bus speed (slot 0) = 50000000Hz (slot req 400000Hz, actual 396825HZ div = 63)
[    3.348847] exynos-nocp: new NoC Probe device registered: 10ca1000.nocp
[    3.355080] exynos-nocp: new NoC Probe device registered: 10ca1400.nocp
[    3.361568] exynos-nocp: new NoC Probe device registered: 10ca1800.nocp
[    3.368304] exynos-nocp: new NoC Probe device registered: 10ca1c00.nocp
[    3.375678] exynos-ppmu: new PPMU device registered 10d00000.ppmu (ppmu-event3-dmc0-0)
[    3.383246] exynos-ppmu: new PPMU device registered 10d10000.ppmu (ppmu-event3-dmc0-1)
[    3.384121] mmc_host mmc2: Bus speed (slot 0) = 50000000Hz (slot req 50000000Hz, actual 50000000HZ div = 0)
[    3.391672] exynos-ppmu: new PPMU device registered 10d60000.ppmu (ppmu-event3-dmc1-0)
[    3.409004] exynos-ppmu: new PPMU device registered 10d70000.ppmu (ppmu-event3-dmc1-1)
[    3.425848] exynos5-dmc 10c20000.memory-controller: error -ENXIO: IRQ drex_0 not found
[    3.432425] exynos5-dmc 10c20000.memory-controller: error -ENXIO: IRQ drex_1 not found
[    3.442314] exynos5-dmc 10c20000.memory-controller: DMC initialized, in irq mode: 0
[    3.451286] exynos-adc 12d10000.adc: error -ENXIO: IRQ index 1 not found
[    3.473455] NET: Registered PF_INET6 protocol family
[    3.480847] Segment Routing with IPv6
[    3.483170] In-situ OAM (IOAM) with IPv6
[    3.487822] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    3.495740] NET: Registered PF_PACKET protocol family
[    3.499440] NET: Registered PF_KEY protocol family
[    3.504632] Key type dns_resolver registered
[    3.509474] Registering SWP/SWPB emulation handler
[    3.557927] Loading compiled-in X.509 certificates
[    3.565876] mmc2: new ultra high speed DDR50 SDHC card at address 0001
[    3.575497] mmcblk2: mmc2:0001 00000 29.8 GiB
[    3.593323]  mmcblk2: p1 p2
[    3.864246] panfrost 11800000.gpu: clock rate = 400000000
[    3.890104] 
[    3.890181] ======================================================
[    3.896312] WARNING: possible circular locking dependency detected
[    3.902465] 6.4.0-rc6-next-20230615-00004-g8e95ecb4298d #83 Not tainted
[    3.909049] ------------------------------------------------------
[    3.915200] kworker/u16:0/10 is trying to acquire lock:
[    3.920398] c27200b0 (&data->lock){+.+.}-{3:3}, at: exynos_get_temp+0x3c/0xc8
[    3.927509] 
[    3.927509] but task is already holding lock:
[    3.933312] c2d59b94 (&tz->lock){+.+.}-{3:3}, at: __thermal_cooling_device_register.part.0+0x248/0x2b0
[    3.942595] 
[    3.942595] which lock already depends on the new lock.
[    3.942595] 
[    3.950739] 
[    3.950739] the existing dependency chain (in reverse order) is:
[    3.958187] 
[    3.958187] -> #1 (&tz->lock){+.+.}-{3:3}:
[    3.963727]        __mutex_lock+0xa0/0x95c
[    3.967794]        mutex_lock_nested+0x1c/0x24
[    3.972214]        thermal_zone_get_trip+0x20/0x44
[    3.976981]        exynos_tmu_initialize+0x148/0x1f0
[    3.981920]        exynos_tmu_probe+0x2b0/0x70c
[    3.986433]        platform_probe+0x5c/0xbc
[    3.990589]        really_probe+0xe0/0x400
[    3.994660]        __driver_probe_device+0x9c/0x1f0
[    3.999515]        driver_probe_device+0x30/0xc0
[    4.004107]        __driver_attach+0x124/0x1d4
[    4.008526]        bus_for_each_dev+0x70/0xc4
[    4.012859]        bus_add_driver+0xe0/0x200
[    4.017105]        driver_register+0x7c/0x114
[    4.021437]        do_one_initcall+0x6c/0x310
[    4.025770]        kernel_init_freeable+0x1c0/0x228
[    4.030624]        kernel_init+0x18/0x12c
[    4.034608]        ret_from_fork+0x14/0x2c
[    4.038681] 
[    4.038681] -> #0 (&data->lock){+.+.}-{3:3}:
[    4.044406]        __lock_acquire+0x152c/0x295c
[    4.048907]        lock_acquire+0x130/0x380
[    4.053065]        __mutex_lock+0xa0/0x95c
[    4.057138]        mutex_lock_nested+0x1c/0x24
[    4.061558]        exynos_get_temp+0x3c/0xc8
[    4.065804]        __thermal_zone_get_temp+0x48/0x15c
[    4.070832]        __thermal_zone_device_update.part.0+0x68/0x478
[    4.076900]        __thermal_cooling_device_register.part.0+0x268/0x2b0
[    4.083488]        of_devfreq_cooling_register_power+0x194/0x288
[    4.089466]        devfreq_cooling_em_register+0x40/0xa4
[    4.094750]        panfrost_devfreq_init+0x268/0x2d4
[    4.099688]        panfrost_device_init+0xf8/0x4fc
[    4.104453]        panfrost_probe+0xb0/0x180
[    4.108699]        platform_probe+0x5c/0xbc
[    4.112856]        really_probe+0xe0/0x400
[    4.116929]        __driver_probe_device+0x9c/0x1f0
[    4.121785]        driver_probe_device+0x30/0xc0
[    4.126376]        __device_attach_driver+0xa8/0x120
[    4.131317]        bus_for_each_drv+0x84/0xdc
[    4.135647]        __device_attach+0xb0/0x20c
[    4.139980]        bus_probe_device+0x8c/0x90
[    4.144312]        deferred_probe_work_func+0x98/0xe0
[    4.149341]        process_one_work+0x2b4/0x724
[    4.153845]        worker_thread+0x54/0x518
[    4.158003]        kthread+0x104/0x138
[    4.161729]        ret_from_fork+0x14/0x2c
[    4.165802] 
[    4.165802] other info that might help us debug this:
[    4.165802] 
[    4.173789]  Possible unsafe locking scenario:
[    4.173789] 
[    4.179674]        CPU0                    CPU1
[    4.184175]        ----                    ----
[    4.188679]   lock(&tz->lock);
[    4.191709]                                lock(&data->lock);
[    4.197436]                                lock(&tz->lock);
[    4.202982]   lock(&data->lock);
[    4.206181] 
[    4.206181]  *** DEADLOCK ***
[    4.206181] 
[    4.212084] 5 locks held by kworker/u16:0/10:
[    4.216409]  #0: c1c08ca8 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1f4/0x724
[    4.225785]  #1: f0891f20 (deferred_probe_work){+.+.}-{0:0}, at: process_one_work+0x1f4/0x724
[    4.234274]  #2: c245348c (&dev->mutex){....}-{3:3}, at: __device_attach+0x30/0x20c
[    4.241897]  #3: c1424f4c (thermal_list_lock){+.+.}-{3:3}, at: __thermal_cooling_device_register.part.0+0x15c/0x2b0
[    4.252304]  #4: c2d59b94 (&tz->lock){+.+.}-{3:3}, at: __thermal_cooling_device_register.part.0+0x248/0x2b0
[    4.262008] 
[    4.262008] stack backtrace:
[    4.266327] CPU: 5 PID: 10 Comm: kworker/u16:0 Not tainted 6.4.0-rc6-next-20230615-00004-g8e95ecb4298d #83
[    4.275959] Hardware name: Samsung Exynos (Flattened Device Tree)
[    4.282015] Workqueue: events_unbound deferred_probe_work_func
[    4.287823]  unwind_backtrace from show_stack+0x10/0x14
[    4.293015]  show_stack from dump_stack_lvl+0x58/0x70
[    4.298041]  dump_stack_lvl from check_noncircular+0x178/0x194
[    4.303849]  check_noncircular from __lock_acquire+0x152c/0x295c
[    4.309828]  __lock_acquire from lock_acquire+0x130/0x380
[    4.315201]  lock_acquire from __mutex_lock+0xa0/0x95c
[    4.320311]  __mutex_lock from mutex_lock_nested+0x1c/0x24
[    4.325771]  mutex_lock_nested from exynos_get_temp+0x3c/0xc8
[    4.331491]  exynos_get_temp from __thermal_zone_get_temp+0x48/0x15c
[    4.337818]  __thermal_zone_get_temp from __thermal_zone_device_update.part.0+0x68/0x478
[    4.345884]  __thermal_zone_device_update.part.0 from __thermal_cooling_device_register.part.0+0x268/0x2b0
[    4.355506]  __thermal_cooling_device_register.part.0 from of_devfreq_cooling_register_power+0x194/0x288
[    4.364952]  of_devfreq_cooling_register_power from devfreq_cooling_em_register+0x40/0xa4
[    4.373093]  devfreq_cooling_em_register from panfrost_devfreq_init+0x268/0x2d4
[    4.380371]  panfrost_devfreq_init from panfrost_device_init+0xf8/0x4fc
[    4.386952]  panfrost_device_init from panfrost_probe+0xb0/0x180
[    4.392930]  panfrost_probe from platform_probe+0x5c/0xbc
[    4.398300]  platform_probe from really_probe+0xe0/0x400
[    4.403585]  really_probe from __driver_probe_device+0x9c/0x1f0
[    4.409480]  __driver_probe_device from driver_probe_device+0x30/0xc0
[    4.415893]  driver_probe_device from __device_attach_driver+0xa8/0x120
[    4.422480]  __device_attach_driver from bus_for_each_drv+0x84/0xdc
[    4.428719]  bus_for_each_drv from __device_attach+0xb0/0x20c
[    4.434436]  __device_attach from bus_probe_device+0x8c/0x90
[    4.440068]  bus_probe_device from deferred_probe_work_func+0x98/0xe0
[    4.446482]  deferred_probe_work_func from process_one_work+0x2b4/0x724
[    4.453069]  process_one_work from worker_thread+0x54/0x518
[    4.458612]  worker_thread from kthread+0x104/0x138
[    4.463462]  kthread from ret_from_fork+0x14/0x2c
[    4.468141] Exception stack(0xf0891fb0 to 0xf0891ff8)
[    4.473170] 1fa0:                                     00000000 00000000 00000000 00000000
[    4.481327] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    4.489472] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[    4.496433] panfrost 11800000.gpu: mali-t620 id 0x620 major 0x0 minor 0x1 status 0x0
[    4.503774] panfrost 11800000.gpu: features: 00000000,00000408, issues: 00000000,21eee400
[    4.512036] panfrost 11800000.gpu: Features: L2:0x07110206 Shader:0x00000000 Tiler:0x00000809 Mem:0x1 MMU:0x00002830 AS:0xf JS:0x7
[    4.523683] panfrost 11800000.gpu: shader_present=0x3f l2_present=0x11
[    4.530576] panfrost 11800000.gpu: using only 1st core group (4 cores from 6)
[    4.542176] [drm] Initialized panfrost 1.2.0 20180908 for 11800000.gpu on minor 0
[    4.555151] xhci-hcd xhci-hcd.7.auto: xHCI Host Controller
[    4.559273] xhci-hcd xhci-hcd.7.auto: new USB bus registered, assigned bus number 3
[    4.567917] xhci-hcd xhci-hcd.7.auto: hcc params 0x0220f04c hci version 0x100 quirks 0x0000000002000010
[    4.576466] xhci-hcd xhci-hcd.7.auto: irq 142, io mem 0x12000000
[    4.582848] xhci-hcd xhci-hcd.7.auto: xHCI Host Controller
[    4.587712] xhci-hcd xhci-hcd.7.auto: new USB bus registered, assigned bus number 4
[    4.595329] xhci-hcd xhci-hcd.7.auto: Host supports USB 3.0 SuperSpeed
[    4.602196] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.04
[    4.610089] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    4.617255] usb usb3: Product: xHCI Host Controller
[    4.622033] usb usb3: Manufacturer: Linux 6.4.0-rc6-next-20230615-00004-g8e95ecb4298d xhci-hcd
[    4.630679] usb usb3: SerialNumber: xhci-hcd.7.auto
[    4.637045] hub 3-0:1.0: USB hub found
[    4.639425] hub 3-0:1.0: 1 port detected
[    4.644484] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[    4.651646] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.04
[    4.659620] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    4.667071] usb usb4: Product: xHCI Host Controller
[    4.671592] usb usb4: Manufacturer: Linux 6.4.0-rc6-next-20230615-00004-g8e95ecb4298d xhci-hcd
[    4.680249] usb usb4: SerialNumber: xhci-hcd.7.auto
[    4.686468] hub 4-0:1.0: USB hub found
[    4.688869] hub 4-0:1.0: 1 port detected
[    4.700645] xhci-hcd xhci-hcd.8.auto: xHCI Host Controller
[    4.704812] xhci-hcd xhci-hcd.8.auto: new USB bus registered, assigned bus number 5
[    4.713313] xhci-hcd xhci-hcd.8.auto: hcc params 0x0220f04c hci version 0x100 quirks 0x0000000002000010
[    4.721958] xhci-hcd xhci-hcd.8.auto: irq 143, io mem 0x12400000
[    4.728362] xhci-hcd xhci-hcd.8.auto: xHCI Host Controller
[    4.733140] xhci-hcd xhci-hcd.8.auto: new USB bus registered, assigned bus number 6
[    4.740834] xhci-hcd xhci-hcd.8.auto: Host supports USB 3.0 SuperSpeed
[    4.747691] usb usb5: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.04
[    4.755559] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    4.762673] usb usb5: Product: xHCI Host Controller
[    4.767580] usb usb5: Manufacturer: Linux 6.4.0-rc6-next-20230615-00004-g8e95ecb4298d xhci-hcd
[    4.776157] usb usb5: SerialNumber: xhci-hcd.8.auto
[    4.782432] hub 5-0:1.0: USB hub found
[    4.784894] hub 5-0:1.0: 1 port detected
[    4.789749] usb usb6: We don't know the algorithms for LPM for this host, disabling LPM.
[    4.797084] usb usb6: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.04
[    4.804999] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    4.812135] usb usb6: Product: xHCI Host Controller
[    4.817215] usb usb6: Manufacturer: Linux 6.4.0-rc6-next-20230615-00004-g8e95ecb4298d xhci-hcd
[    4.825614] usb usb6: SerialNumber: xhci-hcd.8.auto
[    4.831820] hub 6-0:1.0: USB hub found
[    4.834262] hub 6-0:1.0: 1 port detected
[    4.841535] s3c-rtc 101e0000.rtc: rtc disabled, re-enabling
[    4.846135] rtc rtc1: invalid alarm value: 1900-01-14T00:00:00
[    4.851943] s3c-rtc 101e0000.rtc: registered as rtc1
[    4.880372] cpu cpu0: EM: OPP:1200000 is inefficient
[    4.883926] cpu cpu0: EM: OPP:1100000 is inefficient
[    4.888904] cpu cpu0: EM: OPP:900000 is inefficient
[    4.893703] cpu cpu0: EM: OPP:800000 is inefficient
[    4.898597] cpu cpu0: EM: OPP:600000 is inefficient
[    4.903408] cpu cpu0: EM: OPP:500000 is inefficient
[    4.908301] cpu cpu0: EM: OPP:400000 is inefficient
[    4.913113] cpu cpu0: EM: OPP:200000 is inefficient
[    4.919737] cpu cpu0: EM: created perf domain
[    4.929774] cpu cpu4: EM: OPP:1700000 is inefficient
[    4.933324] cpu cpu4: EM: OPP:1600000 is inefficient
[    4.938405] cpu cpu4: EM: OPP:1400000 is inefficient
[    4.943192] cpu cpu4: EM: OPP:1300000 is inefficient
[    4.948195] cpu cpu4: EM: OPP:1100000 is inefficient
[    4.953070] cpu cpu4: EM: OPP:1000000 is inefficient
[    4.958056] cpu cpu4: EM: OPP:900000 is inefficient
[    4.962861] cpu cpu4: EM: OPP:700000 is inefficient
[    4.967756] cpu cpu4: EM: OPP:600000 is inefficient
[    4.972567] cpu cpu4: EM: OPP:500000 is inefficient
[    4.977492] cpu cpu4: EM: OPP:400000 is inefficient
[    4.982271] cpu cpu4: EM: OPP:300000 is inefficient
[    4.987192] cpu cpu4: EM: OPP:200000 is inefficient
[    4.994634] cpu cpu4: EM: created perf domain
[    5.042727] exynos-bus: new bus device registered: bus-wcore ( 88700 KHz ~ 532000 KHz)
[    5.049914] exynos-chipid 10000000.chipid: cpu0 opp0, freq: 1500 missing
[    5.056057] exynos-chipid 10000000.chipid: cpu4 opp0, freq: 2100 missing
[    5.062656] exynos-chipid 10000000.chipid: Exynos: CPU[EXYNOS5800] PRO_ID[0xe5422000] REV[0x1] Detected
[    5.073683] exynos-bus: new bus device registered: bus-disp1 (120000 KHz ~ 300000 KHz)
[    5.074296] usb 5-1: new high-speed USB device number 2 using xhci-hcd
[    5.085871] exynos-bus: new bus device registered: bus-disp1-fimd (120000 KHz ~ 200000 KHz)
[    5.096558] exynos-bus: new bus device registered: bus-fsys2 ( 75000 KHz ~ 200000 KHz)
[    5.104143] exynos-bus: new bus device registered: bus-fsys-apb (111000 KHz ~ 222000 KHz)
[    5.112741] exynos-bus: new bus device registered: bus-g2d ( 83250 KHz ~ 333000 KHz)
[    5.120484] exynos-bus: new bus device registered: bus-g2d-acp ( 66500 KHz ~ 266000 KHz)
[    5.128481] exynos-bus: new bus device registered: bus-gen ( 88700 KHz ~ 266000 KHz)
[    5.136058] exynos-bus: new bus device registered: bus-gscl-scaler (150000 KHz ~ 300000 KHz)
[    5.144593] exynos-bus: new bus device registered: bus-jpeg ( 75000 KHz ~ 300000 KHz)
[    5.152381] exynos-bus: new bus device registered: bus-jpeg-apb ( 83250 KHz ~ 166500 KHz)
[    5.161013] exynos-bus: new bus device registered: bus-mfc ( 83250 KHz ~ 333000 KHz)
[    5.168841] exynos-bus: new bus device registered: bus-mscl ( 84000 KHz ~ 666000 KHz)
[    5.176582] exynos-bus: new bus device registered: bus-noc ( 66600 KHz ~ 111000 KHz)
[    5.183773] exynos-bus: new bus device registered: bus-peri ( 66600 KHz ~  66600 KHz)
[    5.192931] Trying to probe devices needed for running init ...
[    5.266383] usb 5-1: New USB device found, idVendor=0bda, idProduct=8153, bcdDevice=30.00
[    5.273145] usb 5-1: New USB device strings: Mfr=1, Product=2, SerialNumber=6
[    5.280305] usb 5-1: Product: USB 10/100/1000 LAN
[    5.285124] usb 5-1: Manufacturer: Realtek
[    5.289037] usb 5-1: SerialNumber: 000001000000
[    5.516231] r8152-cfgselector 5-1: reset high-speed USB device number 2 using xhci-hcd
[    5.719723] r8152 5-1:1.0: Direct firmware load for rtl_nic/rtl8153a-3.fw failed with error -2
[    5.727018] r8152 5-1:1.0: unable to load firmware patch rtl_nic/rtl8153a-3.fw (-2)
[    5.777272] r8152 5-1:1.0 eth0: v1.12.13
[    9.504859] r8152 5-1:1.0 eth0: carrier on
[    9.544113] IP-Config: Complete:
[    9.545921]      device=eth0, hwaddr=00:1e:06:30:bf:ac, ipaddr=192.168.1.12, mask=255.255.255.0, gw=192.168.1.1
[    9.556219]      host=192.168.1.12, domain=, nis-domain=(none)
[    9.561825]      bootserver=192.168.1.10, rootserver=192.168.1.10, rootpath=
[    9.564891] clk: Disabling unused clocks
[    9.580459] ALSA device list:
[    9.581984]   No soundcards found.
[    9.590171] Freeing unused kernel image (initmem) memory: 1024K
[    9.635179] Run /init as init process
[    9.637406]   with arguments:
[    9.640348]     /init
[    9.642598]     nfsrootdebug
[    9.645688]   with environment:
[    9.648627]     HOME=/
[    9.650984]     TERM=linux
:: running early hook [udev]
Starting version 242.29-1-arch
:: running hook [udev]
:: Triggering uevents...
:: running hook [net_nfs4]
IP-Config: eth0 hardware address 00:1e:06:30:bf:ac mtu 1500
IP-Config: eth0 guessed broadcast address 192.168.1.255
IP-Config: eth0 complete (from 192.168.1.10):
 address: 192.168.1.12     broadcast: 192.168.1.255    netmask: 255.255.255.0   
 gateway: 192.168.1.1      dns0     : 0.0.0.0          dns1   : 0.0.0.0         
 rootserver: 192.168.1.10 rootpath: 
 filename  : 
NFS-Mount: 192.168.1.10:/srv/nfs/odroidhc1
Waiting 10 seconds for device /dev/nfs ...
ERROR: device '/dev/nfs' not found. Skipping fsck.
Mount cmd: 
mount.nfs4 -o vers=4,nolock 192.168.1.10:/srv/nfs/odroidhc1 /new_root
:: running cleanup hook [udev]
[   22.463514] 
[   22.463649] nfs_read_alloc_scratch(hdr=c2c15180, size=16) = c4320dc0
[   22.475450] nfs4_xdr_dec_read_plus(hdr=c2c15180, scratch=c4320dc0)
[   22.480441]     buf = {c4320dc0, 16}
[   22.483921]     buf = {c4320dc0, 16}
[   22.487712]     buf = {c4320dc0, 16}
[   22.491085]     buf = {c4320dc0, 16}
[   22.495006]     buf = {c4320dc0, 16}
[   22.498499] nfs_readhdr_free(hdr=c2c15180, scratch=c4320dc0)
[   22.508977] 
[   22.509056] nfs_read_alloc_scratch(hdr=c2c15400, size=16) = c4320f40
[   22.517942] nfs4_xdr_dec_read_plus(hdr=c2c15400, scratch=c4320f40)
[   22.522768]     buf = {c4320f40, 16}
[   22.526364]     buf = {c4320f40, 16}
[   22.529822]     buf = {c4320f40, 16}
[   22.533372]     buf = {c4320f40, 16}
[   22.537011]     buf = {c4320f40, 16}
[   22.540652] nfs_readhdr_free(hdr=c2c15400, scratch=c4320f40)
[   22.541734] 
[   22.547615] nfs_read_alloc_scratch(hdr=c2c15680, size=16) = c4320d40
[   22.558143] nfs4_xdr_dec_read_plus(hdr=c2c15680, scratch=c4320d40)
[   22.562951]     buf = {c4320d40, 16}
[   22.566564]     buf = {c4320d40, 16}
[   22.570014]     buf = {c4320d40, 16}
[   22.573560]     buf = {c4320d40, 16}
[   22.577376]     buf = {c4320d40, 16}
[   22.581029] nfs_readhdr_free(hdr=c2c15680, scratch=c4320d40)
[   22.581315] 
[   22.587967] nfs_read_alloc_scratch(hdr=c2c15900, size=16) = c4320540
[   22.598379] nfs4_xdr_dec_read_plus(hdr=c2c15900, scratch=c4320540)
[   22.603275]     buf = {c4320540, 16}
[   22.606857]     buf = {c4320540, 16}
[   22.610316]     buf = {c4320540, 16}
[   22.613836]     buf = {c4320540, 16}
[   22.617507]     buf = {c4320540, 16}
[   22.621250] nfs_readhdr_free(hdr=c2c15900, scratch=c4320540)
[   22.623226] 
[   22.628048] nfs_read_alloc_scratch(hdr=c2c15b80, size=16) = c4320b00
[   22.639506] nfs4_xdr_dec_read_plus(hdr=c2c15b80, scratch=c4320b00)
[   22.644431]     buf = {c4320b00, 16}
[   22.647837]     buf = {c4320b00, 16}
[   22.651398]     buf = {c4320b00, 16}
[   22.655080]     buf = {c4320b00, 16}
[   22.658531]     buf = {c4320b00, 16}
[   22.662366] nfs_readhdr_free(hdr=c2c15b80, scratch=c4320b00)
[   22.664550] 
[   22.669233] nfs_read_alloc_scratch(hdr=c2c15e00, size=16) = c4320380
[   22.678407] nfs4_xdr_dec_read_plus(hdr=c2c15e00, scratch=c4320380)
[   22.683170]     buf = {c4320380, 16}
[   22.686924]     buf = {c4320380, 16}
[   22.690312]     buf = {c4320380, 16}
[   22.693886]     buf = {c4320380, 16}
[   22.697491]     buf = {c4320380, 16}
[   22.701165] nfs_readhdr_free(hdr=c2c15e00, scratch=c4320380)
[   22.718745] 
[   22.718829] nfs_read_alloc_scratch(hdr=c2c16080, size=16) = c4320fc0
[   22.727562] nfs4_xdr_dec_read_plus(hdr=c2c16080, scratch=c4320fc0)
[   22.732420]     buf = {c4320fc0, 16}
[   22.735987]     buf = {c4320fc0, 16}
[   22.739458]     buf = {c4320fc0, 16}
[   22.743008]     buf = {c4320fc0, 16}
[   22.746630]     buf = {c4320fc0, 16}
[   22.750312] nfs_readhdr_free(hdr=c2c16080, scratch=c4320fc0)
[   22.752712] 
[   22.757275] nfs_read_alloc_scratch(hdr=c2c16300, size=16) = c4320500
[   22.767519] nfs4_xdr_dec_read_plus(hdr=c2c16300, scratch=c4320500)
[   22.772335]     buf = {c4320500, 16}
[   22.775929]     buf = {c4320500, 16}
[   22.779389]     buf = {c4320500, 16}
[   22.782936]     buf = {c4320500, 16}
[   22.786799]     buf = {c4320500, 16}
[   22.790417] nfs_readhdr_free(hdr=c2c16300, scratch=c4320500)
[   22.791088] 
[   22.797174] nfs_read_alloc_scratch(hdr=c2c16580, size=16) = c4320dc0
[   22.809766] nfs4_xdr_dec_read_plus(hdr=c2c16580, scratch=c4320dc0)
[   22.814829]     buf = {c4320dc0, 16}
[   22.818099]     buf = {c4320dc0, 16}
[   22.821648]     buf = {c4320dc0, 16}
[   22.825291]     buf = {c4320dc0, 16}
[   22.828802]     buf = {c4320dc0, 16}
[   22.832661] nfs_readhdr_free(hdr=c2c16580, scratch=c4320dc0)
[   22.835016] 
[   22.839447] nfs_read_alloc_scratch(hdr=c2c16800, size=16) = c4320800
[   22.848164] nfs4_xdr_dec_read_plus(hdr=c2c16800, scratch=c4320800)
[   22.852983]     buf = {c4320800, 16}
[   22.856948]     buf = {c4320800, 16}
[   22.860084]     buf = {c4320800, 16}
[   22.863652]     buf = {c4320800, 16}
[   22.867305]     buf = {c4320800, 16}
[   22.870944] nfs_readhdr_free(hdr=c2c16800, scratch=c4320800)
[   22.873194] 
[   22.877828] nfs_read_alloc_scratch(hdr=c2c16a80, size=16) = c43205c0
[   22.888677] nfs4_xdr_dec_read_plus(hdr=c2c16a80, scratch=c43205c0)
[   22.893516]     buf = {c43205c0, 16}
[   22.897097]     buf = {c43205c0, 16}
[   22.900565]     buf = {c43205c0, 16}
[   22.904178]     buf = {c43205c0, 16}
[   22.907694]     buf = {c43205c0, 16}
[   22.911503] nfs_readhdr_free(hdr=c2c16a80, scratch=c43205c0)
[   22.912395] 
[   22.918355] nfs_read_alloc_scratch(hdr=c2c16d00, size=16) = c4320e40
[   22.930449] nfs4_xdr_dec_read_plus(hdr=c2c16d00, scratch=c4320e40)
[   22.935274]     buf = {c4320e40, 16}
[   22.938750]     buf = {c4320e40, 16}
[   22.942300]     buf = {c4320e40, 16}
[   22.945897]     buf = {c4320e40, 16}
[   22.949450]     buf = {c4320e40, 16}
[   22.953323] nfs_readhdr_free(hdr=c2c16d00, scratch=c4320e40)
[   22.960209] 
[   22.960287] nfs_read_alloc_scratch(hdr=c2c16f80, size=16) = c4320d00
[   22.970187] nfs4_xdr_dec_read_plus(hdr=c2c16f80, scratch=c4320d00)
[   22.974970]     buf = {c4320d00, 16}
[   22.978454]     buf = {c4320d00, 16}
[   22.982007]     buf = {c4320d00, 16}
[   22.985600]     buf = {c4320d00, 16}
[   22.989133]     buf = {c4320d00, 16}
[   22.992791] nfs_readhdr_free(hdr=c2c16f80, scratch=c4320d00)
[   22.996851] 
[   22.999790] nfs_read_alloc_scratch(hdr=c410c000, size=16) = c4922700
[   23.008418] nfs4_xdr_dec_read_plus(hdr=c410c000, scratch=c4922700)
[   23.013141]     buf = {c4922700, 16}
[   23.016750]     buf = {c4922700, 16}
[   23.020242]     buf = {c4922700, 16}
[   23.023794]     buf = {c4922700, 16}
[   23.027577]     buf = {c4922700, 16}
[   23.031027] nfs_readhdr_free(hdr=c410c000, scratch=c4922700)
[   23.034545] 
[   23.038021] nfs_read_alloc_scratch(hdr=c2c17200, size=16) = c4320600
[   23.051879] nfs4_xdr_dec_read_plus(hdr=c2c17200, scratch=c4320600)
[   23.056657]     buf = {c4320600, 16}
[   23.060154]     buf = {c4320600, 16}
[   23.063707]     buf = {c4320600, 16}
[   23.067306]     buf = {c4320600, 16}
[   23.070855]     buf = {c4320600, 16}
[   23.074776] nfs_readhdr_free(hdr=c2c17200, scratch=c4320600)
[   23.082147] 
[   23.082223] nfs_read_alloc_scratch(hdr=c410c280, size=16) = c4922940
[   23.090980] nfs4_xdr_dec_read_plus(hdr=c410c280, scratch=c4922940)
[   23.095764]     buf = {c4922940, 16}
[   23.099257]     buf = {c4922940, 16}
[   23.102810]     buf = {c4922940, 16}
[   23.106416]     buf = {c4922940, 16}
[   23.109925]     buf = {c4922940, 16}
[   23.113571] nfs_readhdr_free(hdr=c410c280, scratch=c4922940)
[   23.117957] 
[   23.120593] nfs_read_alloc_scratch(hdr=c410c500, size=16) = c4922f40
[   23.133373] nfs4_xdr_dec_read_plus(hdr=c410c500, scratch=c4922f40)
[   23.138374]     buf = {c4922f40, 16}
[   23.141693]     buf = {c4922f40, 16}
[   23.145398]     buf = {c4922f40, 16}
[   23.148800]     buf = {c4922f40, 16}
[   23.152397]     buf = {c4922f40, 16}
[   23.156257] nfs_readhdr_free(hdr=c410c500, scratch=c4922f40)
[   23.157856] 
[   23.162982] nfs_read_alloc_scratch(hdr=c410c780, size=16) = c4922680
[   23.175910] nfs4_xdr_dec_read_plus(hdr=c410c780, scratch=c4922680)
[   23.180690]     buf = {c4922680, 16}
[   23.184264]     buf = {c4922680, 16}
[   23.187742]     buf = {c4922680, 16}
[   23.191294]     buf = {c4922680, 16}
[   23.194922]     buf = {c4922680, 16}
[   23.198679] nfs_readhdr_free(hdr=c410c780, scratch=c4922680)
[   23.208945] 
[   23.209010] nfs_read_alloc_scratch(hdr=c2c17480, size=16) = c43205c0
[   23.218191] nfs4_xdr_dec_read_plus(hdr=c2c17480, scratch=c43205c0)
[   23.222919]     buf = {c43205c0, 16}
[   23.226514]     buf = {c43205c0, 16}
[   23.230016]     buf = {c43205c0, 16}
[   23.233569]     buf = {c43205c0, 16}
[   23.237165]     buf = {c43205c0, 16}
[   23.240784] nfs_readhdr_free(hdr=c2c17480, scratch=c43205c0)
[   23.244632] 
[   23.247788] nfs_read_alloc_scratch(hdr=c410ca00, size=16) = c49228c0
[   23.259824] nfs4_xdr_dec_read_plus(hdr=c410ca00, scratch=c49228c0)
[   23.264635]     buf = {c49228c0, 16}
[   23.268101]     buf = {c49228c0, 16}
[   23.271654]     buf = {c49228c0, 16}
[   23.275347]     buf = {c49228c0, 16}
[   23.278826]     buf = {c49228c0, 16}
[   23.282553] nfs_readhdr_free(hdr=c410ca00, scratch=c49228c0)
[   23.294118] 
[   23.294191] nfs_read_alloc_scratch(hdr=c2c17700, size=16) = c4320d40
[   23.303200] nfs4_xdr_dec_read_plus(hdr=c2c17700, scratch=c4320d40)
[   23.308027]     buf = {c4320d40, 16}
[   23.311521]     buf = {c4320d40, 16}
[   23.315081]     buf = {c4320d40, 16}
[   23.318575]     buf = {c4320d40, 16}
[   23.322136]     buf = {c4320d40, 16}
[   23.325860] nfs_readhdr_free(hdr=c2c17700, scratch=c4320d40)
[   23.330059] 
[   23.332805] nfs_read_alloc_scratch(hdr=c2c17980, size=16) = c43200c0
[   23.342743] nfs4_xdr_dec_read_plus(hdr=c2c17980, scratch=c43200c0)
[   23.347526]     buf = {c43200c0, 16}
[   23.351012]     buf = {c43200c0, 16}
[   23.354616]     buf = {c43200c0, 16}
[   23.358118]     buf = {c43200c0, 16}
[   23.361689]     buf = {c43200c0, 16}
[   23.365553] nfs_readhdr_free(hdr=c2c17980, scratch=c43200c0)
[   23.377666] 
[   23.377756] nfs_read_alloc_scratch(hdr=c2c17c00, size=16) = c43209c0
[   23.386771] nfs4_xdr_dec_read_plus(hdr=c2c17c00, scratch=c43209c0)
[   23.391497]     buf = {c43209c0, 16}
[   23.395126]     buf = {c43209c0, 16}
[   23.398608]     buf = {c43209c0, 16}
[   23.402159]     buf = {c43209c0, 16}
[   23.405773]     buf = {c43209c0, 16}
[   23.409364] nfs_readhdr_free(hdr=c2c17c00, scratch=c43209c0)
[   23.412746] 
[   23.416486] nfs_read_alloc_scratch(hdr=c2c17980, size=16) = c4320640
[   23.427968] nfs4_xdr_dec_read_plus(hdr=c2c17980, scratch=c4320640)
[   23.432694]     buf = {c4320640, 16}
[   23.436322]     buf = {c4320640, 16}
[   23.439802]     buf = {c4320640, 16}
[   23.443354]     buf = {c4320640, 16}
[   23.446979]     buf = {c4320640, 16}
[   23.450671] nfs_readhdr_free(hdr=c2c17980, scratch=c4320640)
[   23.452524] 
[   23.457678] nfs_read_alloc_scratch(hdr=c2c17700, size=16) = c4320780
[   23.467825] nfs4_xdr_dec_read_plus(hdr=c2c17700, scratch=c4320780)
[   23.472555]     buf = {c4320780, 16}
[   23.476390]     buf = {c4320780, 16}
[   23.479283] 
[   23.479689]     buf = {c4320780, 16}
[   23.481181] nfs_read_alloc_scratch(hdr=c316c000, size=16) = c4843b40
[   23.484812]     buf = {c4320780, 16}
[   23.484838]     buf = {c4320780, 16}
[   23.484960] nfs_readhdr_free(hdr=c2c17700, scratch=c4320780)
[   23.507499] nfs4_xdr_dec_read_plus(hdr=c316c000, scratch=c4843b40)
[   23.512328]     buf = {c4843b40, 16}
[   23.515892]     buf = {c4843b40, 16}
[   23.519397]     buf = {c4843b40, 16}
[   23.522947]     buf = {c4843b40, 16}
[   23.526548]     buf = {c4843b40, 16}
[   23.530172] nfs_readhdr_free(hdr=c316c000, scratch=c4843b40)
[   23.533131] 
[   23.537265] nfs_read_alloc_scratch(hdr=c316c280, size=16) = c4843ac0
[   23.547263] nfs4_xdr_dec_read_plus(hdr=c316c280, scratch=c4843ac0)
[   23.552014]     buf = {c4843ac0, 16}
[   23.555620]     buf = {c4843ac0, 16}
[   23.559104]     buf = {c4843ac0, 16}
[   23.562657]     buf = {c4843ac0, 16}
[   23.566445]     buf = {c4843ac0, 16}
[   23.570122] nfs_readhdr_free(hdr=c316c280, scratch=c4843ac0)
[   23.571529] 
[   23.577174] nfs_read_alloc_scratch(hdr=c316c500, size=16) = c4843a40
[   23.585909] nfs4_xdr_dec_read_plus(hdr=c316c500, scratch=c4843a40)
[   23.590662]     buf = {c4843a40, 16}
[   23.594262]     buf = {c4843a40, 16}
[   23.597744]     buf = {c4843a40, 16}
[   23.599088] 
[   23.601279]     buf = {c4843a40, 16}
[   23.601296]     buf = {c4843a40, 16}
[   23.602755] nfs_read_alloc_scratch(hdr=c2738000, size=16) = c2535600
[   23.606455] nfs_readhdr_free(hdr=c316c500, scratch=c4843a40)
[   23.612409] nfs4_xdr_dec_read_plus(hdr=c2738000, scratch=c2535600)
[   23.628219]     buf = {c2535600, 16}
[   23.631553]     buf = {c2535600, 16}
[   23.635351]     buf = {c2535600, 16}
[   23.638706]     buf = {c2535600, 16}
[   23.642308]     buf = {c2535600, 16}
[   23.646034] nfs_readhdr_free(hdr=c2738000, scratch=c2535600)
[   23.648765] 
[   23.652910] nfs_read_alloc_scratch(hdr=c2738280, size=16) = c2535b40
[   23.663443] nfs4_xdr_dec_read_plus(hdr=c2738280, scratch=c2535b40)
[   23.668307]     buf = {c2535b40, 16}
[   23.671757]     buf = {c2535b40, 16}
[   23.675376]     buf = {c2535b40, 16}
[   23.678862]     buf = {c2535b40, 16}
[   23.682445]     buf = {c2535b40, 16}
[   23.686305] nfs_readhdr_free(hdr=c2738280, scratch=c2535b40)
[   23.687396] 
[   23.693098] nfs_read_alloc_scratch(hdr=c316c780, size=16) = cfdc5e80
[   23.704573] nfs4_xdr_dec_read_plus(hdr=c316c780, scratch=cfdc5e80)
[   23.709364]     buf = {cfdc5e80, 16}
[   23.712885]     buf = {cfdc5e80, 16}
[   23.716513]     buf = {cfdc5e80, 16}
[   23.719990]     buf = {cfdc5e80, 16}
[   23.723587]     buf = {cfdc5e80, 16}
[   23.727737] nfs_readhdr_free(hdr=c316c780, scratch=cfdc5e80)
[   23.737065] 
[   23.737186] nfs_read_alloc_scratch(hdr=c2738500, size=16) = c2535dc0
[   23.745462] nfs4_xdr_dec_read_plus(hdr=c2738500, scratch=c2535dc0)
[   23.750227]     buf = {c2535dc0, 16}
[   23.753759]     buf = {c2535dc0, 16}
[   23.757730]     buf = {c2535dc0, 16}
[   23.760920]     buf = {c2535dc0, 16}
[   23.764595]     buf = {c2535dc0, 16}
[   23.768185] nfs_readhdr_free(hdr=c2738500, scratch=c2535dc0)
[   23.770766] 
[   23.775264] nfs_read_alloc_scratch(hdr=c2738780, size=16) = c2535fc0
[   23.783655] nfs4_xdr_dec_read_plus(hdr=c2738780, scratch=c2535fc0)
[   23.788503]     buf = {c2535fc0, 16}
[   23.791964]     buf = {c2535fc0, 16}
[   23.795615]     buf = {c2535fc0, 16}
[   23.799069]     buf = {c2535fc0, 16}
[   23.802633]     buf = {c2535fc0, 16}
[   23.806608] nfs_readhdr_free(hdr=c2738780, scratch=c2535fc0)
[   23.816770] 
[   23.816886] nfs_read_alloc_scratch(hdr=c1cfc000, size=16) = c419b4c0
[   23.825406] nfs4_xdr_dec_read_plus(hdr=c1cfc000, scratch=c419b4c0)
[   23.830177]     buf = {c419b4c0, 16}
[   23.833707]     buf = {c419b4c0, 16}
[   23.837535]     buf = {c419b4c0, 16}
[   23.840864]     buf = {c419b4c0, 16}
[   23.844499]     buf = {c419b4c0, 16}
[   23.848202] nfs_readhdr_free(hdr=c1cfc000, scratch=c419b4c0)
[   23.855380] 
[   23.855502] nfs_read_alloc_scratch(hdr=c2738a00, size=16) = c2535a40
[   23.866459] nfs4_xdr_dec_read_plus(hdr=c2738a00, scratch=c2535a40)
[   23.871269]     buf = {c2535a40, 16}
[   23.874875]     buf = {c2535a40, 16}
[   23.878343]     buf = {c2535a40, 16}
[   23.881892]     buf = {c2535a40, 16}
[   23.885738]     buf = {c2535a40, 16}
[   23.889397] nfs_readhdr_free(hdr=c2738a00, scratch=c2535a40)
[   23.899433] 
[   23.899524] nfs_read_alloc_scratch(hdr=cfeac000, size=16) = c40b23c0
[   23.908636] nfs4_xdr_dec_read_plus(hdr=cfeac000, scratch=c40b23c0)
[   23.913492]     buf = {c40b23c0, 16}
[   23.917072]     buf = {c40b23c0, 16}
[   23.920536]     buf = {c40b23c0, 16}
[   23.924169]     buf = {c40b23c0, 16}
[   23.927661]     buf = {c40b23c0, 16}
[   23.931378] nfs_readhdr_free(hdr=cfeac000, scratch=c40b23c0)
[   23.933492] 
[   23.938314] nfs_read_alloc_scratch(hdr=cfeac280, size=16) = c40b2ec0
[   23.947186] nfs4_xdr_dec_read_plus(hdr=cfeac280, scratch=c40b2ec0)
[   23.951985]     buf = {c40b2ec0, 16}
[   23.955868]     buf = {c40b2ec0, 16}
[   23.959091]     buf = {c40b2ec0, 16}
[   23.962663]     buf = {c40b2ec0, 16}
[   23.966286]     buf = {c40b2ec0, 16}
[   23.969972] nfs_readhdr_free(hdr=cfeac280, scratch=c40b2ec0)
[   23.979033] 
[   23.979126] nfs_read_alloc_scratch(hdr=cfeac500, size=16) = c40b2e80
[   23.987741] nfs4_xdr_dec_read_plus(hdr=cfeac500, scratch=c40b2e80)
[   23.992573]     buf = {c40b2e80, 16}
[   23.996138]     buf = {c40b2e80, 16}
[   23.999619]     buf = {c40b2e80, 16}
[   24.003168]     buf = {c40b2e80, 16}
[   24.006825]     buf = {c40b2e80, 16}
[   24.010463] nfs_readhdr_free(hdr=cfeac500, scratch=c40b2e80)
[   24.012679] 
[   24.017416] nfs_read_alloc_scratch(hdr=cfeac780, size=16) = c40b2900
[   24.027747] nfs4_xdr_dec_read_plus(hdr=cfeac780, scratch=c40b2900)
[   24.032582]     buf = {c40b2900, 16}
[   24.036188]     buf = {c40b2900, 16}
[   24.039623]     buf = {c40b2900, 16}
[   24.043174]     buf = {c40b2900, 16}
[   24.046830]     buf = {c40b2900, 16}
[   24.050580] nfs_readhdr_free(hdr=cfeac780, scratch=c40b2900)
[   24.059729] 
[   24.059832] nfs_read_alloc_scratch(hdr=cfeaca00, size=16) = c40b2240
[   24.068446] nfs4_xdr_dec_read_plus(hdr=cfeaca00, scratch=c40b2240)
[   24.073281]     buf = {c40b2240, 16}
[   24.076947]     buf = {c40b2240, 16}
[   24.080325]     buf = {c40b2240, 16}
[   24.083875]     buf = {c40b2240, 16}
[   24.087719]     buf = {c40b2240, 16}
[   24.091196] nfs_readhdr_free(hdr=cfeaca00, scratch=c40b2240)
[   24.093381] 
[   24.098116] nfs_read_alloc_scratch(hdr=cfeacc80, size=16) = c40b2880
[   24.108813] nfs4_xdr_dec_read_plus(hdr=cfeacc80, scratch=c40b2880)
[   24.113835]     buf = {c40b2880, 16}
[   24.117400]     buf = {c40b2880, 16}
[   24.120823]     buf = {c40b2880, 16}
[   24.124443]     buf = {c40b2880, 16}
[   24.127969]     buf = {c40b2880, 16}
[   24.131825] nfs_readhdr_free(hdr=cfeacc80, scratch=c40b2880)
[   24.142099] 
[   24.142272] nfs_read_alloc_scratch(hdr=cfeacf00, size=16) = c40b2740
[   24.150889] nfs4_xdr_dec_read_plus(hdr=cfeacf00, scratch=c40b2740)
[   24.155863]     buf = {c40b2740, 16}
[   24.159208]     buf = {c40b2740, 16}
[   24.162762]     buf = {c40b2740, 16}
[   24.166388]     buf = {c40b2740, 16}
[   24.169890]     buf = {c40b2740, 16}
[   24.173604] nfs_readhdr_free(hdr=cfeacf00, scratch=c40b2740)
[   24.175853] 
[   24.180561] nfs_read_alloc_scratch(hdr=cfead180, size=16) = c40b2980
[   24.189010] nfs4_xdr_dec_read_plus(hdr=cfead180, scratch=c40b2980)
[   24.193845]     buf = {c40b2980, 16}
[   24.197419]     buf = {c40b2980, 16}
[   24.200887]     buf = {c40b2980, 16}
[   24.204720]     buf = {c40b2980, 16}
[   24.208047]     buf = {c40b2980, 16}
[   24.211792] nfs_readhdr_free(hdr=cfead180, scratch=c40b2980)
[   24.221076] 
[   24.221180] nfs_read_alloc_scratch(hdr=cfead400, size=16) = c40b2880
[   24.230503] nfs4_xdr_dec_read_plus(hdr=cfead400, scratch=c40b2880)
[   24.235510]     buf = {c40b2880, 16}
[   24.238896]     buf = {c40b2880, 16}
[   24.242415]     buf = {c40b2880, 16}
[   24.246112]     buf = {c40b2880, 16}
[   24.249754]     buf = {c40b2880, 16}
[   24.253255] nfs_readhdr_free(hdr=cfead400, scratch=c40b2880)
[   24.255565] 
[   24.260168] nfs_read_alloc_scratch(hdr=cfead680, size=16) = c40b20c0
[   24.270814] nfs4_xdr_dec_read_plus(hdr=cfead680, scratch=c40b20c0)
[   24.275773]     buf = {c40b20c0, 16}
[   24.279143]     buf = {c40b20c0, 16}
[   24.282698]     buf = {c40b20c0, 16}
[   24.286299]     buf = {c40b20c0, 16}
[   24.289832]     buf = {c40b20c0, 16}
[   24.293667] nfs_readhdr_free(hdr=cfead680, scratch=c40b20c0)
[   24.294292] 
[   24.300499] nfs_read_alloc_scratch(hdr=cfead900, size=16) = c40b2f40
[   24.310017] nfs4_xdr_dec_read_plus(hdr=cfead900, scratch=c40b2f40)
[   24.311901] 
[   24.315262]     buf = {c40b2f40, 16}
[   24.316231] nfs_read_alloc_scratch(hdr=cfeadb80, size=16) = c40b2280
[   24.318482] nfs4_xdr_dec_read_plus(hdr=cfeadb80, scratch=c40b2280)
[   24.319810]     buf = {c40b2f40, 16}
[   24.326213]     buf = {c40b2280, 16}
[   24.332261]     buf = {c40b2f40, 16}
[   24.335910]     buf = {c40b2280, 16}
[   24.339365]     buf = {c40b2f40, 16}
[   24.339403]     buf = {c40b2f40, 16}
[   24.342966]     buf = {c40b2280, 16}
[   24.346777] nfs_readhdr_free(hdr=cfead900, scratch=c40b2f40)
[   24.350024]     buf = {c40b2280, 16}
[   24.366686]     buf = {c40b2280, 16}
[   24.370073] nfs_readhdr_free(hdr=cfeadb80, scratch=c40b2280)
[   24.373387] 
[   24.377081] nfs_read_alloc_scratch(hdr=c2c17480, size=16) = c4320d40
[   24.388593] nfs4_xdr_dec_read_plus(hdr=c2c17480, scratch=c4320d40)
[   24.393316]     buf = {c4320d40, 16}
[   24.396906]     buf = {c4320d40, 16}
[   24.400414]     buf = {c4320d40, 16}
[   24.403967]     buf = {c4320d40, 16}
[   24.407590]     buf = {c4320d40, 16}
[   24.411271] nfs_readhdr_free(hdr=c2c17480, scratch=c4320d40)
[   24.413862] 
[   24.418195] nfs_read_alloc_scratch(hdr=c2c17200, size=16) = cfd82780
[   24.431270] nfs4_xdr_dec_read_plus(hdr=c2c17200, scratch=cfd82780)
[   24.436071]     buf = {cfd82780, 16}
[   24.439542]     buf = {cfd82780, 16}
[   24.443095]     buf = {cfd82780, 16}
[   24.446744]     buf = {cfd82780, 16}
[   24.450250]     buf = {cfd82780, 16}
[   24.454202] nfs_readhdr_free(hdr=c2c17200, scratch=cfd82780)
[   24.464096] 
[   24.464171] nfs_read_alloc_scratch(hdr=c2c16f80, size=16) = cfe3fb80
[   24.472910] nfs4_xdr_dec_read_plus(hdr=c2c16f80, scratch=cfe3fb80)
[   24.477759]     buf = {cfe3fb80, 16}
[   24.481271]     buf = {cfe3fb80, 16}
[   24.484853]     buf = {cfe3fb80, 16}
[   24.488355]     buf = {cfe3fb80, 16}
[   24.491890]     buf = {cfe3fb80, 16}
[   24.495627] nfs_readhdr_free(hdr=c2c16f80, scratch=cfe3fb80)
[   24.498997] 
[   24.502525] nfs_read_alloc_scratch(hdr=c2c16d00, size=16) = c431e080
[   24.514285] nfs4_xdr_dec_read_plus(hdr=c2c16d00, scratch=c431e080)
[   24.519010]     buf = {c431e080, 16}
[   24.522556]     buf = {c431e080, 16}
[   24.526165]     buf = {c431e080, 16}
[   24.529660]     buf = {c431e080, 16}
[   24.533238]     buf = {c431e080, 16}
[   24.537009] nfs_readhdr_free(hdr=c2c16d00, scratch=c431e080)
[   24.538649] 
[   24.543886] nfs_read_alloc_scratch(hdr=c2c16a80, size=16) = c431e600
[   24.555833] nfs4_xdr_dec_read_plus(hdr=c2c16a80, scratch=c431e600)
[   24.560576]     buf = {c431e600, 16}
[   24.564208] 
[   24.564223] nfs_read_alloc_scratch(hdr=c2c16800, size=16) = c431e900
[   24.565577]     buf = {c431e600, 16}
[   24.575556] nfs4_xdr_dec_read_plus(hdr=c2c16800, scratch=c431e900)
[   24.575567]     buf = {c431e900, 16}
[   24.581840]     buf = {c431e600, 16}
[   24.585209]     buf = {c431e900, 16}
[   24.585219]     buf = {c431e900, 16}
[   24.588847]     buf = {c431e600, 16}
[   24.592262]     buf = {c431e900, 16}
[   24.592281]     buf = {c431e900, 16}
[   24.596502]     buf = {c431e600, 16}
[   24.599555] nfs_readhdr_free(hdr=c2c16800, scratch=c431e900)
[   24.603046] 
[   24.603449] nfs_readhdr_free(hdr=c2c16a80, scratch=c431e600)
[   24.622902] nfs_read_alloc_scratch(hdr=c2c16580, size=16) = c431ee40
[   24.634835] nfs4_xdr_dec_read_plus(hdr=c2c16580, scratch=c431ee40)
[   24.639576]     buf = {c431ee40, 16}
[   24.643119]     buf = {c431ee40, 16}
[   24.646748]     buf = {c431ee40, 16}
[   24.650224]     buf = {c431ee40, 16}
[   24.653803]     buf = {c431ee40, 16}
[   24.657587] nfs_readhdr_free(hdr=c2c16580, scratch=c431ee40)
[   24.660414] 
[   24.664507] nfs_read_alloc_scratch(hdr=c2c16300, size=16) = c431e580
[   24.676176] nfs4_xdr_dec_read_plus(hdr=c2c16300, scratch=c431e580)
[   24.680908]     buf = {c431e580, 16}
[   24.684548]     buf = {c431e580, 16}
[   24.688006]     buf = {c431e580, 16}
[   24.691558]     buf = {c431e580, 16}
[   24.695332]     buf = {c431e580, 16}
[   24.699155] nfs_readhdr_free(hdr=c2c16300, scratch=c431e580)
[   24.704497] 
[   24.705767] nfs_read_alloc_scratch(hdr=c2c16080, size=16) = c431e3c0
[   24.715167] nfs4_xdr_dec_read_plus(hdr=c2c16080, scratch=c431e3c0)
[   24.719935]     buf = {c431e3c0, 16}
[   24.723501]     buf = {c431e3c0, 16}
[   24.727092]     buf = {c431e3c0, 16}
[   24.730605]     buf = {c431e3c0, 16}
[   24.734220]     buf = {c431e3c0, 16}
[   24.737798] nfs_readhdr_free(hdr=c2c16080, scratch=c431e3c0)
[   24.741006] 
[   24.744853] nfs_read_alloc_scratch(hdr=c2c15e00, size=16) = c431e040
[   24.756801] nfs4_xdr_dec_read_plus(hdr=c2c15e00, scratch=c431e040)
[   24.761529]     buf = {c431e040, 16}
[   24.765142]     buf = {c431e040, 16}
[   24.768627]     buf = {c431e040, 16}
[   24.772179]     buf = {c431e040, 16}
[   24.775860]     buf = {c431e040, 16}
[   24.779525] nfs_readhdr_free(hdr=c2c15e00, scratch=c431e040)
[   24.781340] 
[   24.786404] nfs_read_alloc_scratch(hdr=c2c15b80, size=16) = c431ea40
[   24.798001] nfs4_xdr_dec_read_plus(hdr=c2c15b80, scratch=c431ea40)
[   24.802726]     buf = {c431ea40, 16}
[   24.806349]     buf = {c431ea40, 16}
[   24.809822]     buf = {c431ea40, 16}
[   24.813374]     buf = {c431ea40, 16}
[   24.817117]     buf = {c431ea40, 16}
[   24.820950] nfs_readhdr_free(hdr=c2c15b80, scratch=c431ea40)
[   24.831710] 
[   24.831783] nfs_read_alloc_scratch(hdr=c2c15900, size=16) = c431ed00
[   24.840814] nfs4_xdr_dec_read_plus(hdr=c2c15900, scratch=c431ed00)
[   24.845628]     buf = {c431ed00, 16}
[   24.849095]     buf = {c431ed00, 16}
[   24.852646]     buf = {c431ed00, 16}
[   24.856245]     buf = {c431ed00, 16}
[   24.859757]     buf = {c431ed00, 16}
[   24.863403] nfs_readhdr_free(hdr=c2c15900, scratch=c431ed00)
[   24.866776] 
[   24.870440] nfs_read_alloc_scratch(hdr=cfeade00, size=16) = c40b22c0
[   24.878902] nfs4_xdr_dec_read_plus(hdr=cfeade00, scratch=c40b22c0)
[   24.883633]     buf = {c40b22c0, 16}
[   24.887296]     buf = {c40b22c0, 16}
[   24.890727]     buf = {c40b22c0, 16}
[   24.894414]     buf = {c40b22c0, 16}
[   24.897837]     buf = {c40b22c0, 16}
[   24.901477] nfs_readhdr_free(hdr=cfeade00, scratch=c40b22c0)
[   24.912619] 
[   24.912694] nfs_read_alloc_scratch(hdr=c2c15680, size=16) = c431e6c0
[   24.920976] nfs4_xdr_dec_read_plus(hdr=c2c15680, scratch=c431e6c0)
[   24.925775]     buf = {c431e6c0, 16}
[   24.929251]     buf = {c431e6c0, 16}
[   24.932803]     buf = {c431e6c0, 16}
[   24.936464]     buf = {c431e6c0, 16}
[   24.939917]     buf = {c431e6c0, 16}
[   24.943555] nfs_readhdr_free(hdr=c2c15680, scratch=c431e6c0)
[   24.947081] 
[   24.950584] nfs_read_alloc_scratch(hdr=c2c15400, size=16) = c431e900
[   24.962274] nfs4_xdr_dec_read_plus(hdr=c2c15400, scratch=c431e900)
[   24.967085]     buf = {c431e900, 16}
[   24.970554]     buf = {c431e900, 16}
[   24.974167]     buf = {c431e900, 16}
[   24.977658]     buf = {c431e900, 16}
[   24.981237]     buf = {c431e900, 16}
[   24.985039] nfs_readhdr_free(hdr=c2c15400, scratch=c431e900)
[   24.998230] 
[   24.998305] nfs_read_alloc_scratch(hdr=cfeae080, size=16) = c40b22c0
[   25.007835] nfs4_xdr_dec_read_plus(hdr=cfeae080, scratch=c40b22c0)
[   25.012834]     buf = {c40b22c0, 16}
[   25.016373]     buf = {c40b22c0, 16}
[   25.019792]     buf = {c40b22c0, 16}
[   25.023340]     buf = {c40b22c0, 16}
[   25.027203]     buf = {c40b22c0, 16}
[   25.030812] nfs_readhdr_free(hdr=cfeae080, scratch=c40b22c0)
[   25.032993] 
[   25.037581] nfs_read_alloc_scratch(hdr=cfeae300, size=16) = c40b2200
[   25.047808] nfs4_xdr_dec_read_plus(hdr=cfeae300, scratch=c40b2200)
[   25.052656]     buf = {c40b2200, 16}
[   25.056244]     buf = {c40b2200, 16}
[   25.059701]     buf = {c40b2200, 16}
[   25.063249]     buf = {c40b2200, 16}
[   25.066881]     buf = {c40b2200, 16}
[   25.070663] nfs_readhdr_free(hdr=cfeae300, scratch=c40b2200)
[   25.071847] 
[   25.077491] nfs_read_alloc_scratch(hdr=cfeae580, size=16) = c40b2e40
[   25.084447] 
[   25.085251] nfs_read_alloc_scratch(hdr=cfeae800, size=16) = c43205c0
[   25.086263] nfs4_xdr_dec_read_plus(hdr=cfeae580, scratch=c40b2e40)
[   25.096698] nfs4_xdr_dec_read_plus(hdr=cfeae800, scratch=c43205c0)
[   25.097842]     buf = {c40b2e40, 16}
[   25.103864]     buf = {c43205c0, 16}
[   25.107464]     buf = {c40b2e40, 16}
[   25.107484]     buf = {c40b2e40, 16}
[   25.107496]     buf = {c40b2e40, 16}
[   25.107525]     buf = {c40b2e40, 16}
[   25.110967]     buf = {c43205c0, 16}
[   25.114821] nfs_readhdr_free(hdr=cfeae580, scratch=c40b2e40)
[   25.115848] 
[   25.115867] nfs_read_alloc_scratch(hdr=cfeaea80, size=16) = c1ea5000
[   25.116907] 
[   25.116924] nfs_read_alloc_scratch(hdr=cfeaed00, size=16) = c1ea50c0
[   25.118072]     buf = {c43205c0, 16}
[   25.118080]     buf = {c43205c0, 16}
[   25.118121]     buf = {c43205c0, 16}
[   25.121710] nfs4_xdr_dec_read_plus(hdr=cfeaea80, scratch=c1ea5000)
[   25.121838] nfs4_xdr_dec_read_plus(hdr=cfeaed00, scratch=c1ea50c0)
[   25.121854]     buf = {c1ea50c0, 16}
[   25.121898]     buf = {c1ea50c0, 16}
[   25.121909]     buf = {c1ea50c0, 16}
[   25.121918]     buf = {c1ea50c0, 16}
[   25.121934]     buf = {c1ea50c0, 16}
[   25.122039] nfs_readhdr_free(hdr=cfeaed00, scratch=c1ea50c0)
[   25.125411] nfs_readhdr_free(hdr=cfeae800, scratch=c43205c0)
[   25.128810]     buf = {c1ea5000, 16}
[   25.205942]     buf = {c1ea5000, 16}
[   25.209113]     buf = {c1ea5000, 16}
[   25.212729]     buf = {c1ea5000, 16}
[   25.216360]     buf = {c1ea5000, 16}
[   25.220226] nfs_readhdr_free(hdr=cfeaea80, scratch=c1ea5000)
[   25.220402] 
[   25.226937] nfs_read_alloc_scratch(hdr=cfeaef80, size=16) = c1ea5340
[   25.233624] 
[   25.234781] nfs_read_alloc_scratch(hdr=cfeaf200, size=16) = c1ea5440
[   25.238739] nfs4_xdr_dec_read_plus(hdr=cfeaef80, scratch=c1ea5340)
[   25.245085] nfs4_xdr_dec_read_plus(hdr=cfeaf200, scratch=c1ea5440)
[   25.247157]     buf = {c1ea5340, 16}
[   25.253255]     buf = {c1ea5440, 16}
[   25.256846]     buf = {c1ea5340, 16}
[   25.256859]     buf = {c1ea5340, 16}
[   25.256868]     buf = {c1ea5340, 16}
[   25.256891]     buf = {c1ea5340, 16}
[   25.260361]     buf = {c1ea5440, 16}
[   25.264149] nfs_readhdr_free(hdr=cfeaef80, scratch=c1ea5340)
[   25.264611] 
[   25.264619] nfs_read_alloc_scratch(hdr=cfeaf480, size=16) = c1ea5ac0
[   25.265138] 
[   25.265146] nfs_read_alloc_scratch(hdr=cfeaf700, size=16) = c1ceb600
[   25.265472] 
[   25.265480] nfs_read_alloc_scratch(hdr=cfeaf980, size=16) = c1f41e80
[   25.270426]     buf = {c1ea5440, 16}
[   25.272002] nfs4_xdr_dec_read_plus(hdr=cfeaf480, scratch=c1ea5ac0)
[   25.277706]     buf = {c1ea5440, 16}
[   25.278125]     buf = {c1ea5ac0, 16}
[   25.278140]     buf = {c1ea5ac0, 16}
[   25.278150]     buf = {c1ea5ac0, 16}
[   25.278164]     buf = {c1ea5ac0, 16}
[   25.279669] nfs4_xdr_dec_read_plus(hdr=cfeaf700, scratch=c1ceb600)
[   25.279684]     buf = {c1ceb600, 16}
[   25.279693]     buf = {c1ceb600, 16}
[   25.279702]     buf = {c1ceb600, 16}
[   25.279711]     buf = {c1ceb600, 16}
[   25.279775]     buf = {c1ceb600, 16}
[   25.279834] nfs4_xdr_dec_read_plus(hdr=cfeaf980, scratch=c1f41e80)
[   25.279844]     buf = {c1f41e80, 16}
[   25.279853]     buf = {c1f41e80, 16}
[   25.279862]     buf = {c1f41e80, 16}
[   25.279871]     buf = {c1f41e80, 16}
[   25.279907]     buf = {c1f41e80, 16}
[   25.280183] nfs_readhdr_free(hdr=cfeaf980, scratch=c1f41e80)
[   25.280271] nfs_readhdr_free(hdr=cfeaf700, scratch=c1ceb600)
[   25.280967] 
[   25.280979] nfs_read_alloc_scratch(hdr=cfeafc00, size=16) = c430a4c0
[   25.281294] 
[   25.281302] nfs_read_alloc_scratch(hdr=cfeaf700, size=16) = c430a5c0
[   25.281833] 
[   25.281841] nfs_read_alloc_scratch(hdr=cfeaf980, size=16) = c430ae00
[   25.282659] 
[   25.282668] nfs_read_alloc_scratch(hdr=cfeaef80, size=16) = cfe50680
[   25.285594] 
[   25.286002]     buf = {c1ea5440, 16}
[   25.286037] nfs4_xdr_dec_read_plus(hdr=cfeaf700, scratch=c430a5c0)
[   25.286067]     buf = {c430a5c0, 16}
[   25.286092]     buf = {c430a5c0, 16}
[   25.286117]     buf = {c430a5c0, 16}
[   25.286141]     buf = {c430a5c0, 16}
[   25.286213]     buf = {c430a5c0, 16}
[   25.286211] nfs_readhdr_free(hdr=cfeaf200, scratch=c1ea5440)
[   25.286446] nfs_readhdr_free(hdr=cfeaf700, scratch=c430a5c0)
[   25.288951] nfs4_xdr_dec_read_plus(hdr=cfeafc00, scratch=c430a4c0)
[   25.288992]     buf = {c430a4c0, 16}
[   25.289007]     buf = {c430a4c0, 16}
[   25.289021]     buf = {c430a4c0, 16}
[   25.289033]     buf = {c430a4c0, 16}
[   25.289104]     buf = {c430a4c0, 16}
[   25.289601] nfs_readhdr_free(hdr=cfeafc00, scratch=c430a4c0)
[   25.291295] nfs4_xdr_dec_read_plus(hdr=cfeaf980, scratch=c430ae00)
[   25.291323]     buf = {c430ae00, 16}
[   25.291336]     buf = {c430ae00, 16}
[   25.291349]     buf = {c430ae00, 16}
[   25.291360]     buf = {c430ae00, 16}
[   25.291424]     buf = {c430ae00, 16}
[   25.291747]     buf = {c1ea5ac0, 16}
[   25.291850] nfs_readhdr_free(hdr=cfeaf980, scratch=c430ae00)
[   25.293027] nfs_read_alloc_scratch(hdr=cfeaea80, size=16) = cfe50ec0
[   25.295060] nfs4_xdr_dec_read_plus(hdr=cfeaef80, scratch=cfe50680)
[   25.295105]     buf = {cfe50680, 16}
[   25.295122]     buf = {cfe50680, 16}
[   25.295135]     buf = {cfe50680, 16}
[   25.295148]     buf = {cfe50680, 16}
[   25.295230]     buf = {cfe50680, 16}
[   25.295646] nfs_readhdr_free(hdr=cfeaef80, scratch=cfe50680)
[   25.299744] nfs_readhdr_free(hdr=cfeaf480, scratch=c1ea5ac0)
[   25.306839] nfs4_xdr_dec_read_plus(hdr=cfeaea80, scratch=cfe50ec0)
[   25.575963]     buf = {cfe50ec0, 16}
[   25.579018]     buf = {cfe50ec0, 16}
[   25.582565]     buf = {cfe50ec0, 16}
[   25.586214]     buf = {cfe50ec0, 16}
[   25.589720]     buf = {cfe50ec0, 16}
[   25.593616] nfs_readhdr_free(hdr=cfeaea80, scratch=cfe50ec0)
[   25.594594] 
[   25.600365] nfs_read_alloc_scratch(hdr=cfeae800, size=16) = cfe50b80
[   25.607738] 
[   25.608124] nfs_read_alloc_scratch(hdr=cfeaed00, size=16) = cfe50340
[   25.614631] nfs4_xdr_dec_read_plus(hdr=cfeae800, scratch=cfe50b80)
[   25.620625]     buf = {cfe50b80, 16}
[   25.624233]     buf = {cfe50b80, 16}
[   25.627713]     buf = {cfe50b80, 16}
[   25.631268]     buf = {cfe50b80, 16}
[   25.632349] nfs4_xdr_dec_read_plus(hdr=cfeaed00, scratch=cfe50340)
[   25.634894]     buf = {cfe50b80, 16}
[   25.635183] nfs_readhdr_free(hdr=cfeae800, scratch=cfe50b80)
[   25.635960] 
[   25.635973] nfs_read_alloc_scratch(hdr=cfeae580, size=16) = c2c0f140
[   25.640951]     buf = {cfe50340, 16}
[   25.644667] nfs4_xdr_dec_read_plus(hdr=cfeae580, scratch=c2c0f140)
[   25.650391]     buf = {cfe50340, 16}
[   25.651604]     buf = {c2c0f140, 16}
[   25.651621]     buf = {c2c0f140, 16}
[   25.651635]     buf = {c2c0f140, 16}
[   25.658137]     buf = {cfe50340, 16}
[   25.661480]     buf = {c2c0f140, 16}
[   25.661527]     buf = {c2c0f140, 16}
[   25.667758]     buf = {cfe50340, 16}
[   25.671394] nfs_readhdr_free(hdr=cfeae580, scratch=c2c0f140)
[   25.674943]     buf = {cfe50340, 16}
[   25.706086] nfs_readhdr_free(hdr=cfeaed00, scratch=cfe50340)
[   25.709657] 
[   25.712400] nfs_read_alloc_scratch(hdr=cfeae300, size=16) = c33010c0
[   25.723843] nfs4_xdr_dec_read_plus(hdr=cfeae300, scratch=c33010c0)
[   25.729027]     buf = {c33010c0, 16}
[   25.732188]     buf = {c33010c0, 16}
[   25.735846]     buf = {c33010c0, 16}
[   25.739289]     buf = {c33010c0, 16}
[   25.742897]     buf = {c33010c0, 16}
[   25.746765] nfs_readhdr_free(hdr=cfeae300, scratch=c33010c0)
[   25.747509] 
[   25.753525] nfs_read_alloc_scratch(hdr=cfeae080, size=16) = c3301940
[   25.761383] 
[   25.761458] nfs_read_alloc_scratch(hdr=cfeade00, size=16) = c3301000
[   25.765512] nfs4_xdr_dec_read_plus(hdr=cfeae080, scratch=c3301940)
[   25.773117] nfs4_xdr_dec_read_plus(hdr=cfeade00, scratch=c3301000)
[   25.773893]     buf = {c3301940, 16}
[   25.780095]     buf = {c3301000, 16}
[   25.780108]     buf = {c3301000, 16}
[   25.780119]     buf = {c3301000, 16}
[   25.780128]     buf = {c3301000, 16}
[   25.780186]     buf = {c3301000, 16}
[   25.783598]     buf = {c3301940, 16}
[   25.787591] nfs_readhdr_free(hdr=cfeade00, scratch=c3301000)
[   25.787993] 
[   25.788003] nfs_read_alloc_scratch(hdr=cfeadb80, size=16) = cffff080
[   25.790503] 
[   25.790518] nfs_read_alloc_scratch(hdr=cfead900, size=16) = cffff3c0
[   25.790708]     buf = {c3301940, 16}
[   25.794996] 
[   25.796362] nfs4_xdr_dec_read_plus(hdr=cfeadb80, scratch=cffff080)
[   25.796381]     buf = {cffff080, 16}
[   25.796395]     buf = {cffff080, 16}
[   25.796406]     buf = {cffff080, 16}
[   25.796418]     buf = {cffff080, 16}
[   25.796492]     buf = {cffff080, 16}
[   25.796934] nfs_readhdr_free(hdr=cfeadb80, scratch=cffff080)
[   25.797220] nfs4_xdr_dec_read_plus(hdr=cfead900, scratch=cffff3c0)
[   25.797235]     buf = {cffff3c0, 16}
[   25.797248]     buf = {cffff3c0, 16}
[   25.797258]     buf = {cffff3c0, 16}
[   25.797267]     buf = {cffff3c0, 16}
[   25.797293]     buf = {cffff3c0, 16}
[   25.797485] nfs_readhdr_free(hdr=cfead900, scratch=cffff3c0)
[   25.797807]     buf = {c3301940, 16}
[   25.801359] nfs_read_alloc_scratch(hdr=cfead680, size=16) = cffffc40
[   25.805062]     buf = {c3301940, 16}
[   25.815966] 
[   25.816198] nfs4_xdr_dec_read_plus(hdr=cfead680, scratch=cffffc40)
[   25.816214]     buf = {cffffc40, 16}
[   25.816226]     buf = {cffffc40, 16}
[   25.816235]     buf = {cffffc40, 16}
[   25.816243]     buf = {cffffc40, 16}
[   25.816295]     buf = {cffffc40, 16}
[   25.816581] nfs_readhdr_free(hdr=cfead680, scratch=cffffc40)
[   25.818938] nfs_readhdr_free(hdr=cfeae080, scratch=c3301940)
[   25.819813] nfs_read_alloc_scratch(hdr=cfead400, size=16) = c2c0f180
[   25.947570] 
[   25.948157] nfs_read_alloc_scratch(hdr=cfead180, size=16) = c1ea5400
[   25.954933] nfs4_xdr_dec_read_plus(hdr=cfead400, scratch=c2c0f180)
[   25.960642]     buf = {c2c0f180, 16}
[   25.964241]     buf = {c2c0f180, 16}
[   25.965814] nfs4_xdr_dec_read_plus(hdr=cfead180, scratch=c1ea5400)
[   25.967732]     buf = {c2c0f180, 16}
[   25.973883]     buf = {c1ea5400, 16}
[   25.977462]     buf = {c2c0f180, 16}
[   25.980989]     buf = {c1ea5400, 16}
[   25.985038]     buf = {c2c0f180, 16}
[   25.988097]     buf = {c1ea5400, 16}
[   25.992214] nfs_readhdr_free(hdr=cfead400, scratch=c2c0f180)
[   25.995248]     buf = {c1ea5400, 16}
[   26.004604]     buf = {c1ea5400, 16}
[   26.008285] nfs_readhdr_free(hdr=cfead180, scratch=c1ea5400)
[   26.012569] 
[   26.015155] nfs_read_alloc_scratch(hdr=c2c15180, size=16) = c431e680
[   26.026277] nfs4_xdr_dec_read_plus(hdr=c2c15180, scratch=c431e680)
[   26.031006]     buf = {c431e680, 16}
[   26.034608]     buf = {c431e680, 16}
[   26.038107]     buf = {c431e680, 16}
[   26.041663]     buf = {c431e680, 16}
[   26.045398]     buf = {c431e680, 16}
[   26.048958] nfs_readhdr_free(hdr=c2c15180, scratch=c431e680)
[   26.054963] 
[   26.055866] nfs_read_alloc_scratch(hdr=cfeacf00, size=16) = c1ea5440
[   26.065441] 
[   26.065530] nfs_read_alloc_scratch(hdr=cfeacc80, size=16) = cfe50500
[   26.073720] nfs4_xdr_dec_read_plus(hdr=cfeacf00, scratch=c1ea5440)
[   26.078634]     buf = {c1ea5440, 16}
[   26.082067]     buf = {c1ea5440, 16}
[   26.085715]     buf = {c1ea5440, 16}
[   26.086829] nfs4_xdr_dec_read_plus(hdr=cfeacc80, scratch=cfe50500)
[   26.089178]     buf = {c1ea5440, 16}
[   26.095297]     buf = {cfe50500, 16}
[   26.095308]     buf = {cfe50500, 16}
[   26.095318]     buf = {cfe50500, 16}
[   26.099427]     buf = {c1ea5440, 16}
[   26.102358]     buf = {cfe50500, 16}
[   26.102622]     buf = {cfe50500, 16}
[   26.107855] nfs_readhdr_free(hdr=cfeacf00, scratch=c1ea5440)
[   26.109065] 
[   26.109083] nfs_read_alloc_scratch(hdr=cfeaca00, size=16) = cfe505c0
[   26.109793] nfs_readhdr_free(hdr=cfeacc80, scratch=cfe50500)
[   26.110697] 
[   26.110714] nfs_read_alloc_scratch(hdr=cfeac780, size=16) = cfe50cc0
[   26.111744] nfs4_xdr_dec_read_plus(hdr=cfeaca00, scratch=cfe505c0)
[   26.111764]     buf = {cfe505c0, 16}
[   26.111782]     buf = {cfe505c0, 16}
[   26.111800]     buf = {cfe505c0, 16}
[   26.111818]     buf = {cfe505c0, 16}
[   26.111842]     buf = {cfe505c0, 16}
[   26.111982] nfs_readhdr_free(hdr=cfeaca00, scratch=cfe505c0)
[   26.122142] nfs4_xdr_dec_read_plus(hdr=cfeac780, scratch=cfe50cc0)
[   26.182784]     buf = {cfe50cc0, 16}
[   26.186325]     buf = {cfe50cc0, 16}
[   26.189802]     buf = {cfe50cc0, 16}
[   26.193355]     buf = {cfe50cc0, 16}
[   26.196981]     buf = {cfe50cc0, 16}
[   26.200687] nfs_readhdr_free(hdr=cfeac780, scratch=cfe50cc0)
[   26.203035] 
[   26.207578] nfs_read_alloc_scratch(hdr=cfeac500, size=16) = cfe50a40
[   26.221561] nfs4_xdr_dec_read_plus(hdr=cfeac500, scratch=cfe50a40)
[   26.226345]     buf = {cfe50a40, 16}
[   26.229834]     buf = {cfe50a40, 16}
[   26.233386]     buf = {cfe50a40, 16}
[   26.237046]     buf = {cfe50a40, 16}
[   26.240540]     buf = {cfe50a40, 16}
[   26.244445] nfs_readhdr_free(hdr=cfeac500, scratch=cfe50a40)
[   26.247041] 
[   26.251166] nfs_read_alloc_scratch(hdr=cfeac280, size=16) = cfe50e40
[   26.264601] nfs4_xdr_dec_read_plus(hdr=cfeac280, scratch=cfe50e40)
[   26.269328]     buf = {cfe50e40, 16}
[   26.272875]     buf = {cfe50e40, 16}
[   26.276473]     buf = {cfe50e40, 16}
[   26.279981]     buf = {cfe50e40, 16}
[   26.283580]     buf = {cfe50e40, 16}
[   26.287484] nfs_readhdr_free(hdr=cfeac280, scratch=cfe50e40)
[   26.292821] 
[   26.294335] nfs_read_alloc_scratch(hdr=c2c15180, size=16) = c431eb40
[   26.305964] nfs4_xdr_dec_read_plus(hdr=c2c15180, scratch=c431eb40)
[   26.310824]     buf = {c431eb40, 16}
[   26.314399]     buf = {c431eb40, 16}
[   26.317919]     buf = {c431eb40, 16}
[   26.321448]     buf = {c431eb40, 16}
[   26.325103]     buf = {c431eb40, 16}
[   26.328931] nfs_readhdr_free(hdr=c2c15180, scratch=c431eb40)
[   26.334905] 
[   26.335617] nfs_read_alloc_scratch(hdr=c2c15400, size=16) = c431e240
[   26.350476] nfs4_xdr_dec_read_plus(hdr=c2c15400, scratch=c431e240)
[   26.355283]     buf = {c431e240, 16}
[   26.358748]     buf = {c431e240, 16}
[   26.362300]     buf = {c431e240, 16}
[   26.365886]     buf = {c431e240, 16}
[   26.369451]     buf = {c431e240, 16}
[   26.373289] nfs_readhdr_free(hdr=c2c15400, scratch=c431e240)
[   26.375789] 
[   26.380079] nfs_read_alloc_scratch(hdr=c2c15680, size=16) = c431e3c0
[   26.391438] nfs4_xdr_dec_read_plus(hdr=c2c15680, scratch=c431e3c0)
[   26.396218]     buf = {c431e3c0, 16}
[   26.399710]     buf = {c431e3c0, 16}
[   26.403263]     buf = {c431e3c0, 16}
[   26.406962]     buf = {c431e3c0, 16}
[   26.410430]     buf = {c431e3c0, 16}
[   26.414473] nfs_readhdr_free(hdr=c2c15680, scratch=c431e3c0)
[   26.417327] 
[   26.421042] nfs_read_alloc_scratch(hdr=c2c15900, size=16) = c431e700
[   26.436488] nfs4_xdr_dec_read_plus(hdr=c2c15900, scratch=c431e700)
[   26.441217]     buf = {c431e700, 16}
[   26.444830]     buf = {c431e700, 16}
[   26.448314]     buf = {c431e700, 16}
[   26.451866]     buf = {c431e700, 16}
[   26.455505]     buf = {c431e700, 16}
[   26.459263] nfs_readhdr_free(hdr=c2c15900, scratch=c431e700)
[   26.461510] 
[   26.466093] nfs_read_alloc_scratch(hdr=c2c15b80, size=16) = c431e900
[   26.478407] nfs4_xdr_dec_read_plus(hdr=c2c15b80, scratch=c431e900)
[   26.483131]     buf = {c431e900, 16}
[   26.486725]     buf = {c431e900, 16}
[   26.490229]     buf = {c431e900, 16}
[   26.493780]     buf = {c431e900, 16}
[   26.497528]     buf = {c431e900, 16}
[   26.501325] nfs_readhdr_free(hdr=c2c15b80, scratch=c431e900)
[   26.502544] 
[   26.508007] nfs_read_alloc_scratch(hdr=c2c15e00, size=16) = c431e1c0
[   26.522742] nfs4_xdr_dec_read_plus(hdr=c2c15e00, scratch=c431e1c0)
[   26.527515]     buf = {c431e1c0, 16}
[   26.531016]     buf = {c431e1c0, 16}
[   26.534600]     buf = {c431e1c0, 16}
[   26.538121]     buf = {c431e1c0, 16}
[   26.541717]     buf = {c431e1c0, 16}
[   26.545556] nfs_readhdr_free(hdr=c2c15e00, scratch=c431e1c0)
[   26.546941] 
[   26.552347] nfs_read_alloc_scratch(hdr=c2c16080, size=16) = c431e380
[   26.563784] nfs4_xdr_dec_read_plus(hdr=c2c16080, scratch=c431e380)
[   26.568570]     buf = {c431e380, 16}
[   26.572060]     buf = {c431e380, 16}
[   26.575651]     buf = {c431e380, 16}
[   26.579163]     buf = {c431e380, 16}
[   26.582745]     buf = {c431e380, 16}
[   26.586788] nfs_readhdr_free(hdr=c2c16080, scratch=c431e380)
[   26.587969] 
[   26.593388] nfs_read_alloc_scratch(hdr=c2c16300, size=16) = c431e040
[   26.608223] nfs4_xdr_dec_read_plus(hdr=c2c16300, scratch=c431e040)
[   26.613135]     buf = {c431e040, 16}
[   26.616934]     buf = {c431e040, 16}
[   26.620111]     buf = {c431e040, 16}
[   26.623646]     buf = {c431e040, 16}
[   26.627330]     buf = {c431e040, 16}
[   26.631189] nfs_readhdr_free(hdr=c2c16300, scratch=c431e040)
[   26.632532] 
[   26.637836] nfs_read_alloc_scratch(hdr=c2c16580, size=16) = c431ec00
[   26.648074] nfs4_xdr_dec_read_plus(hdr=c2c16580, scratch=c431ec00)
[   26.652853]     buf = {c431ec00, 16}
[   26.656451]     buf = {c431ec00, 16}
[   26.659975]     buf = {c431ec00, 16}
[   26.663492]     buf = {c431ec00, 16}
[   26.667153]     buf = {c431ec00, 16}
[   26.670770] nfs_readhdr_free(hdr=c2c16580, scratch=c431ec00)
[   26.672225] 
[   26.677676] nfs_read_alloc_scratch(hdr=c2c16800, size=16) = c431e740
[   26.690475] nfs4_xdr_dec_read_plus(hdr=c2c16800, scratch=c431e740)
[   26.695271]     buf = {c431e740, 16}
[   26.698752]     buf = {c431e740, 16}
[   26.702304]     buf = {c431e740, 16}
[   26.705959]     buf = {c431e740, 16}
[   26.709455]     buf = {c431e740, 16}
[   26.713361] nfs_readhdr_free(hdr=c2c16800, scratch=c431e740)
[   26.716145] 
[   26.720084] nfs_read_alloc_scratch(hdr=c2c16a80, size=16) = c431e400
[   26.733651] nfs4_xdr_dec_read_plus(hdr=c2c16a80, scratch=c431e400)
[   26.738431]     buf = {c431e400, 16}
[   26.741926]     buf = {c431e400, 16}
[   26.745514]     buf = {c431e400, 16}
[   26.749030]     buf = {c431e400, 16}
[   26.752617]     buf = {c431e400, 16}
[   26.756426] nfs_readhdr_free(hdr=c2c16a80, scratch=c431e400)
[   26.759415] 
[   26.763302] nfs_read_alloc_scratch(hdr=c2c16d00, size=16) = c431ee00
[   26.775934] nfs4_xdr_dec_read_plus(hdr=c2c16d00, scratch=c431ee00)
[   26.780745]     buf = {c431ee00, 16}
[   26.784333]     buf = {c431ee00, 16}
[   26.787810]     buf = {c431ee00, 16}
[   26.791424]     buf = {c431ee00, 16}
[   26.794977]     buf = {c431ee00, 16}
[   26.798665] nfs_readhdr_free(hdr=c2c16d00, scratch=c431ee00)
[   26.800550] 
[   26.805542] nfs_read_alloc_scratch(hdr=c2c16f80, size=16) = c431eac0
[   26.819846] nfs4_xdr_dec_read_plus(hdr=c2c16f80, scratch=c431eac0)
[   26.824670]     buf = {c431eac0, 16}
[   26.828288]     buf = {c431eac0, 16}
[   26.831671]     buf = {c431eac0, 16}
[   26.835313]     buf = {c431eac0, 16}
[   26.838808]     buf = {c431eac0, 16}
[   26.842577] nfs_readhdr_free(hdr=c2c16f80, scratch=c431eac0)
[   26.843894] 
[   26.849446] nfs_read_alloc_scratch(hdr=c2c17200, size=16) = c431ed40
[   26.861910] nfs4_xdr_dec_read_plus(hdr=c2c17200, scratch=c431ed40)
[   26.866691]     buf = {c431ed40, 16}
[   26.870181]     buf = {c431ed40, 16}
[   26.873734]     buf = {c431ed40, 16}
[   26.877427]     buf = {c431ed40, 16}
[   26.880901]     buf = {c431ed40, 16}
[   26.884713] nfs_readhdr_free(hdr=c2c17200, scratch=c431ed40)
[   26.888476] 
[   26.891514] nfs_read_alloc_scratch(hdr=c2c17480, size=16) = c431eac0
[   26.906476] nfs4_xdr_dec_read_plus(hdr=c2c17480, scratch=c431eac0)
[   26.911272]     buf = {c431eac0, 16}
[   26.914888]     buf = {c431eac0, 16}
[   26.918352]     buf = {c431eac0, 16}
[   26.921913]     buf = {c431eac0, 16}
[   26.925612]     buf = {c431eac0, 16}
[   26.929422] nfs_readhdr_free(hdr=c2c17480, scratch=c431eac0)
[   26.937332] 
[   26.937407] nfs_read_alloc_scratch(hdr=c2c17700, size=16) = c3301900
[   26.951173] nfs4_xdr_dec_read_plus(hdr=c2c17700, scratch=c3301900)
[   26.956182]     buf = {c3301900, 16}
[   26.959492]     buf = {c3301900, 16}
[   26.963066]     buf = {c3301900, 16}
[   26.966658]     buf = {c3301900, 16}
[   26.970239]     buf = {c3301900, 16}
[   26.974227] nfs_readhdr_free(hdr=c2c17700, scratch=c3301900)
[   26.978465] 
[   26.980789] nfs_read_alloc_scratch(hdr=c2c17980, size=16) = c3301200
[   26.992712] nfs4_xdr_dec_read_plus(hdr=c2c17980, scratch=c3301200)
[   26.997717]     buf = {c3301200, 16}
[   27.000993]     buf = {c3301200, 16}
[   27.004597]     buf = {c3301200, 16}
[   27.008096]     buf = {c3301200, 16}
[   27.011676]     buf = {c3301200, 16}
[   27.015439] nfs_readhdr_free(hdr=c2c17980, scratch=c3301200)
[   27.018320] 
[   27.022319] nfs_read_alloc_scratch(hdr=c2c17c00, size=16) = c3301400
[   27.036513] nfs4_xdr_dec_read_plus(hdr=c2c17c00, scratch=c3301400)
[   27.041234]     buf = {c3301400, 16}
[   27.044828]     buf = {c3301400, 16}
[   27.048332]     buf = {c3301400, 16}
[   27.051884]     buf = {c3301400, 16}
[   27.055564]     buf = {c3301400, 16}
[   27.059229] nfs_readhdr_free(hdr=c2c17c00, scratch=c3301400)
[   27.060387] 
[   27.066109] nfs_read_alloc_scratch(hdr=c2c17980, size=16) = c3301f00
[   27.078948] nfs4_xdr_dec_read_plus(hdr=c2c17980, scratch=c3301f00)
[   27.083667]     buf = {c3301f00, 16}
[   27.087264]     buf = {c3301f00, 16}
[   27.090766]     buf = {c3301f00, 16}
[   27.094358]     buf = {c3301f00, 16}
[   27.097909]     buf = {c3301f00, 16}
[   27.101691] nfs_readhdr_free(hdr=c2c17980, scratch=c3301f00)
[   27.103450] 
[   27.108543] nfs_read_alloc_scratch(hdr=c2c17700, size=16) = c3301380
[   27.122269] nfs4_xdr_dec_read_plus(hdr=c2c17700, scratch=c3301380)
[   27.127111]     buf = {c3301380, 16}
[   27.130587]     buf = {c3301380, 16}
[   27.134208]     buf = {c3301380, 16}
[   27.137690]     buf = {c3301380, 16}
[   27.141297]     buf = {c3301380, 16}
[   27.145433] nfs_readhdr_free(hdr=c2c17700, scratch=c3301380)
[   27.147695] 
[   27.151875] nfs_read_alloc_scratch(hdr=c2c17480, size=16) = c33014c0
[   27.163586] nfs4_xdr_dec_read_plus(hdr=c2c17480, scratch=c33014c0)
[   27.168424]     buf = {c33014c0, 16}
[   27.171902]     buf = {c33014c0, 16}
[   27.175546]     buf = {c33014c0, 16}
[   27.179009]     buf = {c33014c0, 16}
[   27.182597]     buf = {c33014c0, 16}
[   27.186323] nfs_readhdr_free(hdr=c2c17480, scratch=c33014c0)
[   27.187665] 
[   27.193198] nfs_read_alloc_scratch(hdr=c2c17200, size=16) = c3301000
[   27.205375] nfs4_xdr_dec_read_plus(hdr=c2c17200, scratch=c3301000)
[   27.210097]     buf = {c3301000, 16}
[   27.213642]     buf = {c3301000, 16}
[   27.217251]     buf = {c3301000, 16}
[   27.220748]     buf = {c3301000, 16}
[   27.224378]     buf = {c3301000, 16}
[   27.228108] nfs_readhdr_free(hdr=c2c17200, scratch=c3301000)
[   27.229701] 
[   27.235019] nfs_read_alloc_scratch(hdr=c2c16f80, size=16) = c3301d40
[   27.246885] nfs4_xdr_dec_read_plus(hdr=c2c16f80, scratch=c3301d40)
[   27.251643]     buf = {c3301d40, 16}
[   27.255262]     buf = {c3301d40, 16}
[   27.258742]     buf = {c3301d40, 16}
[   27.262309]     buf = {c3301d40, 16}
[   27.265941]     buf = {c3301d40, 16}
[   27.269626] nfs_readhdr_free(hdr=c2c16f80, scratch=c3301d40)
[   27.270588] 
[   27.276481] nfs_read_alloc_scratch(hdr=c2c16d00, size=16) = c33011c0
[   27.290941] nfs4_xdr_dec_read_plus(hdr=c2c16d00, scratch=c33011c0)
[   27.295704]     buf = {c33011c0, 16}
[   27.299203]     buf = {c33011c0, 16}
[   27.302756]     buf = {c33011c0, 16}
[   27.306397]     buf = {c33011c0, 16}
[   27.309901]     buf = {c33011c0, 16}
[   27.313689] nfs_readhdr_free(hdr=c2c16d00, scratch=c33011c0)
[   27.316010] 
[   27.320532] nfs_read_alloc_scratch(hdr=c2c16a80, size=16) = c3301400
[   27.332233] nfs4_xdr_dec_read_plus(hdr=c2c16a80, scratch=c3301400)
[   27.337015]     buf = {c3301400, 16}
[   27.340506]     buf = {c3301400, 16}
[   27.344109]     buf = {c3301400, 16}
[   27.347611]     buf = {c3301400, 16}
[   27.351187]     buf = {c3301400, 16}
[   27.354998] nfs_readhdr_free(hdr=c2c16a80, scratch=c3301400)
[   27.359187] 
[   27.361838] nfs_read_alloc_scratch(hdr=c2c16800, size=16) = c3301580
[   27.369885] nfs4_xdr_dec_read_plus(hdr=c2c16800, scratch=c3301580)
[   27.374914]     buf = {c3301580, 16}
[   27.378205]     buf = {c3301580, 16}
[   27.381780]     buf = {c3301580, 16}
[   27.385359]     buf = {c3301580, 16}
[   27.388889]     buf = {c3301580, 16}
[   27.392500] nfs_readhdr_free(hdr=c2c16800, scratch=c3301580)
[   27.396072] 
[   27.399494] nfs_read_alloc_scratch(hdr=c2c16580, size=16) = c3301900
[   27.410416] nfs4_xdr_dec_read_plus(hdr=c2c16580, scratch=c3301900)
[   27.415182]     buf = {c3301900, 16}
[   27.418684]     buf = {c3301900, 16}
[   27.422236]     buf = {c3301900, 16}
[   27.425823]     buf = {c3301900, 16}
[   27.429357]     buf = {c3301900, 16}
[   27.433036] nfs_readhdr_free(hdr=c2c16580, scratch=c3301900)
[   27.436415] systemd[1]: System time before build time, advancing clock.
[   27.445420] 
[   27.446586] nfs_read_alloc_scratch(hdr=c2c16300, size=16) = c3301b40
[   27.457870] nfs4_xdr_dec_read_plus(hdr=c2c16300, scratch=c3301b40)
[   27.462589]     buf = {c3301b40, 16}
[   27.466214]     buf = {c3301b40, 16}
[   27.469689]     buf = {c3301b40, 16}
[   27.473241]     buf = {c3301b40, 16}
[   27.476873]     buf = {c3301b40, 16}
[   27.480589] nfs_readhdr_free(hdr=c2c16300, scratch=c3301b40)
[   27.486339] 
[   27.487450] nfs_read_alloc_scratch(hdr=c2c16080, size=16) = c3301280
[   27.500934] nfs4_xdr_dec_read_plus(hdr=c2c16080, scratch=c3301280)
[   27.505726]     buf = {c3301280, 16}
[   27.509216]     buf = {c3301280, 16}
[   27.512769]     buf = {c3301280, 16}
[   27.516501]     buf = {c3301280, 16}
[   27.519961]     buf = {c3301280, 16}
[   27.523870] nfs_readhdr_free(hdr=c2c16080, scratch=c3301280)
[   27.554694] 
[   27.554775] nfs_read_alloc_scratch(hdr=c2c15e00, size=16) = c3301540
[   27.567281] nfs4_xdr_dec_read_plus(hdr=c2c15e00, scratch=c3301540)
[   27.572039]     buf = {c3301540, 16}
[   27.575649]     buf = {c3301540, 16}
[   27.579127]     buf = {c3301540, 16}
[   27.582684]     buf = {c3301540, 16}
[   27.586328]     buf = {c3301540, 16}
[   27.589996] nfs_readhdr_free(hdr=c2c15e00, scratch=c3301540)
[   27.607132] 
[   27.607209] nfs_read_alloc_scratch(hdr=cfeac280, size=16) = cfe50740
[   27.615212] nfs4_xdr_dec_read_plus(hdr=cfeac280, scratch=cfe50740)
[   27.619937]     buf = {cfe50740, 16}
[   27.623483]     buf = {cfe50740, 16}
[   27.627116]     buf = {cfe50740, 16}
[   27.630603]     buf = {cfe50740, 16}
[   27.634206]     buf = {cfe50740, 16}
[   27.637797] nfs_readhdr_free(hdr=cfeac280, scratch=cfe50740)
[   27.643251] 
[   27.644895] nfs_read_alloc_scratch(hdr=cfeac000, size=16) = cfe50800
[   27.653096] nfs4_xdr_dec_read_plus(hdr=cfeac000, scratch=cfe50800)
[   27.657924]     buf = {cfe50800, 16}
[   27.661389]     buf = {cfe50800, 16}
[   27.665005]     buf = {cfe50800, 16}
[   27.668489]     buf = {cfe50800, 16}
[   27.672043]     buf = {cfe50800, 16}
[   27.675725] nfs_readhdr_free(hdr=cfeac000, scratch=cfe50800)
[   27.681758] 
[   27.682695] nfs_read_alloc_scratch(hdr=cfeac000, size=16) = cfe50a40
[   27.692571] nfs4_xdr_dec_read_plus(hdr=cfeac000, scratch=cfe50a40)
[   27.697357]     buf = {cfe50a40, 16}
[   27.700839]     buf = {cfe50a40, 16}
[   27.704444]     buf = {cfe50a40, 16}
[   27.707945]     buf = {cfe50a40, 16}
[   27.711511]     buf = {cfe50a40, 16}
[   27.715225] nfs_readhdr_free(hdr=cfeac000, scratch=cfe50a40)
[   27.719958] 
[   27.722204] nfs_read_alloc_scratch(hdr=cfeac280, size=16) = cfe50100
[   27.731978] nfs4_xdr_dec_read_plus(hdr=cfeac280, scratch=cfe50100)
[   27.736998]     buf = {cfe50100, 16}
[   27.740299]     buf = {cfe50100, 16}
[   27.743873]     buf = {cfe50100, 16}
[   27.747451]     buf = {cfe50100, 16}
[   27.750996]     buf = {cfe50100, 16}
[   27.754730] nfs_readhdr_free(hdr=cfeac280, scratch=cfe50100)
[   27.758163] 
[   27.761587] nfs_read_alloc_scratch(hdr=cfeac500, size=16) = cfe501c0
[   27.773275] nfs4_xdr_dec_read_plus(hdr=cfeac500, scratch=cfe501c0)
[   27.778049]     buf = {cfe501c0, 16}
[   27.781547]     buf = {cfe501c0, 16}
[   27.785145]     buf = {cfe501c0, 16}
[   27.788653]     buf = {cfe501c0, 16}
[   27.792225]     buf = {cfe501c0, 16}
[   27.796058] nfs_readhdr_free(hdr=cfeac500, scratch=cfe501c0)
[   27.798996] 
[   27.802878] nfs_read_alloc_scratch(hdr=cfeac780, size=16) = cfe50e80
[   27.811927] nfs4_xdr_dec_read_plus(hdr=cfeac780, scratch=cfe50e80)
[   27.816957]     buf = {cfe50e80, 16}
[   27.820246]     buf = {cfe50e80, 16}
[   27.823819]     buf = {cfe50e80, 16}
[   27.827405]     buf = {cfe50e80, 16}
[   27.830939]     buf = {cfe50e80, 16}
[   27.834662] nfs_readhdr_free(hdr=cfeac780, scratch=cfe50e80)
[   27.837955] 
[   27.841534] nfs_read_alloc_scratch(hdr=cfeacc80, size=16) = cfe50700
[   27.850799] nfs4_xdr_dec_read_plus(hdr=cfeacc80, scratch=cfe50700)
[   27.855598]     buf = {cfe50700, 16}
[   27.859068]     buf = {cfe50700, 16}
[   27.862622]     buf = {cfe50700, 16}
[   27.866217]     buf = {cfe50700, 16}
[   27.869735]     buf = {cfe50700, 16}
[   27.873390] nfs_readhdr_free(hdr=cfeacc80, scratch=cfe50700)
[   27.895233] 
[   27.895328] nfs_read_alloc_scratch(hdr=cfeacf00, size=16) = c4320440
[   27.909709] nfs4_xdr_dec_read_plus(hdr=cfeacf00, scratch=c4320440)
[   27.914490]     buf = {c4320440, 16}
[   27.917981]     buf = {c4320440, 16}
[   27.921535]     buf = {c4320440, 16}
[   27.925140]     buf = {c4320440, 16}
[   27.928695]     buf = {c4320440, 16}
[   27.932504] nfs_readhdr_free(hdr=cfeacf00, scratch=c4320440)
[   27.937140] 
[   27.939315] nfs_read_alloc_scratch(hdr=cfeaca00, size=16) = c4320480
[   27.951202] nfs4_xdr_dec_read_plus(hdr=cfeaca00, scratch=c4320480)
[   27.955988]     buf = {c4320480, 16}
[   27.959477]     buf = {c4320480, 16}
[   27.963029]     buf = {c4320480, 16}
[   27.966618]     buf = {c4320480, 16}
[   27.970173]     buf = {c4320480, 16}
[   27.973957] nfs_readhdr_free(hdr=cfeaca00, scratch=c4320480)
[   27.977536] 
[   27.980809] nfs_read_alloc_scratch(hdr=cfead180, size=16) = c4320c00
[   27.994684] nfs4_xdr_dec_read_plus(hdr=cfead180, scratch=c4320c00)
[   27.999424]     buf = {c4320c00, 16}
[   28.002962]     buf = {c4320c00, 16}
[   28.006554]     buf = {c4320c00, 16}
[   28.010063]     buf = {c4320c00, 16}
[   28.013652]     buf = {c4320c00, 16}
[   28.017428] nfs_readhdr_free(hdr=cfead180, scratch=c4320c00)
[   28.022825] 
[   28.024312] nfs_read_alloc_scratch(hdr=cfead400, size=16) = c4320480
[   28.032369] nfs4_xdr_dec_read_plus(hdr=cfead400, scratch=c4320480)
[   28.037134]     buf = {c4320480, 16}
[   28.040634]     buf = {c4320480, 16}
[   28.044217]     buf = {c4320480, 16}
[   28.047739]     buf = {c4320480, 16}
[   28.051295]     buf = {c4320480, 16}
[   28.055041] nfs_readhdr_free(hdr=cfead400, scratch=c4320480)
[   28.060249] 
[   28.061967] nfs_read_alloc_scratch(hdr=cfeae080, size=16) = c4320140
[   28.073702] nfs4_xdr_dec_read_plus(hdr=cfeae080, scratch=c4320140)
[   28.078658]     buf = {c4320140, 16}
[   28.082001]     buf = {c4320140, 16}
[   28.085614]     buf = {c4320140, 16}
[   28.089107]     buf = {c4320140, 16}
[   28.092702]     buf = {c4320140, 16}
[   28.096434] nfs_readhdr_free(hdr=cfeae080, scratch=c4320140)
[   28.103610] systemd[1]: systemd 253.4-1-arch running in system mode (+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified)
[   28.134592] systemd[1]: Detected architecture arm.
[   28.141749] 
[   28.141829] nfs_read_alloc_scratch(hdr=cfead680, size=16) = c43204c0
[   28.150487] nfs4_xdr_dec_read_plus(hdr=cfead680, scratch=c43204c0)
[   28.155344]     buf = {c43204c0, 16}
[   28.158758]     buf = {c43204c0, 16}
[   28.162311]     buf = {c43204c0, 16}
[   28.165912]     buf = {c43204c0, 16}
[   28.169424]     buf = {c43204c0, 16}
[   28.173067] nfs_readhdr_free(hdr=cfead680, scratch=c43204c0)
[   28.185235] 
[   28.185319] nfs_read_alloc_scratch(hdr=c2c15b80, size=16) = c3301bc0
[   28.193199] nfs4_xdr_dec_read_plus(hdr=c2c15b80, scratch=c3301bc0)
[   28.197980]     buf = {c3301bc0, 16}
[   28.201469]     buf = {c3301bc0, 16}
[   28.205070]     buf = {c3301bc0, 16}
[   28.208577]     buf = {c3301bc0, 16}
[   28.212132]     buf = {c3301bc0, 16}
[   28.215853] nfs_readhdr_free(hdr=c2c15b80, scratch=c3301bc0)

Welcome to Arch Linux ARM!

[   28.254625] 
[   28.254722] nfs_read_alloc_scratch(hdr=c2c15900, size=16) = c3301b40
[   28.268595] nfs4_xdr_dec_read_plus(hdr=c2c15900, scratch=c3301b40)
[   28.273353]     buf = {c3301b40, 16}
[   28.276981]     buf = {c3301b40, 16}
[   28.280445]     buf = {c3301b40, 16}
[   28.284049]     buf = {c3301b40, 16}
[   28.287615]     buf = {c3301b40, 16}
[   28.291505] nfs_readhdr_free(hdr=c2c15900, scratch=c3301b40)
[   28.297686] 
[   28.298288] nfs_read_alloc_scratch(hdr=c2c15680, size=16) = c3301080
[   28.306938] nfs4_xdr_dec_read_plus(hdr=c2c15680, scratch=c3301080)
[   28.311666]     buf = {c3301080, 16}
[   28.315274]     buf = {c3301080, 16}
[   28.318777]     buf = {c3301080, 16}
[   28.322327]     buf = {c3301080, 16}
[   28.325925]     buf = {c3301080, 16}
[   28.329522] nfs_readhdr_free(hdr=c2c15680, scratch=c3301080)
[   28.332123] systemd[1]: Hostname set to <odroidhc1>.
[   28.344817] 
[   28.344898] nfs_read_alloc_scratch(hdr=c2c15400, size=16) = c3301fc0
[   28.358458] nfs4_xdr_dec_read_plus(hdr=c2c15400, scratch=c3301fc0)
[   28.363466]     buf = {c3301fc0, 16}
[   28.367097]     buf = {c3301fc0, 16}
[   28.370409]     buf = {c3301fc0, 16}
[   28.373958]     buf = {c3301fc0, 16}
[   28.377616]     buf = {c3301fc0, 16}
[   28.381453] nfs_readhdr_free(hdr=c2c15400, scratch=c3301fc0)
[   28.384340] 
[   28.388203] nfs_read_alloc_scratch(hdr=c2c15180, size=16) = c3301080
[   28.400163] nfs4_xdr_dec_read_plus(hdr=c2c15180, scratch=c3301080)
[   28.405395]     buf = {c3301080, 16}
[   28.408613]     buf = {c3301080, 16}
[   28.412161]     buf = {c3301080, 16}
[   28.415750]     buf = {c3301080, 16}
[   28.419330]     buf = {c3301080, 16}
[   28.423321] nfs_readhdr_free(hdr=c2c15180, scratch=c3301080)
[   28.439671] 
[   28.439778] nfs_read_alloc_scratch(hdr=c2c15180, size=16) = c3301600
[   28.447824] nfs4_xdr_dec_read_plus(hdr=c2c15180, scratch=c3301600)
[   28.452610]     buf = {c3301600, 16}
[   28.456454]     buf = {c3301600, 16}
[   28.459742]     buf = {c3301600, 16}
[   28.463313]     buf = {c3301600, 16}
[   28.466928]     buf = {c3301600, 16}
[   28.470553] nfs_readhdr_free(hdr=c2c15180, scratch=c3301600)
[   28.572140] 
[   28.572249] nfs_read_alloc_scratch(hdr=cfead900, size=16) = c430acc0
[   28.589575] nfs4_xdr_dec_read_plus(hdr=cfead900, scratch=c430acc0)
[   28.594836]     buf = {c430acc0, 16}
[   28.598175]     buf = {c430acc0, 16}
[   28.601677]     buf = {c430acc0, 16}
[   28.605372]     buf = {c430acc0, 16}
[   28.609094]     buf = {c430acc0, 16}
[   28.612690] nfs_readhdr_free(hdr=cfead900, scratch=c430acc0)
[   28.624737] 
[   28.624859] nfs_read_alloc_scratch(hdr=cfeadb80, size=16) = c430a940
[   28.633351] nfs4_xdr_dec_read_plus(hdr=cfeadb80, scratch=c430a940)
[   28.638180]     buf = {c430a940, 16}
[   28.641637]     buf = {c430a940, 16}
[   28.645290]     buf = {c430a940, 16}
[   28.648840]     buf = {c430a940, 16}
[   28.652362]     buf = {c430a940, 16}
[   28.656119] nfs_readhdr_free(hdr=cfeadb80, scratch=c430a940)
[   28.659788] 
[   28.662979] nfs_read_alloc_scratch(hdr=cfeade00, size=16) = c430a9c0
[   28.674412] nfs4_xdr_dec_read_plus(hdr=cfeade00, scratch=c430a9c0)
[   28.679142]     buf = {c430a9c0, 16}
[   28.682678]     buf = {c430a9c0, 16}
[   28.686277]     buf = {c430a9c0, 16}
[   28.689784]     buf = {c430a9c0, 16}
[   28.693356]     buf = {c430a9c0, 16}
[   28.697056] nfs_readhdr_free(hdr=cfeade00, scratch=c430a9c0)
[   28.698499] 
[   28.704050] nfs_read_alloc_scratch(hdr=cfeae300, size=16) = c430a800
[   28.715384] nfs4_xdr_dec_read_plus(hdr=cfeae300, scratch=c430a800)
[   28.720118]     buf = {c430a800, 16}
[   28.723660]     buf = {c430a800, 16}
[   28.727255]     buf = {c430a800, 16}
[   28.730765]     buf = {c430a800, 16}
[   28.734476]     buf = {c430a800, 16}
[   28.738200] nfs_readhdr_free(hdr=cfeae300, scratch=c430a800)
[   28.748183] 
[   28.748249] nfs_read_alloc_scratch(hdr=cfeaed00, size=16) = c430a800
[   28.757654] nfs4_xdr_dec_read_plus(hdr=cfeaed00, scratch=c430a800)
[   28.762418]     buf = {c430a800, 16}
[   28.766101]     buf = {c430a800, 16}
[   28.769471]     buf = {c430a800, 16}
[   28.773023]     buf = {c430a800, 16}
[   28.776828]     buf = {c430a800, 16}
[   28.780286] nfs_readhdr_free(hdr=cfeaed00, scratch=c430a800)
[   28.783612] 
[   28.787257] nfs_read_alloc_scratch(hdr=cfeae580, size=16) = c430aa40
[   28.798663] nfs4_xdr_dec_read_plus(hdr=cfeae580, scratch=c430aa40)
[   28.803393]     buf = {c430aa40, 16}
[   28.807006]     buf = {c430aa40, 16}
[   28.810489]     buf = {c430aa40, 16}
[   28.814085]     buf = {c430aa40, 16}
[   28.817622]     buf = {c430aa40, 16}
[   28.821342] nfs_readhdr_free(hdr=cfeae580, scratch=c430aa40)
[   28.824660] 
[   28.828302] nfs_read_alloc_scratch(hdr=cfeae800, size=16) = c430a5c0
[   28.835875] 
[   28.836051] nfs_read_alloc_scratch(hdr=cfeaea80, size=16) = c1f44500
[   28.837660] nfs4_xdr_dec_read_plus(hdr=cfeae800, scratch=c430a5c0)
[   28.848698]     buf = {c430a5c0, 16}
[   28.852084]     buf = {c430a5c0, 16}
[   28.855694]     buf = {c430a5c0, 16}
[   28.855818] nfs4_xdr_dec_read_plus(hdr=cfeaea80, scratch=c1f44500)
[   28.859174]     buf = {c430a5c0, 16}
[   28.859192]     buf = {c430a5c0, 16}
[   28.859305] nfs_readhdr_free(hdr=cfeae800, scratch=c430a5c0)
[   28.865707]     buf = {c1f44500, 16}
[   28.882582]     buf = {c1f44500, 16}
[   28.885573]     buf = {c1f44500, 16}
[   28.889051]     buf = {c1f44500, 16}
[   28.892728]     buf = {c1f44500, 16}
[   28.897308] nfs_readhdr_free(hdr=cfeaea80, scratch=c1f44500)
[   28.904760] systemd[1]: bpf-lsm: BPF LSM hook not enabled in the kernel, BPF LSM not supported
[   28.919484] memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL, pid=1 'systemd'
[   28.954276] 
[   28.954468] nfs_read_alloc_scratch(hdr=c1cfc280, size=16) = c419ba00
[   28.962633] nfs4_xdr_dec_read_plus(hdr=c1cfc280, scratch=c419ba00)
[   28.967537]     buf = {c419ba00, 16}
[   28.970956]     buf = {c419ba00, 16}
[   28.974606]     buf = {c419ba00, 16}
[   28.978086]     buf = {c419ba00, 16}
[   28.981653]     buf = {c419ba00, 16}
[   28.985467] nfs_readhdr_free(hdr=c1cfc280, scratch=c419ba00)
[   28.999962] 
[   29.000078] nfs_read_alloc_scratch(hdr=c1cfc500, size=16) = c419be80
[   29.008358] nfs4_xdr_dec_read_plus(hdr=c1cfc500, scratch=c419be80)
[   29.013109]     buf = {c419be80, 16}
[   29.016887]     buf = {c419be80, 16}
[   29.020246]     buf = {c419be80, 16}
[   29.023818]     buf = {c419be80, 16}
[   29.027417]     buf = {c419be80, 16}
[   29.031036] nfs_readhdr_free(hdr=c1cfc500, scratch=c419be80)
[   29.035636] 
[   29.038180] nfs_read_alloc_scratch(hdr=c1cfc780, size=16) = c419b940
[   29.048518] nfs4_xdr_dec_read_plus(hdr=c1cfc780, scratch=c419b940)
[   29.053321]     buf = {c419b940, 16}
[   29.056922]     buf = {c419b940, 16}
[   29.060421]     buf = {c419b940, 16}
[   29.063952]     buf = {c419b940, 16}
[   29.067660]     buf = {c419b940, 16}
[   29.071335] nfs_readhdr_free(hdr=c1cfc780, scratch=c419b940)
[   29.077087] 
[   29.078201] nfs_read_alloc_scratch(hdr=c1cfca00, size=16) = c419b240
[   29.091395] nfs4_xdr_dec_read_plus(hdr=c1cfca00, scratch=c419b240)
[   29.096168]     buf = {c419b240, 16}
[   29.099669]     buf = {c419b240, 16}
[   29.103222]     buf = {c419b240, 16}
[   29.106815]     buf = {c419b240, 16}
[   29.110372]     buf = {c419b240, 16}
[   29.114260] nfs_readhdr_free(hdr=c1cfca00, scratch=c419b240)
[   29.125839] 
[   29.126092] nfs_read_alloc_scratch(hdr=c1cfcc80, size=16) = c419b480
[   29.135262] nfs4_xdr_dec_read_plus(hdr=c1cfcc80, scratch=c419b480)
[   29.140008]     buf = {c419b480, 16}
[   29.143544]     buf = {c419b480, 16}
[   29.147168]     buf = {c419b480, 16}
[   29.150653]     buf = {c419b480, 16}
[   29.154280]     buf = {c419b480, 16}
[   29.157882] nfs_readhdr_free(hdr=c1cfcc80, scratch=c419b480)
[   29.163389] 
[   29.165165] nfs_read_alloc_scratch(hdr=c1cfcf00, size=16) = c419bc40
[   29.177496] nfs4_xdr_dec_read_plus(hdr=c1cfcf00, scratch=c419bc40)
[   29.182292]     buf = {c419bc40, 16}
[   29.186145]     buf = {c419bc40, 16}
[   29.189334]     buf = {c419bc40, 16}
[   29.192886]     buf = {c419bc40, 16}
[   29.196516]     buf = {c419bc40, 16}
[   29.200211] nfs_readhdr_free(hdr=c1cfcf00, scratch=c419bc40)
[   29.201413] 
[   29.207288] nfs_read_alloc_scratch(hdr=c1cfd180, size=16) = c419b7c0
[   29.218982] nfs4_xdr_dec_read_plus(hdr=c1cfd180, scratch=c419b7c0)
[   29.223723]     buf = {c419b7c0, 16}
[   29.227312]     buf = {c419b7c0, 16}
[   29.230816]     buf = {c419b7c0, 16}
[   29.234400]     buf = {c419b7c0, 16}
[   29.237962]     buf = {c419b7c0, 16}
[   29.241774] nfs_readhdr_free(hdr=c1cfd180, scratch=c419b7c0)
[   29.263275] 
[   29.263615] nfs_read_alloc_scratch(hdr=c1cfd400, size=16) = c419b1c0
[   29.273036] nfs4_xdr_dec_read_plus(hdr=c1cfd400, scratch=c419b1c0)
[   29.277870]     buf = {c419b1c0, 16}
[   29.281324]     buf = {c419b1c0, 16}
[   29.284956]     buf = {c419b1c0, 16}
[   29.288431]     buf = {c419b1c0, 16}
[   29.291996]     buf = {c419b1c0, 16}
[   29.295710] nfs_readhdr_free(hdr=c1cfd400, scratch=c419b1c0)
[   29.298705] 
[   29.302742] nfs_read_alloc_scratch(hdr=c1cfd680, size=16) = c419bf40
[   29.313243] nfs4_xdr_dec_read_plus(hdr=c1cfd680, scratch=c419bf40)
[   29.318031]     buf = {c419bf40, 16}
[   29.321522]     buf = {c419bf40, 16}
[   29.325131]     buf = {c419bf40, 16}
[   29.328627]     buf = {c419bf40, 16}
[   29.332204]     buf = {c419bf40, 16}
[   29.335970] nfs_readhdr_free(hdr=c1cfd680, scratch=c419bf40)
[   29.338560] 
[   29.343157] nfs_read_alloc_scratch(hdr=c1cfd900, size=16) = c419b7c0
[   29.357807] nfs4_xdr_dec_read_plus(hdr=c1cfd900, scratch=c419b7c0)
[   29.362569]     buf = {c419b7c0, 16}
[   29.366159]     buf = {c419b7c0, 16}
[   29.369652]     buf = {c419b7c0, 16}
[   29.373204]     buf = {c419b7c0, 16}
[   29.376855]     buf = {c419b7c0, 16}
[   29.380641] nfs_readhdr_free(hdr=c1cfd900, scratch=c419b7c0)
[   29.385299] 
[   29.387737] nfs_read_alloc_scratch(hdr=c1cfdb80, size=16) = c419bbc0
[   29.396539] 
[   29.396809] nfs_read_alloc_scratch(hdr=c1cfde00, size=16) = cfdc5d00
[   29.399709] nfs4_xdr_dec_read_plus(hdr=c1cfdb80, scratch=c419bbc0)
[   29.406103] 
[   29.409150]     buf = {c419bbc0, 16}
[   29.410521] nfs4_xdr_dec_read_plus(hdr=c1cfde00, scratch=cfdc5d00)
[   29.410537]     buf = {cfdc5d00, 16}
[   29.410549]     buf = {cfdc5d00, 16}
[   29.410561]     buf = {cfdc5d00, 16}
[   29.410575]     buf = {cfdc5d00, 16}
[   29.410571] nfs_read_alloc_scratch(hdr=c1cfe080, size=16) = cfd81500
[   29.414169]     buf = {c419bbc0, 16}
[   29.414186]     buf = {c419bbc0, 16}
[   29.414198]     buf = {c419bbc0, 16}
[   29.420533]     buf = {cfdc5d00, 16}
[   29.422246] nfs4_xdr_dec_read_plus(hdr=c1cfe080, scratch=cfd81500)
[   29.422339]     buf = {cfd81500, 16}
[   29.422393]     buf = {cfd81500, 16}
[   29.422443]     buf = {cfd81500, 16}
[   29.422471]     buf = {cfd81500, 16}
[   29.422620]     buf = {cfd81500, 16}
[   29.423334] nfs_readhdr_free(hdr=c1cfe080, scratch=cfd81500)
[   29.424133]     buf = {c419bbc0, 16}
[   29.428064] nfs_readhdr_free(hdr=c1cfde00, scratch=cfdc5d00)
[   29.430268] 
[   29.430303] nfs_read_alloc_scratch(hdr=c1cfe300, size=16) = cfd81c40
[   29.431209] nfs_readhdr_free(hdr=c1cfdb80, scratch=c419bbc0)
[   29.438584] nfs4_xdr_dec_read_plus(hdr=c1cfe300, scratch=cfd81c40)
[   29.513630]     buf = {cfd81c40, 16}
[   29.516960]     buf = {cfd81c40, 16}
[   29.520462]     buf = {cfd81c40, 16}
[   29.524066]     buf = {cfd81c40, 16}
[   29.527619]     buf = {cfd81c40, 16}
[   29.531334] nfs_readhdr_free(hdr=c1cfe300, scratch=cfd81c40)
[   29.533484] 
[   29.538405] nfs_read_alloc_scratch(hdr=c1cfe580, size=16) = cfd81640
[   29.552292] nfs4_xdr_dec_read_plus(hdr=c1cfe580, scratch=cfd81640)
[   29.557238]     buf = {cfd81640, 16}
[   29.560632]     buf = {cfd81640, 16}
[   29.564300]     buf = {cfd81640, 16}
[   29.567736]     buf = {cfd81640, 16}
[   29.571371]     buf = {cfd81640, 16}
[   29.575410] nfs_readhdr_free(hdr=c1cfe580, scratch=cfd81640)
[   29.588176] 
[   29.588443] nfs_read_alloc_scratch(hdr=c2738c80, size=16) = c2535880
[   29.600483] nfs4_xdr_dec_read_plus(hdr=c2738c80, scratch=c2535880)
[   29.605328]     buf = {c2535880, 16}
[   29.608774]     buf = {c2535880, 16}
[   29.612326]     buf = {c2535880, 16}
[   29.615972]     buf = {c2535880, 16}
[   29.619450] 8<--- cut here ---
[   29.622464] Unable to handle kernel NULL pointer dereference at virtual address 00000004 when read
[   29.631443] [00000004] *pgd=00000000
[   29.634992] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[   29.640237] Modules linked in:
[   29.643263] CPU: 4 PID: 72 Comm: kworker/u16:3 Not tainted 6.4.0-rc6-next-20230615-00004-g8e95ecb4298d #83
[   29.652895] Hardware name: Samsung Exynos (Flattened Device Tree)
[   29.658954] Workqueue: rpciod rpc_async_schedule
[   29.663539] PC is at mmiocpy+0x4c/0x334
[   29.667356] LR is at xdr_inline_decode+0xfc/0x2a4
[   29.672028] pc : [<c0bc658c>]    lr : [<c0b8dd5c>]    psr: 20030013
[   29.678269] sp : f0a99db8  ip : 00000000  fp : f0a99dc8
[   29.683467] r10: 0000000b  r9 : cfedce18  r8 : cfedce00
[   29.688666] r7 : 00000ffc  r6 : c2535880  r5 : 00000004  r4 : f0a99e70
[   29.695167] r3 : 00000000  r2 : 00000f7c  r1 : 00000004  r0 : c2535880
[   29.701666] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   29.708773] Control: 10c5387d  Table: 4fed806a  DAC: 00000051
[   29.714490] Register r0 information: slab kmalloc-64 start c2535880 pointer offset 0 size 64
[   29.722899] Register r1 information: non-paged memory
[   29.727920] Register r2 information: non-paged memory
[   29.732947] Register r3 information: NULL pointer
[   29.737625] Register r4 information: 2-page vmalloc region starting at 0xf0a98000 allocated at kernel_clone+0x58/0x3b4
[   29.748291] Register r5 information: non-paged memory
[   29.753310] Register r6 information: slab kmalloc-64 start c2535880 pointer offset 0 size 64
[   29.761720] Register r7 information: non-paged memory
[   29.766741] Register r8 information: slab kmalloc-512 start cfedce00 pointer offset 0 size 512
[   29.775325] Register r9 information: slab kmalloc-512 start cfedce00 pointer offset 24 size 512
[   29.783990] Register r10 information: non-paged memory
[   29.789098] Register r11 information: 2-page vmalloc region starting at 0xf0a98000 allocated at kernel_clone+0x58/0x3b4
[   29.799850] Register r12 information: NULL pointer
[   29.804609] Process kworker/u16:3 (pid: 72, stack limit = 0x768cef0c)
[   29.811024] Stack: (0xf0a99db8 to 0xf0a9a000)
[   29.815354] 9da0:                                                       00000004 c2535880
[   29.823507] 9dc0: cfedce00 cfedce18 c2535880 f0a99e70 00003000 c0b8dd5c 0000000b f0a99e70
[   29.831651] 9de0: c2738e80 00000001 cfedce00 c0bf7ac4 00000000 a2d887e1 00000000 00000000
[   29.839797] 9e00: 00000003 f0a99e2c 00000000 00000000 c1e64000 c0c05d88 c495da1c 00000000
[   29.847943] 9e20: c495da94 00000000 00000000 c3341600 00000000 00003000 f0a99e70 00003000
[   29.856088] 9e40: 00003000 a2d887e1 f0a99e70 c2738cd8 c2725200 c1308f58 c2725230 c1e64000
[   29.864233] 9e60: c0b6226c c0b7cedc c2738cf4 c0b6246c 00001000 c2725230 00001000 00000000
[   29.872379] 9e80: c2535880 00000010 c334160c 0000302d c2725200 a2d887e1 c2738cd8 c1308f58
[   29.880524] 9ea0: c1e64000 c0b6226c c1465440 c0b7e3b8 00000001 00000001 04248060 c145d595
[   29.888670] 9ec0: c1e64000 c0199110 c01475c4 a2d887e1 c1c20000 c1e64000 04208060 c2591400
[   29.896815] 9ee0: c1c20000 c1e64000 c1308f58 c145d8c0 c2591405 c0b7ea40 c2738cfc c2514600
[   29.904963] 9f00: c2591400 c0147528 00000001 00000000 c0147468 a2d887e1 c0d02b8c c1c20000
[   29.913107] 9f20: c1af01a4 00000000 00000000 c10b4f6c 00000000 a2d887e1 c1c20000 c2514600
[   29.921252] 9f40: c1c20000 c251462c c1c20040 c1305d40 00000088 c1e64000 c1c20000 c01479ec
[   29.929397] 9f60: c1e64000 c145c89e c1e7ae80 c1e64000 c2c989c0 c0147998 c2514600 c1e7ae80
[   29.937543] 9f80: f0a85ebc 00000000 00000000 c0151330 c2c989c0 c015122c 00000000 00000000
[   29.945688] 9fa0: 00000000 00000000 00000000 c0100108 00000000 00000000 00000000 00000000
[   29.953834] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   29.961979] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[   29.970136]  mmiocpy from xdr_inline_decode+0xfc/0x2a4
[   29.975235]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x198/0x3a8
[   29.981736]  nfs4_xdr_dec_read_plus from call_decode+0x200/0x2bc
[   29.987715]  call_decode from __rpc_execute+0xf8/0x764
[   29.992825]  __rpc_execute from rpc_async_schedule+0x1c/0x34
[   29.998456]  rpc_async_schedule from process_one_work+0x2b4/0x724
[   30.004528]  process_one_work from worker_thread+0x54/0x518
[   30.010067]  worker_thread from kthread+0x104/0x138
[   30.014926]  kthread from ret_from_fork+0x14/0x2c
[   30.019598] Exception stack(0xf0a99fb0 to 0xf0a99ff8)
[   30.024625] 9fa0:                                     00000000 00000000 00000000 00000000
[   30.032775] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   30.040920] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[   30.047509] Code: ba000002 f5d1f03c f5d1f05c f5d1f07c (e8b15378) 
[   30.053707] ---[ end trace 0000000000000000 ]---
�
Terminating...
Thanks for using picocom
kozik@pi3:~$ 

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

On 15/06/2023 10:52, Krzysztof Kozlowski wrote:
> On 14/06/2023 22:55, Anna Schumaker wrote:
>>>>> Still null ptr (built on 420b2d4 with your patch):
>>>>
>>>> We're through the merge window and at rc1 now, so I can spend more
>>>> time scratching my head over your bug again. We've come up with a
>>>> patch (attached) that adds a bunch of printks to show us what the
>>>> kernel thinks is going on. Do you mind trying it out and letting us
>>>> know what gets printed out? You'll need to make sure
>>>> CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.
>>>
>>> The patch does not apply. I tried: v6.4-rc1, v6.4-rc5, next-20230609.
>>
>> Can you try the attached patch on top of my 3-patch series from the
>> other day, and let me know what gets printed out? It adds a bunch of
>> printk()s at strategic points to print out what is going on with the
>> xdr scratch buffer since it's suddenly a bad memory address after
>> working for a bit on your machine.
>>
> 
> Here you have entire log - attached (113 kB, I hope goes past mailing
> lists/spam filters).

As expected this bounced from the mailing lists, but I hope you got it.
If not, let me know.

Best regards,
Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Anna Schumaker]

On Thu, Jun 15, 2023 at 4:55 AM Krzysztof Kozlowski
<krzysztof.kozlowski@linaro.org> wrote:
>
> On 15/06/2023 10:52, Krzysztof Kozlowski wrote:
> > On 14/06/2023 22:55, Anna Schumaker wrote:
> >>>>> Still null ptr (built on 420b2d4 with your patch):
> >>>>
> >>>> We're through the merge window and at rc1 now, so I can spend more
> >>>> time scratching my head over your bug again. We've come up with a
> >>>> patch (attached) that adds a bunch of printks to show us what the
> >>>> kernel thinks is going on. Do you mind trying it out and letting us
> >>>> know what gets printed out? You'll need to make sure
> >>>> CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.
> >>>
> >>> The patch does not apply. I tried: v6.4-rc1, v6.4-rc5, next-20230609.
> >>
> >> Can you try the attached patch on top of my 3-patch series from the
> >> other day, and let me know what gets printed out? It adds a bunch of
> >> printk()s at strategic points to print out what is going on with the
> >> xdr scratch buffer since it's suddenly a bad memory address after
> >> working for a bit on your machine.
> >>
> >
> > Here you have entire log - attached (113 kB, I hope goes past mailing
> > lists/spam filters).
>
> As expected this bounced from the mailing lists, but I hope you got it.
> If not, let me know.

I did still receive it. Thanks!

Anna
>
> Best regards,
> Krzysztof
>

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Anna Schumaker]

[-- Attachment #1: Type: text/plain, Size: 1936 bytes --]

On Thu, Jun 15, 2023 at 9:01 AM Anna Schumaker <schumaker.anna@gmail.com> wrote:
>
> On Thu, Jun 15, 2023 at 4:55 AM Krzysztof Kozlowski
> <krzysztof.kozlowski@linaro.org> wrote:
> >
> > On 15/06/2023 10:52, Krzysztof Kozlowski wrote:
> > > On 14/06/2023 22:55, Anna Schumaker wrote:
> > >>>>> Still null ptr (built on 420b2d4 with your patch):
> > >>>>
> > >>>> We're through the merge window and at rc1 now, so I can spend more
> > >>>> time scratching my head over your bug again. We've come up with a
> > >>>> patch (attached) that adds a bunch of printks to show us what the
> > >>>> kernel thinks is going on. Do you mind trying it out and letting us
> > >>>> know what gets printed out? You'll need to make sure
> > >>>> CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.
> > >>>
> > >>> The patch does not apply. I tried: v6.4-rc1, v6.4-rc5, next-20230609.
> > >>
> > >> Can you try the attached patch on top of my 3-patch series from the
> > >> other day, and let me know what gets printed out? It adds a bunch of
> > >> printk()s at strategic points to print out what is going on with the
> > >> xdr scratch buffer since it's suddenly a bad memory address after
> > >> working for a bit on your machine.
> > >>
> > >
> > > Here you have entire log - attached (113 kB, I hope goes past mailing
> > > lists/spam filters).
> >
> > As expected this bounced from the mailing lists, but I hope you got it.
> > If not, let me know.
>
> I did still receive it. Thanks!

Can you swap out yesterday's patch with this patch? I've adjusted what
gets printed out, and added printk()s to xdr_copy_to_scratch().  I'm
starting to think that the xdr scratch buffer is fine, and that it's
the other pointer passed to memcpy() in that function that's the
problem, and the output from this patch will confirm for me.

Thanks,
Anna

>
> Anna
> >
> > Best regards,
> > Krzysztof
> >

[-- Attachment #2: v2-0001-NFS-Add-debugging-printk-s-to-trace-the-xdr-scrat.patch --]
[-- Type: text/x-patch, Size: 3404 bytes --]

From d41f4304007d2954f8513f3c3d845028fefe79ec Mon Sep 17 00:00:00 2001
From: Anna Schumaker <Anna.Schumaker@Netapp.com>
Date: Wed, 14 Jun 2023 16:49:37 -0400
Subject: [RFC v2] NFS: Add debugging printk()s to trace the xdr->scratch
 buffer

I'm trying to figure out at what point the xdr->scratch buffer is
allocated, freed, set, and reset to figure out why READ_PLUS suddenly
thinks it's a NULL pointer with length 16.

Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
---
 fs/nfs/nfs42xdr.c          | 2 ++
 fs/nfs/read.c              | 8 +++++++-
 include/linux/sunrpc/xdr.h | 1 +
 net/sunrpc/xdr.c           | 4 ++++
 4 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index 20aa5e746497..6a4eade2400b 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -1351,6 +1351,8 @@ static int nfs4_xdr_dec_read_plus(struct rpc_rqst *rqstp,
 	struct compound_hdr hdr;
 	int status;
 
+	printk(KERN_INFO "%s(hdr=%px, scratch=%px)\n", __func__,
+		container_of(res, struct nfs_pgio_header, res), res->scratch);
 	xdr_set_scratch_buffer(xdr, res->scratch, READ_PLUS_SCRATCH_SIZE);
 
 	status = decode_compound_hdr(xdr, &hdr);
diff --git a/fs/nfs/read.c b/fs/nfs/read.c
index 7dc21a48e3e7..7b93316a52de 100644
--- a/fs/nfs/read.c
+++ b/fs/nfs/read.c
@@ -47,8 +47,11 @@ static struct nfs_pgio_header *nfs_readhdr_alloc(void)
 
 static void nfs_readhdr_free(struct nfs_pgio_header *rhdr)
 {
-	if (rhdr->res.scratch != NULL)
+	if (rhdr->res.scratch != NULL) {
+		printk(KERN_INFO "%s(hdr=%px, scratch=%px)\n",
+			__func__, rhdr, rhdr->res.scratch);
 		kfree(rhdr->res.scratch);
+	}
 	kmem_cache_free(nfs_rdata_cachep, rhdr);
 }
 
@@ -114,6 +117,9 @@ bool nfs_read_alloc_scratch(struct nfs_pgio_header *hdr, size_t size)
 {
 	WARN_ON(hdr->res.scratch != NULL);
 	hdr->res.scratch = kmalloc(size, GFP_KERNEL);
+	printk(KERN_INFO "\n");
+	printk(KERN_INFO "%s(hdr=%px, size=%zd) = %px\n",
+		__func__, hdr, size, hdr->res.scratch);
 	return hdr->res.scratch != NULL;
 }
 EXPORT_SYMBOL_GPL(nfs_read_alloc_scratch);
diff --git a/include/linux/sunrpc/xdr.h b/include/linux/sunrpc/xdr.h
index d917618a3058..1c9a54e9efac 100644
--- a/include/linux/sunrpc/xdr.h
+++ b/include/linux/sunrpc/xdr.h
@@ -286,6 +286,7 @@ extern unsigned int xdr_stream_zero(struct xdr_stream *xdr, unsigned int offset,
 static inline void
 xdr_set_scratch_buffer(struct xdr_stream *xdr, void *buf, size_t buflen)
 {
+	WARN_ON(buf != NULL && xdr->scratch.iov_base != NULL);
 	xdr->scratch.iov_base = buf;
 	xdr->scratch.iov_len = buflen;
 }
diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
index 391b336d97de..37c8f7e519dd 100644
--- a/net/sunrpc/xdr.c
+++ b/net/sunrpc/xdr.c
@@ -1440,6 +1440,8 @@ static __be32 *xdr_copy_to_scratch(struct xdr_stream *xdr, size_t nbytes)
 	p = __xdr_inline_decode(xdr, cplen);
 	if (p == NULL)
 		return NULL;
+	printk(KERN_INFO "    %s(%d): memcpy(%px, %px, %zd)\n",
+		__func__, __LINE__, cpdest, p, cplen);
 	memcpy(cpdest, p, cplen);
 	if (!xdr_set_next_buffer(xdr))
 		goto out_overflow;
@@ -1448,6 +1450,8 @@ static __be32 *xdr_copy_to_scratch(struct xdr_stream *xdr, size_t nbytes)
 	p = __xdr_inline_decode(xdr, nbytes);
 	if (p == NULL)
 		return NULL;
+	printk(KERN_INFO "    %s(%d): memcpy(%px, %px, %zd)\n",
+		__func__, __LINE__, cpdest, p, nbytes);
 	memcpy(cpdest, p, nbytes);
 	return xdr->scratch.iov_base;
 out_overflow:
-- 
2.41.0

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Anna Schumaker]

[-- Attachment #1: Type: text/plain, Size: 2199 bytes --]

On Thu, Jun 15, 2023 at 1:04 PM Anna Schumaker <schumaker.anna@gmail.com> wrote:
>
> On Thu, Jun 15, 2023 at 9:01 AM Anna Schumaker <schumaker.anna@gmail.com> wrote:
> >
> > On Thu, Jun 15, 2023 at 4:55 AM Krzysztof Kozlowski
> > <krzysztof.kozlowski@linaro.org> wrote:
> > >
> > > On 15/06/2023 10:52, Krzysztof Kozlowski wrote:
> > > > On 14/06/2023 22:55, Anna Schumaker wrote:
> > > >>>>> Still null ptr (built on 420b2d4 with your patch):
> > > >>>>
> > > >>>> We're through the merge window and at rc1 now, so I can spend more
> > > >>>> time scratching my head over your bug again. We've come up with a
> > > >>>> patch (attached) that adds a bunch of printks to show us what the
> > > >>>> kernel thinks is going on. Do you mind trying it out and letting us
> > > >>>> know what gets printed out? You'll need to make sure
> > > >>>> CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.
> > > >>>
> > > >>> The patch does not apply. I tried: v6.4-rc1, v6.4-rc5, next-20230609.
> > > >>
> > > >> Can you try the attached patch on top of my 3-patch series from the
> > > >> other day, and let me know what gets printed out? It adds a bunch of
> > > >> printk()s at strategic points to print out what is going on with the
> > > >> xdr scratch buffer since it's suddenly a bad memory address after
> > > >> working for a bit on your machine.
> > > >>
> > > >
> > > > Here you have entire log - attached (113 kB, I hope goes past mailing
> > > > lists/spam filters).
> > >
> > > As expected this bounced from the mailing lists, but I hope you got it.
> > > If not, let me know.
> >
> > I did still receive it. Thanks!
>
> Can you swap out yesterday's patch with this patch? I've adjusted what
> gets printed out, and added printk()s to xdr_copy_to_scratch().  I'm
> starting to think that the xdr scratch buffer is fine, and that it's
> the other pointer passed to memcpy() in that function that's the
> problem, and the output from this patch will confirm for me.

Oh, and can you add this one on top of the v2 patch as well?

Thanks,
Anna

>
> Thanks,
> Anna
>
> >
> > Anna
> > >
> > > Best regards,
> > > Krzysztof
> > >

[-- Attachment #2: 0001-NFS-Olga-s-prink-patch.patch --]
[-- Type: text/x-patch, Size: 3266 bytes --]

From 8252138376a2ba3e53807fafab9c382b6acf503f Mon Sep 17 00:00:00 2001
From: Anna Schumaker <Anna.Schumaker@Netapp.com>
Date: Thu, 15 Jun 2023 13:13:47 -0400
Subject: [PATCH] NFS: Olga's prink() patch

Adds a handful of printk() statements to print out the state of the xdr
stream.
---
 fs/nfs/nfs42xdr.c | 5 +++++
 net/sunrpc/clnt.c | 1 +
 net/sunrpc/xdr.c  | 2 ++
 3 files changed, 8 insertions(+)

diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index 6a4eade2400b..22628057d080 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -352,6 +352,7 @@ static void encode_read_plus(struct xdr_stream *xdr,
 			     const struct nfs_pgio_args *args,
 			     struct compound_hdr *hdr)
 {
+	printk("AGLO: %s setting up decode buffer size=%d\n", __func__, decode_read_plus_maxsz);
 	encode_op_hdr(xdr, OP_READ_PLUS, decode_read_plus_maxsz, hdr);
 	encode_nfs4_stateid(xdr, &args->stateid);
 	encode_uint64(xdr, args->offset);
@@ -1065,6 +1066,7 @@ static int decode_read_plus_segment(struct xdr_stream *xdr,
 {
 	__be32 *p;
 
+	printk("AGLO: %s start seg=%p\n", __func__, seg);
 	p = xdr_inline_decode(xdr, 4);
 	if (!p)
 		return -EIO;
@@ -1088,6 +1090,7 @@ static int decode_read_plus_segment(struct xdr_stream *xdr,
 		xdr_decode_hyper(p, &seg->hole.length);
 	} else
 		return -EINVAL;
+	printk("AGLO: %s end seg=%p xdr->nwords=%d\n", __func__, seg, xdr->nwords);
 	return 0;
 }
 
@@ -1130,6 +1133,7 @@ static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res *res)
 	int status, i;
 	__be32 *p;
 
+	printk("AGLO: %s START\n", __func__);
 	status = decode_op_hdr(xdr, OP_READ_PLUS);
 	if (status)
 		return status;
@@ -1161,6 +1165,7 @@ static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res *res)
 
 out:
 	kfree(segs);
+	printk("AGLO: %s END\n", __func__);
 	return status;
 }
 
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index d2ee56634308..4426a25d0152 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1337,6 +1337,7 @@ void rpc_prepare_reply_pages(struct rpc_rqst *req, struct page **pages,
 {
 	hdrsize += RPC_REPHDRSIZE + req->rq_cred->cr_auth->au_ralign;
 
+	printk("AGLO: %s hdrsize %d (<<2 %d) HDRSIZE %d auligh %d\n", __func__, hdrsize, hdrsize << 2, RPC_REPHDRSIZE, req->rq_cred->cr_auth->au_ralign);
 	xdr_inline_pages(&req->rq_rcv_buf, hdrsize << 2, pages, base, len);
 	trace_rpc_xdr_reply_pages(req->rq_task, &req->rq_rcv_buf);
 }
diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
index 37c8f7e519dd..7f635f601dd4 100644
--- a/net/sunrpc/xdr.c
+++ b/net/sunrpc/xdr.c
@@ -1422,6 +1422,7 @@ static __be32 * __xdr_inline_decode(struct xdr_stream *xdr, size_t nbytes)
 	__be32 *p = xdr->p;
 	__be32 *q = p + nwords;
 
+	printk("AGLO: %s nwords=%d xdr->nwords=%d q=%p xdr->end=%p p=%p (%d %d)\n", __func__, nwords, xdr->nwords, q, xdr->end, p, q > xdr->end, q < p);
 	if (unlikely(nwords > xdr->nwords || q > xdr->end || q < p))
 		return NULL;
 	xdr->p = q;
@@ -1435,6 +1436,7 @@ static __be32 *xdr_copy_to_scratch(struct xdr_stream *xdr, size_t nbytes)
 	char *cpdest = xdr->scratch.iov_base;
 	size_t cplen = (char *)xdr->end - (char *)xdr->p;
 
+	printk("AGLO: %s here\n", __func__);
 	if (nbytes > xdr->scratch.iov_len)
 		goto out_overflow;
 	p = __xdr_inline_decode(xdr, cplen);
-- 
2.41.0

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Anna Schumaker]

[-- Attachment #1: Type: text/plain, Size: 2548 bytes --]

On Thu, Jun 15, 2023 at 1:16 PM Anna Schumaker <schumaker.anna@gmail.com> wrote:
>
> On Thu, Jun 15, 2023 at 1:04 PM Anna Schumaker <schumaker.anna@gmail.com> wrote:
> >
> > On Thu, Jun 15, 2023 at 9:01 AM Anna Schumaker <schumaker.anna@gmail.com> wrote:
> > >
> > > On Thu, Jun 15, 2023 at 4:55 AM Krzysztof Kozlowski
> > > <krzysztof.kozlowski@linaro.org> wrote:
> > > >
> > > > On 15/06/2023 10:52, Krzysztof Kozlowski wrote:
> > > > > On 14/06/2023 22:55, Anna Schumaker wrote:
> > > > >>>>> Still null ptr (built on 420b2d4 with your patch):
> > > > >>>>
> > > > >>>> We're through the merge window and at rc1 now, so I can spend more
> > > > >>>> time scratching my head over your bug again. We've come up with a
> > > > >>>> patch (attached) that adds a bunch of printks to show us what the
> > > > >>>> kernel thinks is going on. Do you mind trying it out and letting us
> > > > >>>> know what gets printed out? You'll need to make sure
> > > > >>>> CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.
> > > > >>>
> > > > >>> The patch does not apply. I tried: v6.4-rc1, v6.4-rc5, next-20230609.
> > > > >>
> > > > >> Can you try the attached patch on top of my 3-patch series from the
> > > > >> other day, and let me know what gets printed out? It adds a bunch of
> > > > >> printk()s at strategic points to print out what is going on with the
> > > > >> xdr scratch buffer since it's suddenly a bad memory address after
> > > > >> working for a bit on your machine.
> > > > >>
> > > > >
> > > > > Here you have entire log - attached (113 kB, I hope goes past mailing
> > > > > lists/spam filters).
> > > >
> > > > As expected this bounced from the mailing lists, but I hope you got it.
> > > > If not, let me know.
> > >
> > > I did still receive it. Thanks!
> >
> > Can you swap out yesterday's patch with this patch? I've adjusted what
> > gets printed out, and added printk()s to xdr_copy_to_scratch().  I'm
> > starting to think that the xdr scratch buffer is fine, and that it's
> > the other pointer passed to memcpy() in that function that's the
> > problem, and the output from this patch will confirm for me.
>
> Oh, and can you add this one on top of the v2 patch as well?

Sorry about the noise today. Can you use this patch instead of the two
I attached earlier? I cleaned up the output and cut down on extra
output..

>
> Thanks,
> Anna
>
> >
> > Thanks,
> > Anna
> >
> > >
> > > Anna
> > > >
> > > > Best regards,
> > > > Krzysztof
> > > >

[-- Attachment #2: 0001-NFS-Add-debugging-printk-s-to-trace-the-READ_PLUS-xd.patch --]
[-- Type: text/x-patch, Size: 6616 bytes --]

From 22f96107e7fe5874d4f7915294576e93422d9fbc Mon Sep 17 00:00:00 2001
From: Anna Schumaker <Anna.Schumaker@Netapp.com>
Date: Wed, 14 Jun 2023 16:49:37 -0400
Subject: [PATCH] NFS: Add debugging printk()s to trace the READ_PLUS xdr
 decoding

In addition to tracing the lifetime of the xdr->scratch buffer, it adds
information about the xdr stream such as remaining bytes, current decode
offset, and some other (hopefully) helpful stuff.

Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
---
 fs/nfs/nfs42xdr.c          | 18 ++++++++++++++++--
 fs/nfs/read.c              |  8 +++++++-
 include/linux/sunrpc/xdr.h |  1 +
 net/sunrpc/xdr.c           | 17 +++++++++++++----
 4 files changed, 37 insertions(+), 7 deletions(-)

diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index 20aa5e746497..8786a27fe731 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -789,6 +789,11 @@ static void nfs4_xdr_enc_read_plus(struct rpc_rqst *req,
 
 	rpc_prepare_reply_pages(req, args->pages, args->pgbase, args->count,
 				hdr.replen - READ_PLUS_SEGMENT_SIZE_DIFF);
+	printk(KERN_INFO "  %s(hdr=%px, decode_read_plus_maxsz=%d)\n"
+		"    xdr->buf = {head->iov_len=%zd, page_base=%u, page_len=%u}\n",
+		__func__, container_of(args, struct nfs_pgio_header, args),
+		decode_read_plus_maxsz, xdr->buf->head->iov_len,
+		xdr->buf->page_base, xdr->buf->page_len);
 	encode_nops(&hdr);
 }
 
@@ -1061,10 +1066,12 @@ static inline uint64_t read_plus_segment_length(struct read_plus_segment *seg)
 }
 
 static int decode_read_plus_segment(struct xdr_stream *xdr,
-				    struct read_plus_segment *seg)
+				    struct read_plus_segment *seg,
+				    unsigned int n)
 {
 	__be32 *p;
 
+	printk(KERN_INFO "    %s(n=%u)\n", __func__, n);
 	p = xdr_inline_decode(xdr, 4);
 	if (!p)
 		return -EIO;
@@ -1079,6 +1086,8 @@ static int decode_read_plus_segment(struct xdr_stream *xdr,
 		struct xdr_buf buf;
 		uint32_t len = be32_to_cpup(p);
 
+		printk(KERN_INFO "      %s(n=%u): DATA {offset=%lld, length=%u}\n",
+			__func__, n, seg->offset, len);
 		seg->data.length = len;
 		seg->data.from = xdr_stream_pos(xdr);
 
@@ -1086,8 +1095,11 @@ static int decode_read_plus_segment(struct xdr_stream *xdr,
 			return -EIO;
 	} else if (seg->type == NFS4_CONTENT_HOLE) {
 		xdr_decode_hyper(p, &seg->hole.length);
+		printk(KERN_INFO "      %s(n=%u): HOLE {offset=%lld, length=%llu}\n",
+			__func__, n, seg->offset, seg->hole.length);
 	} else
 		return -EINVAL;
+	printk(KERN_INFO "      %s(n=%u): xdr->nwords = %d", __func__, n, xdr->nwords);
 	return 0;
 }
 
@@ -1149,7 +1161,7 @@ static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res *res)
 		return -ENOMEM;
 
 	for (i = 0; i < segments; i++) {
-		status = decode_read_plus_segment(xdr, &segs[i]);
+		status = decode_read_plus_segment(xdr, &segs[i], i);
 		if (status < 0)
 			goto out;
 	}
@@ -1351,6 +1363,8 @@ static int nfs4_xdr_dec_read_plus(struct rpc_rqst *rqstp,
 	struct compound_hdr hdr;
 	int status;
 
+	printk(KERN_INFO "  %s(hdr=%px, scratch=%px)\n", __func__,
+		container_of(res, struct nfs_pgio_header, res), res->scratch);
 	xdr_set_scratch_buffer(xdr, res->scratch, READ_PLUS_SCRATCH_SIZE);
 
 	status = decode_compound_hdr(xdr, &hdr);
diff --git a/fs/nfs/read.c b/fs/nfs/read.c
index 7dc21a48e3e7..0d8c86be4242 100644
--- a/fs/nfs/read.c
+++ b/fs/nfs/read.c
@@ -47,8 +47,11 @@ static struct nfs_pgio_header *nfs_readhdr_alloc(void)
 
 static void nfs_readhdr_free(struct nfs_pgio_header *rhdr)
 {
-	if (rhdr->res.scratch != NULL)
+	if (rhdr->res.scratch != NULL) {
+		printk(KERN_INFO "  %s(hdr=%px, scratch=%px)\n",
+			__func__, rhdr, rhdr->res.scratch);
 		kfree(rhdr->res.scratch);
+	}
 	kmem_cache_free(nfs_rdata_cachep, rhdr);
 }
 
@@ -114,6 +117,9 @@ bool nfs_read_alloc_scratch(struct nfs_pgio_header *hdr, size_t size)
 {
 	WARN_ON(hdr->res.scratch != NULL);
 	hdr->res.scratch = kmalloc(size, GFP_KERNEL);
+	printk(KERN_INFO "\n");
+	printk(KERN_INFO "%s(hdr=%px, size=%zd) = %px\n",
+		__func__, hdr, size, hdr->res.scratch);
 	return hdr->res.scratch != NULL;
 }
 EXPORT_SYMBOL_GPL(nfs_read_alloc_scratch);
diff --git a/include/linux/sunrpc/xdr.h b/include/linux/sunrpc/xdr.h
index d917618a3058..1c9a54e9efac 100644
--- a/include/linux/sunrpc/xdr.h
+++ b/include/linux/sunrpc/xdr.h
@@ -286,6 +286,7 @@ extern unsigned int xdr_stream_zero(struct xdr_stream *xdr, unsigned int offset,
 static inline void
 xdr_set_scratch_buffer(struct xdr_stream *xdr, void *buf, size_t buflen)
 {
+	WARN_ON(buf != NULL && xdr->scratch.iov_base != NULL);
 	xdr->scratch.iov_base = buf;
 	xdr->scratch.iov_len = buflen;
 }
diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
index 391b336d97de..ff8406e668b2 100644
--- a/net/sunrpc/xdr.c
+++ b/net/sunrpc/xdr.c
@@ -1416,12 +1416,17 @@ void xdr_init_decode_pages(struct xdr_stream *xdr, struct xdr_buf *buf,
 }
 EXPORT_SYMBOL_GPL(xdr_init_decode_pages);
 
-static __be32 * __xdr_inline_decode(struct xdr_stream *xdr, size_t nbytes)
+static __be32 * __xdr_inline_decode(struct xdr_stream *xdr, size_t nbytes,
+				    bool debug_print)
 {
 	unsigned int nwords = XDR_QUADLEN(nbytes);
 	__be32 *p = xdr->p;
 	__be32 *q = p + nwords;
 
+	if (unlikely(debug_print))
+		printk(KERN_INFO "        %s(nbytes=%zd, nwords=%d, xdr, q=%px)\n"
+			"          xdr = {nwords=%u, p=%px, end=%px}\n",
+	 		__func__, nbytes, nwords, q, xdr->nwords, xdr->p, xdr->end);
 	if (unlikely(nwords > xdr->nwords || q > xdr->end || q < p))
 		return NULL;
 	xdr->p = q;
@@ -1437,17 +1442,21 @@ static __be32 *xdr_copy_to_scratch(struct xdr_stream *xdr, size_t nbytes)
 
 	if (nbytes > xdr->scratch.iov_len)
 		goto out_overflow;
-	p = __xdr_inline_decode(xdr, cplen);
+	p = __xdr_inline_decode(xdr, cplen, true);
 	if (p == NULL)
 		return NULL;
+	printk(KERN_INFO "      %s(%d): memcpy(%px, %px, %zd)\n",
+		__func__, __LINE__, cpdest, p, cplen);
 	memcpy(cpdest, p, cplen);
 	if (!xdr_set_next_buffer(xdr))
 		goto out_overflow;
 	cpdest += cplen;
 	nbytes -= cplen;
-	p = __xdr_inline_decode(xdr, nbytes);
+	p = __xdr_inline_decode(xdr, nbytes, true);
 	if (p == NULL)
 		return NULL;
+	printk(KERN_INFO "      %s(%d): memcpy(%px, %px, %zd)\n",
+		__func__, __LINE__, cpdest, p, nbytes);
 	memcpy(cpdest, p, nbytes);
 	return xdr->scratch.iov_base;
 out_overflow:
@@ -1473,7 +1482,7 @@ __be32 * xdr_inline_decode(struct xdr_stream *xdr, size_t nbytes)
 		return xdr->p;
 	if (xdr->p == xdr->end && !xdr_set_next_buffer(xdr))
 		goto out_overflow;
-	p = __xdr_inline_decode(xdr, nbytes);
+	p = __xdr_inline_decode(xdr, nbytes, false);
 	if (p != NULL)
 		return p;
 	return xdr_copy_to_scratch(xdr, nbytes);
-- 
2.41.0

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

[-- Attachment #1: Type: text/plain, Size: 2396 bytes --]

On 15/06/2023 21:38, Anna Schumaker wrote:
> On Thu, Jun 15, 2023 at 1:16 PM Anna Schumaker <schumaker.anna@gmail.com> wrote:
>>
>> On Thu, Jun 15, 2023 at 1:04 PM Anna Schumaker <schumaker.anna@gmail.com> wrote:
>>>
>>> On Thu, Jun 15, 2023 at 9:01 AM Anna Schumaker <schumaker.anna@gmail.com> wrote:
>>>>
>>>> On Thu, Jun 15, 2023 at 4:55 AM Krzysztof Kozlowski
>>>> <krzysztof.kozlowski@linaro.org> wrote:
>>>>>
>>>>> On 15/06/2023 10:52, Krzysztof Kozlowski wrote:
>>>>>> On 14/06/2023 22:55, Anna Schumaker wrote:
>>>>>>>>>> Still null ptr (built on 420b2d4 with your patch):
>>>>>>>>>
>>>>>>>>> We're through the merge window and at rc1 now, so I can spend more
>>>>>>>>> time scratching my head over your bug again. We've come up with a
>>>>>>>>> patch (attached) that adds a bunch of printks to show us what the
>>>>>>>>> kernel thinks is going on. Do you mind trying it out and letting us
>>>>>>>>> know what gets printed out? You'll need to make sure
>>>>>>>>> CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.
>>>>>>>>
>>>>>>>> The patch does not apply. I tried: v6.4-rc1, v6.4-rc5, next-20230609.
>>>>>>>
>>>>>>> Can you try the attached patch on top of my 3-patch series from the
>>>>>>> other day, and let me know what gets printed out? It adds a bunch of
>>>>>>> printk()s at strategic points to print out what is going on with the
>>>>>>> xdr scratch buffer since it's suddenly a bad memory address after
>>>>>>> working for a bit on your machine.
>>>>>>>
>>>>>>
>>>>>> Here you have entire log - attached (113 kB, I hope goes past mailing
>>>>>> lists/spam filters).
>>>>>
>>>>> As expected this bounced from the mailing lists, but I hope you got it.
>>>>> If not, let me know.
>>>>
>>>> I did still receive it. Thanks!
>>>
>>> Can you swap out yesterday's patch with this patch? I've adjusted what
>>> gets printed out, and added printk()s to xdr_copy_to_scratch().  I'm
>>> starting to think that the xdr scratch buffer is fine, and that it's
>>> the other pointer passed to memcpy() in that function that's the
>>> problem, and the output from this patch will confirm for me.
>>
>> Oh, and can you add this one on top of the v2 patch as well?
> 
> Sorry about the noise today. Can you use this patch instead of the two
> I attached earlier? I cleaned up the output and cut down on extra
> output..
> 

Here you have - attached.


Best regards,
Krzysztof

[-- Attachment #2: oops-nfs-read-plus-20230617.txt --]
[-- Type: text/plain, Size: 106344 bytes --]

[    9.391361] IP-Config: Complete:
[    9.393172]      device=eth0, hwaddr=00:1e:06:30:bf:ac, ipaddr=192.168.1.12, mask=255.255.255.0, gw=192.168.1.1
[    9.403379]      host=192.168.1.12, domain=, nis-domain=(none)
[    9.409023]      bootserver=192.168.1.10, rootserver=192.168.1.10, rootpath=
[    9.411961] clk: Disabling unused clocks
[    9.428187] ALSA device list:
[    9.429785]   No soundcards found.
[    9.439456] Freeing unused kernel image (initmem) memory: 1024K
[    9.482351] Run /init as init process
[    9.484579]   with arguments:
[    9.487521]     /init
[    9.489771]     nfsrootdebug
[    9.492889]   with environment:
[    9.495813]     HOME=/
[    9.498134]     TERM=linux
:: running early hook [udev]
Starting version 242.29-1-arch
:: running hook [udev]
:: Triggering uevents...
:: running hook [net_nfs4]
IP-Config: eth0 hardware address 00:1e:06:30:bf:ac mtu 1500
IP-Config: eth0 guessed broadcast address 192.168.1.255
IP-Config: eth0 complete (from 192.168.1.10):
 address: 192.168.1.12     broadcast: 192.168.1.255    netmask: 255.255.255.0   
 gateway: 192.168.1.1      dns0     : 0.0.0.0          dns1   : 0.0.0.0         
 rootserver: 192.168.1.10 rootpath: 
 filename  : 
NFS-Mount: 192.168.1.10:/srv/nfs/odroidhc1
Waiting 10 seconds for device /dev/nfs ...
ERROR: device '/dev/nfs' not found. Skipping fsck.
Mount cmd: 
mount.nfs4 -o vers=4,nolock 192.168.1.10:/srv/nfs/odroidhc1 /new_root
:: running cleanup hook [udev]
[   22.371083] 
[   22.371180] nfs_read_alloc_scratch(hdr=c3194000, size=16) = c2c42700
[   22.377641]   nfs4_xdr_enc_read_plus(hdr=c3194000, decode_read_plus_maxsz=9)
[   22.377641]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   22.397474]   nfs4_xdr_dec_read_plus(hdr=c3194000, scratch=c2c42700)
[   22.402598]     decode_read_plus_segment(n=0)
[   22.406783]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   22.414306]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   22.414630]   nfs_readhdr_free(hdr=c3194000, scratch=c2c42700)
[   22.421148] 
[   22.427274] nfs_read_alloc_scratch(hdr=c1d20000, size=16) = c4338040
[   22.433702]   nfs4_xdr_enc_read_plus(hdr=c1d20000, decode_read_plus_maxsz=9)
[   22.433702]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   22.449860]   nfs4_xdr_dec_read_plus(hdr=c1d20000, scratch=c4338040)
[   22.454930]     decode_read_plus_segment(n=0)
[   22.459129]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   22.466410]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   22.466605]   nfs_readhdr_free(hdr=c1d20000, scratch=c4338040)
[   22.468109] 
[   22.479588] nfs_read_alloc_scratch(hdr=c1d20280, size=16) = c4338500
[   22.486065]   nfs4_xdr_enc_read_plus(hdr=c1d20280, decode_read_plus_maxsz=9)
[   22.486065]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   22.504290]   nfs4_xdr_dec_read_plus(hdr=c1d20280, scratch=c4338500)
[   22.509306]     decode_read_plus_segment(n=0)
[   22.513673]       decode_read_plus_segment(n=0): DATA {offset=20480, length=67096}
[   22.521169]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   22.521546]   nfs_readhdr_free(hdr=c1d20280, scratch=c4338500)
[   22.522697] 
[   22.534455] nfs_read_alloc_scratch(hdr=c3194280, size=16) = c3301800
[   22.540981]   nfs4_xdr_enc_read_plus(hdr=c3194280, decode_read_plus_maxsz=9)
[   22.540981]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   22.558506]   nfs4_xdr_dec_read_plus(hdr=c3194280, scratch=c3301800)
[   22.563617]     decode_read_plus_segment(n=0)
[   22.567792]       decode_read_plus_segment(n=0): DATA {offset=114688, length=70120}
[   22.575766]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   22.576143]   nfs_readhdr_free(hdr=c3194280, scratch=c3301800)
[   22.578969] 
[   22.588747] nfs_read_alloc_scratch(hdr=c3194500, size=16) = c1f46180
[   22.595231]   nfs4_xdr_enc_read_plus(hdr=c3194500, decode_read_plus_maxsz=9)
[   22.595231]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   22.613908]   nfs4_xdr_dec_read_plus(hdr=c3194500, scratch=c1f46180)
[   22.619050]     decode_read_plus_segment(n=0)
[   22.623323]       decode_read_plus_segment(n=0): DATA {offset=40960, length=73728}
[   22.630804]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   22.631194]   nfs_readhdr_free(hdr=c3194500, scratch=c1f46180)
[   22.633694] 
[   22.644066] nfs_read_alloc_scratch(hdr=c3194780, size=16) = c40d9d00
[   22.650685]   nfs4_xdr_enc_read_plus(hdr=c3194780, decode_read_plus_maxsz=9)
[   22.650685]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   22.666943]   nfs4_xdr_dec_read_plus(hdr=c3194780, scratch=c40d9d00)
[   22.671993]     decode_read_plus_segment(n=0)
[   22.676218]       decode_read_plus_segment(n=0): DATA {offset=16384, length=24576}
[   22.683829]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   22.684048]   nfs_readhdr_free(hdr=c3194780, scratch=c40d9d00)
[   22.702872] 
[   22.703066] nfs_read_alloc_scratch(hdr=c40f8000, size=16) = c2c29580
[   22.709437]   nfs4_xdr_enc_read_plus(hdr=c40f8000, decode_read_plus_maxsz=9)
[   22.709437]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   22.725576]   nfs4_xdr_dec_read_plus(hdr=c40f8000, scratch=c2c29580)
[   22.730534]     decode_read_plus_segment(n=0)
[   22.734914]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   22.742101]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   22.742290]   nfs_readhdr_free(hdr=c40f8000, scratch=c2c29580)
[   22.745605] 
[   22.755379] nfs_read_alloc_scratch(hdr=c40f8280, size=16) = c2c29ac0
[   22.761855]   nfs4_xdr_enc_read_plus(hdr=c40f8280, decode_read_plus_maxsz=9)
[   22.761855]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   22.779517]   nfs4_xdr_dec_read_plus(hdr=c40f8280, scratch=c2c29ac0)
[   22.784580]     decode_read_plus_segment(n=0)
[   22.788794]       decode_read_plus_segment(n=0): DATA {offset=1679360, length=67900}
[   22.796827]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   22.797159]   nfs_readhdr_free(hdr=c40f8280, scratch=c2c29ac0)
[   22.798033] 
[   22.809776] nfs_read_alloc_scratch(hdr=c40f8500, size=16) = c2c29740
[   22.816258]   nfs4_xdr_enc_read_plus(hdr=c40f8500, decode_read_plus_maxsz=9)
[   22.816258]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   22.835848]   nfs4_xdr_dec_read_plus(hdr=c40f8500, scratch=c2c29740)
[   22.840842]     decode_read_plus_segment(n=0)
[   22.845249]       decode_read_plus_segment(n=0): DATA {offset=20480, length=131072}
[   22.852881]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   22.853332]   nfs_readhdr_free(hdr=c40f8500, scratch=c2c29740)
[   22.856523] 
[   22.866316] nfs_read_alloc_scratch(hdr=c40f8780, size=16) = c2c29540
[   22.872776]   nfs4_xdr_enc_read_plus(hdr=c40f8780, decode_read_plus_maxsz=9)
[   22.872776]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   22.888816]   nfs4_xdr_dec_read_plus(hdr=c40f8780, scratch=c2c29540)
[   22.893891]     decode_read_plus_segment(n=0)
[   22.898083]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   22.905379]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   22.905594]   nfs_readhdr_free(hdr=c40f8780, scratch=c2c29540)
[   22.909249] 
[   22.918636] nfs_read_alloc_scratch(hdr=c2730000, size=16) = c41c9d00
[   22.925106]   nfs4_xdr_enc_read_plus(hdr=c2730000, decode_read_plus_maxsz=9)
[   22.925106]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   22.943005]   nfs4_xdr_dec_read_plus(hdr=c2730000, scratch=c41c9d00)
[   22.947999]     decode_read_plus_segment(n=0)
[   22.952389]       decode_read_plus_segment(n=0): DATA {offset=3215360, length=68384}
[   22.960030]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   22.960341]   nfs_readhdr_free(hdr=c2730000, scratch=c41c9d00)
[   22.961528] 
[   22.973585] nfs_read_alloc_scratch(hdr=c2730280, size=16) = c2c42f80
[   22.979783]   nfs4_xdr_enc_read_plus(hdr=c2730280, decode_read_plus_maxsz=9)
[   22.979783]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   22.999719]   nfs4_xdr_dec_read_plus(hdr=c2730280, scratch=c2c42f80)
[   23.004731]     decode_read_plus_segment(n=0)
[   23.008978]       decode_read_plus_segment(n=0): DATA {offset=139264, length=131072}
[   23.016809]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.017227]   nfs_readhdr_free(hdr=c2730280, scratch=c2c42f80)
[   23.026397] 
[   23.030164] nfs_read_alloc_scratch(hdr=c2730500, size=16) = c2c42b40
[   23.036694]   nfs4_xdr_enc_read_plus(hdr=c2730500, decode_read_plus_maxsz=9)
[   23.036694]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.053245]   nfs4_xdr_dec_read_plus(hdr=c2730500, scratch=c2c42b40)
[   23.058159]     decode_read_plus_segment(n=0)
[   23.062605]       decode_read_plus_segment(n=0): DATA {offset=0, length=31295}
[   23.069689]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.069843]   nfs_readhdr_free(hdr=c2730500, scratch=c2c42b40)
[   23.074247] 
[   23.083164] nfs_read_alloc_scratch(hdr=c3194a00, size=16) = c40d9ac0
[   23.089543]   nfs4_xdr_enc_read_plus(hdr=c3194a00, decode_read_plus_maxsz=9)
[   23.089543]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.106318]   nfs4_xdr_dec_read_plus(hdr=c3194a00, scratch=c40d9ac0)
[   23.111304]     decode_read_plus_segment(n=0)
[   23.115544]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   23.122903]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.123017]   nfs_readhdr_free(hdr=c3194a00, scratch=c40d9ac0)
[   23.127794] 
[   23.136663] nfs_read_alloc_scratch(hdr=c3194c80, size=16) = c24cc540
[   23.142685]   nfs4_xdr_enc_read_plus(hdr=c3194c80, decode_read_plus_maxsz=9)
[   23.142685]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.163608]   nfs4_xdr_dec_read_plus(hdr=c3194c80, scratch=c24cc540)
[   23.168570]     decode_read_plus_segment(n=0)
[   23.172953]       decode_read_plus_segment(n=0): DATA {offset=49152, length=131072}
[   23.180608]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.181010]   nfs_readhdr_free(hdr=c3194c80, scratch=c24cc540)
[   23.187533] 
[   23.193840] nfs_read_alloc_scratch(hdr=c40f8a00, size=16) = c2c293c0
[   23.200176]   nfs4_xdr_enc_read_plus(hdr=c40f8a00, decode_read_plus_maxsz=9)
[   23.200176]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.215983]   nfs4_xdr_dec_read_plus(hdr=c40f8a00, scratch=c2c293c0)
[   23.220888]     decode_read_plus_segment(n=0)
[   23.225396]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   23.232518]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.232647]   nfs_readhdr_free(hdr=c40f8a00, scratch=c2c293c0)
[   23.237246] 
[   23.245969] nfs_read_alloc_scratch(hdr=c40f8c80, size=16) = c2c29880
[   23.252418]   nfs4_xdr_enc_read_plus(hdr=c40f8c80, decode_read_plus_maxsz=9)
[   23.252418]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.270539]   nfs4_xdr_dec_read_plus(hdr=c40f8c80, scratch=c2c29880)
[   23.275514]     decode_read_plus_segment(n=0)
[   23.279782]       decode_read_plus_segment(n=0): DATA {offset=1486848, length=79348}
[   23.287566]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.287814]   nfs_readhdr_free(hdr=c40f8c80, scratch=c2c29880)
[   23.291694] 
[   23.301084] nfs_read_alloc_scratch(hdr=c40f8f00, size=16) = c2c296c0
[   23.307475]   nfs4_xdr_enc_read_plus(hdr=c40f8f00, decode_read_plus_maxsz=9)
[   23.307475]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.326355]   nfs4_xdr_dec_read_plus(hdr=c40f8f00, scratch=c2c296c0)
[   23.331339]     decode_read_plus_segment(n=0)
[   23.335595]       decode_read_plus_segment(n=0): DATA {offset=32768, length=131072}
[   23.343324]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.343693]   nfs_readhdr_free(hdr=c40f8f00, scratch=c2c296c0)
[   23.358012] 
[   23.358261] nfs_read_alloc_scratch(hdr=c40f9180, size=16) = c2c29100
[   23.365343]   nfs4_xdr_enc_read_plus(hdr=c40f9180, decode_read_plus_maxsz=9)
[   23.365343]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.380432]   nfs4_xdr_dec_read_plus(hdr=c40f9180, scratch=c2c29100)
[   23.385397]     decode_read_plus_segment(n=0)
[   23.389667]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   23.396919]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.397023]   nfs_readhdr_free(hdr=c40f9180, scratch=c2c29100)
[   23.401981] 
[   23.410273] nfs_read_alloc_scratch(hdr=c2730780, size=16) = c2c42e00
[   23.416804]   nfs4_xdr_enc_read_plus(hdr=c2730780, decode_read_plus_maxsz=9)
[   23.416804]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.433980]   nfs4_xdr_dec_read_plus(hdr=c2730780, scratch=c2c42e00)
[   23.438894]     decode_read_plus_segment(n=0)
[   23.443293]       decode_read_plus_segment(n=0): DATA {offset=57344, length=67032}
[   23.450788]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.451036]   nfs_readhdr_free(hdr=c2730780, scratch=c2c42e00)
[   23.463877] 
[   23.464071] nfs_read_alloc_scratch(hdr=c2730a00, size=16) = c2c42d80
[   23.470449]   nfs4_xdr_enc_read_plus(hdr=c2730a00, decode_read_plus_maxsz=9)
[   23.470449]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.486158]   nfs4_xdr_dec_read_plus(hdr=c2730a00, scratch=c2c42d80)
[   23.491066]     decode_read_plus_segment(n=0)
[   23.495440]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   23.502643]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.502771]   nfs_readhdr_free(hdr=c2730a00, scratch=c2c42d80)
[   23.507709] 
[   23.516136] nfs_read_alloc_scratch(hdr=c2730c80, size=16) = c2c42940
[   23.522617]   nfs4_xdr_enc_read_plus(hdr=c2730c80, decode_read_plus_maxsz=9)
[   23.522617]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.539587]   nfs4_xdr_dec_read_plus(hdr=c2730c80, scratch=c2c42940)
[   23.544566]     decode_read_plus_segment(n=0)
[   23.548815]       decode_read_plus_segment(n=0): DATA {offset=16384, length=38368}
[   23.556550]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.556785]   nfs_readhdr_free(hdr=c2730c80, scratch=c2c42940)
[   23.577338] 
[   23.577593] nfs_read_alloc_scratch(hdr=c2730f00, size=16) = c2c42840
[   23.584144]   nfs4_xdr_enc_read_plus(hdr=c2730f00, decode_read_plus_maxsz=9)
[   23.584144]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.600171]   nfs4_xdr_dec_read_plus(hdr=c2730f00, scratch=c2c42840)
[   23.605191]     decode_read_plus_segment(n=0)
[   23.609451]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   23.616717]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.616820]   nfs_readhdr_free(hdr=c2730f00, scratch=c2c42840)
[   23.620944] 
[   23.630013] nfs_read_alloc_scratch(hdr=c2731180, size=16) = c323dd80
[   23.636404]   nfs4_xdr_enc_read_plus(hdr=c2731180, decode_read_plus_maxsz=9)
[   23.636404]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.653578]   nfs4_xdr_dec_read_plus(hdr=c2731180, scratch=c323dd80)
[   23.658478]     decode_read_plus_segment(n=0)
[   23.662856]       decode_read_plus_segment(n=0): DATA {offset=40960, length=66996}
[   23.670367]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.670551]   nfs_readhdr_free(hdr=c2731180, scratch=c323dd80)
[   23.672341] 
[   23.683761] nfs_read_alloc_scratch(hdr=c2731400, size=16) = c2637080
[   23.690355]   nfs4_xdr_enc_read_plus(hdr=c2731400, decode_read_plus_maxsz=9)
[   23.690355]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.707411]   nfs4_xdr_dec_read_plus(hdr=c2731400, scratch=c2637080)
[   23.710774] 
[   23.712386]     decode_read_plus_segment(n=0)
[   23.712399]       decode_read_plus_segment(n=0): DATA {offset=16384, length=24576}
[   23.712423]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.713765] nfs_read_alloc_scratch(hdr=c2731680, size=16) = c41c7200
[   23.718217]   nfs_readhdr_free(hdr=c2731400, scratch=c2637080)
[   23.725813]   nfs4_xdr_enc_read_plus(hdr=c2731680, decode_read_plus_maxsz=9)
[   23.725813]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.761104]   nfs4_xdr_dec_read_plus(hdr=c2731680, scratch=c41c7200)
[   23.766418]     decode_read_plus_segment(n=0)
[   23.770532]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   23.777742]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.777995]   nfs_readhdr_free(hdr=c2731680, scratch=c41c7200)
[   23.785347] 
[   23.790863] nfs_read_alloc_scratch(hdr=c2731900, size=16) = c41c7740
[   23.797386]   nfs4_xdr_enc_read_plus(hdr=c2731900, decode_read_plus_maxsz=9)
[   23.797386]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.817276]   nfs4_xdr_dec_read_plus(hdr=c2731900, scratch=c41c7740)
[   23.822382]     decode_read_plus_segment(n=0)
[   23.826525]       decode_read_plus_segment(n=0): DATA {offset=20480, length=67188}
[   23.834190]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.834476]   nfs_readhdr_free(hdr=c2731900, scratch=c41c7740)
[   23.835993] 
[   23.847474] nfs_read_alloc_scratch(hdr=c2731b80, size=16) = c41c7800
[   23.854027]   nfs4_xdr_enc_read_plus(hdr=c2731b80, decode_read_plus_maxsz=9)
[   23.854027]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.868523] 
[   23.868776] nfs_read_alloc_scratch(hdr=c2731e00, size=16) = c41c7ac0
[   23.868996]   nfs4_xdr_dec_read_plus(hdr=c2731b80, scratch=c41c7800)
[   23.875576]   nfs4_xdr_enc_read_plus(hdr=c2731e00, decode_read_plus_maxsz=9)
[   23.875576]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.881528]     decode_read_plus_segment(n=0)
[   23.897623]   nfs4_xdr_dec_read_plus(hdr=c2731e00, scratch=c41c7ac0)
[   23.899320]       decode_read_plus_segment(n=0): DATA {offset=16384, length=4096}
[   23.905729]     decode_read_plus_segment(n=0)
[   23.913237]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.917400]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   23.917527]   nfs_readhdr_free(hdr=c2731b80, scratch=c41c7800)
[   23.923440]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.936959]   nfs_readhdr_free(hdr=c2731e00, scratch=c41c7ac0)
[   23.939929] 
[   23.949676] nfs_read_alloc_scratch(hdr=c2732080, size=16) = c41c7b40
[   23.956179]   nfs4_xdr_enc_read_plus(hdr=c2732080, decode_read_plus_maxsz=9)
[   23.956179]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   23.974016]   nfs4_xdr_dec_read_plus(hdr=c2732080, scratch=c41c7b40)
[   23.978948]     decode_read_plus_segment(n=0)
[   23.983326]       decode_read_plus_segment(n=0): DATA {offset=294912, length=67400}
[   23.990909]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   23.991166]   nfs_readhdr_free(hdr=c2732080, scratch=c41c7b40)
[   23.993316] 
[   24.004298] nfs_read_alloc_scratch(hdr=c2732300, size=16) = c41c7200
[   24.010685]   nfs4_xdr_enc_read_plus(hdr=c2732300, decode_read_plus_maxsz=9)
[   24.010685]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.029421]   nfs4_xdr_dec_read_plus(hdr=c2732300, scratch=c41c7200)
[   24.034576]     decode_read_plus_segment(n=0)
[   24.038729]       decode_read_plus_segment(n=0): DATA {offset=16384, length=114688}
[   24.046537]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.046966]   nfs_readhdr_free(hdr=c2732300, scratch=c41c7200)
[   24.063312] 
[   24.063543] nfs_read_alloc_scratch(hdr=c2732580, size=16) = c41c7280
[   24.070144]   nfs4_xdr_enc_read_plus(hdr=c2732580, decode_read_plus_maxsz=9)
[   24.070144]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.091084]   nfs4_xdr_dec_read_plus(hdr=c2732580, scratch=c41c7280)
[   24.096072]     decode_read_plus_segment(n=0)
[   24.100341]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   24.107597]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.107700]   nfs_readhdr_free(hdr=c2732580, scratch=c41c7280)
[   24.111690] 
[   24.120915] nfs_read_alloc_scratch(hdr=c2732800, size=16) = c41c74c0
[   24.127461]   nfs4_xdr_enc_read_plus(hdr=c2732800, decode_read_plus_maxsz=9)
[   24.127461]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.142664]   nfs4_xdr_dec_read_plus(hdr=c2732800, scratch=c41c74c0)
[   24.147561]     decode_read_plus_segment(n=0)
[   24.151938]       decode_read_plus_segment(n=0): DATA {offset=16384, length=13736}
[   24.159440]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.159549]   nfs_readhdr_free(hdr=c2732800, scratch=c41c74c0)
[   24.175242] 
[   24.175491] nfs_read_alloc_scratch(hdr=c2732a80, size=16) = c41c7600
[   24.182255]   nfs4_xdr_enc_read_plus(hdr=c2732a80, decode_read_plus_maxsz=9)
[   24.182255]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.198244]   nfs4_xdr_dec_read_plus(hdr=c2732a80, scratch=c41c7600)
[   24.203368]     decode_read_plus_segment(n=0)
[   24.207516]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   24.214775]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.214968]   nfs_readhdr_free(hdr=c2732a80, scratch=c41c7600)
[   24.217966] 
[   24.228363] nfs_read_alloc_scratch(hdr=c2732d00, size=16) = c41c7f00
[   24.234565]   nfs4_xdr_enc_read_plus(hdr=c2732d00, decode_read_plus_maxsz=9)
[   24.234565]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.252198]   nfs4_xdr_dec_read_plus(hdr=c2732d00, scratch=c41c7f00)
[   24.257184]     decode_read_plus_segment(n=0)
[   24.261589]       decode_read_plus_segment(n=0): DATA {offset=249856, length=67584}
[   24.269143]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.269458]   nfs_readhdr_free(hdr=c2732d00, scratch=c41c7f00)
[   24.280878] 
[   24.282439] nfs_read_alloc_scratch(hdr=c2732f80, size=16) = c41c7c00
[   24.288783]   nfs4_xdr_enc_read_plus(hdr=c2732f80, decode_read_plus_maxsz=9)
[   24.288783]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.304932]   nfs4_xdr_dec_read_plus(hdr=c2732f80, scratch=c41c7c00)
[   24.309892]     decode_read_plus_segment(n=0)
[   24.314482]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   24.321533]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.321717]   nfs_readhdr_free(hdr=c2732f80, scratch=c41c7c00)
[   24.324292] 
[   24.334656] nfs_read_alloc_scratch(hdr=c2733200, size=16) = c41c7e00
[   24.341400]   nfs4_xdr_enc_read_plus(hdr=c2733200, decode_read_plus_maxsz=9)
[   24.341400]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.356873]   nfs4_xdr_dec_read_plus(hdr=c2733200, scratch=c41c7e00)
[   24.361914]     decode_read_plus_segment(n=0)
[   24.366127]       decode_read_plus_segment(n=0): DATA {offset=16384, length=17904}
[   24.373744]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.373936]   nfs_readhdr_free(hdr=c2733200, scratch=c41c7e00)
[   24.387331] 
[   24.387424] nfs_read_alloc_scratch(hdr=c3194f00, size=16) = c24cce40
[   24.393883]   nfs4_xdr_enc_read_plus(hdr=c3194f00, decode_read_plus_maxsz=9)
[   24.393883]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.409658]   nfs4_xdr_dec_read_plus(hdr=c3194f00, scratch=c24cce40)
[   24.414638]     decode_read_plus_segment(n=0)
[   24.418901]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   24.426315]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.426463]   nfs_readhdr_free(hdr=c3194f00, scratch=c24cce40)
[   24.429802] 
[   24.439434] nfs_read_alloc_scratch(hdr=c3195180, size=16) = c24cc480
[   24.446084]   nfs4_xdr_enc_read_plus(hdr=c3195180, decode_read_plus_maxsz=9)
[   24.446084]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.464095]   nfs4_xdr_dec_read_plus(hdr=c3195180, scratch=c24cc480)
[   24.469480]     decode_read_plus_segment(n=0)
[   24.473443]       decode_read_plus_segment(n=0): DATA {offset=126976, length=67124}
[   24.481052]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.481394]   nfs_readhdr_free(hdr=c3195180, scratch=c24cc480)
[   24.499263] 
[   24.499554] nfs_read_alloc_scratch(hdr=c3195400, size=16) = c24cc280
[   24.506360]   nfs4_xdr_enc_read_plus(hdr=c3195400, decode_read_plus_maxsz=9)
[   24.506360]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.522941]   nfs4_xdr_dec_read_plus(hdr=c3195400, scratch=c24cc280)
[   24.527837]     decode_read_plus_segment(n=0)
[   24.532218]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   24.539368]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.539482]   nfs_readhdr_free(hdr=c3195400, scratch=c24cc280)
[   24.542914] 
[   24.552980] nfs_read_alloc_scratch(hdr=c3195680, size=16) = c24cc540
[   24.559186]   nfs4_xdr_enc_read_plus(hdr=c3195680, decode_read_plus_maxsz=9)
[   24.559186]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.576804]   nfs4_xdr_dec_read_plus(hdr=c3195680, scratch=c24cc540)
[   24.581815]     decode_read_plus_segment(n=0)
[   24.586112]       decode_read_plus_segment(n=0): DATA {offset=888832, length=70620}
[   24.593804]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.594114]   nfs_readhdr_free(hdr=c3195680, scratch=c24cc540)
[   24.608394] 
[   24.608595] nfs_read_alloc_scratch(hdr=c2733480, size=16) = c41c7a80
[   24.615022]   nfs4_xdr_enc_read_plus(hdr=c2733480, decode_read_plus_maxsz=9)
[   24.615022]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.630859]   nfs4_xdr_dec_read_plus(hdr=c2733480, scratch=c41c7a80)
[   24.635808]     decode_read_plus_segment(n=0)
[   24.640084]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   24.647329]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.647435]   nfs_readhdr_free(hdr=c2733480, scratch=c41c7a80)
[   24.651534] 
[   24.660693] nfs_read_alloc_scratch(hdr=c2733700, size=16) = c41c7900
[   24.667234]   nfs4_xdr_enc_read_plus(hdr=c2733700, decode_read_plus_maxsz=9)
[   24.667234]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.682715]   nfs4_xdr_dec_read_plus(hdr=c2733700, scratch=c41c7900)
[   24.687622]     decode_read_plus_segment(n=0)
[   24.691991]       decode_read_plus_segment(n=0): DATA {offset=16384, length=9852}
[   24.699414]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.699518]   nfs_readhdr_free(hdr=c2733700, scratch=c41c7900)
[   24.715516] 
[   24.715743] nfs_read_alloc_scratch(hdr=c2733980, size=16) = c41c7500
[   24.722346]   nfs4_xdr_enc_read_plus(hdr=c2733980, decode_read_plus_maxsz=9)
[   24.722346]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.738296]   nfs4_xdr_dec_read_plus(hdr=c2733980, scratch=c41c7500)
[   24.743334]     decode_read_plus_segment(n=0)
[   24.747569]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   24.754850]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.754974]   nfs_readhdr_free(hdr=c2733980, scratch=c41c7500)
[   24.759219] 
[   24.768140] nfs_read_alloc_scratch(hdr=c2733c00, size=16) = c41c7900
[   24.775303]   nfs4_xdr_enc_read_plus(hdr=c2733c00, decode_read_plus_maxsz=9)
[   24.775303]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.792119]   nfs4_xdr_dec_read_plus(hdr=c2733c00, scratch=c41c7900)
[   24.797100]     decode_read_plus_segment(n=0)
[   24.801460]       decode_read_plus_segment(n=0): DATA {offset=61440, length=66936}
[   24.808999]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.809258]   nfs_readhdr_free(hdr=c2733c00, scratch=c41c7900)
[   24.810666] 
[   24.822565] nfs_read_alloc_scratch(hdr=c2733980, size=16) = c41c7f00
[   24.828871]   nfs4_xdr_enc_read_plus(hdr=c2733980, decode_read_plus_maxsz=9)
[   24.828871]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.836727] 
[   24.844009] nfs_read_alloc_scratch(hdr=c2733700, size=16) = c41c7c00
[   24.846236]   nfs4_xdr_dec_read_plus(hdr=c2733980, scratch=c41c7f00)
[   24.850517]   nfs4_xdr_enc_read_plus(hdr=c2733700, decode_read_plus_maxsz=9)
[   24.850517]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.856268]     decode_read_plus_segment(n=0)
[   24.856283]       decode_read_plus_segment(n=0): DATA {offset=16384, length=45056}
[   24.875866]   nfs4_xdr_dec_read_plus(hdr=c2733700, scratch=c41c7c00)
[   24.881738]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.881990]   nfs_readhdr_free(hdr=c2733980, scratch=c41c7f00)
[   24.888679]     decode_read_plus_segment(n=0)
[   24.904321]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   24.911454]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.911573]   nfs_readhdr_free(hdr=c2733700, scratch=c41c7c00)
[   24.916211] 
[   24.924830] nfs_read_alloc_scratch(hdr=c3195900, size=16) = c24cc480
[   24.931164]   nfs4_xdr_enc_read_plus(hdr=c3195900, decode_read_plus_maxsz=9)
[   24.931164]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   24.950194]   nfs4_xdr_dec_read_plus(hdr=c3195900, scratch=c24cc480)
[   24.955252]     decode_read_plus_segment(n=0)
[   24.959473]       decode_read_plus_segment(n=0): DATA {offset=3162112, length=67264}
[   24.967287]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   24.967541]   nfs_readhdr_free(hdr=c3195900, scratch=c24cc480)
[   24.970453] 
[   24.980659] nfs_read_alloc_scratch(hdr=c3195b80, size=16) = c40000c0
[   24.987348]   nfs4_xdr_enc_read_plus(hdr=c3195b80, decode_read_plus_maxsz=9)
[   24.987348]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.006208]   nfs4_xdr_dec_read_plus(hdr=c3195b80, scratch=c40000c0)
[   25.011176]     decode_read_plus_segment(n=0)
[   25.015532]       decode_read_plus_segment(n=0): DATA {offset=167936, length=131072}
[   25.023323]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.023694]   nfs_readhdr_free(hdr=c3195b80, scratch=c40000c0)
[   25.045086] 
[   25.045421] nfs_read_alloc_scratch(hdr=c3195e00, size=16) = c40006c0
[   25.052322]   nfs4_xdr_enc_read_plus(hdr=c3195e00, decode_read_plus_maxsz=9)
[   25.052322]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.067658]   nfs4_xdr_dec_read_plus(hdr=c3195e00, scratch=c40006c0)
[   25.072641]     decode_read_plus_segment(n=0)
[   25.076893]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   25.084158]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.084291]   nfs_readhdr_free(hdr=c3195e00, scratch=c40006c0)
[   25.089549] 
[   25.097987] nfs_read_alloc_scratch(hdr=c3196080, size=16) = c4000b80
[   25.104315]   nfs4_xdr_enc_read_plus(hdr=c3196080, decode_read_plus_maxsz=9)
[   25.104315]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.122105]   nfs4_xdr_dec_read_plus(hdr=c3196080, scratch=c4000b80)
[   25.127032]     decode_read_plus_segment(n=0)
[   25.131415]       decode_read_plus_segment(n=0): DATA {offset=98304, length=67036}
[   25.138938]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.139262]   nfs_readhdr_free(hdr=c3196080, scratch=c4000b80)
[   25.140438] 
[   25.152756] nfs_read_alloc_scratch(hdr=c3196300, size=16) = c4000940
[   25.158731]   nfs4_xdr_enc_read_plus(hdr=c3196300, decode_read_plus_maxsz=9)
[   25.158731]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.173201] 
[   25.173601] nfs_read_alloc_scratch(hdr=c2733480, size=16) = c41c7c80
[   25.176209]   nfs4_xdr_dec_read_plus(hdr=c3196300, scratch=c4000940)
[   25.180043]   nfs4_xdr_enc_read_plus(hdr=c2733480, decode_read_plus_maxsz=9)
[   25.180043]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.186230]     decode_read_plus_segment(n=0)
[   25.203538]   nfs4_xdr_dec_read_plus(hdr=c2733480, scratch=c41c7c80)
[   25.204173]       decode_read_plus_segment(n=0): DATA {offset=16384, length=81920}
[   25.210441]     decode_read_plus_segment(n=0)
[   25.218093]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.222464]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   25.222799]   nfs_readhdr_free(hdr=c3196300, scratch=c4000940)
[   25.228461]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.241542]   nfs_readhdr_free(hdr=c2733480, scratch=c41c7c80)
[   25.245029] 
[   25.254574] nfs_read_alloc_scratch(hdr=c40e8000, size=16) = c24afec0
[   25.260935]   nfs4_xdr_enc_read_plus(hdr=c40e8000, decode_read_plus_maxsz=9)
[   25.260935]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.279387]   nfs4_xdr_dec_read_plus(hdr=c40e8000, scratch=c24afec0)
[   25.284337]     decode_read_plus_segment(n=0)
[   25.288608]       decode_read_plus_segment(n=0): DATA {offset=667648, length=67028}
[   25.296300]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.296510]   nfs_readhdr_free(hdr=c40e8000, scratch=c24afec0)
[   25.298331] 
[   25.309730] nfs_read_alloc_scratch(hdr=c40e8280, size=16) = c24af700
[   25.315976]   nfs4_xdr_enc_read_plus(hdr=c40e8280, decode_read_plus_maxsz=9)
[   25.315976]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.336463]   nfs4_xdr_dec_read_plus(hdr=c40e8280, scratch=c24af700)
[   25.341428]     decode_read_plus_segment(n=0)
[   25.345690]       decode_read_plus_segment(n=0): DATA {offset=16384, length=114688}
[   25.353412]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.353809]   nfs_readhdr_free(hdr=c40e8280, scratch=c24af700)
[   25.360091] 
[   25.366595] nfs_read_alloc_scratch(hdr=c2749180, size=16) = c4080440
[   25.373252]   nfs4_xdr_enc_read_plus(hdr=c2749180, decode_read_plus_maxsz=9)
[   25.373252]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.389021]   nfs4_xdr_dec_read_plus(hdr=c2749180, scratch=c4080440)
[   25.394083]     decode_read_plus_segment(n=0)
[   25.398288]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   25.405558]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.405682]   nfs_readhdr_free(hdr=c2749180, scratch=c4080440)
[   25.409011] 
[   25.418717] nfs_read_alloc_scratch(hdr=c2749400, size=16) = c4080dc0
[   25.425106]   nfs4_xdr_enc_read_plus(hdr=c2749400, decode_read_plus_maxsz=9)
[   25.425106]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.444489]   nfs4_xdr_dec_read_plus(hdr=c2749400, scratch=c4080dc0)
[   25.449464]     decode_read_plus_segment(n=0)
[   25.453820]       decode_read_plus_segment(n=0): DATA {offset=233472, length=67284}
[   25.461498]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.461765]   nfs_readhdr_free(hdr=c2749400, scratch=c4080dc0)
[   25.463678] 
[   25.474622] nfs_read_alloc_scratch(hdr=c2749680, size=16) = c4080480
[   25.480978]   nfs4_xdr_enc_read_plus(hdr=c2749680, decode_read_plus_maxsz=9)
[   25.480978]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.502384]   nfs4_xdr_dec_read_plus(hdr=c2749680, scratch=c4080480)
[   25.507304]     decode_read_plus_segment(n=0)
[   25.511693]       decode_read_plus_segment(n=0): DATA {offset=16384, length=114688}
[   25.519299]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.519712]   nfs_readhdr_free(hdr=c2749680, scratch=c4080480)
[   25.531537] 
[   25.532505] nfs_read_alloc_scratch(hdr=c2749900, size=16) = c4080440
[   25.538885]   nfs4_xdr_enc_read_plus(hdr=c2749900, decode_read_plus_maxsz=9)
[   25.538885]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.555220]   nfs4_xdr_dec_read_plus(hdr=c2749900, scratch=c4080440)
[   25.560121]     decode_read_plus_segment(n=0)
[   25.564501]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   25.571699]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.571818]   nfs_readhdr_free(hdr=c2749900, scratch=c4080440)
[   25.574944] 
[   25.584917] nfs_read_alloc_scratch(hdr=c2749b80, size=16) = c4080a00
[   25.591332]   nfs4_xdr_enc_read_plus(hdr=c2749b80, decode_read_plus_maxsz=9)
[   25.591332]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.607079]   nfs4_xdr_dec_read_plus(hdr=c2749b80, scratch=c4080a00)
[   25.612053]     decode_read_plus_segment(n=0)
[   25.616314]       decode_read_plus_segment(n=0): DATA {offset=16384, length=5540}
[   25.623847]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.623965]   nfs_readhdr_free(hdr=c2749b80, scratch=c4080a00)
[   25.638404] 
[   25.638659] nfs_read_alloc_scratch(hdr=c40f9400, size=16) = c2c29880
[   25.645319]   nfs4_xdr_enc_read_plus(hdr=c40f9400, decode_read_plus_maxsz=9)
[   25.645319]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.663208]   nfs4_xdr_dec_read_plus(hdr=c40f9400, scratch=c2c29880)
[   25.668151]     decode_read_plus_segment(n=0)
[   25.672522]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   25.679673]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.679824]   nfs_readhdr_free(hdr=c40f9400, scratch=c2c29880)
[   25.683112] 
[   25.692961] nfs_read_alloc_scratch(hdr=c40f9680, size=16) = c2c29c00
[   25.699511]   nfs4_xdr_enc_read_plus(hdr=c40f9680, decode_read_plus_maxsz=9)
[   25.699511]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.717286]   nfs4_xdr_dec_read_plus(hdr=c40f9680, scratch=c2c29c00)
[   25.722352]     decode_read_plus_segment(n=0)
[   25.726592]       decode_read_plus_segment(n=0): DATA {offset=16384, length=62984}
[   25.734249]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.734560]   nfs_readhdr_free(hdr=c40f9680, scratch=c2c29c00)
[   25.750574] 
[   25.750834] nfs_read_alloc_scratch(hdr=c1d20280, size=16) = c4338ec0
[   25.757813]   nfs4_xdr_enc_read_plus(hdr=c1d20280, decode_read_plus_maxsz=9)
[   25.757813]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.773255]   nfs4_xdr_dec_read_plus(hdr=c1d20280, scratch=c4338ec0)
[   25.778220]     decode_read_plus_segment(n=0)
[   25.782579]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   25.789769]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.789902]   nfs_readhdr_free(hdr=c1d20280, scratch=c4338ec0)
[   25.794230] 
[   25.803095] nfs_read_alloc_scratch(hdr=c40f9900, size=16) = c2c29200
[   25.809597]   nfs4_xdr_enc_read_plus(hdr=c40f9900, decode_read_plus_maxsz=9)
[   25.809597]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.828832]   nfs4_xdr_dec_read_plus(hdr=c40f9900, scratch=c2c29200)
[   25.833893]     decode_read_plus_segment(n=0)
[   25.838191]       decode_read_plus_segment(n=0): DATA {offset=65536, length=67072}
[   25.845783]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.846064]   nfs_readhdr_free(hdr=c40f9900, scratch=c2c29200)
[   25.849295] 
[   25.859222] nfs_read_alloc_scratch(hdr=c40f9b80, size=16) = c2c29340
[   25.865756]   nfs4_xdr_enc_read_plus(hdr=c40f9b80, decode_read_plus_maxsz=9)
[   25.865756]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.867406] 
[   25.880916] nfs_read_alloc_scratch(hdr=c40f9e00, size=16) = c2c29f00
[   25.882263]   nfs4_xdr_dec_read_plus(hdr=c40f9b80, scratch=c2c29340)
[   25.887115]   nfs4_xdr_enc_read_plus(hdr=c40f9e00, decode_read_plus_maxsz=9)
[   25.887115]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.893011]     decode_read_plus_segment(n=0)
[   25.911010]       decode_read_plus_segment(n=0): DATA {offset=16384, length=16384}
[   25.918557]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.918684]   nfs_readhdr_free(hdr=c40f9b80, scratch=c2c29340)
[   25.919653] 
[   25.922792]   nfs4_xdr_dec_read_plus(hdr=c40f9e00, scratch=c2c29f00)
[   25.922847]     decode_read_plus_segment(n=0)
[   25.922883]       decode_read_plus_segment(n=0): DATA {offset=163840, length=114688}
[   25.923316]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   25.923770]   nfs_readhdr_free(hdr=c40f9e00, scratch=c2c29f00)
[   25.961955] nfs_read_alloc_scratch(hdr=c1d20500, size=16) = c43387c0
[   25.968304]   nfs4_xdr_enc_read_plus(hdr=c1d20500, decode_read_plus_maxsz=9)
[   25.968304]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.968800] 
[   25.983572] nfs_read_alloc_scratch(hdr=c1d20780, size=16) = c43386c0
[   25.989837]   nfs4_xdr_enc_read_plus(hdr=c1d20780, decode_read_plus_maxsz=9)
[   25.989837]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   25.998102]   nfs4_xdr_dec_read_plus(hdr=c1d20500, scratch=c43387c0)
[   26.005675]   nfs4_xdr_dec_read_plus(hdr=c1d20780, scratch=c43386c0)
[   26.009744]     decode_read_plus_segment(n=0)
[   26.015892]     decode_read_plus_segment(n=0)
[   26.020300]       decode_read_plus_segment(n=0): DATA {offset=16384, length=122880}
[   26.024557]       decode_read_plus_segment(n=0): DATA {offset=16384, length=4096}
[   26.024571]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.032872]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.039748]   nfs_readhdr_free(hdr=c1d20780, scratch=c43386c0)
[   26.046271]   nfs_readhdr_free(hdr=c1d20500, scratch=c43387c0)
[   26.047450] 
[   26.047465] nfs_read_alloc_scratch(hdr=c1ff4500, size=16) = c24ce980
[   26.047728]   nfs4_xdr_enc_read_plus(hdr=c1ff4500, decode_read_plus_maxsz=9)
[   26.047728]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.048356] 
[   26.048396] nfs_read_alloc_scratch(hdr=c1ff4780, size=16) = c24cef80
[   26.048682]   nfs4_xdr_enc_read_plus(hdr=c1ff4780, decode_read_plus_maxsz=9)
[   26.048682]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.055088]   nfs4_xdr_dec_read_plus(hdr=c1ff4500, scratch=c24ce980)
[   26.055942]   nfs4_xdr_dec_read_plus(hdr=c1ff4780, scratch=c24cef80)
[   26.055973]     decode_read_plus_segment(n=0)
[   26.056006]       decode_read_plus_segment(n=0): DATA {offset=270336, length=8192}
[   26.056069]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.056323]   nfs_readhdr_free(hdr=c1ff4780, scratch=c24cef80)
[   26.142519]     decode_read_plus_segment(n=0)
[   26.146615]       decode_read_plus_segment(n=0): DATA {offset=16384, length=32768}
[   26.154427]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.154597]   nfs_readhdr_free(hdr=c1ff4500, scratch=c24ce980)
[   26.156047] 
[   26.167432] nfs_read_alloc_scratch(hdr=c32c0000, size=16) = c4261a40
[   26.173960]   nfs4_xdr_enc_read_plus(hdr=c32c0000, decode_read_plus_maxsz=9)
[   26.173960]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.175298] 
[   26.188855] nfs_read_alloc_scratch(hdr=c32c0280, size=16) = c1e7c980
[   26.195287]   nfs4_xdr_enc_read_plus(hdr=c32c0280, decode_read_plus_maxsz=9)
[   26.195287]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.196357] 
[   26.200799]   nfs4_xdr_dec_read_plus(hdr=c32c0000, scratch=c4261a40)
[   26.200843]     decode_read_plus_segment(n=0)
[   26.200883]       decode_read_plus_segment(n=0): DATA {offset=180224, length=98304}
[   26.201146]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.201657]   nfs_readhdr_free(hdr=c32c0000, scratch=c4261a40)
[   26.227449]   nfs4_xdr_dec_read_plus(hdr=c32c0280, scratch=c1e7c980)
[   26.228544] nfs_read_alloc_scratch(hdr=c32c0500, size=16) = c2c42f00
[   26.234518]     decode_read_plus_segment(n=0)
[   26.240505]   nfs4_xdr_enc_read_plus(hdr=c32c0500, decode_read_plus_maxsz=9)
[   26.240505]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.246646]       decode_read_plus_segment(n=0): DATA {offset=16384, length=131072}
[   26.246880]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.259807]   nfs4_xdr_dec_read_plus(hdr=c32c0500, scratch=c2c42f00)
[   26.271511]   nfs_readhdr_free(hdr=c32c0280, scratch=c1e7c980)
[   26.273442] 
[   26.273459] nfs_read_alloc_scratch(hdr=c32c0780, size=16) = c4261e40
[   26.273747]   nfs4_xdr_enc_read_plus(hdr=c32c0780, decode_read_plus_maxsz=9)
[   26.273747]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.274208] 
[   26.274230] nfs_read_alloc_scratch(hdr=c32c0a00, size=16) = c42611c0
[   26.274433]   nfs4_xdr_enc_read_plus(hdr=c32c0a00, decode_read_plus_maxsz=9)
[   26.274433]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.275600] 
[   26.275621] nfs_read_alloc_scratch(hdr=c32c0c80, size=16) = c40d9cc0
[   26.275870]   nfs4_xdr_enc_read_plus(hdr=c32c0c80, decode_read_plus_maxsz=9)
[   26.275870]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.278344] 
[   26.278378] nfs_read_alloc_scratch(hdr=c32c0f00, size=16) = c3d7e2c0
[   26.278636]     decode_read_plus_segment(n=0)
[   26.278802]   nfs4_xdr_enc_read_plus(hdr=c32c0f00, decode_read_plus_maxsz=9)
[   26.278802]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.281558] 
[   26.281594] nfs_read_alloc_scratch(hdr=c32c1180, size=16) = c3d7eb00
[   26.282891]   nfs4_xdr_enc_read_plus(hdr=c32c1180, decode_read_plus_maxsz=9)
[   26.282891]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.290579]   nfs4_xdr_dec_read_plus(hdr=c32c0780, scratch=c4261e40)
[   26.290792]       decode_read_plus_segment(n=0): DATA {offset=16384, length=49152}
[   26.292430]   nfs4_xdr_dec_read_plus(hdr=c32c0a00, scratch=c42611c0)
[   26.292460]     decode_read_plus_segment(n=0)
[   26.292474]       decode_read_plus_segment(n=0): DATA {offset=147456, length=20480}
[   26.292504]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.292675]   nfs_readhdr_free(hdr=c32c0a00, scratch=c42611c0)
[   26.296685]     decode_read_plus_segment(n=0)
[   26.298175]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.299628]   nfs4_xdr_dec_read_plus(hdr=c32c0c80, scratch=c40d9cc0)
[   26.299654]     decode_read_plus_segment(n=0)
[   26.299666]       decode_read_plus_segment(n=0): DATA {offset=131072, length=102400}
[   26.299739]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.300024]   nfs_readhdr_free(hdr=c32c0c80, scratch=c40d9cc0)
[   26.304432]       decode_read_plus_segment(n=0): DATA {offset=131072, length=131072}
[   26.304602]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.306189]   nfs4_xdr_dec_read_plus(hdr=c32c0f00, scratch=c3d7e2c0)
[   26.306230]     decode_read_plus_segment(n=0)
[   26.306244]       decode_read_plus_segment(n=0): DATA {offset=131072, length=131072}
[   26.307032]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.308963]   nfs_readhdr_free(hdr=c32c0f00, scratch=c3d7e2c0)
[   26.309017]   nfs4_xdr_dec_read_plus(hdr=c32c1180, scratch=c3d7eb00)
[   26.309042]     decode_read_plus_segment(n=0)
[   26.309055]       decode_read_plus_segment(n=0): DATA {offset=2973696, length=131072}
[   26.309124]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.309443]   nfs_readhdr_free(hdr=c32c1180, scratch=c3d7eb00)
[   26.311702] 
[   26.311720] nfs_read_alloc_scratch(hdr=c32c1400, size=16) = c3d7ef40
[   26.311846]   nfs4_xdr_enc_read_plus(hdr=c32c1400, decode_read_plus_maxsz=9)
[   26.311846]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.313839] 
[   26.313858] nfs_read_alloc_scratch(hdr=c32c1680, size=16) = c3d7e380
[   26.314089]   nfs4_xdr_enc_read_plus(hdr=c32c1680, decode_read_plus_maxsz=9)
[   26.314089]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.315155]   nfs4_xdr_dec_read_plus(hdr=c32c1400, scratch=c3d7ef40)
[   26.315190]     decode_read_plus_segment(n=0)
[   26.315204]       decode_read_plus_segment(n=0): DATA {offset=3104768, length=57344}
[   26.315243]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.315432]   nfs_readhdr_free(hdr=c32c1400, scratch=c3d7ef40)
[   26.317729] 
[   26.317751] nfs_read_alloc_scratch(hdr=c32c1900, size=16) = c3f6a080
[   26.317887]   nfs4_xdr_enc_read_plus(hdr=c32c1900, decode_read_plus_maxsz=9)
[   26.317887]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.318429]   nfs_readhdr_free(hdr=c32c0500, scratch=c2c42f00)
[   26.319870]   nfs_readhdr_free(hdr=c32c0780, scratch=c4261e40)
[   26.327470]   nfs4_xdr_dec_read_plus(hdr=c32c1680, scratch=c3d7e380)
[   26.336572]   nfs4_xdr_dec_read_plus(hdr=c32c1900, scratch=c3f6a080)
[   26.336603]     decode_read_plus_segment(n=0)
[   26.336613]       decode_read_plus_segment(n=0): DATA {offset=430080, length=131072}
[   26.336678]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.336951]   nfs_readhdr_free(hdr=c32c1900, scratch=c3f6a080)
[   26.709214]     decode_read_plus_segment(n=0)
[   26.713302]       decode_read_plus_segment(n=0): DATA {offset=299008, length=131072}
[   26.721007]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.721652]   nfs_readhdr_free(hdr=c32c1680, scratch=c3d7e380)
[   26.734193] 
[   26.734270] nfs_read_alloc_scratch(hdr=c32c1b80, size=16) = c3f6a800
[   26.740682]   nfs4_xdr_enc_read_plus(hdr=c32c1b80, decode_read_plus_maxsz=9)
[   26.740682]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.763427]   nfs4_xdr_dec_read_plus(hdr=c32c1b80, scratch=c3f6a800)
[   26.768354]     decode_read_plus_segment(n=0)
[   26.772729]       decode_read_plus_segment(n=0): DATA {offset=16384, length=114688}
[   26.780349]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.780769]   nfs_readhdr_free(hdr=c32c1b80, scratch=c3f6a800)
[   26.782671] 
[   26.793569] nfs_read_alloc_scratch(hdr=c32c1e00, size=16) = c3f6a640
[   26.800015]   nfs4_xdr_enc_read_plus(hdr=c32c1e00, decode_read_plus_maxsz=9)
[   26.800015]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.803537] 
[   26.814983] nfs_read_alloc_scratch(hdr=c32c2080, size=16) = c40d9880
[   26.821479]   nfs4_xdr_enc_read_plus(hdr=c32c2080, decode_read_plus_maxsz=9)
[   26.821479]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.835207]   nfs4_xdr_dec_read_plus(hdr=c32c1e00, scratch=c3f6a640)
[   26.841307]     decode_read_plus_segment(n=0)
[   26.845544]       decode_read_plus_segment(n=0): DATA {offset=131072, length=131072}
[   26.851171]   nfs4_xdr_dec_read_plus(hdr=c32c2080, scratch=c40d9880)
[   26.853442]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.859565]     decode_read_plus_segment(n=0)
[   26.859930]   nfs_readhdr_free(hdr=c32c1e00, scratch=c3f6a640)
[   26.865573]       decode_read_plus_segment(n=0): DATA {offset=16384, length=114688}
[   26.883416]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.883675]   nfs_readhdr_free(hdr=c32c2080, scratch=c40d9880)
[   26.884374] 
[   26.896587] nfs_read_alloc_scratch(hdr=c1ff4500, size=16) = c24ceb40
[   26.903034]   nfs4_xdr_enc_read_plus(hdr=c1ff4500, decode_read_plus_maxsz=9)
[   26.903034]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.908995] 
[   26.918056] nfs_read_alloc_scratch(hdr=c1ff4780, size=16) = c3f6a200
[   26.924475]   nfs4_xdr_enc_read_plus(hdr=c1ff4780, decode_read_plus_maxsz=9)
[   26.924475]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.929494]   nfs4_xdr_dec_read_plus(hdr=c1ff4500, scratch=c24ceb40)
[   26.932632] 
[   26.932649] nfs_read_alloc_scratch(hdr=c1ff4a00, size=16) = c3f6ad00
[   26.932859]   nfs4_xdr_enc_read_plus(hdr=c1ff4a00, decode_read_plus_maxsz=9)
[   26.932859]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   26.948365] 
[   26.951209]   nfs4_xdr_dec_read_plus(hdr=c1ff4a00, scratch=c3f6ad00)
[   26.951366]     decode_read_plus_segment(n=0)
[   26.951387]       decode_read_plus_segment(n=0): DATA {offset=278528, length=131072}
[   26.951496]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.952335]   nfs_readhdr_free(hdr=c1ff4a00, scratch=c3f6ad00)
[   26.952423]     decode_read_plus_segment(n=0)
[   26.952446]       decode_read_plus_segment(n=0): DATA {offset=131072, length=118784}
[   26.952885]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.954689]   nfs_readhdr_free(hdr=c1ff4500, scratch=c24ceb40)
[   26.961010]   nfs4_xdr_dec_read_plus(hdr=c1ff4780, scratch=c3f6a200)
[   26.961047]     decode_read_plus_segment(n=0)
[   26.961084]       decode_read_plus_segment(n=0): DATA {offset=16384, length=40960}
[   26.961411]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   26.961854]   nfs_readhdr_free(hdr=c1ff4780, scratch=c3f6a200)
[   26.965779] nfs_read_alloc_scratch(hdr=c1ff4c80, size=16) = c40b0400
[   27.057626]   nfs4_xdr_enc_read_plus(hdr=c1ff4c80, decode_read_plus_maxsz=9)
[   27.057626]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.059826] 
[   27.072554] nfs_read_alloc_scratch(hdr=c1ff4f00, size=16) = c40b0c80
[   27.078935]   nfs4_xdr_enc_read_plus(hdr=c1ff4f00, decode_read_plus_maxsz=9)
[   27.078935]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.085431]   nfs4_xdr_dec_read_plus(hdr=c1ff4c80, scratch=c40b0400)
[   27.098782]     decode_read_plus_segment(n=0)
[   27.103140]       decode_read_plus_segment(n=0): DATA {offset=409600, length=131072}
[   27.110794]   nfs4_xdr_dec_read_plus(hdr=c1ff4f00, scratch=c40b0c80)
[   27.111124]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.117206]     decode_read_plus_segment(n=0)
[   27.117531]   nfs_readhdr_free(hdr=c1ff4c80, scratch=c40b0400)
[   27.123194]       decode_read_plus_segment(n=0): DATA {offset=1511424, length=131072}
[   27.141176]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.141692]   nfs_readhdr_free(hdr=c1ff4f00, scratch=c40b0c80)
[   27.144005] 
[   27.154301] nfs_read_alloc_scratch(hdr=c32c2080, size=16) = c40d9900
[   27.160733]   nfs4_xdr_enc_read_plus(hdr=c32c2080, decode_read_plus_maxsz=9)
[   27.160733]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.177918]   nfs4_xdr_dec_read_plus(hdr=c32c2080, scratch=c40d9900)
[   27.182870]     decode_read_plus_segment(n=0)
[   27.187149]       decode_read_plus_segment(n=0): DATA {offset=1642496, length=36864}
[   27.194920]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.195107]   nfs_readhdr_free(hdr=c32c2080, scratch=c40d9900)
[   27.202139] 
[   27.208135] nfs_read_alloc_scratch(hdr=c32c2300, size=16) = c3f6aa40
[   27.214556]   nfs4_xdr_enc_read_plus(hdr=c32c2300, decode_read_plus_maxsz=9)
[   27.214556]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.219270] 
[   27.229565] nfs_read_alloc_scratch(hdr=c32c2580, size=16) = c3f6aec0
[   27.236032]   nfs4_xdr_enc_read_plus(hdr=c32c2580, decode_read_plus_maxsz=9)
[   27.236032]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.248830]   nfs4_xdr_dec_read_plus(hdr=c32c2300, scratch=c3f6aa40)
[   27.255890]     decode_read_plus_segment(n=0)
[   27.260128]       decode_read_plus_segment(n=0): DATA {offset=151552, length=131072}
[   27.267956]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.268296]   nfs4_xdr_dec_read_plus(hdr=c32c2580, scratch=c3f6aec0)
[   27.268518]   nfs_readhdr_free(hdr=c32c2300, scratch=c3f6aa40)
[   27.270592] 
[   27.270618] nfs_read_alloc_scratch(hdr=c2749e00, size=16) = c4080b80
[   27.270832]   nfs4_xdr_enc_read_plus(hdr=c2749e00, decode_read_plus_maxsz=9)
[   27.270832]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.273326]   nfs4_xdr_dec_read_plus(hdr=c2749e00, scratch=c4080b80)
[   27.273360]     decode_read_plus_segment(n=0)
[   27.273373] 
[   27.273376]       decode_read_plus_segment(n=0): DATA {offset=16384, length=4096}
[   27.273401]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.273412] nfs_read_alloc_scratch(hdr=c274a080, size=16) = c40d9700
[   27.273572]   nfs4_xdr_enc_read_plus(hdr=c274a080, decode_read_plus_maxsz=9)
[   27.273572]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.273795]   nfs_readhdr_free(hdr=c2749e00, scratch=c4080b80)
[   27.273933]     decode_read_plus_segment(n=0)
[   27.280809]   nfs4_xdr_dec_read_plus(hdr=c274a080, scratch=c40d9700)
[   27.285961]       decode_read_plus_segment(n=0): DATA {offset=282624, length=131072}
[   27.287394]     decode_read_plus_segment(n=0)
[   27.293922]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.307362]       decode_read_plus_segment(n=0): DATA {offset=1302528, length=131072}
[   27.314036]   nfs_readhdr_free(hdr=c32c2580, scratch=c3f6aec0)
[   27.318179]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.401495]   nfs_readhdr_free(hdr=c274a080, scratch=c40d9700)
[   27.404492] 
[   27.414202] nfs_read_alloc_scratch(hdr=c32c2800, size=16) = c3f6a040
[   27.420608]   nfs4_xdr_enc_read_plus(hdr=c32c2800, decode_read_plus_maxsz=9)
[   27.420608]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.443423]   nfs4_xdr_dec_read_plus(hdr=c32c2800, scratch=c3f6a040)
[   27.448335]     decode_read_plus_segment(n=0)
[   27.452708]       decode_read_plus_segment(n=0): DATA {offset=933888, length=131072}
[   27.460417]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.460847]   nfs_readhdr_free(hdr=c32c2800, scratch=c3f6a040)
[   27.465185] 
[   27.473646] nfs_read_alloc_scratch(hdr=c274a080, size=16) = c4080e00
[   27.480054]   nfs4_xdr_enc_read_plus(hdr=c274a080, decode_read_plus_maxsz=9)
[   27.480054]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.500967]   nfs4_xdr_dec_read_plus(hdr=c274a080, scratch=c4080e00)
[   27.505944]     decode_read_plus_segment(n=0)
[   27.510201]       decode_read_plus_segment(n=0): DATA {offset=450560, length=131072}
[   27.518014]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.518400]   nfs_readhdr_free(hdr=c274a080, scratch=c4080e00)
[   27.523317] 
[   27.531188] nfs_read_alloc_scratch(hdr=c32c2800, size=16) = c3f6a5c0
[   27.537611]   nfs4_xdr_enc_read_plus(hdr=c32c2800, decode_read_plus_maxsz=9)
[   27.537611]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.557871]   nfs4_xdr_dec_read_plus(hdr=c32c2800, scratch=c3f6a5c0)
[   27.562835]     decode_read_plus_segment(n=0)
[   27.567103]       decode_read_plus_segment(n=0): DATA {offset=581632, length=86016}
[   27.574807]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.575110]   nfs_readhdr_free(hdr=c32c2800, scratch=c3f6a5c0)
[   27.582507] 
[   27.588001] nfs_read_alloc_scratch(hdr=c32c2a80, size=16) = c3f6aa00
[   27.594544]   nfs4_xdr_enc_read_plus(hdr=c32c2a80, decode_read_plus_maxsz=9)
[   27.594544]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.613757]   nfs4_xdr_dec_read_plus(hdr=c32c2a80, scratch=c3f6aa00)
[   27.618660]     decode_read_plus_segment(n=0)
[   27.623052]       decode_read_plus_segment(n=0): DATA {offset=708608, length=131072}
[   27.630761]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.631156]   nfs_readhdr_free(hdr=c32c2a80, scratch=c3f6aa00)
[   27.633745] 
[   27.643977] nfs_read_alloc_scratch(hdr=c32c2d00, size=16) = c3f6a0c0
[   27.650358]   nfs4_xdr_enc_read_plus(hdr=c32c2d00, decode_read_plus_maxsz=9)
[   27.650358]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.668977]   nfs4_xdr_dec_read_plus(hdr=c32c2d00, scratch=c3f6a0c0)
[   27.674029]     decode_read_plus_segment(n=0)
[   27.678289]       decode_read_plus_segment(n=0): DATA {offset=1433600, length=53248}
[   27.686067]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.686363]   nfs_readhdr_free(hdr=c32c2d00, scratch=c3f6a0c0)
[   27.688695] 
[   27.699200] nfs_read_alloc_scratch(hdr=c32c2f80, size=16) = c3f6aec0
[   27.705621]   nfs4_xdr_enc_read_plus(hdr=c32c2f80, decode_read_plus_maxsz=9)
[   27.705621]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.726739]   nfs4_xdr_dec_read_plus(hdr=c32c2f80, scratch=c3f6aec0)
[   27.731712]     decode_read_plus_segment(n=0)
[   27.735967]       decode_read_plus_segment(n=0): DATA {offset=2768896, length=131072}
[   27.743967]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.744367]   nfs_readhdr_free(hdr=c32c2f80, scratch=c3f6aec0)
[   27.746776] 
[   27.757038] nfs_read_alloc_scratch(hdr=c32c3200, size=16) = c3f6a8c0
[   27.763440]   nfs4_xdr_enc_read_plus(hdr=c32c3200, decode_read_plus_maxsz=9)
[   27.763440]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.785476]   nfs4_xdr_dec_read_plus(hdr=c32c3200, scratch=c3f6a8c0)
[   27.790386]     decode_read_plus_segment(n=0)
[   27.794769]       decode_read_plus_segment(n=0): DATA {offset=1830912, length=131072}
[   27.802662]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.803033]   nfs_readhdr_free(hdr=c32c3200, scratch=c3f6a8c0)
[   27.804768] 
[   27.815784] nfs_read_alloc_scratch(hdr=c274a080, size=16) = c4080380
[   27.822227]   nfs4_xdr_enc_read_plus(hdr=c274a080, decode_read_plus_maxsz=9)
[   27.822227]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.843392]   nfs4_xdr_dec_read_plus(hdr=c274a080, scratch=c4080380)
[   27.848293]     decode_read_plus_segment(n=0)
[   27.852674]       decode_read_plus_segment(n=0): DATA {offset=2613248, length=131072}
[   27.860472]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.860916]   nfs_readhdr_free(hdr=c274a080, scratch=c4080380)
[   27.863360] 
[   27.873686] nfs_read_alloc_scratch(hdr=c2749e00, size=16) = c4080cc0
[   27.880051]   nfs4_xdr_enc_read_plus(hdr=c2749e00, decode_read_plus_maxsz=9)
[   27.880051]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.898607]   nfs4_xdr_dec_read_plus(hdr=c2749e00, scratch=c4080cc0)
[   27.903633]     decode_read_plus_segment(n=0)
[   27.907834]       decode_read_plus_segment(n=0): DATA {offset=1961984, length=77824}
[   27.915621]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.915902]   nfs_readhdr_free(hdr=c2749e00, scratch=c4080cc0)
[   27.917214] 
[   27.928816] nfs_read_alloc_scratch(hdr=c274a300, size=16) = c40d9c80
[   27.935331]   nfs4_xdr_enc_read_plus(hdr=c274a300, decode_read_plus_maxsz=9)
[   27.935331]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   27.956423]   nfs4_xdr_dec_read_plus(hdr=c274a300, scratch=c40d9c80)
[   27.961387]     decode_read_plus_segment(n=0)
[   27.965652]       decode_read_plus_segment(n=0): DATA {offset=1654784, length=131072}
[   27.973662]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   27.974090]   nfs_readhdr_free(hdr=c274a300, scratch=c40d9c80)
[   27.975339] 
[   27.986727] nfs_read_alloc_scratch(hdr=c274a580, size=16) = c40d9740
[   27.993158]   nfs4_xdr_enc_read_plus(hdr=c274a580, decode_read_plus_maxsz=9)
[   27.993158]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.011519]   nfs4_xdr_dec_read_plus(hdr=c274a580, scratch=c40d9740)
[   28.016420]     decode_read_plus_segment(n=0)
[   28.020746]       decode_read_plus_segment(n=0): DATA {offset=1785856, length=45056}
[   28.028625]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.028817]   nfs_readhdr_free(hdr=c274a580, scratch=c40d9740)
[   28.030107] 
[   28.041792] nfs_read_alloc_scratch(hdr=c1ff4c80, size=16) = c40b0bc0
[   28.048114]   nfs4_xdr_enc_read_plus(hdr=c1ff4c80, decode_read_plus_maxsz=9)
[   28.048114]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.068746]   nfs4_xdr_dec_read_plus(hdr=c1ff4c80, scratch=c40b0bc0)
[   28.073750]     decode_read_plus_segment(n=0)
[   28.077971]       decode_read_plus_segment(n=0): DATA {offset=2912256, length=131072}
[   28.085867]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.086228]   nfs_readhdr_free(hdr=c1ff4c80, scratch=c40b0bc0)
[   28.087529] 
[   28.099039] nfs_read_alloc_scratch(hdr=c1ff5180, size=16) = c40b0580
[   28.105505]   nfs4_xdr_enc_read_plus(hdr=c1ff5180, decode_read_plus_maxsz=9)
[   28.105505]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.125049]   nfs4_xdr_dec_read_plus(hdr=c1ff5180, scratch=c40b0580)
[   28.129957]     decode_read_plus_segment(n=0)
[   28.134336]       decode_read_plus_segment(n=0): DATA {offset=1560576, length=94208}
[   28.142183]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.142488]   nfs_readhdr_free(hdr=c1ff5180, scratch=c40b0580)
[   28.144312] 
[   28.155264] nfs_read_alloc_scratch(hdr=c1ff5400, size=16) = c40b0a80
[   28.161726]   nfs4_xdr_enc_read_plus(hdr=c1ff5400, decode_read_plus_maxsz=9)
[   28.161726]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.181585]   nfs4_xdr_dec_read_plus(hdr=c1ff5400, scratch=c40b0a80)
[   28.186490]     decode_read_plus_segment(n=0)
[   28.190814]       decode_read_plus_segment(n=0): DATA {offset=360448, length=90112}
[   28.198578]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.198823]   nfs_readhdr_free(hdr=c1ff5400, scratch=c40b0a80)
[   28.201164] 
[   28.211749] nfs_read_alloc_scratch(hdr=c274a580, size=16) = c24cebc0
[   28.218078]   nfs4_xdr_enc_read_plus(hdr=c274a580, decode_read_plus_maxsz=9)
[   28.218078]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.239073]   nfs4_xdr_dec_read_plus(hdr=c274a580, scratch=c24cebc0)
[   28.244032]     decode_read_plus_segment(n=0)
[   28.248304]       decode_read_plus_segment(n=0): DATA {offset=1445888, length=114688}
[   28.256332]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.256714]   nfs_readhdr_free(hdr=c274a580, scratch=c24cebc0)
[   28.258228] 
[   28.269376] nfs_read_alloc_scratch(hdr=c274a800, size=16) = c24cee00
[   28.275826]   nfs4_xdr_enc_read_plus(hdr=c274a800, decode_read_plus_maxsz=9)
[   28.275826]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.294545]   nfs4_xdr_dec_read_plus(hdr=c274a800, scratch=c24cee00)
[   28.299449]     decode_read_plus_segment(n=0)
[   28.304060]       decode_read_plus_segment(n=0): DATA {offset=278528, length=81920}
[   28.311542]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.311912]   nfs_readhdr_free(hdr=c274a800, scratch=c24cee00)
[   28.315946] 
[   28.324681] nfs_read_alloc_scratch(hdr=c274aa80, size=16) = c323dcc0
[   28.331171]   nfs4_xdr_enc_read_plus(hdr=c274aa80, decode_read_plus_maxsz=9)
[   28.331171]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.353295]   nfs4_xdr_dec_read_plus(hdr=c274aa80, scratch=c323dcc0)
[   28.358211]     decode_read_plus_segment(n=0)
[   28.362585]       decode_read_plus_segment(n=0): DATA {offset=2744320, length=131072}
[   28.370374]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.370731]   nfs_readhdr_free(hdr=c274aa80, scratch=c323dcc0)
[   28.384138] 
[   28.384211] nfs_read_alloc_scratch(hdr=c40e8500, size=16) = c24afcc0
[   28.390544]   nfs4_xdr_enc_read_plus(hdr=c40e8500, decode_read_plus_maxsz=9)
[   28.390544]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.412382]   nfs4_xdr_dec_read_plus(hdr=c40e8500, scratch=c24afcc0)
[   28.417285]     decode_read_plus_segment(n=0)
[   28.421671]       decode_read_plus_segment(n=0): DATA {offset=1048576, length=131072}
[   28.429459]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.429861]   nfs_readhdr_free(hdr=c40e8500, scratch=c24afcc0)
[   28.435018] 
[   28.442715] nfs_read_alloc_scratch(hdr=c40e8780, size=16) = c24afc40
[   28.449046]   nfs4_xdr_enc_read_plus(hdr=c40e8780, decode_read_plus_maxsz=9)
[   28.449046]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.469338]   nfs4_xdr_dec_read_plus(hdr=c40e8780, scratch=c24afc40)
[   28.474359]     decode_read_plus_segment(n=0)
[   28.478565]       decode_read_plus_segment(n=0): DATA {offset=1064960, length=106496}
[   28.486435]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.486694]   nfs_readhdr_free(hdr=c40e8780, scratch=c24afc40)
[   28.489834] 
[   28.499872] nfs_read_alloc_scratch(hdr=c40e8a00, size=16) = c2c42200
[   28.506128]   nfs4_xdr_enc_read_plus(hdr=c40e8a00, decode_read_plus_maxsz=9)
[   28.506128]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.526456]   nfs4_xdr_dec_read_plus(hdr=c40e8a00, scratch=c2c42200)
[   28.531403]     decode_read_plus_segment(n=0)
[   28.535679]       decode_read_plus_segment(n=0): DATA {offset=1183744, length=131072}
[   28.543566]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.543895]   nfs_readhdr_free(hdr=c40e8a00, scratch=c2c42200)
[   28.546284] 
[   28.556795] nfs_read_alloc_scratch(hdr=c40e8c80, size=16) = c24cc680
[   28.563188]   nfs4_xdr_enc_read_plus(hdr=c40e8c80, decode_read_plus_maxsz=9)
[   28.563188]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.583517]   nfs4_xdr_dec_read_plus(hdr=c40e8c80, scratch=c24cc680)
[   28.588414]     decode_read_plus_segment(n=0)
[   28.592793]       decode_read_plus_segment(n=0): DATA {offset=782336, length=131072}
[   28.600502]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.600844]   nfs_readhdr_free(hdr=c40e8c80, scratch=c24cc680)
[   28.602458] 
[   28.613942] nfs_read_alloc_scratch(hdr=c40e8f00, size=16) = c24cc0c0
[   28.620133]   nfs4_xdr_enc_read_plus(hdr=c40e8f00, decode_read_plus_maxsz=9)
[   28.620133]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.640446]   nfs4_xdr_dec_read_plus(hdr=c40e8f00, scratch=c24cc0c0)
[   28.645418]     decode_read_plus_segment(n=0)
[   28.649674]       decode_read_plus_segment(n=0): DATA {offset=1388544, length=122880}
[   28.657675]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.658053]   nfs_readhdr_free(hdr=c40e8f00, scratch=c24cc0c0)
[   28.660601] 
[   28.670743] nfs_read_alloc_scratch(hdr=c40e9180, size=16) = c24cc700
[   28.677141]   nfs4_xdr_enc_read_plus(hdr=c40e9180, decode_read_plus_maxsz=9)
[   28.677141]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.697033]   nfs4_xdr_dec_read_plus(hdr=c40e9180, scratch=c24cc700)
[   28.701992]     decode_read_plus_segment(n=0)
[   28.706268]       decode_read_plus_segment(n=0): DATA {offset=1318912, length=69632}
[   28.714109]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.714309]   nfs_readhdr_free(hdr=c40e9180, scratch=c24cc700)
[   28.715686] 
[   28.727251] nfs_read_alloc_scratch(hdr=c274ad00, size=16) = c4080fc0
[   28.733654]   nfs4_xdr_enc_read_plus(hdr=c274ad00, decode_read_plus_maxsz=9)
[   28.733654]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.753976]   nfs4_xdr_dec_read_plus(hdr=c274ad00, scratch=c4080fc0)
[   28.758883]     decode_read_plus_segment(n=0)
[   28.763375]       decode_read_plus_segment(n=0): DATA {offset=913408, length=110592}
[   28.771007]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.771486]   nfs_readhdr_free(hdr=c274ad00, scratch=c4080fc0)
[   28.772890] 
[   28.784191] nfs_read_alloc_scratch(hdr=c274af80, size=16) = c4080e00
[   28.790711]   nfs4_xdr_enc_read_plus(hdr=c274af80, decode_read_plus_maxsz=9)
[   28.790711]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.809141]   nfs4_xdr_dec_read_plus(hdr=c274af80, scratch=c4080e00)
[   28.814120]     decode_read_plus_segment(n=0)
[   28.818372]       decode_read_plus_segment(n=0): DATA {offset=1024000, length=77824}
[   28.826157]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.826428]   nfs_readhdr_free(hdr=c274af80, scratch=c4080e00)
[   28.827411] 
[   28.839354] nfs_read_alloc_scratch(hdr=c40e9180, size=16) = c24ccf00
[   28.845810]   nfs4_xdr_enc_read_plus(hdr=c40e9180, decode_read_plus_maxsz=9)
[   28.845810]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.867507]   nfs4_xdr_dec_read_plus(hdr=c40e9180, scratch=c24ccf00)
[   28.872481]     decode_read_plus_segment(n=0)
[   28.876746]       decode_read_plus_segment(n=0): DATA {offset=581632, length=131072}
[   28.884562]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.884948]   nfs_readhdr_free(hdr=c40e9180, scratch=c24ccf00)
[   28.889822] 
[   28.897789] nfs_read_alloc_scratch(hdr=c40e9400, size=16) = c24cc680
[   28.904263]   nfs4_xdr_enc_read_plus(hdr=c40e9400, decode_read_plus_maxsz=9)
[   28.904263]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.922622]   nfs4_xdr_dec_read_plus(hdr=c40e9400, scratch=c24cc680)
[   28.927530]     decode_read_plus_segment(n=0)
[   28.931916]       decode_read_plus_segment(n=0): DATA {offset=540672, length=65536}
[   28.939518]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.939785]   nfs_readhdr_free(hdr=c40e9400, scratch=c24cc680)
[   28.944405] 
[   28.952856] nfs_read_alloc_scratch(hdr=c40e9680, size=16) = c24cc3c0
[   28.959176]   nfs4_xdr_enc_read_plus(hdr=c40e9680, decode_read_plus_maxsz=9)
[   28.959176]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   28.974518]   nfs4_xdr_dec_read_plus(hdr=c40e9680, scratch=c24cc3c0)
[   28.979415]     decode_read_plus_segment(n=0)
[   28.983813]       decode_read_plus_segment(n=0): DATA {offset=0, length=44}
[   28.990681]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   28.990775]   nfs_readhdr_free(hdr=c40e9680, scratch=c24cc3c0)
[   28.994980] 
[   29.003941] nfs_read_alloc_scratch(hdr=c40e9900, size=16) = c24cccc0
[   29.010302]   nfs4_xdr_enc_read_plus(hdr=c40e9900, decode_read_plus_maxsz=9)
[   29.010302]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.027970]   nfs4_xdr_dec_read_plus(hdr=c40e9900, scratch=c24cccc0)
[   29.032962]     decode_read_plus_segment(n=0)
[   29.037203]       decode_read_plus_segment(n=0): DATA {offset=667648, length=40960}
[   29.044919]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.045094]   nfs_readhdr_free(hdr=c40e9900, scratch=c24cccc0)
[   29.049508] systemd[1]: System time before build time, advancing clock.
[   29.063558] 
[   29.064664] nfs_read_alloc_scratch(hdr=c40e9b80, size=16) = c24cc100
[   29.071044]   nfs4_xdr_enc_read_plus(hdr=c40e9b80, decode_read_plus_maxsz=9)
[   29.071044]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.090969]   nfs4_xdr_dec_read_plus(hdr=c40e9b80, scratch=c24cc100)
[   29.096015]     decode_read_plus_segment(n=0)
[   29.100210]       decode_read_plus_segment(n=0): DATA {offset=606208, length=73728}
[   29.108045]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.108395]   nfs_readhdr_free(hdr=c40e9b80, scratch=c24cc100)
[   29.120045] 
[   29.121093] nfs_read_alloc_scratch(hdr=c40e9e00, size=16) = c24cc280
[   29.127532]   nfs4_xdr_enc_read_plus(hdr=c40e9e00, decode_read_plus_maxsz=9)
[   29.127532]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.147771]   nfs4_xdr_dec_read_plus(hdr=c40e9e00, scratch=c24cc280)
[   29.152737]     decode_read_plus_segment(n=0)
[   29.156998]       decode_read_plus_segment(n=0): DATA {offset=3043328, length=106496}
[   29.164932]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.165245]   nfs_readhdr_free(hdr=c40e9e00, scratch=c24cc280)
[   29.192075] 
[   29.192166] nfs_read_alloc_scratch(hdr=c1ff5400, size=16) = c40b0cc0
[   29.198502]   nfs4_xdr_enc_read_plus(hdr=c1ff5400, decode_read_plus_maxsz=9)
[   29.198502]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.218315]   nfs4_xdr_dec_read_plus(hdr=c1ff5400, scratch=c40b0cc0)
[   29.223308]     decode_read_plus_segment(n=0)
[   29.227555]       decode_read_plus_segment(n=0): DATA {offset=712704, length=69632}
[   29.235247]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.235446]   nfs_readhdr_free(hdr=c1ff5400, scratch=c40b0cc0)
[   29.247730] 
[   29.248433] nfs_read_alloc_scratch(hdr=c1ff5680, size=16) = c40b0200
[   29.254963]   nfs4_xdr_enc_read_plus(hdr=c1ff5680, decode_read_plus_maxsz=9)
[   29.254963]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.270143]   nfs4_xdr_dec_read_plus(hdr=c1ff5680, scratch=c40b0200)
[   29.275204]     decode_read_plus_segment(n=0)
[   29.279413]       decode_read_plus_segment(n=0): DATA {offset=0, length=97}
[   29.286408]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.286496]   nfs_readhdr_free(hdr=c1ff5680, scratch=c40b0200)
[   29.292009] 
[   29.299585] nfs_read_alloc_scratch(hdr=c1ff5900, size=16) = c40b0d00
[   29.306010]   nfs4_xdr_enc_read_plus(hdr=c1ff5900, decode_read_plus_maxsz=9)
[   29.306010]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.321467]   nfs4_xdr_dec_read_plus(hdr=c1ff5900, scratch=c40b0d00)
[   29.326372]     decode_read_plus_segment(n=0)
[   29.330702]       decode_read_plus_segment(n=0): DATA {offset=0, length=773}
[   29.337765]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.337853]   nfs_readhdr_free(hdr=c1ff5900, scratch=c40b0d00)
[   29.344326] 
[   29.350988] nfs_read_alloc_scratch(hdr=c1ff5b80, size=16) = c40b0840
[   29.357416]   nfs4_xdr_enc_read_plus(hdr=c1ff5b80, decode_read_plus_maxsz=9)
[   29.357416]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.374165]   nfs4_xdr_dec_read_plus(hdr=c1ff5b80, scratch=c40b0840)
[   29.379108]     decode_read_plus_segment(n=0)
[   29.383496]       decode_read_plus_segment(n=0): DATA {offset=0, length=16667}
[   29.390640]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.390781]   nfs_readhdr_free(hdr=c1ff5b80, scratch=c40b0840)
[   29.394281] 
[   29.403860] nfs_read_alloc_scratch(hdr=c1ff5e00, size=16) = c40b0bc0
[   29.410271]   nfs4_xdr_enc_read_plus(hdr=c1ff5e00, decode_read_plus_maxsz=9)
[   29.410271]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.427279]   nfs4_xdr_dec_read_plus(hdr=c1ff5e00, scratch=c40b0bc0)
[   29.432247]     decode_read_plus_segment(n=0)
[   29.436513]       decode_read_plus_segment(n=0): DATA {offset=0, length=37338}
[   29.443764]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.443942]   nfs_readhdr_free(hdr=c1ff5e00, scratch=c40b0bc0)
[   29.447498] 
[   29.456969] nfs_read_alloc_scratch(hdr=c1ff6080, size=16) = c40b02c0
[   29.463393]   nfs4_xdr_enc_read_plus(hdr=c1ff6080, decode_read_plus_maxsz=9)
[   29.463393]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.481197]   nfs4_xdr_dec_read_plus(hdr=c1ff6080, scratch=c40b02c0)
[   29.486166]     decode_read_plus_segment(n=0)
[   29.490431]       decode_read_plus_segment(n=0): DATA {offset=0, length=55526}
[   29.497846]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.498130]   nfs_readhdr_free(hdr=c1ff6080, scratch=c40b02c0)
[   29.501673] 
[   29.510898] nfs_read_alloc_scratch(hdr=c1ff6300, size=16) = c40b0c80
[   29.517297]   nfs4_xdr_enc_read_plus(hdr=c1ff6300, decode_read_plus_maxsz=9)
[   29.517297]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.534230]   nfs4_xdr_dec_read_plus(hdr=c1ff6300, scratch=c40b0c80)
[   29.539135]     decode_read_plus_segment(n=0)
[   29.543516]       decode_read_plus_segment(n=0): DATA {offset=0, length=30967}
[   29.550670]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.550883]   nfs_readhdr_free(hdr=c1ff6300, scratch=c40b0c80)
[   29.554521] 
[   29.563923] nfs_read_alloc_scratch(hdr=c40ea080, size=16) = c24ccb00
[   29.570294]   nfs4_xdr_enc_read_plus(hdr=c40ea080, decode_read_plus_maxsz=9)
[   29.570294]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.586783]   nfs4_xdr_dec_read_plus(hdr=c40ea080, scratch=c24ccb00)
[   29.591748]     decode_read_plus_segment(n=0)
[   29.596007]       decode_read_plus_segment(n=0): DATA {offset=0, length=17240}
[   29.603276]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.603412]   nfs_readhdr_free(hdr=c40ea080, scratch=c24ccb00)
[   29.626802] 
[   29.626927] nfs_read_alloc_scratch(hdr=c1ff6580, size=16) = c40b21c0
[   29.633321]   nfs4_xdr_enc_read_plus(hdr=c1ff6580, decode_read_plus_maxsz=9)
[   29.633321]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.651893]   nfs4_xdr_dec_read_plus(hdr=c1ff6580, scratch=c40b21c0)
[   29.656810]     decode_read_plus_segment(n=0)
[   29.661130]       decode_read_plus_segment(n=0): DATA {offset=942080, length=106496}
[   29.669100]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.669472]   nfs_readhdr_free(hdr=c1ff6580, scratch=c40b21c0)
[   29.674252] 
[   29.682339] nfs_read_alloc_scratch(hdr=c1ff6800, size=16) = c40b2880
[   29.688478]   nfs4_xdr_enc_read_plus(hdr=c1ff6800, decode_read_plus_maxsz=9)
[   29.688478]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.708244]   nfs4_xdr_dec_read_plus(hdr=c1ff6800, scratch=c40b2880)
[   29.713201]     decode_read_plus_segment(n=0)
[   29.717469]       decode_read_plus_segment(n=0): DATA {offset=479232, length=102400}
[   29.725275]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.725507]   nfs_readhdr_free(hdr=c1ff6800, scratch=c40b2880)
[   29.728674] 
[   29.738488] nfs_read_alloc_scratch(hdr=c1ff6a80, size=16) = c40b2740
[   29.744846]   nfs4_xdr_enc_read_plus(hdr=c1ff6a80, decode_read_plus_maxsz=9)
[   29.744846]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.765169]   nfs4_xdr_dec_read_plus(hdr=c1ff6a80, scratch=c40b2740)
[   29.770078]     decode_read_plus_segment(n=0)
[   29.774450]       decode_read_plus_segment(n=0): DATA {offset=679936, length=131072}
[   29.782216]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.782518]   nfs_readhdr_free(hdr=c1ff6a80, scratch=c40b2740)
[   29.792547] 
[   29.795379] nfs_read_alloc_scratch(hdr=c1ff6d00, size=16) = c40b2380
[   29.801799]   nfs4_xdr_enc_read_plus(hdr=c1ff6d00, decode_read_plus_maxsz=9)
[   29.801799]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.817235]   nfs4_xdr_dec_read_plus(hdr=c1ff6d00, scratch=c40b2380)
[   29.822214]     decode_read_plus_segment(n=0)
[   29.826453]       decode_read_plus_segment(n=0): DATA {offset=0, length=2127}
[   29.833643]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.833730]   nfs_readhdr_free(hdr=c1ff6d00, scratch=c40b2380)
[   29.839616] 
[   29.847024] nfs_read_alloc_scratch(hdr=c1ff6f80, size=16) = c40b2e80
[   29.853291]   nfs4_xdr_enc_read_plus(hdr=c1ff6f80, decode_read_plus_maxsz=9)
[   29.853291]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.874230]   nfs4_xdr_dec_read_plus(hdr=c1ff6f80, scratch=c40b2e80)
[   29.879144]     decode_read_plus_segment(n=0)
[   29.883508]       decode_read_plus_segment(n=0): DATA {offset=839680, length=94208}
[   29.891129]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.891519]   nfs_readhdr_free(hdr=c1ff6f80, scratch=c40b2e80)
[   29.905321] systemd[1]: systemd 253.4-1-arch running in system mode (+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified)
[   29.936351] systemd[1]: Detected architecture arm.
[   29.943658] 
[   29.943752] nfs_read_alloc_scratch(hdr=c40ea300, size=16) = c24cc200
[   29.950256]   nfs4_xdr_enc_read_plus(hdr=c40ea300, decode_read_plus_maxsz=9)
[   29.950256]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   29.966232]   nfs4_xdr_dec_read_plus(hdr=c40ea300, scratch=c24cc200)
[   29.971184]     decode_read_plus_segment(n=0)
[   29.975596]       decode_read_plus_segment(n=0): DATA {offset=0, length=33}
[   29.982518]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   29.982749]   nfs_readhdr_free(hdr=c40ea300, scratch=c24cc200)
[   29.991742] 
[   29.995708] nfs_read_alloc_scratch(hdr=c40ea580, size=16) = c24cccc0
[   30.002456]   nfs4_xdr_enc_read_plus(hdr=c40ea580, decode_read_plus_maxsz=9)
[   30.002456]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.017693]   nfs4_xdr_dec_read_plus(hdr=c40ea580, scratch=c24cccc0)
[   30.022807]     decode_read_plus_segment(n=0)
[   30.026999]       decode_read_plus_segment(n=0): DATA {offset=0, length=339}
[   30.034134]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.034327]   nfs_readhdr_free(hdr=c40ea580, scratch=c24cccc0)

Welcome to Arch Linux ARM!

[   30.082009] 
[   30.082116] nfs_read_alloc_scratch(hdr=c274af80, size=16) = c2637080
[   30.088620]   nfs4_xdr_enc_read_plus(hdr=c274af80, decode_read_plus_maxsz=9)
[   30.088620]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.110543]   nfs4_xdr_dec_read_plus(hdr=c274af80, scratch=c2637080)
[   30.115874]     decode_read_plus_segment(n=0)
[   30.119869]       decode_read_plus_segment(n=0): DATA {offset=811008, length=126976}
[   30.127696]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.128140]   nfs_readhdr_free(hdr=c274af80, scratch=c2637080)
[   30.131147] 
[   30.140858] nfs_read_alloc_scratch(hdr=c274b200, size=16) = c40b0800
[   30.147502]   nfs4_xdr_enc_read_plus(hdr=c274b200, decode_read_plus_maxsz=9)
[   30.147502]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.163306]   nfs4_xdr_dec_read_plus(hdr=c274b200, scratch=c40b0800)
[   30.168303]     decode_read_plus_segment(n=0)
[   30.172682]       decode_read_plus_segment(n=0): DATA {offset=0, length=10}
[   30.179547]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.179768]   nfs_readhdr_free(hdr=c274b200, scratch=c40b0800)
[   30.182414] systemd[1]: Hostname set to <odroidhc1>.
[   30.200796] 
[   30.200886] nfs_read_alloc_scratch(hdr=c1ff6f80, size=16) = c40b2b40
[   30.207442]   nfs4_xdr_enc_read_plus(hdr=c1ff6f80, decode_read_plus_maxsz=9)
[   30.207442]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.226834]   nfs4_xdr_dec_read_plus(hdr=c1ff6f80, scratch=c40b2b40)
[   30.231953]     decode_read_plus_segment(n=0)
[   30.236129]       decode_read_plus_segment(n=0): DATA {offset=2510848, length=102400}
[   30.244046]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.244439]   nfs_readhdr_free(hdr=c1ff6f80, scratch=c40b2b40)
[   30.247288] 
[   30.257201] nfs_read_alloc_scratch(hdr=c1ff7200, size=16) = c40b2100
[   30.264135]   nfs4_xdr_enc_read_plus(hdr=c1ff7200, decode_read_plus_maxsz=9)
[   30.264135]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.284068]   nfs4_xdr_dec_read_plus(hdr=c1ff7200, scratch=c40b2100)
[   30.289081]     decode_read_plus_segment(n=0)
[   30.293459]       decode_read_plus_segment(n=0): DATA {offset=2310144, length=131072}
[   30.301291]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.301758]   nfs_readhdr_free(hdr=c1ff7200, scratch=c40b2100)
[   30.316097] 
[   30.316203] nfs_read_alloc_scratch(hdr=c1ff7480, size=16) = c40b24c0
[   30.322644]   nfs4_xdr_enc_read_plus(hdr=c1ff7480, decode_read_plus_maxsz=9)
[   30.322644]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.337964]   nfs4_xdr_dec_read_plus(hdr=c1ff7480, scratch=c40b24c0)
[   30.342996]     decode_read_plus_segment(n=0)
[   30.347218]       decode_read_plus_segment(n=0): DATA {offset=0, length=17}
[   30.354240]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.354425]   nfs_readhdr_free(hdr=c1ff7480, scratch=c40b24c0)
[   30.426052] 
[   30.426157] nfs_read_alloc_scratch(hdr=c1ff7700, size=16) = c40b3140
[   30.432928]   nfs4_xdr_enc_read_plus(hdr=c1ff7700, decode_read_plus_maxsz=9)
[   30.432928]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.456115]   nfs4_xdr_dec_read_plus(hdr=c1ff7700, scratch=c40b3140)
[   30.461097]     decode_read_plus_segment(n=0)
[   30.465448]       decode_read_plus_segment(n=0): DATA {offset=1191936, length=126976}
[   30.473463]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.473952]   nfs_readhdr_free(hdr=c1ff7700, scratch=c40b3140)
[   30.487594] 
[   30.487693] nfs_read_alloc_scratch(hdr=c1ff7980, size=16) = c40b34c0
[   30.494388]   nfs4_xdr_enc_read_plus(hdr=c1ff7980, decode_read_plus_maxsz=9)
[   30.494388]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.510083]   nfs4_xdr_dec_read_plus(hdr=c1ff7980, scratch=c40b34c0)
[   30.515220]     decode_read_plus_segment(n=0)
[   30.519349]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   30.526636]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.526829]   nfs_readhdr_free(hdr=c1ff7980, scratch=c40b34c0)
[   30.529103] 
[   30.539812] nfs_read_alloc_scratch(hdr=c274b480, size=16) = c4261fc0
[   30.546298]   nfs4_xdr_enc_read_plus(hdr=c274b480, decode_read_plus_maxsz=9)
[   30.546298]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.563487]   nfs4_xdr_dec_read_plus(hdr=c274b480, scratch=c4261fc0)
[   30.568464]     decode_read_plus_segment(n=0)
[   30.572850]       decode_read_plus_segment(n=0): DATA {offset=262144, length=67316}
[   30.580427]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.580740]   nfs_readhdr_free(hdr=c274b480, scratch=c4261fc0)
[   30.581541] 
[   30.593659] nfs_read_alloc_scratch(hdr=c1ff7c00, size=16) = c40b3b80
[   30.600067]   nfs4_xdr_enc_read_plus(hdr=c1ff7c00, decode_read_plus_maxsz=9)
[   30.600067]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.619509]   nfs4_xdr_dec_read_plus(hdr=c1ff7c00, scratch=c40b3b80)
[   30.624680]     decode_read_plus_segment(n=0)
[   30.628816]       decode_read_plus_segment(n=0): DATA {offset=16384, length=114688}
[   30.636600]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.637111]   nfs_readhdr_free(hdr=c1ff7c00, scratch=c40b3b80)
[   30.645912] 
[   30.649713] nfs_read_alloc_scratch(hdr=c1ff7980, size=16) = c40b3e80
[   30.656659]   nfs4_xdr_enc_read_plus(hdr=c1ff7980, decode_read_plus_maxsz=9)
[   30.656659]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.672688]   nfs4_xdr_dec_read_plus(hdr=c1ff7980, scratch=c40b3e80)
[   30.677663]     decode_read_plus_segment(n=0)
[   30.682041]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   30.689171]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.689379]   nfs_readhdr_free(hdr=c1ff7980, scratch=c40b3e80)
[   30.691682] 
[   30.702477] nfs_read_alloc_scratch(hdr=c1ff7700, size=16) = c40b39c0
[   30.708845]   nfs4_xdr_enc_read_plus(hdr=c1ff7700, decode_read_plus_maxsz=9)
[   30.708845]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.726456]   nfs4_xdr_dec_read_plus(hdr=c1ff7700, scratch=c40b39c0)
[   30.731549]     decode_read_plus_segment(n=0)
[   30.735727]       decode_read_plus_segment(n=0): DATA {offset=24576, length=67084}
[   30.743561]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.743917]   nfs_readhdr_free(hdr=c1ff7700, scratch=c40b39c0)
[   30.744419] 
[   30.756588] nfs_read_alloc_scratch(hdr=c1ff7480, size=16) = c40b3940
[   30.763009]   nfs4_xdr_enc_read_plus(hdr=c1ff7480, decode_read_plus_maxsz=9)
[   30.763009]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.764041] 
[   30.778097] nfs_read_alloc_scratch(hdr=c1ff7200, size=16) = c40b3040
[   30.778564]   nfs4_xdr_dec_read_plus(hdr=c1ff7480, scratch=c40b3940)
[   30.784465]   nfs4_xdr_enc_read_plus(hdr=c1ff7200, decode_read_plus_maxsz=9)
[   30.784465]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.790554]     decode_read_plus_segment(n=0)
[   30.790571]       decode_read_plus_segment(n=0): DATA {offset=16384, length=8192}
[   30.790594]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.810968]   nfs4_xdr_dec_read_plus(hdr=c1ff7200, scratch=c40b3040)
[   30.816167]   nfs_readhdr_free(hdr=c1ff7480, scratch=c40b3940)
[   30.821987]     decode_read_plus_segment(n=0)
[   30.838645]       decode_read_plus_segment(n=0): DATA {offset=131072, length=131072}
[   30.846324]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.846880]   nfs_readhdr_free(hdr=c1ff7200, scratch=c40b3040)
[   30.855857] systemd[1]: bpf-lsm: BPF LSM hook not enabled in the kernel, BPF LSM not supported
[   30.875790] memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL, pid=1 'systemd'
[   30.921601] 
[   30.921803] nfs_read_alloc_scratch(hdr=c2733200, size=16) = c41c7540
[   30.928145]   nfs4_xdr_enc_read_plus(hdr=c2733200, decode_read_plus_maxsz=9)
[   30.928145]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   30.943426]   nfs4_xdr_dec_read_plus(hdr=c2733200, scratch=c41c7540)
[   30.948351]     decode_read_plus_segment(n=0)
[   30.952738]       decode_read_plus_segment(n=0): DATA {offset=0, length=63}
[   30.959622]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   30.959778]   nfs_readhdr_free(hdr=c2733200, scratch=c41c7540)
[   30.977895] 
[   30.978020] nfs_read_alloc_scratch(hdr=c2732f80, size=16) = c41c7700
[   30.984453]   nfs4_xdr_enc_read_plus(hdr=c2732f80, decode_read_plus_maxsz=9)
[   30.984453]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.000030]   nfs4_xdr_dec_read_plus(hdr=c2732f80, scratch=c41c7700)
[   31.005001]     decode_read_plus_segment(n=0)
[   31.009293]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   31.016516]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   31.016596]   nfs_readhdr_free(hdr=c2732f80, scratch=c41c7700)
[   31.020617] 
[   31.029854] nfs_read_alloc_scratch(hdr=c2732d00, size=16) = c41c7e00
[   31.036392]   nfs4_xdr_enc_read_plus(hdr=c2732d00, decode_read_plus_maxsz=9)
[   31.036392]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.053169]   nfs4_xdr_dec_read_plus(hdr=c2732d00, scratch=c41c7e00)
[   31.058066]     decode_read_plus_segment(n=0)
[   31.062435]       decode_read_plus_segment(n=0): DATA {offset=811008, length=69728}
[   31.070047]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   31.070193]   nfs_readhdr_free(hdr=c2732d00, scratch=c41c7e00)
[   31.077390] 
[   31.083880] nfs_read_alloc_scratch(hdr=c2732a80, size=16) = c41c70c0
[   31.089983]   nfs4_xdr_enc_read_plus(hdr=c2732a80, decode_read_plus_maxsz=9)
[   31.089983]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.108844]   nfs4_xdr_dec_read_plus(hdr=c2732a80, scratch=c41c70c0)
[   31.114011]     decode_read_plus_segment(n=0)
[   31.118153]       decode_read_plus_segment(n=0): DATA {offset=16384, length=114688}
[   31.126210]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   31.126865]   nfs_readhdr_free(hdr=c2732a80, scratch=c41c70c0)
[   31.141767] 
[   31.141901] nfs_read_alloc_scratch(hdr=c2732800, size=16) = c41c7140
[   31.148607]   nfs4_xdr_enc_read_plus(hdr=c2732800, decode_read_plus_maxsz=9)
[   31.148607]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.164604]   nfs4_xdr_dec_read_plus(hdr=c2732800, scratch=c41c7140)
[   31.169862]     decode_read_plus_segment(n=0)
[   31.174101]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   31.181086]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   31.181371]   nfs_readhdr_free(hdr=c2732800, scratch=c41c7140)
[   31.183900] 
[   31.194345] nfs_read_alloc_scratch(hdr=c40fa080, size=16) = c2c29ac0
[   31.200745]   nfs4_xdr_enc_read_plus(hdr=c40fa080, decode_read_plus_maxsz=9)
[   31.200745]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.217911]   nfs4_xdr_dec_read_plus(hdr=c40fa080, scratch=c2c29ac0)
[   31.222960]     decode_read_plus_segment(n=0)
[   31.227172]       decode_read_plus_segment(n=0): DATA {offset=212992, length=69596}
[   31.235105]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   31.235377]   nfs_readhdr_free(hdr=c40fa080, scratch=c2c29ac0)
[   31.236155] 
[   31.248067] nfs_read_alloc_scratch(hdr=c40fa300, size=16) = c2c29fc0
[   31.254504]   nfs4_xdr_enc_read_plus(hdr=c40fa300, decode_read_plus_maxsz=9)
[   31.254504]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.273203]   nfs4_xdr_dec_read_plus(hdr=c40fa300, scratch=c2c29fc0)
[   31.278167]     decode_read_plus_segment(n=0)
[   31.282549]       decode_read_plus_segment(n=0): DATA {offset=16384, length=114688}
[   31.290126]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   31.290414]   nfs_readhdr_free(hdr=c40fa300, scratch=c2c29fc0)
[   31.309856] 
[   31.309986] nfs_read_alloc_scratch(hdr=c40fa580, size=16) = c2c29f40
[   31.316516]   nfs4_xdr_enc_read_plus(hdr=c40fa580, decode_read_plus_maxsz=9)
[   31.316516]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.333056]   nfs4_xdr_dec_read_plus(hdr=c40fa580, scratch=c2c29f40)
[   31.337996]     decode_read_plus_segment(n=0)
[   31.342380]       decode_read_plus_segment(n=0): DATA {offset=0, length=16384}
[   31.349520]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   31.349672]   nfs_readhdr_free(hdr=c40fa580, scratch=c2c29f40)
[   31.353977] 
[   31.362908] nfs_read_alloc_scratch(hdr=c40fa800, size=16) = c2c29bc0
[   31.369463]   nfs4_xdr_enc_read_plus(hdr=c40fa800, decode_read_plus_maxsz=9)
[   31.369463]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.390312]   nfs4_xdr_dec_read_plus(hdr=c40fa800, scratch=c2c29bc0)
[   31.395780]     decode_read_plus_segment(n=0)
[   31.399650]       decode_read_plus_segment(n=0): DATA {offset=315392, length=67848}
[   31.407423]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   31.407761]   nfs_readhdr_free(hdr=c40fa800, scratch=c2c29bc0)
[   31.408613] 
[   31.420505] nfs_read_alloc_scratch(hdr=c40faa80, size=16) = c2c29940
[   31.427029]   nfs4_xdr_enc_read_plus(hdr=c40faa80, decode_read_plus_maxsz=9)
[   31.427029]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.445821]   nfs4_xdr_dec_read_plus(hdr=c40faa80, scratch=c2c29940)
[   31.450806]     decode_read_plus_segment(n=0)
[   31.455227]       decode_read_plus_segment(n=0): DATA {offset=16384, length=114688}
[   31.463086]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   31.463598]   nfs_readhdr_free(hdr=c40faa80, scratch=c2c29940)
[   31.465006] 
[   31.475998] nfs_read_alloc_scratch(hdr=c40fad00, size=16) = c2c29d40
[   31.482472]   nfs4_xdr_enc_read_plus(hdr=c40fad00, decode_read_plus_maxsz=9)
[   31.482472]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.483031] 
[   31.497435] nfs_read_alloc_scratch(hdr=c40faf80, size=16) = c40b03c0
[   31.500200]   nfs4_xdr_dec_read_plus(hdr=c40fad00, scratch=c2c29d40)
[   31.503864]   nfs4_xdr_enc_read_plus(hdr=c40faf80, decode_read_plus_maxsz=9)
[   31.503864]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.505085] 
[   31.505100] nfs_read_alloc_scratch(hdr=c40fb200, size=16) = c3f1b440
[   31.505233]   nfs4_xdr_enc_read_plus(hdr=c40fb200, decode_read_plus_maxsz=9)
[   31.505233]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.510026]     decode_read_plus_segment(n=0)
[   31.513444]   nfs4_xdr_dec_read_plus(hdr=c40fb200, scratch=c3f1b440)
[   31.513471]     decode_read_plus_segment(n=0)
[   31.513480]       decode_read_plus_segment(n=0): DATA {offset=131072, length=131072}
[   31.513544]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   31.513834]   nfs_readhdr_free(hdr=c40fb200, scratch=c3f1b440)
[   31.528960]   nfs4_xdr_dec_read_plus(hdr=c40faf80, scratch=c40b03c0)
[   31.531568]       decode_read_plus_segment(n=0): DATA {offset=131072, length=81920}
[   31.545276]     decode_read_plus_segment(n=0)
[   31.549405]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   31.555782]       decode_read_plus_segment(n=0): DATA {offset=131072, length=131072}
[   31.560263]   nfs_readhdr_free(hdr=c40fad00, scratch=c2c29d40)
[   31.567886]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   31.618011]   nfs_readhdr_free(hdr=c40faf80, scratch=c40b03c0)
[   31.619514] 
[   31.630591] nfs_read_alloc_scratch(hdr=c1ff6f80, size=16) = c40b3800
[   31.637034]   nfs4_xdr_enc_read_plus(hdr=c1ff6f80, decode_read_plus_maxsz=9)
[   31.637034]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.658407]   nfs4_xdr_dec_read_plus(hdr=c1ff6f80, scratch=c40b3800)
[   31.663427]     decode_read_plus_segment(n=0)
[   31.667722]       decode_read_plus_segment(n=0): DATA {offset=700416, length=110592}
[   31.675505]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   31.675882]   nfs_readhdr_free(hdr=c1ff6f80, scratch=c40b3800)
[   31.678202] 
[   31.688617] nfs_read_alloc_scratch(hdr=c1ff6d00, size=16) = c40b3580
[   31.695039]   nfs4_xdr_enc_read_plus(hdr=c1ff6d00, decode_read_plus_maxsz=9)
[   31.695039]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.714452]   nfs4_xdr_dec_read_plus(hdr=c1ff6d00, scratch=c40b3580)
[   31.719364]     decode_read_plus_segment(n=0)
[   31.723749]       decode_read_plus_segment(n=0): DATA {offset=475136, length=131072}
[   31.731518]       decode_read_plus_segment(n=0): xdr->nwords = 0
[   31.732116]   nfs_readhdr_free(hdr=c1ff6d00, scratch=c40b3580)
[   31.744364] 
[   31.744660] nfs_read_alloc_scratch(hdr=c32c3200, size=16) = c3f6a100
[   31.751035]   nfs4_xdr_enc_read_plus(hdr=c32c3200, decode_read_plus_maxsz=9)
[   31.751035]     xdr->buf = {head->iov_len=196, page_base=0, page_len=0}
[   31.770512]   nfs4_xdr_dec_read_plus(hdr=c32c3200, scratch=c3f6a100)
[   31.775484]     decode_read_plus_segment(n=0)
[   31.779742]       decode_read_plus_segment(n=0): DATA {offset=0, length=12288}
[   31.786998]       decode_read_plus_segment(n=0): xdr->nwords = 13357
[   31.787015]     decode_read_plus_segment(n=1)
[   31.797592]         __xdr_inline_decode(nbytes=4092, nwords=1023, xdr, q=00001000)
[   31.797592]           xdr = {nwords=13356, p=00000004, end=00001000}
[   31.811530]       xdr_copy_to_scratch(1428): memcpy(c3f6a100, 00000004, 4092)
[   31.818572] 8<--- cut here ---
[   31.821635] Unable to handle kernel NULL pointer dereference at virtual address 00000004 when read
[   31.830528] [00000004] *pgd=00000000
[   31.834161] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[   31.839363] Modules linked in:
[   31.842394] CPU: 6 PID: 163 Comm: kworker/u16:4 Not tainted 6.4.0-rc6-next-20230616-00004-g399ba25dd3af #151
[   31.852189] Hardware name: Samsung Exynos (Flattened Device Tree)
[   31.858250] Workqueue: rpciod rpc_async_schedule
[   31.862841] PC is at mmiocpy+0x4c/0x334
[   31.866652] LR is at xdr_inline_decode+0x140/0x2a4
[   31.871426] pc : [<c0bc6a8c>]    lr : [<c0b8e278>]    psr: 20010013
[   31.877660] sp : f0d7dd98  ip : 00000000  fp : f0d7dda8
[   31.882857] r10: c0d19288  r9 : 0000000b  r8 : c3f6a100
[   31.888057] r7 : 00000004  r6 : 00000ffc  r5 : 00000004  r4 : f0d7de70
[   31.894557] r3 : 00000000  r2 : 00000f7c  r1 : 00000004  r0 : c3f6a100
[   31.901058] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   31.908163] Control: 10c5387d  Table: 448a006a  DAC: 00000051
[   31.913880] Register r0 information: slab kmalloc-64 start c3f6a100 pointer offset 0 size 64
[   31.922290] Register r1 information: non-paged memory
[   31.927310] Register r2 information: non-paged memory
[   31.932337] Register r3 information: NULL pointer
[   31.937016] Register r4 information: 2-page vmalloc region starting at 0xf0d7c000 allocated at kernel_clone+0x58/0x3b4
[   31.947684] Register r5 information: non-paged memory
[   31.952704] Register r6 information: non-paged memory
[   31.957726] Register r7 information: non-paged memory
[   31.962752] Register r8 information: slab kmalloc-64 start c3f6a100 pointer offset 0 size 64
[   31.971163] Register r9 information: non-paged memory
[   31.976184] Register r10 information: non-slab/vmalloc memory
[   31.981905] Register r11 information: 2-page vmalloc region starting at 0xf0d7c000 allocated at kernel_clone+0x58/0x3b4
[   31.992655] Register r12 information: NULL pointer
[   31.997414] Process kworker/u16:4 (pid: 163, stack limit = 0x167f457e)
[   32.003916] Stack: (0xf0d7dd98 to 0xf0d7e000)
[   32.008246] dd80:                                                       00000004 00000ffc
[   32.016398] dda0: c3f6a100 0000000b c3f6a100 f0d7de70 c100bd00 c0b8e278 00000004 00000ffc
[   32.024543] ddc0: 00003000 c32c3400 f0d7de70 00000001 c40c8a18 c0bf7fdc 00000000 00000000
[   32.032689] dde0: 00003000 c1c20000 c2759405 c40c8a00 c100bd44 0000000b 00000000 00000000
[   32.040833] de00: 00000003 f0d7de2c 00000000 00000000 c4363000 c0c062e0 c3367a1c 00000000
[   32.048979] de20: c3367a94 00000000 00000000 c43cc100 00000000 00003000 f0d7de70 00003000
[   32.057125] de40: 00003000 6f8bfcf9 f0d7de70 c32c3258 c3fb8200 c1308f58 c3fb8230 c4363000
[   32.065271] de60: c0b6260c c0b7d27c c32c3274 c0b6280c 00001000 c3fb8230 00001000 00000000
[   32.073416] de80: c3f6a100 00000010 c43cc10c 0000302d c3fb8200 6f8bfcf9 c32c3258 c1308f58
[   32.081561] dea0: c4363000 c0b6260c c1465480 c0b7e758 00000001 00000001 04248060 c145d5d7
[   32.089707] dec0: c4363000 c0199110 c01475c4 6f8bfcf9 c1c20000 c4363000 04208060 c2759400
[   32.097853] dee0: c1c20000 c4363000 c1308f58 c145d900 c2759405 c0b7ede0 c32c327c c41c6980
[   32.105998] df00: c2759400 c0147528 00000001 00000000 c0147468 6f8bfcf9 c0d02b90 c1c20000
[   32.114143] df20: c1af01a4 00000000 00000000 c10b5264 00000000 6f8bfcf9 c1c20000 c41c6980
[   32.122288] df40: c1c20000 c41c69ac c1c20040 c1305d40 00000088 c4363000 c1c20000 c01479ec
[   32.130434] df60: c4363000 c145c8de c24cd700 c4363000 c41c7bc0 c0147998 c41c6980 c24cd700
[   32.138580] df80: f0a99ebc 00000000 00000000 c0151330 c41c7bc0 c015122c 00000000 00000000
[   32.146725] dfa0: 00000000 00000000 00000000 c0100108 00000000 00000000 00000000 00000000
[   32.154871] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   32.163016] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[   32.171174]  mmiocpy from xdr_inline_decode+0x140/0x2a4
[   32.176359]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x1b0/0x3fc
[   32.182860]  nfs4_xdr_dec_read_plus from call_decode+0x200/0x2bc
[   32.188840]  call_decode from __rpc_execute+0xf8/0x764
[   32.193949]  __rpc_execute from rpc_async_schedule+0x1c/0x34
[   32.199579]  rpc_async_schedule from process_one_work+0x2b4/0x724
[   32.205652]  process_one_work from worker_thread+0x54/0x518
[   32.211191]  worker_thread from kthread+0x104/0x138
[   32.216047]  kthread from ret_from_fork+0x14/0x2c
[   32.220721] Exception stack(0xf0d7dfb0 to 0xf0d7dff8)
[   32.225749] dfa0:                                     00000000 00000000 00000000 00000000
[   32.233900] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   32.242044] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[   32.248633] Code: ba000002 f5d1f03c f5d1f05c f5d1f07c (e8b15378) 
[   32.254858] ---[ end trace 0000000000000000 ]---
[   39.512253] vddq_mmc0: disabling
[   39.515712] vdd_ldo12: disabling
[   39.519751] vdd_ldo16: disabling
[   39.525706] vdd_ldo24: disabling
[   39.529730] vdd_ldo26: disabling
[   39.546463] vdd_vmem: disabling

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Anna Schumaker]

On Sat, Jun 17, 2023 at 6:09 AM Krzysztof Kozlowski
<krzysztof.kozlowski@linaro.org> wrote:
>
> On 15/06/2023 21:38, Anna Schumaker wrote:
> > On Thu, Jun 15, 2023 at 1:16 PM Anna Schumaker <schumaker.anna@gmail.com> wrote:
> >>
> >> On Thu, Jun 15, 2023 at 1:04 PM Anna Schumaker <schumaker.anna@gmail.com> wrote:
> >>>
> >>> On Thu, Jun 15, 2023 at 9:01 AM Anna Schumaker <schumaker.anna@gmail.com> wrote:
> >>>>
> >>>> On Thu, Jun 15, 2023 at 4:55 AM Krzysztof Kozlowski
> >>>> <krzysztof.kozlowski@linaro.org> wrote:
> >>>>>
> >>>>> On 15/06/2023 10:52, Krzysztof Kozlowski wrote:
> >>>>>> On 14/06/2023 22:55, Anna Schumaker wrote:
> >>>>>>>>>> Still null ptr (built on 420b2d4 with your patch):
> >>>>>>>>>
> >>>>>>>>> We're through the merge window and at rc1 now, so I can spend more
> >>>>>>>>> time scratching my head over your bug again. We've come up with a
> >>>>>>>>> patch (attached) that adds a bunch of printks to show us what the
> >>>>>>>>> kernel thinks is going on. Do you mind trying it out and letting us
> >>>>>>>>> know what gets printed out? You'll need to make sure
> >>>>>>>>> CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.
> >>>>>>>>
> >>>>>>>> The patch does not apply. I tried: v6.4-rc1, v6.4-rc5, next-20230609.
> >>>>>>>
> >>>>>>> Can you try the attached patch on top of my 3-patch series from the
> >>>>>>> other day, and let me know what gets printed out? It adds a bunch of
> >>>>>>> printk()s at strategic points to print out what is going on with the
> >>>>>>> xdr scratch buffer since it's suddenly a bad memory address after
> >>>>>>> working for a bit on your machine.
> >>>>>>>
> >>>>>>
> >>>>>> Here you have entire log - attached (113 kB, I hope goes past mailing
> >>>>>> lists/spam filters).
> >>>>>
> >>>>> As expected this bounced from the mailing lists, but I hope you got it.
> >>>>> If not, let me know.
> >>>>
> >>>> I did still receive it. Thanks!
> >>>
> >>> Can you swap out yesterday's patch with this patch? I've adjusted what
> >>> gets printed out, and added printk()s to xdr_copy_to_scratch().  I'm
> >>> starting to think that the xdr scratch buffer is fine, and that it's
> >>> the other pointer passed to memcpy() in that function that's the
> >>> problem, and the output from this patch will confirm for me.
> >>
> >> Oh, and can you add this one on top of the v2 patch as well?
> >
> > Sorry about the noise today. Can you use this patch instead of the two
> > I attached earlier? I cleaned up the output and cut down on extra
> > output..
> >
>
> Here you have - attached.

This is good, thanks! I was finally able to figure out how to hit the
bug using a 32bit x86 VM, so hopefully the next thing you hear from me
is a patch fixing the bug!

Anna

>
>
> Best regards,
> Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

On 21/06/2023 14:49, Anna Schumaker wrote:
> On Sat, Jun 17, 2023 at 6:09 AM Krzysztof Kozlowski
> <krzysztof.kozlowski@linaro.org> wrote:
>>
>> On 15/06/2023 21:38, Anna Schumaker wrote:
>>> On Thu, Jun 15, 2023 at 1:16 PM Anna Schumaker <schumaker.anna@gmail.com> wrote:
>>>>
>>>> On Thu, Jun 15, 2023 at 1:04 PM Anna Schumaker <schumaker.anna@gmail.com> wrote:
>>>>>
>>>>> On Thu, Jun 15, 2023 at 9:01 AM Anna Schumaker <schumaker.anna@gmail.com> wrote:
>>>>>>
>>>>>> On Thu, Jun 15, 2023 at 4:55 AM Krzysztof Kozlowski
>>>>>> <krzysztof.kozlowski@linaro.org> wrote:
>>>>>>>
>>>>>>> On 15/06/2023 10:52, Krzysztof Kozlowski wrote:
>>>>>>>> On 14/06/2023 22:55, Anna Schumaker wrote:
>>>>>>>>>>>> Still null ptr (built on 420b2d4 with your patch):
>>>>>>>>>>>
>>>>>>>>>>> We're through the merge window and at rc1 now, so I can spend more
>>>>>>>>>>> time scratching my head over your bug again. We've come up with a
>>>>>>>>>>> patch (attached) that adds a bunch of printks to show us what the
>>>>>>>>>>> kernel thinks is going on. Do you mind trying it out and letting us
>>>>>>>>>>> know what gets printed out? You'll need to make sure
>>>>>>>>>>> CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.
>>>>>>>>>>
>>>>>>>>>> The patch does not apply. I tried: v6.4-rc1, v6.4-rc5, next-20230609.
>>>>>>>>>
>>>>>>>>> Can you try the attached patch on top of my 3-patch series from the
>>>>>>>>> other day, and let me know what gets printed out? It adds a bunch of
>>>>>>>>> printk()s at strategic points to print out what is going on with the
>>>>>>>>> xdr scratch buffer since it's suddenly a bad memory address after
>>>>>>>>> working for a bit on your machine.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Here you have entire log - attached (113 kB, I hope goes past mailing
>>>>>>>> lists/spam filters).
>>>>>>>
>>>>>>> As expected this bounced from the mailing lists, but I hope you got it.
>>>>>>> If not, let me know.
>>>>>>
>>>>>> I did still receive it. Thanks!
>>>>>
>>>>> Can you swap out yesterday's patch with this patch? I've adjusted what
>>>>> gets printed out, and added printk()s to xdr_copy_to_scratch().  I'm
>>>>> starting to think that the xdr scratch buffer is fine, and that it's
>>>>> the other pointer passed to memcpy() in that function that's the
>>>>> problem, and the output from this patch will confirm for me.
>>>>
>>>> Oh, and can you add this one on top of the v2 patch as well?
>>>
>>> Sorry about the noise today. Can you use this patch instead of the two
>>> I attached earlier? I cleaned up the output and cut down on extra
>>> output..
>>>
>>
>> Here you have - attached.
> 
> This is good, thanks! I was finally able to figure out how to hit the
> bug using a 32bit x86 VM, so hopefully the next thing you hear from me
> is a patch fixing the bug!

QEMU also has 32-bit ARM and x86...

Best regards,
Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Anna Schumaker]

[-- Attachment #1: Type: text/plain, Size: 3291 bytes --]

Hi Krzysztof,

On Wed, Jun 21, 2023 at 9:27 AM Krzysztof Kozlowski
<krzysztof.kozlowski@linaro.org> wrote:
>
> On 21/06/2023 14:49, Anna Schumaker wrote:
> > On Sat, Jun 17, 2023 at 6:09 AM Krzysztof Kozlowski
> > <krzysztof.kozlowski@linaro.org> wrote:
> >>
> >> On 15/06/2023 21:38, Anna Schumaker wrote:
> >>> On Thu, Jun 15, 2023 at 1:16 PM Anna Schumaker <schumaker.anna@gmail.com> wrote:
> >>>>
> >>>> On Thu, Jun 15, 2023 at 1:04 PM Anna Schumaker <schumaker.anna@gmail.com> wrote:
> >>>>>
> >>>>> On Thu, Jun 15, 2023 at 9:01 AM Anna Schumaker <schumaker.anna@gmail.com> wrote:
> >>>>>>
> >>>>>> On Thu, Jun 15, 2023 at 4:55 AM Krzysztof Kozlowski
> >>>>>> <krzysztof.kozlowski@linaro.org> wrote:
> >>>>>>>
> >>>>>>> On 15/06/2023 10:52, Krzysztof Kozlowski wrote:
> >>>>>>>> On 14/06/2023 22:55, Anna Schumaker wrote:
> >>>>>>>>>>>> Still null ptr (built on 420b2d4 with your patch):
> >>>>>>>>>>>
> >>>>>>>>>>> We're through the merge window and at rc1 now, so I can spend more
> >>>>>>>>>>> time scratching my head over your bug again. We've come up with a
> >>>>>>>>>>> patch (attached) that adds a bunch of printks to show us what the
> >>>>>>>>>>> kernel thinks is going on. Do you mind trying it out and letting us
> >>>>>>>>>>> know what gets printed out? You'll need to make sure
> >>>>>>>>>>> CONFIG_NFS_V4_2_READ_PLUS is enabled when compiling the kernel.
> >>>>>>>>>>
> >>>>>>>>>> The patch does not apply. I tried: v6.4-rc1, v6.4-rc5, next-20230609.
> >>>>>>>>>
> >>>>>>>>> Can you try the attached patch on top of my 3-patch series from the
> >>>>>>>>> other day, and let me know what gets printed out? It adds a bunch of
> >>>>>>>>> printk()s at strategic points to print out what is going on with the
> >>>>>>>>> xdr scratch buffer since it's suddenly a bad memory address after
> >>>>>>>>> working for a bit on your machine.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> Here you have entire log - attached (113 kB, I hope goes past mailing
> >>>>>>>> lists/spam filters).
> >>>>>>>
> >>>>>>> As expected this bounced from the mailing lists, but I hope you got it.
> >>>>>>> If not, let me know.
> >>>>>>
> >>>>>> I did still receive it. Thanks!
> >>>>>
> >>>>> Can you swap out yesterday's patch with this patch? I've adjusted what
> >>>>> gets printed out, and added printk()s to xdr_copy_to_scratch().  I'm
> >>>>> starting to think that the xdr scratch buffer is fine, and that it's
> >>>>> the other pointer passed to memcpy() in that function that's the
> >>>>> problem, and the output from this patch will confirm for me.
> >>>>
> >>>> Oh, and can you add this one on top of the v2 patch as well?
> >>>
> >>> Sorry about the noise today. Can you use this patch instead of the two
> >>> I attached earlier? I cleaned up the output and cut down on extra
> >>> output..
> >>>
> >>
> >> Here you have - attached.
> >
> > This is good, thanks! I was finally able to figure out how to hit the
> > bug using a 32bit x86 VM, so hopefully the next thing you hear from me
> > is a patch fixing the bug!

I'm really hopeful that the attached patch finally fixes the issue.
Can you try it and let me know?

Thanks,
Anna

>
> QEMU also has 32-bit ARM and x86...
>
> Best regards,
> Krzysztof
>

[-- Attachment #2: 0001-SUNRPC-kmap-the-xdr-pages-during-decode.patch --]
[-- Type: text/x-patch, Size: 3943 bytes --]

From 88be621fc687d85891636ed4695ab993e1f2cb3b Mon Sep 17 00:00:00 2001
From: Anna Schumaker <Anna.Schumaker@Netapp.com>
Date: Fri, 23 Jun 2023 11:43:14 -0400
Subject: [PATCH] SUNRPC: kmap() the xdr pages during decode

If the pages are in HIGHMEM then we need to make sure they're mapped
before trying to read data off of them, otherwise we could end up with a
NULL pointer dereference.

Reported-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
---
 include/linux/sunrpc/xdr.h |  2 ++
 net/sunrpc/clnt.c          |  1 +
 net/sunrpc/xdr.c           | 20 ++++++++++++++------
 3 files changed, 17 insertions(+), 6 deletions(-)

diff --git a/include/linux/sunrpc/xdr.h b/include/linux/sunrpc/xdr.h
index d917618a3058..f562aab468f5 100644
--- a/include/linux/sunrpc/xdr.h
+++ b/include/linux/sunrpc/xdr.h
@@ -228,6 +228,7 @@ struct xdr_stream {
 	struct kvec *iov;	/* pointer to the current kvec */
 	struct kvec scratch;	/* Scratch buffer */
 	struct page **page_ptr;	/* pointer to the current page */
+	void *page_kaddr;	/* kmapped address of the current page */
 	unsigned int nwords;	/* Remaining decode buffer length */
 
 	struct rpc_rqst *rqst;	/* For debugging */
@@ -254,6 +255,7 @@ extern void xdr_truncate_decode(struct xdr_stream *xdr, size_t len);
 extern int xdr_restrict_buflen(struct xdr_stream *xdr, int newbuflen);
 extern void xdr_write_pages(struct xdr_stream *xdr, struct page **pages,
 		unsigned int base, unsigned int len);
+extern void xdr_stream_unmap_current_page(struct xdr_stream *xdr);
 extern unsigned int xdr_stream_pos(const struct xdr_stream *xdr);
 extern unsigned int xdr_page_pos(const struct xdr_stream *xdr);
 extern void xdr_init_decode(struct xdr_stream *xdr, struct xdr_buf *buf,
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index d2ee56634308..3b7e676d8935 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -2590,6 +2590,7 @@ call_decode(struct rpc_task *task)
 	case 0:
 		task->tk_action = rpc_exit_task;
 		task->tk_status = rpcauth_unwrap_resp(task, &xdr);
+		xdr_stream_unmap_current_page(&xdr);
 		return;
 	case -EAGAIN:
 		task->tk_status = 0;
diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
index 391b336d97de..b073f51e6018 100644
--- a/net/sunrpc/xdr.c
+++ b/net/sunrpc/xdr.c
@@ -1308,6 +1308,14 @@ static unsigned int xdr_set_tail_base(struct xdr_stream *xdr,
 	return xdr_set_iov(xdr, buf->tail, base, len);
 }
 
+void xdr_stream_unmap_current_page(struct xdr_stream *xdr)
+{
+	if (xdr->page_kaddr) {
+		kunmap_local(xdr->page_kaddr);
+		xdr->page_kaddr = NULL;
+	}
+}
+
 static unsigned int xdr_set_page_base(struct xdr_stream *xdr,
 				      unsigned int base, unsigned int len)
 {
@@ -1315,7 +1323,6 @@ static unsigned int xdr_set_page_base(struct xdr_stream *xdr,
 	unsigned int maxlen;
 	unsigned int pgoff;
 	unsigned int pgend;
-	void *kaddr;
 
 	maxlen = xdr->buf->page_len;
 	if (base >= maxlen)
@@ -1330,15 +1337,15 @@ static unsigned int xdr_set_page_base(struct xdr_stream *xdr,
 
 	pgnr = base >> PAGE_SHIFT;
 	xdr->page_ptr = &xdr->buf->pages[pgnr];
-	kaddr = page_address(*xdr->page_ptr);
+	xdr->page_kaddr = kmap_local_page(*xdr->page_ptr);
 
 	pgoff = base & ~PAGE_MASK;
-	xdr->p = (__be32*)(kaddr + pgoff);
+	xdr->p = (__be32*)(xdr->page_kaddr + pgoff);
 
 	pgend = pgoff + len;
 	if (pgend > PAGE_SIZE)
 		pgend = PAGE_SIZE;
-	xdr->end = (__be32*)(kaddr + pgend);
+	xdr->end = (__be32*)(xdr->page_kaddr + pgend);
 	xdr->iov = NULL;
 	return len;
 }
@@ -1366,9 +1373,10 @@ static void xdr_set_next_page(struct xdr_stream *xdr)
 
 static bool xdr_set_next_buffer(struct xdr_stream *xdr)
 {
-	if (xdr->page_ptr != NULL)
+	if (xdr->page_ptr != NULL) {
+		xdr_stream_unmap_current_page(xdr);
 		xdr_set_next_page(xdr);
-	else if (xdr->iov == xdr->buf->head)
+	} else if (xdr->iov == xdr->buf->head)
 		xdr_set_page(xdr, 0, xdr_stream_remaining(xdr));
 	return xdr->p != xdr->end;
 }
-- 
2.41.0

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Krzysztof Kozlowski]

On 23/06/2023 19:59, Anna Schumaker wrote:
>>>>>>>
>>>>>>> Can you swap out yesterday's patch with this patch? I've adjusted what
>>>>>>> gets printed out, and added printk()s to xdr_copy_to_scratch().  I'm
>>>>>>> starting to think that the xdr scratch buffer is fine, and that it's
>>>>>>> the other pointer passed to memcpy() in that function that's the
>>>>>>> problem, and the output from this patch will confirm for me.
>>>>>>
>>>>>> Oh, and can you add this one on top of the v2 patch as well?
>>>>>
>>>>> Sorry about the noise today. Can you use this patch instead of the two
>>>>> I attached earlier? I cleaned up the output and cut down on extra
>>>>> output..
>>>>>
>>>>
>>>> Here you have - attached.
>>>
>>> This is good, thanks! I was finally able to figure out how to hit the
>>> bug using a 32bit x86 VM, so hopefully the next thing you hear from me
>>> is a patch fixing the bug!
> 
> I'm really hopeful that the attached patch finally fixes the issue.
> Can you try it and let me know?

Just test it yourself on 32-bit system... There is absolutely nothing
special in the system I reproduced it on. Nothing.


IP-Config: eth0 hardware address 00:1e:06:30:bf:ac mtu 1500
IP-Config: eth0 guessed broadcast address 192.168.1.255
IP-Config: eth0 complete (from 192.168.1.10):
 address: 192.168.1.12     broadcast: 192.168.1.255    netmask:
255.255.255.0
 gateway: 192.168.1.1      dns0     : 0.0.0.0          dns1   : 0.0.0.0

 rootserver: 192.168.1.10 rootpath:
 filename  :
NFS-Mount: 192.168.1.10:/srv/nfs/odroidhc1
Waiting 10 seconds for device /dev/nfs ...
ERROR: device '/dev/nfs' not found. Skipping fsck.
Mount cmd:
mount.nfs4 -o vers=4,nolock 192.168.1.10:/srv/nfs/odroidhc1 /new_root
[   21.800626] ------------[ cut here ]------------
[   21.803891] WARNING: CPU: 7 PID: 154 at mm/highmem.c:603
xdr_stream_unmap_current_page+0x18/0x24
[   21.812729] Modules linked in:
[   21.815642] CPU: 7 PID: 154 Comm: mount.nfs4 Not tainted
6.4.0-00001-gfbb103bb8df0 #8
[   21.823444] Hardware name: Samsung Exynos (Flattened Device Tree)
[   21.829525]  unwind_backtrace from show_stack+0x10/0x14
[   21.834698]  show_stack from dump_stack_lvl+0x58/0x70
[   21.839730]  dump_stack_lvl from __warn+0x7c/0x1bc
[   21.844491]  __warn from warn_slowpath_fmt+0xbc/0x1b8
[   21.849518]  warn_slowpath_fmt from
xdr_stream_unmap_current_page+0x18/0x24
[   21.856437]  xdr_stream_unmap_current_page from call_decode+0x210/0x2c8
[   21.863020]  call_decode from __rpc_execute+0xf8/0x764
[   21.868134]  __rpc_execute from rpc_execute+0xc0/0x1d0
[   21.873243]  rpc_execute from rpc_run_task+0x148/0x190
[   21.878348]  rpc_run_task from rpc_create_xprt+0x1a4/0x284
[   21.883805]  rpc_create_xprt from rpc_create+0xf8/0x254
[   21.889004]  rpc_create from nfs_create_rpc_client+0x150/0x17c
[   21.894812]  nfs_create_rpc_client from nfs4_alloc_client+0x360/0x374
[   21.901226]  nfs4_alloc_client from nfs_get_client+0x16c/0x3e8
[   21.907030]  nfs_get_client from nfs4_set_client+0xfc/0x1a4
[   21.912574]  nfs4_set_client from nfs4_create_server+0x11c/0x2fc
[   21.918554]  nfs4_create_server from nfs4_try_get_tree+0x10/0x50
[   21.924534]  nfs4_try_get_tree from vfs_get_tree+0x24/0xe4
[   21.929993]  vfs_get_tree from path_mount+0x3e8/0xb04
[   21.935019]  path_mount from sys_mount+0x20c/0x254
[   21.939784]  sys_mount from ret_fast_syscall+0x0/0x1c
[   21.944809] Exception stack(0xf0cf9fa8 to 0xf0cf9ff0)
[   21.949837] 9fa0:                   0047ebe0 00479c64 0047e960
0047e9b8 0047e9c8 00000000
[   21.957986] 9fc0: 0047ebe0 00479c64 b6f058c8 00000015 00466c08
00000010 00479c64 00466bfc
[   21.966139] 9fe0: 00479e70 befb69b0 0045a708 b6dca610
[   21.971245] irq event stamp: 0
[   21.974188] hardirqs last  enabled at (0): [<00000000>] 0x0
[   21.979736] hardirqs last disabled at (0): [<c012357c>]
copy_process+0x810/0x1ffc
[   21.987227] softirqs last  enabled at (0): [<c012357c>]
copy_process+0x810/0x1ffc
[   21.994679] softirqs last disabled at (0): [<00000000>] 0x0
[   22.000264] ---[ end trace 0000000000000000 ]---
[   22.004781] BUG: sleeping function called from invalid context at
net/sunrpc/sched.c:953
[   22.012876] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid:
154, name: mount.nfs4
[   22.020936] preempt_count: 1, expected: 0
[   22.024881] RCU nest depth: 0, expected: 0
[   22.028955] INFO: lockdep is turned off.
[   22.032889] CPU: 7 PID: 154 Comm: mount.nfs4 Tainted: G        W
    6.4.0-00001-gfbb103bb8df0 #8
[   22.042131] Hardware name: Samsung Exynos (Flattened Device Tree)
[   22.048196]  unwind_backtrace from show_stack+0x10/0x14
[   22.053393]  show_stack from dump_stack_lvl+0x58/0x70
[   22.058417]  dump_stack_lvl from __might_resched+0x194/0x260
[   22.064054]  __might_resched from __rpc_execute+0x118/0x764
[   22.069596]  __rpc_execute from rpc_execute+0xc0/0x1d0
[   22.074708]  rpc_execute from rpc_run_task+0x148/0x190
[   22.079821]  rpc_run_task from rpc_create_xprt+0x1a4/0x284
[   22.085281]  rpc_create_xprt from rpc_create+0xf8/0x254
[   22.090483]  rpc_create from nfs_create_rpc_client+0x150/0x17c
[   22.096286]  nfs_create_rpc_client from nfs4_alloc_client+0x360/0x374
[   22.102700]  nfs4_alloc_client from nfs_get_client+0x16c/0x3e8
[   22.108504]  nfs_get_client from nfs4_set_client+0xfc/0x1a4
[   22.114050]  nfs4_set_client from nfs4_create_server+0x11c/0x2fc
[   22.120029]  nfs4_create_server from nfs4_try_get_tree+0x10/0x50
[   22.126009]  nfs4_try_get_tree from vfs_get_tree+0x24/0xe4
[   22.131467]  vfs_get_tree from path_mount+0x3e8/0xb04
[   22.136493]  path_mount from sys_mount+0x20c/0x254
[   22.141258]  sys_mount from ret_fast_syscall+0x0/0x1c
[   22.146284] Exception stack(0xf0cf9fa8 to 0xf0cf9ff0)
[   22.151322] 9fa0:                   0047ebe0 00479c64 0047e960
0047e9b8 0047e9c8 00000000
[   22.159461] 9fc0: 0047ebe0 00479c64 b6f058c8 00000015 00466c08
00000010 00479c64 00466bfc
[   22.167606] 9fe0: 00479e70 befb69b0 0045a708 b6dca610
[   22.172820] BUG: scheduling while atomic: mount.nfs4/154/0x00000002
[   22.178871] INFO: lockdep is turned off.
[   22.182803] Modules linked in:
[   22.185798] CPU: 7 PID: 154 Comm: mount.nfs4 Tainted: G        W
    6.4.0-00001-gfbb103bb8df0 #8
[   22.195076] Hardware name: Samsung Exynos (Flattened Device Tree)
[   22.201139]  unwind_backtrace from show_stack+0x10/0x14
[   22.206337]  show_stack from dump_stack_lvl+0x58/0x70
[   22.211365]  dump_stack_lvl from __schedule_bug+0x70/0x84
[   22.216736]  __schedule_bug from __schedule+0x9c0/0xc80
[   22.221936]  __schedule from schedule+0x58/0xf8
[   22.226439]  schedule from schedule_timeout+0x134/0x200
[   22.231641]  schedule_timeout from __wait_for_common+0xac/0x1f8
[   22.237533]  __wait_for_common from
wait_for_completion_killable+0x18/0x24
[   22.244379]  wait_for_completion_killable from
__kthread_create_on_node+0xe0/0x168
[   22.251923]  __kthread_create_on_node from
kthread_create_on_node+0x30/0x60
[   22.258851]  kthread_create_on_node from svc_set_num_threads+0x1c8/0x420
[   22.265525]  svc_set_num_threads from nfs_callback_up+0x150/0x3c0
[   22.271597]  nfs_callback_up from nfs4_init_client+0x98/0x144
[   22.277306]  nfs4_init_client from nfs4_set_client+0xfc/0x1a4
[   22.283026]  nfs4_set_client from nfs4_create_server+0x11c/0x2fc
[   22.289005]  nfs4_create_server from nfs4_try_get_tree+0x10/0x50
[   22.294985]  nfs4_try_get_tree from vfs_get_tree+0x24/0xe4
[   22.300444]  vfs_get_tree from path_mount+0x3e8/0xb04
[   22.305468]  path_mount from sys_mount+0x20c/0x254
[   22.310249]  sys_mount from ret_fast_syscall+0x0/0x1c
[   22.315261] Exception stack(0xf0cf9fa8 to 0xf0cf9ff0)
[   22.320300] 9fa0:                   0047ebe0 00479c64 0047e960
0047e9b8 0047e9c8 00000000
[   22.328438] 9fc0: 0047ebe0 00479c64 b6f058c8 00000015 00466c08
00000010 00479c64 00466bfc
[   22.336582] 9fe0: 00479e70 befb69b0 0045a708 b6dca610
:: running cleanup hook [udev]
[   26.235349] systemd[1]: System time before build time, advancing clock.
[   26.435536] systemd[1]: systemd 253.4-1-arch running in system mode
(+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS
+OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD
+LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2
+BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT
default-hierarchy=unified)
[   26.466749] systemd[1]: Detected architecture arm.



Best regards,
Krzysztof

Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)

[Anna Schumaker]

[-- Attachment #1: Type: text/plain, Size: 9891 bytes --]

Hi Krzysztof,

On Mon, Jun 26, 2023 at 6:28 AM Krzysztof Kozlowski
<krzysztof.kozlowski@linaro.org> wrote:
>
> On 23/06/2023 19:59, Anna Schumaker wrote:
> >>>>>>>
> >>>>>>> Can you swap out yesterday's patch with this patch? I've adjusted what
> >>>>>>> gets printed out, and added printk()s to xdr_copy_to_scratch().  I'm
> >>>>>>> starting to think that the xdr scratch buffer is fine, and that it's
> >>>>>>> the other pointer passed to memcpy() in that function that's the
> >>>>>>> problem, and the output from this patch will confirm for me.
> >>>>>>
> >>>>>> Oh, and can you add this one on top of the v2 patch as well?
> >>>>>
> >>>>> Sorry about the noise today. Can you use this patch instead of the two
> >>>>> I attached earlier? I cleaned up the output and cut down on extra
> >>>>> output..
> >>>>>
> >>>>
> >>>> Here you have - attached.
> >>>
> >>> This is good, thanks! I was finally able to figure out how to hit the
> >>> bug using a 32bit x86 VM, so hopefully the next thing you hear from me
> >>> is a patch fixing the bug!
> >
> > I'm really hopeful that the attached patch finally fixes the issue.
> > Can you try it and let me know?
>
> Just test it yourself on 32-bit system... There is absolutely nothing
> special in the system I reproduced it on. Nothing.
>

I have an updated set of patches for you to try out, hopefully fixing
that last set of warnings from the other day. I gave them the same
amount of testing as my previous patch: connectathon tests, xfstests,
and my own read plus test using x86_64 and i686 VMs mounted with NFS
versions 3, 4.0, 4.1, and 4.2 and sec=sys, sec=krb5, sec=krb5i, and
sec=krb5p. I have not hit any oopses, warnings, or newly-failing
tests.

I would appreciate it if you could try it out on your hardware and let
me know if there is still an issue since trying to compile an
exynos_defconfig for an i686 VM will result in a different set of
Kconfig options getting selected compared to what you have.
Additionally, your odroid-hc1 has a unique CPU setup containing both a
Cortex-A15 and Cortex-A7 which isn't possible to virtualize with
libvirt.

Thanks.
Anna
>
> IP-Config: eth0 hardware address 00:1e:06:30:bf:ac mtu 1500
> IP-Config: eth0 guessed broadcast address 192.168.1.255
> IP-Config: eth0 complete (from 192.168.1.10):
>  address: 192.168.1.12     broadcast: 192.168.1.255    netmask:
> 255.255.255.0
>  gateway: 192.168.1.1      dns0     : 0.0.0.0          dns1   : 0.0.0.0
>
>  rootserver: 192.168.1.10 rootpath:
>  filename  :
> NFS-Mount: 192.168.1.10:/srv/nfs/odroidhc1
> Waiting 10 seconds for device /dev/nfs ...
> ERROR: device '/dev/nfs' not found. Skipping fsck.
> Mount cmd:
> mount.nfs4 -o vers=4,nolock 192.168.1.10:/srv/nfs/odroidhc1 /new_root
> [   21.800626] ------------[ cut here ]------------
> [   21.803891] WARNING: CPU: 7 PID: 154 at mm/highmem.c:603
> xdr_stream_unmap_current_page+0x18/0x24
> [   21.812729] Modules linked in:
> [   21.815642] CPU: 7 PID: 154 Comm: mount.nfs4 Not tainted
> 6.4.0-00001-gfbb103bb8df0 #8
> [   21.823444] Hardware name: Samsung Exynos (Flattened Device Tree)
> [   21.829525]  unwind_backtrace from show_stack+0x10/0x14
> [   21.834698]  show_stack from dump_stack_lvl+0x58/0x70
> [   21.839730]  dump_stack_lvl from __warn+0x7c/0x1bc
> [   21.844491]  __warn from warn_slowpath_fmt+0xbc/0x1b8
> [   21.849518]  warn_slowpath_fmt from
> xdr_stream_unmap_current_page+0x18/0x24
> [   21.856437]  xdr_stream_unmap_current_page from call_decode+0x210/0x2c8
> [   21.863020]  call_decode from __rpc_execute+0xf8/0x764
> [   21.868134]  __rpc_execute from rpc_execute+0xc0/0x1d0
> [   21.873243]  rpc_execute from rpc_run_task+0x148/0x190
> [   21.878348]  rpc_run_task from rpc_create_xprt+0x1a4/0x284
> [   21.883805]  rpc_create_xprt from rpc_create+0xf8/0x254
> [   21.889004]  rpc_create from nfs_create_rpc_client+0x150/0x17c
> [   21.894812]  nfs_create_rpc_client from nfs4_alloc_client+0x360/0x374
> [   21.901226]  nfs4_alloc_client from nfs_get_client+0x16c/0x3e8
> [   21.907030]  nfs_get_client from nfs4_set_client+0xfc/0x1a4
> [   21.912574]  nfs4_set_client from nfs4_create_server+0x11c/0x2fc
> [   21.918554]  nfs4_create_server from nfs4_try_get_tree+0x10/0x50
> [   21.924534]  nfs4_try_get_tree from vfs_get_tree+0x24/0xe4
> [   21.929993]  vfs_get_tree from path_mount+0x3e8/0xb04
> [   21.935019]  path_mount from sys_mount+0x20c/0x254
> [   21.939784]  sys_mount from ret_fast_syscall+0x0/0x1c
> [   21.944809] Exception stack(0xf0cf9fa8 to 0xf0cf9ff0)
> [   21.949837] 9fa0:                   0047ebe0 00479c64 0047e960
> 0047e9b8 0047e9c8 00000000
> [   21.957986] 9fc0: 0047ebe0 00479c64 b6f058c8 00000015 00466c08
> 00000010 00479c64 00466bfc
> [   21.966139] 9fe0: 00479e70 befb69b0 0045a708 b6dca610
> [   21.971245] irq event stamp: 0
> [   21.974188] hardirqs last  enabled at (0): [<00000000>] 0x0
> [   21.979736] hardirqs last disabled at (0): [<c012357c>]
> copy_process+0x810/0x1ffc
> [   21.987227] softirqs last  enabled at (0): [<c012357c>]
> copy_process+0x810/0x1ffc
> [   21.994679] softirqs last disabled at (0): [<00000000>] 0x0
> [   22.000264] ---[ end trace 0000000000000000 ]---
> [   22.004781] BUG: sleeping function called from invalid context at
> net/sunrpc/sched.c:953
> [   22.012876] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid:
> 154, name: mount.nfs4
> [   22.020936] preempt_count: 1, expected: 0
> [   22.024881] RCU nest depth: 0, expected: 0
> [   22.028955] INFO: lockdep is turned off.
> [   22.032889] CPU: 7 PID: 154 Comm: mount.nfs4 Tainted: G        W
>     6.4.0-00001-gfbb103bb8df0 #8
> [   22.042131] Hardware name: Samsung Exynos (Flattened Device Tree)
> [   22.048196]  unwind_backtrace from show_stack+0x10/0x14
> [   22.053393]  show_stack from dump_stack_lvl+0x58/0x70
> [   22.058417]  dump_stack_lvl from __might_resched+0x194/0x260
> [   22.064054]  __might_resched from __rpc_execute+0x118/0x764
> [   22.069596]  __rpc_execute from rpc_execute+0xc0/0x1d0
> [   22.074708]  rpc_execute from rpc_run_task+0x148/0x190
> [   22.079821]  rpc_run_task from rpc_create_xprt+0x1a4/0x284
> [   22.085281]  rpc_create_xprt from rpc_create+0xf8/0x254
> [   22.090483]  rpc_create from nfs_create_rpc_client+0x150/0x17c
> [   22.096286]  nfs_create_rpc_client from nfs4_alloc_client+0x360/0x374
> [   22.102700]  nfs4_alloc_client from nfs_get_client+0x16c/0x3e8
> [   22.108504]  nfs_get_client from nfs4_set_client+0xfc/0x1a4
> [   22.114050]  nfs4_set_client from nfs4_create_server+0x11c/0x2fc
> [   22.120029]  nfs4_create_server from nfs4_try_get_tree+0x10/0x50
> [   22.126009]  nfs4_try_get_tree from vfs_get_tree+0x24/0xe4
> [   22.131467]  vfs_get_tree from path_mount+0x3e8/0xb04
> [   22.136493]  path_mount from sys_mount+0x20c/0x254
> [   22.141258]  sys_mount from ret_fast_syscall+0x0/0x1c
> [   22.146284] Exception stack(0xf0cf9fa8 to 0xf0cf9ff0)
> [   22.151322] 9fa0:                   0047ebe0 00479c64 0047e960
> 0047e9b8 0047e9c8 00000000
> [   22.159461] 9fc0: 0047ebe0 00479c64 b6f058c8 00000015 00466c08
> 00000010 00479c64 00466bfc
> [   22.167606] 9fe0: 00479e70 befb69b0 0045a708 b6dca610
> [   22.172820] BUG: scheduling while atomic: mount.nfs4/154/0x00000002
> [   22.178871] INFO: lockdep is turned off.
> [   22.182803] Modules linked in:
> [   22.185798] CPU: 7 PID: 154 Comm: mount.nfs4 Tainted: G        W
>     6.4.0-00001-gfbb103bb8df0 #8
> [   22.195076] Hardware name: Samsung Exynos (Flattened Device Tree)
> [   22.201139]  unwind_backtrace from show_stack+0x10/0x14
> [   22.206337]  show_stack from dump_stack_lvl+0x58/0x70
> [   22.211365]  dump_stack_lvl from __schedule_bug+0x70/0x84
> [   22.216736]  __schedule_bug from __schedule+0x9c0/0xc80
> [   22.221936]  __schedule from schedule+0x58/0xf8
> [   22.226439]  schedule from schedule_timeout+0x134/0x200
> [   22.231641]  schedule_timeout from __wait_for_common+0xac/0x1f8
> [   22.237533]  __wait_for_common from
> wait_for_completion_killable+0x18/0x24
> [   22.244379]  wait_for_completion_killable from
> __kthread_create_on_node+0xe0/0x168
> [   22.251923]  __kthread_create_on_node from
> kthread_create_on_node+0x30/0x60
> [   22.258851]  kthread_create_on_node from svc_set_num_threads+0x1c8/0x420
> [   22.265525]  svc_set_num_threads from nfs_callback_up+0x150/0x3c0
> [   22.271597]  nfs_callback_up from nfs4_init_client+0x98/0x144
> [   22.277306]  nfs4_init_client from nfs4_set_client+0xfc/0x1a4
> [   22.283026]  nfs4_set_client from nfs4_create_server+0x11c/0x2fc
> [   22.289005]  nfs4_create_server from nfs4_try_get_tree+0x10/0x50
> [   22.294985]  nfs4_try_get_tree from vfs_get_tree+0x24/0xe4
> [   22.300444]  vfs_get_tree from path_mount+0x3e8/0xb04
> [   22.305468]  path_mount from sys_mount+0x20c/0x254
> [   22.310249]  sys_mount from ret_fast_syscall+0x0/0x1c
> [   22.315261] Exception stack(0xf0cf9fa8 to 0xf0cf9ff0)
> [   22.320300] 9fa0:                   0047ebe0 00479c64 0047e960
> 0047e9b8 0047e9c8 00000000
> [   22.328438] 9fc0: 0047ebe0 00479c64 b6f058c8 00000015 00466c08
> 00000010 00479c64 00466bfc
> [   22.336582] 9fe0: 00479e70 befb69b0 0045a708 b6dca610
> :: running cleanup hook [udev]
> [   26.235349] systemd[1]: System time before build time, advancing clock.
> [   26.435536] systemd[1]: systemd 253.4-1-arch running in system mode
> (+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS
> +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD
> +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2
> +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT
> default-hierarchy=unified)
> [   26.466749] systemd[1]: Detected architecture arm.
>
>
>
> Best regards,
> Krzysztof
>

[-- Attachment #2: v4-0003-NFSv4.2-Rework-scratch-handling-for-READ_PLUS-aga.patch --]
[-- Type: text/x-patch, Size: 4620 bytes --]

From 19419202420ee47183c4e81118d17dfaf08bdea9 Mon Sep 17 00:00:00 2001
From: Anna Schumaker <Anna.Schumaker@Netapp.com>
Date: Fri, 9 Jun 2023 15:26:25 -0400
Subject: [PATCH v4 3/4] NFSv4.2: Rework scratch handling for READ_PLUS (again)

I found that the read code might send multiple requests using the same
nfs_pgio_header, but nfs4_proc_read_setup() is only called once. This is
how we ended up occasionally double-freeing the scratch buffer, but also
means we set a NULL pointer but non-zero length to the xdr scratch
buffer. This results in an oops the first time decoding needs to copy
something to scratch, which frequently happens when decoding READ_PLUS
hole segments.

I fix this by moving scratch handling into the pageio read code. I
provide a function to allocate scratch space for decoding read replies,
and free the scratch buffer when the nfs_pgio_header is freed.

Fixes: fbd2a05f29a9 (NFSv4.2: Rework scratch handling for READ_PLUS)
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
---
 fs/nfs/internal.h |  1 +
 fs/nfs/nfs42.h    |  1 +
 fs/nfs/nfs42xdr.c |  2 +-
 fs/nfs/nfs4proc.c | 13 +------------
 fs/nfs/read.c     | 10 ++++++++++
 5 files changed, 14 insertions(+), 13 deletions(-)

diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index 3cc027d3bd58..1607c23f68d4 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -489,6 +489,7 @@ extern const struct nfs_pgio_completion_ops nfs_async_read_completion_ops;
 extern void nfs_pageio_init_read(struct nfs_pageio_descriptor *pgio,
 			struct inode *inode, bool force_mds,
 			const struct nfs_pgio_completion_ops *compl_ops);
+extern bool nfs_read_alloc_scratch(struct nfs_pgio_header *hdr, size_t size);
 extern int nfs_read_add_folio(struct nfs_pageio_descriptor *pgio,
 			       struct nfs_open_context *ctx,
 			       struct folio *folio);
diff --git a/fs/nfs/nfs42.h b/fs/nfs/nfs42.h
index 0fe5aacbcfdf..b59876b01a1e 100644
--- a/fs/nfs/nfs42.h
+++ b/fs/nfs/nfs42.h
@@ -13,6 +13,7 @@
  * more? Need to consider not to pre-alloc too much for a compound.
  */
 #define PNFS_LAYOUTSTATS_MAXDEV (4)
+#define READ_PLUS_SCRATCH_SIZE (16)
 
 /* nfs4.2proc.c */
 #ifdef CONFIG_NFS_V4_2
diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index 75765382cc0e..20aa5e746497 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -1351,7 +1351,7 @@ static int nfs4_xdr_dec_read_plus(struct rpc_rqst *rqstp,
 	struct compound_hdr hdr;
 	int status;
 
-	xdr_set_scratch_buffer(xdr, res->scratch, sizeof(res->scratch));
+	xdr_set_scratch_buffer(xdr, res->scratch, READ_PLUS_SCRATCH_SIZE);
 
 	status = decode_compound_hdr(xdr, &hdr);
 	if (status)
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index d3665390c4cb..73dc8a793ae9 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -5437,18 +5437,8 @@ static bool nfs4_read_plus_not_supported(struct rpc_task *task,
 	return false;
 }
 
-static inline void nfs4_read_plus_scratch_free(struct nfs_pgio_header *hdr)
-{
-	if (hdr->res.scratch) {
-		kfree(hdr->res.scratch);
-		hdr->res.scratch = NULL;
-	}
-}
-
 static int nfs4_read_done(struct rpc_task *task, struct nfs_pgio_header *hdr)
 {
-	nfs4_read_plus_scratch_free(hdr);
-
 	if (!nfs4_sequence_done(task, &hdr->res.seq_res))
 		return -EAGAIN;
 	if (nfs4_read_stateid_changed(task, &hdr->args))
@@ -5468,8 +5458,7 @@ static bool nfs42_read_plus_support(struct nfs_pgio_header *hdr,
 	/* Note: We don't use READ_PLUS with pNFS yet */
 	if (nfs_server_capable(hdr->inode, NFS_CAP_READ_PLUS) && !hdr->ds_clp) {
 		msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ_PLUS];
-		hdr->res.scratch = kmalloc(32, GFP_KERNEL);
-		return hdr->res.scratch != NULL;
+		return nfs_read_alloc_scratch(hdr, READ_PLUS_SCRATCH_SIZE);
 	}
 	return false;
 }
diff --git a/fs/nfs/read.c b/fs/nfs/read.c
index f71eeee67e20..7dc21a48e3e7 100644
--- a/fs/nfs/read.c
+++ b/fs/nfs/read.c
@@ -47,6 +47,8 @@ static struct nfs_pgio_header *nfs_readhdr_alloc(void)
 
 static void nfs_readhdr_free(struct nfs_pgio_header *rhdr)
 {
+	if (rhdr->res.scratch != NULL)
+		kfree(rhdr->res.scratch);
 	kmem_cache_free(nfs_rdata_cachep, rhdr);
 }
 
@@ -108,6 +110,14 @@ void nfs_pageio_reset_read_mds(struct nfs_pageio_descriptor *pgio)
 }
 EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
+bool nfs_read_alloc_scratch(struct nfs_pgio_header *hdr, size_t size)
+{
+	WARN_ON(hdr->res.scratch != NULL);
+	hdr->res.scratch = kmalloc(size, GFP_KERNEL);
+	return hdr->res.scratch != NULL;
+}
+EXPORT_SYMBOL_GPL(nfs_read_alloc_scratch);
+
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
 	struct folio *folio = nfs_page_to_folio(req);
-- 
2.41.0


[-- Attachment #3: v4-0002-NFSv4.2-Fix-READ_PLUS-size-calculations.patch --]
[-- Type: text/x-patch, Size: 2023 bytes --]

From ec97f469ce6560ae695a94addfbaeb6d468913bb Mon Sep 17 00:00:00 2001
From: Anna Schumaker <Anna.Schumaker@Netapp.com>
Date: Wed, 31 May 2023 17:02:54 -0400
Subject: [PATCH v4 2/4] NFSv4.2: Fix READ_PLUS size calculations

I bump the decode_read_plus_maxsz to account for hole segments, but I
need to subtract out this increase when calling
rpc_prepare_reply_pages() so the common case of single data segment
replies can be directly placed into the xdr pages without needing to be
shifted around.

Reported-by: Chuck Lever <chuck.lever@oracle.com>
Fixes: d3b00a802c845 ("NFS: Replace the READ_PLUS decoding code")
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
---
 fs/nfs/nfs42xdr.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index ef3b150970ff..75765382cc0e 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -51,10 +51,16 @@
 					(1 /* data_content4 */ + \
 					 2 /* data_info4.di_offset */ + \
 					 1 /* data_info4.di_length */)
+#define NFS42_READ_PLUS_HOLE_SEGMENT_SIZE \
+					(1 /* data_content4 */ + \
+					 2 /* data_info4.di_offset */ + \
+					 2 /* data_info4.di_length */)
+#define READ_PLUS_SEGMENT_SIZE_DIFF	(NFS42_READ_PLUS_HOLE_SEGMENT_SIZE - \
+					 NFS42_READ_PLUS_DATA_SEGMENT_SIZE)
 #define decode_read_plus_maxsz		(op_decode_hdr_maxsz + \
 					 1 /* rpr_eof */ + \
 					 1 /* rpr_contents count */ + \
-					 NFS42_READ_PLUS_DATA_SEGMENT_SIZE)
+					 NFS42_READ_PLUS_HOLE_SEGMENT_SIZE)
 #define encode_seek_maxsz		(op_encode_hdr_maxsz + \
 					 encode_stateid_maxsz + \
 					 2 /* offset */ + \
@@ -781,8 +787,8 @@ static void nfs4_xdr_enc_read_plus(struct rpc_rqst *req,
 	encode_putfh(xdr, args->fh, &hdr);
 	encode_read_plus(xdr, args, &hdr);
 
-	rpc_prepare_reply_pages(req, args->pages, args->pgbase,
-				args->count, hdr.replen);
+	rpc_prepare_reply_pages(req, args->pages, args->pgbase, args->count,
+				hdr.replen - READ_PLUS_SEGMENT_SIZE_DIFF);
 	encode_nops(&hdr);
 }
 
-- 
2.41.0


[-- Attachment #4: v4-0001-NFSv4.2-Fix-READ_PLUS-smatch-warnings.patch --]
[-- Type: text/x-patch, Size: 1492 bytes --]

From e33f2202055b50815990134c80b4680bb80f819e Mon Sep 17 00:00:00 2001
From: Anna Schumaker <Anna.Schumaker@Netapp.com>
Date: Wed, 24 May 2023 17:27:08 -0400
Subject: [PATCH v4 1/4] NFSv4.2: Fix READ_PLUS smatch warnings

Smatch reports:
  fs/nfs/nfs42xdr.c:1131 decode_read_plus() warn: missing error code? 'status'

Which Dan suggests to fix by doing a hardcoded "return 0" from the
"if (segments == 0)" check.

Additionally, smatch reports that the "status = -EIO" assignment is not
used. This patch addresses both these issues.

Reported-by: kernel test robot <lkp@intel.com>
Reported-by: Dan Carpenter <error27@gmail.com>
Closes: https://lore.kernel.org/r/202305222209.6l5VM2lL-lkp@intel.com/
Fixes: d3b00a802c845 ("NFS: Replace the READ_PLUS decoding code")
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
---
 fs/nfs/nfs42xdr.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index a6df815a140c..ef3b150970ff 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -1136,13 +1136,12 @@ static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res *res)
 	res->eof = be32_to_cpup(p++);
 	segments = be32_to_cpup(p++);
 	if (segments == 0)
-		return status;
+		return 0;
 
 	segs = kmalloc_array(segments, sizeof(*segs), GFP_KERNEL);
 	if (!segs)
 		return -ENOMEM;
 
-	status = -EIO;
 	for (i = 0; i < segments; i++) {
 		status = decode_read_plus_segment(xdr, &segs[i]);
 		if (status < 0)
-- 
2.41.0


[-- Attachment #5: v4-0004-SUNRPC-kmap-the-xdr-pages-during-decode.patch --]
[-- Type: text/x-patch, Size: 3621 bytes --]

From 4f322029488f9721836de3a478889c7c523e1f95 Mon Sep 17 00:00:00 2001
From: Anna Schumaker <Anna.Schumaker@Netapp.com>
Date: Fri, 23 Jun 2023 11:43:14 -0400
Subject: [PATCH v4 4/4] SUNRPC: kmap() the xdr pages during decode

If the pages are in HIGHMEM then we need to make sure they're mapped
before trying to read data off of them, otherwise we could end up with a
NULL pointer dereference.

Reported-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
---
 include/linux/sunrpc/xdr.h |  2 ++
 net/sunrpc/clnt.c          |  1 +
 net/sunrpc/xdr.c           | 17 ++++++++++++++++-
 3 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/include/linux/sunrpc/xdr.h b/include/linux/sunrpc/xdr.h
index d917618a3058..f562aab468f5 100644
--- a/include/linux/sunrpc/xdr.h
+++ b/include/linux/sunrpc/xdr.h
@@ -228,6 +228,7 @@ struct xdr_stream {
 	struct kvec *iov;	/* pointer to the current kvec */
 	struct kvec scratch;	/* Scratch buffer */
 	struct page **page_ptr;	/* pointer to the current page */
+	void *page_kaddr;	/* kmapped address of the current page */
 	unsigned int nwords;	/* Remaining decode buffer length */
 
 	struct rpc_rqst *rqst;	/* For debugging */
@@ -254,6 +255,7 @@ extern void xdr_truncate_decode(struct xdr_stream *xdr, size_t len);
 extern int xdr_restrict_buflen(struct xdr_stream *xdr, int newbuflen);
 extern void xdr_write_pages(struct xdr_stream *xdr, struct page **pages,
 		unsigned int base, unsigned int len);
+extern void xdr_stream_unmap_current_page(struct xdr_stream *xdr);
 extern unsigned int xdr_stream_pos(const struct xdr_stream *xdr);
 extern unsigned int xdr_page_pos(const struct xdr_stream *xdr);
 extern void xdr_init_decode(struct xdr_stream *xdr, struct xdr_buf *buf,
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index d2ee56634308..3b7e676d8935 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -2590,6 +2590,7 @@ call_decode(struct rpc_task *task)
 	case 0:
 		task->tk_action = rpc_exit_task;
 		task->tk_status = rpcauth_unwrap_resp(task, &xdr);
+		xdr_stream_unmap_current_page(&xdr);
 		return;
 	case -EAGAIN:
 		task->tk_status = 0;
diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
index 391b336d97de..fb5203337608 100644
--- a/net/sunrpc/xdr.c
+++ b/net/sunrpc/xdr.c
@@ -1308,6 +1308,14 @@ static unsigned int xdr_set_tail_base(struct xdr_stream *xdr,
 	return xdr_set_iov(xdr, buf->tail, base, len);
 }
 
+void xdr_stream_unmap_current_page(struct xdr_stream *xdr)
+{
+	if (xdr->page_kaddr) {
+		kunmap_local(xdr->page_kaddr);
+		xdr->page_kaddr = NULL;
+	}
+}
+
 static unsigned int xdr_set_page_base(struct xdr_stream *xdr,
 				      unsigned int base, unsigned int len)
 {
@@ -1325,12 +1333,18 @@ static unsigned int xdr_set_page_base(struct xdr_stream *xdr,
 	if (len > maxlen)
 		len = maxlen;
 
+	xdr_stream_unmap_current_page(xdr);
 	xdr_stream_page_set_pos(xdr, base);
 	base += xdr->buf->page_base;
 
 	pgnr = base >> PAGE_SHIFT;
 	xdr->page_ptr = &xdr->buf->pages[pgnr];
-	kaddr = page_address(*xdr->page_ptr);
+
+	if (PageHighMem(*xdr->page_ptr)) {
+		xdr->page_kaddr = kmap_local_page(*xdr->page_ptr);
+		kaddr = xdr->page_kaddr;
+	} else
+		kaddr = page_address(*xdr->page_ptr);
 
 	pgoff = base & ~PAGE_MASK;
 	xdr->p = (__be32*)(kaddr + pgoff);
@@ -1384,6 +1398,7 @@ void xdr_init_decode(struct xdr_stream *xdr, struct xdr_buf *buf, __be32 *p,
 		     struct rpc_rqst *rqst)
 {
 	xdr->buf = buf;
+	xdr->page_kaddr = NULL;
 	xdr_reset_scratch_buffer(xdr);
 	xdr->nwords = XDR_QUADLEN(buf->len);
 	if (xdr_set_iov(xdr, buf->head, 0, buf->len) == 0 &&
-- 
2.41.0