From: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
To: Anna Schumaker <schumaker.anna@gmail.com>
Cc: Trond Myklebust <trondmy@hammerspace.com>,
Anna Schumaker <Anna.Schumaker@netapp.com>,
linux-nfs <linux-nfs@vger.kernel.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
"regressions@lists.linux.dev" <regressions@lists.linux.dev>
Subject: Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47)
Date: Mon, 23 Jan 2023 08:58:43 +0100 [thread overview]
Message-ID: <2add1769-1458-b185-bc78-6d573f61b6fc@linaro.org> (raw)
In-Reply-To: <CAFX2Jf=5X3zyZEWQmD6Rg9jQAD7ccDbae5LQCwrAyPCVVoFumg@mail.gmail.com>
On 22/01/2023 23:25, Anna Schumaker wrote:
> Hi Krzysztof,
>
> On Mon, Jan 9, 2023 at 3:46 AM Krzysztof Kozlowski
> <krzysztof.kozlowski@linaro.org> wrote:
>>
>> On 09/01/2023 09:14, Krzysztof Kozlowski wrote:
>>> On 08/01/2023 18:09, Trond Myklebust wrote:
>>>> Hi Krzysztof,
>>>>
>>>>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>>>>
>>>>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
>>>>>
>>>>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>>>>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>>>>>> when mounting NFS root on NFSv4 client:
>>>>>>
>>>>>> [ 25.739003] systemd[1]: Set hostname to <odroidhc1>.
>>>>>> [ 25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>>> argument
>>>>>> [ 26.199478] 8<--- cut here ---
>>>>>> [ 26.201366] Unable to handle kernel NULL pointer dereference at
>>>>>> virtual address 00000004
>>>>>> ...
>>>>>> [ 26.555522] mmiocpy from xdr_inline_decode+0xec/0x16c
>>>>>> [ 26.560628] xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
>>>>>> [ 26.567130] nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>>>
>>>>>> Full OOPS attached. Full log available here:
>>>>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
>>>>>>
>>>>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
>>>>>> not the cause, but rather making it default caused the regression.
>>>>>>
>>>>>> I did not make the bisect yet which commit introduced it, if every
>>>>>> config includes NFS_V4_2_READ_PLUS.
>>>>>
>>>>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
>>>>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
>>>>>
>>>>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
>>>>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>> Date: Thu Jul 21 14:21:34 2022 -0400
>>>>>
>>>>> NFS: Replace the READ_PLUS decoding code
>>>>>
>>>>> We now take a 2-step process that allows us to place data and hole
>>>>> segments directly at their final position in the xdr_stream without
>>>>> needing to do a bunch of redundant copies to expand holes. Due to the
>>>>> variable lengths of each segment, the xdr metadata might cross page
>>>>> boundaries which I account for by setting a small scratch buffer so
>>>>> xdr_inline_decode() won't fail.
>>>>>
>>>>> Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>>>> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
>>>>>
>>>>> With a trace:
>>>>> [ 25.898462] systemd[1]: Set hostname to <odroidhc1>.
>>>>> [ 25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>>>> argument
>>>>> [ 25.986237] random: crng init done
>>>>> [ 26.264564] 8<--- cut here ---
>>>>> [ 26.266823] Unable to handle kernel NULL pointer dereference at
>>>>> virtual address 00000fe8
>>>>> ...
>>>>> [ 26.597263] nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>>> [ 26.603222] call_decode from __rpc_execute+0xd0/0x890
>>>>> [ 26.608328] __rpc_execute from rpc_async_schedule+0x1c/0x34
>>>>> [ 26.613960] rpc_async_schedule from process_one_work+0x294/0x790
>>>>> [ 26.620030] process_one_work from worker_thread+0x54/0x518
>>>>> [ 26.625570] worker_thread from kthread+0xf4/0x128
>>>>> [ 26.630336] kthread from ret_from_fork+0x14/0x2c
>>>>>
>>>>
>>>> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?
>>>
>>> I would say that buggy server should not cause NULL pointer dereferences
>>> on the client. Otherwise this is a perfect recipe for a rogue server in
>>> the network to start crashing clients and running exploits... Imagine a
>>> compromised machine (through some other means) in a local company
>>> network running now a server with NFS share "HR salary data" or "HR
>>> planned layoffs", where unsuspected people in that network access it
>>> leading to exploit of NFS code on their side...
>>>
>>> Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH
>>>
>>> Which points that it is not latest stable, so anyway I need to update.
>>
>> I updated the kernel to 5.15.84-3-rpi-ARCH which is pretty close to
>> latest stable and I can reproduce the issue. Therefore:
>> 1. It is visible on two stable (one new, one old) kernels on the server,
>> 2. Buggy or rogue server should not cause NULL pointer on remote devices...
>
> I'm still working on this issue, but I haven't been able to reproduce
> it with my setup at all yet. I was hoping I could ask you a couple of
> questions?
>
> - Are both client and server run on a Raspberry Pi 3?
No, server is Rpi3, client is Odroid HC1.
> - Is there a specific workload that triggers the bug, or is it just
> during boot when using nfsroot?
No specific workload - mounting system with nfsroot. You have there full
logs as it is reproducible every time.
> - Would it be possible to get a Wireshark network trace of the crash
> (you'll have to run this on the server due to nfsroot).
I'll check.
> - Can you share your export options from /etc/exports and the mount
> options that the client uses?
exports are the same since like 5 years or more:
/srv/nfs/odroidhc1 192.168.1.0/24(rw,async,no_root_squash,no_subtree_check)
/srv/nfs/odroidhc1-home
192.168.1.0/24(rw,async,no_root_squash,no_subtree_check)
>
> Thanks,
> Anna
>
>>
>> Best regards,
>> Krzysztof
>>
Best regards,
Krzysztof
next prev parent reply other threads:[~2023-01-23 7:58 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-07 15:44 Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS (commit 7fd461c47) Krzysztof Kozlowski
2023-01-08 8:52 ` Linux kernel regression tracking (#adding)
2023-02-18 4:42 ` Linux regression tracking #update (Thorsten Leemhuis)
2023-02-18 15:09 ` Krzysztof Kozlowski
2023-02-21 14:24 ` Linux regression tracking (Thorsten Leemhuis)
2023-01-08 13:25 ` Krzysztof Kozlowski
2023-01-08 17:09 ` Trond Myklebust
2023-01-09 8:14 ` Krzysztof Kozlowski
2023-01-09 8:42 ` Krzysztof Kozlowski
2023-01-09 14:44 ` Trond Myklebust
2023-01-09 15:07 ` Chuck Lever III
2023-01-09 15:26 ` Anna Schumaker
2023-01-09 18:25 ` Chuck Lever III
2023-01-09 15:38 ` Trond Myklebust
2023-01-09 17:11 ` Chuck Lever III
2023-01-09 17:20 ` Chuck Lever III
2023-01-22 22:25 ` Anna Schumaker
2023-01-23 7:58 ` Krzysztof Kozlowski [this message]
2023-02-09 18:22 ` Anna Schumaker
2023-02-10 8:41 ` Krzysztof Kozlowski
2023-02-10 8:53 ` Krzysztof Kozlowski
2023-02-10 20:55 ` Anna Schumaker
2023-02-11 11:23 ` Krzysztof Kozlowski
2023-02-12 14:05 ` Anna Schumaker
2023-02-14 11:02 ` Krzysztof Kozlowski
2023-02-16 17:40 ` Olga Kornievskaia
2023-03-06 17:09 ` Anna Schumaker
2023-04-04 1:01 ` Olga Kornievskaia
2023-04-04 5:29 ` Krzysztof Kozlowski
2023-06-10 10:15 ` Krzysztof Kozlowski
2023-06-14 20:55 ` Anna Schumaker
2023-06-15 8:52 ` Krzysztof Kozlowski
2023-06-15 8:55 ` Krzysztof Kozlowski
2023-06-15 13:01 ` Anna Schumaker
2023-06-15 17:04 ` Anna Schumaker
2023-06-15 17:16 ` Anna Schumaker
2023-06-15 19:38 ` Anna Schumaker
2023-06-17 10:09 ` Krzysztof Kozlowski
2023-06-21 12:49 ` Anna Schumaker
2023-06-21 13:27 ` Krzysztof Kozlowski
2023-06-23 17:59 ` Anna Schumaker
2023-06-26 10:28 ` Krzysztof Kozlowski
2023-06-27 19:08 ` Anna Schumaker
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2add1769-1458-b185-bc78-6d573f61b6fc@linaro.org \
--to=krzysztof.kozlowski@linaro.org \
--cc=Anna.Schumaker@netapp.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=regressions@lists.linux.dev \
--cc=schumaker.anna@gmail.com \
--cc=trondmy@hammerspace.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).