From: Stefan Majer <stefan.majer@gmail.com>
To: sagi grimberg <sagi@grimberg.me>
Cc: Keith Busch <kbusch@kernel.org>,
linux-nvme <linux-nvme@lists.infradead.org>
Subject: Re: null pointer dereference in nvme_tcp_io_work
Date: Fri, 27 Dec 2019 08:54:23 +0100 [thread overview]
Message-ID: <CADdPHGuJjpY6WNBw5hGFUKbJdrwM-oQ9A3xCy3e2O6fY4yzEPQ@mail.gmail.com> (raw)
In-Reply-To: <CAB5Wxwco3KD1e_nRGQ_mWAMa_2d-wP2-1Aao4ZXtDeVgFQQM_w@mail.gmail.com>
Hi,
no problem, i am also on vacation.
the issue is not reproducible in a pure bare metal environment, target
and host are physical machines.
The environment where it happens both machines are kvm based.
I first have to figure out howto gdb on the kernel crash, thats not my
daily jobs, so please be patient.
Greetings
Stefan
On Fri, Dec 27, 2019 at 8:49 AM sagi grimberg <sagi@grimberg.me> wrote:
>
> Hey,
>
> On vacation so not able to take a look right now, but can you provide a line info from gdb on the RIP line?
>
> Also, did you say that the issue is not reproducible when the host is on bare metal but only on kvm? ( You said the target, but I'm asking about the host).
>
> On Thu, Dec 26, 2019, 23:18 Stefan Majer <stefan.majer@gmail.com> wrote:
>>
>> Hi,
>>
>> i have to add that doing the same on bare metal does work without any problems.
>> I suspect that this is probably caused by the fact that in the above
>> example my target is a qemu-kvm machine with a emulated nvme device.
>> Greetings
>> Stefan
>>
>> On Thu, Dec 26, 2019 at 6:47 PM Keith Busch <kbusch@kernel.org> wrote:
>> >
>> > Adding Sagi.
>> >
>> > On Wed, Dec 25, 2019 at 11:06:17AM +0100, Stefan Majer wrote:
>> > > Hi,
>> > >
>> > > im trying to setup a nvme-over-tcp test environment with a qemu-kvm
>> > > based nvmet-tcp target based on ubuntu-19.10 and a ubuntu-19.10 host
>> > > with kernel 5.4.6 installed. Kernel was taken from
>> > > https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.4.6/ . Same Panic
>> > > occurs with ubuntu 19.10 kernel 5.3.x
>> > >
>> > > After setup the target i can discover and connect the exported nvme
>> > > device on the host with:
>> > > modprobe nvme
>> > > modprobe nvme-tcp
>> > > nvme discover -t tcp -a 192.168.22.1 -s 4420
>> > > nvme connect -t tcp -n nvmet-test -a 192.168.22.1 -s 4420
>> > >
>> > > No errors so far, but when i try to format the device with:
>> > >
>> > > mkfs.ext4 /dev/nvme0n1
>> > >
>> > > The kernel panics with:
>> > > Writing inode tables:
>> > > [ 692.651243] BUG: kernel NULL pointer dereference, address: 0000000000000008
>> > > [ 692.653158] #PF: supervisor read access in kernel mode
>> > > [ 692.653922] #PF: error_code(0x0000) - not-present page
>> > > [ 692.653922] PGD 0 P4D 0
>> > > [ 692.653922] Oops: 0000 [#1] SMP PTI
>> > > [ 692.653922] CPU: 0 PID: 224 Comm: kworker/0:1H Not tainted
>> > > 5.4.6-050406-generic #201912211140
>> > > [ 692.653922] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
>> > > BIOS 0.0.0 02/06/2015
>> > > [ 692.653922] Workqueue: nvme_tcp_wq nvme_tcp_io_work [nvme_tcp]
>> > > [ 692.653922] RIP: 0010:nvme_tcp_io_work+0x308/0x790 [nvme_tcp]
>> > > [ 692.653922] Code: 8b 86 98 00 00 00 83 f8 02 0f 85 6d fd ff ff 49
>> > > 8b 46 28 4d 89 f7 48 89 45 a8 49 8b 47 78 49 8b 57 68 45 8b 67 34 45
>> > > 2b 67 38 <8b> 58 08 8b 48 0c 4c 8b 28 48 29 d3 48 8d 34 11 4c 39 e3 48
>> > > 89 75
>> > > [ 692.653922] RSP: 0018:ffffa49a00447dd8 EFLAGS: 00010206
>> > > [ 692.653922] RAX: 0000000000000000 RBX: 0000000077bd3601 RCX: 0000000000000000
>> > > [ 692.653922] RDX: 0000000000000000 RSI: 0000000000000011 RDI: ffff9376781c0500
>> > > [ 692.653922] RBP: ffffa49a00447e60 R08: 0000000000001000 R09: 0000000005000809
>> > > [ 692.653922] R10: 0000000000000009 R11: 0000000000000000 R12: 0000000000001000
>> > > [ 692.653922] R13: 0000000000000048 R14: ffff9376781c04a0 R15: ffff9376781c04a0
>> > > [ 692.653922] FS: 0000000000000000(0000) GS:ffff93767f600000(0000)
>> > > knlGS:0000000000000000
>> > > [ 692.653922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> > > [ 692.653922] CR2: 0000000000000008 CR3: 000000007b488003 CR4: 0000000000360ef0
>> > > [ 692.653922] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> > > [ 692.653922] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> > > [ 692.653922] Call Trace:
>> > > [ 692.653922] process_one_work+0x1ec/0x3a0
>> > > [ 692.653922] worker_thread+0x4d/0x400
>> > > [ 692.653922] kthread+0x104/0x140
>> > > [ 692.653922] ? process_one_work+0x3a0/0x3a0
>> > > [ 692.653922] ? kthread_park+0x90/0x90
>> > > [ 692.653922] ret_from_fork+0x35/0x40
>> > > [ 692.653922] Modules linked in: binfmt_misc nvme_tcp nvme_fabrics
>> > > nvme nvme_core xt_conntrack xt_MASQUERADE nf_conntrack_netlink
>> > > nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat
>> > > nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c bpfilter
>> > > br_netfilter bridge stp llc overlay intel_rapl_msr intel_rapl_common
>> > > kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul
>> > > ghash_clmulni_intel aesni_intel nls_iso8859_1 crypto_simd cryptd
>> > > cirrus glue_helper drm_kms_helper drm input_leds fb_sys_fops joydev
>> > > serio_raw syscopyarea sysfillrect sysimgblt mac_hid qemu_fw_cfg
>> > > bonding sch_fq_codel ipmi_watchdog ipmi_devintf ipmi_msghandler
>> > > virtio_rng ip_tables x_tables autofs4 ahci psmouse virtio_net
>> > > net_failover failover libahci i2c_piix4 pata_acpi floppy
>> > > [ 692.653922] CR2: 0000000000000008
>> > > [ 692.653922] ---[ end trace d688c2c182feef87 ]---
>> > > [ 692.653922] RIP: 0010:nvme_tcp_io_work+0x308/0x790 [nvme_tcp]
>> > > [ 692.653922] Code: 8b 86 98 00 00 00 83 f8 02 0f 85 6d fd ff ff 49
>> > > 8b 46 28 4d 89 f7 48 89 45 a8 49 8b 47 78 49 8b 57 68 45 8b 67 34 45
>> > > 2b 67 38 <8b> 58 08 8b 48 0c 4c 8b 28 48 29 d3 48 8d 34 11 4c 39 e3 48
>> > > 89 75
>> > > [ 692.653922] RSP: 0018:ffffa49a00447dd8 EFLAGS: 00010206
>> > > [ 692.653922] RAX: 0000000000000000 RBX: 0000000077bd3601 RCX: 0000000000000000
>> > > [ 692.653922] RDX: 0000000000000000 RSI: 0000000000000011 RDI: ffff9376781c0500
>> > > [ 692.653922] RBP: ffffa49a00447e60 R08: 0000000000001000 R09: 0000000005000809
>> > > [ 692.653922] R10: 0000000000000009 R11: 0000000000000000 R12: 0000000000001000
>> > > [ 692.653922] R13: 0000000000000048 R14: ffff9376781c04a0 R15: ffff9376781c04a0
>> > > [ 692.653922] FS: 0000000000000000(0000) GS:ffff93767f600000(0000)
>> > > knlGS:0000000000000000
>> > > [ 692.653922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> > > [ 692.653922] CR2: 0000000000000008 CR3: 000000007b488003 CR4: 0000000000360ef0
>> > > [ 692.653922] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> > > [ 692.653922] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> > >
>> > >
>> > > Any help appreciated.
>> > >
>> > > Greetings
>> > >
>> > > --
>> > > Stefan Majer
>>
>>
>>
>> --
>> Stefan Majer
--
Stefan Majer
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
next prev parent reply other threads:[~2019-12-27 7:54 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-12-25 10:06 null pointer dereference in nvme_tcp_io_work Stefan Majer
2019-12-26 17:47 ` Keith Busch
2019-12-27 7:18 ` Stefan Majer
[not found] ` <CAB5Wxwco3KD1e_nRGQ_mWAMa_2d-wP2-1Aao4ZXtDeVgFQQM_w@mail.gmail.com>
2019-12-27 7:54 ` Stefan Majer [this message]
2019-12-28 17:49 ` Stefan Majer
2019-12-28 17:53 ` Stefan Majer
2020-01-07 15:41 ` Stefan Majer
2020-01-07 16:48 ` Nadolski, Edmund
2020-01-15 20:03 ` Sagi Grimberg
2020-01-16 6:28 ` Stefan Majer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CADdPHGuJjpY6WNBw5hGFUKbJdrwM-oQ9A3xCy3e2O6fY4yzEPQ@mail.gmail.com \
--to=stefan.majer@gmail.com \
--cc=kbusch@kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.