Linux-RDMA Archive on lore.kernel.org
 help / color / Atom feed
From: Jason Gunthorpe <jgg@ziepe.ca>
To: Leon Romanovsky <leon@kernel.org>
Cc: Doug Ledford <dledford@redhat.com>,
	Leon Romanovsky <leonro@mellanox.com>,
	RDMA mailing list <linux-rdma@vger.kernel.org>,
	Daniel Jurgens <danielj@mellanox.com>,
	Erez Shitrit <erezsh@mellanox.com>,
	Maor Gottlieb <maorg@mellanox.com>,
	Michael Guralnik <michaelgur@mellanox.com>,
	Moni Shoua <monis@mellanox.com>,
	Parav Pandit <parav@mellanox.com>,
	Sean Hefty <sean.hefty@intel.com>,
	Valentine Fatiev <valentinef@mellanox.com>,
	Yishai Hadas <yishaih@mellanox.com>,
	Yonatan Cohen <yonatanc@mellanox.com>,
	Zhu Yanjun <yanjunz@mellanox.com>
Subject: Re: [PATCH rdma-rc 8/9] IB/umad: Fix kernel crash while unloading ib_umad
Date: Thu, 13 Feb 2020 10:28:18 -0400
Message-ID: <20200213142818.GA16120@ziepe.ca> (raw)
In-Reply-To: <20200212072635.682689-9-leon@kernel.org>

On Wed, Feb 12, 2020 at 09:26:34AM +0200, Leon Romanovsky wrote:
> From: Yonatan Cohen <yonatanc@mellanox.com>
> 
> When unloading ib_umad, remove ibdev sys file 1st before
> port removal to prevent kernel oops.
> 
> ib_mad's method ibdev_show() might access a umad port
> whoes ibdev field has already been NULLed when rmmod ib_umad
> was issued from another shell.
> 
> Consider this scenario
> 	         shell-1            	shell-2
> 	        rmmod ib_mod    	cat /sys/devices/../ibdev
> 	            |           		|
> 	        ib_umad_kill_port()        ibdev_show()
> 	     port->ib_dev = NULL	dev_name(port->ib_dev)
> 
> kernel stack
> PF: error_code(0x0000) - not-present page
> Oops: 0000 [#1] SMP DEBUG_PAGEALLOC PTI
> RIP: 0010:ibdev_show+0x18/0x50 [ib_umad]
> RSP: 0018:ffffc9000097fe40 EFLAGS: 00010282
> RAX: 0000000000000000 RBX: ffffffffa0441120 RCX: ffff8881df514000
> RDX: ffff8881df514000 RSI: ffffffffa0441120 RDI: ffff8881df1e8870
> RBP: ffffffff81caf000 R08: ffff8881df1e8870 R09: 0000000000000000
> R10: 0000000000001000 R11: 0000000000000003 R12: ffff88822f550b40
> R13: 0000000000000001 R14: ffffc9000097ff08 R15: ffff8882238bad58
> FS:  00007f1437ff3740(0000) GS:ffff888236940000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000004e8 CR3: 00000001e0dfc001 CR4: 00000000001606e0
> Call Trace:
>  dev_attr_show+0x15/0x50
>  sysfs_kf_seq_show+0xb8/0x1a0
>  seq_read+0x12d/0x350
>  vfs_read+0x89/0x140
>  ksys_read+0x55/0xd0
>  do_syscall_64+0x55/0x1b0
>  entry_SYSCALL_64_after_hwframe+0x44/0xa9:
> 
> Fixes: e9dd5daf884c ("IB/umad: Refactor code to use cdev_device_add()")

This is the wrong fixes line, this ordering change was actually
deliberately done:

commit cf7ad3030271c55a7119a8c2162563e3f6e93879
Author: Parav Pandit <parav@mellanox.com>
Date:   Fri Dec 21 16:19:24 2018 +0200

    IB/umad: Avoid destroying device while it is accessed
    
    ib_umad_reg_agent2() and ib_umad_reg_agent() access the device name in
    dev_notice(), while concurrently, ib_umad_kill_port() can destroy the
    device using device_destroy().
    
            cpu-0                               cpu-1
            -----                               -----
        ib_umad_ioctl()
            [...]                            ib_umad_kill_port()
                                                  device_destroy(dev)
    
            ib_umad_reg_agent()
                dev_notice(dev)

The mistake in the above was to move the device_dstroy() down, not
split it into device_del() above and put_device() below.

Now that is already split we are OK.

Jason

  reply index

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-02-12  7:26 [PATCH rdma-rc 0/9] Fixes for v5.6 Leon Romanovsky
2020-02-12  7:26 ` [PATCH rdma-rc 1/9] RDMA/ucma: Mask QPN to be 24 bits according to IBTA Leon Romanovsky
2020-02-12  7:26 ` [PATCH rdma-rc 2/9] RDMA/core: Fix protection fault in get_pkey_idx_qp_list Leon Romanovsky
2020-02-12  8:01   ` Leon Romanovsky
2020-02-12  8:06   ` Leon Romanovsky
2020-02-12  7:26 ` [PATCH rdma-rc 3/9] Revert "RDMA/cma: Simplify rdma_resolve_addr() error flow" Leon Romanovsky
2020-02-13 13:30   ` Jason Gunthorpe
2020-02-14  3:11     ` Parav Pandit
2020-02-14 14:08       ` Jason Gunthorpe
2020-02-14 14:48         ` Parav Pandit
2020-02-19 20:40   ` Jason Gunthorpe
2020-02-12  7:26 ` [PATCH rdma-rc 4/9] IB/ipoib: Fix double free of skb in case of multicast traffic in CM mode Leon Romanovsky
2020-02-13 15:37   ` Jason Gunthorpe
2020-02-13 18:10     ` Leon Romanovsky
2020-02-13 18:26       ` Jason Gunthorpe
2020-02-13 18:36         ` Leon Romanovsky
2020-02-13 19:09           ` Jason Gunthorpe
2020-02-12  7:26 ` [PATCH rdma-rc 5/9] RDMA/core: Add missing list deletion on freeing event queue Leon Romanovsky
2020-02-12  7:26 ` [PATCH rdma-rc 6/9] IB/mlx5: Fix async events cleanup flows Leon Romanovsky
2020-02-12  7:26 ` [PATCH rdma-rc 7/9] RDMA/rxe: Fix soft lockup problem due to using tasklets in softirq Leon Romanovsky
2020-02-12  7:26 ` [PATCH rdma-rc 8/9] IB/umad: Fix kernel crash while unloading ib_umad Leon Romanovsky
2020-02-13 14:28   ` Jason Gunthorpe [this message]
2020-02-13 18:03     ` Leon Romanovsky
2020-02-12  7:26 ` [PATCH rdma-rc 9/9] RDMA/mlx5: Prevent overflow in mmap offset calculations Leon Romanovsky
2020-02-13 18:03 ` [PATCH rdma-rc 0/9] Fixes for v5.6 Jason Gunthorpe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200213142818.GA16120@ziepe.ca \
    --to=jgg@ziepe.ca \
    --cc=danielj@mellanox.com \
    --cc=dledford@redhat.com \
    --cc=erezsh@mellanox.com \
    --cc=leon@kernel.org \
    --cc=leonro@mellanox.com \
    --cc=linux-rdma@vger.kernel.org \
    --cc=maorg@mellanox.com \
    --cc=michaelgur@mellanox.com \
    --cc=monis@mellanox.com \
    --cc=parav@mellanox.com \
    --cc=sean.hefty@intel.com \
    --cc=valentinef@mellanox.com \
    --cc=yanjunz@mellanox.com \
    --cc=yishaih@mellanox.com \
    --cc=yonatanc@mellanox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-RDMA Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rdma/0 linux-rdma/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rdma linux-rdma/ https://lore.kernel.org/linux-rdma \
		linux-rdma@vger.kernel.org
	public-inbox-index linux-rdma

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rdma


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git