RE: Race condition in Kernel

From: Gulam Mohamed <gulam.mohamed@oracle.com>
To: Ming Lei <ming.lei@redhat.com>
Cc: "hch@infradead.org" <hch@infradead.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
	Junxiao Bi <junxiao.bi@oracle.com>,
	Martin Petersen <martin.petersen@oracle.com>,
	"axboe@kernel.dk" <axboe@kernel.dk>
Subject: RE: Race condition in Kernel
Date: Thu, 1 Apr 2021 16:27:37 +0000	[thread overview]
Message-ID: <CO1PR10MB4563673A20326DF5170CF60F987B9@CO1PR10MB4563.namprd10.prod.outlook.com> (raw)
In-Reply-To: <YFvrZUzmdLpj7E4F@T590>

Hi Ming,

      Thanks for taking a look into this. Can you please see my inline comments in below mail?

Regards,
Gulam Mohamed.

-----Original Message-----
From: Ming Lei <ming.lei@redhat.com> 
Sent: Thursday, March 25, 2021 7:16 AM
To: Gulam Mohamed <gulam.mohamed@oracle.com>
Cc: hch@infradead.org; linux-kernel@vger.kernel.org; linux-block@vger.kernel.org; Junxiao Bi <junxiao.bi@oracle.com>; Martin Petersen <martin.petersen@oracle.com>; axboe@kernel.dk
Subject: Re: Race condition in Kernel

On Wed, Mar 24, 2021 at 12:37:03PM +0000, Gulam Mohamed wrote:
> Hi All,
> 
> We are facing a stale link (of the device) issue during the iscsi-logout process if we use parted command just before the iscsi logout. Here are the details:
> 	 	 
> As part of iscsi logout, the partitions and the disk will be removed. The parted command, used to list the partitions, will open the disk in RW mode which results in systemd-udevd re-reading the partitions. This will trigger the rescan partitions which will also delete and re-add the partitions. So, both iscsi logout processing and the parted (through systemd-udevd) will be involved in add/delete of partitions. In our case, the following sequence of operations happened (the iscsi device is /dev/sdb with partition sdb1):
> 	
> 	1. sdb1 was removed by PARTED
> 	2. kworker, as part of iscsi logout, couldn't remove sdb1 as it was already removed by PARTED
> 	3. sdb1 was added by parted

After kworker is started for logout, I guess all IOs are supposed to be failed at that time, so just wondering why 'sdb1' is still added by parted(systemd-udev)? 
ioctl(BLKRRPART) needs to read partition table for adding back partitions, if IOs are failed by iscsi logout, I guess the issue can be avoided too?

[GULAM]: Yes, the ioctl(BLKRRPART) reads the partition table for adding back the partitions. I kept a printk in the code just after the partition table is read. Noticed that the partition table was read before the iscsi-logout kworker started the logout processing.
                   Following are the logs for your reference:

 Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: ORA:: Calling sysfs_delete_link() for dev: sdb3 command: systemd-udevd		<== sdb3 Removed by PARTED 
Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: ORA:: rescan_partitions() Read Complete to the disk: sdb command: systemd-udevd   <== Reading sdb completed, before iscsi-logout worker started
Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: ORA:: Calling sysfs_delete_link() for dev: 3:0:0:0 command: kworker/u16:3
Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: sdb: sdb1 sdb2 sdb3
Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: ORA:: device: 'sdb3': device_add command: systemd-udevd		<== sdb3 Added by PARTED 
Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: ORA:: Calling sysfs_delete_link() for dev: 8:16 command: kworker/u16:3
Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: ORA:: Calling sysfs_delete_link() for dev: sdb command: kworker/u16:3	<== sdb Removed by iscsi 
Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: scsi 3:0:0:0: alua: Detached

--
Ming