Re: More parallel atomic_open/d_splice_alias fun with NFS and possibly more FSes.

From: James Simmons <jsimmons@infradead.org>
To: Oleg Drokin <green@linuxhacker.ru>
Cc: Al Viro <viro@ZenIV.linux.org.uk>,
	Mailing List <linux-kernel@vger.kernel.org>,
	"<linux-fsdevel@vger.kernel.org>" <linux-fsdevel@vger.kernel.org>
Subject: Re: More parallel atomic_open/d_splice_alias fun with NFS and possibly more FSes.
Date: Sun, 10 Jul 2016 19:14:18 +0100 (BST)	[thread overview]
Message-ID: <alpine.LFD.2.20.1607101913170.6483@casper.infradead.org> (raw)
In-Reply-To: <88A6EDE1-10F4-48FA-BCDC-D507C3E22DFA@linuxhacker.ru>

> On Jul 4, 2016, at 10:25 PM, Al Viro wrote:
> 
> > BTW, could you take a look at
> > git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.git#sendmsg.lustre?
> > It's a bunch of simplifications that became possible once sendmsg()/recvmsg()
> > switched to iov_iter, stopped mangling the iovecs and went for predictable
> > behaviour re advancing the iterator.
> 
> Thanks, this looks good to me and passes my testing (on tcp).
> 
> +typedef struct bio_vec lnet_kiov_t;
> 
> This I guess we'll need to just get rid of all lnet_kiov_t usage, but that's
> something we can do ourselves, I guess.
> 
> Anyway, your patchset is based on old tree that no longer applies cleanly,
> I rebased it to current staging tree to save you time in case
> you want to go forward with it.
> It's at git@github.com:verygreen/linux.git branch lustre-next-sendmsg
> 
> James, can you please give it a try on IB?

Its broke for the ko2iblnd driver.

[  110.840583] LNet: Using FMR for registration
[  110.991747] LNet: Added LNI 10.37.248.137@o2ib1 [63/2560/0/180]
[  110.998211] ------------[ cut here ]------------
[  111.003012] kernel BUG at lib/iov_iter.c:513!
[  111.007545] invalid opcode: 0000 [#1] SMP
[  111.011731] Modules linked in: ko2iblnd(C) ptlrpc(C+) obdclass(C) 
ksocklnd(C) lnet(C) sha512_generic sha256_generic md5 crc32_generic crc3
2_pclmul libcfs(C) autofs4 ipmi_devintf auth_rpcgss nfsv4 dns_resolver 
8021q iptable_filter ip_tables x_tables ib_ipoib rdma_ucm ib_ucm ib_uv
erbs ib_umad rdma_cm configfs ib_cm iw_cm mlx4_ib ib_core dm_mirror 
dm_region_hash dm_log dm_multipath sg sd_mod joydev pcspkr dm_mod mpt3sas
 raid_class acpi_cpufreq ipmi_ssif ipmi_si ipmi_msghandler isci libsas 
scsi_transport_sas wmi tpm_tis tpm i2c_i801 ahci libahci libata scsi_m
od ehci_pci ehci_hcd button tcp_cubic nfsv3(E) nfs_acl(E) ipv6(E) nfs(E) 
lockd(E) sunrpc(E) grace(E) mlx4_en(E) mlx4_core(E) igb(E) i2c_algo_
bit(E) i2c_core(E) ptp(E) pps_core(E) hwmon(E)
[  111.086669] CPU: 6 PID: 11899 Comm: router_checker Tainted: G         C  
E   4.7.0-rc6+ #1
[  111.095248] Hardware name: Supermicro X9DRT/X9DRT, BIOS 3.0a 02/19/2014
[  111.102040] task: ffff880826d32d80 ti: ffff880811d24000 task.ti: 
ffff880811d24000
[  111.109818] RIP: 0010:[<ffffffff8128daf2>]  [<ffffffff8128daf2>] 
iov_iter_kvec+0x22/0x30
[  111.118302] RSP: 0018:ffff880811d27b28  EFLAGS: 00010246
[  111.123806] RAX: 0000000000000000 RBX: ffff88105e037c00 RCX: 
0000000000000000
[  111.131111] RDX: 0000000000000000 RSI: 0000000000000005 RDI: 
ffff880811d27b78
[  111.138426] RBP: ffff880811d27b28 R08: 0000000000000000 R09: 
0000000000000000
[  111.145751] R10: 0000000000000000 R11: 00000000fffd19f7 R12: 
0000000000000000
[  111.153083] R13: 0000000000000000 R14: 000500010a25ca3b R15: 
ffff880811d27b78
[  111.160407] FS:  0000000000000000(0000) GS:ffff88107fd00000(0000) 
knlGS:0000000000000000
[  111.168797] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  111.174717] CR2: 00007fd075c48945 CR3: 000000105a9ec000 CR4: 
00000000000406e0
[  111.182031] Stack:
[  111.184224]  ffff880811d27be8 ffffffffa04012bd 0000000000000000 
ffff880811d27b40
[  111.192206]  0000000000000000 0000000000000000 0000000000000000 
0000000000000000
[  111.192206]  0000000000000000 0000000000000000 0000000000000000 
0000000000000000
[  111.200196]  0000880800000002 ffff88105e394000 0000000000000000 
0000000000000000
[  111.208179] Call Trace:
[  111.210818]  [<ffffffffa04012bd>] kiblnd_send+0x51d/0x9e0 [ko2iblnd]
[  111.217370]  [<ffffffffa06ec6bd>] lnet_ni_send+0x3d/0xe0 [lnet]
[  111.223487]  [<ffffffffa06ee223>] lnet_send+0x6b3/0xc80 [lnet]
[  111.229501]  [<ffffffffa06eeb58>] LNetGet+0x368/0x650 [lnet]
[  111.235346]  [<ffffffffa0692a50>] ? cfs_percpt_lock+0x50/0x110 [libcfs]
[  111.242139]  [<ffffffffa06f4d8f>] lnet_ping_router_locked+0x20f/0x840 
[lnet]
[  111.249384]  [<ffffffffa06f5909>] lnet_router_checker+0xd9/0x490 [lnet]
[  111.256192]  [<ffffffff8108347d>] ? default_wake_function+0xd/0x10
[  111.262549]  [<ffffffff810923f1>] ? __wake_up_common+0x51/0x80
[  111.268562]  [<ffffffffa06f5830>] ? lnet_prune_rc_data+0x470/0x470 
[lnet]
[  111.275544]  [<ffffffff81508f9b>] ? schedule+0x3b/0xa0
[  111.280871]  [<ffffffffa06f5830>] ? lnet_prune_rc_data+0x470/0x470 
[lnet]
[  111.287849]  [<ffffffff810779d7>] kthread+0xc7/0xe0
[  111.292904]  [<ffffffff8150c3cf>] ret_from_fork+0x1f/0x40
[  111.298475]  [<ffffffff81077910>] ? 
kthread_freezable_should_stop+0x70/0x70
[  111.305631] Code: 2e 0f 1f 84 00 00 00 00 00 55 40 f6 c6 02 48 89 e5 74 
18 89 37 48 89 57 18 48 89 4f 20 48 c7 47 08 00 00 00 00 4c 89 47 10 c9 c3 
<0f> 0b eb fe 66 2e 0f 1f 84 00 00 00 00 00 48 8b 47 10 55 48 89
[  111.329528] RIP  [<ffffffff8128daf2>] iov_iter_kvec+0x22/0x30
[  111.335533]  RSP <ffff880811d27b28>
[  111.339360] ---[ end trace 1ea9288f558e2c8d ]---