Re: Orangefs ABI documentation

[Mike Marshall <hubcap@omnibond.com>]

I added the patches, and ran a bunch of tests.

Stuff works fine when left unbothered, and also
when wrenches are thrown into the works.

I had multiple userspace things going on at the
same time, dbench, ls -R, find... kill -9 or control-C on
any of them is handled well. When I killed both
the client-core and its restarter, the kernel
dealt with swarm of ops that had nowhere
to go... the WARN_ON in service_operation
was hit.

Feb 12 16:19:12 be1 kernel: [ 3658.167544] orangefs: please confirm
that pvfs2-client daemon is running.
Feb 12 16:19:12 be1 kernel: [ 3658.167547] fs/orangefs/dir.c line 264:
orangefs_readdir: orangefs_readdir_index_get() failure (-5)
Feb 12 16:19:12 be1 kernel: [ 3658.170741] ------------[ cut here ]------------
Feb 12 16:19:12 be1 kernel: [ 3658.170746] WARNING: CPU: 0 PID: 1667
at fs/orangefs/waitqueue.c:203 service_operation+0x4f6/0x7f0()
Feb 12 16:19:12 be1 kernel: [ 3658.170747] Modules linked in:
ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6
nf_defrag_ipv6 bnep bluetooth nf_conntrack_ipv4 nf_defrag_ipv4 rfkill
xt_conntrack nf_conntrack ebtable_nat ebtable_broute bridge stp llc
ebtable_filter ebtables ip6table_mangle ip6table_security ip6table_raw
ip6table_filter ip6_tables iptable_mangle iptable_security iptable_raw
ppdev parport_pc virtio_balloon pvpanic parport serio_raw 8139too
i2c_piix4 virtio_console uinput qxl drm_kms_helper ttm drm 8139cp
i2c_core virtio_pci virtio virtio_ring mii ata_generic pata_acpi
Feb 12 16:19:12 be1 kernel: [ 3658.170770] CPU: 0 PID: 1667 Comm:
dbench Not tainted 4.4.0-161995-gda335c6 #62
Feb 12 16:19:12 be1 kernel: [ 3658.170771] Hardware name: Red Hat KVM,
BIOS 0.5.1 01/01/2011
Feb 12 16:19:12 be1 kernel: [ 3658.170772]  0000000000000000
000000001371c2af ffff88000c89fc08 ffffffff8139c7dd
Feb 12 16:19:12 be1 kernel: [ 3658.170774]  ffff88000c89fc40
ffffffff8108e510 ffff88000cb58000 ffff88000cb5c1d0
Feb 12 16:19:12 be1 kernel: [ 3658.170776]  ffff88000cb5c188
00000000fffffff5 0000000000000000 ffff88000c89fc50
Feb 12 16:19:12 be1 kernel: [ 3658.170778] Call Trace:
Feb 12 16:19:12 be1 kernel: [ 3658.170782]  [<ffffffff8139c7dd>]
dump_stack+0x19/0x1c
Feb 12 16:19:12 be1 kernel: [ 3658.170786]  [<ffffffff8108e510>]
warn_slowpath_common+0x80/0xc0
Feb 12 16:19:12 be1 kernel: [ 3658.170787]  [<ffffffff8108e65a>]
warn_slowpath_null+0x1a/0x20
Feb 12 16:19:12 be1 kernel: [ 3658.170788]  [<ffffffff812fe6a6>]
service_operation+0x4f6/0x7f0
Feb 12 16:19:12 be1 kernel: [ 3658.170791]  [<ffffffff810c28b0>] ?
prepare_to_wait_event+0x100/0x100
Feb 12 16:19:12 be1 kernel: [ 3658.170792]  [<ffffffff810c28b0>] ?
prepare_to_wait_event+0x100/0x100
Feb 12 16:19:12 be1 kernel: [ 3658.170794]  [<ffffffff812f2557>]
wait_for_direct_io+0x157/0x520
Feb 12 16:19:12 be1 kernel: [ 3658.170796]  [<ffffffff812f29d6>]
do_readv_writev+0xb6/0x2a0
Feb 12 16:19:12 be1 kernel: [ 3658.170797]  [<ffffffff812f2c75>]
orangefs_file_write_iter+0xb5/0x1a0
Feb 12 16:19:12 be1 kernel: [ 3658.170801]  [<ffffffff811f962c>]
__vfs_write+0xcc/0x100
Feb 12 16:19:12 be1 kernel: [ 3658.170802]  [<ffffffff811f9c81>]
vfs_write+0xa1/0x190
Feb 12 16:19:12 be1 kernel: [ 3658.170804]  [<ffffffff811fabe7>]
SyS_pwrite64+0x87/0xb0
Feb 12 16:19:12 be1 kernel: [ 3658.170807]  [<ffffffff81782faf>]
entry_SYSCALL_64_fastpath+0x12/0x76
Feb 12 16:19:12 be1 kernel: [ 3658.170808] ---[ end trace 9335703ea9225d7b ]---

I run xfstests a very clunky way... I left it running when I left
the office on Friday. I'll grep through the output on my
terminal <blush> on Monday to see if there's any regressions...

-Mike

On Fri, Feb 12, 2016 at 1:00 PM, Martin Brandenburg <martin@omnibond.com> wrote:
> I have some patches for the kernel and our userspace code which
> eliminates the useless readdir buffers. They're a few months old at
> this point.
>
> The problem is that this is already part of the protocol. Unless we
> decide to change it, we can't very well get out of supporting this.
> Personally I want to clean this up while we still have the chance. We
> already plan to only support this module from the latest OrangeFS and
> up.
>
> In either case there's no reason it needs to be so confusing and imply
> it's shared.
>
> Mike, there wouldn't be an unlimited number of buffers. It's still
> limited by the number of ops which are pre-allocated.
>
> -- Martin
>
> On 2/12/16, Mike Marshall <hubcap@omnibond.com> wrote:
>> I'll get the patches today... I have about five small patches
>> that aren't pushed out to github or kernel.org yet, some
>> cosmetic patches and a couple of things you suggested
>> in mail messages... if they get in a fight with your
>> new patches I'll just ditch them and re-do whichever
>> ones of them are still needed after I've got your
>> new stuff tested.
>>
>> Thanks!
>>
>> -Mike
>>
>> On Thu, Feb 11, 2016 at 11:27 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
>>> On Wed, Feb 10, 2016 at 10:22:40PM -0500, Mike Marshall wrote:
>>>> > If there is (or at least supposed to be) something that prevents
>>>> > completions
>>>> > of readdir requests (on unrelated directories, by different processes,
>>>> > etc.)
>>>> > out of order, PLEASE SAY SO.  I would really prefer not to have to
>>>> > fight
>>>> > the readdir side of that mess; cancels are already bad enough ;-/
>>>>
>>>> Hi Al... your ideas sound good to me, I'll try to get you good
>>>> answers on stuff like the above sometime tomorrow...
>>>
>>> OK, this is really, really completely untested, might chew your data,
>>> bugger your dog, etc.  OTOH, if it somehow fails to do the above, it
>>> ought to deal with cancels properly.
>>>
>>> Pushed into #orangefs-untested, along with two wait_for_direct_io() fixes
>>> discussed upthread.  This is _not_ all - it still needs saner "wait for
>>> slot"
>>> logics, switching op->waitq to completion/killing loop in
>>> wait_for_matching_downcall(), etc.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>