much about ah objects in rxe

[Bob Pearson <rpearsonhpe@gmail.com>]

Jason,

I am confused a little.

 - xa_alloc_xxx internally takes xa->xa_lock with a spinlock but
   has a gfp_t parameter which is normally GFP_KERNEL. So I trust them when they say
   that it releases the lock around kmalloc's by 'magic' as you say.

 - The only read side operation on the rxe pool xarrays is in rxe_pool_get_index() but
   that will be protected by a rcu_read_lock so it can't deadlock with the write
   side spinlocks regardless of type (plain, _bh, _saveirq)

 - Apparently CM is calling ib_create_ah while holding spin locks. This would
   call xa_alloc_xxx which would unlock xa_lock and call kmalloc(..., GFP_KERNEL)
   which should cause a warning for AH. You say it does not because xarray doesn't
   call might_sleep().

I am not sure how might_sleep() works. When I add might_sleep() just ahead of
xa_alloc_xxx() it does not cause warnings for CM test cases (e.g. rping.)
Another way to study this would be to test for in_atomic() but
that seems to be discouraged and may not work as assumed. It's hard to reproduce
evidence that ib_create_ah really has spinlocks held by the caller. I think it
was seen in lockdep traces but I have a hard time reading them.

 - There is a lot of effort trying to make 'deadlocks' go away. But the read side
   is going to end as up rcu_read_lock so there soon will be no deadlocks with
   rxe_pool_get_index() possible. xarrays were designed to work well with rcu
   so it would better to just go ahead and do it. Verbs objects tend to be long
   lived with lots of IO on each instance. This is a perfect use case for rcu.

I think this means there is no reason for anything but a plain spinlock in rxe_alloc
and rxe_add_to_pool.

To sum up once we have rcu enabled the only required change is to use GFP_ATOMIC
or find a way to pre-allocate for AH objects (assuming that I can convince myself
that ib_create_ah really comes with spinlocks held).

Bob