[bug report] libceph: add new iov_iter-based ceph_msg_data_type and ceph_osd_data_type

[bug report] libceph: add new iov_iter-based ceph_msg_data_type and ceph_osd_data_type

[Dan Carpenter]

Hello Jeff Layton,

To be honest, I'm not sure why I am only seeing this now.  These
warnings are hard to analyse because they involve such a long call tree.
Anyway, hopefully it's not too complicated for you since you know the
code.

The patch dee0c5f83460: "libceph: add new iov_iter-based
ceph_msg_data_type and ceph_osd_data_type" from Jul 1, 2022
(linux-next), leads to the following Smatch static checker warning:

	lib/iov_iter.c:905 want_pages_array()
	warn: sleeping in atomic context

lib/iov_iter.c
    896 static int want_pages_array(struct page ***res, size_t size,
    897                             size_t start, unsigned int maxpages)
    898 {
    899         unsigned int count = DIV_ROUND_UP(size + start, PAGE_SIZE);
    900 
    901         if (count > maxpages)
    902                 count = maxpages;
    903         WARN_ON(!count);        // caller should've prevented that
    904         if (!*res) {
--> 905                 *res = kvmalloc_array(count, sizeof(struct page *), GFP_KERNEL);
    906                 if (!*res)
    907                         return 0;
    908         }
    909         return count;
    910 }


prep_next_sparse_read() <- disables preempt
-> advance_cursor()
   -> ceph_msg_data_next()
      -> ceph_msg_data_iter_next()
         -> iov_iter_get_pages2()
            -> __iov_iter_get_pages_alloc()
               -> want_pages_array()

The prep_next_sparse_read() functions hold the spin_lock(&o->o_requests_lock);
lock so it can't sleep.  But iov_iter_get_pages2() seems like a sleeping
operation.

regards,
dan carpenter

Re: [bug report] libceph: add new iov_iter-based ceph_msg_data_type and ceph_osd_data_type

[Jeff Layton]

On Wed, 2023-10-11 at 12:50 +0300, Dan Carpenter wrote:
> Hello Jeff Layton,
> 
> To be honest, I'm not sure why I am only seeing this now.  These
> warnings are hard to analyse because they involve such a long call tree.
> Anyway, hopefully it's not too complicated for you since you know the
> code.
> 
> The patch dee0c5f83460: "libceph: add new iov_iter-based
> ceph_msg_data_type and ceph_osd_data_type" from Jul 1, 2022
> (linux-next), leads to the following Smatch static checker warning:
> 
> 	lib/iov_iter.c:905 want_pages_array()
> 	warn: sleeping in atomic context
> 
> lib/iov_iter.c
>     896 static int want_pages_array(struct page ***res, size_t size,
>     897                             size_t start, unsigned int maxpages)
>     898 {
>     899         unsigned int count = DIV_ROUND_UP(size + start, PAGE_SIZE);
>     900 
>     901         if (count > maxpages)
>     902                 count = maxpages;
>     903         WARN_ON(!count);        // caller should've prevented that
>     904         if (!*res) {
> --> 905                 *res = kvmalloc_array(count, sizeof(struct page *), GFP_KERNEL);
>     906                 if (!*res)
>     907                         return 0;
>     908         }
>     909         return count;
>     910 }
> 
> 
> prep_next_sparse_read() <- disables preempt
> -> advance_cursor()
>    -> ceph_msg_data_next()
>       -> ceph_msg_data_iter_next()
>          -> iov_iter_get_pages2()
>             -> __iov_iter_get_pages_alloc()
>                -> want_pages_array()
> 
> The prep_next_sparse_read() functions hold the spin_lock(&o->o_requests_lock);
> lock so it can't sleep.  But iov_iter_get_pages2() seems like a sleeping
> operation.
> 
> 

I think this is a false alarm, but I'd appreciate a sanity check:

iov_iter_get_pages2 has this:

	BUG_ON(!pages);

...which should ensure that *res won't be NULL when want_pages_array is
called. That said, this seems like kind of a fragile thing to rely on.
Should we do something to make this a bit less subtle?
-- 
Jeff Layton <jlayton@kernel.org>

Re: [bug report] libceph: add new iov_iter-based ceph_msg_data_type and ceph_osd_data_type

[Dan Carpenter]

On Wed, Oct 11, 2023 at 08:06:59AM -0400, Jeff Layton wrote:
> On Wed, 2023-10-11 at 12:50 +0300, Dan Carpenter wrote:
> > Hello Jeff Layton,
> > 
> > To be honest, I'm not sure why I am only seeing this now.  These
> > warnings are hard to analyse because they involve such a long call tree.
> > Anyway, hopefully it's not too complicated for you since you know the
> > code.
> > 
> > The patch dee0c5f83460: "libceph: add new iov_iter-based
> > ceph_msg_data_type and ceph_osd_data_type" from Jul 1, 2022
> > (linux-next), leads to the following Smatch static checker warning:
> > 
> > 	lib/iov_iter.c:905 want_pages_array()
> > 	warn: sleeping in atomic context
> > 
> > lib/iov_iter.c
> >     896 static int want_pages_array(struct page ***res, size_t size,
> >     897                             size_t start, unsigned int maxpages)
> >     898 {
> >     899         unsigned int count = DIV_ROUND_UP(size + start, PAGE_SIZE);
> >     900 
> >     901         if (count > maxpages)
> >     902                 count = maxpages;
> >     903         WARN_ON(!count);        // caller should've prevented that
> >     904         if (!*res) {
> > --> 905                 *res = kvmalloc_array(count, sizeof(struct page *), GFP_KERNEL);
> >     906                 if (!*res)
> >     907                         return 0;
> >     908         }
> >     909         return count;
> >     910 }
> > 
> > 
> > prep_next_sparse_read() <- disables preempt
> > -> advance_cursor()
> >    -> ceph_msg_data_next()
> >       -> ceph_msg_data_iter_next()
> >          -> iov_iter_get_pages2()
> >             -> __iov_iter_get_pages_alloc()
> >                -> want_pages_array()
> > 
> > The prep_next_sparse_read() functions hold the spin_lock(&o->o_requests_lock);
> > lock so it can't sleep.  But iov_iter_get_pages2() seems like a sleeping
> > operation.
> > 
> > 
> 
> I think this is a false alarm, but I'd appreciate a sanity check:
> 
> iov_iter_get_pages2 has this:
> 
> 	BUG_ON(!pages);
> 
> ...which should ensure that *res won't be NULL when want_pages_array is
> called. That said, this seems like kind of a fragile thing to rely on.
> Should we do something to make this a bit less subtle?

Nah, forget about it.  Let's just leave this conversation up on lore so
if anyone has questions about it they can read this thread.  The
BUG_ON(!pages) is really straight forward to understand.

regards,
dan carpenter