From: Jan Kara <jack@suse.cz>
To: Paul Moore <paul@paul-moore.com>
Cc: jack@suse.cz, willy@infradead.org, baijiaju1990@gmail.com,
Eric Paris <eparis@redhat.com>,
amir73il@gmail.com, linux-audit@redhat.com,
linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org
Subject: Re: [PATCH] kernel: audit_tree: Fix a sleep-in-atomic-context bug
Date: Mon, 25 Jun 2018 14:55:40 +0200 [thread overview]
Message-ID: <20180625125540.gmhupkrkjrhwtj2p@quack2.suse.cz> (raw)
In-Reply-To: <20180625092257.kyqnmn4ki7cuqkat@quack2.suse.cz>
On Mon 25-06-18 11:22:57, Jan Kara wrote:
> On Fri 22-06-18 14:56:09, Paul Moore wrote:
> > On Fri, Jun 22, 2018 at 5:23 AM Jan Kara <jack@suse.cz> wrote:
> > > On Wed 20-06-18 21:29:12, Matthew Wilcox wrote:
> > > > On Thu, Jun 21, 2018 at 11:32:45AM +0800, Jia-Ju Bai wrote:
> > > > > The kernel may sleep with holding a spinlock.
> > > > > The function call paths (from bottom to top) in Linux-4.16.7 are:
> > > > >
> > > > > [FUNC] kmem_cache_alloc(GFP_KERNEL)
> > > > > fs/notify/mark.c, 439:
> > > > > kmem_cache_alloc in fsnotify_attach_connector_to_object
> > > > > fs/notify/mark.c, 520:
> > > > > fsnotify_attach_connector_to_object in fsnotify_add_mark_list
> > > > > fs/notify/mark.c, 590:
> > > > > fsnotify_add_mark_list in fsnotify_add_mark_locked
> > > > > kernel/audit_tree.c, 437:
> > > > > fsnotify_add_mark_locked in tag_chunk
> > > > > kernel/audit_tree.c, 423:
> > > > > spin_lock in tag_chunk
> > > >
> > > > There are several locks here; your report would be improved by saying
> > > > which one is the problem. I'm assuming it's old_entry->lock.
> > > >
> > > > spin_lock(&old_entry->lock);
> > > > ...
> > > > if (fsnotify_add_inode_mark_locked(chunk_entry,
> > > > old_entry->connector->inode, 1)) {
> > > > ...
> > > > return fsnotify_add_mark_locked(mark, inode, NULL, allow_dups);
> > > > ...
> > > > ret = fsnotify_add_mark_list(mark, inode, mnt, allow_dups);
> > > > ...
> > > > if (inode)
> > > > connp = &inode->i_fsnotify_marks;
> > > > conn = fsnotify_grab_connector(connp);
> > > > if (!conn) {
> > > > err = fsnotify_attach_connector_to_object(connp, inode, mnt);
> > > >
> > > > It seems to me that this is safe because old_entry is looked up from
> > > > fsnotify_find_mark, and it can't be removed while its lock is held.
> > > > Therefore there's always a 'conn' returned from fsnotify_grab_connector(),
> > > > and so this path will never be taken.
> > > >
> > > > But this code path is confusing to me, and I could be wrong. Jan, please
> > > > confirm my analysis is correct?
> > >
> > > Yes, you are correct. The presence of another mark in the list (and the
> > > fact we pin it there using refcount & mark_mutex) guarantees we won't need
> > > to allocate the connector. I agree the audit code's use of fsnotify would
> > > deserve some cleanup.
> >
> > I'm always open to suggestions and patches (hint, hint) from the
> > fsnotify experts ;)
>
> Yeah, I was looking into it on Friday and today :). Currently I've got a
> bit stuck because I think I've found some races in audit_tree code and I
> haven't yet decided how to fix them. E.g. am I right the following can
> happen?
>
> CPU1 CPU2
> tag_chunk(inode, tree1) tag_chunk(inode, tree2)
> old_entry = fsnotify_find_mark(); old_entry = fsnotify_find_mark();
> old = container_of(old_entry); old = container_of(old_entry);
> chunk = alloc_chunk(old->count + 1); chunk = alloc_chunk(old->count + 1);
> mutex_lock(&group->mark_mutex);
> adds new mark
> replaces chunk
> old->dead = 1;
> mutex_unlock(&group->mark_mutex);
> mutex_lock(&group->mark_mutex);
> if (!(old_entry->flags &
> FSNOTIFY_MARK_FLAG_ATTACHED)) {
> Check fails as old_entry is
> not yet destroyed
> adds new mark
> replaces old chunk again ->
> list corruption, lost refs, ...
> mutex_unlock(&group->mark_mutex);
>
> Generally there's a bigger problem that audit_tree code can have multiple
> marks attached to one inode but only one of them is the "valid" one (i.e.,
> the one embedded in the latest chunk). This is only a temporary state until
> fsnotify_destroy_mark() detaches the mark and then on last reference drop
> we really remove the mark from inode's list but during that window it is
> undefined which mark is returned from fsnotify_find_mark()...
>
> So am I right the above can really happen or is there some higher level
> synchronization I'm missing? If this can really happen, I think I'll need
> to rework the code so that audit_tree has just one mark attached and
> let it probably point to the current chunk.
Also am I right to assume that if two tag_chunk() calls race, both try to
add new fsnotify mark in create_chunk() and one of them fails, then the
resulting ENOSPC error from create_chunk() is actually a bug? Because from
looking at the code it seems that the desired functionality is that
tag_chunk() should add 'tree' to the chunk, expanding chunk as needed.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
next prev parent reply other threads:[~2018-06-25 12:55 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-06-21 3:32 [PATCH] kernel: audit_tree: Fix a sleep-in-atomic-context bug Jia-Ju Bai
2018-06-21 4:29 ` Matthew Wilcox
2018-06-22 9:23 ` Jan Kara
2018-06-22 18:56 ` Paul Moore
2018-06-25 9:22 ` Jan Kara
2018-06-25 12:55 ` Jan Kara [this message]
2018-06-25 22:25 ` Paul Moore
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180625125540.gmhupkrkjrhwtj2p@quack2.suse.cz \
--to=jack@suse.cz \
--cc=amir73il@gmail.com \
--cc=baijiaju1990@gmail.com \
--cc=eparis@redhat.com \
--cc=linux-audit@redhat.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=paul@paul-moore.com \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).