From: Jeff Layton <jlayton-vpEMnDpepFuMZCB2o+C8xQ@public.gmane.org>
To: "J. Bruce Fields" <bfields-uC3wQj2KruNg9hUCZPvPmw@public.gmane.org>
Cc: linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-fsdevel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Al Viro <viro-3bDd1+5oDREiFSDQTTA3OLVCufUGDwFn@public.gmane.org>
Subject: Re: [PATCH v5 00/20] nfsd: open file caching
Date: Sat, 10 Oct 2015 07:19:23 -0400 [thread overview]
Message-ID: <20151010071923.435ce037@synchrony.poochiereds.net> (raw)
In-Reply-To: <20151008180400.GB496-uC3wQj2KruNg9hUCZPvPmw@public.gmane.org>
On Thu, 8 Oct 2015 14:04:00 -0400
"J. Bruce Fields" <bfields-uC3wQj2KruNg9hUCZPvPmw@public.gmane.org> wrote:
> On Thu, Oct 08, 2015 at 12:55:29PM -0400, Jeff Layton wrote:
> > My bad...it needs this patch. I'll roll this into the set before the
> > next posting.
>
> Oh, good, thanks.
>
> Also, just seen on the server side--not sure what was going on at the
> time.
>
> There were a ton of these:
>
> Oct 08 12:35:07 f21-1.fieldses.org kernel: ------------[ cut here ]------------
> Oct 08 12:35:07 f21-1.fieldses.org kernel: WARNING: CPU: 1 PID: 584 at lib/list_debug.c:59 __list_del_entry+0x9e/0xc0()
> Oct 08 12:35:07 f21-1.fieldses.org kernel: list_del corruption. prev->next should be ffff88004cb23f80, but was b6a7e8df8948e4eb
> Oct 08 12:35:07 f21-1.fieldses.org kernel: Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
> Oct 08 12:35:07 f21-1.fieldses.org kernel: CPU: 1 PID: 584 Comm: fsnotify_mark Not tainted 4.3.0-rc3-14186-g7619b8e #322
> Oct 08 12:35:07 f21-1.fieldses.org kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140709_153950- 04/01/2014
> Oct 08 12:35:07 f21-1.fieldses.org kernel: ffffffff81f62683 ffff880071af3d50 ffffffff8160540c ffff880071af3d98
> Oct 08 12:35:07 f21-1.fieldses.org kernel: ffff880071af3d88 ffffffff81077692 ffff88004cb23f80 ffffffff8109c160
> Oct 08 12:35:07 f21-1.fieldses.org kernel: ffff880071af3e08 ffff880071af3e30 ffff88004cb23f70 ffff880071af3de8
> Oct 08 12:35:07 f21-1.fieldses.org kernel: Call Trace:
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff8160540c>] dump_stack+0x4e/0x82
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81077692>] warn_slowpath_common+0x82/0xc0
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff8109c160>] ? sort_range+0x20/0x30
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff8107771c>] warn_slowpath_fmt+0x4c/0x50
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff8162219e>] __list_del_entry+0x9e/0xc0
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff811ef485>] fsnotify_mark_destroy+0x95/0x140
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff810baa10>] ? wait_woken+0x90/0x90
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff811ef3f0>] ? fsnotify_put_mark+0x30/0x30
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81098d6f>] kthread+0xef/0x110
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81a767dc>] ? _raw_spin_unlock_irq+0x2c/0x50
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81098c80>] ? kthread_create_on_node+0x200/0x200
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81a7748f>] ret_from_fork+0x3f/0x70
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81098c80>] ? kthread_create_on_node+0x200/0x200
> Oct 08 12:35:07 f21-1.fieldses.org kernel: ---[ end trace 687abd8552e06b32 ]---
>
Thanks for the bug report! I think I understand the problem now:
It's in the way this patchset embeds a fsnotify_mark inside the
nfsd_file. The way fsnotify_destroy_mark works sort of requires that it
be freed separately since it wants to traverse these objects under a
srcu read lock. The rest of the stack traces are probably collateral
damage from that mem corruption.
I think I'll have to change the code to allocate the fsnotify_mark objects
separately. It may also be better to have just one mark per inode and
have each nfsd_file take a reference to the mark. I'll need to stare at
the code a bit longer to see what makes the most sense.
--
Jeff Layton <jlayton-vpEMnDpepFuMZCB2o+C8xQ@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2015-10-10 11:19 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-10-05 11:02 [PATCH v5 00/20] nfsd: open file caching Jeff Layton
2015-10-05 11:02 ` [PATCH v5 01/20] list_lru: add list_lru_rotate Jeff Layton
[not found] ` <1444042962-6947-2-git-send-email-jeff.layton-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org>
2015-10-05 21:47 ` Dave Chinner
2015-10-06 11:43 ` Jeff Layton
[not found] ` <20151006074341.0e2f796e-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org>
2015-10-07 1:09 ` Dave Chinner
2015-10-05 11:02 ` [PATCH v5 02/20] fs: have flush_delayed_fput flush the workqueue job Jeff Layton
2015-10-05 11:02 ` [PATCH v5 03/20] fs: add a kerneldoc header to fput Jeff Layton
2015-10-05 11:02 ` [PATCH v5 04/20] fs: add fput_queue Jeff Layton
2015-10-05 11:02 ` [PATCH v5 07/20] locks: create a new notifier chain for lease attempts Jeff Layton
2015-10-05 11:02 ` [PATCH v5 09/20] sunrpc: add a new cache_detail operation for when a cache is flushed Jeff Layton
2015-10-05 11:02 ` [PATCH v5 10/20] nfsd: add a new struct file caching facility to nfsd Jeff Layton
2015-10-05 11:02 ` [PATCH v5 11/20] nfsd: keep some rudimentary stats on nfsd_file cache Jeff Layton
[not found] ` <1444042962-6947-1-git-send-email-jeff.layton-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org>
2015-10-05 11:02 ` [PATCH v5 05/20] fs: export flush_delayed_fput Jeff Layton
2015-10-05 11:02 ` [PATCH v5 06/20] fsnotify: export several symbols Jeff Layton
2015-10-05 11:02 ` [PATCH v5 08/20] nfsd: move include of state.h from trace.c to trace.h Jeff Layton
2015-10-05 11:02 ` [PATCH v5 12/20] nfsd: allow filecache open to skip fh_verify check Jeff Layton
2015-10-05 11:02 ` [PATCH v5 13/20] nfsd: hook up nfsd_write to the new nfsd_file cache Jeff Layton
2015-10-05 11:02 ` [PATCH v5 14/20] nfsd: hook up nfsd_read to the " Jeff Layton
2015-10-05 11:02 ` [PATCH v5 18/20] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file Jeff Layton
2015-10-08 16:42 ` [PATCH v5 00/20] nfsd: open file caching J. Bruce Fields
2015-10-08 16:55 ` Jeff Layton
[not found] ` <20151008125529.3f30308e-08S845evdOaAjSkqwZiSMmfYqLom42DlXqFh9Ls21Oc@public.gmane.org>
2015-10-08 18:04 ` J. Bruce Fields
[not found] ` <20151008180400.GB496-uC3wQj2KruNg9hUCZPvPmw@public.gmane.org>
2015-10-10 11:19 ` Jeff Layton [this message]
2015-10-10 13:48 ` J. Bruce Fields
2015-10-05 11:02 ` [PATCH v5 15/20] nfsd: hook nfsd_commit up to the nfsd_file cache Jeff Layton
2015-10-05 11:02 ` [PATCH v5 16/20] nfsd: convert nfs4_file->fi_fds array to use nfsd_files Jeff Layton
2015-10-05 11:02 ` [PATCH v5 17/20] nfsd: have nfsd_test_lock use the nfsd_file cache Jeff Layton
2015-10-05 11:02 ` [PATCH v5 19/20] nfsd: hook up nfs4_preprocess_stateid_op to " Jeff Layton
2015-10-05 11:02 ` [PATCH v5 20/20] nfsd: rip out the raparms cache Jeff Layton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20151010071923.435ce037@synchrony.poochiereds.net \
--to=jlayton-vpemndpepfumzcb2o+c8xq@public.gmane.org \
--cc=bfields-uC3wQj2KruNg9hUCZPvPmw@public.gmane.org \
--cc=linux-fsdevel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=viro-3bDd1+5oDREiFSDQTTA3OLVCufUGDwFn@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).