From: Jan Kara <jack@suse.cz>
To: Alexander Beregalov <a.beregalov@gmail.com>
Cc: Theodore Tso <tytso@mit.edu>,
"linux-next@vger.kernel.org" <linux-next@vger.kernel.org>,
linux-ext4@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>,
sparclinux@vger.kernel.org
Subject: Re: next-20090310: ext4 hangs
Date: Tue, 31 Mar 2009 14:33:07 +0200 [thread overview]
Message-ID: <20090331123307.GG11808@duck.suse.cz> (raw)
In-Reply-To: <a4423d670903310307i7acd31f0r4836beae14cfb92d@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2902 bytes --]
On Tue 31-03-09 14:07:30, Alexander Beregalov wrote:
> 2009/3/31 Jan Kara <jack@suse.cz>:
> > On Thu 26-03-09 01:38:32, Alexander Beregalov wrote:
> >> 2009/3/25 Jan Kara <jack@suse.cz>:
> >> > On Wed 25-03-09 20:07:46, Alexander Beregalov wrote:
> >> >> 2009/3/25 Jan Kara <jack@suse.cz>:
> >> >> > On Wed 25-03-09 18:29:10, Alexander Beregalov wrote:
> >> >> >> 2009/3/25 Jan Kara <jack@suse.cz>:
> >> >> >> > On Wed 25-03-09 18:18:43, Alexander Beregalov wrote:
> >> >> >> >> 2009/3/25 Jan Kara <jack@suse.cz>:
> >> >> >> >> >> > So, I think I need to try it on 2.6.29-rc7 again.
> >> >> >> >> >> I've looked into this. Obviously, what's happenning is that we delete
> >> >> >> >> >> an inode and jbd2_journal_release_jbd_inode() finds inode is just under
> >> >> >> >> >> writeout in transaction commit and thus it waits. But it gets never woken
> >> >> >> >> >> up and because it has a handle from the transaction, every one eventually
> >> >> >> >> >> blocks on waiting for a transaction to finish.
> >> >> >> >> >> But I don't really see how that can happen. The code is really
> >> >> >> >> >> straightforward and everything happens under j_list_lock... Strange.
> >> >> >> >> > BTW: Is the system SMP?
> >> >> >> >> No, it is UP system.
> >> >> >> > Even stranger. And do you have CONFIG_PREEMPT set?
> >> >> >> >
> >> >> >> >> The bug exists even in 2.6.29, I posted it with a new topic.
> >> >> >> > OK, I've sort-of expected this.
> >> >> >>
> >> >> >> CONFIG_PREEMPT_RCU=y
> >> >> >> CONFIG_PREEMPT_RCU_TRACE=y
> >> >> >> # CONFIG_PREEMPT_NONE is not set
> >> >> >> # CONFIG_PREEMPT_VOLUNTARY is not set
> >> >> >> CONFIG_PREEMPT=y
> >> >> >> CONFIG_DEBUG_PREEMPT=y
> >> >> >> # CONFIG_PREEMPT_TRACER is not set
> >> >> >>
> >> >> >> config is attached.
> >> >> > Thanks for the data. I still don't see how the wakeup can get lost. The
> >> >> > process even cannot be preempted when we are in the section protected by
> >> >> > j_list_lock... Can you send me a disassembly of functions
> >> >> > jbd2_journal_release_jbd_inode() and journal_submit_data_buffers() so that
> >> >> > I can see whether the compiler has not reordered something unexpectedly?
> >> > Thanks for the disassembly...
> >> >
> >> >> By default gcc inlines journal_submit_data_buffers()
> >> >> Here is -fno-inline version. Default version is in attach.
> > <snip>
> >
> > I'm helpless here. I don't see how we can miss a wakeup (plus you seem to
> > be the only one reporting the bug). Could you please compile and test the kernel
> > with the attached patch? It will print to kernel log when we go to sleep
> > waiting for inode commit and when we send wakeups etc. When you hit the
> > deadlock, please send me your kernel log. It should help with debugging why do
> > we miss the wakeup. Thanks.
>
> Which patch?
Ups. Forgot to attach ;).
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
[-- Attachment #2: 0001-ext4-Debug-sleepers-in-iput.patch --]
[-- Type: text/x-patch, Size: 1983 bytes --]
>From 123ab7510c04c698077e5756b4de6c66ce8ee71e Mon Sep 17 00:00:00 2001
From: Jan Kara <jack@suse.cz>
Date: Tue, 31 Mar 2009 11:57:10 +0200
Subject: [PATCH] ext4: Debug sleepers in iput()
Signed-off-by: Jan Kara <jack@suse.cz>
---
fs/jbd2/commit.c | 4 ++++
fs/jbd2/journal.c | 6 ++++++
2 files changed, 10 insertions(+), 0 deletions(-)
diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c
index 62804e5..f47b8a3 100644
--- a/fs/jbd2/commit.c
+++ b/fs/jbd2/commit.c
@@ -259,6 +259,8 @@ static int journal_submit_data_buffers(journal_t *journal,
spin_lock(&journal->j_list_lock);
J_ASSERT(jinode->i_transaction == commit_transaction);
jinode->i_flags &= ~JI_COMMIT_RUNNING;
+ if (jinode->i_flags & 4)
+ printk(KERN_INFO "JBD2: Waking up sleeper on ino %lu\n", jinode->i_vfs_inode->i_ino);
wake_up_bit(&jinode->i_flags, __JI_COMMIT_RUNNING);
}
spin_unlock(&journal->j_list_lock);
@@ -296,6 +298,8 @@ static int journal_finish_inode_data_buffers(journal_t *journal,
}
spin_lock(&journal->j_list_lock);
jinode->i_flags &= ~JI_COMMIT_RUNNING;
+ if (jinode->i_flags & 4)
+ printk(KERN_INFO "JBD2: Waking up sleeper on ino %lu\n", jinode->i_vfs_inode->i_ino);
wake_up_bit(&jinode->i_flags, __JI_COMMIT_RUNNING);
}
diff --git a/fs/jbd2/journal.c b/fs/jbd2/journal.c
index 5814410..5459fd9 100644
--- a/fs/jbd2/journal.c
+++ b/fs/jbd2/journal.c
@@ -2225,11 +2225,17 @@ restart:
if (jinode->i_flags & JI_COMMIT_RUNNING) {
wait_queue_head_t *wq;
DEFINE_WAIT_BIT(wait, &jinode->i_flags, __JI_COMMIT_RUNNING);
+ unsigned long ino = jinode->i_vfs_inode->i_ino;
+
+ jinode->i_flags |= 4;
+ printk(KERN_INFO "JBD2: Waiting for ino %lu\n", ino);
+
wq = bit_waitqueue(&jinode->i_flags, __JI_COMMIT_RUNNING);
prepare_to_wait(wq, &wait.wait, TASK_UNINTERRUPTIBLE);
spin_unlock(&journal->j_list_lock);
schedule();
finish_wait(wq, &wait.wait);
+ printk(KERN_INFO "JBD2: Woken on ino %lu\n", ino);
goto restart;
}
--
1.6.0.2
next prev parent reply other threads:[~2009-03-31 12:33 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-03-10 12:17 next-20090310: ext4 hangs Alexander Beregalov
2009-03-10 12:46 ` Theodore Tso
2009-03-10 12:54 ` Alexander Beregalov
2009-03-10 14:18 ` Alexander Beregalov
2009-03-10 15:47 ` Theodore Tso
2009-03-11 16:07 ` Alexander Beregalov
2009-03-25 15:11 ` Jan Kara
2009-03-25 15:15 ` Jan Kara
2009-03-25 15:18 ` Alexander Beregalov
2009-03-25 15:22 ` Jan Kara
2009-03-25 15:29 ` Alexander Beregalov
2009-03-25 16:15 ` Jan Kara
2009-03-25 17:07 ` Alexander Beregalov
2009-03-25 19:43 ` Jan Kara
2009-03-25 22:38 ` Alexander Beregalov
2009-03-26 0:00 ` Jan Kara
2009-03-26 0:17 ` Jiri Gaisler
2009-03-26 0:25 ` Jan Kara
2009-03-31 10:01 ` Jan Kara
2009-03-31 10:07 ` Alexander Beregalov
2009-03-31 12:33 ` Jan Kara [this message]
2009-04-02 18:50 ` Alexander Beregalov
2009-04-04 21:09 ` Alexander Beregalov
2009-04-06 9:20 ` Jan Kara
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20090331123307.GG11808@duck.suse.cz \
--to=jack@suse.cz \
--cc=a.beregalov@gmail.com \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-next@vger.kernel.org \
--cc=sparclinux@vger.kernel.org \
--cc=tytso@mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).