Re: hangs on boot in 9984d7394618df9

Re: hangs on boot in 9984d7394618df9

[Stephen Warren]

On 04/08/2013 03:48 PM, Al Viro wrote:
> On Mon, Apr 08, 2013 at 10:23:27PM +0100, Al Viro wrote:
>> On Mon, Apr 08, 2013 at 03:52:08PM -0500, Nathan Zimmer wrote:
>>
>>> Further digging seems to indicate 9984d7394618df9, the one right
>>> after the commit I previously identified.
>>> Not sure what I did wrong with my bisect to put it off by one.
>>
>> Ugh...  Can't reproduce here ;-/  Could you give more details on your
>> setup?
> 
> Anyway, I've just pushed a splitup of that commit (carved in 3 pieces)
> into vfs.git#pipe-splitup; could you check which part triggers that
> hang?  Should propagate in a few...

It looks like "pipe: unify ->release() and ->open()" introduces the
problem. Note that I had to add a prototype for fifo_open() before the
structs that reference it for that commit to compile.

Re: hangs on boot in 9984d7394618df9

[Doug Anderson]

Hi,

On Mon, Apr 8, 2013 at 3:17 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> Anyway, I've just pushed a splitup of that commit (carved in 3 pieces)
>> into vfs.git#pipe-splitup; could you check which part triggers that
>> hang?  Should propagate in a few...
>
> It looks like "pipe: unify ->release() and ->open()" introduces the
> problem. Note that I had to add a prototype for fifo_open() before the
> structs that reference it for that commit to compile.

It sounds like Stephen has provided you the info you needed so not
doing any extra testing now, but I figured I'd chime in that I hit
problems this morning with linux-next and it appears to be the same
thing.  I did a revert of 9984d7394618df9 and (plus a revert of a
handful of patches to the same file) and problems are resolved.  The
failure case is really weird in that everything works well booting to
a simple /bin/bash but fails when you do more complex tasks.

Anyway: If you need some extra testing feel free to CC me.

-Doug

Re: hangs on boot in 9984d7394618df9

[Al Viro]

On Mon, Apr 08, 2013 at 04:17:17PM -0600, Stephen Warren wrote:
> On 04/08/2013 03:48 PM, Al Viro wrote:
> > On Mon, Apr 08, 2013 at 10:23:27PM +0100, Al Viro wrote:
> >> On Mon, Apr 08, 2013 at 03:52:08PM -0500, Nathan Zimmer wrote:
> >>
> >>> Further digging seems to indicate 9984d7394618df9, the one right
> >>> after the commit I previously identified.
> >>> Not sure what I did wrong with my bisect to put it off by one.
> >>
> >> Ugh...  Can't reproduce here ;-/  Could you give more details on your
> >> setup?
> > 
> > Anyway, I've just pushed a splitup of that commit (carved in 3 pieces)
> > into vfs.git#pipe-splitup; could you check which part triggers that
> > hang?  Should propagate in a few...
> 
> It looks like "pipe: unify ->release() and ->open()" introduces the
> problem. Note that I had to add a prototype for fifo_open() before the
> structs that reference it for that commit to compile.

Very interesting...  Just in case, could you try this on top of that
branch and see if it triggers?

diff --git a/fs/pipe.c b/fs/pipe.c
index 8ce279b..b6cd51b 100644
--- a/fs/pipe.c
+++ b/fs/pipe.c
@@ -1022,6 +1022,11 @@ static int fifo_open(struct inode *inode, struct file *filp)
 	/* We can only do regular read/write on fifos */
 	filp->f_mode &= (FMODE_READ | FMODE_WRITE);
 
+	if (inode->i_sb->s_magic == PIPEFS_MAGIC) {
+		WARN_ON(filp->f_flags & O_NONBLOCK);
+		filp->f_flags &= ~O_NONBLOCK;
+	}
+
 	switch (filp->f_mode) {
 	case FMODE_READ:
 	/*

Re: hangs on boot in 9984d7394618df9

[Al Viro]

On Mon, Apr 08, 2013 at 11:46:37PM +0100, Al Viro wrote:

> Very interesting...  Just in case, could you try this on top of that
> branch and see if it triggers?
> 
> diff --git a/fs/pipe.c b/fs/pipe.c
> index 8ce279b..b6cd51b 100644
> --- a/fs/pipe.c
> +++ b/fs/pipe.c
> @@ -1022,6 +1022,11 @@ static int fifo_open(struct inode *inode, struct file *filp)
>  	/* We can only do regular read/write on fifos */
>  	filp->f_mode &= (FMODE_READ | FMODE_WRITE);
>  
> +	if (inode->i_sb->s_magic == PIPEFS_MAGIC) {
> +		WARN_ON(filp->f_flags & O_NONBLOCK);
> +		filp->f_flags &= ~O_NONBLOCK;
> +	}

*d'oh*

OK, I'm an idiot.  I see what's going on now...  Give me a few and I'll push
a fix.

Re: hangs on boot in 9984d7394618df9

[Al Viro]

On Mon, Apr 08, 2013 at 03:45:31PM -0700, Doug Anderson wrote:
> Hi,
> 
> On Mon, Apr 8, 2013 at 3:17 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
> >> Anyway, I've just pushed a splitup of that commit (carved in 3 pieces)
> >> into vfs.git#pipe-splitup; could you check which part triggers that
> >> hang?  Should propagate in a few...
> >
> > It looks like "pipe: unify ->release() and ->open()" introduces the
> > problem. Note that I had to add a prototype for fifo_open() before the
> > structs that reference it for that commit to compile.
> 
> It sounds like Stephen has provided you the info you needed so not
> doing any extra testing now, but I figured I'd chime in that I hit
> problems this morning with linux-next and it appears to be the same
> thing.  I did a revert of 9984d7394618df9 and (plus a revert of a
> handful of patches to the same file) and problems are resolved.  The
> failure case is really weird in that everything works well booting to
> a simple /bin/bash but fails when you do more complex tasks.
> 
> Anyway: If you need some extra testing feel free to CC me.

Folks, see if vfs.git#experimental works for you; the PITA had apparently
been caused by change of open() semantics for /proc/<pid>/fd/<some_pipe> -
it started to behave like a FIFO, i.e. wait for peer to show up.  Normally
that's not a problem, but if you have closed e.g. the write end of a pipe
and try to open /proc/<pid>/fd/<read_end_of_pipe>, you'll get open() waiting
for writers to appear.  Which isn't what we used to do here (open succeeded
immediately) and apparently that was enough to trip drakut.

Branch head should be at 574179469f7370aadb9cbac1ceca7c3723c17bee.

Re: hangs on boot in 9984d7394618df9

[Stephen Warren]

On 04/08/2013 05:06 PM, Al Viro wrote:
> On Mon, Apr 08, 2013 at 03:45:31PM -0700, Doug Anderson wrote:
>> Hi,
>>
>> On Mon, Apr 8, 2013 at 3:17 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>>> Anyway, I've just pushed a splitup of that commit (carved in 3 pieces)
>>>> into vfs.git#pipe-splitup; could you check which part triggers that
>>>> hang?  Should propagate in a few...
>>>
>>> It looks like "pipe: unify ->release() and ->open()" introduces the
>>> problem. Note that I had to add a prototype for fifo_open() before the
>>> structs that reference it for that commit to compile.
>>
>> It sounds like Stephen has provided you the info you needed so not
>> doing any extra testing now, but I figured I'd chime in that I hit
>> problems this morning with linux-next and it appears to be the same
>> thing.  I did a revert of 9984d7394618df9 and (plus a revert of a
>> handful of patches to the same file) and problems are resolved.  The
>> failure case is really weird in that everything works well booting to
>> a simple /bin/bash but fails when you do more complex tasks.
>>
>> Anyway: If you need some extra testing feel free to CC me.
> 
> Folks, see if vfs.git#experimental works for you; the PITA had apparently
> been caused by change of open() semantics for /proc/<pid>/fd/<some_pipe> -
> it started to behave like a FIFO, i.e. wait for peer to show up.  Normally
> that's not a problem, but if you have closed e.g. the write end of a pipe
> and try to open /proc/<pid>/fd/<read_end_of_pipe>, you'll get open() waiting
> for writers to appear.  Which isn't what we used to do here (open succeeded
> immediately) and apparently that was enough to trip drakut.
> 
> Branch head should be at 574179469f7370aadb9cbac1ceca7c3723c17bee.

That branch works.

Thanks!

Re: hangs on boot in 9984d7394618df9

[Doug Anderson]

Al,

On Mon, Apr 8, 2013 at 4:06 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> Folks, see if vfs.git#experimental works for you; the PITA had apparently
> been caused by change of open() semantics for /proc/<pid>/fd/<some_pipe> -
> it started to behave like a FIFO, i.e. wait for peer to show up.  Normally
> that's not a problem, but if you have closed e.g. the write end of a pipe
> and try to open /proc/<pid>/fd/<read_end_of_pipe>, you'll get open() waiting
> for writers to appear.  Which isn't what we used to do here (open succeeded
> immediately) and apparently that was enough to trip drakut.
>
> Branch head should be at 574179469f7370aadb9cbac1ceca7c3723c17bee.

That branch booted fine for me and didn't show any problems.

I wasn't easily able to merge onto linux-next and test there though.
I tried applying these the 4 top commits of your branch to
"next-20130408" and it didn't solve my problems.  A full merge of your
branch to linux-next showed conflicts and I didn't dig.

-Doug

Re: hangs on boot in 9984d7394618df9

[Nathan Zimmer]

On 04/08/2013 06:46 PM, Doug Anderson wrote:
> Al,
>
> On Mon, Apr 8, 2013 at 4:06 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
>> Folks, see if vfs.git#experimental works for you; the PITA had apparently
>> been caused by change of open() semantics for /proc/<pid>/fd/<some_pipe> -
>> it started to behave like a FIFO, i.e. wait for peer to show up.  Normally
>> that's not a problem, but if you have closed e.g. the write end of a pipe
>> and try to open /proc/<pid>/fd/<read_end_of_pipe>, you'll get open() waiting
>> for writers to appear.  Which isn't what we used to do here (open succeeded
>> immediately) and apparently that was enough to trip drakut.
>>
>> Branch head should be at 574179469f7370aadb9cbac1ceca7c3723c17bee.
> That branch booted fine for me and didn't show any problems.
>
> I wasn't easily able to merge onto linux-next and test there though.
> I tried applying these the 4 top commits of your branch to
> "next-20130408" and it didn't solve my problems.  A full merge of your
> branch to linux-next showed conflicts and I didn't dig.
>
> -Doug
I booted great for me too.
Also the numbers from the scaling test are improved also, at least on my 
128p box.
I'll verify on a 512 or larger box when I have the chance.
However I think I still should to provide some relief for the older kernels.

Re: hangs on boot in 9984d7394618df9

[Sedat Dilek]

The issue is fixed with Linux-Next (next-20130410).

- Sedat -

[1] http://marc.info/?l=linux-next&m=136559043516192&w=2