[Virtio-fs] xfstest generic/503 hangs

[Virtio-fs] xfstest generic/503 hangs

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 2626 bytes --]

Hi,

I have this bug report here:
https://bugzilla.redhat.com/show_bug.cgi?id=1813885

And I’m afraid I’m not really making progress on debugging it, so I was
wondering whether any of you might have some insights.

The problem is that the generic/503 xfstest hangs on virtio-fs.  Now, I
don’t know how the reporter got that test to run in the first place,
because for me, it requires fcollapse and fzero, which as far as I can
tell are currently not supported for virtio-fs.

So I first had to disable those requirements, and then let the helper
program (src/t_mmap_collision.c) not test those operations.

Then, the test hangs.  What I could find out so far is that the hang
occurs in src/t_mmap_collision.c’s truncate_down_fn() (run through
run_test(&truncate_down_fn), namely in one of the pread()s.  I can also
see that some of the pread()s before fail with EFAULT.

A bit more context: t_mmap_collision.c opens a test file twice (I think
the idea is that you open it once on an FS with DAX, and once without,
but AFAIU it should work either way).  For the relevant test, it mmap()s
the DAX FD, truncates it, then fallocates it to increase the size again.
 Then it reads from the non-DAX FD.

It does all of that in two threads simultaneously for a second.

The EFAULT seems to come from the guest kernel.  I don’t see virtiofsd
returning an error anywhere.  I don’t know where it comes from exactly,
only that when I replace all occurrences of “EFAULT” by e.g. “EBADSLT”
in mm/, the test crashes instead of hanging, so I take that to mean that
the error comes from something in mm/ (which I suppose isn’t too
unexpected).

The test passes if running the test function in a single thread instead
of two, or if you use a separate TEST_DEV and SCRATCH_DEV – but in the
latter case, you really have two separate files, so the test becomes
rather moot (AFAIU).

The fact that truncate_down_fn() uses fallocate() seems irrelevant.
When you replace it by ftruncate() (i.e. the dax_fd is just first
truncated to 0, and then truncated back to @file_size), the test fails
in the same way.  So maybe there is some interaction between the
ftruncate() and a concurrent pread()?  But where does the EFAULT come from?

Does anyone have any spontaneous ideas? :/


In any case, thanks already for reading this,

Max


(I suppose my plan now is that instead of debugging the kernel further,
I should come up with a simpler reproducer, to see whether the problem
is really just a concurrent ftruncate() + pread() on two FDs that point
to the same file.)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Virtio-fs] xfstest generic/503 hangs

[Liu Bo]

On Mon, Mar 23, 2020 at 07:18:57PM +0100, Max Reitz wrote:
> Hi,
> 
> I have this bug report here:
> https://bugzilla.redhat.com/show_bug.cgi?id=1813885
> 
> And I’m afraid I’m not really making progress on debugging it, so I was
> wondering whether any of you might have some insights.
> 
> The problem is that the generic/503 xfstest hangs on virtio-fs.  Now, I
> don’t know how the reporter got that test to run in the first place,
> because for me, it requires fcollapse and fzero, which as far as I can
> tell are currently not supported for virtio-fs.
> 
> So I first had to disable those requirements, and then let the helper
> program (src/t_mmap_collision.c) not test those operations.
> 
> Then, the test hangs.  What I could find out so far is that the hang
> occurs in src/t_mmap_collision.c’s truncate_down_fn() (run through
> run_test(&truncate_down_fn), namely in one of the pread()s.  I can also
> see that some of the pread()s before fail with EFAULT.
> 
> A bit more context: t_mmap_collision.c opens a test file twice (I think
> the idea is that you open it once on an FS with DAX, and once without,
> but AFAIU it should work either way).  For the relevant test, it mmap()s
> the DAX FD, truncates it, then fallocates it to increase the size again.
>  Then it reads from the non-DAX FD.
> 
> It does all of that in two threads simultaneously for a second.
> 
> The EFAULT seems to come from the guest kernel.  I don’t see virtiofsd
> returning an error anywhere.  I don’t know where it comes from exactly,
> only that when I replace all occurrences of “EFAULT” by e.g. “EBADSLT”
> in mm/, the test crashes instead of hanging, so I take that to mean that
> the error comes from something in mm/ (which I suppose isn’t too
> unexpected).
> 
> The test passes if running the test function in a single thread instead
> of two, or if you use a separate TEST_DEV and SCRATCH_DEV – but in the
> latter case, you really have two separate files, so the test becomes
> rather moot (AFAIU).
> 
> The fact that truncate_down_fn() uses fallocate() seems irrelevant.
> When you replace it by ftruncate() (i.e. the dax_fd is just first
> truncated to 0, and then truncated back to @file_size), the test fails
> in the same way.  So maybe there is some interaction between the
> ftruncate() and a concurrent pread()?  But where does the EFAULT come from?
> 
> Does anyone have any spontaneous ideas? :/
>

When it comes to "hang", I guess it's a similar problem of guest dax
reading while host truncates the file, this can be verify by the hang
stack, did it hang at dax_iomap_actor() -> dax_copy_to_iter()?

I think we did use fuse_break_layout() to avoid pinned blocks being
truncated underneath in this kind of mmap problems, but seems it
doesn't work as expected.

thanks,
-liubo

Re: [Virtio-fs] xfstest generic/503 hangs

[Dr. David Alan Gilbert]

* Max Reitz (mreitz@redhat.com) wrote:
> Hi,
> 
> I have this bug report here:
> https://bugzilla.redhat.com/show_bug.cgi?id=1813885
> 
> And I’m afraid I’m not really making progress on debugging it, so I was
> wondering whether any of you might have some insights.
> 
> The problem is that the generic/503 xfstest hangs on virtio-fs.  Now, I
> don’t know how the reporter got that test to run in the first place,
> because for me, it requires fcollapse and fzero, which as far as I can
> tell are currently not supported for virtio-fs.
> 
> So I first had to disable those requirements, and then let the helper
> program (src/t_mmap_collision.c) not test those operations.
> 
> Then, the test hangs.  What I could find out so far is that the hang
> occurs in src/t_mmap_collision.c’s truncate_down_fn() (run through
> run_test(&truncate_down_fn), namely in one of the pread()s.  I can also
> see that some of the pread()s before fail with EFAULT.
> 
> A bit more context: t_mmap_collision.c opens a test file twice (I think
> the idea is that you open it once on an FS with DAX, and once without,
> but AFAIU it should work either way).  For the relevant test, it mmap()s
> the DAX FD, truncates it, then fallocates it to increase the size again.
>  Then it reads from the non-DAX FD.

Can you just confirm where the DAX is happening here?  As I read that bz
entry it's using the qemu which doesn't have DAX code yet.

Dave

> It does all of that in two threads simultaneously for a second.
> 
> The EFAULT seems to come from the guest kernel.  I don’t see virtiofsd
> returning an error anywhere.  I don’t know where it comes from exactly,
> only that when I replace all occurrences of “EFAULT” by e.g. “EBADSLT”
> in mm/, the test crashes instead of hanging, so I take that to mean that
> the error comes from something in mm/ (which I suppose isn’t too
> unexpected).
> 
> The test passes if running the test function in a single thread instead
> of two, or if you use a separate TEST_DEV and SCRATCH_DEV – but in the
> latter case, you really have two separate files, so the test becomes
> rather moot (AFAIU).
> 
> The fact that truncate_down_fn() uses fallocate() seems irrelevant.
> When you replace it by ftruncate() (i.e. the dax_fd is just first
> truncated to 0, and then truncated back to @file_size), the test fails
> in the same way.  So maybe there is some interaction between the
> ftruncate() and a concurrent pread()?  But where does the EFAULT come from?
> 
> Does anyone have any spontaneous ideas? :/
> 
> 
> In any case, thanks already for reading this,
> 
> Max
> 
> 
> (I suppose my plan now is that instead of debugging the kernel further,
> I should come up with a simpler reproducer, to see whether the problem
> is really just a concurrent ftruncate() + pread() on two FDs that point
> to the same file.)
> 




> _______________________________________________
> Virtio-fs mailing list
> Virtio-fs@redhat.com
> https://www.redhat.com/mailman/listinfo/virtio-fs

--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Virtio-fs] xfstest generic/503 hangs

[Liu Bo]

On Tue, Mar 24, 2020 at 02:40:23AM +0800, Liu Bo wrote:
> On Mon, Mar 23, 2020 at 07:18:57PM +0100, Max Reitz wrote:
> > Hi,
> > 
> > I have this bug report here:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1813885
> > 
> > And I’m afraid I’m not really making progress on debugging it, so I was
> > wondering whether any of you might have some insights.
> > 
> > The problem is that the generic/503 xfstest hangs on virtio-fs.  Now, I
> > don’t know how the reporter got that test to run in the first place,
> > because for me, it requires fcollapse and fzero, which as far as I can
> > tell are currently not supported for virtio-fs.
> > 
> > So I first had to disable those requirements, and then let the helper
> > program (src/t_mmap_collision.c) not test those operations.
> > 
> > Then, the test hangs.  What I could find out so far is that the hang
> > occurs in src/t_mmap_collision.c’s truncate_down_fn() (run through
> > run_test(&truncate_down_fn), namely in one of the pread()s.  I can also
> > see that some of the pread()s before fail with EFAULT.
> > 
> > A bit more context: t_mmap_collision.c opens a test file twice (I think
> > the idea is that you open it once on an FS with DAX, and once without,
> > but AFAIU it should work either way).  For the relevant test, it mmap()s
> > the DAX FD, truncates it, then fallocates it to increase the size again.
> >  Then it reads from the non-DAX FD.
> > 
> > It does all of that in two threads simultaneously for a second.
> > 
> > The EFAULT seems to come from the guest kernel.  I don’t see virtiofsd
> > returning an error anywhere.  I don’t know where it comes from exactly,
> > only that when I replace all occurrences of “EFAULT” by e.g. “EBADSLT”
> > in mm/, the test crashes instead of hanging, so I take that to mean that
> > the error comes from something in mm/ (which I suppose isn’t too
> > unexpected).
> > 
> > The test passes if running the test function in a single thread instead
> > of two, or if you use a separate TEST_DEV and SCRATCH_DEV – but in the
> > latter case, you really have two separate files, so the test becomes
> > rather moot (AFAIU).
> > 
> > The fact that truncate_down_fn() uses fallocate() seems irrelevant.
> > When you replace it by ftruncate() (i.e. the dax_fd is just first
> > truncated to 0, and then truncated back to @file_size), the test fails
> > in the same way.  So maybe there is some interaction between the
> > ftruncate() and a concurrent pread()?  But where does the EFAULT come from?
> > 
> > Does anyone have any spontaneous ideas? :/
> >
> 
> When it comes to "hang", I guess it's a similar problem of guest dax
> reading while host truncates the file, this can be verify by the hang
> stack, did it hang at dax_iomap_actor() -> dax_copy_to_iter()?
> 
> I think we did use fuse_break_layout() to avoid pinned blocks being
> truncated underneath in this kind of mmap problems, but seems it
> doesn't work as expected.

I spoke too quickly, the stack in bz is not about dax hang I described above.

thanks,
-liubo

Re: [Virtio-fs] xfstest generic/503 hangs

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 1930 bytes --]

On 23.03.20 19:43, Dr. David Alan Gilbert wrote:
> * Max Reitz (mreitz@redhat.com) wrote:
>> Hi,
>>
>> I have this bug report here:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1813885
>>
>> And I’m afraid I’m not really making progress on debugging it, so I was
>> wondering whether any of you might have some insights.
>>
>> The problem is that the generic/503 xfstest hangs on virtio-fs.  Now, I
>> don’t know how the reporter got that test to run in the first place,
>> because for me, it requires fcollapse and fzero, which as far as I can
>> tell are currently not supported for virtio-fs.
>>
>> So I first had to disable those requirements, and then let the helper
>> program (src/t_mmap_collision.c) not test those operations.
>>
>> Then, the test hangs.  What I could find out so far is that the hang
>> occurs in src/t_mmap_collision.c’s truncate_down_fn() (run through
>> run_test(&truncate_down_fn), namely in one of the pread()s.  I can also
>> see that some of the pread()s before fail with EFAULT.
>>
>> A bit more context: t_mmap_collision.c opens a test file twice (I think
>> the idea is that you open it once on an FS with DAX, and once without,
>> but AFAIU it should work either way).  For the relevant test, it mmap()s
>> the DAX FD, truncates it, then fallocates it to increase the size again.
>>  Then it reads from the non-DAX FD.
> 
> Can you just confirm where the DAX is happening here?  As I read that bz
> entry it's using the qemu which doesn't have DAX code yet.

I actually don’t know whether it really uses DAX.  When I say “DAX” here
(and “dax_fd”), I mean it in the spirit of the test, which tries to open
a file once with DAX and once without.  But it isn’t like it verifies
that the first instance actually uses DAX.

So I think it rather likely that it doesn’t use DAX, and just two FDs
for a single file concurrently.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Virtio-fs] xfstest generic/503 hangs

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 3219 bytes --]

On 23.03.20 20:12, Liu Bo wrote:
> On Tue, Mar 24, 2020 at 02:40:23AM +0800, Liu Bo wrote:
>> On Mon, Mar 23, 2020 at 07:18:57PM +0100, Max Reitz wrote:
>>> Hi,
>>>
>>> I have this bug report here:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1813885
>>>
>>> And I’m afraid I’m not really making progress on debugging it, so I was
>>> wondering whether any of you might have some insights.
>>>
>>> The problem is that the generic/503 xfstest hangs on virtio-fs.  Now, I
>>> don’t know how the reporter got that test to run in the first place,
>>> because for me, it requires fcollapse and fzero, which as far as I can
>>> tell are currently not supported for virtio-fs.
>>>
>>> So I first had to disable those requirements, and then let the helper
>>> program (src/t_mmap_collision.c) not test those operations.
>>>
>>> Then, the test hangs.  What I could find out so far is that the hang
>>> occurs in src/t_mmap_collision.c’s truncate_down_fn() (run through
>>> run_test(&truncate_down_fn), namely in one of the pread()s.  I can also
>>> see that some of the pread()s before fail with EFAULT.
>>>
>>> A bit more context: t_mmap_collision.c opens a test file twice (I think
>>> the idea is that you open it once on an FS with DAX, and once without,
>>> but AFAIU it should work either way).  For the relevant test, it mmap()s
>>> the DAX FD, truncates it, then fallocates it to increase the size again.
>>>  Then it reads from the non-DAX FD.
>>>
>>> It does all of that in two threads simultaneously for a second.
>>>
>>> The EFAULT seems to come from the guest kernel.  I don’t see virtiofsd
>>> returning an error anywhere.  I don’t know where it comes from exactly,
>>> only that when I replace all occurrences of “EFAULT” by e.g. “EBADSLT”
>>> in mm/, the test crashes instead of hanging, so I take that to mean that
>>> the error comes from something in mm/ (which I suppose isn’t too
>>> unexpected).
>>>
>>> The test passes if running the test function in a single thread instead
>>> of two, or if you use a separate TEST_DEV and SCRATCH_DEV – but in the
>>> latter case, you really have two separate files, so the test becomes
>>> rather moot (AFAIU).
>>>
>>> The fact that truncate_down_fn() uses fallocate() seems irrelevant.
>>> When you replace it by ftruncate() (i.e. the dax_fd is just first
>>> truncated to 0, and then truncated back to @file_size), the test fails
>>> in the same way.  So maybe there is some interaction between the
>>> ftruncate() and a concurrent pread()?  But where does the EFAULT come from?
>>>
>>> Does anyone have any spontaneous ideas? :/
>>>
>>
>> When it comes to "hang", I guess it's a similar problem of guest dax
>> reading while host truncates the file,

That does sound similar...

>> this can be verify by the hang
>> stack, did it hang at dax_iomap_actor() -> dax_copy_to_iter()?
>>
>> I think we did use fuse_break_layout() to avoid pinned blocks being
>> truncated underneath in this kind of mmap problems, but seems it
>> doesn't work as expected.
> 
> I spoke too quickly, the stack in bz is not about dax hang I described above.

...but then maybe not. :-/

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Virtio-fs] xfstest generic/503 hangs

[Dr. David Alan Gilbert]

* Max Reitz (mreitz@redhat.com) wrote:
> On 23.03.20 19:43, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mreitz@redhat.com) wrote:
> >> Hi,
> >>
> >> I have this bug report here:
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1813885
> >>
> >> And I’m afraid I’m not really making progress on debugging it, so I was
> >> wondering whether any of you might have some insights.
> >>
> >> The problem is that the generic/503 xfstest hangs on virtio-fs.  Now, I
> >> don’t know how the reporter got that test to run in the first place,
> >> because for me, it requires fcollapse and fzero, which as far as I can
> >> tell are currently not supported for virtio-fs.
> >>
> >> So I first had to disable those requirements, and then let the helper
> >> program (src/t_mmap_collision.c) not test those operations.
> >>
> >> Then, the test hangs.  What I could find out so far is that the hang
> >> occurs in src/t_mmap_collision.c’s truncate_down_fn() (run through
> >> run_test(&truncate_down_fn), namely in one of the pread()s.  I can also
> >> see that some of the pread()s before fail with EFAULT.
> >>
> >> A bit more context: t_mmap_collision.c opens a test file twice (I think
> >> the idea is that you open it once on an FS with DAX, and once without,
> >> but AFAIU it should work either way).  For the relevant test, it mmap()s
> >> the DAX FD, truncates it, then fallocates it to increase the size again.
> >>  Then it reads from the non-DAX FD.
> > 
> > Can you just confirm where the DAX is happening here?  As I read that bz
> > entry it's using the qemu which doesn't have DAX code yet.
> 
> I actually don’t know whether it really uses DAX.  When I say “DAX” here
> (and “dax_fd”), I mean it in the spirit of the test, which tries to open
> a file once with DAX and once without.  But it isn’t like it verifies
> that the first instance actually uses DAX.
> 
> So I think it rather likely that it doesn’t use DAX, and just two FDs
> for a single file concurrently.

OK, I don't think DAX is really involved.

Does Liu Bo's: 
https://www.redhat.com/archives/virtio-fs/2020-March/msg00074.html

help?

Dave

> Max
> 



--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Virtio-fs] xfstest generic/503 hangs

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 2474 bytes --]

On 24.03.20 20:52, Dr. David Alan Gilbert wrote:
> * Max Reitz (mreitz@redhat.com) wrote:
>> On 23.03.20 19:43, Dr. David Alan Gilbert wrote:
>>> * Max Reitz (mreitz@redhat.com) wrote:
>>>> Hi,
>>>>
>>>> I have this bug report here:
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1813885
>>>>
>>>> And I’m afraid I’m not really making progress on debugging it, so I was
>>>> wondering whether any of you might have some insights.
>>>>
>>>> The problem is that the generic/503 xfstest hangs on virtio-fs.  Now, I
>>>> don’t know how the reporter got that test to run in the first place,
>>>> because for me, it requires fcollapse and fzero, which as far as I can
>>>> tell are currently not supported for virtio-fs.
>>>>
>>>> So I first had to disable those requirements, and then let the helper
>>>> program (src/t_mmap_collision.c) not test those operations.
>>>>
>>>> Then, the test hangs.  What I could find out so far is that the hang
>>>> occurs in src/t_mmap_collision.c’s truncate_down_fn() (run through
>>>> run_test(&truncate_down_fn), namely in one of the pread()s.  I can also
>>>> see that some of the pread()s before fail with EFAULT.
>>>>
>>>> A bit more context: t_mmap_collision.c opens a test file twice (I think
>>>> the idea is that you open it once on an FS with DAX, and once without,
>>>> but AFAIU it should work either way).  For the relevant test, it mmap()s
>>>> the DAX FD, truncates it, then fallocates it to increase the size again.
>>>>  Then it reads from the non-DAX FD.
>>>
>>> Can you just confirm where the DAX is happening here?  As I read that bz
>>> entry it's using the qemu which doesn't have DAX code yet.
>>
>> I actually don’t know whether it really uses DAX.  When I say “DAX” here
>> (and “dax_fd”), I mean it in the spirit of the test, which tries to open
>> a file once with DAX and once without.  But it isn’t like it verifies
>> that the first instance actually uses DAX.
>>
>> So I think it rather likely that it doesn’t use DAX, and just two FDs
>> for a single file concurrently.
> 
> OK, I don't think DAX is really involved.
> 
> Does Liu Bo's: 
> https://www.redhat.com/archives/virtio-fs/2020-March/msg00074.html
> 
> help?

No, unfortunately it doesn’t.

(I tried it last week, I was looking for patches that might be related;
the only other thing I tried was
https://www.redhat.com/archives/virtio-fs/2020-March/msg00002.html)

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]