Two persistent problems

Two persistent problems

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2328 bytes --]

   Chris, Josef, anyone else who's interested,

   On IRC, I've been seeing reports of two persistent unsolved
problems. Neither is showing up very often, but both have turned up
often enough to indicate that there's something specific going on
worthy of investigation.

   One of them is definitely a btrfs problem. The other may be btrfs,
or something in the block layer, or just broken hardware; it's hard to
tell from where I sit.

Problem 1: ENOSPC on balance

   This has been going on since about March this year. I can
reasonably certainly recall 8-10 cases, possibly a number more. When
running a balance, the operation fails with ENOSPC when there's plenty
of space remaining unallocated. This happens on full balance, filtered
balance, and device delete. Other than the ENOSPC on balance, the FS
seems to work OK. It seems to be more prevalent on filesystems
converted from ext*. The first few or more reports of this didn't make
it to bugzilla, but a few of them since then have gone in.

Problem 2: Unexplained zeroes

   Failure to mount. Transid failure, "expected xyz, have 0". Chris
looked at an early one of these (for Ke, on IRC) back in September
(the 27th -- sadly, the public IRC logs aren't there for it, but I can
supply a copy of the private log). He rapidly came to the conclusion
that it was something bad going on with TRIM, replacing some blocks
with zeroes. Since then, I've seen a bunch of these coming past on
IRC. It seems to be a 3.17 thing. I can successfully predict the
presence of an SSD and -odiscard from the "have 0". I've successfully
persuaded several people to put this into bugzilla and capture
btrfs-images.  btrfs recover doesn't generally seem to be helpful in
recovering data.


   I think Josef had problem 1 in his sights, but I don't know if
additional images or reports are helpful at this point. For problem 2,
there's obviously something bad going on, but there's not much else to
go on -- and the inability to recover data isn't good.

   For each of these, what more information should I be trying to
collect from any future reporters?

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
            --- Great films about cricket:  Forrest Stump ---            

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Two persistent problems

[Josef Bacik]

On 11/14/2014 04:51 PM, Hugo Mills wrote:
>     Chris, Josef, anyone else who's interested,
>
>     On IRC, I've been seeing reports of two persistent unsolved
> problems. Neither is showing up very often, but both have turned up
> often enough to indicate that there's something specific going on
> worthy of investigation.
>
>     One of them is definitely a btrfs problem. The other may be btrfs,
> or something in the block layer, or just broken hardware; it's hard to
> tell from where I sit.
>
> Problem 1: ENOSPC on balance
>
>     This has been going on since about March this year. I can
> reasonably certainly recall 8-10 cases, possibly a number more. When
> running a balance, the operation fails with ENOSPC when there's plenty
> of space remaining unallocated. This happens on full balance, filtered
> balance, and device delete. Other than the ENOSPC on balance, the FS
> seems to work OK. It seems to be more prevalent on filesystems
> converted from ext*. The first few or more reports of this didn't make
> it to bugzilla, but a few of them since then have gone in.
>
> Problem 2: Unexplained zeroes
>
>     Failure to mount. Transid failure, "expected xyz, have 0". Chris
> looked at an early one of these (for Ke, on IRC) back in September
> (the 27th -- sadly, the public IRC logs aren't there for it, but I can
> supply a copy of the private log). He rapidly came to the conclusion
> that it was something bad going on with TRIM, replacing some blocks
> with zeroes. Since then, I've seen a bunch of these coming past on
> IRC. It seems to be a 3.17 thing. I can successfully predict the
> presence of an SSD and -odiscard from the "have 0". I've successfully
> persuaded several people to put this into bugzilla and capture
> btrfs-images.  btrfs recover doesn't generally seem to be helpful in
> recovering data.
>
>
>     I think Josef had problem 1 in his sights, but I don't know if
> additional images or reports are helpful at this point. For problem 2,
> there's obviously something bad going on, but there's not much else to
> go on -- and the inability to recover data isn't good.
>
>     For each of these, what more information should I be trying to
> collect from any future reporters?
>
>

So for #2 I've been looking at that the last two weeks.  I'm always 
paranoid we're screwing up one of our data integrity sort of things, 
either not waiting on IO to complete properly or something like that. 
I've built a dm target to be as evil as possible and have been running 
it trying to make bad things happen.  I got slightly side tracked since 
my stress test exposed a bug in the tree log stuff an csums which I just 
fixed.  Now that I've fixed that I'm going back to try and make the 
"expected blah, have 0" type errors happen.

As for the ENOSPC I keep meaning to look into it and I keep getting 
distracted with other more horrible things.  Ideally I'd like to 
reproduce it myself, so more info on that front would be good, like do 
all reports use RAID/compression/some other odd set of features?  Thanks 
for taking care of this stuff Hugo, #2 is the worst one and I'd like to 
be absolutely sure it's not our bug, once I'm happy we aren't I'll look 
at the balance thing.

Josef

Re: Two persistent problems

[Konstantin]

Josef Bacik wrote on 14.11.2014 at 23:00:
> On 11/14/2014 04:51 PM, Hugo Mills wrote:
>>     Chris, Josef, anyone else who's interested,
>>
>>     On IRC, I've been seeing reports of two persistent unsolved
>> problems. Neither is showing up very often, but both have turned up
>> often enough to indicate that there's something specific going on
>> worthy of investigation.
>>
>>     One of them is definitely a btrfs problem. The other may be btrfs,
>> or something in the block layer, or just broken hardware; it's hard to
>> tell from where I sit.
>>
>> Problem 1: ENOSPC on balance
>>
>>     This has been going on since about March this year. I can
>> reasonably certainly recall 8-10 cases, possibly a number more. When
>> running a balance, the operation fails with ENOSPC when there's plenty
>> of space remaining unallocated. This happens on full balance, filtered
>> balance, and device delete. Other than the ENOSPC on balance, the FS
>> seems to work OK. It seems to be more prevalent on filesystems
>> converted from ext*. The first few or more reports of this didn't make
>> it to bugzilla, but a few of them since then have gone in.
>>
>> Problem 2: Unexplained zeroes
>>
>>     Failure to mount. Transid failure, "expected xyz, have 0". Chris
>> looked at an early one of these (for Ke, on IRC) back in September
>> (the 27th -- sadly, the public IRC logs aren't there for it, but I can
>> supply a copy of the private log). He rapidly came to the conclusion
>> that it was something bad going on with TRIM, replacing some blocks
>> with zeroes. Since then, I've seen a bunch of these coming past on
>> IRC. It seems to be a 3.17 thing. I can successfully predict the
>> presence of an SSD and -odiscard from the "have 0". I've successfully
>> persuaded several people to put this into bugzilla and capture
>> btrfs-images.  btrfs recover doesn't generally seem to be helpful in
>> recovering data.
>>
>>
>>     I think Josef had problem 1 in his sights, but I don't know if
>> additional images or reports are helpful at this point. For problem 2,
>> there's obviously something bad going on, but there's not much else to
>> go on -- and the inability to recover data isn't good.
>>
>>     For each of these, what more information should I be trying to
>> collect from any future reporters?
>>
>>
>
> So for #2 I've been looking at that the last two weeks.  I'm always
> paranoid we're screwing up one of our data integrity sort of things,
> either not waiting on IO to complete properly or something like that.
> I've built a dm target to be as evil as possible and have been running
> it trying to make bad things happen.  I got slightly side tracked
> since my stress test exposed a bug in the tree log stuff an csums
> which I just fixed.  Now that I've fixed that I'm going back to try
> and make the "expected blah, have 0" type errors happen.
>
> As for the ENOSPC I keep meaning to look into it and I keep getting
> distracted with other more horrible things.  Ideally I'd like to
> reproduce it myself, so more info on that front would be good, like do
> all reports use RAID/compression/some other odd set of features? 
> Thanks for taking care of this stuff Hugo, #2 is the worst one and I'd
> like to be absolutely sure it's not our bug, once I'm happy we aren't
> I'll look at the balance thing.
>
> Josef

For #2, I had a strangely damaged BTRFS I reported a week or so ago
which may have similar background. Dmesg gives:

parent transid verify failed on 586239082496 wanted 13329746340512024838
found 588
BTRFS: open_ctree failed

The thing is that btrfsck crashes when trying to check this. As nobody
seemed to be interested I reformatted this disk today.

Re: Two persistent problems

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 4363 bytes --]

On Fri, Nov 14, 2014 at 05:00:26PM -0500, Josef Bacik wrote:
> On 11/14/2014 04:51 PM, Hugo Mills wrote:
> >    Chris, Josef, anyone else who's interested,
> >
> >    On IRC, I've been seeing reports of two persistent unsolved
> >problems. Neither is showing up very often, but both have turned up
> >often enough to indicate that there's something specific going on
> >worthy of investigation.
> >
> >    One of them is definitely a btrfs problem. The other may be btrfs,
> >or something in the block layer, or just broken hardware; it's hard to
> >tell from where I sit.
> >
> >Problem 1: ENOSPC on balance
> >
> >    This has been going on since about March this year. I can
> >reasonably certainly recall 8-10 cases, possibly a number more. When
> >running a balance, the operation fails with ENOSPC when there's plenty
> >of space remaining unallocated. This happens on full balance, filtered
> >balance, and device delete. Other than the ENOSPC on balance, the FS
> >seems to work OK. It seems to be more prevalent on filesystems
> >converted from ext*. The first few or more reports of this didn't make
> >it to bugzilla, but a few of them since then have gone in.
> >
> >Problem 2: Unexplained zeroes
> >
> >    Failure to mount. Transid failure, "expected xyz, have 0". Chris
> >looked at an early one of these (for Ke, on IRC) back in September
> >(the 27th -- sadly, the public IRC logs aren't there for it, but I can
> >supply a copy of the private log). He rapidly came to the conclusion
> >that it was something bad going on with TRIM, replacing some blocks
> >with zeroes. Since then, I've seen a bunch of these coming past on
> >IRC. It seems to be a 3.17 thing. I can successfully predict the
> >presence of an SSD and -odiscard from the "have 0". I've successfully
> >persuaded several people to put this into bugzilla and capture
> >btrfs-images.  btrfs recover doesn't generally seem to be helpful in
> >recovering data.
> >
> >
> >    I think Josef had problem 1 in his sights, but I don't know if
> >additional images or reports are helpful at this point. For problem 2,
> >there's obviously something bad going on, but there's not much else to
> >go on -- and the inability to recover data isn't good.
> >
> >    For each of these, what more information should I be trying to
> >collect from any future reporters?
> >
> >
> 
> So for #2 I've been looking at that the last two weeks.  I'm always
> paranoid we're screwing up one of our data integrity sort of things,
> either not waiting on IO to complete properly or something like
> that. I've built a dm target to be as evil as possible and have been
> running it trying to make bad things happen.  I got slightly side
> tracked since my stress test exposed a bug in the tree log stuff an
> csums which I just fixed.  Now that I've fixed that I'm going back
> to try and make the "expected blah, have 0" type errors happen.

   I've searched the bugzilla archive and found the two reports that I
know of (87061 and 87021); I couldn't see any others. I've requested
more information on both -- nothing obviously in common, except SSD
and (probably) discard. I tried to tag them both with "trim" for easy
finding, but that seems to have been lost somewhere. I'll try that
again when I get home this evening and have access to my password.

> As for the ENOSPC I keep meaning to look into it and I keep getting
> distracted with other more horrible things.  Ideally I'd like to
> reproduce it myself, so more info on that front would be good, like
> do all reports use RAID/compression/some other odd set of features?
> Thanks for taking care of this stuff Hugo, #2 is the worst one and
> I'd like to be absolutely sure it's not our bug, once I'm happy we
> aren't I'll look at the balance thing.

   OK, good to know you're on both of these. I think the "easy"
solution to reproduce the ENOSPC is to convert an ext4 filesystem. It
doesn't seem to be a unique characteristic, but it is a frequent
correlation. We had another one today, after an FS conversion -- I've
asked them to attach a btrfs-image dump and the enospc_debug log to
the bugzilla report.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
         --- 2 + 2 = 5,  for sufficiently large values of 2. ---         

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Two persistent problems

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2469 bytes --]

On Mon, Nov 17, 2014 at 11:59:48AM +0100, Konstantin wrote:
> Josef Bacik wrote on 14.11.2014 at 23:00:
> > On 11/14/2014 04:51 PM, Hugo Mills wrote:
[snip]
> >> Problem 2: Unexplained zeroes
> >>
> >>     Failure to mount. Transid failure, "expected xyz, have 0". Chris
> >> looked at an early one of these (for Ke, on IRC) back in September
> >> (the 27th -- sadly, the public IRC logs aren't there for it, but I can
> >> supply a copy of the private log). He rapidly came to the conclusion
> >> that it was something bad going on with TRIM, replacing some blocks
> >> with zeroes. Since then, I've seen a bunch of these coming past on
> >> IRC. It seems to be a 3.17 thing. I can successfully predict the
> >> presence of an SSD and -odiscard from the "have 0". I've successfully
> >> persuaded several people to put this into bugzilla and capture
> >> btrfs-images.  btrfs recover doesn't generally seem to be helpful in
> >> recovering data.
[snip]
> > So for #2 I've been looking at that the last two weeks.  I'm always
> > paranoid we're screwing up one of our data integrity sort of things,
> > either not waiting on IO to complete properly or something like that.
> > I've built a dm target to be as evil as possible and have been running
> > it trying to make bad things happen.  I got slightly side tracked
> > since my stress test exposed a bug in the tree log stuff an csums
> > which I just fixed.  Now that I've fixed that I'm going back to try
> > and make the "expected blah, have 0" type errors happen.
[snip]
> For #2, I had a strangely damaged BTRFS I reported a week or so ago
> which may have similar background. Dmesg gives:
> 
> parent transid verify failed on 586239082496 wanted 13329746340512024838
> found 588
> BTRFS: open_ctree failed
> 
> The thing is that btrfsck crashes when trying to check this. As nobody
> seemed to be interested I reformatted this disk today.

   Whilst that's a genuine problem, it's not specifically the one I
was referring to here, which shows up with "want=X, have=0" from btrfs
check, and seems to be related to TRIM on SSDs.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
    --- Turning,  pages turning in the widening bath, / The spine ---    
        cannot bear the humidity. / Books fall apart; the binding        
            cannot hold. / Page 129 is loosed upon the world.            

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Two persistent problems

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 3569 bytes --]

Am Fri, 14 Nov 2014 17:00:26 -0500
schrieb Josef Bacik <jbacik@fb.com>:

> On 11/14/2014 04:51 PM, Hugo Mills wrote:
> >     Chris, Josef, anyone else who's interested,
> >
> >     On IRC, I've been seeing reports of two persistent unsolved
> > problems. Neither is showing up very often, but both have turned up
> > often enough to indicate that there's something specific going on
> > worthy of investigation.
> >
> >     One of them is definitely a btrfs problem. The other may be btrfs,
> > or something in the block layer, or just broken hardware; it's hard to
> > tell from where I sit.
> >
> > Problem 1: ENOSPC on balance
> >
> >     This has been going on since about March this year. I can
> > reasonably certainly recall 8-10 cases, possibly a number more. When
> > running a balance, the operation fails with ENOSPC when there's plenty
> > of space remaining unallocated. This happens on full balance, filtered
> > balance, and device delete. Other than the ENOSPC on balance, the FS
> > seems to work OK. It seems to be more prevalent on filesystems
> > converted from ext*. The first few or more reports of this didn't make
> > it to bugzilla, but a few of them since then have gone in.
> >
> > Problem 2: Unexplained zeroes
> >
> >     Failure to mount. Transid failure, "expected xyz, have 0". Chris
> > looked at an early one of these (for Ke, on IRC) back in September
> > (the 27th -- sadly, the public IRC logs aren't there for it, but I can
> > supply a copy of the private log). He rapidly came to the conclusion
> > that it was something bad going on with TRIM, replacing some blocks
> > with zeroes. Since then, I've seen a bunch of these coming past on
> > IRC. It seems to be a 3.17 thing. I can successfully predict the
> > presence of an SSD and -odiscard from the "have 0". I've successfully
> > persuaded several people to put this into bugzilla and capture
> > btrfs-images.  btrfs recover doesn't generally seem to be helpful in
> > recovering data.
> >
> >
> >     I think Josef had problem 1 in his sights, but I don't know if
> > additional images or reports are helpful at this point. For problem 2,
> > there's obviously something bad going on, but there's not much else to
> > go on -- and the inability to recover data isn't good.
> >
> >     For each of these, what more information should I be trying to
> > collect from any future reporters?
> >
> >
> 
> So for #2 I've been looking at that the last two weeks.  I'm always 
> paranoid we're screwing up one of our data integrity sort of things, 
> either not waiting on IO to complete properly or something like that. 
> I've built a dm target to be as evil as possible and have been running 
> it trying to make bad things happen.  I got slightly side tracked since 
> my stress test exposed a bug in the tree log stuff an csums which I just 
> fixed.  Now that I've fixed that I'm going back to try and make the 
> "expected blah, have 0" type errors happen.

Just a quick question from a user: does Filipe's patch "Btrfs: fix race between
fs trimming and block group remove/allocation" fix this?  Judging by the commit
message, it looks like it.  If so, can you say whether it will make it into
3.17.x?

Maybe I'm being overly paranoid, but I stuck with 3.16.7 because of this.  (I
mean, I have backups, but there's no need to provoke a situation where I will
need them ;-) .)

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]