* Ran into "invalid block group size" bug, unclear how to proceed. @ 2018-12-04 3:32 Mike Javorski 2018-12-04 4:00 ` Mike Javorski ` (2 more replies) 0 siblings, 3 replies; 8+ messages in thread From: Mike Javorski @ 2018-12-04 3:32 UTC (permalink / raw) To: linux-btrfs Need a bit of advice here ladies / gents. I am running into an issue which Qu Wenruo seems to have posted a patch for several weeks ago (see https://patchwork.kernel.org/patch/10694997/). Here is the relevant dmesg output which led me to Qu's patch. ---- [ 10.032475] BTRFS critical (device sdb): corrupt leaf: root=2 block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104, invalid block group size, have 10804527104 expect (0, 10737418240] [ 10.032493] BTRFS error (device sdb): failed to read block groups: -5 [ 10.053365] BTRFS error (device sdb): open_ctree failed ---- This server has a 16 disk btrfs filesystem (RAID6) which I boot periodically to btrfs-send snapshots to. This machine is running ArchLinux and I had just updated to their latest 4.19.4 kernel package (from 4.18.10 which was working fine). I've tried updating to the 4.19.6 kernel that is in testing, but that doesn't seem to resolve the issue. From what I can see on kernel.org, the patch above is not pushed to stable or to Linus' tree. At this point the question is what to do. Is my FS toast? Could I revert to the 4.18.10 kernel and boot safely? I don't know if the 4.19 boot process may have flipped some bits which would make reverting problematic. Thanks much, - mike ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ran into "invalid block group size" bug, unclear how to proceed. 2018-12-04 3:32 Ran into "invalid block group size" bug, unclear how to proceed Mike Javorski @ 2018-12-04 4:00 ` Mike Javorski 2018-12-04 5:56 ` Chris Murphy 2018-12-04 10:18 ` Qu Wenruo 2 siblings, 0 replies; 8+ messages in thread From: Mike Javorski @ 2018-12-04 4:00 UTC (permalink / raw) To: linux-btrfs Apologies for not scouring the mailing list completely (just subscribed in fact) as It appears that Patrick Dijkgraaf also ran into this issue. He went ahead with a volume rebuild, whereas I am hoping I can recover having not run anything more than a "btrfs scan" and "btrfs device ready" on this machine to this point since the kernel upgrade and the FS was intact up to that point. - mike On Mon, Dec 3, 2018 at 7:32 PM Mike Javorski <mike.javorski@gmail.com> wrote: > > Need a bit of advice here ladies / gents. I am running into an issue > which Qu Wenruo seems to have posted a patch for several weeks ago > (see https://patchwork.kernel.org/patch/10694997/). > > Here is the relevant dmesg output which led me to Qu's patch. > ---- > [ 10.032475] BTRFS critical (device sdb): corrupt leaf: root=2 > block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104, > invalid block group size, have 10804527104 expect (0, 10737418240] > [ 10.032493] BTRFS error (device sdb): failed to read block groups: -5 > [ 10.053365] BTRFS error (device sdb): open_ctree failed > ---- > > This server has a 16 disk btrfs filesystem (RAID6) which I boot > periodically to btrfs-send snapshots to. This machine is running > ArchLinux and I had just updated to their latest 4.19.4 kernel > package (from 4.18.10 which was working fine). I've tried updating to > the 4.19.6 kernel that is in testing, but that doesn't seem to resolve > the issue. From what I can see on kernel.org, the patch above is not > pushed to stable or to Linus' tree. > > At this point the question is what to do. Is my FS toast? Could I > revert to the 4.18.10 kernel and boot safely? I don't know if the 4.19 > boot process may have flipped some bits which would make reverting > problematic. > > Thanks much, > > - mike ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ran into "invalid block group size" bug, unclear how to proceed. 2018-12-04 3:32 Ran into "invalid block group size" bug, unclear how to proceed Mike Javorski 2018-12-04 4:00 ` Mike Javorski @ 2018-12-04 5:56 ` Chris Murphy 2018-12-04 6:46 ` Mike Javorski 2018-12-04 10:18 ` Qu Wenruo 2 siblings, 1 reply; 8+ messages in thread From: Chris Murphy @ 2018-12-04 5:56 UTC (permalink / raw) To: mike.javorski; +Cc: Btrfs BTRFS On Mon, Dec 3, 2018 at 8:32 PM Mike Javorski <mike.javorski@gmail.com> wrote: > > Need a bit of advice here ladies / gents. I am running into an issue > which Qu Wenruo seems to have posted a patch for several weeks ago > (see https://patchwork.kernel.org/patch/10694997/). > > Here is the relevant dmesg output which led me to Qu's patch. > ---- > [ 10.032475] BTRFS critical (device sdb): corrupt leaf: root=2 > block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104, > invalid block group size, have 10804527104 expect (0, 10737418240] > [ 10.032493] BTRFS error (device sdb): failed to read block groups: -5 > [ 10.053365] BTRFS error (device sdb): open_ctree failed > ---- > > This server has a 16 disk btrfs filesystem (RAID6) which I boot > periodically to btrfs-send snapshots to. This machine is running > ArchLinux and I had just updated to their latest 4.19.4 kernel > package (from 4.18.10 which was working fine). I've tried updating to > the 4.19.6 kernel that is in testing, but that doesn't seem to resolve > the issue. From what I can see on kernel.org, the patch above is not > pushed to stable or to Linus' tree. > > At this point the question is what to do. Is my FS toast? Could I > revert to the 4.18.10 kernel and boot safely? I don't know if the 4.19 > boot process may have flipped some bits which would make reverting > problematic. That patch is not yet merged in linux-next so to use it, you'd need to apply yourself and compile a kernel. I can't tell for sure if it'd help. But, the less you change the file system, the better chance of saving it. I have no idea why there'd be a corrupt leaf just due to a kernel version change, though. Needless to say, raid56 just seems fragile once it runs into any kind of trouble. I personally wouldn't boot off it at all. I would only mount it from another system, ideally an installed system but a live system with the kernel versions you need would also work. That way you can get more information without changes, and booting will almost immediately mount rw, if mount succeeds at all, and will write a bunch of changes to the file system. Whether it's a case of 4.18.10 not detecting corruption that 4.19 sees, or if 4.19 already caused it, the best chance is to not mount it rw, and not run check --repair, until you get some feedback from a developer. The thing I'd like to see is # btrfs rescue super -v /anydevice/ # btrfs insp dump-s -f /anydevice/ First command will tell us if all the supers are the same and valid across all devices. And the second one, hopefully it's pointed to a device with valid super, will tell us if there's a log root value other than 0. Both of those are read only commands. -- Chris Murphy ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ran into "invalid block group size" bug, unclear how to proceed. 2018-12-04 5:56 ` Chris Murphy @ 2018-12-04 6:46 ` Mike Javorski 0 siblings, 0 replies; 8+ messages in thread From: Mike Javorski @ 2018-12-04 6:46 UTC (permalink / raw) To: lists; +Cc: linux-btrfs Apologies for the dupe Chris, I neglected to hit Reply-All.. Comments below. On Mon, Dec 3, 2018 at 9:56 PM Chris Murphy <lists@colorremedies.com> wrote: > > On Mon, Dec 3, 2018 at 8:32 PM Mike Javorski <mike.javorski@gmail.com> wrote: > > > > Need a bit of advice here ladies / gents. I am running into an issue > > which Qu Wenruo seems to have posted a patch for several weeks ago > > (see https://patchwork.kernel.org/patch/10694997/). > > > > Here is the relevant dmesg output which led me to Qu's patch. > > ---- > > [ 10.032475] BTRFS critical (device sdb): corrupt leaf: root=2 > > block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104, > > invalid block group size, have 10804527104 expect (0, 10737418240] > > [ 10.032493] BTRFS error (device sdb): failed to read block groups: -5 > > [ 10.053365] BTRFS error (device sdb): open_ctree failed > > ---- > > > > This server has a 16 disk btrfs filesystem (RAID6) which I boot > > periodically to btrfs-send snapshots to. This machine is running > > ArchLinux and I had just updated to their latest 4.19.4 kernel > > package (from 4.18.10 which was working fine). I've tried updating to > > the 4.19.6 kernel that is in testing, but that doesn't seem to resolve > > the issue. From what I can see on kernel.org, the patch above is not > > pushed to stable or to Linus' tree. > > > > At this point the question is what to do. Is my FS toast? Could I > > revert to the 4.18.10 kernel and boot safely? I don't know if the 4.19 > > boot process may have flipped some bits which would make reverting > > problematic. > > That patch is not yet merged in linux-next so to use it, you'd need to > apply yourself and compile a kernel. I can't tell for sure if it'd > help. > > But, the less you change the file system, the better chance of saving > it. I have no idea why there'd be a corrupt leaf just due to a kernel > version change, though. > > Needless to say, raid56 just seems fragile once it runs into any kind > of trouble. I personally wouldn't boot off it at all. I would only > mount it from another system, ideally an installed system but a live > system with the kernel versions you need would also work. That way you > can get more information without changes, and booting will almost > immediately mount rw, if mount succeeds at all, and will write a bunch > of changes to the file system. > If the boot could corrupt the disk, that ship has already sailed as I have previously attempted to mount the volume with the 4.19.4 and 4.19.6 kernels, but they both failed reporting the log lines in my original message. I am hoping that Qu notices this thread at some point as they are the author of the original patch which introduced the check which is now failing, as well as the un-merged patch linked earlier that adjusts the check condition. What I don't know is if the checks up until that mount failure have been read-only and thus I can revert to the older kernel, or if something would have been written to disk prior to the mount call failing. I don't want to risk the 23 TiB of snapshot data stored if it's an easy workaround :-). I realize there are risks with the RAID56 code, but I've done my best to mitigate with server-grade hardware, ECC memory, a UPS and a redundant copy of the data via btrfs-send to this machine. Losing this snapshot volume is not the end of the world, but I am about to upgrade the primary server (which is currently running 4.19.4 without issue btw) and want to have a best effort snapshot/backup in place before I do so. > Whether it's a case of 4.18.10 not detecting corruption that 4.19 > sees, or if 4.19 already caused it, the best chance is to not mount it > rw, and not run check --repair, until you get some feedback from a > developer. > +1 > The thing I'd like to see is > # btrfs rescue super -v /anydevice/ > # btrfs insp dump-s -f /anydevice/ > > First command will tell us if all the supers are the same and valid > across all devices. And the second one, hopefully it's pointed to a > device with valid super, will tell us if there's a log root value > other than 0. Both of those are read only commands. > It was my understanding that rescue writes to the disk (the man page indicates it recovers superblocks which would imply writes). If you are sure they both leave the on-disk data structures completely intact, I would be willing to run them. - mike ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ran into "invalid block group size" bug, unclear how to proceed. 2018-12-04 3:32 Ran into "invalid block group size" bug, unclear how to proceed Mike Javorski 2018-12-04 4:00 ` Mike Javorski 2018-12-04 5:56 ` Chris Murphy @ 2018-12-04 10:18 ` Qu Wenruo 2018-12-04 22:33 ` Mike Javorski 2 siblings, 1 reply; 8+ messages in thread From: Qu Wenruo @ 2018-12-04 10:18 UTC (permalink / raw) To: Mike Javorski, linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 1927 bytes --] On 2018/12/4 上午11:32, Mike Javorski wrote: > Need a bit of advice here ladies / gents. I am running into an issue > which Qu Wenruo seems to have posted a patch for several weeks ago > (see https://patchwork.kernel.org/patch/10694997/). > > Here is the relevant dmesg output which led me to Qu's patch. > ---- > [ 10.032475] BTRFS critical (device sdb): corrupt leaf: root=2 > block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104, > invalid block group size, have 10804527104 expect (0, 10737418240] > [ 10.032493] BTRFS error (device sdb): failed to read block groups: -5 > [ 10.053365] BTRFS error (device sdb): open_ctree failed > ---- Exactly the same symptom. > > This server has a 16 disk btrfs filesystem (RAID6) which I boot > periodically to btrfs-send snapshots to. This machine is running > ArchLinux and I had just updated to their latest 4.19.4 kernel > package (from 4.18.10 which was working fine). I've tried updating to > the 4.19.6 kernel that is in testing, but that doesn't seem to resolve > the issue. From what I can see on kernel.org, the patch above is not > pushed to stable or to Linus' tree. > > At this point the question is what to do. Is my FS toast? If there is no other problem at all, your fs is just fine. It's my original patch too sensitive (the excuse for not checking chunk allocator carefully enough). But since you have the down time, it's never a bad idea to run a btrfs check --readonly to see if your fs is really OK. > Could I > revert to the 4.18.10 kernel and boot safely? If your btrfs check --readonly doesn't report any problem, then you're completely fine to do so. Although I still recommend to go RAID10 other than RAID5/6. Thanks, Qu > I don't know if the 4.19 > boot process may have flipped some bits which would make reverting > problematic. > > Thanks much, > > - mike > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ran into "invalid block group size" bug, unclear how to proceed. 2018-12-04 10:18 ` Qu Wenruo @ 2018-12-04 22:33 ` Mike Javorski 2018-12-05 5:24 ` Qu Wenruo 0 siblings, 1 reply; 8+ messages in thread From: Mike Javorski @ 2018-12-04 22:33 UTC (permalink / raw) To: quwenruo.btrfs; +Cc: linux-btrfs On Tue, Dec 4, 2018 at 2:18 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote: > > > > On 2018/12/4 上午11:32, Mike Javorski wrote: > > Need a bit of advice here ladies / gents. I am running into an issue > > which Qu Wenruo seems to have posted a patch for several weeks ago > > (see https://patchwork.kernel.org/patch/10694997/). > > > > Here is the relevant dmesg output which led me to Qu's patch. > > ---- > > [ 10.032475] BTRFS critical (device sdb): corrupt leaf: root=2 > > block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104, > > invalid block group size, have 10804527104 expect (0, 10737418240] > > [ 10.032493] BTRFS error (device sdb): failed to read block groups: -5 > > [ 10.053365] BTRFS error (device sdb): open_ctree failed > > ---- > > Exactly the same symptom. > > > > > This server has a 16 disk btrfs filesystem (RAID6) which I boot > > periodically to btrfs-send snapshots to. This machine is running > > ArchLinux and I had just updated to their latest 4.19.4 kernel > > package (from 4.18.10 which was working fine). I've tried updating to > > the 4.19.6 kernel that is in testing, but that doesn't seem to resolve > > the issue. From what I can see on kernel.org, the patch above is not > > pushed to stable or to Linus' tree. > > > > At this point the question is what to do. Is my FS toast? > > If there is no other problem at all, your fs is just fine. > It's my original patch too sensitive (the excuse for not checking chunk > allocator carefully enough). > > But since you have the down time, it's never a bad idea to run a btrfs > check --readonly to see if your fs is really OK. > After running for 4 hours... UUID: 25b16375-b90b-408e-b592-fb07ed116d58 [1/7] checking root items [2/7] checking extents [3/7] checking free space cache [4/7] checking fs roots [5/7] checking only csums items (without verifying data) [6/7] checking root refs [7/7] checking quota groups found 24939616169984 bytes used, no error found total csum bytes: 24321980768 total tree bytes: 41129721856 total fs tree bytes: 9854648320 total extent tree bytes: 737804288 btree space waste bytes: 7483785005 file data blocks allocated: 212883520618496 referenced 212876546314240 So things appear good to go. I will keep an eye out for the patch to land before upgrading the kernel again. > > Could I > > revert to the 4.18.10 kernel and boot safely? > > If your btrfs check --readonly doesn't report any problem, then you're > completely fine to do so. > Although I still recommend to go RAID10 other than RAID5/6. I understand the risk, but don't have the funds to buy sufficient disks to operate in RAID10. The data is mostly large files and activity is predominantly reads, so risk is currently acceptable given the backup server. All super critical data is backed up to (very slow) cloud storage. > > Thanks, > Qu > > > I don't know if the 4.19 > > boot process may have flipped some bits which would make reverting > > problematic. > > > > Thanks much, > > > > - mike > > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ran into "invalid block group size" bug, unclear how to proceed. 2018-12-04 22:33 ` Mike Javorski @ 2018-12-05 5:24 ` Qu Wenruo 2018-12-05 5:47 ` Mike Javorski 0 siblings, 1 reply; 8+ messages in thread From: Qu Wenruo @ 2018-12-05 5:24 UTC (permalink / raw) To: Mike Javorski; +Cc: linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 3503 bytes --] On 2018/12/5 上午6:33, Mike Javorski wrote: > On Tue, Dec 4, 2018 at 2:18 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote: >> >> >> >> On 2018/12/4 上午11:32, Mike Javorski wrote: >>> Need a bit of advice here ladies / gents. I am running into an issue >>> which Qu Wenruo seems to have posted a patch for several weeks ago >>> (see https://patchwork.kernel.org/patch/10694997/). >>> >>> Here is the relevant dmesg output which led me to Qu's patch. >>> ---- >>> [ 10.032475] BTRFS critical (device sdb): corrupt leaf: root=2 >>> block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104, >>> invalid block group size, have 10804527104 expect (0, 10737418240] >>> [ 10.032493] BTRFS error (device sdb): failed to read block groups: -5 >>> [ 10.053365] BTRFS error (device sdb): open_ctree failed >>> ---- >> >> Exactly the same symptom. >> >>> >>> This server has a 16 disk btrfs filesystem (RAID6) which I boot >>> periodically to btrfs-send snapshots to. This machine is running >>> ArchLinux and I had just updated to their latest 4.19.4 kernel >>> package (from 4.18.10 which was working fine). I've tried updating to >>> the 4.19.6 kernel that is in testing, but that doesn't seem to resolve >>> the issue. From what I can see on kernel.org, the patch above is not >>> pushed to stable or to Linus' tree. >>> >>> At this point the question is what to do. Is my FS toast? >> >> If there is no other problem at all, your fs is just fine. >> It's my original patch too sensitive (the excuse for not checking chunk >> allocator carefully enough). >> >> But since you have the down time, it's never a bad idea to run a btrfs >> check --readonly to see if your fs is really OK. >> > > After running for 4 hours... > > UUID: 25b16375-b90b-408e-b592-fb07ed116d58 > [1/7] checking root items > [2/7] checking extents > [3/7] checking free space cache > [4/7] checking fs roots > [5/7] checking only csums items (without verifying data) > [6/7] checking root refs > [7/7] checking quota groups > found 24939616169984 bytes used, no error found > total csum bytes: 24321980768 > total tree bytes: 41129721856 > total fs tree bytes: 9854648320 > total extent tree bytes: 737804288 > btree space waste bytes: 7483785005 > file data blocks allocated: 212883520618496 > referenced 212876546314240 > > So things appear good to go. I will keep an eye out for the patch to > land before upgrading the kernel again. > >>> Could I >>> revert to the 4.18.10 kernel and boot safely? >> >> If your btrfs check --readonly doesn't report any problem, then you're >> completely fine to do so. >> Although I still recommend to go RAID10 other than RAID5/6. > > I understand the risk, but don't have the funds to buy sufficient > disks to operate in RAID10. Then my advice would be, for any powerloss, please run a full-disk scrub (and of course ensure there is not another powerloss during scrubbing). I know this sounds silly and slow, but at least it should workaround the write hole problem. Thanks, Qu > The data is mostly large files and > activity is predominantly reads, so risk is currently acceptable given > the backup server. All super critical data is backed up to (very slow) > cloud storage. > >> >> Thanks, >> Qu >> >>> I don't know if the 4.19 >>> boot process may have flipped some bits which would make reverting >>> problematic. >>> >>> Thanks much, >>> >>> - mike >>> >> [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ran into "invalid block group size" bug, unclear how to proceed. 2018-12-05 5:24 ` Qu Wenruo @ 2018-12-05 5:47 ` Mike Javorski 0 siblings, 0 replies; 8+ messages in thread From: Mike Javorski @ 2018-12-05 5:47 UTC (permalink / raw) To: quwenruo.btrfs; +Cc: linux-btrfs Will do, thanks! - mike On Tue, Dec 4, 2018 at 9:24 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote: > > > > On 2018/12/5 上午6:33, Mike Javorski wrote: > > On Tue, Dec 4, 2018 at 2:18 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote: > >> > >> > >> > >> On 2018/12/4 上午11:32, Mike Javorski wrote: > >>> Need a bit of advice here ladies / gents. I am running into an issue > >>> which Qu Wenruo seems to have posted a patch for several weeks ago > >>> (see https://patchwork.kernel.org/patch/10694997/). > >>> > >>> Here is the relevant dmesg output which led me to Qu's patch. > >>> ---- > >>> [ 10.032475] BTRFS critical (device sdb): corrupt leaf: root=2 > >>> block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104, > >>> invalid block group size, have 10804527104 expect (0, 10737418240] > >>> [ 10.032493] BTRFS error (device sdb): failed to read block groups: -5 > >>> [ 10.053365] BTRFS error (device sdb): open_ctree failed > >>> ---- > >> > >> Exactly the same symptom. > >> > >>> > >>> This server has a 16 disk btrfs filesystem (RAID6) which I boot > >>> periodically to btrfs-send snapshots to. This machine is running > >>> ArchLinux and I had just updated to their latest 4.19.4 kernel > >>> package (from 4.18.10 which was working fine). I've tried updating to > >>> the 4.19.6 kernel that is in testing, but that doesn't seem to resolve > >>> the issue. From what I can see on kernel.org, the patch above is not > >>> pushed to stable or to Linus' tree. > >>> > >>> At this point the question is what to do. Is my FS toast? > >> > >> If there is no other problem at all, your fs is just fine. > >> It's my original patch too sensitive (the excuse for not checking chunk > >> allocator carefully enough). > >> > >> But since you have the down time, it's never a bad idea to run a btrfs > >> check --readonly to see if your fs is really OK. > >> > > > > After running for 4 hours... > > > > UUID: 25b16375-b90b-408e-b592-fb07ed116d58 > > [1/7] checking root items > > [2/7] checking extents > > [3/7] checking free space cache > > [4/7] checking fs roots > > [5/7] checking only csums items (without verifying data) > > [6/7] checking root refs > > [7/7] checking quota groups > > found 24939616169984 bytes used, no error found > > total csum bytes: 24321980768 > > total tree bytes: 41129721856 > > total fs tree bytes: 9854648320 > > total extent tree bytes: 737804288 > > btree space waste bytes: 7483785005 > > file data blocks allocated: 212883520618496 > > referenced 212876546314240 > > > > So things appear good to go. I will keep an eye out for the patch to > > land before upgrading the kernel again. > > > >>> Could I > >>> revert to the 4.18.10 kernel and boot safely? > >> > >> If your btrfs check --readonly doesn't report any problem, then you're > >> completely fine to do so. > >> Although I still recommend to go RAID10 other than RAID5/6. > > > > I understand the risk, but don't have the funds to buy sufficient > > disks to operate in RAID10. > > Then my advice would be, for any powerloss, please run a full-disk scrub > (and of course ensure there is not another powerloss during scrubbing). > > I know this sounds silly and slow, but at least it should workaround the > write hole problem. > > Thanks, > Qu > > > The data is mostly large files and > > activity is predominantly reads, so risk is currently acceptable given > > the backup server. All super critical data is backed up to (very slow) > > cloud storage. > > > >> > >> Thanks, > >> Qu > >> > >>> I don't know if the 4.19 > >>> boot process may have flipped some bits which would make reverting > >>> problematic. > >>> > >>> Thanks much, > >>> > >>> - mike > >>> > >> > ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-12-05 5:47 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-12-04 3:32 Ran into "invalid block group size" bug, unclear how to proceed Mike Javorski 2018-12-04 4:00 ` Mike Javorski 2018-12-04 5:56 ` Chris Murphy 2018-12-04 6:46 ` Mike Javorski 2018-12-04 10:18 ` Qu Wenruo 2018-12-04 22:33 ` Mike Javorski 2018-12-05 5:24 ` Qu Wenruo 2018-12-05 5:47 ` Mike Javorski
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).