Linux-BTRFS Archive on lore.kernel.org
 help / color / Atom feed
* How to Fix 'Error: could not find extent items for root 257'?
@ 2020-02-05 10:18 Chiung-Ming Huang
  2020-02-05 10:29 ` Qu Wenruo
  0 siblings, 1 reply; 16+ messages in thread
From: Chiung-Ming Huang @ 2020-02-05 10:18 UTC (permalink / raw)
  To: linux-btrfs

Hi everyone

It's a long story. I try to describe it shortly. My btrfs RAID1
includes 5 HDDs, 10Tx2, 1Tx1, 2Tx1 and 3Tx1. They all based on bcache
(1Tx1 SSD as cache) and luks. I tried to reorder it to ` Luks -->
Bcache --> SSD --> HDD` with only one layer of luks on bcache. But I
failed because of power-off accidentally. Please help me to fix it.
Thanks.

1. OS: Ubuntu 18.04

2. $ uname -a
Linux rescue 5.3.0-26-generic #28~18.04.1-Ubuntu SMP Wed Dec 18
16:40:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

3. $ btrfs --version
btrfs-progs v5.4.1

4. $ btrfs fi show
Label: none  uuid: 0b79cf54-c424-40ed-adca-bd66b38ad57a
        Total devices 5 FS bytes used 496.00KiB
        devid    1 size 9.09TiB used 3.93TiB path /dev/bcache2
        devid    3 size 2.73TiB used 746.00GiB path /dev/bcache3
        devid    5 size 9.09TiB used 7.09TiB path /dev/bcache4
        devid    6 size 931.51GiB used 0.00B path /dev/mapper/disk-1t
        devid    7 size 1.82TiB used 0.00B path /dev/mapper/disk-2t

5. $ mount /dev/bcache4 /mnt
It showed the second part of messages after about 10 seconds and
remount it as readonly
------------------ dmesg part 1/2 ------------------

[Wed Feb  5 17:09:04 2020] BTRFS info (device bcache2): disk space
caching is enabled
[Wed Feb  5 17:09:04 2020] BTRFS info (device bcache2): has skinny
extents
[Wed Feb  5 17:09:04 2020] BTRFS info (device bcache2): bdev
/dev/bcache2 errs: wr 0, rd 0, flush 0, corrupt 266140, gen 8928
[Wed Feb  5 17:09:04 2020] BTRFS info (device bcache2): bdev
/dev/bcache3 errs: wr 0, rd 0, flush 0, corrupt 0, gen 3
[Wed Feb  5 17:09:04 2020] BTRFS info (device bcache2): enabling ssd
optimizations
[Wed Feb  5 17:09:04 2020] BTRFS info (device bcache2): checking UUID
tree
[Wed Feb  5 17:09:04 2020] BTRFS error (device bcache2): tree level
mismatch detected, bytenr=19499133206528 level expected=0 has=2
[Wed Feb  5 17:09:04 2020] BTRFS error (device bcache2): tree level
mismatch detected, bytenr=19499133206528 level expected=0 has=2
[Wed Feb  5 17:09:04 2020] BTRFS warning (device bcache2): iterating
uuid_tree failed -117
btrfs fi df /

------------------ dmesg part 2/2 ------------------

[Wed Feb  5 17:09:36 2020] BTRFS error (device bcache2): tree block
14963956514816 owner 3 already locked by pid=3187, extent tree
corruption detected
[Wed Feb  5 17:09:36 2020] ------------[ cut here ]------------
[Wed Feb  5 17:09:36 2020] BTRFS: Transaction aborted (error -117)
[Wed Feb  5 17:09:36 2020] WARNING: CPU: 4 PID: 3187 at
/build/linux-hwe-3HpQOB/linux-hwe-5.3.0/fs/btrfs/volumes.c:3025
btrfs_remove_chunk+0x76e/0x8a0 [btrfs]
[Wed Feb  5 17:09:36 2020] Modules linked in: cmac bnep nls_iso8859_1
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic
ledtrig_audio snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep
edac_mce_amd kvm_amd snd_pcm ccp kvm snd_seq_midi snd_seq_midi_event
irqby
pass k10temp snd_rawmidi btusb fam15h_power btrtl btbcm btintel joydev
nouveau snd_seq input_leds bluetooth snd_seq_device mxm_wmi snd_timer
video ecdh_generic snd ecc ttm i2c_algo_bit soundcore mac_hid
sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables autofs4 b
trfs xor zstd_compress raid6_pq libcrc32c algif_skcipher af_alg
dm_crypt hid_logitech_hidpp bcache crc64 hid_logitech_dj hid_generic
usbhid hid uas usb_storage nvidia_drm(POE) nvidia_modeset(POE)
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
nvidia(POE) ae
s_x86_64 crypto_simd drm_kms_helper syscopyarea cryptd sysfillrect
glue_helper sysimgblt fb_sys_fops drm r8169 nvme realtek i2c_piix4
ahci ipmi_devintf nvme_core(E) libahci ipmi_msghandler wmi
[Wed Feb  5 17:09:36 2020] CPU: 4 PID: 3187 Comm: btrfs-cleaner
Tainted: P        W  OE     5.3.0-26-generic #28~18.04.1-Ubuntu
[Wed Feb  5 17:09:36 2020] Hardware name: MSI MS-7974/970A-G43 PLUS
(MS-7974), BIOS V1.1 07/04/2016
[Wed Feb  5 17:09:36 2020] RIP: 0010:btrfs_remove_chunk+0x76e/0x8a0 [btrfs]
[Wed Feb  5 17:09:36 2020] Code: 48 8b 50 50 f0 48 0f ba aa 40 ce 00
00 02 8b 45 a0 72 1c 83 f8 fb 0f 84 af 00 00 00 89 c6 48 c7 c7 f0 52
7d c1 e8 72 fa 73 eb <0f> 0b 8b 45 a0 48 8b 7d 90 89 c1 ba d1 0b 00 00
48 c7 c6 90 54 7c
[Wed Feb  5 17:09:36 2020] RSP: 0018:ffffa7a5035d3d98 EFLAGS: 00010282
[Wed Feb  5 17:09:36 2020] RAX: 0000000000000000 RBX: 0000000040000000
RCX: 0000000000000006
[Wed Feb  5 17:09:36 2020] RDX: 0000000000000007 RSI: 0000000000000092
RDI: ffff97cea7b17440
[Wed Feb  5 17:09:36 2020] RBP: ffffa7a5035d3e48 R08: 00000000000005d3
R09: 0000000000000004
[Wed Feb  5 17:09:36 2020] R10: 00000000ffffffff R11: 0000000000000001
R12: ffff97ce9b647c00
[Wed Feb  5 17:09:36 2020] R13: ffff97ce95c2e800 R14: ffff97ce9c1d03b8
R15: ffff97ce9253ec40
[Wed Feb  5 17:09:36 2020] FS:  0000000000000000(0000)
GS:ffff97cea7b00000(0000) knlGS:0000000000000000
[Wed Feb  5 17:09:36 2020] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Wed Feb  5 17:09:36 2020] CR2: 000055b322010290 CR3: 000000061d628000
CR4: 00000000000406e0
[Wed Feb  5 17:09:36 2020] Call Trace:
[Wed Feb  5 17:09:36 2020]  btrfs_delete_unused_bgs+0x36a/0x490 [btrfs]
[Wed Feb  5 17:09:36 2020]  cleaner_kthread+0xed/0x130 [btrfs]
[Wed Feb  5 17:09:36 2020]  kthread+0x121/0x140
[Wed Feb  5 17:09:36 2020]  ? __btrfs_btree_balance_dirty+0x60/0x60 [btrfs]
[Wed Feb  5 17:09:36 2020]  ? kthread_park+0xb0/0xb0
[Wed Feb  5 17:09:36 2020]  ret_from_fork+0x22/0x40
[Wed Feb  5 17:09:36 2020] ---[ end trace c34270cb20778d7d ]---
[Wed Feb  5 17:09:36 2020] BTRFS: error (device bcache2) in
btrfs_remove_chunk:3025: errno=-117 unknown
[Wed Feb  5 17:09:36 2020] BTRFS info (device bcache2): forced readonly
[Wed Feb  5 17:09:36 2020] ------------[ cut here ]------------
[Wed Feb  5 17:09:36 2020] WARNING: CPU: 4 PID: 3187 at
/build/linux-hwe-3HpQOB/linux-hwe-5.3.0/fs/btrfs/space-info.h:106
btrfs_space_info_update_bytes_may_use.part.10+0x14/0x21 [btrfs]
[Wed Feb  5 17:09:36 2020] Modules linked in: cmac bnep nls_iso8859_1
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic
ledtrig_audio snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep
edac_mce_amd kvm_amd snd_pcm ccp kvm snd_seq_midi snd_seq_midi_event
irqby
pass k10temp snd_rawmidi btusb fam15h_power btrtl btbcm btintel joydev
nouveau snd_seq input_leds bluetooth snd_seq_device mxm_wmi snd_timer
video ecdh_generic snd ecc ttm i2c_algo_bit soundcore mac_hid
sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables autofs4 b
trfs xor zstd_compress raid6_pq libcrc32c algif_skcipher af_alg
dm_crypt hid_logitech_hidpp bcache crc64 hid_logitech_dj hid_generic
usbhid hid uas usb_storage nvidia_drm(POE) nvidia_modeset(POE)
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
nvidia(POE) ae
s_x86_64 crypto_simd drm_kms_helper syscopyarea cryptd sysfillrect
glue_helper sysimgblt fb_sys_fops drm r8169 nvme realtek i2c_piix4
ahci ipmi_devintf nvme_core(E) libahci ipmi_msghandler wmi
[Wed Feb  5 17:09:36 2020] CPU: 4 PID: 3187 Comm: btrfs-cleaner
Tainted: P        W  OE     5.3.0-26-generic #28~18.04.1-Ubuntu
[Wed Feb  5 17:09:36 2020] Hardware name: MSI MS-7974/970A-G43 PLUS
(MS-7974), BIOS V1.1 07/04/2016
[Wed Feb  5 17:09:36 2020] RIP:
0010:btrfs_space_info_update_bytes_may_use.part.10+0x14/0x21 [btrfs]
[Wed Feb  5 17:09:36 2020] Code: 74 05 e8 22 a5 6d eb 48 8d 65 d8 5b
41 5a 41 5c 41 5d 41 5e 5d c3 55 48 89 e5 53 48 89 fb 48 c7 c7 e8 a4
7d c1 e8 d2 84 74 eb <0f> 0b 48 c7 43 28 00 00 00 00 5b 5d c3 55 48 89
e5 53 48 89 fb 48
[Wed Feb  5 17:09:36 2020] RSP: 0018:ffffa7a5035d3ca8 EFLAGS: 00010286
[Wed Feb  5 17:09:36 2020] RAX: 0000000000000024 RBX: ffff97ce96ed5800
RCX: 0000000000000006
[Wed Feb  5 17:09:36 2020] RDX: 0000000000000000 RSI: 0000000000000092
RDI: ffff97cea7b17440
[Wed Feb  5 17:09:36 2020] RBP: ffffa7a5035d3cb0 R08: 00000000000005ed
R09: 0000000000000004
[Wed Feb  5 17:09:36 2020] R10: 0000000000000002 R11: 0000000000000001
R12: ffff97ce96ed5800
[Wed Feb  5 17:09:36 2020] R13: 0000000000080000 R14: 000000000007c000
R15: ffff97ce9c1d0000
[Wed Feb  5 17:09:36 2020] FS:  0000000000000000(0000)
GS:ffff97cea7b00000(0000) knlGS:0000000000000000
[Wed Feb  5 17:09:36 2020] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Wed Feb  5 17:09:36 2020] CR2: 000055b322010290 CR3: 000000061d628000
CR4: 00000000000406e0
[Wed Feb  5 17:09:36 2020] Call Trace:
[Wed Feb  5 17:09:36 2020]  btrfs_space_info_add_old_bytes+0x261/0x280 [btrfs]
[Wed Feb  5 17:09:36 2020]  __btrfs_block_rsv_release+0x16e/0x1a0 [btrfs]
[Wed Feb  5 17:09:36 2020]  btrfs_trans_release_chunk_metadata+0x35/0x50 [btrfs]
[Wed Feb  5 17:09:36 2020]
btrfs_create_pending_block_groups+0x13d/0x240 [btrfs]
[Wed Feb  5 17:09:36 2020]  __btrfs_end_transaction+0x6e/0x1e0 [btrfs]
[Wed Feb  5 17:09:36 2020]  btrfs_end_transaction+0x10/0x20 [btrfs]
[Wed Feb  5 17:09:36 2020]  btrfs_delete_unused_bgs+0x28b/0x490 [btrfs]
[Wed Feb  5 17:09:36 2020]  cleaner_kthread+0xed/0x130 [btrfs]
[Wed Feb  5 17:09:36 2020]  kthread+0x121/0x140
[Wed Feb  5 17:09:36 2020]  ? __btrfs_btree_balance_dirty+0x60/0x60 [btrfs]
[Wed Feb  5 17:09:36 2020]  ? kthread_park+0xb0/0xb0
[Wed Feb  5 17:09:36 2020]  ret_from_fork+0x22/0x40
[Wed Feb  5 17:09:36 2020] ---[ end trace c34270cb20778d7e ]---

6. $ btrfs fi df /mnt
Data, RAID1: total=4.21TiB, used=0.00B
Data, single: total=3.30TiB, used=0.00B
System, RAID1: total=32.00MiB, used=0.00B
Metadata, RAID1: total=12.00GiB, used=496.00KiB
Metadata, single: total=8.00GiB, used=0.00B
GlobalReserve, single: total=512.00MiB, used=0.00B

7. $ btrfs check -p /dev/bcache4
Opening filesystem to check...
Checking filesystem on /dev/bcache4
UUID: 0b79cf54-c424-40ed-adca-bd66b38ad57a
Error: could not find extent items for root 257(0:00:00 elapsed, 1199
items checked)
[1/7] checking root items                      (0:00:00 elapsed, 7748
items checked)
ERROR: failed to repair root items: No such file or directory

8. $ btrfs scrub start -B -R /mnt
The status is aborted because the file system was forcely re-mounted readonly.

9. $ lsblk -o NAME,SIZE,TYPE,FSTYPE
sda                     931.5G disk  bcache
└─bcache0               931.5G disk  crypto_LUKS
  └─disk-1t                931.5G crypt btrfs
sdb                     232.9G disk
└─sdb6                     10G part  crypto_LUKS
  └─rescue                 10G crypt btrfs
sdc                       2.7T disk  crypto_LUKS
└─disk-3t             2.7T crypt bcache
  └─bcache3               2.7T disk  btrfs
sdd                       9.1T disk  crypto_LUKS
└─disk-10t           9.1T crypt bcache
  └─bcache2               9.1T disk  btrfs
sde                       1.8T disk  bcache
└─bcache1                 1.8T disk  crypto_LUKS
  └─disk-2t          1.8T crypt btrfs
sdf                       9.1T disk  crypto_LUKS
└─disk-10t          9.1T crypt bcache
  └─bcache4               9.1T disk  btrfs
nvme0n1                 953.9G disk
└─nvme0n1p1               636G part  crypto_LUKS
  └─cache                 636G crypt bcache
    ├─bcache0           931.5G disk  crypto_LUKS
    │ └─disk-1t           931.5G crypt btrfs
    ├─bcache1             1.8T disk  crypto_LUKS
    │ └─disk-2t          1.8T crypt btrfs
    ├─bcache2             9.1T disk  btrfs
    ├─bcache3             2.7T disk  btrfs
    └─bcache4             9.1T disk  btrfs


Regards,
Chiung-Ming Huang

^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, back to index

Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-05 10:18 How to Fix 'Error: could not find extent items for root 257'? Chiung-Ming Huang
2020-02-05 10:29 ` Qu Wenruo
2020-02-05 15:29   ` Chiung-Ming Huang
2020-02-05 19:38     ` Chris Murphy
2020-02-06  3:11       ` Chiung-Ming Huang
     [not found]   ` <CAEOGEKHf9F0VM=au-42MwD63_V8RwtqiskV0LsGpq-c=J_qyPg@mail.gmail.com>
     [not found]     ` <f2ad6b4f-b011-8954-77e1-5162c84f7c1f@gmx.com>
2020-02-06  4:13       ` Chiung-Ming Huang
2020-02-06  4:35         ` Qu Wenruo
2020-02-06  6:50           ` Chiung-Ming Huang
2020-02-07  3:49           ` Chiung-Ming Huang
2020-02-07  4:00             ` Qu Wenruo
2020-02-07  6:16               ` Chiung-Ming Huang
2020-02-07  7:16                 ` Qu Wenruo
2020-02-10  6:50                   ` Chiung-Ming Huang
2020-02-10  7:03                     ` Qu Wenruo
2020-02-15  3:47                       ` Chiung-Ming Huang
2020-02-15  4:29                         ` Qu Wenruo

Linux-BTRFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \
		linux-btrfs@vger.kernel.org
	public-inbox-index linux-btrfs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git