* BTRFS suddenly moving to read-only
@ 2020-08-11 9:39 Thommandra Gowtham
2020-08-11 10:38 ` Qu Wenruo
0 siblings, 1 reply; 4+ messages in thread
From: Thommandra Gowtham @ 2020-08-11 9:39 UTC (permalink / raw)
To: linux-btrfs
Hi,
Need some help to understand if there are any issues in BTRFS/Linux Kernel.
Running BTRFS as root filesystem and we see that suddenly the entire
disk is moved to read-only due to errors.
Did the SSD run out of life? If yes, then
- What are the best BTRFS options for frequent small amount of
writes(log files) on low quality SSD? If we want to increase the life
of the Disk.
- How do we determine the Disk health apart from SMART attributes? Can
we do a Disk write/read test to figure it out?
mount options used:
rw,noatime,compress=lzo,ssd,space_cache,commit=60,subvolid=263
# btrfs --version
btrfs-progs v4.4
Ubuntu 16.04: 4.15.0-36-generic #1 SMP Mon Oct 22 21:20:30 PDT 2018
x86_64 x86_64 x86_64 GNU/Linux
mkstemp: Read-only file system
[35816007.175210] print_req_error: I/O error, dev sda, sector 4472632
[35816007.182192] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
66, rd 725, flush 0, corrupt 0, gen 0
[35816007.192913] print_req_error: I/O error, dev sda, sector 4472632
[35816007.199855] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
66, rd 726, flush 0, corrupt 0, gen 0
[35816007.210675] print_req_error: I/O error, dev sda, sector 10180680
[35816007.217748] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
66, rd 727, flush 0, corrupt 0, gen 0
[35816007.461941] print_req_error: I/O error, dev sda, sector 4472048
[35816007.468903] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
66, rd 728, flush 0, corrupt 0, gen 0
[35816007.479611] systemd[7035]: serial-getty@ttyS0.service: Failed at
step EXEC spawning /sbin/agetty: Input/output error
[35816007.712006] print_req_error: I/O error, dev sda, sector 4472048
# dmesg | tail
bash: /bin/dmesg: Input/output error
Doesn't Input/output error mean the disk is inaccessible?
# btrfs fi show
Label: 'rpool' uuid: 42d39990-e4eb-414b-8b17-0c4a2f76cc76
Total devices 1 FS bytes used 11.80GiB
devid 1 size 27.20GiB used 19.01GiB path /dev/sda4
# smartctl -a /dev/sda
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.15.0-36-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
Short INQUIRY response, skip product id
A mandatory SMART command failed: exiting. To continue, add one or
more '-T permissive' options.
We were able to get smartctl o/p after a power-cycle
# smartctl -a /dev/sda
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.15.0-36-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: FS032GM242I-AC
Serial Number: AA010520170000000489
Firmware Version: O1026A
User Capacity: 31,488,000,000 bytes [31.4 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sun Aug 9 04:26:10 2020 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
...
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x0000 100 100 000 Old_age
Offline - 0
5 Reallocated_Sector_Ct 0x0000 100 100 000 Old_age
Offline - 0
9 Power_On_Hours 0x0000 100 100 000 Old_age
Offline - 735
12 Power_Cycle_Count 0x0000 100 100 000 Old_age
Offline - 20
160 Unknown_Attribute 0x0000 100 100 000 Old_age
Offline - 0
161 Unknown_Attribute 0x0000 100 100 000 Old_age
Offline - 58
163 Unknown_Attribute 0x0000 100 100 000 Old_age
Offline - 2
164 Unknown_Attribute 0x0000 100 100 000 Old_age
Offline - 1045371
165 Unknown_Attribute 0x0000 100 100 000 Old_age
Offline - 1075
166 Unknown_Attribute 0x0000 100 100 000 Old_age
Offline - 972
167 Unknown_Attribute 0x0000 100 100 000 Old_age
Offline - 1030
168 Unknown_Attribute 0x0000 100 100 000 Old_age
Offline - 3000
169 Unknown_Attribute 0x0000 100 100 000 Old_age
Offline - 66
175 Program_Fail_Count_Chip 0x0000 100 100 000 Old_age
Offline - 0
176 Erase_Fail_Count_Chip 0x0000 100 100 000 Old_age
Offline - 0
177 Wear_Leveling_Count 0x0000 100 100 050 Old_age
Offline - 3733
178 Used_Rsvd_Blk_Cnt_Chip 0x0000 100 100 000 Old_age
Offline - 0
181 Program_Fail_Cnt_Total 0x0000 100 100 000 Old_age
Offline - 0
182 Erase_Fail_Count_Total 0x0000 100 100 000 Old_age
Offline - 0
192 Power-Off_Retract_Count 0x0000 100 100 000 Old_age
Offline - 5
194 Temperature_Celsius 0x0000 100 100 000 Old_age
Offline - 40
195 Hardware_ECC_Recovered 0x0000 100 100 000 Old_age
Offline - 0
196 Reallocated_Event_Count 0x0000 100 100 016 Old_age
Offline - 0
197 Current_Pending_Sector 0x0000 100 100 000 Old_age
Offline - 0
198 Offline_Uncorrectable 0x0000 100 100 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0000 100 100 050 Old_age
Offline - 0
232 Available_Reservd_Space 0x0000 100 100 000 Old_age
Offline - 100
241 Total_LBAs_Written 0x0000 100 100 000 Old_age
Offline - 766189
242 Total_LBAs_Read 0x0000 100 100 000 Old_age
Offline - 11847
245 Unknown_Attribute 0x0000 100 100 000 Old_age
Offline - 1045371
Regards,
Gowtham
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: BTRFS suddenly moving to read-only
2020-08-11 9:39 BTRFS suddenly moving to read-only Thommandra Gowtham
@ 2020-08-11 10:38 ` Qu Wenruo
2020-08-11 15:12 ` Thommandra Gowtham
0 siblings, 1 reply; 4+ messages in thread
From: Qu Wenruo @ 2020-08-11 10:38 UTC (permalink / raw)
To: Thommandra Gowtham, linux-btrfs
[-- Attachment #1.1: Type: text/plain, Size: 7266 bytes --]
On 2020/8/11 下午5:39, Thommandra Gowtham wrote:
> Hi,
>
> Need some help to understand if there are any issues in BTRFS/Linux Kernel.
>
> Running BTRFS as root filesystem and we see that suddenly the entire
> disk is moved to read-only due to errors.
>
> Did the SSD run out of life? If yes, then
> - What are the best BTRFS options for frequent small amount of
> writes(log files) on low quality SSD? If we want to increase the life
> of the Disk.
If using systemd, you can config systemd to go memory only journal, so
that the life of the ssd can be expanded.
> - How do we determine the Disk health apart from SMART attributes? Can
> we do a Disk write/read test to figure it out?
AFAIK SMART is the only thing we can rely on now.
>
> mount options used:
> rw,noatime,compress=lzo,ssd,space_cache,commit=60,subvolid=263
>
> # btrfs --version
> btrfs-progs v4.4
>
> Ubuntu 16.04: 4.15.0-36-generic #1 SMP Mon Oct 22 21:20:30 PDT 2018
> x86_64 x86_64 x86_64 GNU/Linux
>
> mkstemp: Read-only file system
> [35816007.175210] print_req_error: I/O error, dev sda, sector 4472632
> [35816007.182192] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
> 66, rd 725, flush 0, corrupt 0, gen 0
This means some read error happened.
Do you have extra log context?
> [35816007.192913] print_req_error: I/O error, dev sda, sector 4472632
> [35816007.199855] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
> 66, rd 726, flush 0, corrupt 0, gen 0
> [35816007.210675] print_req_error: I/O error, dev sda, sector 10180680
> [35816007.217748] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
> 66, rd 727, flush 0, corrupt 0, gen 0
> [35816007.461941] print_req_error: I/O error, dev sda, sector 4472048
> [35816007.468903] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
> 66, rd 728, flush 0, corrupt 0, gen 0
> [35816007.479611] systemd[7035]: serial-getty@ttyS0.service: Failed at
> step EXEC spawning /sbin/agetty: Input/output error
> [35816007.712006] print_req_error: I/O error, dev sda, sector 4472048
This means, we failed to read some data from sda.
It's not the error from btrfs checksum verification, but directly read
error from the device driver.
So the command, agetty can't be executed due to we failed to read the
content of that executable file.
>
> # dmesg | tail
> bash: /bin/dmesg: Input/output error
>
> Doesn't Input/output error mean the disk is inaccessible?
This means, we can't even access /bin/dmesg the file itself.
>
> # btrfs fi show
> Label: 'rpool' uuid: 42d39990-e4eb-414b-8b17-0c4a2f76cc76
> Total devices 1 FS bytes used 11.80GiB
> devid 1 size 27.20GiB used 19.01GiB path /dev/sda4
>
> # smartctl -a /dev/sda
> smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.15.0-36-generic] (local build)
> Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
>
> Short INQUIRY response, skip product id
> A mandatory SMART command failed: exiting. To continue, add one or
> more '-T permissive' options.
>
>
> We were able to get smartctl o/p after a power-cycle
Did you get dmesg/agetty run after a power-cycle?
Or it still triggers the same -EIO error?
BTW, if the smartctl doesn't record above read error as error, maybe
it's some unstable cables causing temporary errors?
Thanks,
Qu
>
> # smartctl -a /dev/sda
> smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.15.0-36-generic] (local build)
> Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Device Model: FS032GM242I-AC
> Serial Number: AA010520170000000489
> Firmware Version: O1026A
> User Capacity: 31,488,000,000 bytes [31.4 GB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: Solid State Device
> Device is: Not in smartctl database [for details use: -P showall]
> ATA Version is: ACS-2 (minor revision not indicated)
> SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is: Sun Aug 9 04:26:10 2020 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> ...
> SMART Attributes Data Structure revision number: 1
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> UPDATED WHEN_FAILED RAW_VALUE
> 1 Raw_Read_Error_Rate 0x0000 100 100 000 Old_age
> Offline - 0
> 5 Reallocated_Sector_Ct 0x0000 100 100 000 Old_age
> Offline - 0
> 9 Power_On_Hours 0x0000 100 100 000 Old_age
> Offline - 735
> 12 Power_Cycle_Count 0x0000 100 100 000 Old_age
> Offline - 20
> 160 Unknown_Attribute 0x0000 100 100 000 Old_age
> Offline - 0
> 161 Unknown_Attribute 0x0000 100 100 000 Old_age
> Offline - 58
> 163 Unknown_Attribute 0x0000 100 100 000 Old_age
> Offline - 2
> 164 Unknown_Attribute 0x0000 100 100 000 Old_age
> Offline - 1045371
> 165 Unknown_Attribute 0x0000 100 100 000 Old_age
> Offline - 1075
> 166 Unknown_Attribute 0x0000 100 100 000 Old_age
> Offline - 972
> 167 Unknown_Attribute 0x0000 100 100 000 Old_age
> Offline - 1030
> 168 Unknown_Attribute 0x0000 100 100 000 Old_age
> Offline - 3000
> 169 Unknown_Attribute 0x0000 100 100 000 Old_age
> Offline - 66
> 175 Program_Fail_Count_Chip 0x0000 100 100 000 Old_age
> Offline - 0
> 176 Erase_Fail_Count_Chip 0x0000 100 100 000 Old_age
> Offline - 0
> 177 Wear_Leveling_Count 0x0000 100 100 050 Old_age
> Offline - 3733
> 178 Used_Rsvd_Blk_Cnt_Chip 0x0000 100 100 000 Old_age
> Offline - 0
> 181 Program_Fail_Cnt_Total 0x0000 100 100 000 Old_age
> Offline - 0
> 182 Erase_Fail_Count_Total 0x0000 100 100 000 Old_age
> Offline - 0
> 192 Power-Off_Retract_Count 0x0000 100 100 000 Old_age
> Offline - 5
> 194 Temperature_Celsius 0x0000 100 100 000 Old_age
> Offline - 40
> 195 Hardware_ECC_Recovered 0x0000 100 100 000 Old_age
> Offline - 0
> 196 Reallocated_Event_Count 0x0000 100 100 016 Old_age
> Offline - 0
> 197 Current_Pending_Sector 0x0000 100 100 000 Old_age
> Offline - 0
> 198 Offline_Uncorrectable 0x0000 100 100 000 Old_age
> Offline - 0
> 199 UDMA_CRC_Error_Count 0x0000 100 100 050 Old_age
> Offline - 0
> 232 Available_Reservd_Space 0x0000 100 100 000 Old_age
> Offline - 100
> 241 Total_LBAs_Written 0x0000 100 100 000 Old_age
> Offline - 766189
> 242 Total_LBAs_Read 0x0000 100 100 000 Old_age
> Offline - 11847
> 245 Unknown_Attribute 0x0000 100 100 000 Old_age
> Offline - 1045371
>
> Regards,
> Gowtham
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: BTRFS suddenly moving to read-only
2020-08-11 10:38 ` Qu Wenruo
@ 2020-08-11 15:12 ` Thommandra Gowtham
2020-08-11 22:58 ` Qu Wenruo
0 siblings, 1 reply; 4+ messages in thread
From: Thommandra Gowtham @ 2020-08-11 15:12 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs
Thank you for the response.
>
> > - How do we determine the Disk health apart from SMART attributes? Can
> > we do a Disk write/read test to figure it out?
>
> AFAIK SMART is the only thing we can rely on now.
Thank you. The reason I asked the question is sometimes, though SMART
reports the Percent Life remaining as > 80, we see issues with the
disk.
So I was looking if we can use dd or other tools to determine disk
write speed and compare with the new SSD's. Like below.
# dd if=/dev/zero of=/var/tmp/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.90537 s, 564 MB/s
>
> >
> > mount options used:
> > rw,noatime,compress=lzo,ssd,space_cache,commit=60,subvolid=263
> >
> > # btrfs --version
> > btrfs-progs v4.4
> >
> > Ubuntu 16.04: 4.15.0-36-generic #1 SMP Mon Oct 22 21:20:30 PDT 2018
> > x86_64 x86_64 x86_64 GNU/Linux
> >
> > mkstemp: Read-only file system
> > [35816007.175210] print_req_error: I/O error, dev sda, sector 4472632
> > [35816007.182192] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
> > 66, rd 725, flush 0, corrupt 0, gen 0
>
> This means some read error happened.
Yes. The errors started to occur when we were upgrading the packages.
Eventually the upgrade failed with read-only filesystem errors.
>
> Do you have extra log context?
Not much on this system as we couldn't get anything from syslog after
power-cycle.
But on other instances, we see errors like below
# cat syslog | grep error
Jun 25 13:12:13 kernel: [154559.788764] res
41/04:00:80:08:00/00:00:00:00:00/60 Emask 0x1 (device error)
Jun 25 13:12:13 kernel: [154559.821041] res
41/04:00:80:08:00/00:00:00:00:00/60 Emask 0x1 (device error)
Jun 25 13:12:13 kernel: [154559.900810] res
41/04:00:00:08:02/00:00:00:00:00/60 Emask 0x1 (device error)
Jun 25 13:12:13 kernel: [154559.933070] res
41/04:00:00:08:02/00:00:00:00:00/60 Emask 0x1 (device error)
Jun 25 13:12:13 kernel: [154560.016591] res
41/04:00:80:08:00/00:00:00:00:00/60 Emask 0x1 (device error)
Jun 25 13:12:13 kernel: [154560.048882] res
41/04:00:80:08:00/00:00:00:00:00/60 Emask 0x1 (device error)
Jun 25 13:12:13 kernel: [154560.114361] ata2.00: NCQ disabled due to
excessive errors
Jun 25 13:12:13 kernel: [154560.132361] res
41/04:00:00:08:02/00:00:00:00:00/60 Emask 0x1 (device error)
Jun 25 13:12:13 kernel: [154560.154507] ata2.00: error: { ABRT }
Jun 25 13:12:13 kernel: [154560.164580] res
41/04:00:00:08:02/00:00:00:00:00/60 Emask 0x1 (device error)
Jun 25 13:12:14 kernel: [154560.339129] ata2.00: error: { ABRT }
Jun 25 13:12:14 kernel: [154560.346548] print_req_error: I/O error,
dev sdb, sector 67111040
Jun 25 13:12:14 kernel: [154560.360192] BTRFS error (device sdb3):
bdev /dev/sdb3 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Jun 25 13:12:14 kernel: [154560.417322] res
51/04:00:00:08:02/00:00:04:00:00/60 Emask 0x1 (device error)
Jun 25 13:12:14 kernel: [154560.511036] ata2.00: error: { ABRT }
Jun 25 13:12:14 kernel: [154560.518434] print_req_error: I/O error,
dev sdb, sector 67241984
Jun 25 13:12:14 kernel: [154560.525291] BTRFS error (device sdb3):
bdev /dev/sdb3 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
>
> > [35816007.192913] print_req_error: I/O error, dev sda, sector 4472632
> > [35816007.199855] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
> > 66, rd 726, flush 0, corrupt 0, gen 0
> > [35816007.210675] print_req_error: I/O error, dev sda, sector 10180680
> > [35816007.217748] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
> > 66, rd 727, flush 0, corrupt 0, gen 0
> > [35816007.461941] print_req_error: I/O error, dev sda, sector 4472048
> > [35816007.468903] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
> > 66, rd 728, flush 0, corrupt 0, gen 0
> > [35816007.479611] systemd[7035]: serial-getty@ttyS0.service: Failed at
> > step EXEC spawning /sbin/agetty: Input/output error
> > [35816007.712006] print_req_error: I/O error, dev sda, sector 4472048
>
> This means, we failed to read some data from sda.
>
> It's not the error from btrfs checksum verification, but directly read
> error from the device driver.
>
> So the command, agetty can't be executed due to we failed to read the
> content of that executable file.
>
> >
> > # dmesg | tail
> > bash: /bin/dmesg: Input/output error
> >
> > Doesn't Input/output error mean the disk is inaccessible?
>
> This means, we can't even access /bin/dmesg the file itself.
Yes. That would technically mean that the Disk is not accessible
though it is being reported as read-only by 'mount -t btrfs'.
If a disk is missing or offline, is it done by kernel (bug) or
something related to hardware. This is being seen on multiple systems.
So there has to be some commonality among them and as the disk moves
to sudden read-only, we are unable to get much logs on all cases.
How can we debug these instances? Can you please give some pointers?
>
> >
> > # btrfs fi show
> > Label: 'rpool' uuid: 42d39990-e4eb-414b-8b17-0c4a2f76cc76
> > Total devices 1 FS bytes used 11.80GiB
> > devid 1 size 27.20GiB used 19.01GiB path /dev/sda4
> >
> > # smartctl -a /dev/sda
> > smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.15.0-36-generic] (local build)
> > Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
> >
> > Short INQUIRY response, skip product id
> > A mandatory SMART command failed: exiting. To continue, add one or
> > more '-T permissive' options.
> >
> >
> > We were able to get smartctl o/p after a power-cycle
>
> Did you get dmesg/agetty run after a power-cycle?
>
> Or it still triggers the same -EIO error?
No. After power-cycle everything is back to normal(rw mounted) with
logs not showing any abnormalities.
Subsequent IO activity(upgrading the packages) was successful as well.
>
> BTW, if the smartctl doesn't record above read error as error, maybe
> it's some unstable cables causing temporary errors?
> > 169 Unknown_Attribute 0x0000 100 100 000 Old_age
> > Offline - 66
The Disk Percent life remaining is at '66' for this system which is
low in my opinion. Can a disk go offline suddenly when the health
drops low?
Regards,
Gowtham
>
> Thanks,
> Qu
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: BTRFS suddenly moving to read-only
2020-08-11 15:12 ` Thommandra Gowtham
@ 2020-08-11 22:58 ` Qu Wenruo
0 siblings, 0 replies; 4+ messages in thread
From: Qu Wenruo @ 2020-08-11 22:58 UTC (permalink / raw)
To: Thommandra Gowtham; +Cc: linux-btrfs
[-- Attachment #1.1: Type: text/plain, Size: 7158 bytes --]
On 2020/8/11 下午11:12, Thommandra Gowtham wrote:
> Thank you for the response.
>
>>
>>> - How do we determine the Disk health apart from SMART attributes? Can
>>> we do a Disk write/read test to figure it out?
>>
>> AFAIK SMART is the only thing we can rely on now.
>
> Thank you. The reason I asked the question is sometimes, though SMART
> reports the Percent Life remaining as > 80, we see issues with the
> disk.
> So I was looking if we can use dd or other tools to determine disk
> write speed and compare with the new SSD's. Like below.
>
> # dd if=/dev/zero of=/var/tmp/test1.img bs=1G count=1 oflag=dsync
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.90537 s, 564 MB/s
>
>>
>>>
>>> mount options used:
>>> rw,noatime,compress=lzo,ssd,space_cache,commit=60,subvolid=263
>>>
>>> # btrfs --version
>>> btrfs-progs v4.4
>>>
>>> Ubuntu 16.04: 4.15.0-36-generic #1 SMP Mon Oct 22 21:20:30 PDT 2018
>>> x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> mkstemp: Read-only file system
>>> [35816007.175210] print_req_error: I/O error, dev sda, sector 4472632
>>> [35816007.182192] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
>>> 66, rd 725, flush 0, corrupt 0, gen 0
>>
>> This means some read error happened.
>
> Yes. The errors started to occur when we were upgrading the packages.
> Eventually the upgrade failed with read-only filesystem errors.
>
>>
>> Do you have extra log context?
>
> Not much on this system as we couldn't get anything from syslog after
> power-cycle.
>
> But on other instances, we see errors like below
>
> # cat syslog | grep error
> Jun 25 13:12:13 kernel: [154559.788764] res
> 41/04:00:80:08:00/00:00:00:00:00/60 Emask 0x1 (device error)
> Jun 25 13:12:13 kernel: [154559.821041] res
> 41/04:00:80:08:00/00:00:00:00:00/60 Emask 0x1 (device error)
> Jun 25 13:12:13 kernel: [154559.900810] res
> 41/04:00:00:08:02/00:00:00:00:00/60 Emask 0x1 (device error)
> Jun 25 13:12:13 kernel: [154559.933070] res
> 41/04:00:00:08:02/00:00:00:00:00/60 Emask 0x1 (device error)
> Jun 25 13:12:13 kernel: [154560.016591] res
> 41/04:00:80:08:00/00:00:00:00:00/60 Emask 0x1 (device error)
> Jun 25 13:12:13 kernel: [154560.048882] res
> 41/04:00:80:08:00/00:00:00:00:00/60 Emask 0x1 (device error)
> Jun 25 13:12:13 kernel: [154560.114361] ata2.00: NCQ disabled due to
> excessive errors
> Jun 25 13:12:13 kernel: [154560.132361] res
> 41/04:00:00:08:02/00:00:00:00:00/60 Emask 0x1 (device error)
> Jun 25 13:12:13 kernel: [154560.154507] ata2.00: error: { ABRT }
This means the disk has some command failed to be executed.
Full context would help to locate the problem.
> Jun 25 13:12:13 kernel: [154560.164580] res
> 41/04:00:00:08:02/00:00:00:00:00/60 Emask 0x1 (device error)
> Jun 25 13:12:14 kernel: [154560.339129] ata2.00: error: { ABRT }
> Jun 25 13:12:14 kernel: [154560.346548] print_req_error: I/O error,
> dev sdb, sector 67111040
> Jun 25 13:12:14 kernel: [154560.360192] BTRFS error (device sdb3):
> bdev /dev/sdb3 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
> Jun 25 13:12:14 kernel: [154560.417322] res
> 51/04:00:00:08:02/00:00:04:00:00/60 Emask 0x1 (device error)
> Jun 25 13:12:14 kernel: [154560.511036] ata2.00: error: { ABRT }
> Jun 25 13:12:14 kernel: [154560.518434] print_req_error: I/O error,
> dev sdb, sector 67241984
> Jun 25 13:12:14 kernel: [154560.525291] BTRFS error (device sdb3):
> bdev /dev/sdb3 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
>
>>
>>> [35816007.192913] print_req_error: I/O error, dev sda, sector 4472632
>>> [35816007.199855] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
>>> 66, rd 726, flush 0, corrupt 0, gen 0
>>> [35816007.210675] print_req_error: I/O error, dev sda, sector 10180680
>>> [35816007.217748] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
>>> 66, rd 727, flush 0, corrupt 0, gen 0
>>> [35816007.461941] print_req_error: I/O error, dev sda, sector 4472048
>>> [35816007.468903] BTRFS error (device sda4): bdev /dev/sda4 errs: wr
>>> 66, rd 728, flush 0, corrupt 0, gen 0
>>> [35816007.479611] systemd[7035]: serial-getty@ttyS0.service: Failed at
>>> step EXEC spawning /sbin/agetty: Input/output error
>>> [35816007.712006] print_req_error: I/O error, dev sda, sector 4472048
>>
>> This means, we failed to read some data from sda.
>>
>> It's not the error from btrfs checksum verification, but directly read
>> error from the device driver.
>>
>> So the command, agetty can't be executed due to we failed to read the
>> content of that executable file.
>>
>>>
>>> # dmesg | tail
>>> bash: /bin/dmesg: Input/output error
>>>
>>> Doesn't Input/output error mean the disk is inaccessible?
>>
>> This means, we can't even access /bin/dmesg the file itself.
>
> Yes. That would technically mean that the Disk is not accessible
> though it is being reported as read-only by 'mount -t btrfs'.
>
> If a disk is missing or offline, is it done by kernel (bug) or
> something related to hardware. This is being seen on multiple systems.
> So there has to be some commonality among them and as the disk moves
> to sudden read-only, we are unable to get much logs on all cases.
>
> How can we debug these instances? Can you please give some pointers?
I would recommend to setup a netconsole environment to catch all logs.
Then next time you can provide full context about the problem.
>
>>
>>>
>>> # btrfs fi show
>>> Label: 'rpool' uuid: 42d39990-e4eb-414b-8b17-0c4a2f76cc76
>>> Total devices 1 FS bytes used 11.80GiB
>>> devid 1 size 27.20GiB used 19.01GiB path /dev/sda4
>>>
>>> # smartctl -a /dev/sda
>>> smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.15.0-36-generic] (local build)
>>> Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
>>>
>>> Short INQUIRY response, skip product id
>>> A mandatory SMART command failed: exiting. To continue, add one or
>>> more '-T permissive' options.
>>>
>>>
>>> We were able to get smartctl o/p after a power-cycle
>>
>> Did you get dmesg/agetty run after a power-cycle?
>>
>> Or it still triggers the same -EIO error?
>
> No. After power-cycle everything is back to normal(rw mounted) with
> logs not showing any abnormalities.
> Subsequent IO activity(upgrading the packages) was successful as well.
That's tricky, now looks more like a bug in block layer.
Thus netconsole setup is strongly recommended.
>
>>
>> BTW, if the smartctl doesn't record above read error as error, maybe
>> it's some unstable cables causing temporary errors?
>
>>> 169 Unknown_Attribute 0x0000 100 100 000 Old_age
>>> Offline - 66
>
> The Disk Percent life remaining is at '66' for this system which is
> low in my opinion. Can a disk go offline suddenly when the health
> drops low?
Not sure for the hardware, needs better context to determine though.
Thanks,
Qu
>
> Regards,
> Gowtham
>
>>
>> Thanks,
>> Qu
>>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-08-11 22:58 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-11 9:39 BTRFS suddenly moving to read-only Thommandra Gowtham
2020-08-11 10:38 ` Qu Wenruo
2020-08-11 15:12 ` Thommandra Gowtham
2020-08-11 22:58 ` Qu Wenruo
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.