Ext4 errors in linux /var/log/messages

Ext4 errors in linux /var/log/messages

[Alok Jain]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

Hi Guys,

Thanks for providing the forum to check the issue, I already got
really helpful responses and need one more help wrt to EXT4 log
messages coming in /var/log/messages file

1) I have iSCSI device attached to my VM
2) Copied and extracted files in the attached device
3) Unmount and disconnected the device, post disconnect operation I
can see lots of EXT4 file system errors in /var/log/messages file,
attaching the same (err.txt) for review.

I am not sure what these log messages are correspond to as the device
was already disconnected at  Feb 1:22:42, but the ext4 log messages
came after that, can you please review and share your findings.

Any help is appreciated

Thanks,
Alok

[-- Attachment #2: err.txt --]
[-- Type: text/plain, Size: 7867 bytes --]

Feb 28 01:22:46 str-1 kernel: EXT4-fs warning (device sdd1): ext4_end_bio:323: I/O error 10 writing to inode 9701084 (offset 16777216 size 8388608 starting block 25836288)
Feb 28 01:22:46 str-1 kernel: Buffer I/O error on device sdd1, logical block 25835520
Feb 28 01:22:46 str-1 kernel: Buffer I/O error on device sdd1, logical block 25835521
Feb 28 01:22:46 str-1 kernel: Buffer I/O error on device sdd1, logical block 25835522
Feb 28 01:22:46 str-1 kernel: Buffer I/O error on device sdd1, logical block 25835523
Feb 28 01:22:46 str-1 kernel: Buffer I/O error on device sdd1, logical block 25835524
Feb 28 01:22:46 str-1 kernel: Buffer I/O error on device sdd1, logical block 25835525
Feb 28 01:22:46 str-1 kernel: Buffer I/O error on device sdd1, logical block 25835526
Feb 28 01:22:46 str-1 kernel: Buffer I/O error on device sdd1, logical block 25835527
Feb 28 01:22:46 str-1 kernel: Buffer I/O error on device sdd1, logical block 25835528
Feb 28 01:22:46 str-1 kernel: Buffer I/O error on device sdd1, logical block 25835529
Feb 28 01:22:46 str-1 kernel: EXT4-fs warning (device sdd1): ext4_end_bio:323: I/O error 10 writing to inode 9701084 (offset 16777216 size 8388608 starting block 25836800)
Feb 28 01:22:46 str-1 kernel: EXT4-fs warning (device sdd1): ext4_end_bio:323: I/O error 10 writing to inode 9701084 (offset 16777216 size 8388608 starting block 25837312)
Feb 28 01:22:46 str-1 kernel: EXT4-fs warning (device sdd1): ext4_end_bio:323: I/O error 10 writing to inode 9701084 (offset 16777216 size 8388608 starting block 25837824)
Feb 28 01:22:46 str-1 kernel: EXT4-fs warning (device sdd1): ext4_end_bio:323: I/O error 10 writing to inode 9701084 (offset 25165824 size 4968448 starting block 25838336)
Feb 28 01:22:46 str-1 kernel: EXT4-fs warning (device sdd1): ext4_end_bio:323: I/O error 10 writing to inode 9701084 (offset 25165824 size 4968448 starting block 25838848)
Feb 28 01:22:46 str-1 kernel: EXT4-fs warning (device sdd1): ext4_end_bio:323: I/O error 10 writing to inode 9701084 (offset 25165824 size 4968448 starting block 25839037)
Feb 28 01:22:46 str-1 kernel: EXT4-fs warning (device sdd1): ext4_end_bio:323: I/O error 10 writing to inode 1050417 (offset 8388608 size 3104768 starting block 25839549)
Feb 28 01:22:46 str-1 kernel: EXT4-fs warning (device sdd1): ext4_end_bio:323: I/O error 10 writing to inode 1050417 (offset 8388608 size 3104768 starting block 25839795)
Feb 28 01:22:46 str-1 kernel: EXT4-fs warning (device sdd1): ext4_end_bio:323: I/O error 10 writing to inode 1050417 (offset 0 size 8388608 starting block 25840384)
Feb 28 01:22:46 str-1 kernel: capability: warning: `turbostat.4.14.' uses 32-bit capabilities (legacy support in use)
Feb 28 01:22:47 str-1 kernel: EXT4-fs error (device sdd1): ext4_wait_block_bitmap:517: comm kworker/u96:3: Cannot read block bitmap - block_group = 178, block_bitmap = 5767170
Feb 28 01:22:47 str-1 kernel: EXT4-fs (sdd1): Delayed block allocation failed for inode 1442438 at logical offset 0 with max blocks 636 with error 5
Feb 28 01:22:47 str-1 kernel: EXT4-fs (sdd1): This should not happen!! Data will be lost
Feb 28 01:22:47 str-1 kernel: EXT4-fs error (device sdd1): ext4_wait_block_bitmap:517: comm kworker/u96:3: Cannot read block bitmap - block_group = 178, block_bitmap = 5767170
Feb 28 01:22:47 str-1 kernel: EXT4-fs (sdd1): previous I/O error to superblock detected
Feb 28 01:22:47 str-1 kernel: EXT4-fs (sdd1): Delayed block allocation failed for inode 1442534 at logical offset 0 with max blocks 771 with error 5
Feb 28 01:22:47 str-1 kernel: EXT4-fs (sdd1): This should not happen!! Data will be lost
Feb 28 01:22:47 str-1 kernel: EXT4-fs error (device sdd1): ext4_wait_block_bitmap:517: comm kworker/u96:3: Cannot read block bitmap - block_group = 178, block_bitmap = 5767170
Feb 28 01:22:47 str-1 kernel: EXT4-fs (sdd1): previous I/O error to superblock detected
Feb 28 01:22:47 str-1 kernel: EXT4-fs (sdd1): Delayed block allocation failed for inode 1442873 at logical offset 0 with max blocks 215 with error 5
Feb 28 01:22:47 str-1 kernel: EXT4-fs (sdd1): This should not happen!! Data will be lost
Feb 28 01:22:47 str-1 kernel: EXT4-fs error (device sdd1): ext4_wait_block_bitmap:517: comm kworker/u96:3: Cannot read block bitmap - block_group = 178, block_bitmap = 5767170
Feb 28 01:22:47 str-1 kernel: EXT4-fs (sdd1): previous I/O error to superblock detected
Feb 28 01:22:47 str-1 kernel: EXT4-fs (sdd1): Delayed block allocation failed for inode 1442878 at logical offset 0 with max blocks 155 with error 5
Feb 28 01:22:47 str-1 kernel: EXT4-fs error (device sdd1): ext4_wait_block_bitmap:517: comm kworker/u96:3: Cannot read block bitmap - block_group = 178, block_bitmap = 5767170
Feb 28 01:22:47 str-1 kernel: EXT4-fs error (device sdd1): ext4_wait_block_bitmap:517: comm kworker/u96:3: Cannot read block bitmap - block_group = 178, block_bitmap = 5767170
Feb 28 01:22:47 str-1 kernel: EXT4-fs error (device sdd1): ext4_wait_block_bitmap:517: comm kworker/u96:3: Cannot read block bitmap - block_group = 178, block_bitmap = 5767170
Feb 28 01:22:47 str-1 kernel: EXT4-fs error (device sdd1): ext4_wait_block_bitmap:517: comm kworker/u96:3: Cannot read block bitmap - block_group = 178, block_bitmap = 5767170
Feb 28 01:22:47 str-1 kernel: EXT4-fs error (device sdd1): ext4_wait_block_bitmap:517: comm kworker/u96:3: Cannot read block bitmap - block_group = 178, block_bitmap = 5767170
Feb 28 01:22:47 str-1 kernel: EXT4-fs error (device sdd1): ext4_wait_block_bitmap:517: comm kworker/u96:3: Cannot read block bitmap - block_group = 178, block_bitmap = 5767170
Feb 28 01:22:49 str-1 kernel: JBD2: Error -5 detected when updating journal superblock for sdd1-8.
Feb 28 01:22:49 str-1 kernel: Aborting journal on device sdd1-8.
Feb 28 01:22:49 str-1 kernel: JBD2: Detected IO errors while flushing file data on sdd1-8
Feb 28 01:22:49 str-1 kernel: Aborting journal on device sdd1-8.
Feb 28 01:22:49 str-1 kernel: JBD2: Error -5 detected when updating journal superblock for sdd1-8.
Feb 28 01:22:49 str-1 kernel: JBD2: Error -5 detected when updating journal superblock for sdd1-8.
Feb 28 01:22:49 str-1 kernel: EXT4-fs error (device sdd1): ext4_journal_check_start:61: Detected aborted journal
Feb 28 01:22:51 str-1 audispd: node=instance20181212042353 type=SYSCALL msg=audit(1582852971.728:844284): arch=c000003e syscall=2 success=yes exit=5 a0=55f5b68831b0 a1=2 a2=180 a3=fffffff8 items=1 ppid=30763 pid=30765 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="faillock" exe="/usr/sbin/faillock" subj=system_u:system_r:sosreport_t:s0-s0:c0.c1023 key="logins"
Feb 28 01:22:51 str-1 audispd: node=instance20181212042353 type=CWD msg=audit(1582852971.728:844284): cwd="/var/spool/abrt/ccpp-2020-02-28-01:22:27-3609"
Feb 28 01:22:51 str-1 audispd: node=instance20181212042353 type=PATH msg=audit(1582852971.728:844284): item=0 name="/var/run/faillock/pandwive" inode=42865364 dev=00:17 mode=0100600 ouid=1154689 ogid=0 rdev=00:00 obj=system_u:object_r:faillog_t:s0 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
Feb 28 01:22:51 str-1 audispd: node=instance20181212042353 type=PROCTITLE msg=audit(1582852971.728:844284): proctitle="faillock"
Feb 28 01:22:52 str-1 kernel: EXT4-fs error: 3814 callbacks suppressed
Feb 28 01:22:52 str-1 kernel: EXT4-fs error (device sdd1): ext4_wait_block_bitmap:517: comm umount: Cannot read block bitmap - block_group = 773, block_bitmap = 25165829
Feb 28 01:22:52 str-1 kernel: EXT4-fs: 11466 callbacks suppressed
Feb 28 01:22:52 str-1 kernel: EXT4-fs (sdd1): previous I/O error to superblock detected
Feb 28 01:22:52 str-1 kernel: EXT4-fs error (device sdd1): ext4_discard_preallocations:4104: comm umount: Error -5 reading block bitmap for 773
Feb 28 01:22:52 str-1 kernel: EXT4-fs (sdd1): previous I/O error to superblock detected

Re: Ext4 errors in linux /var/log/messages

[Theodore Y. Ts'o]

On Fri, Mar 06, 2020 at 07:29:22PM +0530, Alok Jain wrote:
> Hi Guys,
> 
> Thanks for providing the forum to check the issue, I already got
> really helpful responses and need one more help wrt to EXT4 log
> messages coming in /var/log/messages file
> 
> 1) I have iSCSI device attached to my VM
> 2) Copied and extracted files in the attached device
> 3) Unmount and disconnected the device, post disconnect operation I
> can see lots of EXT4 file system errors in /var/log/messages file,
> attaching the same (err.txt) for review.
> 
> I am not sure what these log messages are correspond to as the device
> was already disconnected at  Feb I1:22:42, but th*e ext4 log messages
> came after that, can you please review and share your findings.

There's nothing really deep here.  Ext4 wasn't able to write to
/dev/sdd1.  Assuming that /dev/sdd is your iSCSI device, it's probably
because the connection to the iSCSI device had gotten disrupted.  This
is similar to what you might see if you were writing to a USB
thumbstick and the USB device is unceremoniously yanked out.

One possible scenario is thar while the umount was *started*, but it
had not completed writing out dirty blocks before the iSCSI device was
disconnected.  If that's true, this would be a case of operator error.

	       	  	       	    	  - Ted