All of lore.kernel.org
 help / color / mirror / Atom feed
* Atmel Nand PMECC UBI ECC issue
@ 2018-03-26 14:56 Olivier Schonken
  2018-03-26 20:07 ` Richard Weinberger
  0 siblings, 1 reply; 5+ messages in thread
From: Olivier Schonken @ 2018-03-26 14:56 UTC (permalink / raw)
  To: linux-mtd
  Cc: David Woodhouse, Brian Norris, Boris Brezillon, Marek Vasut,
	Richard Weinberger

[-- Attachment #1: Type: text/plain, Size: 3420 bytes --]

Sorry for the resend, seems my gmail editor was in HTML mode which got
rejected by the mailing list.  Humble apologies.

I have run into an issue with the Atmel nand controller on the
SAMA5D36, which I am struggling to debug.

We are using custom hardware based on the SAMA5D36. With Micron
MT29F8G08ABBCAH4 NAND flash.  Kernel version is 4.14.29 - mainline
from kernel.org.  ECC strength is 24 bits with 1024 byte sector size.
The PMECC settings was calculated as per
https://www.at91.com/linux4sam/bin/view/Linux4SAM/PmeccConfigure, with
the nand HEADER value at 0xc0e18e05.

The system works, and only some units present the error, the baffling
part of it, is that a unit can work properly for a long while, and
then suddenly the error presents itself. (Once traced it to a glibc
library file, which means it isn't even due to heavy writing on the
filesystem.) I have noticed that most of the time the PEB in which the
error occurs is the same.  Even after reprogramming the device via
ubiformat, or SAM-BA.

In the attached log output, you will see that there is a UBIFS error,
where it detects a bitflip, which I confirmed by comparing the binary
sequence to the Buildroot generated ubi file.

Using Atmel's SAM-BA to read back the contents of the NAND flash,
yields the correct contents for the page causing the ECC error.

31 18 10 06 00 FE A2 74 FB CF 00 00 00 00 00 00 C5 05 00 00 01 00 00
00 AB 0C 00 00

Starting up linux again results in the same issue.
This extract shows the ubifs magic number with the bitflip. The rest
of the binary sequence matches a unique part of the ubi image.

[   75.140000] 7fe0: b6f8f8e4 becf7a40 b6be5788 b6e9c000 60000010 ffffffff
[   75.150000] UBIFS error (ubi0:0 pid 1): ubifs_check_node: bad magic
0x6101830, expected 0x6101831
[   75.160000] UBIFS error (ubi0:0 pid 1): ubifs_check_node: bad node
at LEB 325:216208
[   75.160000] Not a node, first 24 bytes:
[   75.160000] 00000000: 30 18 10 06 00 fe a2 74 fb cf 00 00 00 00 00
00 c5 05 00 00 01 00 00 00
0......t................
[   75.180000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #706

mtdinfo for the partition in question
Type:                           nand
Eraseblock size:                262144 bytes, 256.0 KiB
Amount of eraseblocks:          2048 (536870912 bytes, 512.0 MiB)
Minimum input/output unit size: 4096 bytes
Sub-page size:                  4096 bytes
OOB size:                       224 bytes
Character device major/minor:   90:10
Bad blocks are allowed:         true
Device is writable:             true

Device tree entry:
        nand_controller: nand-controller {
                status = "okay";

                nand@3 {
                    reg = <0x3 0x0 0x800000>;
                    atmel,rb = <0>;
                    nand-bus-width = <8>;
                    nand-ecc-mode = "hw";
                    nand-ecc-strength = <24>;
                    nand-ecc-step-size = <1024>;
                    nand-on-flash-bbt;
                    label = "atmel_nand";
                };
            };

Attached are the dmesg traces with the ECC issue.  A nanddump of the
block with the ECC error, including OOB contents as per "nanddump -f
nandblock-withoob.ubi /dev/mtd5 -s 0x51c0000 -o -l 262144 &>
nanddump-cmdline-output.txt"

I am almost constantly banging my head against a metaphorical wall on
this one.  Any help in solving it will be much appreciated.

Regards

Olivier Schonken

[-- Attachment #2: dmesg-ubi-nand-issue.txt --]
[-- Type: text/plain, Size: 39489 bytes --]

[    6.410000] LBCal=0x84126888
[    6.510000] ProSLIC_Init done
[    6.510000] ubi0: attaching mtd5
[    6.600000] random: crng init done
[   10.770000] ubi0: scanning is finished
[   10.790000] ubi0: attached mtd5 (name "rootfs", size 512 MiB)
[   10.790000] ubi0: PEB size: 262144 bytes (256 KiB), LEB size: 253952 bytes
[   10.800000] ubi0: min./max. I/O unit sizes: 4096/4096, sub-page size 4096
[   10.810000] ubi0: VID header offset: 4096 (aligned 4096), data offset: 8192
[   10.810000] ubi0: good PEBs: 2048, bad PEBs: 0, corrupted PEBs: 0
[   10.820000] ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128
[   10.830000] ubi0: max/mean erase counter: 3/1, WL threshold: 4096, image sequence number: 412636547
[   10.840000] ubi0: available PEBs: 0, total reserved PEBs: 2048, PEBs reserved for bad PEB handling: 80
[   10.850000] ubi0: background thread "ubi_bgt0d" started, PID 69
[   10.850000] input: gpio_keys as /devices/platform/gpio_keys/input/input4
[   10.860000] rtc-ds1307 1-0068: setting system clock to 2000-01-04 01:12:06 UTC (946948326)
[   10.870000] backlight-power-supply: disabling
[   10.880000] panel-power-supply: disabling
[   10.880000] FUSE_2V5: disabling
[   10.890000] ALSA device list:
[   10.890000]   #0: wm5102_SAMA5D3EK
[   10.900000] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 71
[   11.060000] UBIFS (ubi0:0): recovery needed
[   11.380000] UBIFS (ubi0:0): recovery completed
[   11.390000] UBIFS (ubi0:0): UBIFS: mounted UBI device 0, volume 0, name "rootfs"
[   11.400000] UBIFS (ubi0:0): LEB size: 253952 bytes (248 KiB), min./max. I/O unit sizes: 4096 bytes/4096 bytes
[   11.410000] UBIFS (ubi0:0): FS size: 496222208 bytes (473 MiB, 1954 LEBs), journal size 9404416 bytes (8 MiB, 38 LEBs)
[   11.420000] UBIFS (ubi0:0): reserved for root: 0 bytes (0 KiB)
[   11.420000] UBIFS (ubi0:0): media format: w4/r0 (latest is w5/r0), UUID 4BCB0AE6-7577-4B0F-B9E3-4E857356E020, small LPT model
[   11.440000] VFS: Mounted root (ubifs filesystem) on device 0:15.
[   11.450000] devtmpfs: mounted
[   11.460000] Freeing unused kernel memory: 1024K
[   12.130000] systemd[1]: System time before build time, advancing clock.
[   12.300000] systemd[1]: systemd 232 running in system mode. (-PAM -AUDIT -SELINUX -IMA -APPARMOR -SMACK -SYSVINIT -UTMP -LIBCRYPTSETUP +GCRYPT -GNUTLS +ACL -XZ -LZ4 -SECCOMP -BLKID -ELFUTILS +KMOD -IDN)
[   12.320000] systemd[1]: Detected architecture arm.

Welcome to Buildroot 2017.02!

[   12.370000] systemd[1]: Set hostname to <SAMA5>.
[   12.410000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1609 bytes from PEB 327:221224, read only 1609 bytes, retry
[   12.420000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1609 bytes from PEB 327:221224, read only 1609 bytes, retry
[   12.430000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1609 bytes from PEB 327:221224, read only 1609 bytes, retry
[   12.450000] ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1609 bytes from PEB 327:221224, read 1609 bytes
[   12.460000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   12.460000] Hardware name: Atmel SAMA5
[   12.470000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   12.470000] [<c010b164>] (show_stack) from [<c046cba0>] (ubi_io_read+0x12c/0x304)
[   12.480000] [<c046cba0>] (ubi_io_read) from [<c046a9d8>] (ubi_eba_read_leb+0xb0/0x40c)
[   12.490000] [<c046a9d8>] (ubi_eba_read_leb) from [<c046968c>] (ubi_leb_read+0x70/0xc0)
[   12.500000] [<c046968c>] (ubi_leb_read) from [<c02c2860>] (ubifs_leb_read+0x2c/0x78)
[   12.500000] [<c02c2860>] (ubifs_leb_read) from [<c02c5474>] (fallible_read_node+0x48/0x120)
[   12.510000] [<c02c5474>] (fallible_read_node) from [<c02c75a0>] (ubifs_tnc_locate+0x100/0x1b0)
[   12.520000] [<c02c75a0>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   12.530000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   12.540000] [<c02badb8>] (ubifs_readpage) from [<c01842b4>] (filemap_fault+0x458/0x5d0)
[   12.550000] [<c01842b4>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   12.550000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   12.560000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   12.570000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   12.580000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   12.590000] Exception stack(0xde437fb0 to 0xde437ff8)
[   12.590000] 7fa0:                                     00000000 00000000 b6c544d4 00000000
[   12.600000] 7fc0: 00000000 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a54
[   12.610000] 7fe0: b6f61884 be830a40 b6e748f4 b6e6c440 60000010 ffffffff
[   12.610000] UBIFS error (ubi0:0 pid 1): ubifs_leb_read: reading 1609 bytes from LEB 325:213032 failed, error -74
[   12.620000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   12.630000] Hardware name: Atmel SAMA5
[   12.630000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   12.640000] [<c010b164>] (show_stack) from [<c02c28a8>] (ubifs_leb_read+0x74/0x78)
[   12.650000] [<c02c28a8>] (ubifs_leb_read) from [<c02c5474>] (fallible_read_node+0x48/0x120)
[   12.660000] [<c02c5474>] (fallible_read_node) from [<c02c75a0>] (ubifs_tnc_locate+0x100/0x1b0)
[   12.670000] [<c02c75a0>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   12.670000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   12.680000] [<c02badb8>] (ubifs_readpage) from [<c01842b4>] (filemap_fault+0x458/0x5d0)
[   12.690000] [<c01842b4>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   12.700000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   12.710000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   12.710000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   12.720000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   12.730000] Exception stack(0xde437fb0 to 0xde437ff8)
[   12.740000] 7fa0:                                     00000000 00000000 b6c544d4 00000000
[   12.740000] 7fc0: 00000000 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a54
[   12.750000] 7fe0: b6f61884 be830a40 b6e748f4 b6e6c440 60000010 ffffffff
[   12.760000] UBIFS error (ubi0:0 pid 1): fallible_read_node: cannot read node type 1 from LEB 325:213032, error -74
[   12.770000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1609 bytes from PEB 327:221224, read only 1609 bytes, retry
[   12.780000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1609 bytes from PEB 327:221224, read only 1609 bytes, retry
[   12.800000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1609 bytes from PEB 327:221224, read only 1609 bytes, retry
[   12.810000] ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1609 bytes from PEB 327:221224, read 1609 bytes
[   12.820000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   12.830000] Hardware name: Atmel SAMA5
[   12.830000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   12.840000] [<c010b164>] (show_stack) from [<c046cba0>] (ubi_io_read+0x12c/0x304)
[   12.850000] [<c046cba0>] (ubi_io_read) from [<c046a9d8>] (ubi_eba_read_leb+0xb0/0x40c)
[   12.850000] [<c046a9d8>] (ubi_eba_read_leb) from [<c046968c>] (ubi_leb_read+0x70/0xc0)
[   12.860000] [<c046968c>] (ubi_leb_read) from [<c02c2860>] (ubifs_leb_read+0x2c/0x78)
[   12.870000] [<c02c2860>] (ubifs_leb_read) from [<c02c4274>] (ubifs_read_node+0x90/0x280)
[   12.880000] [<c02c4274>] (ubifs_read_node) from [<c02e0054>] (ubifs_tnc_read_node+0xcc/0xd4)
[   12.890000] [<c02e0054>] (ubifs_tnc_read_node) from [<c02c7634>] (ubifs_tnc_locate+0x194/0x1b0)
[   12.900000] [<c02c7634>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   12.900000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   12.910000] [<c02badb8>] (ubifs_readpage) from [<c01842b4>] (filemap_fault+0x458/0x5d0)
[   12.920000] [<c01842b4>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   12.930000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   12.940000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   12.940000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   12.950000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   12.960000] Exception stack(0xde437fb0 to 0xde437ff8)
[   12.960000] 7fa0:                                     00000000 00000000 b6c544d4 00000000
[   12.970000] 7fc0: 00000000 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a54
[   12.980000] 7fe0: b6f61884 be830a40 b6e748f4 b6e6c440 60000010 ffffffff
[   13.010000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1745 bytes from PEB 327:219472, read only 1745 bytes, retry
[   13.020000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1745 bytes from PEB 327:219472, read only 1745 bytes, retry
[   13.040000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1745 bytes from PEB 327:219472, read only 1745 bytes, retry
[   13.050000] ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1745 bytes from PEB 327:219472, read 1745 bytes
[   13.060000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   13.070000] Hardware name: Atmel SAMA5
[   13.070000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   13.080000] [<c010b164>] (show_stack) from [<c046cba0>] (ubi_io_read+0x12c/0x304)
[   13.090000] [<c046cba0>] (ubi_io_read) from [<c046a9d8>] (ubi_eba_read_leb+0xb0/0x40c)
[   13.090000] [<c046a9d8>] (ubi_eba_read_leb) from [<c046968c>] (ubi_leb_read+0x70/0xc0)
[   13.100000] [<c046968c>] (ubi_leb_read) from [<c02c2860>] (ubifs_leb_read+0x2c/0x78)
[   13.110000] [<c02c2860>] (ubifs_leb_read) from [<c02c5474>] (fallible_read_node+0x48/0x120)
[   13.120000] [<c02c5474>] (fallible_read_node) from [<c02c75a0>] (ubifs_tnc_locate+0x100/0x1b0)
[   13.130000] [<c02c75a0>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   13.130000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   13.140000] [<c02badb8>] (ubifs_readpage) from [<c01842b4>] (filemap_fault+0x458/0x5d0)
[   13.150000] [<c01842b4>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   13.160000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   13.170000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   13.170000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   13.180000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   13.190000] Exception stack(0xde437fb0 to 0xde437ff8)
[   13.200000] 7fa0:                                     00574518 0057423c 00453b84 005741d0
[   13.200000] 7fc0: 005711a8 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a8c
[   13.210000] 7fe0: 00571b8c be830a60 0044fbc4 b6e6bb78 20000010 ffffffff
[   13.220000] UBIFS error (ubi0:0 pid 1): ubifs_leb_read: reading 1745 bytes from LEB 325:211280 failed, error -74
[   13.230000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   13.240000] Hardware name: Atmel SAMA5
[   13.240000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   13.250000] [<c010b164>] (show_stack) from [<c02c28a8>] (ubifs_leb_read+0x74/0x78)
[   13.250000] [<c02c28a8>] (ubifs_leb_read) from [<c02c5474>] (fallible_read_node+0x48/0x120)
[   13.260000] [<c02c5474>] (fallible_read_node) from [<c02c75a0>] (ubifs_tnc_locate+0x100/0x1b0)
[   13.270000] [<c02c75a0>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   13.280000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   13.290000] [<c02badb8>] (ubifs_readpage) from [<c01842b4>] (filemap_fault+0x458/0x5d0)
[   13.300000] [<c01842b4>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   13.300000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   13.310000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   13.320000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   13.330000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   13.340000] Exception stack(0xde437fb0 to 0xde437ff8)
[   13.340000] 7fa0:                                     00574518 0057423c 00453b84 005741d0
[   13.350000] 7fc0: 005711a8 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a8c
[   13.360000] 7fe0: 00571b8c be830a60 0044fbc4 b6e6bb78 20000010 ffffffff
[   13.360000] UBIFS error (ubi0:0 pid 1): fallible_read_node: cannot read node type 1 from LEB 325:211280, error -74
[   13.380000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1745 bytes from PEB 327:219472, read only 1745 bytes, retry
[   13.390000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1745 bytes from PEB 327:219472, read only 1745 bytes, retry
[   13.400000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1745 bytes from PEB 327:219472, read only 1745 bytes, retry
[   13.420000] ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1745 bytes from PEB 327:219472, read 1745 bytes
[   13.430000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   13.430000] Hardware name: Atmel SAMA5
[   13.440000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   13.450000] [<c010b164>] (show_stack) from [<c046cba0>] (ubi_io_read+0x12c/0x304)
[   13.450000] [<c046cba0>] (ubi_io_read) from [<c046a9d8>] (ubi_eba_read_leb+0xb0/0x40c)
[   13.460000] [<c046a9d8>] (ubi_eba_read_leb) from [<c046968c>] (ubi_leb_read+0x70/0xc0)
[   13.470000] [<c046968c>] (ubi_leb_read) from [<c02c2860>] (ubifs_leb_read+0x2c/0x78)
[   13.480000] [<c02c2860>] (ubifs_leb_read) from [<c02c4274>] (ubifs_read_node+0x90/0x280)
[   13.490000] [<c02c4274>] (ubifs_read_node) from [<c02e0054>] (ubifs_tnc_read_node+0xcc/0xd4)
[   13.490000] [<c02e0054>] (ubifs_tnc_read_node) from [<c02c7634>] (ubifs_tnc_locate+0x194/0x1b0)
[   13.500000] [<c02c7634>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   13.510000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   13.520000] [<c02badb8>] (ubifs_readpage) from [<c01842b4>] (filemap_fault+0x458/0x5d0)
[   13.530000] [<c01842b4>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   13.530000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   13.540000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   13.550000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   13.560000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   13.570000] Exception stack(0xde437fb0 to 0xde437ff8)
[   13.570000] 7fa0:                                     00574518 0057423c 00453b84 005741d0
[   13.580000] 7fc0: 005711a8 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a8c
[   13.590000] 7fe0: 00571b8c be830a60 0044fbc4 b6e6bb78 20000010 ffffffff
[   13.600000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1558 bytes from PEB 327:222840, read only 1558 bytes, retry
[   13.610000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1558 bytes from PEB 327:222840, read only 1558 bytes, retry
[   13.620000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1558 bytes from PEB 327:222840, read only 1558 bytes, retry
[   13.640000] ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1558 bytes from PEB 327:222840, read 1558 bytes
[   13.650000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   13.650000] Hardware name: Atmel SAMA5
[   13.660000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   13.670000] [<c010b164>] (show_stack) from [<c046cba0>] (ubi_io_read+0x12c/0x304)
[   13.670000] [<c046cba0>] (ubi_io_read) from [<c046a9d8>] (ubi_eba_read_leb+0xb0/0x40c)
[   13.680000] [<c046a9d8>] (ubi_eba_read_leb) from [<c046968c>] (ubi_leb_read+0x70/0xc0)
[   13.690000] [<c046968c>] (ubi_leb_read) from [<c02c2860>] (ubifs_leb_read+0x2c/0x78)
[   13.700000] [<c02c2860>] (ubifs_leb_read) from [<c02c5474>] (fallible_read_node+0x48/0x120)
[   13.700000] [<c02c5474>] (fallible_read_node) from [<c02c75a0>] (ubifs_tnc_locate+0x100/0x1b0)
[   13.710000] [<c02c75a0>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   13.720000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   13.730000] [<c02badb8>] (ubifs_readpage) from [<c01842b4>] (filemap_fault+0x458/0x5d0)
[   13.740000] [<c01842b4>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   13.750000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   13.750000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   13.760000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   13.770000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   13.780000] Exception stack(0xde437fb0 to 0xde437ff8)
[   13.780000] 7fa0:                                     00574760 00000000 00000064 00000000
[   13.790000] 7fc0: 005711a8 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a8c
[   13.800000] 7fe0: 0057162c be830a60 0044fbf0 b6e6d96c 60000010 ffffffff
[   13.810000] UBIFS error (ubi0:0 pid 1): ubifs_leb_read: reading 1558 bytes from LEB 325:214648 failed, error -74
[   13.820000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   13.820000] Hardware name: Atmel SAMA5
[   13.830000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   13.830000] [<c010b164>] (show_stack) from [<c02c28a8>] (ubifs_leb_read+0x74/0x78)
[   13.840000] [<c02c28a8>] (ubifs_leb_read) from [<c02c5474>] (fallible_read_node+0x48/0x120)
[   13.850000] [<c02c5474>] (fallible_read_node) from [<c02c75a0>] (ubifs_tnc_locate+0x100/0x1b0)
[   13.860000] [<c02c75a0>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   13.870000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   13.870000] [<c02badb8>] (ubifs_readpage) from [<c01842b4>] (filemap_fault+0x458/0x5d0)
[   13.880000] [<c01842b4>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   13.890000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   13.900000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   13.910000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   13.910000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   13.920000] Exception stack(0xde437fb0 to 0xde437ff8)
[   13.930000] 7fa0:                                     00574760 00000000 00000064 00000000
[   13.940000] 7fc0: 005711a8 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a8c
[   13.940000] 7fe0: 0057162c be830a60 0044fbf0 b6e6d96c 60000010 ffffffff
[   13.950000] UBIFS error (ubi0:0 pid 1): fallible_read_node: cannot read node type 1 from LEB 325:214648, error -74
[   13.960000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1558 bytes from PEB 327:222840, read only 1558 bytes, retry
[   13.980000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1558 bytes from PEB 327:222840, read only 1558 bytes, retry
[   13.990000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1558 bytes from PEB 327:222840, read only 1558 bytes, retry
[   14.000000] ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1558 bytes from PEB 327:222840, read 1558 bytes
[   14.010000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   14.020000] Hardware name: Atmel SAMA5
[   14.020000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   14.030000] [<c010b164>] (show_stack) from [<c046cba0>] (ubi_io_read+0x12c/0x304)
[   14.040000] [<c046cba0>] (ubi_io_read) from [<c046a9d8>] (ubi_eba_read_leb+0xb0/0x40c)
[   14.050000] [<c046a9d8>] (ubi_eba_read_leb) from [<c046968c>] (ubi_leb_read+0x70/0xc0)
[   14.050000] [<c046968c>] (ubi_leb_read) from [<c02c2860>] (ubifs_leb_read+0x2c/0x78)
[   14.060000] [<c02c2860>] (ubifs_leb_read) from [<c02c4274>] (ubifs_read_node+0x90/0x280)
[   14.070000] [<c02c4274>] (ubifs_read_node) from [<c02e0054>] (ubifs_tnc_read_node+0xcc/0xd4)
[   14.080000] [<c02e0054>] (ubifs_tnc_read_node) from [<c02c7634>] (ubifs_tnc_locate+0x194/0x1b0)
[   14.090000] [<c02c7634>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   14.090000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   14.100000] [<c02badb8>] (ubifs_readpage) from [<c01842b4>] (filemap_fault+0x458/0x5d0)
[   14.110000] [<c01842b4>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   14.120000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   14.130000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   14.130000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   14.140000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   14.150000] Exception stack(0xde437fb0 to 0xde437ff8)
[   14.160000] 7fa0:                                     00574760 00000000 00000064 00000000
[   14.160000] 7fc0: 005711a8 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a8c
[   14.170000] 7fe0: 0057162c be830a60 0044fbf0 b6e6d96c 60000010 ffffffff
[   14.180000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read only 1477 bytes, retry
[   14.190000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read only 1477 bytes, retry
[   14.210000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read only 1477 bytes, retry
[   14.220000] ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read 1477 bytes
[   14.230000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   14.240000] Hardware name: Atmel SAMA5
[   14.240000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   14.250000] [<c010b164>] (show_stack) from [<c046cba0>] (ubi_io_read+0x12c/0x304)
[   14.260000] [<c046cba0>] (ubi_io_read) from [<c046a9d8>] (ubi_eba_read_leb+0xb0/0x40c)
[   14.270000] [<c046a9d8>] (ubi_eba_read_leb) from [<c046968c>] (ubi_leb_read+0x70/0xc0)
[   14.270000] [<c046968c>] (ubi_leb_read) from [<c02c2860>] (ubifs_leb_read+0x2c/0x78)
[   14.280000] [<c02c2860>] (ubifs_leb_read) from [<c02c5474>] (fallible_read_node+0x48/0x120)
[   14.290000] [<c02c5474>] (fallible_read_node) from [<c02c75a0>] (ubifs_tnc_locate+0x100/0x1b0)
[   14.300000] [<c02c75a0>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   14.310000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   14.320000] [<c02badb8>] (ubifs_readpage) from [<c01842b4>] (filemap_fault+0x458/0x5d0)
[   14.320000] [<c01842b4>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   14.330000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   14.340000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   14.350000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   14.360000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   14.360000] Exception stack(0xde437fb0 to 0xde437ff8)
[   14.370000] 7fa0:                                     00000000 00000000 b6f854c0 00574518
[   14.380000] 7fc0: 005711a8 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a5c
[   14.380000] 7fe0: b6f618e4 be830a40 b6bb7788 b6e6e000 60000010 ffffffff
[   14.390000] UBIFS error (ubi0:0 pid 1): ubifs_leb_read: reading 1477 bytes from LEB 325:216208 failed, error -74
[   14.400000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   14.410000] Hardware name: Atmel SAMA5
[   14.410000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   14.420000] [<c010b164>] (show_stack) from [<c02c28a8>] (ubifs_leb_read+0x74/0x78)
[   14.430000] [<c02c28a8>] (ubifs_leb_read) from [<c02c5474>] (fallible_read_node+0x48/0x120)
[   14.440000] [<c02c5474>] (fallible_read_node) from [<c02c75a0>] (ubifs_tnc_locate+0x100/0x1b0)
[   14.440000] [<c02c75a0>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   14.450000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   14.460000] [<c02badb8>] (ubifs_readpage) from [<c01842b4>] (filemap_fault+0x458/0x5d0)
[   14.470000] [<c01842b4>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   14.480000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   14.480000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   14.490000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   14.500000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   14.510000] Exception stack(0xde437fb0 to 0xde437ff8)
[   14.510000] 7fa0:                                     00000000 00000000 b6f854c0 00574518
[   14.520000] 7fc0: 005711a8 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a5c
[   14.530000] 7fe0: b6f618e4 be830a40 b6bb7788 b6e6e000 60000010 ffffffff
[   14.540000] UBIFS error (ubi0:0 pid 1): fallible_read_node: cannot read node type 1 from LEB 325:216208, error -74
[   14.550000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read only 1477 bytes, retry
[   14.560000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read only 1477 bytes, retry
[   14.580000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read only 1477 bytes, retry
[   14.590000] ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read 1477 bytes
[   14.600000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   14.610000] Hardware name: Atmel SAMA5
[   14.610000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   14.620000] [<c010b164>] (show_stack) from [<c046cba0>] (ubi_io_read+0x12c/0x304)
[   14.630000] [<c046cba0>] (ubi_io_read) from [<c046a9d8>] (ubi_eba_read_leb+0xb0/0x40c)
[   14.630000] [<c046a9d8>] (ubi_eba_read_leb) from [<c046968c>] (ubi_leb_read+0x70/0xc0)
[   14.640000] [<c046968c>] (ubi_leb_read) from [<c02c2860>] (ubifs_leb_read+0x2c/0x78)
[   14.650000] [<c02c2860>] (ubifs_leb_read) from [<c02c4274>] (ubifs_read_node+0x90/0x280)
[   14.660000] [<c02c4274>] (ubifs_read_node) from [<c02e0054>] (ubifs_tnc_read_node+0xcc/0xd4)
[   14.670000] [<c02e0054>] (ubifs_tnc_read_node) from [<c02c7634>] (ubifs_tnc_locate+0x194/0x1b0)
[   14.680000] [<c02c7634>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   14.680000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   14.690000] [<c02badb8>] (ubifs_readpage) from [<c01842b4>] (filemap_fault+0x458/0x5d0)
[   14.700000] [<c01842b4>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   14.710000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   14.710000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   14.720000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   14.730000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   14.740000] Exception stack(0xde437fb0 to 0xde437ff8)
[   14.740000] 7fa0:                                     00000000 00000000 b6f854c0 00574518
[   14.750000] 7fc0: 005711a8 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a5c
[   14.760000] 7fe0: b6f618e4 be830a40 b6bb7788 b6e6e000 60000010 ffffffff
[   14.770000] UBIFS error (ubi0:0 pid 1): ubifs_check_node: bad magic 0x6101830, expected 0x6101831
[   14.780000] UBIFS error (ubi0:0 pid 1): ubifs_check_node: bad node at LEB 325:216208
[   14.780000] Not a node, first 24 bytes:
[   14.780000] 00000000: 30 18 10 06 00 fe a2 74 fb cf 00 00 00 00 00 00 c5 05 00 00 01 00 00 00                          0......t................
[   14.800000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   14.810000] Hardware name: Atmel SAMA5
[   14.810000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   14.820000] [<c010b164>] (show_stack) from [<c02c2df0>] (ubifs_check_node+0xc4/0x224)
[   14.830000] [<c02c2df0>] (ubifs_check_node) from [<c02c4334>] (ubifs_read_node+0x150/0x280)
[   14.830000] [<c02c4334>] (ubifs_read_node) from [<c02e0054>] (ubifs_tnc_read_node+0xcc/0xd4)
[   14.840000] [<c02e0054>] (ubifs_tnc_read_node) from [<c02c7634>] (ubifs_tnc_locate+0x194/0x1b0)
[   14.850000] [<c02c7634>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   14.860000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   14.870000] [<c02badb8>] (ubifs_readpage) from [<c01842b4>] (filemap_fault+0x458/0x5d0)
[   14.880000] [<c01842b4>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   14.880000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   14.890000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   14.900000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   14.910000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   14.920000] Exception stack(0xde437fb0 to 0xde437ff8)
[   14.920000] 7fa0:                                     00000000 00000000 b6f854c0 00574518
[   14.930000] 7fc0: 005711a8 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a5c
[   14.940000] 7fe0: b6f618e4 be830a40 b6bb7788 b6e6e000 60000010 ffffffff
[   14.940000] UBIFS error (ubi0:0 pid 1): ubifs_read_node: expected node type 1
[   14.950000] UBIFS error (ubi0:0 pid 1): do_readpage: cannot read page 327 of inode 3243, error -117
[   14.960000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read only 1477 bytes, retry
[   14.980000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read only 1477 bytes, retry
[   14.990000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read only 1477 bytes, retry
[   15.000000] ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read 1477 bytes
[   15.010000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   15.020000] Hardware name: Atmel SAMA5
[   15.020000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   15.030000] [<c010b164>] (show_stack) from [<c046cba0>] (ubi_io_read+0x12c/0x304)
[   15.040000] [<c046cba0>] (ubi_io_read) from [<c046a9d8>] (ubi_eba_read_leb+0xb0/0x40c)
[   15.050000] [<c046a9d8>] (ubi_eba_read_leb) from [<c046968c>] (ubi_leb_read+0x70/0xc0)
[   15.060000] [<c046968c>] (ubi_leb_read) from [<c02c2860>] (ubifs_leb_read+0x2c/0x78)
[   15.060000] [<c02c2860>] (ubifs_leb_read) from [<c02c5474>] (fallible_read_node+0x48/0x120)
[   15.070000] [<c02c5474>] (fallible_read_node) from [<c02c75a0>] (ubifs_tnc_locate+0x100/0x1b0)
[   15.080000] [<c02c75a0>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   15.090000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   15.100000] [<c02badb8>] (ubifs_readpage) from [<c0184030>] (filemap_fault+0x1d4/0x5d0)
[   15.100000] [<c0184030>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   15.110000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   15.120000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   15.130000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   15.140000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   15.140000] Exception stack(0xde437fb0 to 0xde437ff8)
[   15.150000] 7fa0:                                     00000000 00000000 b6f854c0 00574518
[   15.160000] 7fc0: 005711a8 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a5c
[   15.170000] 7fe0: b6f618e4 be830a40 b6bb7788 b6e6e000 60000010 ffffffff
[   15.170000] UBIFS error (ubi0:0 pid 1): ubifs_leb_read: reading 1477 bytes from LEB 325:216208 failed, error -74
[   15.180000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   15.190000] Hardware name: Atmel SAMA5
[   15.190000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   15.200000] [<c010b164>] (show_stack) from [<c02c28a8>] (ubifs_leb_read+0x74/0x78)
[   15.210000] [<c02c28a8>] (ubifs_leb_read) from [<c02c5474>] (fallible_read_node+0x48/0x120)
[   15.220000] [<c02c5474>] (fallible_read_node) from [<c02c75a0>] (ubifs_tnc_locate+0x100/0x1b0)
[   15.230000] [<c02c75a0>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   15.230000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   15.240000] [<c02badb8>] (ubifs_readpage) from [<c0184030>] (filemap_fault+0x1d4/0x5d0)
[   15.250000] [<c0184030>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   15.260000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   15.270000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   15.270000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   15.280000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   15.290000] Exception stack(0xde437fb0 to 0xde437ff8)
[   15.290000] 7fa0:                                     00000000 00000000 b6f854c0 00574518
[   15.300000] 7fc0: 005711a8 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a5c
[   15.310000] 7fe0: b6f618e4 be830a40 b6bb7788 b6e6e000 60000010 ffffffff
[   15.320000] UBIFS error (ubi0:0 pid 1): fallible_read_node: cannot read node type 1 from LEB 325:216208, error -74
[   15.330000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read only 1477 bytes, retry
[   15.340000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read only 1477 bytes, retry
[   15.360000] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read only 1477 bytes, retry
[   15.370000] ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1477 bytes from PEB 327:224400, read 1477 bytes
[   15.380000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   15.390000] Hardware name: Atmel SAMA5
[   15.390000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   15.400000] [<c010b164>] (show_stack) from [<c046cba0>] (ubi_io_read+0x12c/0x304)
[   15.410000] [<c046cba0>] (ubi_io_read) from [<c046a9d8>] (ubi_eba_read_leb+0xb0/0x40c)
[   15.420000] [<c046a9d8>] (ubi_eba_read_leb) from [<c046968c>] (ubi_leb_read+0x70/0xc0)
[   15.420000] [<c046968c>] (ubi_leb_read) from [<c02c2860>] (ubifs_leb_read+0x2c/0x78)
[   15.430000] [<c02c2860>] (ubifs_leb_read) from [<c02c4274>] (ubifs_read_node+0x90/0x280)
[   15.440000] [<c02c4274>] (ubifs_read_node) from [<c02e0054>] (ubifs_tnc_read_node+0xcc/0xd4)
[   15.450000] [<c02e0054>] (ubifs_tnc_read_node) from [<c02c7634>] (ubifs_tnc_locate+0x194/0x1b0)
[   15.460000] [<c02c7634>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   15.460000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   15.470000] [<c02badb8>] (ubifs_readpage) from [<c0184030>] (filemap_fault+0x1d4/0x5d0)
[   15.480000] [<c0184030>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   15.490000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   15.500000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   15.500000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   15.510000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   15.520000] Exception stack(0xde437fb0 to 0xde437ff8)
[   15.530000] 7fa0:                                     00000000 00000000 b6f854c0 00574518
[   15.530000] 7fc0: 005711a8 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a5c
[   15.540000] 7fe0: b6f618e4 be830a40 b6bb7788 b6e6e000 60000010 ffffffff
[   15.550000] UBIFS error (ubi0:0 pid 1): ubifs_check_node: bad magic 0x6101830, expected 0x6101831
[   15.560000] UBIFS error (ubi0:0 pid 1): ubifs_check_node: bad node at LEB 325:216208
[   15.570000] Not a node, first 24 bytes:
[   15.570000] 00000000: 30 18 10 06 00 fe a2 74 fb cf 00 00 00 00 00 00 c5 05 00 00 01 00 00 00                          0......t................
[   15.580000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #713
[   15.590000] Hardware name: Atmel SAMA5
[   15.590000] [<c010da6c>] (unwind_backtrace) from [<c010b164>] (show_stack+0x10/0x14)
[   15.600000] [<c010b164>] (show_stack) from [<c02c2df0>] (ubifs_check_node+0xc4/0x224)
[   15.610000] [<c02c2df0>] (ubifs_check_node) from [<c02c4334>] (ubifs_read_node+0x150/0x280)
[   15.620000] [<c02c4334>] (ubifs_read_node) from [<c02e0054>] (ubifs_tnc_read_node+0xcc/0xd4)
[   15.620000] [<c02e0054>] (ubifs_tnc_read_node) from [<c02c7634>] (ubifs_tnc_locate+0x194/0x1b0)
[   15.630000] [<c02c7634>] (ubifs_tnc_locate) from [<c02b9c5c>] (do_readpage+0x190/0x464)
[   15.640000] [<c02b9c5c>] (do_readpage) from [<c02badb8>] (ubifs_readpage+0x38/0x508)
[   15.650000] [<c02badb8>] (ubifs_readpage) from [<c0184030>] (filemap_fault+0x1d4/0x5d0)
[   15.660000] [<c0184030>] (filemap_fault) from [<c01a8430>] (__do_fault+0x18/0x70)
[   15.660000] [<c01a8430>] (__do_fault) from [<c01ab29c>] (handle_mm_fault+0x544/0x1018)
[   15.670000] [<c01ab29c>] (handle_mm_fault) from [<c010e350>] (do_page_fault+0x1f8/0x288)
[   15.680000] [<c010e350>] (do_page_fault) from [<c010139c>] (do_PrefetchAbort+0x34/0x98)
[   15.690000] [<c010139c>] (do_PrefetchAbort) from [<c010c2c8>] (ret_from_exception+0x0/0x18)
[   15.700000] Exception stack(0xde437fb0 to 0xde437ff8)
[   15.700000] 7fa0:                                     00000000 00000000 b6f854c0 00574518
[   15.710000] 7fc0: 005711a8 00000000 004429cc 00000000 00000000 00000000 005711a8 be830a5c
[   15.720000] 7fe0: b6f618e4 be830a40 b6bb7788 b6e6e000 60000010 ffffffff
[   15.730000] UBIFS error (ubi0:0 pid 1): ubifs_read_node: expected node type 1
[   15.730000] UBIFS error (ubi0:0 pid 1): do_readpage: cannot read page 327 of inode 3243, error -117
[   15.790000] systemd[1]: Caught <BUS>, dumped core as pid 72.
[   15.800000] systemd[1]: Freezing execution.

[-- Attachment #3: nandblock-withoob.ubi --]
[-- Type: application/octet-stream, Size: 276480 bytes --]

[-- Attachment #4: nanddump-cmdline-output.txt --]
[-- Type: text/plain, Size: 244 bytes --]

ECC failed: 1
ECC corrected: 0
Number of bad blocks: 0
Number of bbt blocks: 0
Block size 262144, page size 4096, OOB size 224
Dumping data starting at 0x051c0000 and ending at 0x05200000...
ECC: 1 uncorrectable bitflip(s) at offset 0x051f6000

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Atmel Nand PMECC UBI ECC issue
  2018-03-26 14:56 Atmel Nand PMECC UBI ECC issue Olivier Schonken
@ 2018-03-26 20:07 ` Richard Weinberger
  2018-03-27  8:28   ` Olivier Schonken
  0 siblings, 1 reply; 5+ messages in thread
From: Richard Weinberger @ 2018-03-26 20:07 UTC (permalink / raw)
  To: Olivier Schonken
  Cc: linux-mtd, David Woodhouse, Brian Norris, Boris Brezillon, Marek Vasut

Oliver,

Am Montag, 26. März 2018, 16:56:17 CEST schrieb Olivier Schonken:
> Sorry for the resend, seems my gmail editor was in HTML mode which got
> rejected by the mailing list.  Humble apologies.
> 
> I have run into an issue with the Atmel nand controller on the
> SAMA5D36, which I am struggling to debug.
> 
> We are using custom hardware based on the SAMA5D36. With Micron
> MT29F8G08ABBCAH4 NAND flash.  Kernel version is 4.14.29 - mainline
> from kernel.org.  ECC strength is 24 bits with 1024 byte sector size.
> The PMECC settings was calculated as per
> https://www.at91.com/linux4sam/bin/view/Linux4SAM/PmeccConfigure, with
> the nand HEADER value at 0xc0e18e05.
> 
> The system works, and only some units present the error, the baffling
> part of it, is that a unit can work properly for a long while, and
> then suddenly the error presents itself. (Once traced it to a glibc
> library file, which means it isn't even due to heavy writing on the
> filesystem.) I have noticed that most of the time the PEB in which the
> error occurs is the same.  Even after reprogramming the device via
> ubiformat, or SAM-BA.
> 
> In the attached log output, you will see that there is a UBIFS error,
> where it detects a bitflip, which I confirmed by comparing the binary
> sequence to the Buildroot generated ubi file.
> 
> Using Atmel's SAM-BA to read back the contents of the NAND flash,
> yields the correct contents for the page causing the ECC error.
> 
> 31 18 10 06 00 FE A2 74 FB CF 00 00 00 00 00 00 C5 05 00 00 01 00 00
> 00 AB 0C 00 00

At which offset it this?

> Starting up linux again results in the same issue.
> This extract shows the ubifs magic number with the bitflip. The rest
> of the binary sequence matches a unique part of the ubi image.
> 
> [   75.140000] 7fe0: b6f8f8e4 becf7a40 b6be5788 b6e9c000 60000010 ffffffff
> [   75.150000] UBIFS error (ubi0:0 pid 1): ubifs_check_node: bad magic
> 0x6101830, expected 0x6101831
> [   75.160000] UBIFS error (ubi0:0 pid 1): ubifs_check_node: bad node
> at LEB 325:216208
> [   75.160000] Not a node, first 24 bytes:
> [   75.160000] 00000000: 30 18 10 06 00 fe a2 74 fb cf 00 00 00 00 00
> 00 c5 05 00 00 01 00 00 00
> 0......t................
> [   75.180000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #706
> 
> mtdinfo for the partition in question
> Type:                           nand
> Eraseblock size:                262144 bytes, 256.0 KiB
> Amount of eraseblocks:          2048 (536870912 bytes, 512.0 MiB)
> Minimum input/output unit size: 4096 bytes
> Sub-page size:                  4096 bytes
> OOB size:                       224 bytes
> Character device major/minor:   90:10
> Bad blocks are allowed:         true
> Device is writable:             true
> 
> Device tree entry:
>         nand_controller: nand-controller {
>                 status = "okay";
> 
>                 nand@3 {
>                     reg = <0x3 0x0 0x800000>;
>                     atmel,rb = <0>;
>                     nand-bus-width = <8>;
>                     nand-ecc-mode = "hw";
>                     nand-ecc-strength = <24>;
>                     nand-ecc-step-size = <1024>;
>                     nand-on-flash-bbt;
>                     label = "atmel_nand";
>                 };
>             };
> 
> Attached are the dmesg traces with the ECC issue.  A nanddump of the
> block with the ECC error, including OOB contents as per "nanddump -f
> nandblock-withoob.ubi /dev/mtd5 -s 0x51c0000 -o -l 262144 &>
> nanddump-cmdline-output.txt"

Can you please share the dump without OOB?
UBI does not use OOB, so we don't need it and can use offsets as seen by UBI 
and UBIFS as-is. :)

Thanks,
//richard

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Atmel Nand PMECC UBI ECC issue
  2018-03-26 20:07 ` Richard Weinberger
@ 2018-03-27  8:28   ` Olivier Schonken
  2018-03-27 14:08     ` Richard Weinberger
  0 siblings, 1 reply; 5+ messages in thread
From: Olivier Schonken @ 2018-03-27  8:28 UTC (permalink / raw)
  To: Richard Weinberger
  Cc: linux-mtd, David Woodhouse, Brian Norris, Boris Brezillon, Marek Vasut

[-- Attachment #1: Type: text/plain, Size: 4051 bytes --]

Hi Richard

The bytes in question is at offset 36c90.

Attached is the dump without the OOB.

Regards

Olivier

On Mon, Mar 26, 2018 at 10:07 PM, Richard Weinberger <richard@nod.at> wrote:
> Oliver,
>
> Am Montag, 26. März 2018, 16:56:17 CEST schrieb Olivier Schonken:
>> Sorry for the resend, seems my gmail editor was in HTML mode which got
>> rejected by the mailing list.  Humble apologies.
>>
>> I have run into an issue with the Atmel nand controller on the
>> SAMA5D36, which I am struggling to debug.
>>
>> We are using custom hardware based on the SAMA5D36. With Micron
>> MT29F8G08ABBCAH4 NAND flash.  Kernel version is 4.14.29 - mainline
>> from kernel.org.  ECC strength is 24 bits with 1024 byte sector size.
>> The PMECC settings was calculated as per
>> https://www.at91.com/linux4sam/bin/view/Linux4SAM/PmeccConfigure, with
>> the nand HEADER value at 0xc0e18e05.
>>
>> The system works, and only some units present the error, the baffling
>> part of it, is that a unit can work properly for a long while, and
>> then suddenly the error presents itself. (Once traced it to a glibc
>> library file, which means it isn't even due to heavy writing on the
>> filesystem.) I have noticed that most of the time the PEB in which the
>> error occurs is the same.  Even after reprogramming the device via
>> ubiformat, or SAM-BA.
>>
>> In the attached log output, you will see that there is a UBIFS error,
>> where it detects a bitflip, which I confirmed by comparing the binary
>> sequence to the Buildroot generated ubi file.
>>
>> Using Atmel's SAM-BA to read back the contents of the NAND flash,
>> yields the correct contents for the page causing the ECC error.
>>
>> 31 18 10 06 00 FE A2 74 FB CF 00 00 00 00 00 00 C5 05 00 00 01 00 00
>> 00 AB 0C 00 00
>
> At which offset it this?
>
>> Starting up linux again results in the same issue.
>> This extract shows the ubifs magic number with the bitflip. The rest
>> of the binary sequence matches a unique part of the ubi image.
>>
>> [   75.140000] 7fe0: b6f8f8e4 becf7a40 b6be5788 b6e9c000 60000010 ffffffff
>> [   75.150000] UBIFS error (ubi0:0 pid 1): ubifs_check_node: bad magic
>> 0x6101830, expected 0x6101831
>> [   75.160000] UBIFS error (ubi0:0 pid 1): ubifs_check_node: bad node
>> at LEB 325:216208
>> [   75.160000] Not a node, first 24 bytes:
>> [   75.160000] 00000000: 30 18 10 06 00 fe a2 74 fb cf 00 00 00 00 00
>> 00 c5 05 00 00 01 00 00 00
>> 0......t................
>> [   75.180000] CPU: 0 PID: 1 Comm: systemd Not tainted 4.14.29+ #706
>>
>> mtdinfo for the partition in question
>> Type:                           nand
>> Eraseblock size:                262144 bytes, 256.0 KiB
>> Amount of eraseblocks:          2048 (536870912 bytes, 512.0 MiB)
>> Minimum input/output unit size: 4096 bytes
>> Sub-page size:                  4096 bytes
>> OOB size:                       224 bytes
>> Character device major/minor:   90:10
>> Bad blocks are allowed:         true
>> Device is writable:             true
>>
>> Device tree entry:
>>         nand_controller: nand-controller {
>>                 status = "okay";
>>
>>                 nand@3 {
>>                     reg = <0x3 0x0 0x800000>;
>>                     atmel,rb = <0>;
>>                     nand-bus-width = <8>;
>>                     nand-ecc-mode = "hw";
>>                     nand-ecc-strength = <24>;
>>                     nand-ecc-step-size = <1024>;
>>                     nand-on-flash-bbt;
>>                     label = "atmel_nand";
>>                 };
>>             };
>>
>> Attached are the dmesg traces with the ECC issue.  A nanddump of the
>> block with the ECC error, including OOB contents as per "nanddump -f
>> nandblock-withoob.ubi /dev/mtd5 -s 0x51c0000 -o -l 262144 &>
>> nanddump-cmdline-output.txt"
>
> Can you please share the dump without OOB?
> UBI does not use OOB, so we don't need it and can use offsets as seen by UBI
> and UBIFS as-is. :)
>
> Thanks,
> //richard

[-- Attachment #2: nandblock.ubi --]
[-- Type: application/octet-stream, Size: 262144 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Atmel Nand PMECC UBI ECC issue
  2018-03-27  8:28   ` Olivier Schonken
@ 2018-03-27 14:08     ` Richard Weinberger
  2018-03-28  8:32       ` Boris Brezillon
  0 siblings, 1 reply; 5+ messages in thread
From: Richard Weinberger @ 2018-03-27 14:08 UTC (permalink / raw)
  To: Olivier Schonken
  Cc: linux-mtd, David Woodhouse, Brian Norris, Boris Brezillon, Marek Vasut

Oliver,

Am Dienstag, 27. März 2018, 10:28:52 CEST schrieb Olivier Schonken:
> Hi Richard
> 
> The bytes in question is at offset 36c90.

Okay. At this offset we have a UBIFS data node of length 1477.
One bit of the UBIFS magic changed. 0x30 instead of 0x31
Fun fact, the UBIFS CRC32 is correct. UBIFS magic and the CRC32 are not 
covered by the CRC.
So, really just one bit changed, in the magic.

Thanks,
//richard

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Atmel Nand PMECC UBI ECC issue
  2018-03-27 14:08     ` Richard Weinberger
@ 2018-03-28  8:32       ` Boris Brezillon
  0 siblings, 0 replies; 5+ messages in thread
From: Boris Brezillon @ 2018-03-28  8:32 UTC (permalink / raw)
  To: Richard Weinberger
  Cc: Olivier Schonken, David Woodhouse, Brian Norris, linux-mtd, Marek Vasut

On Tue, 27 Mar 2018 16:08:46 +0200
Richard Weinberger <richard@nod.at> wrote:

> Oliver,
> 
> Am Dienstag, 27. März 2018, 10:28:52 CEST schrieb Olivier Schonken:
> > Hi Richard
> > 
> > The bytes in question is at offset 36c90.  
> 
> Okay. At this offset we have a UBIFS data node of length 1477.
> One bit of the UBIFS magic changed. 0x30 instead of 0x31
> Fun fact, the UBIFS CRC32 is correct. UBIFS magic and the CRC32 are not 
> covered by the CRC.
> So, really just one bit changed, in the magic.

Looks like this problem was caused by a bug in the atmel NFC driver.
Fix is available here [1].

Regards,

Boris

[1]http://patchwork.ozlabs.org/patch/891725/
-- 
Boris Brezillon, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2018-03-28  8:32 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-03-26 14:56 Atmel Nand PMECC UBI ECC issue Olivier Schonken
2018-03-26 20:07 ` Richard Weinberger
2018-03-27  8:28   ` Olivier Schonken
2018-03-27 14:08     ` Richard Weinberger
2018-03-28  8:32       ` Boris Brezillon

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.