* ubi/ubifs performance comparison on two NAND devices @ 2019-02-27 20:39 Tim Harvey 2019-02-27 22:12 ` Richard Weinberger 0 siblings, 1 reply; 8+ messages in thread From: Tim Harvey @ 2019-02-27 20:39 UTC (permalink / raw) To: linux-mtd Greetings, I've got two embedded boards (IMX6 using gpmi-nand driver) that are identical except for the NAND FLASH and am trying to understand why I'm seeing a massive performance difference when it comes to ubi and ubifs (in particular ubi scanning and UBIFS resizing). Micron MT29F16G08ADACAH4: - page size: 4320 bytes (4096 + 224 bytes) - block size: 64 pages (256K + 14K bytes) - device size: 16Gb - 30ns per-block read - 300us per-block program - 2.0ms per-block erase Cypress S34ML16G202BHI000 - page size: 2176 bytes (2048+128) - block size: 64 pages (128K + 8K) - device size: 16Gb - 25ns per-block read - 300us per-block program - 3.5ms per-block erase The parts are relatively close in per-block performance but because the Micron has twice the block size I think of it being twice as fast on a per-byte basis compared to the Cypress and I do see this relative performance when testing read/write/erase both in modern U-Boot and latest Linux. What doesn't add up is that the Cypress part takes about 7x longer to complete the ubi attach (between 'attaching mtd' and 'scanning is finished' and about 100x longer to complete the UBIFS free space fixup (between 'start fixing up free space' and 'free space fixup complete') performed on the first mount of a ubifs filesystem that was created with auto-resize enabled. Both parts are 2GiB and the ubi scanning on attach in the kernel goes from ~4s on Micron to ~28s on Cypress and the UBIFS free space fixup goes from 0.5s on Micron to a whopping 56s on Cypress. Any ideas why would I be seeing this kind of performance difference when the raw erase/read/write performance is (both datasheet and Linux tests) only 2x slower? Anywhere I can look to improve performance of the Cypress part? Best Regards, Tim ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: ubi/ubifs performance comparison on two NAND devices 2019-02-27 20:39 ubi/ubifs performance comparison on two NAND devices Tim Harvey @ 2019-02-27 22:12 ` Richard Weinberger 2019-02-27 22:43 ` Tim Harvey 0 siblings, 1 reply; 8+ messages in thread From: Richard Weinberger @ 2019-02-27 22:12 UTC (permalink / raw) To: Tim Harvey; +Cc: linux-mtd On Wed, Feb 27, 2019 at 9:39 PM Tim Harvey <tharvey@gateworks.com> wrote: > > Greetings, > > I've got two embedded boards (IMX6 using gpmi-nand driver) that are > identical except for the NAND FLASH and am trying to understand why > I'm seeing a massive performance difference when it comes to ubi and > ubifs (in particular ubi scanning and UBIFS resizing). > > Micron MT29F16G08ADACAH4: > - page size: 4320 bytes (4096 + 224 bytes) > - block size: 64 pages (256K + 14K bytes) > - device size: 16Gb > - 30ns per-block read > - 300us per-block program > - 2.0ms per-block erase > > Cypress S34ML16G202BHI000 > - page size: 2176 bytes (2048+128) > - block size: 64 pages (128K + 8K) > - device size: 16Gb > - 25ns per-block read > - 300us per-block program > - 3.5ms per-block erase > > The parts are relatively close in per-block performance but because > the Micron has twice the block size I think of it being twice as fast > on a per-byte basis compared to the Cypress and I do see this relative > performance when testing read/write/erase both in modern U-Boot and > latest Linux. What doesn't add up is that the Cypress part takes about > 7x longer to complete the ubi attach (between 'attaching mtd' and > 'scanning is finished' and about 100x longer to complete the UBIFS > free space fixup (between 'start fixing up free space' and 'free space > fixup complete') performed on the first mount of a ubifs filesystem > that was created with auto-resize enabled. > > Both parts are 2GiB and the ubi scanning on attach in the kernel goes > from ~4s on Micron to ~28s on Cypress and the UBIFS free space fixup > goes from 0.5s on Micron to a whopping 56s on Cypress. > > Any ideas why would I be seeing this kind of performance difference > when the raw erase/read/write performance is (both datasheet and Linux > tests) only 2x slower? Anywhere I can look to improve performance of > the Cypress part? Let's focus as first step on UBI attach by scanning. Here UBI reads the first two (sub)pages of each block. Please figure whether in both setup subpages are used or not. Then measure how long it takes to read a single page. Given this numbers we can start with the detective work. Did you also check what timings both NANDs are using? -- Thanks, //richard ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: ubi/ubifs performance comparison on two NAND devices 2019-02-27 22:12 ` Richard Weinberger @ 2019-02-27 22:43 ` Tim Harvey 2019-02-27 22:59 ` Richard Weinberger 0 siblings, 1 reply; 8+ messages in thread From: Tim Harvey @ 2019-02-27 22:43 UTC (permalink / raw) To: Richard Weinberger; +Cc: linux-mtd On Wed, Feb 27, 2019 at 2:13 PM Richard Weinberger <richard.weinberger@gmail.com> wrote: > > On Wed, Feb 27, 2019 at 9:39 PM Tim Harvey <tharvey@gateworks.com> wrote: > > > > Greetings, > > > > I've got two embedded boards (IMX6 using gpmi-nand driver) that are > > identical except for the NAND FLASH and am trying to understand why > > I'm seeing a massive performance difference when it comes to ubi and > > ubifs (in particular ubi scanning and UBIFS resizing). > > > > Micron MT29F16G08ADACAH4: > > - page size: 4320 bytes (4096 + 224 bytes) > > - block size: 64 pages (256K + 14K bytes) > > - device size: 16Gb > > - 30ns per-block read > > - 300us per-block program > > - 2.0ms per-block erase > > > > Cypress S34ML16G202BHI000 > > - page size: 2176 bytes (2048+128) > > - block size: 64 pages (128K + 8K) > > - device size: 16Gb > > - 25ns per-block read > > - 300us per-block program > > - 3.5ms per-block erase > > > > The parts are relatively close in per-block performance but because > > the Micron has twice the block size I think of it being twice as fast > > on a per-byte basis compared to the Cypress and I do see this relative > > performance when testing read/write/erase both in modern U-Boot and > > latest Linux. What doesn't add up is that the Cypress part takes about > > 7x longer to complete the ubi attach (between 'attaching mtd' and > > 'scanning is finished' and about 100x longer to complete the UBIFS > > free space fixup (between 'start fixing up free space' and 'free space > > fixup complete') performed on the first mount of a ubifs filesystem > > that was created with auto-resize enabled. > > > > Both parts are 2GiB and the ubi scanning on attach in the kernel goes > > from ~4s on Micron to ~28s on Cypress and the UBIFS free space fixup > > goes from 0.5s on Micron to a whopping 56s on Cypress. > > > > Any ideas why would I be seeing this kind of performance difference > > when the raw erase/read/write performance is (both datasheet and Linux > > tests) only 2x slower? Anywhere I can look to improve performance of > > the Cypress part? > > Let's focus as first step on UBI attach by scanning. Here UBI reads > the first two > (sub)pages of each block. > > Please figure whether in both setup subpages are used or not. Then > measure how long > it takes to read a single page. > Given this numbers we can start with the detective work. > Hi Richard, I suppose I should have provided: [ 6.438793] nand: device found, Manufacturer ID: 0x2c, Chip ID: 0xd5 [ 6.445240] nand: Micron MT29F16G08ADACAH4 [ 6.449362] nand: 2048 MiB, SLC, erase size: 256 KiB, page size: 4096, OOB size: 224 [ 6.459037] Scanning device for bad blocks [ 8.784007] 3 fixed-partitions partitions found on MTD device gpmi-nand [ 8.790716] Creating 3 MTD partitions on "gpmi-nand": [ 8.795823] 0x000000000000-0x000001000000 : "uboot" [ 8.810841] 0x000001000000-0x000001100000 : "env" [ 8.819356] 0x000001100000-0x000080000000 : "rootfs" [ 9.355834] gpmi-nand 112000.gpmi-nand: driver registered. and [ 6.014529] nand: device found, Manufacturer ID: 0x01, Chip ID: 0xd5 [ 6.020907] nand: AMD/Spansion S34ML16G2 [ 6.024917] nand: 2048 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 128 [ 6.033385] Scanning device for bad blocks [ 10.689345] 3 fixed-partitions partitions found on MTD device gpmi-nand [ 10.696056] Creating 3 MTD partitions on "gpmi-nand": [ 10.701143] 0x000000000000-0x000001000000 : "uboot" [ 10.720392] 0x000001000000-0x000001100000 : "env" [ 10.729253] 0x000001100000-0x000080000000 : "rootfs" [ 11.793715] gpmi-nand 112000.gpmi-nand: driver registered. So its taking 2.3s to scan the Micron and 5.4s to scan the Cypress which is what I would expect (Micron 2x faster as it has similar read time per block but half as many blocks) > Did you also check what timings both NANDs are using? No, where would I see these? Would this be something that can be provided in device-tree or a nand chip-specific driver? Currently I'm just relying on CFI... the 4.20 kernel I'm testing with has the following enabled: CONFIG_MTD_CFI=y CONFIG_MTD_JEDECPROBE=y CONFIG_MTD_GEN_PROBE=y CONFIG_MTD_MAP_BANK_WIDTH_1=y CONFIG_MTD_MAP_BANK_WIDTH_2=y CONFIG_MTD_MAP_BANK_WIDTH_4=y CONFIG_MTD_CFI_I1=y CONFIG_MTD_CFI_I2=y CONFIG_MTD_CFI_INTELEXT=y CONFIG_MTD_CFI_AMDSTD=y CONFIG_MTD_CFI_STAA=y CONFIG_MTD_CFI_UTIL=y CONFIG_MTD_PHYSMAP_OF=y CONFIG_MTD_DATAFLASH=y CONFIG_MTD_M25P80=m CONFIG_MTD_SST25L=y CONFIG_MTD_NAND_CORE=m CONFIG_MTD_NAND_ECC=y CONFIG_MTD_NAND=y CONFIG_MTD_NAND_GPMI_NAND=y CONFIG_MTD_SPI_NAND=m CONFIG_MTD_SPI_NOR=m CONFIG_MTD_MT81xx_NOR=m CONFIG_MTD_SPI_NOR_USE_4K_SECTORS=y CONFIG_SPI_FSL_QUADSPI=m CONFIG_MTD_UBI=y CONFIG_MTD_UBI_WL_THRESHOLD=4096 CONFIG_MTD_UBI_BEB_LIMIT=20 CONFIG_MTD_UBI_FASTMAP=y CONFIG_MTD_UBI_BLOCK=y Thanks, Tim ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: ubi/ubifs performance comparison on two NAND devices 2019-02-27 22:43 ` Tim Harvey @ 2019-02-27 22:59 ` Richard Weinberger 2019-02-28 16:40 ` Tim Harvey 0 siblings, 1 reply; 8+ messages in thread From: Richard Weinberger @ 2019-02-27 22:59 UTC (permalink / raw) To: Tim Harvey, linux-mtd Tim, Am Mittwoch, 27. Februar 2019, 23:43:17 CET schrieb Tim Harvey: > On Wed, Feb 27, 2019 at 2:13 PM Richard Weinberger > <richard.weinberger@gmail.com> wrote: > > > > On Wed, Feb 27, 2019 at 9:39 PM Tim Harvey <tharvey@gateworks.com> wrote: > > > > > > Greetings, > > > > > > I've got two embedded boards (IMX6 using gpmi-nand driver) that are > > > identical except for the NAND FLASH and am trying to understand why > > > I'm seeing a massive performance difference when it comes to ubi and > > > ubifs (in particular ubi scanning and UBIFS resizing). > > > > > > Micron MT29F16G08ADACAH4: > > > - page size: 4320 bytes (4096 + 224 bytes) > > > - block size: 64 pages (256K + 14K bytes) > > > - device size: 16Gb > > > - 30ns per-block read > > > - 300us per-block program > > > - 2.0ms per-block erase > > > > > > Cypress S34ML16G202BHI000 > > > - page size: 2176 bytes (2048+128) > > > - block size: 64 pages (128K + 8K) > > > - device size: 16Gb > > > - 25ns per-block read > > > - 300us per-block program > > > - 3.5ms per-block erase > > > > > > The parts are relatively close in per-block performance but because > > > the Micron has twice the block size I think of it being twice as fast > > > on a per-byte basis compared to the Cypress and I do see this relative > > > performance when testing read/write/erase both in modern U-Boot and > > > latest Linux. What doesn't add up is that the Cypress part takes about > > > 7x longer to complete the ubi attach (between 'attaching mtd' and Here you say attach takes 7 times longer. > > > 'scanning is finished' and about 100x longer to complete the UBIFS > > > free space fixup (between 'start fixing up free space' and 'free space > > > fixup complete') performed on the first mount of a ubifs filesystem > > > that was created with auto-resize enabled. > > > > > > Both parts are 2GiB and the ubi scanning on attach in the kernel goes > > > from ~4s on Micron to ~28s on Cypress and the UBIFS free space fixup > > > goes from 0.5s on Micron to a whopping 56s on Cypress. > > > > > > Any ideas why would I be seeing this kind of performance difference > > > when the raw erase/read/write performance is (both datasheet and Linux > > > tests) only 2x slower? Anywhere I can look to improve performance of > > > the Cypress part? > > > > Let's focus as first step on UBI attach by scanning. Here UBI reads > > the first two > > (sub)pages of each block. > > > > Please figure whether in both setup subpages are used or not. Then > > measure how long > > it takes to read a single page. > > Given this numbers we can start with the detective work. > > > > Hi Richard, > > I suppose I should have provided: > > [ 6.438793] nand: device found, Manufacturer ID: 0x2c, Chip ID: 0xd5 > [ 6.445240] nand: Micron MT29F16G08ADACAH4 > [ 6.449362] nand: 2048 MiB, SLC, erase size: 256 KiB, page size: > 4096, OOB size: 224 > [ 6.459037] Scanning device for bad blocks > [ 8.784007] 3 fixed-partitions partitions found on MTD device gpmi-nand > [ 8.790716] Creating 3 MTD partitions on "gpmi-nand": > [ 8.795823] 0x000000000000-0x000001000000 : "uboot" > [ 8.810841] 0x000001000000-0x000001100000 : "env" > [ 8.819356] 0x000001100000-0x000080000000 : "rootfs" > [ 9.355834] gpmi-nand 112000.gpmi-nand: driver registered. > > and > > [ 6.014529] nand: device found, Manufacturer ID: 0x01, Chip ID: 0xd5 > [ 6.020907] nand: AMD/Spansion S34ML16G2 > [ 6.024917] nand: 2048 MiB, SLC, erase size: 128 KiB, page size: > 2048, OOB size: 128 > [ 6.033385] Scanning device for bad blocks > [ 10.689345] 3 fixed-partitions partitions found on MTD device gpmi-nand > [ 10.696056] Creating 3 MTD partitions on "gpmi-nand": > [ 10.701143] 0x000000000000-0x000001000000 : "uboot" > [ 10.720392] 0x000001000000-0x000001100000 : "env" > [ 10.729253] 0x000001100000-0x000080000000 : "rootfs" > [ 11.793715] gpmi-nand 112000.gpmi-nand: driver registered. > > So its taking 2.3s to scan the Micron and 5.4s to scan the Cypress > which is what I would expect (Micron 2x faster as it has similar read > time per block but half as many blocks) And now all is good. I'm confused. > > Did you also check what timings both NANDs are using? > > No, where would I see these? Would this be something that can be > provided in device-tree or a nand chip-specific driver? Usually the driver computes them based on various sources. See drivers/mtd/nand/raw/nand_timings.c Thanks, //richard ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: ubi/ubifs performance comparison on two NAND devices 2019-02-27 22:59 ` Richard Weinberger @ 2019-02-28 16:40 ` Tim Harvey 2019-02-28 17:21 ` Steve deRosier 0 siblings, 1 reply; 8+ messages in thread From: Tim Harvey @ 2019-02-28 16:40 UTC (permalink / raw) To: Richard Weinberger; +Cc: linux-mtd On Wed, Feb 27, 2019 at 2:59 PM Richard Weinberger <richard@nod.at> wrote: > > Tim, > > Am Mittwoch, 27. Februar 2019, 23:43:17 CET schrieb Tim Harvey: > > On Wed, Feb 27, 2019 at 2:13 PM Richard Weinberger > > <richard.weinberger@gmail.com> wrote: > > > > > > On Wed, Feb 27, 2019 at 9:39 PM Tim Harvey <tharvey@gateworks.com> wrote: > > > > > > > > Greetings, > > > > > > > > I've got two embedded boards (IMX6 using gpmi-nand driver) that are > > > > identical except for the NAND FLASH and am trying to understand why > > > > I'm seeing a massive performance difference when it comes to ubi and > > > > ubifs (in particular ubi scanning and UBIFS resizing). > > > > > > > > Micron MT29F16G08ADACAH4: > > > > - page size: 4320 bytes (4096 + 224 bytes) > > > > - block size: 64 pages (256K + 14K bytes) > > > > - device size: 16Gb > > > > - 30ns per-block read > > > > - 300us per-block program > > > > - 2.0ms per-block erase > > > > > > > > Cypress S34ML16G202BHI000 > > > > - page size: 2176 bytes (2048+128) > > > > - block size: 64 pages (128K + 8K) > > > > - device size: 16Gb > > > > - 25ns per-block read > > > > - 300us per-block program > > > > - 3.5ms per-block erase > > > > > > > > The parts are relatively close in per-block performance but because > > > > the Micron has twice the block size I think of it being twice as fast > > > > on a per-byte basis compared to the Cypress and I do see this relative > > > > performance when testing read/write/erase both in modern U-Boot and > > > > latest Linux. What doesn't add up is that the Cypress part takes about > > > > 7x longer to complete the ubi attach (between 'attaching mtd' and > > Here you say attach takes 7 times longer. > > > > > 'scanning is finished' and about 100x longer to complete the UBIFS > > > > free space fixup (between 'start fixing up free space' and 'free space > > > > fixup complete') performed on the first mount of a ubifs filesystem > > > > that was created with auto-resize enabled. > > > > > > > > Both parts are 2GiB and the ubi scanning on attach in the kernel goes > > > > from ~4s on Micron to ~28s on Cypress and the UBIFS free space fixup > > > > goes from 0.5s on Micron to a whopping 56s on Cypress. > > > > > > > > Any ideas why would I be seeing this kind of performance difference > > > > when the raw erase/read/write performance is (both datasheet and Linux > > > > tests) only 2x slower? Anywhere I can look to improve performance of > > > > the Cypress part? > > > > > > Let's focus as first step on UBI attach by scanning. Here UBI reads > > > the first two > > > (sub)pages of each block. > > > > > > Please figure whether in both setup subpages are used or not. Then > > > measure how long > > > it takes to read a single page. > > > Given this numbers we can start with the detective work. > > > > > > > Hi Richard, > > > > I suppose I should have provided: > > > > [ 6.438793] nand: device found, Manufacturer ID: 0x2c, Chip ID: 0xd5 > > [ 6.445240] nand: Micron MT29F16G08ADACAH4 > > [ 6.449362] nand: 2048 MiB, SLC, erase size: 256 KiB, page size: > > 4096, OOB size: 224 > > [ 6.459037] Scanning device for bad blocks > > [ 8.784007] 3 fixed-partitions partitions found on MTD device gpmi-nand > > [ 8.790716] Creating 3 MTD partitions on "gpmi-nand": > > [ 8.795823] 0x000000000000-0x000001000000 : "uboot" > > [ 8.810841] 0x000001000000-0x000001100000 : "env" > > [ 8.819356] 0x000001100000-0x000080000000 : "rootfs" > > [ 9.355834] gpmi-nand 112000.gpmi-nand: driver registered. > > > > and > > > > [ 6.014529] nand: device found, Manufacturer ID: 0x01, Chip ID: 0xd5 > > [ 6.020907] nand: AMD/Spansion S34ML16G2 > > [ 6.024917] nand: 2048 MiB, SLC, erase size: 128 KiB, page size: > > 2048, OOB size: 128 > > [ 6.033385] Scanning device for bad blocks > > [ 10.689345] 3 fixed-partitions partitions found on MTD device gpmi-nand > > [ 10.696056] Creating 3 MTD partitions on "gpmi-nand": > > [ 10.701143] 0x000000000000-0x000001000000 : "uboot" > > [ 10.720392] 0x000001000000-0x000001100000 : "env" > > [ 10.729253] 0x000001100000-0x000080000000 : "rootfs" > > [ 11.793715] gpmi-nand 112000.gpmi-nand: driver registered. > > > > So its taking 2.3s to scan the Micron and 5.4s to scan the Cypress > > which is what I would expect (Micron 2x faster as it has similar read > > time per block but half as many blocks) > > And now all is good. > I'm confused. > Richard, Whatever 'scan' mtd is doing achieves reasonable results in my opinion as well as erase/read/write (tested with 'flash_erase' and 'dd' the Micron part is about 2x faster MB/s than the Cypress part). What doesn't provide reasonable results is the ubi scan and UBIFS 'free space fixup' (the fixup only occurs on first mount of a ubifs created with the 'autoresize' flag): With the Cypress FLASH: [ 10.141066] ubi0: default fastmap pool size: 256 [ 10.145968] ubi0: default fastmap WL pool size: 128 [ 10.150875] ubi0: attaching mtd2 [ 10.206733] ubi0: scanning started ^^^ ubi scan takes ~12s compared to ~4s on Micron FLASH [ 22.539385] ubi0: scanning is finished [ 22.587780] ubi0 warning: ubi_eba_init: cannot reserve enough PEBs for bad PEB handling, reserved 316, need 320 [ 22.624958] ubi0: attached mtd2 (name "rootfs", size 2031 MiB) [ 22.630826] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes [ 22.637798] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048 [ 22.644663] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096 [ 22.651647] ubi0: good PEBs: 16248, bad PEBs: 0, corrupted PEBs: 0 [ 22.657897] ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128 [ 22.665188] ubi0: max/mean erase counter: 1/0, WL threshold: 4096, image sequence number: 789781498 [ 22.674302] ubi0: available PEBs: 0, total reserved PEBs: 16248, PEBs reserved for bad PEB handling: 316 [ 22.684179] ubi0: background thread "ubi_bgt0d" started, PID 233 ... [ 24.326892] UBIFS (ubi0:0): start fixing up free space [ 51.460774] UBIFS (ubi0:0): free space fixup complete [ 51.473525] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 274 ^^^ UBIFS free space fixup (first mount of fs created with 'autoresize' flag) takes ~27s compared to ~0.5s on Micron So my question is why is the Cypress NAND taking way more than twice as long compared to Micron with 'ubi' scan and 'ubifs resize' when the read/write/erase performance is only 2x slower between the two. > > > Did you also check what timings both NANDs are using? > > > > No, where would I see these? Would this be something that can be > > provided in device-tree or a nand chip-specific driver? > > Usually the driver computes them based on various sources. > See drivers/mtd/nand/raw/nand_timings.c > Ok, thanks for the pointer. Is there a sysfs node that contains all of those? I didn't see anything obvious. I can printk them for comparison but I don't see this as a raw nand issue, I see it as a ubi/ubifs issue. There is something going on at the ubi/ubifs layer that makes the Cypress FLASH very slow for the ubi scan that occurs on attach and the UBIFS resize that occurs on first mount. Thanks, Tim ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: ubi/ubifs performance comparison on two NAND devices 2019-02-28 16:40 ` Tim Harvey @ 2019-02-28 17:21 ` Steve deRosier 2019-03-01 16:41 ` Tim Harvey 0 siblings, 1 reply; 8+ messages in thread From: Steve deRosier @ 2019-02-28 17:21 UTC (permalink / raw) To: Tim Harvey; +Cc: Richard Weinberger, linux-mtd On Thu, Feb 28, 2019 at 8:41 AM Tim Harvey <tharvey@gateworks.com> wrote: > > Ok, thanks for the pointer. Is there a sysfs node that contains all of > those? I didn't see anything obvious. I can printk them for comparison > but I don't see this as a raw nand issue, I see it as a ubi/ubifs > issue. There is something going on at the ubi/ubifs layer that makes > the Cypress FLASH very slow for the ubi scan that occurs on attach and > the UBIFS resize that occurs on first mount. I don't follow your assertion. IMHO, if it were at the ubi/ubifs layer, the time it takes should be symmetrical for both your flashes. Perhaps you're saying "I don't believe it has to do with the hardware layer."? Maybe. Though I suggest looking at all layers and prove things beyond a reasonable doubt, otherwise you're going to likely spend an inordinate amount of time looking in the wrong place. UBIFS sits on top of UBI. Which sits on top of the raw flash driver. Which sits on top of whatever bus or SoC driver that may be necessary (maybe there, maybe not). Which then sits on the actual hardware. Unless you have another method of testing the raw flash driver, and through the exact same pathway the UBI is using, I don't think you've eliminated it. The most likely scenario is it is doing something pathological with your flash. Looking at the timing parameters it chooses is a good start, since IIRC, you've said you're not choosing them, you're letting the driver do so. Let's give an example - maybe with the new flash you've got a write-protect GPIO setup and that wasn't in the old configuration. And let's say it takes too long to toggle due to some really bad setup/hold times set by default because they're not configured. And the NAND driver writer implemented it with gpiolib, and toggles it even on a read, and there's some horrible timing bug in gpiolib... And since UBIFS touches every page during the scan... boom - crazy extra time. This is a totally made up example, but it illustrates the type of odd non-obvious interaction that could happen even if you think everything is fine with the raw nand. Personally, I'd shove a bus analyzer on your NAND and take a look if the bus sequences it does on the "good" vs the "bad" chip case are similar. Likely that will tell you exactly why it takes so long which in turn will lead you to exactly what the problem is. If I had to guess, either there is a configuration error OR the nand driver you're using is choosing bad defaults OR there's a particular pathway in the driver that the UBI is exercising that isn't what a raw access would exercise and there's a funky bug there. Remember though - I'm not saying it isn't a bug in the UBI or UBIFS code, I just don't feel you've eliminated the more likely spot first: the code that actually deals with the chip in question. Go examine and understand the NAND driver and printk those timing parameters. - Steve ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: ubi/ubifs performance comparison on two NAND devices 2019-02-28 17:21 ` Steve deRosier @ 2019-03-01 16:41 ` Tim Harvey 2019-03-01 16:44 ` Richard Weinberger 0 siblings, 1 reply; 8+ messages in thread From: Tim Harvey @ 2019-03-01 16:41 UTC (permalink / raw) To: Steve deRosier; +Cc: Richard Weinberger, linux-mtd On Thu, Feb 28, 2019 at 9:22 AM Steve deRosier <derosier@gmail.com> wrote: > > On Thu, Feb 28, 2019 at 8:41 AM Tim Harvey <tharvey@gateworks.com> wrote: > > > > Ok, thanks for the pointer. Is there a sysfs node that contains all of > > those? I didn't see anything obvious. I can printk them for comparison > > but I don't see this as a raw nand issue, I see it as a ubi/ubifs > > issue. There is something going on at the ubi/ubifs layer that makes > > the Cypress FLASH very slow for the ubi scan that occurs on attach and > > the UBIFS resize that occurs on first mount. > > I don't follow your assertion. IMHO, if it were at the ubi/ubifs > layer, the time it takes should be symmetrical for both your flashes. > Perhaps you're saying "I don't believe it has to do with the hardware > layer."? Maybe. Though I suggest looking at all layers and prove > things beyond a reasonable doubt, otherwise you're going to likely > spend an inordinate amount of time looking in the wrong place. > > UBIFS sits on top of UBI. Which sits on top of the raw flash driver. > Which sits on top of whatever bus or SoC driver that may be necessary > (maybe there, maybe not). Which then sits on the actual hardware. > Unless you have another method of testing the raw flash driver, and > through the exact same pathway the UBI is using, I don't think you've > eliminated it. The most likely scenario is it is doing something > pathological with your flash. Looking at the timing parameters it > chooses is a good start, since IIRC, you've said you're not choosing > them, you're letting the driver do so. > > Let's give an example - maybe with the new flash you've got a > write-protect GPIO setup and that wasn't in the old configuration. And > let's say it takes too long to toggle due to some really bad > setup/hold times set by default because they're not configured. And > the NAND driver writer implemented it with gpiolib, and toggles it > even on a read, and there's some horrible timing bug in gpiolib... And > since UBIFS touches every page during the scan... boom - crazy extra > time. This is a totally made up example, but it illustrates the type > of odd non-obvious interaction that could happen even if you think > everything is fine with the raw nand. > > Personally, I'd shove a bus analyzer on your NAND and take a look if > the bus sequences it does on the "good" vs the "bad" chip case are > similar. Likely that will tell you exactly why it takes so long which > in turn will lead you to exactly what the problem is. > > If I had to guess, either there is a configuration error OR the nand > driver you're using is choosing bad defaults OR there's a particular > pathway in the driver that the UBI is exercising that isn't what a raw > access would exercise and there's a funky bug there. > > Remember though - I'm not saying it isn't a bug in the UBI or UBIFS > code, I just don't feel you've eliminated the more likely spot first: > the code that actually deals with the chip in question. Go examine and > understand the NAND driver and printk those timing parameters. > Steve, I've compared erase/read/write speeds for both flashes and the Cypress flash is 2x slower than the Micron on a 'per-byte-size' basis (which is what I would expect as the datasheets have pretty much the same timings per 'block' but the micron has 2x larger blocks and the chips are the same overall size meaning the cypress would have 2x as many 'block' operations across the same size). So, at a raw erase/read/write level the Cypress is 2x slower than Micron, but ubi-scan is 7x slower (4s to 28s), and ubifs-space-fixup is 100x slower (0.5s to 50s). I guess i've made a mess of the description of the issue. I can dig in and find the basic flash timings the kernel is using 'but' when I test using flash_erase and dd for erase/read/write over say 60M I find the expected 2x slower performance. I just don't understand why I see a much slower performance at the ubi and ubifs layers. Tim ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: ubi/ubifs performance comparison on two NAND devices 2019-03-01 16:41 ` Tim Harvey @ 2019-03-01 16:44 ` Richard Weinberger 0 siblings, 0 replies; 8+ messages in thread From: Richard Weinberger @ 2019-03-01 16:44 UTC (permalink / raw) To: Tim Harvey; +Cc: Steve deRosier, linux-mtd Am Freitag, 1. März 2019, 17:41:06 CET schrieb Tim Harvey: > Steve, > > I've compared erase/read/write speeds for both flashes and the Cypress > flash is 2x slower than the Micron on a 'per-byte-size' basis (which > is what I would expect as the datasheets have pretty much the same > timings per 'block' but the micron has 2x larger blocks and the chips > are the same overall size meaning the cypress would have 2x as many > 'block' operations across the same size). > > So, at a raw erase/read/write level the Cypress is 2x slower than > Micron, but ubi-scan is 7x slower (4s to 28s), and ubifs-space-fixup > is 100x slower (0.5s to 50s). > > I guess i've made a mess of the description of the issue. I can dig in > and find the basic flash timings the kernel is using 'but' when I test > using flash_erase and dd for erase/read/write over say 60M I find the > expected 2x slower performance. I just don't understand why I see a > much slower performance at the ubi and ubifs layers. Can we please start with decent numbers? Is UBI really slower than expected? Or only UBIFS file read/write? I'm still totally confused by your allegations. Thanks, //richard ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-03-01 16:44 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-02-27 20:39 ubi/ubifs performance comparison on two NAND devices Tim Harvey 2019-02-27 22:12 ` Richard Weinberger 2019-02-27 22:43 ` Tim Harvey 2019-02-27 22:59 ` Richard Weinberger 2019-02-28 16:40 ` Tim Harvey 2019-02-28 17:21 ` Steve deRosier 2019-03-01 16:41 ` Tim Harvey 2019-03-01 16:44 ` Richard Weinberger
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).