ubi/ubifs performance comparison on two NAND devices

ubi/ubifs performance comparison on two NAND devices

[Tim Harvey]

Greetings,

I've got two embedded boards (IMX6 using gpmi-nand driver) that are
identical except for the NAND FLASH and am trying to understand why
I'm seeing a massive performance difference when it comes to ubi and
ubifs (in particular ubi scanning and UBIFS resizing).

Micron MT29F16G08ADACAH4:
- page size: 4320 bytes (4096 + 224 bytes)
- block size: 64 pages (256K + 14K bytes)
- device size: 16Gb
- 30ns per-block read
- 300us per-block program
- 2.0ms per-block erase

Cypress S34ML16G202BHI000
- page size: 2176 bytes (2048+128)
- block size: 64 pages (128K + 8K)
- device size: 16Gb
- 25ns per-block read
- 300us per-block program
- 3.5ms per-block erase

The parts are relatively close in per-block performance but because
the Micron has twice the block size I think of it being twice as fast
on a per-byte basis compared to the Cypress and I do see this relative
performance when testing read/write/erase both in modern U-Boot and
latest Linux. What doesn't add up is that the Cypress part takes about
7x longer to complete the ubi attach (between 'attaching mtd' and
'scanning is finished' and about 100x longer to complete the UBIFS
free space fixup (between 'start fixing up free space' and 'free space
fixup complete') performed on the first mount of a ubifs filesystem
that was created with auto-resize enabled.

Both parts are 2GiB and the ubi scanning on attach in the kernel goes
from ~4s on Micron to ~28s on Cypress and the UBIFS free space fixup
goes from 0.5s on Micron to a whopping 56s on Cypress.

Any ideas why would I be seeing this kind of performance difference
when the raw erase/read/write performance is (both datasheet and Linux
tests) only 2x slower? Anywhere I can look to improve performance of
the Cypress part?

Best Regards,

Tim

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: ubi/ubifs performance comparison on two NAND devices

[Richard Weinberger]

On Wed, Feb 27, 2019 at 9:39 PM Tim Harvey <tharvey@gateworks.com> wrote:
>
> Greetings,
>
> I've got two embedded boards (IMX6 using gpmi-nand driver) that are
> identical except for the NAND FLASH and am trying to understand why
> I'm seeing a massive performance difference when it comes to ubi and
> ubifs (in particular ubi scanning and UBIFS resizing).
>
> Micron MT29F16G08ADACAH4:
> - page size: 4320 bytes (4096 + 224 bytes)
> - block size: 64 pages (256K + 14K bytes)
> - device size: 16Gb
> - 30ns per-block read
> - 300us per-block program
> - 2.0ms per-block erase
>
> Cypress S34ML16G202BHI000
> - page size: 2176 bytes (2048+128)
> - block size: 64 pages (128K + 8K)
> - device size: 16Gb
> - 25ns per-block read
> - 300us per-block program
> - 3.5ms per-block erase
>
> The parts are relatively close in per-block performance but because
> the Micron has twice the block size I think of it being twice as fast
> on a per-byte basis compared to the Cypress and I do see this relative
> performance when testing read/write/erase both in modern U-Boot and
> latest Linux. What doesn't add up is that the Cypress part takes about
> 7x longer to complete the ubi attach (between 'attaching mtd' and
> 'scanning is finished' and about 100x longer to complete the UBIFS
> free space fixup (between 'start fixing up free space' and 'free space
> fixup complete') performed on the first mount of a ubifs filesystem
> that was created with auto-resize enabled.
>
> Both parts are 2GiB and the ubi scanning on attach in the kernel goes
> from ~4s on Micron to ~28s on Cypress and the UBIFS free space fixup
> goes from 0.5s on Micron to a whopping 56s on Cypress.
>
> Any ideas why would I be seeing this kind of performance difference
> when the raw erase/read/write performance is (both datasheet and Linux
> tests) only 2x slower? Anywhere I can look to improve performance of
> the Cypress part?

Let's focus as first step on UBI attach by scanning. Here UBI reads
the first two
(sub)pages of each block.

Please figure whether in both setup subpages are used or not. Then
measure how long
it takes to read a single page.
Given this numbers we can start with the detective work.

Did you also check what timings both NANDs are using?

-- 
Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: ubi/ubifs performance comparison on two NAND devices

[Tim Harvey]

On Wed, Feb 27, 2019 at 2:13 PM Richard Weinberger
<richard.weinberger@gmail.com> wrote:
>
> On Wed, Feb 27, 2019 at 9:39 PM Tim Harvey <tharvey@gateworks.com> wrote:
> >
> > Greetings,
> >
> > I've got two embedded boards (IMX6 using gpmi-nand driver) that are
> > identical except for the NAND FLASH and am trying to understand why
> > I'm seeing a massive performance difference when it comes to ubi and
> > ubifs (in particular ubi scanning and UBIFS resizing).
> >
> > Micron MT29F16G08ADACAH4:
> > - page size: 4320 bytes (4096 + 224 bytes)
> > - block size: 64 pages (256K + 14K bytes)
> > - device size: 16Gb
> > - 30ns per-block read
> > - 300us per-block program
> > - 2.0ms per-block erase
> >
> > Cypress S34ML16G202BHI000
> > - page size: 2176 bytes (2048+128)
> > - block size: 64 pages (128K + 8K)
> > - device size: 16Gb
> > - 25ns per-block read
> > - 300us per-block program
> > - 3.5ms per-block erase
> >
> > The parts are relatively close in per-block performance but because
> > the Micron has twice the block size I think of it being twice as fast
> > on a per-byte basis compared to the Cypress and I do see this relative
> > performance when testing read/write/erase both in modern U-Boot and
> > latest Linux. What doesn't add up is that the Cypress part takes about
> > 7x longer to complete the ubi attach (between 'attaching mtd' and
> > 'scanning is finished' and about 100x longer to complete the UBIFS
> > free space fixup (between 'start fixing up free space' and 'free space
> > fixup complete') performed on the first mount of a ubifs filesystem
> > that was created with auto-resize enabled.
> >
> > Both parts are 2GiB and the ubi scanning on attach in the kernel goes
> > from ~4s on Micron to ~28s on Cypress and the UBIFS free space fixup
> > goes from 0.5s on Micron to a whopping 56s on Cypress.
> >
> > Any ideas why would I be seeing this kind of performance difference
> > when the raw erase/read/write performance is (both datasheet and Linux
> > tests) only 2x slower? Anywhere I can look to improve performance of
> > the Cypress part?
>
> Let's focus as first step on UBI attach by scanning. Here UBI reads
> the first two
> (sub)pages of each block.
>
> Please figure whether in both setup subpages are used or not. Then
> measure how long
> it takes to read a single page.
> Given this numbers we can start with the detective work.
>

Hi Richard,

I suppose I should have provided:

[    6.438793] nand: device found, Manufacturer ID: 0x2c, Chip ID: 0xd5
[    6.445240] nand: Micron MT29F16G08ADACAH4
[    6.449362] nand: 2048 MiB, SLC, erase size: 256 KiB, page size:
4096, OOB size: 224
[    6.459037] Scanning device for bad blocks
[    8.784007] 3 fixed-partitions partitions found on MTD device gpmi-nand
[    8.790716] Creating 3 MTD partitions on "gpmi-nand":
[    8.795823] 0x000000000000-0x000001000000 : "uboot"
[    8.810841] 0x000001000000-0x000001100000 : "env"
[    8.819356] 0x000001100000-0x000080000000 : "rootfs"
[    9.355834] gpmi-nand 112000.gpmi-nand: driver registered.

and

[    6.014529] nand: device found, Manufacturer ID: 0x01, Chip ID: 0xd5
[    6.020907] nand: AMD/Spansion S34ML16G2
[    6.024917] nand: 2048 MiB, SLC, erase size: 128 KiB, page size:
2048, OOB size: 128
[    6.033385] Scanning device for bad blocks
[   10.689345] 3 fixed-partitions partitions found on MTD device gpmi-nand
[   10.696056] Creating 3 MTD partitions on "gpmi-nand":
[   10.701143] 0x000000000000-0x000001000000 : "uboot"
[   10.720392] 0x000001000000-0x000001100000 : "env"
[   10.729253] 0x000001100000-0x000080000000 : "rootfs"
[   11.793715] gpmi-nand 112000.gpmi-nand: driver registered.

So its taking 2.3s to scan the Micron and 5.4s to scan the Cypress
which is what I would expect (Micron 2x faster as it has similar read
time per block but half as many blocks)

> Did you also check what timings both NANDs are using?

No, where would I see these? Would this be something that can be
provided in device-tree or a nand chip-specific driver?

Currently I'm just relying on CFI... the 4.20 kernel I'm testing with
has the following enabled:
CONFIG_MTD_CFI=y
CONFIG_MTD_JEDECPROBE=y
CONFIG_MTD_GEN_PROBE=y
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
CONFIG_MTD_CFI_INTELEXT=y
CONFIG_MTD_CFI_AMDSTD=y
CONFIG_MTD_CFI_STAA=y
CONFIG_MTD_CFI_UTIL=y
CONFIG_MTD_PHYSMAP_OF=y
CONFIG_MTD_DATAFLASH=y
CONFIG_MTD_M25P80=m
CONFIG_MTD_SST25L=y
CONFIG_MTD_NAND_CORE=m
CONFIG_MTD_NAND_ECC=y
CONFIG_MTD_NAND=y
CONFIG_MTD_NAND_GPMI_NAND=y
CONFIG_MTD_SPI_NAND=m
CONFIG_MTD_SPI_NOR=m
CONFIG_MTD_MT81xx_NOR=m
CONFIG_MTD_SPI_NOR_USE_4K_SECTORS=y
CONFIG_SPI_FSL_QUADSPI=m
CONFIG_MTD_UBI=y
CONFIG_MTD_UBI_WL_THRESHOLD=4096
CONFIG_MTD_UBI_BEB_LIMIT=20
CONFIG_MTD_UBI_FASTMAP=y
CONFIG_MTD_UBI_BLOCK=y

Thanks,

Tim

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: ubi/ubifs performance comparison on two NAND devices

[Richard Weinberger]

Tim,

Am Mittwoch, 27. Februar 2019, 23:43:17 CET schrieb Tim Harvey:
> On Wed, Feb 27, 2019 at 2:13 PM Richard Weinberger
> <richard.weinberger@gmail.com> wrote:
> >
> > On Wed, Feb 27, 2019 at 9:39 PM Tim Harvey <tharvey@gateworks.com> wrote:
> > >
> > > Greetings,
> > >
> > > I've got two embedded boards (IMX6 using gpmi-nand driver) that are
> > > identical except for the NAND FLASH and am trying to understand why
> > > I'm seeing a massive performance difference when it comes to ubi and
> > > ubifs (in particular ubi scanning and UBIFS resizing).
> > >
> > > Micron MT29F16G08ADACAH4:
> > > - page size: 4320 bytes (4096 + 224 bytes)
> > > - block size: 64 pages (256K + 14K bytes)
> > > - device size: 16Gb
> > > - 30ns per-block read
> > > - 300us per-block program
> > > - 2.0ms per-block erase
> > >
> > > Cypress S34ML16G202BHI000
> > > - page size: 2176 bytes (2048+128)
> > > - block size: 64 pages (128K + 8K)
> > > - device size: 16Gb
> > > - 25ns per-block read
> > > - 300us per-block program
> > > - 3.5ms per-block erase
> > >
> > > The parts are relatively close in per-block performance but because
> > > the Micron has twice the block size I think of it being twice as fast
> > > on a per-byte basis compared to the Cypress and I do see this relative
> > > performance when testing read/write/erase both in modern U-Boot and
> > > latest Linux. What doesn't add up is that the Cypress part takes about
> > > 7x longer to complete the ubi attach (between 'attaching mtd' and

Here you say attach takes 7 times longer.

> > > 'scanning is finished' and about 100x longer to complete the UBIFS
> > > free space fixup (between 'start fixing up free space' and 'free space
> > > fixup complete') performed on the first mount of a ubifs filesystem
> > > that was created with auto-resize enabled.
> > >
> > > Both parts are 2GiB and the ubi scanning on attach in the kernel goes
> > > from ~4s on Micron to ~28s on Cypress and the UBIFS free space fixup
> > > goes from 0.5s on Micron to a whopping 56s on Cypress.
> > >
> > > Any ideas why would I be seeing this kind of performance difference
> > > when the raw erase/read/write performance is (both datasheet and Linux
> > > tests) only 2x slower? Anywhere I can look to improve performance of
> > > the Cypress part?
> >
> > Let's focus as first step on UBI attach by scanning. Here UBI reads
> > the first two
> > (sub)pages of each block.
> >
> > Please figure whether in both setup subpages are used or not. Then
> > measure how long
> > it takes to read a single page.
> > Given this numbers we can start with the detective work.
> >
> 
> Hi Richard,
> 
> I suppose I should have provided:
> 
> [    6.438793] nand: device found, Manufacturer ID: 0x2c, Chip ID: 0xd5
> [    6.445240] nand: Micron MT29F16G08ADACAH4
> [    6.449362] nand: 2048 MiB, SLC, erase size: 256 KiB, page size:
> 4096, OOB size: 224
> [    6.459037] Scanning device for bad blocks
> [    8.784007] 3 fixed-partitions partitions found on MTD device gpmi-nand
> [    8.790716] Creating 3 MTD partitions on "gpmi-nand":
> [    8.795823] 0x000000000000-0x000001000000 : "uboot"
> [    8.810841] 0x000001000000-0x000001100000 : "env"
> [    8.819356] 0x000001100000-0x000080000000 : "rootfs"
> [    9.355834] gpmi-nand 112000.gpmi-nand: driver registered.
> 
> and
> 
> [    6.014529] nand: device found, Manufacturer ID: 0x01, Chip ID: 0xd5
> [    6.020907] nand: AMD/Spansion S34ML16G2
> [    6.024917] nand: 2048 MiB, SLC, erase size: 128 KiB, page size:
> 2048, OOB size: 128
> [    6.033385] Scanning device for bad blocks
> [   10.689345] 3 fixed-partitions partitions found on MTD device gpmi-nand
> [   10.696056] Creating 3 MTD partitions on "gpmi-nand":
> [   10.701143] 0x000000000000-0x000001000000 : "uboot"
> [   10.720392] 0x000001000000-0x000001100000 : "env"
> [   10.729253] 0x000001100000-0x000080000000 : "rootfs"
> [   11.793715] gpmi-nand 112000.gpmi-nand: driver registered.
> 
> So its taking 2.3s to scan the Micron and 5.4s to scan the Cypress
> which is what I would expect (Micron 2x faster as it has similar read
> time per block but half as many blocks)

And now all is good.
I'm confused.

> > Did you also check what timings both NANDs are using?
> 
> No, where would I see these? Would this be something that can be
> provided in device-tree or a nand chip-specific driver?

Usually the driver computes them based on various sources.
See drivers/mtd/nand/raw/nand_timings.c

Thanks,
//richard




______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: ubi/ubifs performance comparison on two NAND devices

[Tim Harvey]

On Wed, Feb 27, 2019 at 2:59 PM Richard Weinberger <richard@nod.at> wrote:
>
> Tim,
>
> Am Mittwoch, 27. Februar 2019, 23:43:17 CET schrieb Tim Harvey:
> > On Wed, Feb 27, 2019 at 2:13 PM Richard Weinberger
> > <richard.weinberger@gmail.com> wrote:
> > >
> > > On Wed, Feb 27, 2019 at 9:39 PM Tim Harvey <tharvey@gateworks.com> wrote:
> > > >
> > > > Greetings,
> > > >
> > > > I've got two embedded boards (IMX6 using gpmi-nand driver) that are
> > > > identical except for the NAND FLASH and am trying to understand why
> > > > I'm seeing a massive performance difference when it comes to ubi and
> > > > ubifs (in particular ubi scanning and UBIFS resizing).
> > > >
> > > > Micron MT29F16G08ADACAH4:
> > > > - page size: 4320 bytes (4096 + 224 bytes)
> > > > - block size: 64 pages (256K + 14K bytes)
> > > > - device size: 16Gb
> > > > - 30ns per-block read
> > > > - 300us per-block program
> > > > - 2.0ms per-block erase
> > > >
> > > > Cypress S34ML16G202BHI000
> > > > - page size: 2176 bytes (2048+128)
> > > > - block size: 64 pages (128K + 8K)
> > > > - device size: 16Gb
> > > > - 25ns per-block read
> > > > - 300us per-block program
> > > > - 3.5ms per-block erase
> > > >
> > > > The parts are relatively close in per-block performance but because
> > > > the Micron has twice the block size I think of it being twice as fast
> > > > on a per-byte basis compared to the Cypress and I do see this relative
> > > > performance when testing read/write/erase both in modern U-Boot and
> > > > latest Linux. What doesn't add up is that the Cypress part takes about
> > > > 7x longer to complete the ubi attach (between 'attaching mtd' and
>
> Here you say attach takes 7 times longer.
>
> > > > 'scanning is finished' and about 100x longer to complete the UBIFS
> > > > free space fixup (between 'start fixing up free space' and 'free space
> > > > fixup complete') performed on the first mount of a ubifs filesystem
> > > > that was created with auto-resize enabled.
> > > >
> > > > Both parts are 2GiB and the ubi scanning on attach in the kernel goes
> > > > from ~4s on Micron to ~28s on Cypress and the UBIFS free space fixup
> > > > goes from 0.5s on Micron to a whopping 56s on Cypress.
> > > >
> > > > Any ideas why would I be seeing this kind of performance difference
> > > > when the raw erase/read/write performance is (both datasheet and Linux
> > > > tests) only 2x slower? Anywhere I can look to improve performance of
> > > > the Cypress part?
> > >
> > > Let's focus as first step on UBI attach by scanning. Here UBI reads
> > > the first two
> > > (sub)pages of each block.
> > >
> > > Please figure whether in both setup subpages are used or not. Then
> > > measure how long
> > > it takes to read a single page.
> > > Given this numbers we can start with the detective work.
> > >
> >
> > Hi Richard,
> >
> > I suppose I should have provided:
> >
> > [    6.438793] nand: device found, Manufacturer ID: 0x2c, Chip ID: 0xd5
> > [    6.445240] nand: Micron MT29F16G08ADACAH4
> > [    6.449362] nand: 2048 MiB, SLC, erase size: 256 KiB, page size:
> > 4096, OOB size: 224
> > [    6.459037] Scanning device for bad blocks
> > [    8.784007] 3 fixed-partitions partitions found on MTD device gpmi-nand
> > [    8.790716] Creating 3 MTD partitions on "gpmi-nand":
> > [    8.795823] 0x000000000000-0x000001000000 : "uboot"
> > [    8.810841] 0x000001000000-0x000001100000 : "env"
> > [    8.819356] 0x000001100000-0x000080000000 : "rootfs"
> > [    9.355834] gpmi-nand 112000.gpmi-nand: driver registered.
> >
> > and
> >
> > [    6.014529] nand: device found, Manufacturer ID: 0x01, Chip ID: 0xd5
> > [    6.020907] nand: AMD/Spansion S34ML16G2
> > [    6.024917] nand: 2048 MiB, SLC, erase size: 128 KiB, page size:
> > 2048, OOB size: 128
> > [    6.033385] Scanning device for bad blocks
> > [   10.689345] 3 fixed-partitions partitions found on MTD device gpmi-nand
> > [   10.696056] Creating 3 MTD partitions on "gpmi-nand":
> > [   10.701143] 0x000000000000-0x000001000000 : "uboot"
> > [   10.720392] 0x000001000000-0x000001100000 : "env"
> > [   10.729253] 0x000001100000-0x000080000000 : "rootfs"
> > [   11.793715] gpmi-nand 112000.gpmi-nand: driver registered.
> >
> > So its taking 2.3s to scan the Micron and 5.4s to scan the Cypress
> > which is what I would expect (Micron 2x faster as it has similar read
> > time per block but half as many blocks)
>
> And now all is good.
> I'm confused.
>

Richard,

Whatever 'scan' mtd is doing achieves reasonable results in my opinion
as well as erase/read/write (tested with 'flash_erase' and 'dd' the
Micron part is about 2x faster MB/s than the Cypress part).

What doesn't provide reasonable results is the ubi scan and UBIFS
'free space fixup' (the fixup only occurs on first mount of a ubifs
created with the 'autoresize' flag):

With the Cypress FLASH:
[   10.141066] ubi0: default fastmap pool size: 256
[   10.145968] ubi0: default fastmap WL pool size: 128
[   10.150875] ubi0: attaching mtd2
[   10.206733] ubi0: scanning started
^^^ ubi scan takes ~12s compared to ~4s on Micron FLASH
[   22.539385] ubi0: scanning is finished
[   22.587780] ubi0 warning: ubi_eba_init: cannot reserve enough PEBs
for bad PEB handling, reserved 316, need 320
[   22.624958] ubi0: attached mtd2 (name "rootfs", size 2031 MiB)
[   22.630826] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
[   22.637798] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[   22.644663] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
[   22.651647] ubi0: good PEBs: 16248, bad PEBs: 0, corrupted PEBs: 0
[   22.657897] ubi0: user volume: 1, internal volumes: 1, max. volumes
count: 128
[   22.665188] ubi0: max/mean erase counter: 1/0, WL threshold: 4096,
image sequence number: 789781498
[   22.674302] ubi0: available PEBs: 0, total reserved PEBs: 16248,
PEBs reserved for bad PEB handling: 316
[   22.684179] ubi0: background thread "ubi_bgt0d" started, PID 233
...
[   24.326892] UBIFS (ubi0:0): start fixing up free space
[   51.460774] UBIFS (ubi0:0): free space fixup complete
[   51.473525] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 274
^^^ UBIFS free space fixup (first mount of fs created with
'autoresize' flag) takes ~27s compared to ~0.5s on Micron

So my question is why is the Cypress NAND taking way more than twice
as long compared to Micron with 'ubi' scan and 'ubifs resize' when the
read/write/erase performance is only 2x slower between the two.

> > > Did you also check what timings both NANDs are using?
> >
> > No, where would I see these? Would this be something that can be
> > provided in device-tree or a nand chip-specific driver?
>
> Usually the driver computes them based on various sources.
> See drivers/mtd/nand/raw/nand_timings.c
>

Ok, thanks for the pointer. Is there a sysfs node that contains all of
those? I didn't see anything obvious. I can printk them for comparison
but I don't see this as a raw nand issue, I see it as a ubi/ubifs
issue. There is something going on at the ubi/ubifs layer that makes
the Cypress FLASH very slow for the ubi scan that occurs on attach and
the UBIFS resize that occurs on first mount.

Thanks,

Tim

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: ubi/ubifs performance comparison on two NAND devices

[Steve deRosier]

On Thu, Feb 28, 2019 at 8:41 AM Tim Harvey <tharvey@gateworks.com> wrote:
>
> Ok, thanks for the pointer. Is there a sysfs node that contains all of
> those? I didn't see anything obvious. I can printk them for comparison
> but I don't see this as a raw nand issue, I see it as a ubi/ubifs
> issue. There is something going on at the ubi/ubifs layer that makes
> the Cypress FLASH very slow for the ubi scan that occurs on attach and
> the UBIFS resize that occurs on first mount.

I don't follow your assertion. IMHO, if it were at the ubi/ubifs
layer, the time it takes should be symmetrical for both your flashes.
Perhaps you're saying "I don't believe it has to do with the hardware
layer."? Maybe. Though I suggest looking at all layers and prove
things beyond a reasonable doubt, otherwise you're going to likely
spend an inordinate amount of time looking in the wrong place.

UBIFS sits on top of UBI. Which sits on top of the raw flash driver.
Which sits on top of whatever bus or SoC driver that may be necessary
(maybe there, maybe not). Which then sits on the actual hardware.
Unless you have another method of testing the raw flash driver, and
through the exact same pathway the UBI is using, I don't think you've
eliminated it. The most likely scenario is it is doing something
pathological with your flash. Looking at the timing parameters it
chooses is a good start, since IIRC, you've said you're not choosing
them, you're letting the driver do so.

Let's give an example - maybe with the new flash you've got a
write-protect GPIO setup and that wasn't in the old configuration. And
let's say it takes too long to toggle due to some really bad
setup/hold times set by default because they're not configured. And
the NAND driver writer implemented it with gpiolib, and toggles it
even on a read, and there's some horrible timing bug in gpiolib... And
since UBIFS touches every page during the scan...  boom - crazy extra
time. This is a totally made up example, but it illustrates the type
of odd non-obvious interaction that could happen even if you think
everything is fine with the raw nand.

Personally, I'd shove a bus analyzer on your NAND and take a look if
the bus sequences it does on the "good" vs the "bad" chip case are
similar. Likely that will tell you exactly why it takes so long which
in turn will lead you to exactly what the problem is.

If I had to guess, either there is a configuration error OR the nand
driver you're using is choosing bad defaults OR there's a particular
pathway in the driver that the UBI is exercising that isn't what a raw
access would exercise and there's a funky bug there.

Remember though - I'm not saying it isn't a bug in the UBI or UBIFS
code, I just don't feel you've eliminated the more likely spot first:
the code that actually deals with the chip in question. Go examine and
understand the NAND driver and printk those timing parameters.

- Steve

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: ubi/ubifs performance comparison on two NAND devices

[Tim Harvey]

On Thu, Feb 28, 2019 at 9:22 AM Steve deRosier <derosier@gmail.com> wrote:
>
> On Thu, Feb 28, 2019 at 8:41 AM Tim Harvey <tharvey@gateworks.com> wrote:
> >
> > Ok, thanks for the pointer. Is there a sysfs node that contains all of
> > those? I didn't see anything obvious. I can printk them for comparison
> > but I don't see this as a raw nand issue, I see it as a ubi/ubifs
> > issue. There is something going on at the ubi/ubifs layer that makes
> > the Cypress FLASH very slow for the ubi scan that occurs on attach and
> > the UBIFS resize that occurs on first mount.
>
> I don't follow your assertion. IMHO, if it were at the ubi/ubifs
> layer, the time it takes should be symmetrical for both your flashes.
> Perhaps you're saying "I don't believe it has to do with the hardware
> layer."? Maybe. Though I suggest looking at all layers and prove
> things beyond a reasonable doubt, otherwise you're going to likely
> spend an inordinate amount of time looking in the wrong place.
>
> UBIFS sits on top of UBI. Which sits on top of the raw flash driver.
> Which sits on top of whatever bus or SoC driver that may be necessary
> (maybe there, maybe not). Which then sits on the actual hardware.
> Unless you have another method of testing the raw flash driver, and
> through the exact same pathway the UBI is using, I don't think you've
> eliminated it. The most likely scenario is it is doing something
> pathological with your flash. Looking at the timing parameters it
> chooses is a good start, since IIRC, you've said you're not choosing
> them, you're letting the driver do so.
>
> Let's give an example - maybe with the new flash you've got a
> write-protect GPIO setup and that wasn't in the old configuration. And
> let's say it takes too long to toggle due to some really bad
> setup/hold times set by default because they're not configured. And
> the NAND driver writer implemented it with gpiolib, and toggles it
> even on a read, and there's some horrible timing bug in gpiolib... And
> since UBIFS touches every page during the scan...  boom - crazy extra
> time. This is a totally made up example, but it illustrates the type
> of odd non-obvious interaction that could happen even if you think
> everything is fine with the raw nand.
>
> Personally, I'd shove a bus analyzer on your NAND and take a look if
> the bus sequences it does on the "good" vs the "bad" chip case are
> similar. Likely that will tell you exactly why it takes so long which
> in turn will lead you to exactly what the problem is.
>
> If I had to guess, either there is a configuration error OR the nand
> driver you're using is choosing bad defaults OR there's a particular
> pathway in the driver that the UBI is exercising that isn't what a raw
> access would exercise and there's a funky bug there.
>
> Remember though - I'm not saying it isn't a bug in the UBI or UBIFS
> code, I just don't feel you've eliminated the more likely spot first:
> the code that actually deals with the chip in question. Go examine and
> understand the NAND driver and printk those timing parameters.
>

Steve,

I've compared erase/read/write speeds for both flashes and the Cypress
flash is 2x slower than the Micron on a 'per-byte-size' basis (which
is what I would expect as the datasheets have pretty much the same
timings per 'block' but the micron has 2x larger blocks and the chips
are the same overall size meaning the cypress would have 2x as many
'block' operations across the same size).

So, at a raw erase/read/write level the Cypress is 2x slower than
Micron, but ubi-scan is 7x slower (4s to 28s), and ubifs-space-fixup
is 100x slower (0.5s to 50s).

I guess i've made a mess of the description of the issue. I can dig in
and find the basic flash timings the kernel is using 'but' when I test
using flash_erase and dd for erase/read/write over say 60M I find the
expected 2x slower performance. I just don't understand why I see a
much slower performance at the ubi and ubifs layers.

Tim

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: ubi/ubifs performance comparison on two NAND devices

[Richard Weinberger]

Am Freitag, 1. März 2019, 17:41:06 CET schrieb Tim Harvey:
> Steve,
> 
> I've compared erase/read/write speeds for both flashes and the Cypress
> flash is 2x slower than the Micron on a 'per-byte-size' basis (which
> is what I would expect as the datasheets have pretty much the same
> timings per 'block' but the micron has 2x larger blocks and the chips
> are the same overall size meaning the cypress would have 2x as many
> 'block' operations across the same size).
> 
> So, at a raw erase/read/write level the Cypress is 2x slower than
> Micron, but ubi-scan is 7x slower (4s to 28s), and ubifs-space-fixup
> is 100x slower (0.5s to 50s).
> 
> I guess i've made a mess of the description of the issue. I can dig in
> and find the basic flash timings the kernel is using 'but' when I test
> using flash_erase and dd for erase/read/write over say 60M I find the
> expected 2x slower performance. I just don't understand why I see a
> much slower performance at the ubi and ubifs layers.

Can we please start with decent numbers?
Is UBI really slower than expected? Or only UBIFS file read/write?

I'm still totally confused by your allegations.

Thanks,
//richard



______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/