[PATCH] mtd: parsers: Support '[]' for id in mtdparts

[PATCH] mtd: parsers: Support '[]' for id in mtdparts

[Ronald G. Minnich]

The MTD subsystem can support command-line defined partitions
for one or more MTD devices.

The format is:
 * mtdparts=<mtddef>[;<mtddef]
 * <mtddef>  := <mtd-id>:<partdef>[,<partdef>]

The ':' separates the id from the partdef.

On PCI MTD devices, the name can be the PCI slot name,
e.g. 0000:00:1f.5. There are two ':' in the name alone.

Change the definition of <mtd-id> so it can be bracketed
with '[]' and hence contain any number of ':'.
An opening '[' must be matched with a closing ']'.
The ':' continues to separate the mtd-id from the <partdef>.

Signed-off-by: Ronald G. Minnich <rminnich@google.com>
Change-Id: I17a757e65f532b11606c7bb104f08837bcd444b9
---
 drivers/mtd/parsers/cmdlinepart.c | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)

diff --git a/drivers/mtd/parsers/cmdlinepart.c b/drivers/mtd/parsers/cmdlinepart.c
index c86f2db8c882..ef9dc0bd7724 100644
--- a/drivers/mtd/parsers/cmdlinepart.c
+++ b/drivers/mtd/parsers/cmdlinepart.c
@@ -10,7 +10,8 @@
  * mtdparts=<mtddef>[;<mtddef]
  * <mtddef>  := <mtd-id>:<partdef>[,<partdef>]
  * <partdef> := <size>[@<offset>][<name>][ro][lk]
- * <mtd-id>  := unique name used in mapping driver/device (mtd->name)
+ * <mtd-id>  := unique name used in mapping driver/device (mtd->name) |
+ *              '[' unique name as above, not including a "]" ']'
  * <size>    := standard linux memsize OR "-" to denote all remaining space
  *              size is automatically truncated at end of device
  *              if specified or truncated size is 0 the part is skipped
@@ -221,14 +222,38 @@ static int mtdpart_setup_real(char *s)
 		char *p, *mtd_id;
 
 		mtd_id = s;
+		mtd_id_len = 0;
+		p = s;
 
-		/* fetch <mtd-id> */
+		/*
+		 * fetch <mtd-id>
+		 * If the first char is '[',
+		 * the form is [mtd-id]:
+		 * otherwise it is mtd-id:
+		 */
+		if (*s == '[') {
+			mtd_id++;
+			p = strchr(s, ']');
+			if (!p) {
+				pr_err("mtd (%s) has '[' but no ']'", s);
+				return -EINVAL;
+			}
+			mtd_id_len = p - mtd_id;
+		}
+
+		/* There is always a : following mtd-id. */
 		p = strchr(s, ':');
 		if (!p) {
 			pr_err("no mtd-id\n");
 			return -EINVAL;
 		}
-		mtd_id_len = p - mtd_id;
+
+		/*
+		 * If the mtd-id was bracketed, mtd_id_len will be valid.
+		 * If it is still 0, we must set it here.
+		 */
+		if (mtd_id_len == 0)
+			mtd_id_len = p - mtd_id;
 
 		dbg(("parsing <%s>\n", p+1));
 
-- 
2.26.0.292.g33ef6b2f38-goog

[PATCH] mtd: parsers: Support '[]' for id in mtdparts

[Ronald G. Minnich]

The MTD subsystem can support command-line defined partitions
for one or more MTD devices.

The format is:
 * mtdparts=<mtddef>[;<mtddef]
 * <mtddef>  := <mtd-id>:<partdef>[,<partdef>]

The ':' separates the id from the partdef.

On PCI MTD devices, the name can be the PCI slot name,
e.g. 0000:00:1f.5. There are two ':' in the name alone.

Change the definition of <mtd-id> so it can be bracketed
with '[]' and hence contain any number of ':'.
An opening '[' must be matched with a closing ']'.
The ':' continues to separate the mtd-id from the <partdef>.

Signed-off-by: Ronald G. Minnich <rminnich@google.com>
Change-Id: I17a757e65f532b11606c7bb104f08837bcd444b9
---
 drivers/mtd/parsers/cmdlinepart.c | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)

diff --git a/drivers/mtd/parsers/cmdlinepart.c b/drivers/mtd/parsers/cmdlinepart.c
index c86f2db8c882..ef9dc0bd7724 100644
--- a/drivers/mtd/parsers/cmdlinepart.c
+++ b/drivers/mtd/parsers/cmdlinepart.c
@@ -10,7 +10,8 @@
  * mtdparts=<mtddef>[;<mtddef]
  * <mtddef>  := <mtd-id>:<partdef>[,<partdef>]
  * <partdef> := <size>[@<offset>][<name>][ro][lk]
- * <mtd-id>  := unique name used in mapping driver/device (mtd->name)
+ * <mtd-id>  := unique name used in mapping driver/device (mtd->name) |
+ *              '[' unique name as above, not including a "]" ']'
  * <size>    := standard linux memsize OR "-" to denote all remaining space
  *              size is automatically truncated at end of device
  *              if specified or truncated size is 0 the part is skipped
@@ -221,14 +222,38 @@ static int mtdpart_setup_real(char *s)
 		char *p, *mtd_id;
 
 		mtd_id = s;
+		mtd_id_len = 0;
+		p = s;
 
-		/* fetch <mtd-id> */
+		/*
+		 * fetch <mtd-id>
+		 * If the first char is '[',
+		 * the form is [mtd-id]:
+		 * otherwise it is mtd-id:
+		 */
+		if (*s == '[') {
+			mtd_id++;
+			p = strchr(s, ']');
+			if (!p) {
+				pr_err("mtd (%s) has '[' but no ']'", s);
+				return -EINVAL;
+			}
+			mtd_id_len = p - mtd_id;
+		}
+
+		/* There is always a : following mtd-id. */
 		p = strchr(s, ':');
 		if (!p) {
 			pr_err("no mtd-id\n");
 			return -EINVAL;
 		}
-		mtd_id_len = p - mtd_id;
+
+		/*
+		 * If the mtd-id was bracketed, mtd_id_len will be valid.
+		 * If it is still 0, we must set it here.
+		 */
+		if (mtd_id_len == 0)
+			mtd_id_len = p - mtd_id;
 
 		dbg(("parsing <%s>\n", p+1));
 
-- 
2.26.0.292.g33ef6b2f38-goog


______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: [PATCH] mtd: parsers: Support '[]' for id in mtdparts

[Linus Walleij]

On Mon, Apr 6, 2020 at 6:10 PM Ronald G. Minnich <rminnich@gmail.com> wrote:

> The MTD subsystem can support command-line defined partitions
> for one or more MTD devices.
>
> The format is:
>  * mtdparts=<mtddef>[;<mtddef]
>  * <mtddef>  := <mtd-id>:<partdef>[,<partdef>]
>
> The ':' separates the id from the partdef.
>
> On PCI MTD devices, the name can be the PCI slot name,
> e.g. 0000:00:1f.5. There are two ':' in the name alone.
>
> Change the definition of <mtd-id> so it can be bracketed
> with '[]' and hence contain any number of ':'.
> An opening '[' must be matched with a closing ']'.
> The ':' continues to separate the mtd-id from the <partdef>.
>
> Signed-off-by: Ronald G. Minnich <rminnich@google.com>

It's a reasonable approach, I was a bit confused for a while
because [] is both used as literals and as meta-characters in the
syntax description which becomes a bit terse.

But it's no big deal so:
Reviewed- by: Linus Walleij <linus.walleij@linaro.org>

> Change-Id: I17a757e65f532b11606c7bb104f08837bcd444b9

Upstream don't like Gerrit IDs but surely the maintainer can drop
this when applying.

I suppose the use case is using PCI-based MTD devices for testing
something android images on desktops? I'm surprised it didn't
come up earlier.

Yours,
Linus Walleij

Re: [PATCH] mtd: parsers: Support '[]' for id in mtdparts

[Linus Walleij]

On Mon, Apr 6, 2020 at 6:10 PM Ronald G. Minnich <rminnich@gmail.com> wrote:

> The MTD subsystem can support command-line defined partitions
> for one or more MTD devices.
>
> The format is:
>  * mtdparts=<mtddef>[;<mtddef]
>  * <mtddef>  := <mtd-id>:<partdef>[,<partdef>]
>
> The ':' separates the id from the partdef.
>
> On PCI MTD devices, the name can be the PCI slot name,
> e.g. 0000:00:1f.5. There are two ':' in the name alone.
>
> Change the definition of <mtd-id> so it can be bracketed
> with '[]' and hence contain any number of ':'.
> An opening '[' must be matched with a closing ']'.
> The ':' continues to separate the mtd-id from the <partdef>.
>
> Signed-off-by: Ronald G. Minnich <rminnich@google.com>

It's a reasonable approach, I was a bit confused for a while
because [] is both used as literals and as meta-characters in the
syntax description which becomes a bit terse.

But it's no big deal so:
Reviewed- by: Linus Walleij <linus.walleij@linaro.org>

> Change-Id: I17a757e65f532b11606c7bb104f08837bcd444b9

Upstream don't like Gerrit IDs but surely the maintainer can drop
this when applying.

I suppose the use case is using PCI-based MTD devices for testing
something android images on desktops? I'm surprised it didn't
come up earlier.

Yours,
Linus Walleij

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: [PATCH] mtd: parsers: Support '[]' for id in mtdparts

[ron minnich]

On Thu, Apr 16, 2020 at 2:51 AM Linus Walleij <linus.walleij@linaro.org> wrote:

> I suppose the use case is using PCI-based MTD devices for testing
> something android images on desktops? I'm surprised it didn't
> come up earlier.

Thanks. In this case it's for systems that companies are deploying
into their data centers, using linuxboot (linuxboot.org) and Intel
chipsets. On Intel  chipsets, there is a 64 MiB SPI part, but only 16
MiB is directly addressable.

Linux goes in the memory-addressable part of the SPI, and UEFI loads
it into RAM, since to UEFI the kernel is just another UEFI driver --
in fact in most cases we replace the UEFI shell with Linux.

But we need a file system, and with the huge amount of drivers that
come with UEFI there's not much room in the top 16M. (we're working to
fix that glitch, a process we call DXE-ectomy, but it takes time).

We wish to place a file system in the low 48 MiB -- lots of room there.

So what one can do is put a squashfs-formatted file system in that low
part of SPI, and, using this mtdparts capability, point the kernel at
it ("root=/dev/mtd1 mtdparts=[a:b.c]etc.etc"). It's a lifesaver for
those of us using u-root for our userland.

ron

Re: [PATCH] mtd: parsers: Support '[]' for id in mtdparts

[ron minnich]

On Thu, Apr 16, 2020 at 2:51 AM Linus Walleij <linus.walleij@linaro.org> wrote:

> I suppose the use case is using PCI-based MTD devices for testing
> something android images on desktops? I'm surprised it didn't
> come up earlier.

Thanks. In this case it's for systems that companies are deploying
into their data centers, using linuxboot (linuxboot.org) and Intel
chipsets. On Intel  chipsets, there is a 64 MiB SPI part, but only 16
MiB is directly addressable.

Linux goes in the memory-addressable part of the SPI, and UEFI loads
it into RAM, since to UEFI the kernel is just another UEFI driver --
in fact in most cases we replace the UEFI shell with Linux.

But we need a file system, and with the huge amount of drivers that
come with UEFI there's not much room in the top 16M. (we're working to
fix that glitch, a process we call DXE-ectomy, but it takes time).

We wish to place a file system in the low 48 MiB -- lots of room there.

So what one can do is put a squashfs-formatted file system in that low
part of SPI, and, using this mtdparts capability, point the kernel at
it ("root=/dev/mtd1 mtdparts=[a:b.c]etc.etc"). It's a lifesaver for
those of us using u-root for our userland.

ron

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: [PATCH] mtd: parsers: Support '[]' for id in mtdparts

[Linus Walleij]

On Thu, Apr 16, 2020 at 4:55 PM ron minnich <rminnich@gmail.com> wrote:
> On Thu, Apr 16, 2020 at 2:51 AM Linus Walleij <linus.walleij@linaro.org> wrote:
>
> > I suppose the use case is using PCI-based MTD devices for testing
> > something android images on desktops? I'm surprised it didn't
> > come up earlier.
>
> Thanks. In this case it's for systems that companies are deploying
> into their data centers, using linuxboot (linuxboot.org) and Intel
> chipsets. On Intel  chipsets, there is a 64 MiB SPI part, but only 16
> MiB is directly addressable.

Aha, now I get the use case.

> Linux goes in the memory-addressable part of the SPI, and UEFI loads
> it into RAM, since to UEFI the kernel is just another UEFI driver --
> in fact in most cases we replace the UEFI shell with Linux.
>
> But we need a file system, and with the huge amount of drivers that
> come with UEFI there's not much room in the top 16M. (we're working to
> fix that glitch, a process we call DXE-ectomy, but it takes time).
>
> We wish to place a file system in the low 48 MiB -- lots of room there.
>
> So what one can do is put a squashfs-formatted file system in that low
> part of SPI, and, using this mtdparts capability, point the kernel at
> it ("root=/dev/mtd1 mtdparts=[a:b.c]etc.etc"). It's a lifesaver for
> those of us using u-root for our userland.

This makes a lot of sense.

Something I have had ideas about upstreaming is the partition
splitter and automatic rootfs configuration from OpenWrt.
Is this something you would have an interest in for your type
of deployments?

I post some of my (never finished) commit text for your reference:

What the MTD partition splitter code does is to take a
partition, already covering an even number of erase blocks
in the flash, and subdivide it at erase block granularity
into sub-partitions.

This structure is created when the raw images are produced
during compilation of a system: scripts that are aware
of the geometry of the flash (such as erase block size)
will catenate the different parts into a compound
partition that can later be split.

The typical consituents of a split partition are:

 [kernel (z)Image]       "kernel"
 squashfs rootfs         "rootfs"
 JFFS2 writeable area    "rootfs_data"

In the simplified case only the squashfs and JFFS2 are
combined into one split partition, let's say this
partition is named "firmware" (a common convention).

The typical scenario for the above layout is:

- Begin compiling the flash image with the kernel image,
  pad that up to the end of the current erase block
- Catenate the squashfs and pad that up to the end of
  the current erase block
- Catenate a JFFS2 emtpy filesystem" marker at the
  beginning of the next erase block

When the splitter examines this, it will split this
"firmware" partition into a kernel partion, a
squashfs "rootfs" partition, and a partition with just
empty space named "rootfs_data", while still keeping
the overarching "firmware" partition in place.

When the kernel boots, it will mount the squashfs "rootfs"
partition as root filesystem, and then when the system
properly comes up mount the "rootfs_data" partition with
overlayfs so that the root filesystem becomes writeable,
while keeping all the read-only content in the squashfs
and all modifications in the JFFS2 partition.

This way all the available flash memory in the "firmware"
partition is used pretty optimally: "kernel" can grow
to the size it needs (such as a new kernel version taking
up more space) same for the squashfs "rootfs" after it.
Whatever remains after the kernel and the rootfs can be
used for storing data.

When the device kernel and rootfs needs to be upgraded,
it can simply unmount the filesystems and overwrite and
erase the "firmware" partition with the new version,
and the whole system is dynamically repartitioned with
the new images: if they grew over a flash block boundary
then the filesystem will be augmented upwards.

Yours,
Linus Walleij

Re: [PATCH] mtd: parsers: Support '[]' for id in mtdparts

[Linus Walleij]

On Thu, Apr 16, 2020 at 4:55 PM ron minnich <rminnich@gmail.com> wrote:
> On Thu, Apr 16, 2020 at 2:51 AM Linus Walleij <linus.walleij@linaro.org> wrote:
>
> > I suppose the use case is using PCI-based MTD devices for testing
> > something android images on desktops? I'm surprised it didn't
> > come up earlier.
>
> Thanks. In this case it's for systems that companies are deploying
> into their data centers, using linuxboot (linuxboot.org) and Intel
> chipsets. On Intel  chipsets, there is a 64 MiB SPI part, but only 16
> MiB is directly addressable.

Aha, now I get the use case.

> Linux goes in the memory-addressable part of the SPI, and UEFI loads
> it into RAM, since to UEFI the kernel is just another UEFI driver --
> in fact in most cases we replace the UEFI shell with Linux.
>
> But we need a file system, and with the huge amount of drivers that
> come with UEFI there's not much room in the top 16M. (we're working to
> fix that glitch, a process we call DXE-ectomy, but it takes time).
>
> We wish to place a file system in the low 48 MiB -- lots of room there.
>
> So what one can do is put a squashfs-formatted file system in that low
> part of SPI, and, using this mtdparts capability, point the kernel at
> it ("root=/dev/mtd1 mtdparts=[a:b.c]etc.etc"). It's a lifesaver for
> those of us using u-root for our userland.

This makes a lot of sense.

Something I have had ideas about upstreaming is the partition
splitter and automatic rootfs configuration from OpenWrt.
Is this something you would have an interest in for your type
of deployments?

I post some of my (never finished) commit text for your reference:

What the MTD partition splitter code does is to take a
partition, already covering an even number of erase blocks
in the flash, and subdivide it at erase block granularity
into sub-partitions.

This structure is created when the raw images are produced
during compilation of a system: scripts that are aware
of the geometry of the flash (such as erase block size)
will catenate the different parts into a compound
partition that can later be split.

The typical consituents of a split partition are:

 [kernel (z)Image]       "kernel"
 squashfs rootfs         "rootfs"
 JFFS2 writeable area    "rootfs_data"

In the simplified case only the squashfs and JFFS2 are
combined into one split partition, let's say this
partition is named "firmware" (a common convention).

The typical scenario for the above layout is:

- Begin compiling the flash image with the kernel image,
  pad that up to the end of the current erase block
- Catenate the squashfs and pad that up to the end of
  the current erase block
- Catenate a JFFS2 emtpy filesystem" marker at the
  beginning of the next erase block

When the splitter examines this, it will split this
"firmware" partition into a kernel partion, a
squashfs "rootfs" partition, and a partition with just
empty space named "rootfs_data", while still keeping
the overarching "firmware" partition in place.

When the kernel boots, it will mount the squashfs "rootfs"
partition as root filesystem, and then when the system
properly comes up mount the "rootfs_data" partition with
overlayfs so that the root filesystem becomes writeable,
while keeping all the read-only content in the squashfs
and all modifications in the JFFS2 partition.

This way all the available flash memory in the "firmware"
partition is used pretty optimally: "kernel" can grow
to the size it needs (such as a new kernel version taking
up more space) same for the squashfs "rootfs" after it.
Whatever remains after the kernel and the rootfs can be
used for storing data.

When the device kernel and rootfs needs to be upgraded,
it can simply unmount the filesystems and overwrite and
erase the "firmware" partition with the new version,
and the whole system is dynamically repartitioned with
the new images: if they grew over a flash block boundary
then the filesystem will be augmented upwards.

Yours,
Linus Walleij

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/