All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH V2 1/2] mtd: add support for partition parsers
@ 2017-02-21 12:40 Rafał Miłecki
  2017-02-21 12:40 ` [PATCH V2 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module Rafał Miłecki
                   ` (2 more replies)
  0 siblings, 3 replies; 16+ messages in thread
From: Rafał Miłecki @ 2017-02-21 12:40 UTC (permalink / raw)
  To: David Woodhouse, Brian Norris, Boris Brezillon, Marek Vasut,
	Richard Weinberger, Cyrille Pitchen
  Cc: linux-mtd, Rafał Miłecki

From: Rafał Miłecki <rafal@milecki.pl>

Some devices have partitions that are kind of containers with extra
subpartitions / volumes instead of e.g. simple filesystem data. To
support such cases we need to first create normal flash partitions and
then take care of these special ones.

It's very common case for home routers. Depending on the vendor there
are formats like TRX, Seama, uImage, WRGG and more. All of them are used
to embed few partitions into a single one / single firmware file.

Ideally all vendors would use some well documented / standardized format
like UBI (and some probably start doing so), but there are still
countless devices on the market using these poor vendor specific
formats.

This patch extends MTD subsystem by allowing to specify partition format
and trying to use a proper parser when needed. Supporting such poor
formats is highly unlikely to be the top priority so these changes try
to minimize maintenance cost to the minimum. It reuses existing code for
these new parsers and just adds a one property and one new function.

This implementation requires setting partition format in a flash parser.
A proper change of bcm47xxpart will follow and in the future we will
hopefully also find a solution for doing it with ofpart
("fixed-partitions").

Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
---
V2: A totally rebased & refreshed version.
---
 drivers/mtd/mtdpart.c          | 38 ++++++++++++++++++++++++++++++++++++++
 include/linux/mtd/partitions.h |  7 +++++++
 2 files changed, 45 insertions(+)

diff --git a/drivers/mtd/mtdpart.c b/drivers/mtd/mtdpart.c
index ea5e5307f667..9a40c3ea384b 100644
--- a/drivers/mtd/mtdpart.c
+++ b/drivers/mtd/mtdpart.c
@@ -363,6 +363,42 @@ static inline void free_partition(struct mtd_part *p)
 	kfree(p);
 }
 
+/**
+ * mtd_parse_part - parse MTD partition with a matching parser
+ *
+ * Some partitions use a specific format to describe contained subpartitions
+ * (volumes). This function tries to use a proper parser for a given format and
+ * registers found (sub)partitions.
+ */
+static int mtd_parse_part(struct mtd_part *slave, const char *format)
+{
+	struct mtd_partitions parsed;
+	const char *probes[2];
+	int i;
+	int err;
+
+	probes[0] = format; /* Use parser with name matching the format */
+	probes[1] = NULL; /* End of parsers */
+	err = parse_mtd_partitions(&slave->mtd, probes, &parsed, NULL);
+	if (err)
+		return err;
+	else if (!parsed.nr_parts)
+		return -ENOENT;
+
+	for (i = 0; i < parsed.nr_parts; i++) {
+		struct mtd_partition *part;
+
+		part = (struct mtd_partition *)&parsed.parts[i];
+		part->offset += slave->offset;
+	}
+
+	err = add_mtd_partitions(slave->master, parsed.parts, parsed.nr_parts);
+
+	mtd_part_parser_cleanup(&parsed);
+
+	return err;
+}
+
 /*
  * This function unregisters and destroy all slave MTD objects which are
  * attached to the given master MTD object.
@@ -724,6 +760,8 @@ int add_mtd_partitions(struct mtd_info *master,
 
 		add_mtd_device(&slave->mtd);
 		mtd_add_partition_attrs(slave);
+		if (parts[i].format)
+			mtd_parse_part(slave, parts[i].format);
 
 		cur_offset = slave->offset + slave->mtd.size;
 	}
diff --git a/include/linux/mtd/partitions.h b/include/linux/mtd/partitions.h
index 06df1e06b6e0..2787e76c030f 100644
--- a/include/linux/mtd/partitions.h
+++ b/include/linux/mtd/partitions.h
@@ -20,6 +20,12 @@
  *
  * For each partition, these fields are available:
  * name: string that will be used to label the partition's MTD device.
+ * format: some partitions can be containers using specific format to describe
+ *	embedded subpartitions / volumes. E.g. many home routers use "firmware"
+ *	partition that contains at least kernel and rootfs. In such case an
+ *	extra parser is needed that will detect these dynamic partitions and
+ *	report them to the MTD subsystem. This property describes partition
+ *	format and allows MTD core to call a proper parser.
  * size: the partition size; if defined as MTDPART_SIZ_FULL, the partition
  * 	will extend to the end of the master MTD device.
  * offset: absolute starting position within the master MTD device; if
@@ -38,6 +44,7 @@
 
 struct mtd_partition {
 	const char *name;		/* identifier string */
+	const char *format;		/* partition format */
 	uint64_t size;			/* partition size */
 	uint64_t offset;		/* offset within the master MTD space */
 	uint32_t mask_flags;		/* master MTD flags to mask out for this partition */
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [PATCH V2 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module
  2017-02-21 12:40 [PATCH V2 1/2] mtd: add support for partition parsers Rafał Miłecki
@ 2017-02-21 12:40 ` Rafał Miłecki
  2017-02-27  9:36 ` [PATCH V2 1/2] mtd: add support for partition parsers Marek Vasut
  2017-02-27 13:06 ` [PATCH V3 " Rafał Miłecki
  2 siblings, 0 replies; 16+ messages in thread
From: Rafał Miłecki @ 2017-02-21 12:40 UTC (permalink / raw)
  To: David Woodhouse, Brian Norris, Boris Brezillon, Marek Vasut,
	Richard Weinberger, Cyrille Pitchen
  Cc: linux-mtd, Rafał Miłecki

From: Rafał Miłecki <rafal@milecki.pl>

This makes TRX parsing code reusable with other platforms and parsers.

Please note this patch doesn't really change anything in the existing
code, just moves it. There is still some place for improvement (e.g.
working on non-hacky method of checking rootfs format) but it's not
really a subject of this change.

Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
---
V2: A totally rebased & refreshed version.
---
 drivers/mtd/Kconfig              |   4 ++
 drivers/mtd/Makefile             |   1 +
 drivers/mtd/bcm47xxpart.c        |  97 +------------------------------
 drivers/mtd/parsers/Kconfig      |   8 +++
 drivers/mtd/parsers/Makefile     |   1 +
 drivers/mtd/parsers/parser_trx.c | 119 +++++++++++++++++++++++++++++++++++++++
 6 files changed, 136 insertions(+), 94 deletions(-)
 create mode 100644 drivers/mtd/parsers/Kconfig
 create mode 100644 drivers/mtd/parsers/Makefile
 create mode 100644 drivers/mtd/parsers/parser_trx.c

diff --git a/drivers/mtd/Kconfig b/drivers/mtd/Kconfig
index e83a279f1217..5a2d71729b9a 100644
--- a/drivers/mtd/Kconfig
+++ b/drivers/mtd/Kconfig
@@ -155,6 +155,10 @@ config MTD_BCM47XX_PARTS
 	  This provides partitions parser for devices based on BCM47xx
 	  boards.
 
+menu "Partition parsers"
+source "drivers/mtd/parsers/Kconfig"
+endmenu
+
 comment "User Modules And Translation Layers"
 
 #
diff --git a/drivers/mtd/Makefile b/drivers/mtd/Makefile
index 99bb9a1f6e16..151d60df303a 100644
--- a/drivers/mtd/Makefile
+++ b/drivers/mtd/Makefile
@@ -13,6 +13,7 @@ obj-$(CONFIG_MTD_AFS_PARTS)	+= afs.o
 obj-$(CONFIG_MTD_AR7_PARTS)	+= ar7part.o
 obj-$(CONFIG_MTD_BCM63XX_PARTS)	+= bcm63xxpart.o
 obj-$(CONFIG_MTD_BCM47XX_PARTS)	+= bcm47xxpart.o
+obj-y				+= parsers/
 
 # 'Users' - code which presents functionality to userspace.
 obj-$(CONFIG_MTD_BLKDEVS)	+= mtd_blkdevs.o
diff --git a/drivers/mtd/bcm47xxpart.c b/drivers/mtd/bcm47xxpart.c
index d10fa6c8f074..66ce72fd1426 100644
--- a/drivers/mtd/bcm47xxpart.c
+++ b/drivers/mtd/bcm47xxpart.c
@@ -43,7 +43,6 @@
 #define ML_MAGIC2			0x26594131
 #define TRX_MAGIC			0x30524448
 #define SHSQ_MAGIC			0x71736873	/* shsq (weird ZTE H218N endianness) */
-#define UBI_EC_MAGIC			0x23494255	/* UBI# */
 
 struct trx_header {
 	uint32_t magic;
@@ -62,89 +61,6 @@ static void bcm47xxpart_add_part(struct mtd_partition *part, const char *name,
 	part->mask_flags = mask_flags;
 }
 
-static const char *bcm47xxpart_trx_data_part_name(struct mtd_info *master,
-						  size_t offset)
-{
-	uint32_t buf;
-	size_t bytes_read;
-	int err;
-
-	err  = mtd_read(master, offset, sizeof(buf), &bytes_read,
-			(uint8_t *)&buf);
-	if (err && !mtd_is_bitflip(err)) {
-		pr_err("mtd_read error while parsing (offset: 0x%X): %d\n",
-			offset, err);
-		goto out_default;
-	}
-
-	if (buf == UBI_EC_MAGIC)
-		return "ubi";
-
-out_default:
-	return "rootfs";
-}
-
-static int bcm47xxpart_parse_trx(struct mtd_info *master,
-				 struct mtd_partition *trx,
-				 struct mtd_partition *parts,
-				 size_t parts_len)
-{
-	struct trx_header header;
-	size_t bytes_read;
-	int curr_part = 0;
-	int i, err;
-
-	if (parts_len < 3) {
-		pr_warn("No enough space to add TRX partitions!\n");
-		return -ENOMEM;
-	}
-
-	err = mtd_read(master, trx->offset, sizeof(header), &bytes_read,
-		       (uint8_t *)&header);
-	if (err && !mtd_is_bitflip(err)) {
-		pr_err("mtd_read error while reading TRX header: %d\n", err);
-		return err;
-	}
-
-	i = 0;
-
-	/* We have LZMA loader if offset[2] points to sth */
-	if (header.offset[2]) {
-		bcm47xxpart_add_part(&parts[curr_part++], "loader",
-				     trx->offset + header.offset[i], 0);
-		i++;
-	}
-
-	if (header.offset[i]) {
-		bcm47xxpart_add_part(&parts[curr_part++], "linux",
-				     trx->offset + header.offset[i], 0);
-		i++;
-	}
-
-	if (header.offset[i]) {
-		size_t offset = trx->offset + header.offset[i];
-		const char *name = bcm47xxpart_trx_data_part_name(master,
-								  offset);
-
-		bcm47xxpart_add_part(&parts[curr_part++], name, offset, 0);
-		i++;
-	}
-
-	/*
-	 * Assume that every partition ends at the beginning of the one it is
-	 * followed by.
-	 */
-	for (i = 0; i < curr_part; i++) {
-		u64 next_part_offset = (i < curr_part - 1) ?
-					parts[i + 1].offset :
-					trx->offset + trx->size;
-
-		parts[i].size = next_part_offset - parts[i].offset;
-	}
-
-	return curr_part;
-}
-
 /**
  * bcm47xxpart_bootpartition - gets index of TRX partition used by bootloader
  *
@@ -362,17 +278,10 @@ static int bcm47xxpart_parse(struct mtd_info *master,
 	for (i = 0; i < trx_num; i++) {
 		struct mtd_partition *trx = &parts[trx_parts[i]];
 
-		if (i == bcm47xxpart_bootpartition()) {
-			int num_parts;
-
-			num_parts = bcm47xxpart_parse_trx(master, trx,
-							  parts + curr_part,
-							  BCM47XXPART_MAX_PARTS - curr_part);
-			if (num_parts > 0)
-				curr_part += num_parts;
-		} else {
+		if (i == bcm47xxpart_bootpartition())
+			trx->format = "trx";
+		else
 			trx->name = "failsafe";
-		}
 	}
 
 	*pparts = parts;
diff --git a/drivers/mtd/parsers/Kconfig b/drivers/mtd/parsers/Kconfig
new file mode 100644
index 000000000000..ebb697a52698
--- /dev/null
+++ b/drivers/mtd/parsers/Kconfig
@@ -0,0 +1,8 @@
+config MTD_PARSER_TRX
+	tristate "Parser for TRX format partitions"
+	depends on MTD && (BCM47XX || ARCH_BCM_5301X)
+	help
+	  TRX is a firmware format used by Broadcom on their devices. It
+	  may contain up to 3/4 partitions (depending on the version).
+	  This driver will parse TRX header and report at least two partitions:
+	  kernel and rootfs.
diff --git a/drivers/mtd/parsers/Makefile b/drivers/mtd/parsers/Makefile
new file mode 100644
index 000000000000..4d9024e0be3b
--- /dev/null
+++ b/drivers/mtd/parsers/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_MTD_PARSER_TRX)		+= parser_trx.o
diff --git a/drivers/mtd/parsers/parser_trx.c b/drivers/mtd/parsers/parser_trx.c
new file mode 100644
index 000000000000..f35e28148808
--- /dev/null
+++ b/drivers/mtd/parsers/parser_trx.c
@@ -0,0 +1,119 @@
+/*
+ * Parser for TRX format partitions
+ *
+ * Copyright (C) 2012 - 2017 Rafał Miłecki <rafal@milecki.pl>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/mtd/mtd.h>
+#include <linux/mtd/partitions.h>
+
+#define TRX_PARSER_MAX_PARTS		4
+
+/* Magics */
+#define UBI_EC_MAGIC			0x23494255	/* UBI# */
+
+struct trx_header {
+	uint32_t magic;
+	uint32_t length;
+	uint32_t crc32;
+	uint16_t flags;
+	uint16_t version;
+	uint32_t offset[3];
+} __packed;
+
+static const char *parser_trx_data_part_name(struct mtd_info *master,
+					     size_t offset)
+{
+	uint32_t buf;
+	size_t bytes_read;
+	int err;
+
+	err  = mtd_read(master, offset, sizeof(buf), &bytes_read,
+			(uint8_t *)&buf);
+	if (err && !mtd_is_bitflip(err)) {
+		pr_err("mtd_read error while parsing (offset: 0x%X): %d\n",
+			offset, err);
+		goto out_default;
+	}
+
+	if (buf == UBI_EC_MAGIC)
+		return "ubi";
+
+out_default:
+	return "rootfs";
+}
+
+static int parser_trx_parse(struct mtd_info *mtd,
+			    const struct mtd_partition **pparts,
+			    struct mtd_part_parser_data *data)
+{
+	struct mtd_partition *parts;
+	struct mtd_partition *part;
+	struct trx_header trx;
+	size_t bytes_read;
+	uint8_t curr_part = 0, i = 0;
+	int err;
+
+	parts = kzalloc(sizeof(struct mtd_partition) * TRX_PARSER_MAX_PARTS,
+			GFP_KERNEL);
+	if (!parts)
+		return -ENOMEM;
+
+	err = mtd_read(mtd, 0, sizeof(trx), &bytes_read, (uint8_t *)&trx);
+	if (err) {
+		pr_err("MTD reading error: %d\n", err);
+		return err;
+	}
+
+	/* We have LZMA loader if offset[2] points to sth */
+	if (trx.offset[2]) {
+		part = &parts[curr_part++];
+		part->name = "loader";
+		part->offset = trx.offset[i];
+		i++;
+	}
+
+	if (trx.offset[i]) {
+		part = &parts[curr_part++];
+		part->name = "linux";
+		part->offset = trx.offset[i];
+		i++;
+	}
+
+	if (trx.offset[i]) {
+		part = &parts[curr_part++];
+		part->name = parser_trx_data_part_name(mtd, trx.offset[i]);
+		part->offset = trx.offset[i];
+		i++;
+	}
+
+	/*
+	 * Assume that every partition ends at the beginning of the one it is
+	 * followed by.
+	 */
+	for (i = 0; i < curr_part; i++) {
+		u64 next_part_offset = (i < curr_part - 1) ?
+				       parts[i + 1].offset : mtd->size;
+
+		parts[i].size = next_part_offset - parts[i].offset;
+	}
+
+	*pparts = parts;
+	return i;
+};
+
+static struct mtd_part_parser mtd_parser_trx = {
+	.parse_fn = parser_trx_parse,
+	.name = "trx",
+};
+module_mtd_part_parser(mtd_parser_trx);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Parser for TRX format partitions");
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 16+ messages in thread

* Re: [PATCH V2 1/2] mtd: add support for partition parsers
  2017-02-21 12:40 [PATCH V2 1/2] mtd: add support for partition parsers Rafał Miłecki
  2017-02-21 12:40 ` [PATCH V2 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module Rafał Miłecki
@ 2017-02-27  9:36 ` Marek Vasut
  2017-02-27  9:51   ` Rafał Miłecki
  2017-02-27 13:06 ` [PATCH V3 " Rafał Miłecki
  2 siblings, 1 reply; 16+ messages in thread
From: Marek Vasut @ 2017-02-27  9:36 UTC (permalink / raw)
  To: Rafał Miłecki, David Woodhouse, Brian Norris,
	Boris Brezillon, Richard Weinberger, Cyrille Pitchen
  Cc: linux-mtd, Rafał Miłecki

On 02/21/2017 01:40 PM, Rafał Miłecki wrote:
> From: Rafał Miłecki <rafal@milecki.pl>
> 
> Some devices have partitions that are kind of containers with extra
> subpartitions / volumes instead of e.g. simple filesystem data. To
> support such cases we need to first create normal flash partitions and
> then take care of these special ones.
> 
> It's very common case for home routers. Depending on the vendor there
> are formats like TRX, Seama, uImage

uImage is file format, not partition type. It's just a blob of data
(usually kernel) with small header, I don't think we should represent
the content of it as a partition.

> , WRGG and more. All of them are used
> to embed few partitions into a single one / single firmware file.
> 
> Ideally all vendors would use some well documented / standardized format
> like UBI (and some probably start doing so), but there are still
> countless devices on the market using these poor vendor specific
> formats.
> 
> This patch extends MTD subsystem by allowing to specify partition format
> and trying to use a proper parser when needed. Supporting such poor
> formats is highly unlikely to be the top priority so these changes try
> to minimize maintenance cost to the minimum. It reuses existing code for
> these new parsers and just adds a one property and one new function.
> 
> This implementation requires setting partition format in a flash parser.
> A proper change of bcm47xxpart will follow and in the future we will
> hopefully also find a solution for doing it with ofpart
> ("fixed-partitions").
> 
> Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
> ---
> V2: A totally rebased & refreshed version.
> ---
>  drivers/mtd/mtdpart.c          | 38 ++++++++++++++++++++++++++++++++++++++
>  include/linux/mtd/partitions.h |  7 +++++++
>  2 files changed, 45 insertions(+)
> 
> diff --git a/drivers/mtd/mtdpart.c b/drivers/mtd/mtdpart.c
> index ea5e5307f667..9a40c3ea384b 100644
> --- a/drivers/mtd/mtdpart.c
> +++ b/drivers/mtd/mtdpart.c
> @@ -363,6 +363,42 @@ static inline void free_partition(struct mtd_part *p)
>  	kfree(p);
>  }
>  
> +/**
> + * mtd_parse_part - parse MTD partition with a matching parser
> + *
> + * Some partitions use a specific format to describe contained subpartitions
> + * (volumes). This function tries to use a proper parser for a given format and
> + * registers found (sub)partitions.
> + */
> +static int mtd_parse_part(struct mtd_part *slave, const char *format)
> +{
> +	struct mtd_partitions parsed;
> +	const char *probes[2];
> +	int i;
> +	int err;
> +
> +	probes[0] = format; /* Use parser with name matching the format */
> +	probes[1] = NULL; /* End of parsers */
> +	err = parse_mtd_partitions(&slave->mtd, probes, &parsed, NULL);
> +	if (err)
> +		return err;
> +	else if (!parsed.nr_parts)
> +		return -ENOENT;
> +
> +	for (i = 0; i < parsed.nr_parts; i++) {
> +		struct mtd_partition *part;
> +
> +		part = (struct mtd_partition *)&parsed.parts[i];
> +		part->offset += slave->offset;
> +	}
> +
> +	err = add_mtd_partitions(slave->master, parsed.parts, parsed.nr_parts);
> +
> +	mtd_part_parser_cleanup(&parsed);
> +
> +	return err;
> +}
> +
>  /*
>   * This function unregisters and destroy all slave MTD objects which are
>   * attached to the given master MTD object.
> @@ -724,6 +760,8 @@ int add_mtd_partitions(struct mtd_info *master,
>  
>  		add_mtd_device(&slave->mtd);
>  		mtd_add_partition_attrs(slave);
> +		if (parts[i].format)
> +			mtd_parse_part(slave, parts[i].format);
>  
>  		cur_offset = slave->offset + slave->mtd.size;
>  	}
> diff --git a/include/linux/mtd/partitions.h b/include/linux/mtd/partitions.h
> index 06df1e06b6e0..2787e76c030f 100644
> --- a/include/linux/mtd/partitions.h
> +++ b/include/linux/mtd/partitions.h
> @@ -20,6 +20,12 @@
>   *
>   * For each partition, these fields are available:
>   * name: string that will be used to label the partition's MTD device.
> + * format: some partitions can be containers using specific format to describe
> + *	embedded subpartitions / volumes. E.g. many home routers use "firmware"
> + *	partition that contains at least kernel and rootfs. In such case an
> + *	extra parser is needed that will detect these dynamic partitions and
> + *	report them to the MTD subsystem. This property describes partition
> + *	format and allows MTD core to call a proper parser.
>   * size: the partition size; if defined as MTDPART_SIZ_FULL, the partition
>   * 	will extend to the end of the master MTD device.
>   * offset: absolute starting position within the master MTD device; if
> @@ -38,6 +44,7 @@
>  
>  struct mtd_partition {
>  	const char *name;		/* identifier string */
> +	const char *format;		/* partition format */
>  	uint64_t size;			/* partition size */
>  	uint64_t offset;		/* offset within the master MTD space */
>  	uint32_t mask_flags;		/* master MTD flags to mask out for this partition */
> 


-- 
Best regards,
Marek Vasut

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH V2 1/2] mtd: add support for partition parsers
  2017-02-27  9:36 ` [PATCH V2 1/2] mtd: add support for partition parsers Marek Vasut
@ 2017-02-27  9:51   ` Rafał Miłecki
  0 siblings, 0 replies; 16+ messages in thread
From: Rafał Miłecki @ 2017-02-27  9:51 UTC (permalink / raw)
  To: Marek Vasut
  Cc: David Woodhouse, Brian Norris, Boris Brezillon,
	Richard Weinberger, Cyrille Pitchen, linux-mtd,
	Rafał Miłecki

On 27 February 2017 at 10:36, Marek Vasut <marek.vasut@gmail.com> wrote:
> On 02/21/2017 01:40 PM, Rafał Miłecki wrote:
>> From: Rafał Miłecki <rafal@milecki.pl>
>>
>> Some devices have partitions that are kind of containers with extra
>> subpartitions / volumes instead of e.g. simple filesystem data. To
>> support such cases we need to first create normal flash partitions and
>> then take care of these special ones.
>>
>> It's very common case for home routers. Depending on the vendor there
>> are formats like TRX, Seama, uImage
>
> uImage is file format, not partition type. It's just a blob of data
> (usually kernel) with small header, I don't think we should represent
> the content of it as a partition.

You are correct, this isn't any standard container with
subpartitions/volumes. This shouldn't be listed in this commit
message, sorry. I'll drop it in V2.

-- 
Rafał

^ permalink raw reply	[flat|nested] 16+ messages in thread

* [PATCH V3 1/2] mtd: add support for partition parsers
  2017-02-21 12:40 [PATCH V2 1/2] mtd: add support for partition parsers Rafał Miłecki
  2017-02-21 12:40 ` [PATCH V2 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module Rafał Miłecki
  2017-02-27  9:36 ` [PATCH V2 1/2] mtd: add support for partition parsers Marek Vasut
@ 2017-02-27 13:06 ` Rafał Miłecki
  2017-02-27 13:06   ` [PATCH V3 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module Rafał Miłecki
                     ` (3 more replies)
  2 siblings, 4 replies; 16+ messages in thread
From: Rafał Miłecki @ 2017-02-27 13:06 UTC (permalink / raw)
  To: David Woodhouse, Brian Norris, Boris Brezillon, Marek Vasut,
	Richard Weinberger, Cyrille Pitchen
  Cc: linux-mtd, Rafał Miłecki

From: Rafał Miłecki <rafal@milecki.pl>

Some devices have partitions that are kind of containers with extra
subpartitions / volumes instead of e.g. simple filesystem data. To
support such cases we need to first create normal flash partitions and
then take care of these special ones.

It's very common case for home routers. Depending on the vendor there
are formats like TRX, Seama, TP-Link, WRGG & more. All of them are used
to embed few partitions into a single one / single firmware file.

Ideally all vendors would use some well documented / standardized format
like UBI (and some probably start doing so), but there are still
countless devices on the market using these poor vendor specific
formats.

This patch extends MTD subsystem by allowing to specify partition format
and trying to use a proper parser when needed. Supporting such poor
formats is highly unlikely to be the top priority so these changes try
to minimize maintenance cost to the minimum. It reuses existing code for
these new parsers and just adds a one property and one new function.

This implementation requires setting partition format in a flash parser.
A proper change of bcm47xxpart will follow and in the future we will
hopefully also find a solution for doing it with ofpart
("fixed-partitions").

Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
---
V2: A totally rebased & refreshed version.
V3: Don't mention uImage in commit message, it was a mistake.
---
 drivers/mtd/mtdpart.c          | 38 ++++++++++++++++++++++++++++++++++++++
 include/linux/mtd/partitions.h |  7 +++++++
 2 files changed, 45 insertions(+)

diff --git a/drivers/mtd/mtdpart.c b/drivers/mtd/mtdpart.c
index ea5e5307f667..6d47ce6058c7 100644
--- a/drivers/mtd/mtdpart.c
+++ b/drivers/mtd/mtdpart.c
@@ -363,6 +363,42 @@ static inline void free_partition(struct mtd_part *p)
 	kfree(p);
 }
 
+/**
+ * mtd_parse_part - parse MTD partition with a matching parser
+ *
+ * Some partitions use a specific format to describe contained subpartitions
+ * (volumes). This function tries to use a proper parser for a given format and
+ * registers found (sub)partitions.
+ */
+static int mtd_parse_part(struct mtd_part *slave, const char *format)
+{
+	struct mtd_partitions parsed;
+	const char *probes[2];
+	int i;
+	int err;
+
+	probes[0] = format; /* Use parser with name matching the format */
+	probes[1] = NULL; /* End of parsers */
+	err = parse_mtd_partitions(&slave->mtd, probes, &parsed, NULL);
+	if (err)
+		return err;
+	else if (!parsed.nr_parts)
+		return -ENOENT;
+
+	for (i = 0; i < parsed.nr_parts; i++) {
+		struct mtd_partition *part;
+
+		part = (struct mtd_partition *)&parsed.parts[i];
+		part->offset += slave->offset;
+	}
+
+	err = add_mtd_partitions(slave->master, parsed.parts, parsed.nr_parts);
+
+	mtd_part_parser_cleanup(&parsed);
+
+	return err;
+}
+
 /*
  * This function unregisters and destroy all slave MTD objects which are
  * attached to the given master MTD object.
@@ -724,6 +760,8 @@ int add_mtd_partitions(struct mtd_info *master,
 
 		add_mtd_device(&slave->mtd);
 		mtd_add_partition_attrs(slave);
+		if (parts[i].format)
+			mtd_parse_part(slave, parts[i].format);
 
 		cur_offset = slave->offset + slave->mtd.size;
 	}
diff --git a/include/linux/mtd/partitions.h b/include/linux/mtd/partitions.h
index 06df1e06b6e0..2787e76c030f 100644
--- a/include/linux/mtd/partitions.h
+++ b/include/linux/mtd/partitions.h
@@ -20,6 +20,12 @@
  *
  * For each partition, these fields are available:
  * name: string that will be used to label the partition's MTD device.
+ * format: some partitions can be containers using specific format to describe
+ *	embedded subpartitions / volumes. E.g. many home routers use "firmware"
+ *	partition that contains at least kernel and rootfs. In such case an
+ *	extra parser is needed that will detect these dynamic partitions and
+ *	report them to the MTD subsystem. This property describes partition
+ *	format and allows MTD core to call a proper parser.
  * size: the partition size; if defined as MTDPART_SIZ_FULL, the partition
  * 	will extend to the end of the master MTD device.
  * offset: absolute starting position within the master MTD device; if
@@ -38,6 +44,7 @@
 
 struct mtd_partition {
 	const char *name;		/* identifier string */
+	const char *format;		/* partition format */
 	uint64_t size;			/* partition size */
 	uint64_t offset;		/* offset within the master MTD space */
 	uint32_t mask_flags;		/* master MTD flags to mask out for this partition */
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [PATCH V3 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module
  2017-02-27 13:06 ` [PATCH V3 " Rafał Miłecki
@ 2017-02-27 13:06   ` Rafał Miłecki
  2017-03-30 10:31   ` [PATCH V3 1/2] mtd: add support for partition parsers Rafał Miłecki
                     ` (2 subsequent siblings)
  3 siblings, 0 replies; 16+ messages in thread
From: Rafał Miłecki @ 2017-02-27 13:06 UTC (permalink / raw)
  To: David Woodhouse, Brian Norris, Boris Brezillon, Marek Vasut,
	Richard Weinberger, Cyrille Pitchen
  Cc: linux-mtd, Rafał Miłecki

From: Rafał Miłecki <rafal@milecki.pl>

This makes TRX parsing code reusable with other platforms and parsers.

Please note this patch doesn't really change anything in the existing
code, just moves it. There is still some place for improvement (e.g.
working on non-hacky method of checking rootfs format) but it's not
really a subject of this change.

Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
---
V2: A totally rebased & refreshed version.
---
 drivers/mtd/Kconfig              |   4 ++
 drivers/mtd/Makefile             |   1 +
 drivers/mtd/bcm47xxpart.c        |  97 +------------------------------
 drivers/mtd/parsers/Kconfig      |   8 +++
 drivers/mtd/parsers/Makefile     |   1 +
 drivers/mtd/parsers/parser_trx.c | 119 +++++++++++++++++++++++++++++++++++++++
 6 files changed, 136 insertions(+), 94 deletions(-)
 create mode 100644 drivers/mtd/parsers/Kconfig
 create mode 100644 drivers/mtd/parsers/Makefile
 create mode 100644 drivers/mtd/parsers/parser_trx.c

diff --git a/drivers/mtd/Kconfig b/drivers/mtd/Kconfig
index e83a279f1217..5a2d71729b9a 100644
--- a/drivers/mtd/Kconfig
+++ b/drivers/mtd/Kconfig
@@ -155,6 +155,10 @@ config MTD_BCM47XX_PARTS
 	  This provides partitions parser for devices based on BCM47xx
 	  boards.
 
+menu "Partition parsers"
+source "drivers/mtd/parsers/Kconfig"
+endmenu
+
 comment "User Modules And Translation Layers"
 
 #
diff --git a/drivers/mtd/Makefile b/drivers/mtd/Makefile
index 99bb9a1f6e16..151d60df303a 100644
--- a/drivers/mtd/Makefile
+++ b/drivers/mtd/Makefile
@@ -13,6 +13,7 @@ obj-$(CONFIG_MTD_AFS_PARTS)	+= afs.o
 obj-$(CONFIG_MTD_AR7_PARTS)	+= ar7part.o
 obj-$(CONFIG_MTD_BCM63XX_PARTS)	+= bcm63xxpart.o
 obj-$(CONFIG_MTD_BCM47XX_PARTS)	+= bcm47xxpart.o
+obj-y				+= parsers/
 
 # 'Users' - code which presents functionality to userspace.
 obj-$(CONFIG_MTD_BLKDEVS)	+= mtd_blkdevs.o
diff --git a/drivers/mtd/bcm47xxpart.c b/drivers/mtd/bcm47xxpart.c
index d10fa6c8f074..66ce72fd1426 100644
--- a/drivers/mtd/bcm47xxpart.c
+++ b/drivers/mtd/bcm47xxpart.c
@@ -43,7 +43,6 @@
 #define ML_MAGIC2			0x26594131
 #define TRX_MAGIC			0x30524448
 #define SHSQ_MAGIC			0x71736873	/* shsq (weird ZTE H218N endianness) */
-#define UBI_EC_MAGIC			0x23494255	/* UBI# */
 
 struct trx_header {
 	uint32_t magic;
@@ -62,89 +61,6 @@ static void bcm47xxpart_add_part(struct mtd_partition *part, const char *name,
 	part->mask_flags = mask_flags;
 }
 
-static const char *bcm47xxpart_trx_data_part_name(struct mtd_info *master,
-						  size_t offset)
-{
-	uint32_t buf;
-	size_t bytes_read;
-	int err;
-
-	err  = mtd_read(master, offset, sizeof(buf), &bytes_read,
-			(uint8_t *)&buf);
-	if (err && !mtd_is_bitflip(err)) {
-		pr_err("mtd_read error while parsing (offset: 0x%X): %d\n",
-			offset, err);
-		goto out_default;
-	}
-
-	if (buf == UBI_EC_MAGIC)
-		return "ubi";
-
-out_default:
-	return "rootfs";
-}
-
-static int bcm47xxpart_parse_trx(struct mtd_info *master,
-				 struct mtd_partition *trx,
-				 struct mtd_partition *parts,
-				 size_t parts_len)
-{
-	struct trx_header header;
-	size_t bytes_read;
-	int curr_part = 0;
-	int i, err;
-
-	if (parts_len < 3) {
-		pr_warn("No enough space to add TRX partitions!\n");
-		return -ENOMEM;
-	}
-
-	err = mtd_read(master, trx->offset, sizeof(header), &bytes_read,
-		       (uint8_t *)&header);
-	if (err && !mtd_is_bitflip(err)) {
-		pr_err("mtd_read error while reading TRX header: %d\n", err);
-		return err;
-	}
-
-	i = 0;
-
-	/* We have LZMA loader if offset[2] points to sth */
-	if (header.offset[2]) {
-		bcm47xxpart_add_part(&parts[curr_part++], "loader",
-				     trx->offset + header.offset[i], 0);
-		i++;
-	}
-
-	if (header.offset[i]) {
-		bcm47xxpart_add_part(&parts[curr_part++], "linux",
-				     trx->offset + header.offset[i], 0);
-		i++;
-	}
-
-	if (header.offset[i]) {
-		size_t offset = trx->offset + header.offset[i];
-		const char *name = bcm47xxpart_trx_data_part_name(master,
-								  offset);
-
-		bcm47xxpart_add_part(&parts[curr_part++], name, offset, 0);
-		i++;
-	}
-
-	/*
-	 * Assume that every partition ends at the beginning of the one it is
-	 * followed by.
-	 */
-	for (i = 0; i < curr_part; i++) {
-		u64 next_part_offset = (i < curr_part - 1) ?
-					parts[i + 1].offset :
-					trx->offset + trx->size;
-
-		parts[i].size = next_part_offset - parts[i].offset;
-	}
-
-	return curr_part;
-}
-
 /**
  * bcm47xxpart_bootpartition - gets index of TRX partition used by bootloader
  *
@@ -362,17 +278,10 @@ static int bcm47xxpart_parse(struct mtd_info *master,
 	for (i = 0; i < trx_num; i++) {
 		struct mtd_partition *trx = &parts[trx_parts[i]];
 
-		if (i == bcm47xxpart_bootpartition()) {
-			int num_parts;
-
-			num_parts = bcm47xxpart_parse_trx(master, trx,
-							  parts + curr_part,
-							  BCM47XXPART_MAX_PARTS - curr_part);
-			if (num_parts > 0)
-				curr_part += num_parts;
-		} else {
+		if (i == bcm47xxpart_bootpartition())
+			trx->format = "trx";
+		else
 			trx->name = "failsafe";
-		}
 	}
 
 	*pparts = parts;
diff --git a/drivers/mtd/parsers/Kconfig b/drivers/mtd/parsers/Kconfig
new file mode 100644
index 000000000000..ebb697a52698
--- /dev/null
+++ b/drivers/mtd/parsers/Kconfig
@@ -0,0 +1,8 @@
+config MTD_PARSER_TRX
+	tristate "Parser for TRX format partitions"
+	depends on MTD && (BCM47XX || ARCH_BCM_5301X)
+	help
+	  TRX is a firmware format used by Broadcom on their devices. It
+	  may contain up to 3/4 partitions (depending on the version).
+	  This driver will parse TRX header and report at least two partitions:
+	  kernel and rootfs.
diff --git a/drivers/mtd/parsers/Makefile b/drivers/mtd/parsers/Makefile
new file mode 100644
index 000000000000..4d9024e0be3b
--- /dev/null
+++ b/drivers/mtd/parsers/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_MTD_PARSER_TRX)		+= parser_trx.o
diff --git a/drivers/mtd/parsers/parser_trx.c b/drivers/mtd/parsers/parser_trx.c
new file mode 100644
index 000000000000..f35e28148808
--- /dev/null
+++ b/drivers/mtd/parsers/parser_trx.c
@@ -0,0 +1,119 @@
+/*
+ * Parser for TRX format partitions
+ *
+ * Copyright (C) 2012 - 2017 Rafał Miłecki <rafal@milecki.pl>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/mtd/mtd.h>
+#include <linux/mtd/partitions.h>
+
+#define TRX_PARSER_MAX_PARTS		4
+
+/* Magics */
+#define UBI_EC_MAGIC			0x23494255	/* UBI# */
+
+struct trx_header {
+	uint32_t magic;
+	uint32_t length;
+	uint32_t crc32;
+	uint16_t flags;
+	uint16_t version;
+	uint32_t offset[3];
+} __packed;
+
+static const char *parser_trx_data_part_name(struct mtd_info *master,
+					     size_t offset)
+{
+	uint32_t buf;
+	size_t bytes_read;
+	int err;
+
+	err  = mtd_read(master, offset, sizeof(buf), &bytes_read,
+			(uint8_t *)&buf);
+	if (err && !mtd_is_bitflip(err)) {
+		pr_err("mtd_read error while parsing (offset: 0x%X): %d\n",
+			offset, err);
+		goto out_default;
+	}
+
+	if (buf == UBI_EC_MAGIC)
+		return "ubi";
+
+out_default:
+	return "rootfs";
+}
+
+static int parser_trx_parse(struct mtd_info *mtd,
+			    const struct mtd_partition **pparts,
+			    struct mtd_part_parser_data *data)
+{
+	struct mtd_partition *parts;
+	struct mtd_partition *part;
+	struct trx_header trx;
+	size_t bytes_read;
+	uint8_t curr_part = 0, i = 0;
+	int err;
+
+	parts = kzalloc(sizeof(struct mtd_partition) * TRX_PARSER_MAX_PARTS,
+			GFP_KERNEL);
+	if (!parts)
+		return -ENOMEM;
+
+	err = mtd_read(mtd, 0, sizeof(trx), &bytes_read, (uint8_t *)&trx);
+	if (err) {
+		pr_err("MTD reading error: %d\n", err);
+		return err;
+	}
+
+	/* We have LZMA loader if offset[2] points to sth */
+	if (trx.offset[2]) {
+		part = &parts[curr_part++];
+		part->name = "loader";
+		part->offset = trx.offset[i];
+		i++;
+	}
+
+	if (trx.offset[i]) {
+		part = &parts[curr_part++];
+		part->name = "linux";
+		part->offset = trx.offset[i];
+		i++;
+	}
+
+	if (trx.offset[i]) {
+		part = &parts[curr_part++];
+		part->name = parser_trx_data_part_name(mtd, trx.offset[i]);
+		part->offset = trx.offset[i];
+		i++;
+	}
+
+	/*
+	 * Assume that every partition ends at the beginning of the one it is
+	 * followed by.
+	 */
+	for (i = 0; i < curr_part; i++) {
+		u64 next_part_offset = (i < curr_part - 1) ?
+				       parts[i + 1].offset : mtd->size;
+
+		parts[i].size = next_part_offset - parts[i].offset;
+	}
+
+	*pparts = parts;
+	return i;
+};
+
+static struct mtd_part_parser mtd_parser_trx = {
+	.parse_fn = parser_trx_parse,
+	.name = "trx",
+};
+module_mtd_part_parser(mtd_parser_trx);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Parser for TRX format partitions");
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 16+ messages in thread

* Re: [PATCH V3 1/2] mtd: add support for partition parsers
  2017-02-27 13:06 ` [PATCH V3 " Rafał Miłecki
  2017-02-27 13:06   ` [PATCH V3 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module Rafał Miłecki
@ 2017-03-30 10:31   ` Rafał Miłecki
  2017-03-30 11:13   ` Marek Vasut
  2017-03-30 12:35   ` [PATCH V4 " Rafał Miłecki
  3 siblings, 0 replies; 16+ messages in thread
From: Rafał Miłecki @ 2017-03-30 10:31 UTC (permalink / raw)
  To: David Woodhouse, Brian Norris, Boris Brezillon, Marek Vasut,
	Richard Weinberger, Cyrille Pitchen
  Cc: linux-mtd, Rafał Miłecki

On 02/27/2017 02:06 PM, Rafał Miłecki wrote:
> From: Rafał Miłecki <rafal@milecki.pl>
>
> Some devices have partitions that are kind of containers with extra
> subpartitions / volumes instead of e.g. simple filesystem data. To
> support such cases we need to first create normal flash partitions and
> then take care of these special ones.
>
> It's very common case for home routers. Depending on the vendor there
> are formats like TRX, Seama, TP-Link, WRGG & more. All of them are used
> to embed few partitions into a single one / single firmware file.
>
> Ideally all vendors would use some well documented / standardized format
> like UBI (and some probably start doing so), but there are still
> countless devices on the market using these poor vendor specific
> formats.
>
> This patch extends MTD subsystem by allowing to specify partition format
> and trying to use a proper parser when needed. Supporting such poor
> formats is highly unlikely to be the top priority so these changes try
> to minimize maintenance cost to the minimum. It reuses existing code for
> these new parsers and just adds a one property and one new function.
>
> This implementation requires setting partition format in a flash parser.
> A proper change of bcm47xxpart will follow and in the future we will
> hopefully also find a solution for doing it with ofpart
> ("fixed-partitions").
>
> Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
> ---
> V2: A totally rebased & refreshed version.
> V3: Don't mention uImage in commit message, it was a mistake.

Marek: I updated commit message as you suggested.

Can I ask for your Reviewed/Acked tag, please?

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH V3 1/2] mtd: add support for partition parsers
  2017-02-27 13:06 ` [PATCH V3 " Rafał Miłecki
  2017-02-27 13:06   ` [PATCH V3 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module Rafał Miłecki
  2017-03-30 10:31   ` [PATCH V3 1/2] mtd: add support for partition parsers Rafał Miłecki
@ 2017-03-30 11:13   ` Marek Vasut
  2017-03-30 12:35   ` [PATCH V4 " Rafał Miłecki
  3 siblings, 0 replies; 16+ messages in thread
From: Marek Vasut @ 2017-03-30 11:13 UTC (permalink / raw)
  To: Rafał Miłecki, David Woodhouse, Brian Norris,
	Boris Brezillon, Richard Weinberger, Cyrille Pitchen
  Cc: linux-mtd, Rafał Miłecki

On 02/27/2017 02:06 PM, Rafał Miłecki wrote:
> From: Rafał Miłecki <rafal@milecki.pl>
> 
> Some devices have partitions that are kind of containers with extra
> subpartitions / volumes instead of e.g. simple filesystem data. To
> support such cases we need to first create normal flash partitions and
> then take care of these special ones.
> 
> It's very common case for home routers. Depending on the vendor there
> are formats like TRX, Seama, TP-Link, WRGG & more. All of them are used
> to embed few partitions into a single one / single firmware file.
> 
> Ideally all vendors would use some well documented / standardized format
> like UBI (and some probably start doing so), but there are still
> countless devices on the market using these poor vendor specific
> formats.
> 
> This patch extends MTD subsystem by allowing to specify partition format
> and trying to use a proper parser when needed. Supporting such poor
> formats is highly unlikely to be the top priority so these changes try
> to minimize maintenance cost to the minimum. It reuses existing code for
> these new parsers and just adds a one property and one new function.
> 
> This implementation requires setting partition format in a flash parser.
> A proper change of bcm47xxpart will follow and in the future we will
> hopefully also find a solution for doing it with ofpart
> ("fixed-partitions").
> 
> Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
> ---
> V2: A totally rebased & refreshed version.
> V3: Don't mention uImage in commit message, it was a mistake.
> ---
>  drivers/mtd/mtdpart.c          | 38 ++++++++++++++++++++++++++++++++++++++
>  include/linux/mtd/partitions.h |  7 +++++++
>  2 files changed, 45 insertions(+)
> 
> diff --git a/drivers/mtd/mtdpart.c b/drivers/mtd/mtdpart.c
> index ea5e5307f667..6d47ce6058c7 100644
> --- a/drivers/mtd/mtdpart.c
> +++ b/drivers/mtd/mtdpart.c
> @@ -363,6 +363,42 @@ static inline void free_partition(struct mtd_part *p)
>  	kfree(p);
>  }
>  
> +/**
> + * mtd_parse_part - parse MTD partition with a matching parser

It'd be useful to document the parameters as per kerneldoc style.

Otherwise
Acked-by: Marek Vasut <marek.vasut@gmail.com>

> + * Some partitions use a specific format to describe contained subpartitions
> + * (volumes). This function tries to use a proper parser for a given format and
> + * registers found (sub)partitions.
> + */
> +static int mtd_parse_part(struct mtd_part *slave, const char *format)
> +{
> +	struct mtd_partitions parsed;
> +	const char *probes[2];
> +	int i;
> +	int err;
> +
> +	probes[0] = format; /* Use parser with name matching the format */
> +	probes[1] = NULL; /* End of parsers */
> +	err = parse_mtd_partitions(&slave->mtd, probes, &parsed, NULL);
> +	if (err)
> +		return err;
> +	else if (!parsed.nr_parts)
> +		return -ENOENT;
> +
> +	for (i = 0; i < parsed.nr_parts; i++) {
> +		struct mtd_partition *part;
> +
> +		part = (struct mtd_partition *)&parsed.parts[i];
> +		part->offset += slave->offset;
> +	}
> +
> +	err = add_mtd_partitions(slave->master, parsed.parts, parsed.nr_parts);
> +
> +	mtd_part_parser_cleanup(&parsed);
> +
> +	return err;
> +}
> +
>  /*
>   * This function unregisters and destroy all slave MTD objects which are
>   * attached to the given master MTD object.
> @@ -724,6 +760,8 @@ int add_mtd_partitions(struct mtd_info *master,
>  
>  		add_mtd_device(&slave->mtd);
>  		mtd_add_partition_attrs(slave);
> +		if (parts[i].format)
> +			mtd_parse_part(slave, parts[i].format);
>  
>  		cur_offset = slave->offset + slave->mtd.size;
>  	}
> diff --git a/include/linux/mtd/partitions.h b/include/linux/mtd/partitions.h
> index 06df1e06b6e0..2787e76c030f 100644
> --- a/include/linux/mtd/partitions.h
> +++ b/include/linux/mtd/partitions.h
> @@ -20,6 +20,12 @@
>   *
>   * For each partition, these fields are available:
>   * name: string that will be used to label the partition's MTD device.
> + * format: some partitions can be containers using specific format to describe
> + *	embedded subpartitions / volumes. E.g. many home routers use "firmware"
> + *	partition that contains at least kernel and rootfs. In such case an
> + *	extra parser is needed that will detect these dynamic partitions and
> + *	report them to the MTD subsystem. This property describes partition
> + *	format and allows MTD core to call a proper parser.
>   * size: the partition size; if defined as MTDPART_SIZ_FULL, the partition
>   * 	will extend to the end of the master MTD device.
>   * offset: absolute starting position within the master MTD device; if
> @@ -38,6 +44,7 @@
>  
>  struct mtd_partition {
>  	const char *name;		/* identifier string */
> +	const char *format;		/* partition format */
>  	uint64_t size;			/* partition size */
>  	uint64_t offset;		/* offset within the master MTD space */
>  	uint32_t mask_flags;		/* master MTD flags to mask out for this partition */
> 


-- 
Best regards,
Marek Vasut

^ permalink raw reply	[flat|nested] 16+ messages in thread

* [PATCH V4 1/2] mtd: add support for partition parsers
  2017-02-27 13:06 ` [PATCH V3 " Rafał Miłecki
                     ` (2 preceding siblings ...)
  2017-03-30 11:13   ` Marek Vasut
@ 2017-03-30 12:35   ` Rafał Miłecki
  2017-03-30 12:35     ` [PATCH V4 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module Rafał Miłecki
  2017-05-11 18:21     ` [PATCH V4 1/2] mtd: add support for partition parsers Brian Norris
  3 siblings, 2 replies; 16+ messages in thread
From: Rafał Miłecki @ 2017-03-30 12:35 UTC (permalink / raw)
  To: David Woodhouse, Brian Norris, Boris Brezillon, Marek Vasut,
	Richard Weinberger
  Cc: Cyrille Pitchen, linux-mtd, Rafał Miłecki

From: Rafał Miłecki <rafal@milecki.pl>

Some devices have partitions that are kind of containers with extra
subpartitions / volumes instead of e.g. simple filesystem data. To
support such cases we need to first create normal flash partitions and
then take care of these special ones.

It's very common case for home routers. Depending on the vendor there
are formats like TRX, Seama, TP-Link, WRGG & more. All of them are used
to embed few partitions into a single one / single firmware file.

Ideally all vendors would use some well documented / standardized format
like UBI (and some probably start doing so), but there are still
countless devices on the market using these poor vendor specific
formats.

This patch extends MTD subsystem by allowing to specify partition format
and trying to use a proper parser when needed. Supporting such poor
formats is highly unlikely to be the top priority so these changes try
to minimize maintenance cost to the minimum. It reuses existing code for
these new parsers and just adds a one property and one new function.

This implementation requires setting partition format in a flash parser.
A proper change of bcm47xxpart will follow and in the future we will
hopefully also find a solution for doing it with ofpart
("fixed-partitions").

Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
Acked-by: Marek Vasut <marek.vasut@gmail.com>
---
V2: A totally rebased & refreshed version.
V3: Don't mention uImage in commit message, it was a mistake.
V4: Document mtd_parse_part parameters
---
 drivers/mtd/mtdpart.c          | 41 +++++++++++++++++++++++++++++++++++++++++
 include/linux/mtd/partitions.h |  7 +++++++
 2 files changed, 48 insertions(+)

diff --git a/drivers/mtd/mtdpart.c b/drivers/mtd/mtdpart.c
index ea5e5307f667..81e0b80237df 100644
--- a/drivers/mtd/mtdpart.c
+++ b/drivers/mtd/mtdpart.c
@@ -363,6 +363,45 @@ static inline void free_partition(struct mtd_part *p)
 	kfree(p);
 }
 
+/**
+ * mtd_parse_part - parse MTD partition with a matching parser
+ *
+ * @slave: part to be parsed for subpartitions
+ * @format: partition format used to call a proper parser
+ *
+ * Some partitions use a specific format to describe contained subpartitions
+ * (volumes). This function tries to use a proper parser for a given format and
+ * registers found (sub)partitions.
+ */
+static int mtd_parse_part(struct mtd_part *slave, const char *format)
+{
+	struct mtd_partitions parsed;
+	const char *probes[2];
+	int i;
+	int err;
+
+	probes[0] = format; /* Use parser with name matching the format */
+	probes[1] = NULL; /* End of parsers */
+	err = parse_mtd_partitions(&slave->mtd, probes, &parsed, NULL);
+	if (err)
+		return err;
+	else if (!parsed.nr_parts)
+		return -ENOENT;
+
+	for (i = 0; i < parsed.nr_parts; i++) {
+		struct mtd_partition *part;
+
+		part = (struct mtd_partition *)&parsed.parts[i];
+		part->offset += slave->offset;
+	}
+
+	err = add_mtd_partitions(slave->master, parsed.parts, parsed.nr_parts);
+
+	mtd_part_parser_cleanup(&parsed);
+
+	return err;
+}
+
 /*
  * This function unregisters and destroy all slave MTD objects which are
  * attached to the given master MTD object.
@@ -724,6 +763,8 @@ int add_mtd_partitions(struct mtd_info *master,
 
 		add_mtd_device(&slave->mtd);
 		mtd_add_partition_attrs(slave);
+		if (parts[i].format)
+			mtd_parse_part(slave, parts[i].format);
 
 		cur_offset = slave->offset + slave->mtd.size;
 	}
diff --git a/include/linux/mtd/partitions.h b/include/linux/mtd/partitions.h
index 06df1e06b6e0..2787e76c030f 100644
--- a/include/linux/mtd/partitions.h
+++ b/include/linux/mtd/partitions.h
@@ -20,6 +20,12 @@
  *
  * For each partition, these fields are available:
  * name: string that will be used to label the partition's MTD device.
+ * format: some partitions can be containers using specific format to describe
+ *	embedded subpartitions / volumes. E.g. many home routers use "firmware"
+ *	partition that contains at least kernel and rootfs. In such case an
+ *	extra parser is needed that will detect these dynamic partitions and
+ *	report them to the MTD subsystem. This property describes partition
+ *	format and allows MTD core to call a proper parser.
  * size: the partition size; if defined as MTDPART_SIZ_FULL, the partition
  * 	will extend to the end of the master MTD device.
  * offset: absolute starting position within the master MTD device; if
@@ -38,6 +44,7 @@
 
 struct mtd_partition {
 	const char *name;		/* identifier string */
+	const char *format;		/* partition format */
 	uint64_t size;			/* partition size */
 	uint64_t offset;		/* offset within the master MTD space */
 	uint32_t mask_flags;		/* master MTD flags to mask out for this partition */
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [PATCH V4 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module
  2017-03-30 12:35   ` [PATCH V4 " Rafał Miłecki
@ 2017-03-30 12:35     ` Rafał Miłecki
  2017-05-11 18:31       ` Brian Norris
  2017-05-11 18:21     ` [PATCH V4 1/2] mtd: add support for partition parsers Brian Norris
  1 sibling, 1 reply; 16+ messages in thread
From: Rafał Miłecki @ 2017-03-30 12:35 UTC (permalink / raw)
  To: David Woodhouse, Brian Norris, Boris Brezillon, Marek Vasut,
	Richard Weinberger
  Cc: Cyrille Pitchen, linux-mtd, Rafał Miłecki

From: Rafał Miłecki <rafal@milecki.pl>

This makes TRX parsing code reusable with other platforms and parsers.

Please note this patch doesn't really change anything in the existing
code, just moves it. There is still some place for improvement (e.g.
working on non-hacky method of checking rootfs format) but it's not
really a subject of this change.

Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
---
V2: A totally rebased & refreshed version.
---
 drivers/mtd/Kconfig              |   4 ++
 drivers/mtd/Makefile             |   1 +
 drivers/mtd/bcm47xxpart.c        |  97 +------------------------------
 drivers/mtd/parsers/Kconfig      |   8 +++
 drivers/mtd/parsers/Makefile     |   1 +
 drivers/mtd/parsers/parser_trx.c | 119 +++++++++++++++++++++++++++++++++++++++
 6 files changed, 136 insertions(+), 94 deletions(-)
 create mode 100644 drivers/mtd/parsers/Kconfig
 create mode 100644 drivers/mtd/parsers/Makefile
 create mode 100644 drivers/mtd/parsers/parser_trx.c

diff --git a/drivers/mtd/Kconfig b/drivers/mtd/Kconfig
index e83a279f1217..5a2d71729b9a 100644
--- a/drivers/mtd/Kconfig
+++ b/drivers/mtd/Kconfig
@@ -155,6 +155,10 @@ config MTD_BCM47XX_PARTS
 	  This provides partitions parser for devices based on BCM47xx
 	  boards.
 
+menu "Partition parsers"
+source "drivers/mtd/parsers/Kconfig"
+endmenu
+
 comment "User Modules And Translation Layers"
 
 #
diff --git a/drivers/mtd/Makefile b/drivers/mtd/Makefile
index 99bb9a1f6e16..151d60df303a 100644
--- a/drivers/mtd/Makefile
+++ b/drivers/mtd/Makefile
@@ -13,6 +13,7 @@ obj-$(CONFIG_MTD_AFS_PARTS)	+= afs.o
 obj-$(CONFIG_MTD_AR7_PARTS)	+= ar7part.o
 obj-$(CONFIG_MTD_BCM63XX_PARTS)	+= bcm63xxpart.o
 obj-$(CONFIG_MTD_BCM47XX_PARTS)	+= bcm47xxpart.o
+obj-y				+= parsers/
 
 # 'Users' - code which presents functionality to userspace.
 obj-$(CONFIG_MTD_BLKDEVS)	+= mtd_blkdevs.o
diff --git a/drivers/mtd/bcm47xxpart.c b/drivers/mtd/bcm47xxpart.c
index d10fa6c8f074..66ce72fd1426 100644
--- a/drivers/mtd/bcm47xxpart.c
+++ b/drivers/mtd/bcm47xxpart.c
@@ -43,7 +43,6 @@
 #define ML_MAGIC2			0x26594131
 #define TRX_MAGIC			0x30524448
 #define SHSQ_MAGIC			0x71736873	/* shsq (weird ZTE H218N endianness) */
-#define UBI_EC_MAGIC			0x23494255	/* UBI# */
 
 struct trx_header {
 	uint32_t magic;
@@ -62,89 +61,6 @@ static void bcm47xxpart_add_part(struct mtd_partition *part, const char *name,
 	part->mask_flags = mask_flags;
 }
 
-static const char *bcm47xxpart_trx_data_part_name(struct mtd_info *master,
-						  size_t offset)
-{
-	uint32_t buf;
-	size_t bytes_read;
-	int err;
-
-	err  = mtd_read(master, offset, sizeof(buf), &bytes_read,
-			(uint8_t *)&buf);
-	if (err && !mtd_is_bitflip(err)) {
-		pr_err("mtd_read error while parsing (offset: 0x%X): %d\n",
-			offset, err);
-		goto out_default;
-	}
-
-	if (buf == UBI_EC_MAGIC)
-		return "ubi";
-
-out_default:
-	return "rootfs";
-}
-
-static int bcm47xxpart_parse_trx(struct mtd_info *master,
-				 struct mtd_partition *trx,
-				 struct mtd_partition *parts,
-				 size_t parts_len)
-{
-	struct trx_header header;
-	size_t bytes_read;
-	int curr_part = 0;
-	int i, err;
-
-	if (parts_len < 3) {
-		pr_warn("No enough space to add TRX partitions!\n");
-		return -ENOMEM;
-	}
-
-	err = mtd_read(master, trx->offset, sizeof(header), &bytes_read,
-		       (uint8_t *)&header);
-	if (err && !mtd_is_bitflip(err)) {
-		pr_err("mtd_read error while reading TRX header: %d\n", err);
-		return err;
-	}
-
-	i = 0;
-
-	/* We have LZMA loader if offset[2] points to sth */
-	if (header.offset[2]) {
-		bcm47xxpart_add_part(&parts[curr_part++], "loader",
-				     trx->offset + header.offset[i], 0);
-		i++;
-	}
-
-	if (header.offset[i]) {
-		bcm47xxpart_add_part(&parts[curr_part++], "linux",
-				     trx->offset + header.offset[i], 0);
-		i++;
-	}
-
-	if (header.offset[i]) {
-		size_t offset = trx->offset + header.offset[i];
-		const char *name = bcm47xxpart_trx_data_part_name(master,
-								  offset);
-
-		bcm47xxpart_add_part(&parts[curr_part++], name, offset, 0);
-		i++;
-	}
-
-	/*
-	 * Assume that every partition ends at the beginning of the one it is
-	 * followed by.
-	 */
-	for (i = 0; i < curr_part; i++) {
-		u64 next_part_offset = (i < curr_part - 1) ?
-					parts[i + 1].offset :
-					trx->offset + trx->size;
-
-		parts[i].size = next_part_offset - parts[i].offset;
-	}
-
-	return curr_part;
-}
-
 /**
  * bcm47xxpart_bootpartition - gets index of TRX partition used by bootloader
  *
@@ -362,17 +278,10 @@ static int bcm47xxpart_parse(struct mtd_info *master,
 	for (i = 0; i < trx_num; i++) {
 		struct mtd_partition *trx = &parts[trx_parts[i]];
 
-		if (i == bcm47xxpart_bootpartition()) {
-			int num_parts;
-
-			num_parts = bcm47xxpart_parse_trx(master, trx,
-							  parts + curr_part,
-							  BCM47XXPART_MAX_PARTS - curr_part);
-			if (num_parts > 0)
-				curr_part += num_parts;
-		} else {
+		if (i == bcm47xxpart_bootpartition())
+			trx->format = "trx";
+		else
 			trx->name = "failsafe";
-		}
 	}
 
 	*pparts = parts;
diff --git a/drivers/mtd/parsers/Kconfig b/drivers/mtd/parsers/Kconfig
new file mode 100644
index 000000000000..ebb697a52698
--- /dev/null
+++ b/drivers/mtd/parsers/Kconfig
@@ -0,0 +1,8 @@
+config MTD_PARSER_TRX
+	tristate "Parser for TRX format partitions"
+	depends on MTD && (BCM47XX || ARCH_BCM_5301X)
+	help
+	  TRX is a firmware format used by Broadcom on their devices. It
+	  may contain up to 3/4 partitions (depending on the version).
+	  This driver will parse TRX header and report at least two partitions:
+	  kernel and rootfs.
diff --git a/drivers/mtd/parsers/Makefile b/drivers/mtd/parsers/Makefile
new file mode 100644
index 000000000000..4d9024e0be3b
--- /dev/null
+++ b/drivers/mtd/parsers/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_MTD_PARSER_TRX)		+= parser_trx.o
diff --git a/drivers/mtd/parsers/parser_trx.c b/drivers/mtd/parsers/parser_trx.c
new file mode 100644
index 000000000000..f35e28148808
--- /dev/null
+++ b/drivers/mtd/parsers/parser_trx.c
@@ -0,0 +1,119 @@
+/*
+ * Parser for TRX format partitions
+ *
+ * Copyright (C) 2012 - 2017 Rafał Miłecki <rafal@milecki.pl>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/mtd/mtd.h>
+#include <linux/mtd/partitions.h>
+
+#define TRX_PARSER_MAX_PARTS		4
+
+/* Magics */
+#define UBI_EC_MAGIC			0x23494255	/* UBI# */
+
+struct trx_header {
+	uint32_t magic;
+	uint32_t length;
+	uint32_t crc32;
+	uint16_t flags;
+	uint16_t version;
+	uint32_t offset[3];
+} __packed;
+
+static const char *parser_trx_data_part_name(struct mtd_info *master,
+					     size_t offset)
+{
+	uint32_t buf;
+	size_t bytes_read;
+	int err;
+
+	err  = mtd_read(master, offset, sizeof(buf), &bytes_read,
+			(uint8_t *)&buf);
+	if (err && !mtd_is_bitflip(err)) {
+		pr_err("mtd_read error while parsing (offset: 0x%X): %d\n",
+			offset, err);
+		goto out_default;
+	}
+
+	if (buf == UBI_EC_MAGIC)
+		return "ubi";
+
+out_default:
+	return "rootfs";
+}
+
+static int parser_trx_parse(struct mtd_info *mtd,
+			    const struct mtd_partition **pparts,
+			    struct mtd_part_parser_data *data)
+{
+	struct mtd_partition *parts;
+	struct mtd_partition *part;
+	struct trx_header trx;
+	size_t bytes_read;
+	uint8_t curr_part = 0, i = 0;
+	int err;
+
+	parts = kzalloc(sizeof(struct mtd_partition) * TRX_PARSER_MAX_PARTS,
+			GFP_KERNEL);
+	if (!parts)
+		return -ENOMEM;
+
+	err = mtd_read(mtd, 0, sizeof(trx), &bytes_read, (uint8_t *)&trx);
+	if (err) {
+		pr_err("MTD reading error: %d\n", err);
+		return err;
+	}
+
+	/* We have LZMA loader if offset[2] points to sth */
+	if (trx.offset[2]) {
+		part = &parts[curr_part++];
+		part->name = "loader";
+		part->offset = trx.offset[i];
+		i++;
+	}
+
+	if (trx.offset[i]) {
+		part = &parts[curr_part++];
+		part->name = "linux";
+		part->offset = trx.offset[i];
+		i++;
+	}
+
+	if (trx.offset[i]) {
+		part = &parts[curr_part++];
+		part->name = parser_trx_data_part_name(mtd, trx.offset[i]);
+		part->offset = trx.offset[i];
+		i++;
+	}
+
+	/*
+	 * Assume that every partition ends at the beginning of the one it is
+	 * followed by.
+	 */
+	for (i = 0; i < curr_part; i++) {
+		u64 next_part_offset = (i < curr_part - 1) ?
+				       parts[i + 1].offset : mtd->size;
+
+		parts[i].size = next_part_offset - parts[i].offset;
+	}
+
+	*pparts = parts;
+	return i;
+};
+
+static struct mtd_part_parser mtd_parser_trx = {
+	.parse_fn = parser_trx_parse,
+	.name = "trx",
+};
+module_mtd_part_parser(mtd_parser_trx);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Parser for TRX format partitions");
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 16+ messages in thread

* Re: [PATCH V4 1/2] mtd: add support for partition parsers
  2017-03-30 12:35   ` [PATCH V4 " Rafał Miłecki
  2017-03-30 12:35     ` [PATCH V4 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module Rafał Miłecki
@ 2017-05-11 18:21     ` Brian Norris
  2017-05-23  8:14       ` Rafał Miłecki
  1 sibling, 1 reply; 16+ messages in thread
From: Brian Norris @ 2017-05-11 18:21 UTC (permalink / raw)
  To: Rafał Miłecki
  Cc: David Woodhouse, Boris Brezillon, Marek Vasut,
	Richard Weinberger, Cyrille Pitchen, linux-mtd,
	Rafał Miłecki

Hi,

On Thu, Mar 30, 2017 at 02:35:26PM +0200, Rafał Miłecki wrote:
> From: Rafał Miłecki <rafal@milecki.pl>
> 
> Some devices have partitions that are kind of containers with extra
> subpartitions / volumes instead of e.g. simple filesystem data. To
> support such cases we need to first create normal flash partitions and
> then take care of these special ones.
> 
> It's very common case for home routers. Depending on the vendor there
> are formats like TRX, Seama, TP-Link, WRGG & more. All of them are used
> to embed few partitions into a single one / single firmware file.
> 
> Ideally all vendors would use some well documented / standardized format
> like UBI (and some probably start doing so), but there are still
> countless devices on the market using these poor vendor specific
> formats.
> 
> This patch extends MTD subsystem by allowing to specify partition format
> and trying to use a proper parser when needed. Supporting such poor
> formats is highly unlikely to be the top priority so these changes try
> to minimize maintenance cost to the minimum. It reuses existing code for
> these new parsers and just adds a one property and one new function.
> 
> This implementation requires setting partition format in a flash parser.
> A proper change of bcm47xxpart will follow and in the future we will
> hopefully also find a solution for doing it with ofpart
> ("fixed-partitions").
> 
> Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
> Acked-by: Marek Vasut <marek.vasut@gmail.com>
> ---
> V2: A totally rebased & refreshed version.
> V3: Don't mention uImage in commit message, it was a mistake.
> V4: Document mtd_parse_part parameters
> ---
>  drivers/mtd/mtdpart.c          | 41 +++++++++++++++++++++++++++++++++++++++++
>  include/linux/mtd/partitions.h |  7 +++++++
>  2 files changed, 48 insertions(+)
> 
> diff --git a/drivers/mtd/mtdpart.c b/drivers/mtd/mtdpart.c
> index ea5e5307f667..81e0b80237df 100644
> --- a/drivers/mtd/mtdpart.c
> +++ b/drivers/mtd/mtdpart.c
> @@ -363,6 +363,45 @@ static inline void free_partition(struct mtd_part *p)
>  	kfree(p);
>  }
>  
> +/**
> + * mtd_parse_part - parse MTD partition with a matching parser

The naming is a bit confusing to me. We already have partition parsers,
but those are typically expected to apply to the whole device. Is this
infrastructure the same thing? Or different?

(To answer myself) this is nearly the same infrastructure, but it's just
for adding *sub*partitions, not top-level partitions. Maybe it would be
slightly less confusing if this was called mtd_parse_subpartitions() or
mtd_parse_subparts()?

> + *
> + * @slave: part to be parsed for subpartitions
> + * @format: partition format used to call a proper parser
> + *
> + * Some partitions use a specific format to describe contained subpartitions
> + * (volumes). This function tries to use a proper parser for a given format and
> + * registers found (sub)partitions.
> + */
> +static int mtd_parse_part(struct mtd_part *slave, const char *format)
> +{
> +	struct mtd_partitions parsed;
> +	const char *probes[2];
> +	int i;
> +	int err;
> +
> +	probes[0] = format; /* Use parser with name matching the format */
> +	probes[1] = NULL; /* End of parsers */
> +	err = parse_mtd_partitions(&slave->mtd, probes, &parsed, NULL);
> +	if (err)
> +		return err;
> +	else if (!parsed.nr_parts)
> +		return -ENOENT;
> +
> +	for (i = 0; i < parsed.nr_parts; i++) {
> +		struct mtd_partition *part;
> +
> +		part = (struct mtd_partition *)&parsed.parts[i];

I'm not super-happy with the de-constification cast here. What if the
partition array was somehow shared? (I don't expect that, but you're
still potentially violating a caller's assumptions when you modify their
'const' data.)

> +		part->offset += slave->offset;
> +	}
> +
> +	err = add_mtd_partitions(slave->master, parsed.parts, parsed.nr_parts);

Maybe a better way to handle that offset modification is to either pass
in the offset to add_mtd_partitions() (e.g., as a simple parameter), or
else adapt add_mtd_partitions() to handle receiving a non-master as the
first parameter. (Then you just pass "slave", and add_mtd_partitions()
handle it with something like "if (mtd_is_partition(...))".)

Then I wonder how we want the parenting structure to work; should the
sub-partition have the "master" as its parent, or the original
partition? I thought Richard had mentioned some problems with the
existing parenting structure (with CONFIG_MTD_PARTITIONED_MASTER)
already, but I don't remember what those were.

Also, if you're "parsing" the slave, but "adding" to the master, then
the bounds-checking logic in add_mtd_partitions() won't properly apply,
and you'll be able to create sub-partitions that extend beyond the
slave, right? If so, then I think (after auditing add_mtd_partitions() a
little closer, and possibly adjusting some of its comments) you might
really just want to pass 'slave', not 'slave->master'.

> +
> +	mtd_part_parser_cleanup(&parsed);
> +
> +	return err;
> +}
> +
>  /*
>   * This function unregisters and destroy all slave MTD objects which are
>   * attached to the given master MTD object.
> @@ -724,6 +763,8 @@ int add_mtd_partitions(struct mtd_info *master,
>  
>  		add_mtd_device(&slave->mtd);
>  		mtd_add_partition_attrs(slave);
> +		if (parts[i].format)
> +			mtd_parse_part(slave, parts[i].format);
>  
>  		cur_offset = slave->offset + slave->mtd.size;
>  	}
> diff --git a/include/linux/mtd/partitions.h b/include/linux/mtd/partitions.h
> index 06df1e06b6e0..2787e76c030f 100644
> --- a/include/linux/mtd/partitions.h
> +++ b/include/linux/mtd/partitions.h
> @@ -20,6 +20,12 @@
>   *
>   * For each partition, these fields are available:
>   * name: string that will be used to label the partition's MTD device.
> + * format: some partitions can be containers using specific format to describe
> + *	embedded subpartitions / volumes. E.g. many home routers use "firmware"
> + *	partition that contains at least kernel and rootfs. In such case an
> + *	extra parser is needed that will detect these dynamic partitions and
> + *	report them to the MTD subsystem. This property describes partition
> + *	format and allows MTD core to call a proper parser.
>   * size: the partition size; if defined as MTDPART_SIZ_FULL, the partition
>   * 	will extend to the end of the master MTD device.
>   * offset: absolute starting position within the master MTD device; if
> @@ -38,6 +44,7 @@
>  
>  struct mtd_partition {
>  	const char *name;		/* identifier string */
> +	const char *format;		/* partition format */

Why only a single format? Can't this use the same definition as what's
used already in, e.g., parse_mtd_partitions() and mtd_device_register()?
i.e., a NULL-terminated string array? Not that the definition ("const
char *const *types") is perfect (perhaps it could use a struct for
encapsulation), but it would be trivially easy to expand this to support
detecting more than one sub-partition type, no?

But then I see that your TRX parser doesn't actually do any validation;
it assumes that if it's called, it must match. I dunno if that can be
fixed...

Or maybe I'm missing the direction you're wanting to go with this. If
so, please correct me. Do you expect that you're only going to call a
sub-parser when you know *exactly* which one to use? If we were to do as
you suggest in the commit message and extend to "fixed-partitions", then
it seems more likely you'll want to support a list of potential
sub-parsers, and you'll want to validate them (i.e., check for headers
and abort and/or try the next one if they don't match).

>  	uint64_t size;			/* partition size */
>  	uint64_t offset;		/* offset within the master MTD space */
>  	uint32_t mask_flags;		/* master MTD flags to mask out for this partition */

Brian

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH V4 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module
  2017-03-30 12:35     ` [PATCH V4 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module Rafał Miłecki
@ 2017-05-11 18:31       ` Brian Norris
  2017-05-24  9:36         ` Rafał Miłecki
  0 siblings, 1 reply; 16+ messages in thread
From: Brian Norris @ 2017-05-11 18:31 UTC (permalink / raw)
  To: Rafał Miłecki
  Cc: David Woodhouse, Boris Brezillon, Marek Vasut,
	Richard Weinberger, Cyrille Pitchen, linux-mtd,
	Rafał Miłecki

Hi,

On Thu, Mar 30, 2017 at 02:35:27PM +0200, Rafał Miłecki wrote:
> From: Rafał Miłecki <rafal@milecki.pl>
> 
> This makes TRX parsing code reusable with other platforms and parsers.
> 
> Please note this patch doesn't really change anything in the existing
> code, just moves it. There is still some place for improvement (e.g.
> working on non-hacky method of checking rootfs format) but it's not
> really a subject of this change.
> 
> Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
> ---
> V2: A totally rebased & refreshed version.
> ---
>  drivers/mtd/Kconfig              |   4 ++
>  drivers/mtd/Makefile             |   1 +
>  drivers/mtd/bcm47xxpart.c        |  97 +------------------------------
>  drivers/mtd/parsers/Kconfig      |   8 +++
>  drivers/mtd/parsers/Makefile     |   1 +
>  drivers/mtd/parsers/parser_trx.c | 119 +++++++++++++++++++++++++++++++++++++++
>  6 files changed, 136 insertions(+), 94 deletions(-)
>  create mode 100644 drivers/mtd/parsers/Kconfig
>  create mode 100644 drivers/mtd/parsers/Makefile
>  create mode 100644 drivers/mtd/parsers/parser_trx.c
> 
> diff --git a/drivers/mtd/Kconfig b/drivers/mtd/Kconfig
> index e83a279f1217..5a2d71729b9a 100644
> --- a/drivers/mtd/Kconfig
> +++ b/drivers/mtd/Kconfig
> @@ -155,6 +155,10 @@ config MTD_BCM47XX_PARTS
>  	  This provides partitions parser for devices based on BCM47xx
>  	  boards.
>  
> +menu "Partition parsers"
> +source "drivers/mtd/parsers/Kconfig"
> +endmenu

Is "parsers" a better name than "partitions"? I proposed moving
everything to "partitions" previously.

> +
>  comment "User Modules And Translation Layers"
>  
>  #
> diff --git a/drivers/mtd/Makefile b/drivers/mtd/Makefile
> index 99bb9a1f6e16..151d60df303a 100644
> --- a/drivers/mtd/Makefile
> +++ b/drivers/mtd/Makefile
> @@ -13,6 +13,7 @@ obj-$(CONFIG_MTD_AFS_PARTS)	+= afs.o
>  obj-$(CONFIG_MTD_AR7_PARTS)	+= ar7part.o
>  obj-$(CONFIG_MTD_BCM63XX_PARTS)	+= bcm63xxpart.o
>  obj-$(CONFIG_MTD_BCM47XX_PARTS)	+= bcm47xxpart.o
> +obj-y				+= parsers/
>  
>  # 'Users' - code which presents functionality to userspace.
>  obj-$(CONFIG_MTD_BLKDEVS)	+= mtd_blkdevs.o
> diff --git a/drivers/mtd/bcm47xxpart.c b/drivers/mtd/bcm47xxpart.c
> index d10fa6c8f074..66ce72fd1426 100644
> --- a/drivers/mtd/bcm47xxpart.c
> +++ b/drivers/mtd/bcm47xxpart.c
> @@ -43,7 +43,6 @@
>  #define ML_MAGIC2			0x26594131
>  #define TRX_MAGIC			0x30524448
>  #define SHSQ_MAGIC			0x71736873	/* shsq (weird ZTE H218N endianness) */
> -#define UBI_EC_MAGIC			0x23494255	/* UBI# */
>  
>  struct trx_header {
>  	uint32_t magic;
> @@ -62,89 +61,6 @@ static void bcm47xxpart_add_part(struct mtd_partition *part, const char *name,
>  	part->mask_flags = mask_flags;
>  }
>  
> -static const char *bcm47xxpart_trx_data_part_name(struct mtd_info *master,
> -						  size_t offset)
> -{
> -	uint32_t buf;
> -	size_t bytes_read;
> -	int err;
> -
> -	err  = mtd_read(master, offset, sizeof(buf), &bytes_read,
> -			(uint8_t *)&buf);
> -	if (err && !mtd_is_bitflip(err)) {
> -		pr_err("mtd_read error while parsing (offset: 0x%X): %d\n",
> -			offset, err);
> -		goto out_default;
> -	}
> -
> -	if (buf == UBI_EC_MAGIC)
> -		return "ubi";
> -
> -out_default:
> -	return "rootfs";
> -}
> -
> -static int bcm47xxpart_parse_trx(struct mtd_info *master,
> -				 struct mtd_partition *trx,
> -				 struct mtd_partition *parts,
> -				 size_t parts_len)
> -{
> -	struct trx_header header;
> -	size_t bytes_read;
> -	int curr_part = 0;
> -	int i, err;
> -
> -	if (parts_len < 3) {
> -		pr_warn("No enough space to add TRX partitions!\n");
> -		return -ENOMEM;
> -	}
> -
> -	err = mtd_read(master, trx->offset, sizeof(header), &bytes_read,
> -		       (uint8_t *)&header);
> -	if (err && !mtd_is_bitflip(err)) {
> -		pr_err("mtd_read error while reading TRX header: %d\n", err);
> -		return err;
> -	}
> -
> -	i = 0;
> -
> -	/* We have LZMA loader if offset[2] points to sth */
> -	if (header.offset[2]) {
> -		bcm47xxpart_add_part(&parts[curr_part++], "loader",
> -				     trx->offset + header.offset[i], 0);
> -		i++;
> -	}
> -
> -	if (header.offset[i]) {
> -		bcm47xxpart_add_part(&parts[curr_part++], "linux",
> -				     trx->offset + header.offset[i], 0);
> -		i++;
> -	}
> -
> -	if (header.offset[i]) {
> -		size_t offset = trx->offset + header.offset[i];
> -		const char *name = bcm47xxpart_trx_data_part_name(master,
> -								  offset);
> -
> -		bcm47xxpart_add_part(&parts[curr_part++], name, offset, 0);
> -		i++;
> -	}
> -
> -	/*
> -	 * Assume that every partition ends at the beginning of the one it is
> -	 * followed by.
> -	 */
> -	for (i = 0; i < curr_part; i++) {
> -		u64 next_part_offset = (i < curr_part - 1) ?
> -					parts[i + 1].offset :
> -					trx->offset + trx->size;
> -
> -		parts[i].size = next_part_offset - parts[i].offset;
> -	}
> -
> -	return curr_part;
> -}
> -
>  /**
>   * bcm47xxpart_bootpartition - gets index of TRX partition used by bootloader
>   *
> @@ -362,17 +278,10 @@ static int bcm47xxpart_parse(struct mtd_info *master,
>  	for (i = 0; i < trx_num; i++) {
>  		struct mtd_partition *trx = &parts[trx_parts[i]];
>  
> -		if (i == bcm47xxpart_bootpartition()) {
> -			int num_parts;
> -
> -			num_parts = bcm47xxpart_parse_trx(master, trx,
> -							  parts + curr_part,
> -							  BCM47XXPART_MAX_PARTS - curr_part);
> -			if (num_parts > 0)
> -				curr_part += num_parts;
> -		} else {
> +		if (i == bcm47xxpart_bootpartition())
> +			trx->format = "trx";
> +		else
>  			trx->name = "failsafe";
> -		}
>  	}
>  
>  	*pparts = parts;
> diff --git a/drivers/mtd/parsers/Kconfig b/drivers/mtd/parsers/Kconfig
> new file mode 100644
> index 000000000000..ebb697a52698
> --- /dev/null
> +++ b/drivers/mtd/parsers/Kconfig
> @@ -0,0 +1,8 @@
> +config MTD_PARSER_TRX
> +	tristate "Parser for TRX format partitions"
> +	depends on MTD && (BCM47XX || ARCH_BCM_5301X)
> +	help
> +	  TRX is a firmware format used by Broadcom on their devices. It
> +	  may contain up to 3/4 partitions (depending on the version).
> +	  This driver will parse TRX header and report at least two partitions:
> +	  kernel and rootfs.
> diff --git a/drivers/mtd/parsers/Makefile b/drivers/mtd/parsers/Makefile
> new file mode 100644
> index 000000000000..4d9024e0be3b
> --- /dev/null
> +++ b/drivers/mtd/parsers/Makefile
> @@ -0,0 +1 @@
> +obj-$(CONFIG_MTD_PARSER_TRX)		+= parser_trx.o
> diff --git a/drivers/mtd/parsers/parser_trx.c b/drivers/mtd/parsers/parser_trx.c
> new file mode 100644
> index 000000000000..f35e28148808
> --- /dev/null
> +++ b/drivers/mtd/parsers/parser_trx.c
> @@ -0,0 +1,119 @@
> +/*
> + * Parser for TRX format partitions
> + *
> + * Copyright (C) 2012 - 2017 Rafał Miłecki <rafal@milecki.pl>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + */
> +
> +#include <linux/module.h>
> +#include <linux/slab.h>
> +#include <linux/mtd/mtd.h>
> +#include <linux/mtd/partitions.h>
> +
> +#define TRX_PARSER_MAX_PARTS		4
> +
> +/* Magics */
> +#define UBI_EC_MAGIC			0x23494255	/* UBI# */
> +
> +struct trx_header {
> +	uint32_t magic;
> +	uint32_t length;
> +	uint32_t crc32;
> +	uint16_t flags;
> +	uint16_t version;
> +	uint32_t offset[3];
> +} __packed;
> +
> +static const char *parser_trx_data_part_name(struct mtd_info *master,
> +					     size_t offset)
> +{
> +	uint32_t buf;
> +	size_t bytes_read;
> +	int err;
> +
> +	err  = mtd_read(master, offset, sizeof(buf), &bytes_read,
> +			(uint8_t *)&buf);
> +	if (err && !mtd_is_bitflip(err)) {
> +		pr_err("mtd_read error while parsing (offset: 0x%X): %d\n",
> +			offset, err);
> +		goto out_default;
> +	}
> +
> +	if (buf == UBI_EC_MAGIC)
> +		return "ubi";
> +
> +out_default:
> +	return "rootfs";
> +}
> +
> +static int parser_trx_parse(struct mtd_info *mtd,
> +			    const struct mtd_partition **pparts,
> +			    struct mtd_part_parser_data *data)

So, this function doesn't do any validation that this is *actually* a
TRX partition? I think it needs to do *some* level of validation if it's
going to be an independent parser driver like this. See my comments on
your other patch.

(I also can't say that I understand the parent bcm47xxpart format/parser
very well; it really looks like a collection of hacks, so maybe
validation is impossible...)

> +{
> +	struct mtd_partition *parts;
> +	struct mtd_partition *part;
> +	struct trx_header trx;
> +	size_t bytes_read;
> +	uint8_t curr_part = 0, i = 0;
> +	int err;
> +
> +	parts = kzalloc(sizeof(struct mtd_partition) * TRX_PARSER_MAX_PARTS,
> +			GFP_KERNEL);
> +	if (!parts)
> +		return -ENOMEM;
> +
> +	err = mtd_read(mtd, 0, sizeof(trx), &bytes_read, (uint8_t *)&trx);
> +	if (err) {
> +		pr_err("MTD reading error: %d\n", err);
> +		return err;

You're leaking 'parts' here. (smatch noticed this.)

> +	}
> +
> +	/* We have LZMA loader if offset[2] points to sth */

Not that this is new, but I have no idea what "sth" is.

> +	if (trx.offset[2]) {
> +		part = &parts[curr_part++];
> +		part->name = "loader";
> +		part->offset = trx.offset[i];
> +		i++;
> +	}
> +
> +	if (trx.offset[i]) {
> +		part = &parts[curr_part++];
> +		part->name = "linux";
> +		part->offset = trx.offset[i];
> +		i++;
> +	}
> +
> +	if (trx.offset[i]) {
> +		part = &parts[curr_part++];
> +		part->name = parser_trx_data_part_name(mtd, trx.offset[i]);
> +		part->offset = trx.offset[i];
> +		i++;
> +	}
> +
> +	/*
> +	 * Assume that every partition ends at the beginning of the one it is
> +	 * followed by.
> +	 */
> +	for (i = 0; i < curr_part; i++) {
> +		u64 next_part_offset = (i < curr_part - 1) ?
> +				       parts[i + 1].offset : mtd->size;
> +
> +		parts[i].size = next_part_offset - parts[i].offset;
> +	}
> +
> +	*pparts = parts;
> +	return i;
> +};
> +
> +static struct mtd_part_parser mtd_parser_trx = {
> +	.parse_fn = parser_trx_parse,
> +	.name = "trx",
> +};
> +module_mtd_part_parser(mtd_parser_trx);
> +
> +MODULE_LICENSE("GPL v2");
> +MODULE_DESCRIPTION("Parser for TRX format partitions");

Brian

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH V4 1/2] mtd: add support for partition parsers
  2017-05-11 18:21     ` [PATCH V4 1/2] mtd: add support for partition parsers Brian Norris
@ 2017-05-23  8:14       ` Rafał Miłecki
  2017-05-23  9:06         ` Rafał Miłecki
  0 siblings, 1 reply; 16+ messages in thread
From: Rafał Miłecki @ 2017-05-23  8:14 UTC (permalink / raw)
  To: Brian Norris
  Cc: David Woodhouse, Boris Brezillon, Marek Vasut,
	Richard Weinberger, Cyrille Pitchen, linux-mtd,
	Rafał Miłecki

On 05/11/2017 08:21 PM, Brian Norris wrote:
> On Thu, Mar 30, 2017 at 02:35:26PM +0200, Rafał Miłecki wrote:
>> From: Rafał Miłecki <rafal@milecki.pl>
>>
>> Some devices have partitions that are kind of containers with extra
>> subpartitions / volumes instead of e.g. simple filesystem data. To
>> support such cases we need to first create normal flash partitions and
>> then take care of these special ones.
>>
>> It's very common case for home routers. Depending on the vendor there
>> are formats like TRX, Seama, TP-Link, WRGG & more. All of them are used
>> to embed few partitions into a single one / single firmware file.
>>
>> Ideally all vendors would use some well documented / standardized format
>> like UBI (and some probably start doing so), but there are still
>> countless devices on the market using these poor vendor specific
>> formats.
>>
>> This patch extends MTD subsystem by allowing to specify partition format
>> and trying to use a proper parser when needed. Supporting such poor
>> formats is highly unlikely to be the top priority so these changes try
>> to minimize maintenance cost to the minimum. It reuses existing code for
>> these new parsers and just adds a one property and one new function.
>>
>> This implementation requires setting partition format in a flash parser.
>> A proper change of bcm47xxpart will follow and in the future we will
>> hopefully also find a solution for doing it with ofpart
>> ("fixed-partitions").
>>
>> Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
>> Acked-by: Marek Vasut <marek.vasut@gmail.com>
>> ---
>> V2: A totally rebased & refreshed version.
>> V3: Don't mention uImage in commit message, it was a mistake.
>> V4: Document mtd_parse_part parameters
>> ---
>>  drivers/mtd/mtdpart.c          | 41 +++++++++++++++++++++++++++++++++++++++++
>>  include/linux/mtd/partitions.h |  7 +++++++
>>  2 files changed, 48 insertions(+)
>>
>> diff --git a/drivers/mtd/mtdpart.c b/drivers/mtd/mtdpart.c
>> index ea5e5307f667..81e0b80237df 100644
>> --- a/drivers/mtd/mtdpart.c
>> +++ b/drivers/mtd/mtdpart.c
>> @@ -363,6 +363,45 @@ static inline void free_partition(struct mtd_part *p)
>>  	kfree(p);
>>  }
>>
>> +/**
>> + * mtd_parse_part - parse MTD partition with a matching parser
>
> The naming is a bit confusing to me. We already have partition parsers,
> but those are typically expected to apply to the whole device. Is this
> infrastructure the same thing? Or different?
>
> (To answer myself) this is nearly the same infrastructure, but it's just
> for adding *sub*partitions, not top-level partitions. Maybe it would be
> slightly less confusing if this was called mtd_parse_subpartitions() or
> mtd_parse_subparts()?

I'm not native English, but I think the name is accurate.

You use spectrum analyzer to analyze the spectrum. You'd probably call it
foo_analyze_spectum.
Or you use signal analyzer to analyze the signal. You'd probably call it
foo_analyze_signal.

This "mtd_parse_part" function uses parser to parse the partition. What it
*looks for* are subpartitions. Maybe I could extend current "parse MTD
partition with a matching parser" function documentation to be more clear.

The name you posted ("mtd_parse_subparts") suggests this function is
parsing (analyzing) subpartitions looking for something inside them. It's
not exactly the case.


If you prefer to have "subpart" or "subpartitions" phase in the function
name then I suggest to use "mtd_parse_for_subparts".

Let me know what do you think.


>> + *
>> + * @slave: part to be parsed for subpartitions
>> + * @format: partition format used to call a proper parser
>> + *
>> + * Some partitions use a specific format to describe contained subpartitions
>> + * (volumes). This function tries to use a proper parser for a given format and
>> + * registers found (sub)partitions.
>> + */
>> +static int mtd_parse_part(struct mtd_part *slave, const char *format)
>> +{
>> +	struct mtd_partitions parsed;
>> +	const char *probes[2];
>> +	int i;
>> +	int err;
>> +
>> +	probes[0] = format; /* Use parser with name matching the format */
>> +	probes[1] = NULL; /* End of parsers */
>> +	err = parse_mtd_partitions(&slave->mtd, probes, &parsed, NULL);
>> +	if (err)
>> +		return err;
>> +	else if (!parsed.nr_parts)
>> +		return -ENOENT;
>> +
>> +	for (i = 0; i < parsed.nr_parts; i++) {
>> +		struct mtd_partition *part;
>> +
>> +		part = (struct mtd_partition *)&parsed.parts[i];
>
> I'm not super-happy with the de-constification cast here. What if the
> partition array was somehow shared? (I don't expect that, but you're
> still potentially violating a caller's assumptions when you modify their
> 'const' data.)
>
>> +		part->offset += slave->offset;
>> +	}
>> +
>> +	err = add_mtd_partitions(slave->master, parsed.parts, parsed.nr_parts);
>
> Maybe a better way to handle that offset modification is to either pass
> in the offset to add_mtd_partitions() (e.g., as a simple parameter), or
> else adapt add_mtd_partitions() to handle receiving a non-master as the
> first parameter. (Then you just pass "slave", and add_mtd_partitions()
> handle it with something like "if (mtd_is_partition(...))".)
>
> Then I wonder how we want the parenting structure to work; should the
> sub-partition have the "master" as its parent, or the original
> partition? I thought Richard had mentioned some problems with the
> existing parenting structure (with CONFIG_MTD_PARTITIONED_MASTER)
> already, but I don't remember what those were.
>
> Also, if you're "parsing" the slave, but "adding" to the master, then
> the bounds-checking logic in add_mtd_partitions() won't properly apply,
> and you'll be able to create sub-partitions that extend beyond the
> slave, right? If so, then I think (after auditing add_mtd_partitions() a
> little closer, and possibly adjusting some of its comments) you might
> really just want to pass 'slave', not 'slave->master'.

I like this idea!


>> +
>> +	mtd_part_parser_cleanup(&parsed);
>> +
>> +	return err;
>> +}
>> +
>>  /*
>>   * This function unregisters and destroy all slave MTD objects which are
>>   * attached to the given master MTD object.
>> @@ -724,6 +763,8 @@ int add_mtd_partitions(struct mtd_info *master,
>>
>>  		add_mtd_device(&slave->mtd);
>>  		mtd_add_partition_attrs(slave);
>> +		if (parts[i].format)
>> +			mtd_parse_part(slave, parts[i].format);
>>
>>  		cur_offset = slave->offset + slave->mtd.size;
>>  	}
>> diff --git a/include/linux/mtd/partitions.h b/include/linux/mtd/partitions.h
>> index 06df1e06b6e0..2787e76c030f 100644
>> --- a/include/linux/mtd/partitions.h
>> +++ b/include/linux/mtd/partitions.h
>> @@ -20,6 +20,12 @@
>>   *
>>   * For each partition, these fields are available:
>>   * name: string that will be used to label the partition's MTD device.
>> + * format: some partitions can be containers using specific format to describe
>> + *	embedded subpartitions / volumes. E.g. many home routers use "firmware"
>> + *	partition that contains at least kernel and rootfs. In such case an
>> + *	extra parser is needed that will detect these dynamic partitions and
>> + *	report them to the MTD subsystem. This property describes partition
>> + *	format and allows MTD core to call a proper parser.
>>   * size: the partition size; if defined as MTDPART_SIZ_FULL, the partition
>>   * 	will extend to the end of the master MTD device.
>>   * offset: absolute starting position within the master MTD device; if
>> @@ -38,6 +44,7 @@
>>
>>  struct mtd_partition {
>>  	const char *name;		/* identifier string */
>> +	const char *format;		/* partition format */
>
> Why only a single format? Can't this use the same definition as what's
> used already in, e.g., parse_mtd_partitions() and mtd_device_register()?
> i.e., a NULL-terminated string array? Not that the definition ("const
> char *const *types") is perfect (perhaps it could use a struct for
> encapsulation), but it would be trivially easy to expand this to support
> detecting more than one sub-partition type, no?

I think this would actually simplify mtd_parse_part a bit, I like it!

Thanks for the review Brian!

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH V4 1/2] mtd: add support for partition parsers
  2017-05-23  8:14       ` Rafał Miłecki
@ 2017-05-23  9:06         ` Rafał Miłecki
  2017-05-25 20:34           ` Brian Norris
  0 siblings, 1 reply; 16+ messages in thread
From: Rafał Miłecki @ 2017-05-23  9:06 UTC (permalink / raw)
  To: Brian Norris
  Cc: David Woodhouse, Boris Brezillon, Marek Vasut,
	Richard Weinberger, Cyrille Pitchen, linux-mtd,
	Rafał Miłecki

On 05/23/2017 10:14 AM, Rafał Miłecki wrote:
> On 05/11/2017 08:21 PM, Brian Norris wrote:
>> On Thu, Mar 30, 2017 at 02:35:26PM +0200, Rafał Miłecki wrote:
>>> + *
>>> + * @slave: part to be parsed for subpartitions
>>> + * @format: partition format used to call a proper parser
>>> + *
>>> + * Some partitions use a specific format to describe contained subpartitions
>>> + * (volumes). This function tries to use a proper parser for a given format and
>>> + * registers found (sub)partitions.
>>> + */
>>> +static int mtd_parse_part(struct mtd_part *slave, const char *format)
>>> +{
>>> +    struct mtd_partitions parsed;
>>> +    const char *probes[2];
>>> +    int i;
>>> +    int err;
>>> +
>>> +    probes[0] = format; /* Use parser with name matching the format */
>>> +    probes[1] = NULL; /* End of parsers */
>>> +    err = parse_mtd_partitions(&slave->mtd, probes, &parsed, NULL);
>>> +    if (err)
>>> +        return err;
>>> +    else if (!parsed.nr_parts)
>>> +        return -ENOENT;
>>> +
>>> +    for (i = 0; i < parsed.nr_parts; i++) {
>>> +        struct mtd_partition *part;
>>> +
>>> +        part = (struct mtd_partition *)&parsed.parts[i];
>>
>> I'm not super-happy with the de-constification cast here. What if the
>> partition array was somehow shared? (I don't expect that, but you're
>> still potentially violating a caller's assumptions when you modify their
>> 'const' data.)
>>
>>> +        part->offset += slave->offset;
>>> +    }
>>> +
>>> +    err = add_mtd_partitions(slave->master, parsed.parts, parsed.nr_parts);
>>
>> Maybe a better way to handle that offset modification is to either pass
>> in the offset to add_mtd_partitions() (e.g., as a simple parameter), or
>> else adapt add_mtd_partitions() to handle receiving a non-master as the
>> first parameter. (Then you just pass "slave", and add_mtd_partitions()
>> handle it with something like "if (mtd_is_partition(...))".)
>>
>> Then I wonder how we want the parenting structure to work; should the
>> sub-partition have the "master" as its parent, or the original
>> partition? I thought Richard had mentioned some problems with the
>> existing parenting structure (with CONFIG_MTD_PARTITIONED_MASTER)
>> already, but I don't remember what those were.
>>
>> Also, if you're "parsing" the slave, but "adding" to the master, then
>> the bounds-checking logic in add_mtd_partitions() won't properly apply,
>> and you'll be able to create sub-partitions that extend beyond the
>> slave, right? If so, then I think (after auditing add_mtd_partitions() a
>> little closer, and possibly adjusting some of its comments) you might
>> really just want to pass 'slave', not 'slave->master'.
>
> I like this idea!

Oh, it's not that simple. Passing struct mtd_part to the add_mtd_partitions
is simple, but it's getting complex afterwards.

1) We can't simply adjust offset in add_mtd_partitions as we are still
    dealing with const struct mtd_partition (note: const).
2) We can't adjust offset after calling allocate_partition as this would
    bypass some validation happening in the allocate_partition.
3) Passing an extra argume to the allocate_partition is a bad idea as it
    already receives uint64_t cur_offset - this would be confusing.

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH V4 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module
  2017-05-11 18:31       ` Brian Norris
@ 2017-05-24  9:36         ` Rafał Miłecki
  0 siblings, 0 replies; 16+ messages in thread
From: Rafał Miłecki @ 2017-05-24  9:36 UTC (permalink / raw)
  To: Brian Norris
  Cc: David Woodhouse, Boris Brezillon, Marek Vasut,
	Richard Weinberger, Cyrille Pitchen, linux-mtd,
	Rafał Miłecki

On 05/11/2017 08:31 PM, Brian Norris wrote:
> On Thu, Mar 30, 2017 at 02:35:27PM +0200, Rafał Miłecki wrote:
>> From: Rafał Miłecki <rafal@milecki.pl>
>>
>> This makes TRX parsing code reusable with other platforms and parsers.
>>
>> Please note this patch doesn't really change anything in the existing
>> code, just moves it. There is still some place for improvement (e.g.
>> working on non-hacky method of checking rootfs format) but it's not
>> really a subject of this change.
>>
>> Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
>> ---
>> V2: A totally rebased & refreshed version.
>> ---
>>  drivers/mtd/Kconfig              |   4 ++
>>  drivers/mtd/Makefile             |   1 +
>>  drivers/mtd/bcm47xxpart.c        |  97 +------------------------------
>>  drivers/mtd/parsers/Kconfig      |   8 +++
>>  drivers/mtd/parsers/Makefile     |   1 +
>>  drivers/mtd/parsers/parser_trx.c | 119 +++++++++++++++++++++++++++++++++++++++
>>  6 files changed, 136 insertions(+), 94 deletions(-)
>>  create mode 100644 drivers/mtd/parsers/Kconfig
>>  create mode 100644 drivers/mtd/parsers/Makefile
>>  create mode 100644 drivers/mtd/parsers/parser_trx.c
>>
>> diff --git a/drivers/mtd/Kconfig b/drivers/mtd/Kconfig
>> index e83a279f1217..5a2d71729b9a 100644
>> --- a/drivers/mtd/Kconfig
>> +++ b/drivers/mtd/Kconfig
>> @@ -155,6 +155,10 @@ config MTD_BCM47XX_PARTS
>>  	  This provides partitions parser for devices based on BCM47xx
>>  	  boards.
>>
>> +menu "Partition parsers"
>> +source "drivers/mtd/parsers/Kconfig"
>> +endmenu
>
> Is "parsers" a better name than "partitions"? I proposed moving
> everything to "partitions" previously.

I didn't see that or forgot about that, sorry. Since this module contains a
parser, I think "parsers" subdirectory may be a good choice.

Did you plan to put anything else than parsers in the "partitions" subdir?


>> +static const char *parser_trx_data_part_name(struct mtd_info *master,
>> +					     size_t offset)
>> +{
>> +	uint32_t buf;
>> +	size_t bytes_read;
>> +	int err;
>> +
>> +	err  = mtd_read(master, offset, sizeof(buf), &bytes_read,
>> +			(uint8_t *)&buf);
>> +	if (err && !mtd_is_bitflip(err)) {
>> +		pr_err("mtd_read error while parsing (offset: 0x%X): %d\n",
>> +			offset, err);
>> +		goto out_default;
>> +	}
>> +
>> +	if (buf == UBI_EC_MAGIC)
>> +		return "ubi";
>> +
>> +out_default:
>> +	return "rootfs";
>> +}
>> +
>> +static int parser_trx_parse(struct mtd_info *mtd,
>> +			    const struct mtd_partition **pparts,
>> +			    struct mtd_part_parser_data *data)
>
> So, this function doesn't do any validation that this is *actually* a
> TRX partition? I think it needs to do *some* level of validation if it's
> going to be an independent parser driver like this. See my comments on
> your other patch.
>
> (I also can't say that I understand the parent bcm47xxpart format/parser
> very well; it really looks like a collection of hacks, so maybe
> validation is impossible...)

It is possible, I'll add it!


>> +{
>> +	struct mtd_partition *parts;
>> +	struct mtd_partition *part;
>> +	struct trx_header trx;
>> +	size_t bytes_read;
>> +	uint8_t curr_part = 0, i = 0;
>> +	int err;
>> +
>> +	parts = kzalloc(sizeof(struct mtd_partition) * TRX_PARSER_MAX_PARTS,
>> +			GFP_KERNEL);
>> +	if (!parts)
>> +		return -ENOMEM;
>> +
>> +	err = mtd_read(mtd, 0, sizeof(trx), &bytes_read, (uint8_t *)&trx);
>> +	if (err) {
>> +		pr_err("MTD reading error: %d\n", err);
>> +		return err;
>
> You're leaking 'parts' here. (smatch noticed this.)

Right, thanks.


>> +	}
>> +
>> +	/* We have LZMA loader if offset[2] points to sth */
>
> Not that this is new, but I have no idea what "sth" is.

I'll improve this comment.

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH V4 1/2] mtd: add support for partition parsers
  2017-05-23  9:06         ` Rafał Miłecki
@ 2017-05-25 20:34           ` Brian Norris
  0 siblings, 0 replies; 16+ messages in thread
From: Brian Norris @ 2017-05-25 20:34 UTC (permalink / raw)
  To: Rafał Miłecki
  Cc: David Woodhouse, Boris Brezillon, Marek Vasut,
	Richard Weinberger, Cyrille Pitchen, linux-mtd,
	Rafał Miłecki

On Tue, May 23, 2017 at 11:06:37AM +0200, Rafał Miłecki wrote:
> On 05/23/2017 10:14 AM, Rafał Miłecki wrote:
> >On 05/11/2017 08:21 PM, Brian Norris wrote:
> >>Maybe a better way to handle that offset modification is to either pass
> >>in the offset to add_mtd_partitions() (e.g., as a simple parameter), or
> >>else adapt add_mtd_partitions() to handle receiving a non-master as the
> >>first parameter. (Then you just pass "slave", and add_mtd_partitions()
> >>handle it with something like "if (mtd_is_partition(...))".)
> >>
> >>Then I wonder how we want the parenting structure to work; should the
> >>sub-partition have the "master" as its parent, or the original
> >>partition? I thought Richard had mentioned some problems with the
> >>existing parenting structure (with CONFIG_MTD_PARTITIONED_MASTER)
> >>already, but I don't remember what those were.
> >>
> >>Also, if you're "parsing" the slave, but "adding" to the master, then
> >>the bounds-checking logic in add_mtd_partitions() won't properly apply,
> >>and you'll be able to create sub-partitions that extend beyond the
> >>slave, right? If so, then I think (after auditing add_mtd_partitions() a
> >>little closer, and possibly adjusting some of its comments) you might
> >>really just want to pass 'slave', not 'slave->master'.
> >
> >I like this idea!
> 
> Oh, it's not that simple. Passing struct mtd_part to the add_mtd_partitions
> is simple, but it's getting complex afterwards.
> 
> 1) We can't simply adjust offset in add_mtd_partitions as we are still
>    dealing with const struct mtd_partition (note: const).
> 2) We can't adjust offset after calling allocate_partition as this would
>    bypass some validation happening in the allocate_partition.

I think I was assuming you'd use a tree structure, in which case you
don't have to modify the offsets at all. The offsets would get
translated by, e.g., a recursive call to part_read() (the subpartition's
part_read() would call the parent partition's part_read(), which would
adjust the offset and call the master's ->read() callback).

Or maybe the recursion (and tree structure) is not a great idea. You
came up with a mostly-OK solution that keeps a flat tree structure and
no recursion in your next version.

> 3) Passing an extra argume to the allocate_partition is a bad idea as it
>    already receives uint64_t cur_offset - this would be confusing.

Yeah, might be. But I think you came up with a different solution
anyway.

Brian

^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2017-05-25 20:34 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-02-21 12:40 [PATCH V2 1/2] mtd: add support for partition parsers Rafał Miłecki
2017-02-21 12:40 ` [PATCH V2 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module Rafał Miłecki
2017-02-27  9:36 ` [PATCH V2 1/2] mtd: add support for partition parsers Marek Vasut
2017-02-27  9:51   ` Rafał Miłecki
2017-02-27 13:06 ` [PATCH V3 " Rafał Miłecki
2017-02-27 13:06   ` [PATCH V3 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module Rafał Miłecki
2017-03-30 10:31   ` [PATCH V3 1/2] mtd: add support for partition parsers Rafał Miłecki
2017-03-30 11:13   ` Marek Vasut
2017-03-30 12:35   ` [PATCH V4 " Rafał Miłecki
2017-03-30 12:35     ` [PATCH V4 2/2] mtd: extract TRX parser out of bcm47xxpart into a separated module Rafał Miłecki
2017-05-11 18:31       ` Brian Norris
2017-05-24  9:36         ` Rafał Miłecki
2017-05-11 18:21     ` [PATCH V4 1/2] mtd: add support for partition parsers Brian Norris
2017-05-23  8:14       ` Rafał Miłecki
2017-05-23  9:06         ` Rafał Miłecki
2017-05-25 20:34           ` Brian Norris

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.