[PATCH RFC v2] ubi-utils: Introduce a utility ubidump

[PATCH RFC v2] ubi-utils: Introduce a utility ubidump

[hujianyang]

Hi all,

After discussing with Artem, I've applied some of his ideas and
rewrite this utility.

Bill also gives me help. I reference his code about getting data
from MTD device.

Thank you all~!

v1:
http://lists.infradead.org/pipermail/linux-mtd/2014-July/054541.html

Changes in v2:
 - modified ioctl() in ubi_vol which just return pnum now.
 - change libubifs.c/libubifs.h into libdump.c/libdump.h and
   remove ubi-level header dump function into libdump.c.
 - read data in eraseblock from MTD functionality.
 - add a new function "ubidump" to handle peb dumping.
 - modified options -H/-N into --ubi/--ubifs.

I found this utility can't dump lpt area correctly because we use
different structure to format this area. I will enable it soon. I
think we should confirm the mechanism of this utility first.

[PATCH RFC v2] UBI: New ioctl() to support ubidump

[hujianyang]

An ioctl() return pnum of a specified leb.


Signed-off-by: hujianyang <hujianyang@huawei.com>
---
 drivers/mtd/ubi/cdev.c      | 26 ++++++++++++++++++++++++++
 include/uapi/mtd/ubi-user.h | 12 ++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/drivers/mtd/ubi/cdev.c b/drivers/mtd/ubi/cdev.c
index 7646220..6fa7346 100644
--- a/drivers/mtd/ubi/cdev.c
+++ b/drivers/mtd/ubi/cdev.c
@@ -581,6 +581,32 @@ static long vol_cdev_ioctl(struct file *file, unsigned int cmd,
 		break;
 	}

+	/* Get pnum of a specified leb command */
+	case UBI_IOCEBGETPNUM:
+	{
+		struct ubi_lnum2pnum_req req;
+		int pnum;
+
+		err = copy_from_user(&req, argp,
+				sizeof(struct ubi_lnum2pnum_req));
+		if (err) {
+			err = -EFAULT;
+			break;
+		}
+
+		err = ubi_is_mapped(desc, req.lnum);
+		if (err <= 0)
+			break;
+		pnum = vol->eba_tbl[req.lnum];
+		req.pnum = pnum;
+
+		err = copy_to_user(argp, &req,
+				sizeof(struct ubi_lnum2pnum_req));
+		if (err)
+			err = -EFAULT;
+		break;
+	}
+
 	default:
 		err = -ENOTTY;
 		break;
diff --git a/include/uapi/mtd/ubi-user.h b/include/uapi/mtd/ubi-user.h
index 1927b0d..fc41ddb 100644
--- a/include/uapi/mtd/ubi-user.h
+++ b/include/uapi/mtd/ubi-user.h
@@ -205,6 +205,8 @@
 #define UBI_IOCVOLCRBLK _IOW(UBI_VOL_IOC_MAGIC, 7, struct ubi_blkcreate_req)
 /* Remove the R/O block device */
 #define UBI_IOCVOLRMBLK _IO(UBI_VOL_IOC_MAGIC, 8)
+/* Get pnum of a specified leb */
+#define UBI_IOCEBGETPNUM _IOW(UBI_VOL_IOC_MAGIC, 9, struct ubi_lnum2pnum_req)

 /* Maximum MTD device name length supported by UBI */
 #define MAX_UBI_MTD_NAME_LEN 127
@@ -442,4 +444,14 @@ struct ubi_blkcreate_req {
 	__s8  padding[128];
 }  __packed;

+/**
+ * struct ubi_lnum2pnum_req - a data structure used in lnum translate requests.
+ * @lnum: logical eraseblock num to translate
+ * @pnum: physical eraseblock num @lnum mapped
+ */
+struct ubi_lnum2pnum_req {
+	__s32 lnum;
+	__s32 pnum;
+}  __packed;
+
 #endif /* __UBI_USER_H__ */
-- 
1.8.1.4

[PATCH 1/5] ubi-utils: Add libdump files

[hujianyang]

I've changed the name of libubifs.c/libubifs.h into libdump.c/h
and move ubi-level header dump functions into libdump.c.


Signed-off-by: hujianyang <hujianyang@huawei.com>
---
 ubi-utils/include/libdump.h |  72 +++++++
 ubi-utils/libdump.c         | 473 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 545 insertions(+)
 create mode 100644 ubi-utils/include/libdump.h
 create mode 100644 ubi-utils/libdump.c

diff --git a/ubi-utils/include/libdump.h b/ubi-utils/include/libdump.h
new file mode 100644
index 0000000..89a50d7
--- /dev/null
+++ b/ubi-utils/include/libdump.h
@@ -0,0 +1,72 @@
+/*
+ * Copyright (c) International Business Machines Corp., 2006
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
+ * the GNU General Public License for more details.
+ *
+ * Author: Hu Jianyang <hujianyang@huawei.com>
+ *
+ * UBIFS library.
+ */
+
+#ifndef __LIBDUMP_H__
+#define __LIBDUMP_H__
+
+#include <mtd/ubifs-media.h>
+#include <mtd/ubi-media.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#define ALIGN(x,a) __ALIGN_MASK(x,(typeof(x))(a)-1)
+#define __ALIGN_MASK(x,mask) (((x)+(mask))&~(mask))
+
+#define min_t(t,x,y) ({ \
+	typeof((x)) _x = (x); \
+	typeof((y)) _y = (y); \
+	(_x < _y) ? _x : _y; \
+})
+
+/**
+ * ubidump - dump ubi/ubifs information
+ * @buf: buffer to dump
+ * @peb_size: size of buffer/MTD eraseblock size
+ * @ubi: dump ubi-level stuff
+ * @ubifs: dump ubifs-level stuff
+ *
+ * This function dump ubi/ubifs information on the buffer.
+ */
+void ubidump(void *buf, int peb_size, int ubi, int ubifs, int detailed);
+
+/*
+ * 'ubifs_scan_a_node()' return values.
+ *
+ * SCANNED_GARBAGE:  scanned garbage
+ * SCANNED_EMPTY_SPACE: scanned empty space
+ * SCANNED_A_NODE: scanned a valid node
+ * SCANNED_A_CORRUPT_NODE: scanned a corrupted node
+ * SCANNED_A_BAD_PAD_NODE: scanned a padding node with invalid pad length
+ *
+ * Greater than zero means: 'scanned that number of padding bytes'
+ */
+enum {
+	SCANNED_GARBAGE        = 0,
+	SCANNED_EMPTY_SPACE    = -1,
+	SCANNED_A_NODE         = -2,
+	SCANNED_A_CORRUPT_NODE = -3,
+	SCANNED_A_BAD_PAD_NODE = -4,
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /*!__LIBDUMP_H__ */
diff --git a/ubi-utils/libdump.c b/ubi-utils/libdump.c
new file mode 100644
index 0000000..d077859
--- /dev/null
+++ b/ubi-utils/libdump.c
@@ -0,0 +1,473 @@
+/*
+ * Copyright (c) International Business Machines Corp., 2006
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
+ * the GNU General Public License for more details.
+ *
+ * Author: Hu Jianyang <hujianyang@huawei.com>
+ *
+ * UBIFS library.
+ */
+
+#define PROGRAM_NAME "libdump"
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <libdump.h>
+#include <mtd_swab.h>
+#include "common.h"
+
+static const char *get_key_fmt(int fmt)
+{
+	switch (fmt) {
+	case UBIFS_SIMPLE_KEY_FMT:
+		return "simple";
+	default:
+		return "unknown/invalid format";
+	}
+}
+
+static const char *get_key_hash(int hash)
+{
+	switch (hash) {
+	case UBIFS_KEY_HASH_R5:
+		return "R5";
+	case UBIFS_KEY_HASH_TEST:
+		return "test";
+	default:
+		return "unknown/invalid name hash";
+	}
+}
+
+static const char *node_ntype(int type)
+{
+	switch (type) {
+	case UBIFS_PAD_NODE:
+		return "padding node";
+	case UBIFS_SB_NODE:
+		return "superblock node";
+	case UBIFS_MST_NODE:
+		return "master node";
+	case UBIFS_REF_NODE:
+		return "reference node";
+	case UBIFS_INO_NODE:
+		return "inode node";
+	case UBIFS_DENT_NODE:
+		return "direntry node";
+	case UBIFS_XENT_NODE:
+		return "xentry node";
+	case UBIFS_DATA_NODE:
+		return "data node";
+	case UBIFS_TRUN_NODE:
+		return "truncate node";
+	case UBIFS_IDX_NODE:
+		return "indexing node";
+	case UBIFS_CS_NODE:
+		return "commit start node";
+	case UBIFS_ORPH_NODE:
+		return "orphan node";
+	default:
+		return "unknown node";
+	}
+}
+
+static const char *node_gtype(int type)
+{
+	switch (type) {
+	case UBIFS_NO_NODE_GROUP:
+		return "no node group";
+	case UBIFS_IN_NODE_GROUP:
+		return "in node group";
+	case UBIFS_LAST_OF_NODE_GROUP:
+		return "last of node group";
+	default:
+		return "unknown";
+	}
+}
+
+static void dump_ch(const struct ubifs_ch *ch)
+{
+	printf("\tmagic          %#x\n", le32_to_cpu(ch->magic));
+	printf("\tcrc            %#x\n", le32_to_cpu(ch->crc));
+	printf("\tnode_type      %d (%s)\n", ch->node_type,
+	       node_ntype(ch->node_type));
+	printf("\tgroup_type     %d (%s)\n", ch->group_type,
+	       node_gtype(ch->group_type));
+	printf("\tsqnum          %llu\n",
+	       (unsigned long long)le64_to_cpu(ch->sqnum));
+	printf("\tlen            %u\n", le32_to_cpu(ch->len));
+}
+
+static void ubifs_dump_node(const void *node)
+{
+	const struct ubifs_ch *ch = node;
+
+	if (le32_to_cpu(ch->magic) != UBIFS_NODE_MAGIC) {
+		printf("Not a node, first %zu bytes:\n", UBIFS_CH_SZ);
+		return;
+	}
+
+	dump_ch(node);
+
+	switch (ch->node_type) {
+	case UBIFS_PAD_NODE:
+	{
+		const struct ubifs_pad_node *pad = node;
+
+		printf("\tpad_len        %u\n", le32_to_cpu(pad->pad_len));
+		break;
+	}
+	case UBIFS_SB_NODE:
+	{
+		const struct ubifs_sb_node *sup = node;
+		unsigned int sup_flags = le32_to_cpu(sup->flags);
+
+		printf("\tkey_hash       %d (%s)\n",
+		       (int)sup->key_hash, get_key_hash(sup->key_hash));
+		printf("\tkey_fmt        %d (%s)\n",
+		       (int)sup->key_fmt, get_key_fmt(sup->key_fmt));
+		printf("\tflags          %#x\n", sup_flags);
+		printf("\tbig_lpt        %u\n",
+		       !!(sup_flags & UBIFS_FLG_BIGLPT));
+		printf("\tspace_fixup    %u\n",
+		       !!(sup_flags & UBIFS_FLG_SPACE_FIXUP));
+		printf("\tmin_io_size    %u\n", le32_to_cpu(sup->min_io_size));
+		printf("\tleb_size       %u\n", le32_to_cpu(sup->leb_size));
+		printf("\tleb_cnt        %u\n", le32_to_cpu(sup->leb_cnt));
+		printf("\tmax_leb_cnt    %u\n", le32_to_cpu(sup->max_leb_cnt));
+		printf("\tmax_bud_bytes  %llu\n",
+		       (unsigned long long)le64_to_cpu(sup->max_bud_bytes));
+		printf("\tlog_lebs       %u\n", le32_to_cpu(sup->log_lebs));
+		printf("\tlpt_lebs       %u\n", le32_to_cpu(sup->lpt_lebs));
+		printf("\torph_lebs      %u\n", le32_to_cpu(sup->orph_lebs));
+		printf("\tjhead_cnt      %u\n", le32_to_cpu(sup->jhead_cnt));
+		printf("\tfanout         %u\n", le32_to_cpu(sup->fanout));
+		printf("\tlsave_cnt      %u\n", le32_to_cpu(sup->lsave_cnt));
+		printf("\tdefault_compr  %u\n",
+		       (int)le16_to_cpu(sup->default_compr));
+		printf("\trp_size        %llu\n",
+		       (unsigned long long)le64_to_cpu(sup->rp_size));
+		printf("\trp_uid         %u\n", le32_to_cpu(sup->rp_uid));
+		printf("\trp_gid         %u\n", le32_to_cpu(sup->rp_gid));
+		printf("\tfmt_version    %u\n", le32_to_cpu(sup->fmt_version));
+		printf("\ttime_gran      %u\n", le32_to_cpu(sup->time_gran));
+		printf("\tUUID           %pUB\n", sup->uuid);
+		break;
+	}
+	case UBIFS_MST_NODE:
+	{
+		const struct ubifs_mst_node *mst = node;
+
+		printf("\thighest_inum   %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->highest_inum));
+		printf("\tcommit number  %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->cmt_no));
+		printf("\tflags          %#x\n", le32_to_cpu(mst->flags));
+		printf("\tlog_lnum       %u\n", le32_to_cpu(mst->log_lnum));
+		printf("\troot_lnum      %u\n", le32_to_cpu(mst->root_lnum));
+		printf("\troot_offs      %u\n", le32_to_cpu(mst->root_offs));
+		printf("\troot_len       %u\n", le32_to_cpu(mst->root_len));
+		printf("\tgc_lnum        %u\n", le32_to_cpu(mst->gc_lnum));
+		printf("\tihead_lnum     %u\n", le32_to_cpu(mst->ihead_lnum));
+		printf("\tihead_offs     %u\n", le32_to_cpu(mst->ihead_offs));
+		printf("\tindex_size     %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->index_size));
+		printf("\tlpt_lnum       %u\n", le32_to_cpu(mst->lpt_lnum));
+		printf("\tlpt_offs       %u\n", le32_to_cpu(mst->lpt_offs));
+		printf("\tnhead_lnum     %u\n", le32_to_cpu(mst->nhead_lnum));
+		printf("\tnhead_offs     %u\n", le32_to_cpu(mst->nhead_offs));
+		printf("\tltab_lnum      %u\n", le32_to_cpu(mst->ltab_lnum));
+		printf("\tltab_offs      %u\n", le32_to_cpu(mst->ltab_offs));
+		printf("\tlsave_lnum     %u\n", le32_to_cpu(mst->lsave_lnum));
+		printf("\tlsave_offs     %u\n", le32_to_cpu(mst->lsave_offs));
+		printf("\tlscan_lnum     %u\n", le32_to_cpu(mst->lscan_lnum));
+		printf("\tleb_cnt        %u\n", le32_to_cpu(mst->leb_cnt));
+		printf("\tempty_lebs     %u\n", le32_to_cpu(mst->empty_lebs));
+		printf("\tidx_lebs       %u\n", le32_to_cpu(mst->idx_lebs));
+		printf("\ttotal_free     %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->total_free));
+		printf("\ttotal_dirty    %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->total_dirty));
+		printf("\ttotal_used     %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->total_used));
+		printf("\ttotal_dead     %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->total_dead));
+		printf("\ttotal_dark     %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->total_dark));
+		break;
+	}
+	case UBIFS_REF_NODE:
+	{
+		const struct ubifs_ref_node *ref = node;
+
+		printf("\tlnum           %u\n", le32_to_cpu(ref->lnum));
+		printf("\toffs           %u\n", le32_to_cpu(ref->offs));
+		printf("\tjhead          %u\n", le32_to_cpu(ref->jhead));
+		break;
+	}
+	case UBIFS_CS_NODE:
+		break;
+	case UBIFS_INO_NODE:
+	case UBIFS_DENT_NODE:
+	case UBIFS_XENT_NODE:
+	case UBIFS_DATA_NODE:
+	case UBIFS_TRUN_NODE:
+	case UBIFS_IDX_NODE:
+	case UBIFS_ORPH_NODE:
+		printf("cannot dump node, type not support\n");
+	default:
+		printf("node type %d was not recognized\n",
+		       (int)ch->node_type);
+	}
+}
+
+/**
+ * scan_padding_bytes - scan for padding bytes.
+ * @buf: buffer to scan
+ * @len: length of buffer
+ *
+ * This function returns the number of padding bytes on success and
+ * %SCANNED_GARBAGE on failure.
+ */
+static int scan_padding_bytes(void *buf, int len)
+{
+	int pad_len = 0, max_pad_len = min_t(int, UBIFS_PAD_NODE_SZ, len);
+	uint8_t *p = buf;
+
+	printf("not a node\n");
+
+	while (pad_len < max_pad_len && *p++ == UBIFS_PADDING_BYTE)
+		pad_len += 1;
+
+	if (!pad_len || (pad_len & 7))
+		return SCANNED_GARBAGE;
+
+	printf("%d padding bytes\n", pad_len);
+
+	return pad_len;
+}
+
+/**
+ * ubifs_scan_a_node - scan for a node or padding.
+ * @buf: buffer to scan
+ * @size: logical eraseblock size
+ * @len: length of buffer
+ * @lnum: logical eraseblock number
+ * @offs: offset within the logical eraseblock
+ *
+ * This function returns a scanning code to indicate what was scanned.
+ */
+static int ubifs_scan_a_node(void *buf, int leb_size, int len, int lnum, int offs)
+{
+	struct ubifs_ch *ch = buf;
+	uint32_t magic;
+
+	magic = le32_to_cpu(ch->magic);
+
+	if (magic == 0xFFFFFFFF) {
+		printf("hit empty space at LEB %d:%d\n", lnum, offs);
+		return SCANNED_EMPTY_SPACE;
+	}
+
+	if (magic != UBIFS_NODE_MAGIC)
+		return scan_padding_bytes(buf, len);
+
+	if (len < UBIFS_CH_SZ)
+		return SCANNED_GARBAGE;
+
+	printf("scanning %s at LEB %d:%d\n",
+	       node_ntype(ch->node_type), lnum, offs);
+
+	/* No ubifs_check_nodei() perform here */
+
+	if (ch->node_type == UBIFS_PAD_NODE) {
+		struct ubifs_pad_node *pad = buf;
+		int pad_len = le32_to_cpu(pad->pad_len);
+		int node_len = le32_to_cpu(ch->len);
+
+		/* Validate the padding node */
+		if (pad_len < 0 ||
+		    offs + node_len + pad_len > leb_size) {
+			printf("bad pad node at LEB %d:%d\n", lnum, offs);
+			ubifs_dump_node(pad);
+			return SCANNED_A_BAD_PAD_NODE;
+		}
+
+		/* Make the node pads to 8-byte boundary */
+		if ((node_len + pad_len) & 7) {
+			printf("bad padding length %d - %d\n",
+			       offs, offs + node_len + pad_len);
+			return SCANNED_A_BAD_PAD_NODE;
+		}
+
+		printf("%d bytes padded at LEB %d:%d, offset now %d\n", pad_len,
+			lnum, offs, ALIGN(offs + node_len + pad_len, 8));
+
+		return node_len + pad_len;
+	}
+
+	return SCANNED_A_NODE;
+}
+
+/**
+ * ubifs_scan - scan a logical eraseblock.
+ * @lnum: logical eraseblock number
+ * @sbuf: scan buffer
+ * @size: size of @buf in bytes
+ * @offs: offset to start at (usually zero)
+ * @detailed: print NODEs detailed info
+ *
+ * This function scans LEB and prints complete information about
+ * its contents.
+ */
+static void ubifs_scan(int lnum, void *sbuf, int leb_size, int offs, int detailed)
+{
+	void *buf = sbuf + offs;
+	int err, len = leb_size - offs;
+
+	printf("scan LEB %d:%d\n", lnum, offs);
+
+	while (len >= 8) {
+		struct ubifs_ch *ch = buf;
+		int node_len, ret;
+
+		printf("look at LEB %d:%d (%d bytes left)\n",
+		       lnum, offs, len);
+
+		ret = ubifs_scan_a_node(buf, leb_size, len, lnum, offs);
+		if (ret > 0) {
+			/* Padding bytes or a valid padding node */
+			offs += ret;
+			buf += ret;
+			len -= ret;
+			continue;
+		}
+
+		if (ret == SCANNED_EMPTY_SPACE)
+			/* Empty space is checked later */
+			break;
+
+		switch (ret) {
+		case SCANNED_GARBAGE:
+			printf("garbage\n");
+			goto corrupted;
+		case SCANNED_A_NODE:
+			if (detailed)
+				ubifs_dump_node(buf);
+			break;
+		case SCANNED_A_CORRUPT_NODE:
+		case SCANNED_A_BAD_PAD_NODE:
+			printf("bad node\n");
+			goto corrupted;
+		default:
+			printf("unknown\n");
+			err = -EINVAL;
+			goto error;
+		}
+
+		node_len = ALIGN(le32_to_cpu(ch->len), 8);
+		offs += node_len;
+		buf += node_len;
+		len -= node_len;
+	}
+
+	for (; len > 4; offs += 4, buf = buf + 4, len -= 4)
+		if (*(uint32_t *)buf != 0xffffffff)
+			break;
+	for (; len; offs++, buf++, len--)
+		if (*(uint8_t *)buf != 0xff) {
+			printf("corrupt empty space at LEB %d:%d\n",
+			       lnum, offs);
+			goto corrupted;
+		}
+
+	printf("stop scanning LEB %d at offset %d\n", lnum, offs);
+	return;
+
+corrupted:
+	printf("corruption at LEB %d:%d\n", lnum, offs);
+	err = -EUCLEAN;
+error:
+	printf("LEB %d scanning failed, error %d\n", lnum, err);
+}
+
+/**
+ * ubi_dump_ec_hdr - dump an erase counter header.
+ * @ec_hdr: the erase counter header to dump
+ */
+static void ubi_dump_ec_hdr(const struct ubi_ec_hdr *ec_hdr)
+{
+	printf("Erase counter header dump:\n");
+	printf("\tmagic          %#08x\n", be32_to_cpu(ec_hdr->magic));
+	printf("\tversion        %d\n", (int)ec_hdr->version);
+	printf("\tec             %llu\n", (long long)be64_to_cpu(ec_hdr->ec));
+	printf("\tvid_hdr_offset %d\n", be32_to_cpu(ec_hdr->vid_hdr_offset));
+	printf("\tdata_offset    %d\n", be32_to_cpu(ec_hdr->data_offset));
+	printf("\timage_seq      %d\n", be32_to_cpu(ec_hdr->image_seq));
+	printf("\thdr_crc        %#08x\n", be32_to_cpu(ec_hdr->hdr_crc));
+}
+
+/**
+ * ubi_dump_vid_hdr - dump a volume identifier header.
+ * @vid_hdr: the volume identifier header to dump
+ */
+static void ubi_dump_vid_hdr(const struct ubi_vid_hdr *vid_hdr)
+{
+	printf("Volume identifier header dump:\n");
+	printf("\tmagic     %08x\n", be32_to_cpu(vid_hdr->magic));
+	printf("\tversion   %d\n",  (int)vid_hdr->version);
+	printf("\tvol_type  %d\n",  (int)vid_hdr->vol_type);
+	printf("\tcopy_flag %d\n",  (int)vid_hdr->copy_flag);
+	printf("\tcompat    %d\n",  (int)vid_hdr->compat);
+	printf("\tvol_id    %d\n",  be32_to_cpu(vid_hdr->vol_id));
+	printf("\tlnum      %d\n",  be32_to_cpu(vid_hdr->lnum));
+	printf("\tdata_size %d\n",  be32_to_cpu(vid_hdr->data_size));
+	printf("\tused_ebs  %d\n",  be32_to_cpu(vid_hdr->used_ebs));
+	printf("\tdata_pad  %d\n",  be32_to_cpu(vid_hdr->data_pad));
+	printf("\tsqnum     %llu\n",
+		(unsigned long long)be64_to_cpu(vid_hdr->sqnum));
+	printf("\thdr_crc   %08x\n", be32_to_cpu(vid_hdr->hdr_crc));
+}
+
+/**
+ * ubidump - dump ubi/ubifs information
+ * @buf: buffer to dump
+ * @peb_size: size of buffer/MTD eraseblock size
+ * @ubi: dump ubi-level stuff
+ * @ubifs: dump ubifs-level stuff
+ *
+ * This function dump ubi/ubifs information on the buffer.
+ */
+void ubidump(void *buf, int peb_size, int ubi, int ubifs, int detailed)
+{
+	int vid_hdr_offset, data_offset, lnum;
+	struct ubi_ec_hdr *ec;
+	struct ubi_vid_hdr *vid;
+
+	ec = (struct ubi_ec_hdr *)buf;
+	if (be32_to_cpu(ec->magic) != UBI_EC_HDR_MAGIC)
+		printf("magic number error: %d, expect %d\n",
+		       be32_to_cpu(ec->magic), UBI_EC_HDR_MAGIC);
+
+	vid_hdr_offset = be32_to_cpu(ec->vid_hdr_offset);
+	data_offset = be32_to_cpu(ec->data_offset);
+
+	vid = (struct ubi_vid_hdr *)(buf + vid_hdr_offset);
+	lnum = be32_to_cpu(vid->lnum);
+
+	if (ubi) {
+		ubi_dump_ec_hdr(ec);
+		ubi_dump_vid_hdr(vid);
+	}
+
+	if (ubifs)
+		ubifs_scan(lnum, buf + data_offset, peb_size - data_offset,
+			   0, detailed);
+}
-- 
1.8.1.4

[PATCH 2/5] ubi-utils: Enable lnum2pnum ioctl in userspace

[hujianyang]

Introduce ubi_lnum_to_pnum() to translate lnum into pnum by the
a new ioctl.


Signed-off-by: hujianyang <hujianyang@huawei.com>
---
 include/mtd/ubi-user.h     | 12 ++++++++++++
 ubi-utils/include/libubi.h | 11 +++++++++++
 ubi-utils/libubi.c         | 12 ++++++++++++
 3 files changed, 35 insertions(+)

diff --git a/include/mtd/ubi-user.h b/include/mtd/ubi-user.h
index 2b50dad..e4f57c6 100644
--- a/include/mtd/ubi-user.h
+++ b/include/mtd/ubi-user.h
@@ -200,6 +200,8 @@
 #define UBI_IOCVOLCRBLK _IOW(UBI_VOL_IOC_MAGIC, 7, struct ubi_blkcreate_req)
 /* Remove the R/O block device */
 #define UBI_IOCVOLRMBLK _IO(UBI_VOL_IOC_MAGIC, 8)
+/* Get pnum of a specified leb */
+#define UBI_IOCEBGETPNUM _IOW(UBI_VOL_IOC_MAGIC, 9, struct ubi_lnum2pnum_req)

 /* Maximum MTD device name length supported by UBI */
 #define MAX_UBI_MTD_NAME_LEN 127
@@ -437,4 +439,14 @@ struct ubi_blkcreate_req {
 	int8_t  padding[128];
 }  __attribute__((packed));

+/**
+ * struct ubi_lnum2pnum_req - a data structure used in lnum translate requests.
+ * @lnum: logical eraseblock num to translate
+ * @pnum: physical eraseblock num @lnum mapped
+ */
+struct ubi_lnum2pnum_req {
+	int32_t lnum;
+	int32_t pnum;
+}  __attribute__((packed));
+
 #endif /* __UBI_USER_H__ */
diff --git a/ubi-utils/include/libubi.h b/ubi-utils/include/libubi.h
index 4d6a7ee..618a6ba 100644
--- a/ubi-utils/include/libubi.h
+++ b/ubi-utils/include/libubi.h
@@ -478,6 +478,17 @@ int ubi_leb_unmap(int fd, int lnum);
  */
 int ubi_is_mapped(int fd, int lnum);

+/**
+ * ubi_lnum_to_pnum - get the pnum of a specified leb.
+ * @fd: volume character device file descriptor
+ * @lnum: logical eraseblock number to translate
+ * @pnum: physical eraseblock number to return
+ *
+ * This function return %0 in case of success and %-1 in case of failure. The
+ * pnum of the specified leb is returned by @pnum.
+ */
+int ubi_lnum_to_pnum(int fd, int lnum, int *pnum);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/ubi-utils/libubi.c b/ubi-utils/libubi.c
index 1e08b7d..9650b6a 100644
--- a/ubi-utils/libubi.c
+++ b/ubi-utils/libubi.c
@@ -1123,6 +1123,18 @@ int ubi_vol_block_remove(int fd)
 	return ioctl(fd, UBI_IOCVOLRMBLK);
 }

+int ubi_lnum_to_pnum(int fd, int lnum, int *pnum)
+{
+	struct ubi_lnum2pnum_req request;
+
+	request.pnum = -1;
+	request.lnum = lnum;
+	if (ioctl(fd, UBI_IOCEBGETPNUM, &request))
+		return -1;
+	*pnum = request.pnum;
+	return 0;
+}
+
 int ubi_update_start(libubi_t desc, int fd, long long bytes)
 {
 	desc = desc;
-- 
1.8.1.4

[PATCH 3/5] ubi-utils: Introduce ubidump

[hujianyang]

Read data by MTD functionality now. Also move some dump code to
libdump.

rename options:
    -n   --->  -l
    -N   --->  --ubifs
    -H   --->  --ubi


Signed-off-by: hujianyang <hujianyang@huawei.com>
---
 ubi-utils/ubidump.c | 293 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 293 insertions(+)
 create mode 100644 ubi-utils/ubidump.c

diff --git a/ubi-utils/ubidump.c b/ubi-utils/ubidump.c
new file mode 100644
index 0000000..95d055b
--- /dev/null
+++ b/ubi-utils/ubidump.c
@@ -0,0 +1,293 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
+ * the GNU General Public License for more details.
+ */
+
+/*
+ * An utility to dump UBI/UBIFS format data in eraseblock
+ *
+ * Author: Hu Jianyang <hujianyang@huawei.com>
+ */
+
+#define PROGRAM_NAME	"ubidump"
+
+#include <stdlib.h>
+#include <getopt.h>
+#include <unistd.h>
+
+#include <libubi.h>
+#include <libdump.h>
+#include <libmtd.h>
+#include "common.h"
+#include "ubiutils-common.h"
+
+#define MTD_DEV_PATT "/dev/mtd%d"
+
+/* The variables below are set by command line arguments */
+struct args {
+	const char *vol;
+	int lnum;
+	int info:1;
+	int ubi:1;
+	int ubifs:1;
+};
+
+static struct args args =
+{
+	.vol	= NULL,
+	.lnum	= -1,
+	.info	= 0,
+	.ubi	= 0,
+	.ubifs	= 0,
+};
+
+static const char doc[] = PROGRAM_NAME " version " VERSION
+		" - an utility to dump UBI/UBIFS format data in eraseblock";
+
+static const char optionsstr[] =
+"-h, --help		print help message\n"
+"-l, --lnum		logic eraseblock num to dump\n"
+"-i, --info		show explicit information about ubifs-level\n"
+"    --ubi		dump ubi-level stuff only\n"
+"    --ubifs		dump ubifs-level stuff only\n"
+"-V, --version		print program version";
+
+static const char usage[] =
+"Usage: " PROGRAM_NAME " <UBI volume node file name> [-l <lnum>] [-i]\n"
+"\t\t\t[--help] [--version]\n\n"
+"Example 1: " PROGRAM_NAME " /dev/ubi0_1 --lnum 2 - dump leb 2 in volume 1\n"
+"Example 2: " PROGRAM_NAME " /dev/ubi0_0 -l 1234 -i - dump leb 1234 with explicit info\n";
+
+static const struct option long_options[] = {
+	{ .name = "ubi",     .has_arg = 0, .flag = NULL, .val = 0   },
+	{ .name = "ubifs",   .has_arg = 0, .flag = NULL, .val = 0   },
+	{ .name = "help",    .has_arg = 0, .flag = NULL, .val = 'h' },
+	{ .name = "lnum",    .has_arg = 1, .flag = NULL, .val = 'l' },
+	{ .name = "info",    .has_arg = 0, .flag = NULL, .val = 'i' },
+	{ .name = "version", .has_arg = 0, .flag = NULL, .val = 'V' },
+	{ NULL, 0, NULL, 0}
+};
+
+static int parse_opt(int argc, char * const argv[])
+{
+	while (1) {
+		int option_index, key, error = 0;
+
+		key = getopt_long(argc, argv, "h?il:", long_options, &option_index);
+		if (key == -1)
+			break;
+
+		switch (key) {
+		case 0:
+			switch (option_index) {
+			case 0:
+				args.ubi = 1;
+				break;
+			case 1:
+				args.ubifs = 1;
+				break;
+			}
+			break;
+		case 'l':
+			args.lnum = simple_strtoul(optarg, &error);
+			if (error || args.lnum < 0)
+				return errmsg("bad lnum: \"%s\"", optarg);
+			break;
+
+		case 'i':
+			args.info = 1;
+			break;
+
+		case 'h':
+		case '?':
+			printf("%s\n\n", doc);
+			printf("%s\n\n", usage);
+			printf("%s\n", optionsstr);
+			exit(EXIT_SUCCESS);
+
+		case 'V':
+			common_print_version();
+			exit(EXIT_SUCCESS);
+
+		case ':':
+			return errmsg("parameter is missing");
+
+		default:
+			fprintf(stderr, "Use -h for help\n");
+			return -1;
+		}
+	}
+	if (optind == argc)
+		return errmsg("UBI device name was not specified (use -h for help)");
+	else if (optind != argc - 1)
+		return errmsg("more then one UBI device specified (use -h for help)");
+
+	args.vol = argv[optind];
+
+	if (args.lnum < 0)
+		return errmsg("lnum was not specified (use -h for help)");
+	if (args.ubi && (args.info || args.ubifs))
+		return errmsg("cannot specify --ubi with -i or --ubifs");
+
+	return 0;
+}
+
+static int leb_get_pnum(int *pnum)
+{
+	int fd, err;
+
+	fd = open(args.vol, O_RDONLY);
+	if (fd == -1) {
+		sys_errmsg("cannot open UBI volume \"%s\"", args.vol);
+		return -1;
+	}
+
+	*pnum = -1;
+	err = ubi_lnum_to_pnum(fd, args.lnum, pnum);
+	if (err)
+		sys_errmsg("ubi_lnum_to_pnum() failed, err = %d", err);
+
+	close(fd);
+	return err;
+}
+
+static int dump_eraseblock(int mtd_num, int pnum)
+{
+	int err, fd, peb_size, ret;
+	int ubi = !args.ubifs;
+	int ubifs = !args.ubi;
+	off_t offs;
+	libmtd_t libmtd;
+	struct mtd_info mtd_info;
+	struct mtd_dev_info mtd;
+	char file[strlen(MTD_DEV_PATT) + 100];
+	char *buf;
+
+	libmtd = libmtd_open();
+	if (!libmtd)
+		return errmsg("MTD subsystem is not present");
+
+	err = mtd_get_info(libmtd, &mtd_info);
+	if (err) {
+		if (errno == ENODEV)
+			errmsg("MTD is not present");
+		sys_errmsg("cannot get MTD information");
+		goto out;
+	}
+
+	err = mtd_get_dev_info1(libmtd, mtd_num, &mtd);
+	if (err) {
+		sys_errmsg("cannot get information about mtd%d\n", mtd_num);
+		goto out;
+	}
+
+	peb_size = mtd.eb_size;
+	buf = malloc(peb_size);
+	if (!buf) {
+		sys_errmsg("cannot alloc %d bytes memoroy", peb_size);
+		err = -ENOMEM;
+		goto out;
+	}
+
+	sprintf(file, MTD_DEV_PATT, mtd_num);
+	fd = open(file, O_RDONLY);
+	if (fd == -1) {
+		sys_errmsg("cannot open MTD device %s, mtd_num = %d", file, mtd_num);
+		goto out_free;
+	}
+
+	offs = pnum * peb_size;
+	ret = pread(fd, buf, peb_size, offs);
+	if (ret != peb_size) {
+		errmsg("cannot read %d bytes at pnum %d, read %d", peb_size,
+		       pnum, ret);
+		err = -EIO;
+		goto out_fd;
+	}
+
+	printf("leb %d is mapped to peb %d, start to dump:\n", args.lnum, pnum);
+	ubidump(buf, peb_size, ubi, ubifs, args.info);
+
+out_fd:
+	close(fd);
+out_free:
+	free(buf);
+out:
+	libmtd_close(libmtd);
+	return err;
+}
+
+int main(int argc, char * const argv[])
+{
+	int err, pnum, mtd_num;
+	libubi_t libubi;
+	struct ubi_vol_info vol_info;
+	struct ubi_dev_info dev_info;
+
+	err = parse_opt(argc, argv);
+	if (err)
+		return -1;
+
+	libubi = libubi_open();
+	if (!libubi) {
+		if (errno == 0)
+			errmsg("UBI is not present in the system");
+		else
+			sys_errmsg("cannot open libubi");
+		goto out_libubi;
+	}
+
+	err = ubi_probe_node(libubi, args.vol);
+	if (err == 1) {
+		errmsg("\"%s\" is an UBI device node, not an UBI volume node",
+		       args.vol);
+		goto out_libubi;
+	} else if (err < 0) {
+		if (errno == ENODEV)
+			errmsg("\"%s\" is not an UBI volume node", args.vol);
+		else
+			sys_errmsg("error while probing \"%s\"", args.vol);
+		goto out_libubi;
+	}
+
+	err = ubi_get_vol_info(libubi, args.vol, &vol_info);
+	if (err) {
+		sys_errmsg("cannot get information about UBI volume \"%s\"",
+			   args.vol);
+		goto out_libubi;
+	}
+
+	err = leb_get_pnum(&pnum);
+	if (err) {
+		sys_errmsg("cannot get pnum of leb %d", args.lnum);
+		goto out_libubi;
+	} else if (pnum == -1) {
+		errmsg("leb %d is not mapped", args.lnum);
+		goto out_libubi;
+	}
+
+	err = ubi_get_dev_info1(libubi, vol_info.dev_num, &dev_info);
+	if (err) {
+		sys_errmsg("cannot get information about UBI%d", vol_info.dev_num);
+		goto out_libubi;
+	}
+	mtd_num = dev_info.mtd_num;
+
+	err = dump_eraseblock(mtd_num, pnum);
+	if (err)
+		goto out_libubi;
+
+	libubi_close(libubi);
+	return 0;
+
+out_libubi:
+	libubi_close(libubi);
+	return -1;
+}
-- 
1.8.1.4

[PATCH 4/5] ubi-utils: Add ubifs-media.h

[hujianyang]

Add ubifs-media.h to dump ubifs-level stuff.


Signed-off-by: hujianyang <hujianyang@huawei.com>
---
 include/mtd/ubifs-media.h | 295 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 295 insertions(+)
 create mode 100644 include/mtd/ubifs-media.h

diff --git a/include/mtd/ubifs-media.h b/include/mtd/ubifs-media.h
new file mode 100644
index 0000000..e1767fb
--- /dev/null
+++ b/include/mtd/ubifs-media.h
@@ -0,0 +1,295 @@
+/*
+ * Copyright (c) International Business Machines Corp., 2006
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
+ * the GNU General Public License for more details.
+ *
+ * Author: Hu Jianyang <hujianyang@huawei.com>
+ *
+ * UBIFS library.
+ */
+
+/*
+ * This file defines the layout of UBI headers and all the other UBIiFS
+ * on-flash data structures.
+ */
+
+#ifndef __UBIFS_MEDIA_H__
+#define __UBIFS_MEDIA_H__
+
+#include <asm/byteorder.h>
+
+/* UBIFS node magic number (must not have the padding byte first or last) */
+#define UBIFS_NODE_MAGIC  0x06101831
+
+/* UBIFS padding byte pattern (must not be first or last byte of node magic) */
+#define UBIFS_PADDING_BYTE 0xCE
+
+/* Node sizes (N.B. these are guaranteed to be multiples of 8) */
+#define UBIFS_CH_SZ        sizeof(struct ubifs_ch)
+#define UBIFS_PAD_NODE_SZ  sizeof(struct ubifs_pad_node)
+
+/*
+ * UBIFS node types.
+ *
+ * UBIFS_INO_NODE: inode node
+ * UBIFS_DATA_NODE: data node
+ * UBIFS_DENT_NODE: directory entry node
+ * UBIFS_XENT_NODE: extended attribute node
+ * UBIFS_TRUN_NODE: truncation node
+ * UBIFS_PAD_NODE: padding node
+ * UBIFS_SB_NODE: superblock node
+ * UBIFS_MST_NODE: master node
+ * UBIFS_REF_NODE: LEB reference node
+ * UBIFS_IDX_NODE: index node
+ * UBIFS_CS_NODE: commit start node
+ * UBIFS_ORPH_NODE: orphan node
+ * UBIFS_NODE_TYPES_CNT: count of supported node types
+ *
+ * Note, we index arrays by these numbers, so keep them low and contiguous.
+ * Node type constants for inodes, direntries and so on have to be the same as
+ * corresponding key type constants.
+ */
+enum {
+	UBIFS_INO_NODE,
+	UBIFS_DATA_NODE,
+	UBIFS_DENT_NODE,
+	UBIFS_XENT_NODE,
+	UBIFS_TRUN_NODE,
+	UBIFS_PAD_NODE,
+	UBIFS_SB_NODE,
+	UBIFS_MST_NODE,
+	UBIFS_REF_NODE,
+	UBIFS_IDX_NODE,
+	UBIFS_CS_NODE,
+	UBIFS_ORPH_NODE,
+	UBIFS_NODE_TYPES_CNT,
+};
+
+/*
+ * Supported key hash functions.
+ *
+ * UBIFS_KEY_HASH_R5: R5 hash
+ * UBIFS_KEY_HASH_TEST: test hash which just returns first 4 bytes of the name
+ */
+enum {
+	UBIFS_KEY_HASH_R5,
+	UBIFS_KEY_HASH_TEST,
+};
+
+/*
+ * Supported key formats.
+ *
+ * UBIFS_SIMPLE_KEY_FMT: simple key format
+ */
+enum {
+	UBIFS_SIMPLE_KEY_FMT,
+};
+
+/*
+ * Node group type (used by recovery to recover whole group or none).
+ *
+ * UBIFS_NO_NODE_GROUP: this node is not part of a group
+ * UBIFS_IN_NODE_GROUP: this node is a part of a group
+ * UBIFS_LAST_OF_NODE_GROUP: this node is the last in a group
+ */
+enum {
+	UBIFS_NO_NODE_GROUP = 0,
+	UBIFS_IN_NODE_GROUP,
+	UBIFS_LAST_OF_NODE_GROUP,
+};
+
+/*
+ * Superblock flags.
+ *
+ * UBIFS_FLG_BIGLPT: if "big" LPT model is used if set
+ * UBIFS_FLG_SPACE_FIXUP: first-mount "fixup" of free space within LEBs needed
+ */
+enum {
+	UBIFS_FLG_BIGLPT = 0x02,
+	UBIFS_FLG_SPACE_FIXUP = 0x04,
+};
+
+/**
+ * struct ubifs_ch - common header node.
+ * @magic: UBIFS node magic number (%UBIFS_NODE_MAGIC)
+ * @crc: CRC-32 checksum of the node header
+ * @sqnum: sequence number
+ * @len: full node length
+ * @node_type: node type
+ * @group_type: node group type
+ * @padding: reserved for future, zeroes
+ *
+ * Every UBIFS node starts with this common part. If the node has a key, the
+ * key always goes next.
+ */
+struct ubifs_ch {
+	__le32 magic;
+	__le32 crc;
+	__le64 sqnum;
+	__le32 len;
+	__u8 node_type;
+	__u8 group_type;
+	__u8 padding[2];
+} __attribute__ ((packed));
+
+/**
+ * struct ubifs_pad_node - padding node.
+ * @ch: common header
+ * @pad_len: how many bytes after this node are unused (because padded)
+ * @padding: reserved for future, zeroes
+ */
+struct ubifs_pad_node {
+	struct ubifs_ch ch;
+	__le32 pad_len;
+} __attribute__ ((packed));
+
+/**
+ * struct ubifs_sb_node - superblock node.
+ * @ch: common header
+ * @padding: reserved for future, zeroes
+ * @key_hash: type of hash function used in keys
+ * @key_fmt: format of the key
+ * @flags: file-system flags (%UBIFS_FLG_BIGLPT, etc)
+ * @min_io_size: minimal input/output unit size
+ * @leb_size: logical eraseblock size in bytes
+ * @leb_cnt: count of LEBs used by file-system
+ * @max_leb_cnt: maximum count of LEBs used by file-system
+ * @max_bud_bytes: maximum amount of data stored in buds
+ * @log_lebs: log size in logical eraseblocks
+ * @lpt_lebs: number of LEBs used for lprops table
+ * @orph_lebs: number of LEBs used for recording orphans
+ * @jhead_cnt: count of journal heads
+ * @fanout: tree fanout (max. number of links per indexing node)
+ * @lsave_cnt: number of LEB numbers in LPT's save table
+ * @fmt_version: UBIFS on-flash format version
+ * @default_compr: default compression algorithm (%UBIFS_COMPR_LZO, etc)
+ * @padding1: reserved for future, zeroes
+ * @rp_uid: reserve pool UID
+ * @rp_gid: reserve pool GID
+ * @rp_size: size of the reserved pool in bytes
+ * @padding2: reserved for future, zeroes
+ * @time_gran: time granularity in nanoseconds
+ * @uuid: UUID generated when the file system image was created
+ * @ro_compat_version: UBIFS R/O compatibility version
+ */
+struct ubifs_sb_node {
+	struct ubifs_ch ch;
+	__u8 padding[2];
+	__u8 key_hash;
+	__u8 key_fmt;
+	__le32 flags;
+	__le32 min_io_size;
+	__le32 leb_size;
+	__le32 leb_cnt;
+	__le32 max_leb_cnt;
+	__le64 max_bud_bytes;
+	__le32 log_lebs;
+	__le32 lpt_lebs;
+	__le32 orph_lebs;
+	__le32 jhead_cnt;
+	__le32 fanout;
+	__le32 lsave_cnt;
+	__le32 fmt_version;
+	__le16 default_compr;
+	__u8 padding1[2];
+	__le32 rp_uid;
+	__le32 rp_gid;
+	__le64 rp_size;
+	__le32 time_gran;
+	__u8 uuid[16];
+	__le32 ro_compat_version;
+	__u8 padding2[3968];
+} __attribute__ ((packed));
+
+/**
+ * struct ubifs_mst_node - master node.
+ * @ch: common header
+ * @highest_inum: highest inode number in the committed index
+ * @cmt_no: commit number
+ * @flags: various flags (%UBIFS_MST_DIRTY, etc)
+ * @log_lnum: start of the log
+ * @root_lnum: LEB number of the root indexing node
+ * @root_offs: offset within @root_lnum
+ * @root_len: root indexing node length
+ * @gc_lnum: LEB reserved for garbage collection (%-1 value means the LEB was
+ * not reserved and should be reserved on mount)
+ * @ihead_lnum: LEB number of index head
+ * @ihead_offs: offset of index head
+ * @index_size: size of index on flash
+ * @total_free: total free space in bytes
+ * @total_dirty: total dirty space in bytes
+ * @total_used: total used space in bytes (includes only data LEBs)
+ * @total_dead: total dead space in bytes (includes only data LEBs)
+ * @total_dark: total dark space in bytes (includes only data LEBs)
+ * @lpt_lnum: LEB number of LPT root nnode
+ * @lpt_offs: offset of LPT root nnode
+ * @nhead_lnum: LEB number of LPT head
+ * @nhead_offs: offset of LPT head
+ * @ltab_lnum: LEB number of LPT's own lprops table
+ * @ltab_offs: offset of LPT's own lprops table
+ * @lsave_lnum: LEB number of LPT's save table (big model only)
+ * @lsave_offs: offset of LPT's save table (big model only)
+ * @lscan_lnum: LEB number of last LPT scan
+ * @empty_lebs: number of empty logical eraseblocks
+ * @idx_lebs: number of indexing logical eraseblocks
+ * @leb_cnt: count of LEBs used by file-system
+ * @padding: reserved for future, zeroes
+ */
+struct ubifs_mst_node {
+	struct ubifs_ch ch;
+	__le64 highest_inum;
+	__le64 cmt_no;
+	__le32 flags;
+	__le32 log_lnum;
+	__le32 root_lnum;
+	__le32 root_offs;
+	__le32 root_len;
+	__le32 gc_lnum;
+	__le32 ihead_lnum;
+	__le32 ihead_offs;
+	__le64 index_size;
+	__le64 total_free;
+	__le64 total_dirty;
+	__le64 total_used;
+	__le64 total_dead;
+	__le64 total_dark;
+	__le32 lpt_lnum;
+	__le32 lpt_offs;
+	__le32 nhead_lnum;
+	__le32 nhead_offs;
+	__le32 ltab_lnum;
+	__le32 ltab_offs;
+	__le32 lsave_lnum;
+	__le32 lsave_offs;
+	__le32 lscan_lnum;
+	__le32 empty_lebs;
+	__le32 idx_lebs;
+	__le32 leb_cnt;
+	__u8 padding[344];
+} __attribute__ ((packed));
+
+/**
+ * struct ubifs_ref_node - logical eraseblock reference node.
+ * @ch: common header
+ * @lnum: the referred logical eraseblock number
+ * @offs: start offset in the referred LEB
+ * @jhead: journal head number
+ * @padding: reserved for future, zeroes
+ */
+struct ubifs_ref_node {
+	struct ubifs_ch ch;
+	__le32 lnum;
+	__le32 offs;
+	__le32 jhead;
+	__u8 padding[28];
+} __attribute__ ((packed));
+
+#endif /*!__UBIFS_MEDIA_H__ */
-- 
1.8.1.4

[PATCH 5/5] ubi-utils: Modified Makefile

[hujianyang]

Compile support.


Signed-off-by: hujianyang <hujianyang@huawei.com>
---
 Makefile | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index eade234..9dc9645 100644
--- a/Makefile
+++ b/Makefile
@@ -28,7 +28,8 @@ MTD_BINS = \
 	sumtool jffs2reader
 UBI_BINS = \
 	ubiupdatevol ubimkvol ubirmvol ubicrc32 ubinfo ubiattach \
-	ubidetach ubinize ubiformat ubirename mtdinfo ubirsvol ubiblock
+	ubidetach ubinize ubiformat ubirename mtdinfo ubirsvol ubiblock \
+	ubidump

 BINS = $(MTD_BINS)
 BINS += mkfs.ubifs/mkfs.ubifs
@@ -115,10 +116,12 @@ obj-libiniparser.a = libiniparser.o dictionary.o
 obj-libscan.a      = libscan.o
 obj-libubi.a       = libubi.o
 obj-libubigen.a    = libubigen.o
+obj-libdump.a      = libdump.o

 obj-mtdinfo   = libubigen.a
 obj-ubinize   = libubigen.a libiniparser.a
 obj-ubiformat = libubigen.a libscan.a
+obj-ubidump   = libdump.a

-$(foreach v,libubi.a libubigen.a libiniparser.a libscan.a,$(eval $(call _mkdep,ubi-utils/,$(v))))
+$(foreach v,libubi.a libubigen.a libiniparser.a libscan.a libdump.a,$(eval $(call _mkdep,ubi-utils/,$(v))))
 $(foreach v,$(UBI_BINS),$(eval $(call mkdep,ubi-utils/,$(v),libubi.a ubiutils-common.o)))
-- 
1.8.1.4

Re: [PATCH RFC v2] UBI: New ioctl() to support ubidump

[Bill Pringlemeir]

On 29 Jul 2014, hujianyang@huawei.com wrote:

> An ioctl() return pnum of a specified leb.

> Signed-off-by: hujianyang <hujianyang@huawei.com>
> ---
> drivers/mtd/ubi/cdev.c      | 26 ++++++++++++++++++++++++++
> include/uapi/mtd/ubi-user.h | 12 ++++++++++++
> 2 files changed, 38 insertions(+)

Please look at 'ubi_scan()', 

  http://sourcecodebrowser.com/mtd-utils/20110107/libscan_8c.html
  http://sourcecodebrowser.com/mtd-utils/20110107/libscan_8h.html

After this call info->ec[pnum] is,

  EB_EMPTY      (ffffffff) - erased
  EB_CORRUPTED  (fffffffe) - inconsistent UBI data.
  EB_ALIEN      (fffffffd) - non-ubi erase sector
  EB_BAD        (fffffffc) - bad block
  or the LNUM.

To write your code, you can either scan the array (~32-2048 entries)
each time you want an 'lnum' or you could run through the array and
construct the opposite table; a 'logical index' gives a 'physical eb';
this would require keeping track of the 'generation' count.

I think a patch to 'ubi_scan()' to create an 'pmap' array might be
better or more accepted than a Linux/MTD/UBI patch?  Then only the
'ubidump' code is needed and not a properly configured/versioned kernel;
or at least only the nandsim module which is similar to some other
utilities.  If you had a 'ubi_scan()' which has an 'info->pmap[leb]'
which had,

  EB_EMPTY      (ffffffff) - not mapped
  or the PNUM.

Would you need to patch the kernel?

Fwiw,
Bill Pringlemeir.

Re: [PATCH RFC v2] UBI: New ioctl() to support ubidump

[Richard Weinberger]

On Tue, Jul 29, 2014 at 11:26 AM, hujianyang <hujianyang@huawei.com> wrote:
> An ioctl() return pnum of a specified leb.
>
>
> Signed-off-by: hujianyang <hujianyang@huawei.com>
> ---
>  drivers/mtd/ubi/cdev.c      | 26 ++++++++++++++++++++++++++
>  include/uapi/mtd/ubi-user.h | 12 ++++++++++++
>  2 files changed, 38 insertions(+)
>
> diff --git a/drivers/mtd/ubi/cdev.c b/drivers/mtd/ubi/cdev.c
> index 7646220..6fa7346 100644
> --- a/drivers/mtd/ubi/cdev.c
> +++ b/drivers/mtd/ubi/cdev.c
> @@ -581,6 +581,32 @@ static long vol_cdev_ioctl(struct file *file, unsigned int cmd,
>                 break;
>         }
>
> +       /* Get pnum of a specified leb command */
> +       case UBI_IOCEBGETPNUM:
> +       {
> +               struct ubi_lnum2pnum_req req;
> +               int pnum;
> +
> +               err = copy_from_user(&req, argp,
> +                               sizeof(struct ubi_lnum2pnum_req));
> +               if (err) {
> +                       err = -EFAULT;
> +                       break;
> +               }
> +
> +               err = ubi_is_mapped(desc, req.lnum);
> +               if (err <= 0)
> +                       break;
> +               pnum = vol->eba_tbl[req.lnum];
> +               req.pnum = pnum;

Isn't this racy?
i.e. If a LEB change happens between ubi_is_mapped() and "pnum =
vol->eba_tbl[req.lnum]".

> +               err = copy_to_user(argp, &req,
> +                               sizeof(struct ubi_lnum2pnum_req));
> +               if (err)
> +                       err = -EFAULT;
> +               break;
> +       }
> +
>         default:
>                 err = -ENOTTY;
>                 break;
> diff --git a/include/uapi/mtd/ubi-user.h b/include/uapi/mtd/ubi-user.h
> index 1927b0d..fc41ddb 100644
> --- a/include/uapi/mtd/ubi-user.h
> +++ b/include/uapi/mtd/ubi-user.h
> @@ -205,6 +205,8 @@
>  #define UBI_IOCVOLCRBLK _IOW(UBI_VOL_IOC_MAGIC, 7, struct ubi_blkcreate_req)
>  /* Remove the R/O block device */
>  #define UBI_IOCVOLRMBLK _IO(UBI_VOL_IOC_MAGIC, 8)
> +/* Get pnum of a specified leb */
> +#define UBI_IOCEBGETPNUM _IOW(UBI_VOL_IOC_MAGIC, 9, struct ubi_lnum2pnum_req)
>
>  /* Maximum MTD device name length supported by UBI */
>  #define MAX_UBI_MTD_NAME_LEN 127
> @@ -442,4 +444,14 @@ struct ubi_blkcreate_req {
>         __s8  padding[128];
>  }  __packed;
>
> +/**
> + * struct ubi_lnum2pnum_req - a data structure used in lnum translate requests.
> + * @lnum: logical eraseblock num to translate
> + * @pnum: physical eraseblock num @lnum mapped
> + */
> +struct ubi_lnum2pnum_req {
> +       __s32 lnum;
> +       __s32 pnum;
> +}  __packed;
> +
>  #endif /* __UBI_USER_H__ */
> --
> 1.8.1.4
>
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/



-- 
Thanks,
//richard

Re: [PATCH RFC v2] UBI: New ioctl() to support ubidump

[hujianyang]

> 
> I think a patch to 'ubi_scan()' to create an 'pmap' array might be
> better or more accepted than a Linux/MTD/UBI patch?  Then only the
> 'ubidump' code is needed and not a properly configured/versioned kernel;
> or at least only the nandsim module which is similar to some other
> utilities.  If you had a 'ubi_scan()' which has an 'info->pmap[leb]'
> which had,
> 

Yes, I think you are right~! I don't want this utility can't be run
without a properly configured kernel and this full-scan method will
be used again when we dumping with an image file.

But I'm worry about the performance. As we dump one specified peb
now, we can't record the mapping table and need to do full scan
each time running this utility.

Anyway, I will try what you said first~!

Thanks~!

Hu

Re: [PATCH RFC v2] UBI: New ioctl() to support ubidump

[hujianyang]

> 
> Isn't this racy?
> i.e. If a LEB change happens between ubi_is_mapped() and "pnum =
> vol->eba_tbl[req.lnum]".
> 

As I was writing this code, I used this ubi_is_mapped() to make sure
lnum is less than @vol->reserved_pebs and !@vol->upd_marker in order
to avoid kernel panic when we get pnum from eba_tbl.

Even we locked this ioctl, peb may also be changed before we read it
from MTD. That's why I didn't focus on this race. We can use some
condition check to make sure this vol is not mounted but user may use
this utility on rootfs.

Bill showed me a new method to get pnum without UBI driver. I will
try that first but this new method can't avoid peb changes before
we read it either. We should consider more on it.

Thanks!

Hu

Re: [PATCH RFC v2] UBI: New ioctl() to support ubidump

[Bill Pringlemeir]

On 29 Jul 2014, hujianyang@huawei.com wrote:

> But I'm worry about the performance. As we dump one specified peb
> now, we can't record the mapping table and need to do full scan
> each time running this utility.

You can record the table if you know that the MTD is not changing.  If
you introduce the 'ioctl()' to the kernel, it will most likely be on an
active file system.  I guess this is a big decision in the tools design.

Supporting an active file system is much more difficult.  For the case
of an real MTD backing store, I was thinking this would run from another
partition or an initramfs; UBI/UbiFS would not be active.  It could
possibly attempt to repair the UBI/UbiFs, while nothing else is using
it.  The other use case is to read out the flash via some tools and
loopback mount it with nandsim on a PC for diagnosis; a post-mortem
analysis.

If you wish to run the utility on an active MTD, I would suggest some
'freeze/thaw' type function to stop UBI/UbiFs and then just use the tool
on the 'frozen' MTD.  Maybe this exists already for suspend/resume or
something else.  It might only be a debug type interface; so
conditionally compiled for developers.  It needs a hook so you can
trigger the freeze on some condition via code.

I think others don't understand why you would want the 'ubidump' to run
on an active file system?  At least, I don't understand this?  Is there
some use case we are missing?  I am quite sure that you can perform
better diagnostics on a 'frozen' image where you can look at the whole
image state.

Fwiw,
Bill Pringlemeir.

Re: [PATCH RFC v2] UBI: New ioctl() to support ubidump

[hujianyang]

On 2014/7/30 23:53, Bill Pringlemeir wrote:
> 
> You can record the table if you know that the MTD is not changing.  If
> you introduce the 'ioctl()' to the kernel, it will most likely be on an
> active file system.  I guess this is a big decision in the tools design.

This ioctl() doesn't need UBIFS filesystem to be mounted, just needs
UBI driver. We can keep this partition not active during this tool
running.

But a properly configured kernel requirement you said in last mail
worries me most. I think you are right and want to have a try. But
I've discussed these stuff with Artem and he said, just quote:

""
So I envision that the tool would work like this.

$ ubidump my.img --lnum 5
  - dump LEB 5, will need to scan the entire image.

$ ubidump /dev/ubiX_Y --lnum 5
  - will just ask the UBI driver to give the PEB number for LEB 5, then
find out the MTD device for this volume (should be possible by checking
the sysfs files), and then reads the MTD device, and gets the UBI-level
information from there.

Something like this, just quick thoughts.
""
http://lists.infradead.org/pipermail/linux-mtd/2014-July/054674.html


We can keep a temporary file to record the mapping table but that will
make the initial submission much more complicated. But this way you
mentioned to record the table should be considered to add into this tool
in the future.

> 
> I think others don't understand why you would want the 'ubidump' to run
> on an active file system?  At least, I don't understand this?  Is there
> some use case we are missing?  I am quite sure that you can perform
> better diagnostics on a 'frozen' image where you can look at the whole
> image state.
> 

Yes, I want this tool to be flexible before. But dumping with an active
partition really useless as you said now. Because in most cases, the
volumes we want to dump must have something wrong.


I think we need to make a lighter version and submit it to ubi-utils
at beginning, then make it strongly step by step. Despite all that,
we should have a clear design of what we need now. So I'm happy with
your sharing of your thoughts.

The "freeze/thaw" method will result a complex tool. We can use some
functionality to make sure UBIFS partition is not active. Should I need
to add them in this version or enable them in the future? Do you think
it is a key problem of this tool? This utility may tolerance some data
mistake caused by peb update race in my considering.

Re: [PATCH RFC v2] UBI: New ioctl() to support ubidump

[Artem Bityutskiy]

On Thu, 2014-07-31 at 11:01 +0800, hujianyang wrote:
> On 2014/7/30 23:53, Bill Pringlemeir wrote:
> > 
> > You can record the table if you know that the MTD is not changing.  If
> > you introduce the 'ioctl()' to the kernel, it will most likely be on an
> > active file system.  I guess this is a big decision in the tools design.
> 
> This ioctl() doesn't need UBIFS filesystem to be mounted, just needs
> UBI driver. We can keep this partition not active during this tool
> running.
> 
> But a properly configured kernel requirement you said in last mail
> worries me most. I think you are right and want to have a try. But
> I've discussed these stuff with Artem and he said, just quote:
> 
> ""
> So I envision that the tool would work like this.
> 
> $ ubidump my.img --lnum 5
>   - dump LEB 5, will need to scan the entire image.
> 
> $ ubidump /dev/ubiX_Y --lnum 5
>   - will just ask the UBI driver to give the PEB number for LEB 5, then
> find out the MTD device for this volume (should be possible by checking
> the sysfs files), and then reads the MTD device, and gets the UBI-level
> information from there.
> 
> Something like this, just quick thoughts.
> ""

Having a tool which does not need any kernel help is a lot more
preferable. I though we may have an ioctl like this if no one has time
to write full scanning support in user-space, and then maintain it.

> We can keep a temporary file to record the mapping table but that will
> make the initial submission much more complicated. But this way you
> mentioned to record the table should be considered to add into this tool
> in the future.

Hujianyang, AFAIU, right now you need an ability to dump UBIFS stuff.
You do not really need to know PEB number. So ubidump may
analyze /dev/ubiX_Y without knowing the PEB number, right? It can just
read from /dev/ubiX_Y directly. And print you all the UBIFS nodes.

I mean, if what user gives you is an UBI volume, you do not try going to
the MTD level.

-- 
Best Regards,
Artem Bityutskiy

Re: [PATCH RFC v2] UBI: New ioctl() to support ubidump

[Artem Bityutskiy]

On Wed, 2014-07-30 at 11:53 -0400, Bill Pringlemeir wrote:
> I think others don't understand why you would want the 'ubidump' to run
> on an active file system?  At least, I don't understand this?  Is there
> some use case we are missing?  I am quite sure that you can perform
> better diagnostics on a 'frozen' image where you can look at the whole
> image state.

I think you are right that in ideal world ubidump should be able to
figure out the LEB->PEB mapping without asking the driver. It should be
able to scan the image and figure everything out itself.

However, writing such a toll is _hard_. Maintaining it is _hard too_.
You basically need to copy part of the driver to user-space, and
maintain it there. Whenever someone adds a new UBI feature like
"fastmap", you need to port that to the tool too.

This is doable, e.g., e2fsprogs project basically does this. But so far,
no one did this for UBI/UBIFS. And the UBIFS user-base is relatively
small. Many people expressed a desire to have a 'chkfs.ubifs' and
'ubidump', but no one went ahead ad did this - this is not easy and this
costs.

So I initially thought may be a little help from the kernel would not
hurt. But I am not 100% sure this is a good idea.

-- 
Best Regards,
Artem Bityutskiy

Re: [PATCH RFC v2] UBI: New ioctl() to support ubidump

[hujianyang]

On 2014/7/31 21:24, Artem Bityutskiy wrote:
> 
> Hujianyang, AFAIU, right now you need an ability to dump UBIFS stuff.

Yes, but not all of what I want. I need this utility dumping UBI-level
stuff too. The ability, not only to dump UBIFS stuff but also to dump
UBI stuff will make it a useful tool.

> You do not really need to know PEB number. So ubidump may
> analyze /dev/ubiX_Y without knowing the PEB number, right? It can just
> read from /dev/ubiX_Y directly. And print you all the UBIFS nodes.
> 

Last version I sent to this maillist was reading data from UBI driver
directly. But our .read() function in UBI driver jump over UBI stuff, so
I read UBI stuff from an ioctl(). This isn't a common way as you said.

I have to say I am about to leave Beijing for at least 3 months and
have no time for the development of this tool. I aspire if we can push
some of my work to ubi-utils before my leaving. So we should hurry up~!

First, we can leave UBI/UBIFS format dumping behind, just dump binary
data, that will make initial submission smaller.

Then, think about what we want at this time.

1) No UBI stuff and read UBIFS stuff from UBI-driver
This is a easiest way and we don't need any ioctl either. We can't get
UBI-level stuff right now but we can put this into our design of dumping
an image file.

2) Read data from MTD-driver and find a way to get pnum
This way is basing on my v2 work. We can put an ioctl() to kernel and
if kernel don't support this ioctl(), this tool will do a full-scan on
MTD device. It's transparent to user. But we should take some time to
consider how to do this full-scan.

3) Re-design this tool about dropping UBI driver based dumping.
This way is like the exist ubiformat utility. We don't use UBI driver
and just do read/write(maybe we can do some repairing later) with MTD
device or image files. It's not small.


I think you will prefer to solution 1. Although it's not fit all my
desires, I will agree with it if you insist. Or do you have any other
suggestions?


Thanks~!

Hu

Re: [PATCH RFC v2] UBI: New ioctl() to support ubidump

[Artem Bityutskiy]

On Fri, 2014-08-01 at 10:50 +0800, hujianyang wrote:
> On 2014/7/31 21:24, Artem Bityutskiy wrote:
> > 
> > Hujianyang, AFAIU, right now you need an ability to dump UBIFS stuff.
> 
> Yes, but not all of what I want. I need this utility dumping UBI-level
> stuff too. The ability, not only to dump UBIFS stuff but also to dump
> UBI stuff will make it a useful tool.

OK.

> > You do not really need to know PEB number. So ubidump may
> > analyze /dev/ubiX_Y without knowing the PEB number, right? It can just
> > read from /dev/ubiX_Y directly. And print you all the UBIFS nodes.
> > 
> 
> Last version I sent to this maillist was reading data from UBI driver
> directly. But our .read() function in UBI driver jump over UBI stuff, so
> I read UBI stuff from an ioctl(). This isn't a common way as you said.

There are 2 layers involved: MTD and UBI.

On the MTD level, the entire flash contents is available (/dev/mtdX)
On the UBI level, only UBI volumes are available (/dev/ubiX_Y)

If I give you an UBI volume dump (cp /dev/ubiX_Y ubi.img), you will only
see the contents of the UBI volume. If there was UBIFS file-system, you
will see UBIFS nodes in the "ubi.img" file. You will not see UBI headers
there.

The ideal design would be treating /dev/ubiX_Y the same way as
'ubi.img'. IOW:

ubidump ubi.img
and
ubidump /dev/ubiX_Y

should give the same results.

If I give you an MTD image (nanddump -o mtd.img /dev/mtdX), you should
be able to see both UBIFS and UBI stuff in 'mtd.img'. You may see the
same information in /dev/mtdX. To get the LEB<->PEB mapping, you need to
do scanning, etc. So

ubidump mtd.img
and
ubidump /dev/mtdX

should be the same.

This is the basic picture.

Now what you want is to add an exception to this scheme. Namely, for the
'ubidump /dev/ubiX_Y' case: you want to get help from the driver, using
the ioctl you introduced.

Frankly, I am not 100% sure this is a good idea, would be nice to hear
from other people. I'd need to think a bit on this.

So what I suggested is the you do not submit this exception so far. Just
teach ubidump to handle UBI volumes. I.e., make it work with

ubidump ubi.img
ubidump /dev/ubiX_Y

yes, these 2 will dump only ubifs stuff. Limited functionality.

> 1) No UBI stuff and read UBIFS stuff from UBI-driver
> This is a easiest way and we don't need any ioctl either. We can't get
> UBI-level stuff right now but we can put this into our design of dumping
> an image file.

Yes. That's a good first step.

> 2) Read data from MTD-driver and find a way to get pnum
> This way is basing on my v2 work. We can put an ioctl() to kernel and
> if kernel don't support this ioctl(), this tool will do a full-scan on
> MTD device. It's transparent to user. But we should take some time to
> consider how to do this full-scan.

I guess, I'd need to think some more about the ioctl(), and may be
someone else suggests something.

> 3) Re-design this tool about dropping UBI driver based dumping.
> This way is like the exist ubiformat utility. We don't use UBI driver
> and just do read/write(maybe we can do some repairing later) with MTD
> device or image files. It's not small.

Probably, we'll think about this when we are done with 1) and may be 2).

-- 
Best Regards,
Artem Bityutskiy

Re: [PATCH RFC v2] UBI: New ioctl() to support ubidump

[hujianyang]

On 2014/8/1 15:30, Artem Bityutskiy wrote:
> 
> There are 2 layers involved: MTD and UBI.
> 
> On the MTD level, the entire flash contents is available (/dev/mtdX)
> On the UBI level, only UBI volumes are available (/dev/ubiX_Y)
> 
> If I give you an UBI volume dump (cp /dev/ubiX_Y ubi.img), you will only
> see the contents of the UBI volume. If there was UBIFS file-system, you
> will see UBIFS nodes in the "ubi.img" file. You will not see UBI headers
> there.
> 
> The ideal design would be treating /dev/ubiX_Y the same way as
> 'ubi.img'. IOW:
> 
> ubidump ubi.img
> and
> ubidump /dev/ubiX_Y
> 
> should give the same results.
> 
> If I give you an MTD image (nanddump -o mtd.img /dev/mtdX), you should
> be able to see both UBIFS and UBI stuff in 'mtd.img'. You may see the
> same information in /dev/mtdX. To get the LEB<->PEB mapping, you need to
> do scanning, etc. So
> 
> ubidump mtd.img
> and
> ubidump /dev/mtdX
> 
> should be the same.
> 
> This is the basic picture.
> 
> Now what you want is to add an exception to this scheme. Namely, for the
> 'ubidump /dev/ubiX_Y' case: you want to get help from the driver, using
> the ioctl you introduced.
> 
> Frankly, I am not 100% sure this is a good idea, would be nice to hear
> from other people. I'd need to think a bit on this.
> 
> So what I suggested is the you do not submit this exception so far. Just
> teach ubidump to handle UBI volumes. I.e., make it work with
> 
> ubidump ubi.img
> ubidump /dev/ubiX_Y
> 

Oh, I've confused these things before. I want this utility work with
mtd.img and /dev/ubiX_Y. It seems they are not same.

> yes, these 2 will dump only ubifs stuff. Limited functionality.
> 
>> 1) No UBI stuff and read UBIFS stuff from UBI-driver
>> This is a easiest way and we don't need any ioctl either. We can't get
>> UBI-level stuff right now but we can put this into our design of dumping
>> an image file.
> 
> Yes. That's a good first step.
> 

How about writing a new version basing on this?

>> 2) Read data from MTD-driver and find a way to get pnum
>> This way is basing on my v2 work. We can put an ioctl() to kernel and
>> if kernel don't support this ioctl(), this tool will do a full-scan on
>> MTD device. It's transparent to user. But we should take some time to
>> consider how to do this full-scan.
> 
> I guess, I'd need to think some more about the ioctl(), and may be
> someone else suggests something.
> 

OK.


Have a good weekend~!

Hu

Re: [PATCH RFC v2] UBI: New ioctl() to support ubidump

[Bill Pringlemeir]

On  1 Aug 2014, dedekind1@gmail.com wrote:

> On Fri, 2014-08-01 at 10:50 +0800, hujianyang wrote:

>> 1) No UBI stuff and read UBIFS stuff from UBI-driver This is a
>> easiest way and we don't need any ioctl either. We can't get
>> UBI-level stuff right now but we can put this into our design of
>> dumping an image file.

> Yes. That's a good first step.

I agree that this is a good way to partition the functionality.

>> 2) Read data from MTD-driver and find a way to get pnum This way is
>> basing on my v2 work. We can put an ioctl() to kernel and if kernel
>> don't support this ioctl(), this tool will do a full-scan on MTD
>> device. It's transparent to user. But we should take some time to
>> consider how to do this full-scan.

> I guess, I'd need to think some more about the ioctl(), and may be
> someone else suggests something.

Isn't this just more work?  The tool will rely on two things?

>> 3) Re-design this tool about dropping UBI driver based dumping.
>> This way is like the exist ubiformat utility. We don't use UBI driver
>> and just do read/write(maybe we can do some repairing later) with MTD
>> device or image files. It's not small.

> Probably, we'll think about this when we are done with 1) and may be
> 2).

Some of these points I don't understand.  The 'ubiformat' already uses
'libscan.c', which have the basics for creating a static 'PEB->LEB'
mapping.  So while I agree that maintaining two version of UBI is not
good, I think we already have this?

UBI has to deal with writing, especially atomic writes, and concurrency.
I don't think the tool needs either, so the 2nd version of UBI is
significantly simpler.  Just look at 'libscan.c' versus
'driver/mtd/ubi/*.c'.

Also, I think a lot of people will have issues where they can not mount
the device and the only thing they can get is the whole 'nanddump';
which won't be done through Linux, but some other means.  Ie, some ROM
boot code (which doesn't understand either UBI or UbiFS) to read out
NAND, a NAND programmer, etc.

Finally, the separate implementation can be good.  Having two versions
of code use the same data structures will often show issues.  If we want
to debug UBI, then using it is not so helpful.  Having the separate
(simpler) implementation with worse performance is exactly the kind of
thing you want for debugging and repair, isn't it?  Features like
'fastmap' are optional and we should fall back to a linear scan.  I
think that future UBI changes will strive to be backwards compatible
unless there is some really good reason not to be?

The big advantage of doing everything outside the kernel is we can
eventually add lots of extra consistency checks that you would never
want to do in a running file system.  Ie, verify a linear scan and the
fastmap are consistent, etc.

My intent is never to make work for anyone.  Adding a patch to libscan.c
seems easy to me.

 1. Allocate a 'pmap' array and a temporary 'sequence' array.
 2. Initialize 'pmap' to -1.
 3a. For each valid 'eb', get 'lnum' and 'sqnum' from vid header.
  b. if (pmap[lnum] == -1 || sequence[lnum] < sqnum)
      pmap[lnum] = eb, sequence[lnum] = sqnum;

but maybe I don't understand all the complexities.  Isn't it that
simple?

Now if we want to use the tool on live volumes, 'libscan.c' will not
work for that.  I thought that the concepts of a debugger may apply.  It
can look at a core file, or a paused processes memory.  But maybe it is
crazy to 'pause' Ubi/UbiFs.  Someone might want such 'snapshots' to
examine the evolution of storage or some race issues... but I guess that
is not the main reason of using the ioctl().

Fwiw,
Bill Pringlemeir.

Re: [PATCH RFC v2] UBI: New ioctl() to support ubidump

[hujianyang]

On 2014/8/1 23:35, Bill Pringlemeir wrote:
> 
>  1. Allocate a 'pmap' array and a temporary 'sequence' array.
>  2. Initialize 'pmap' to -1.
>  3a. For each valid 'eb', get 'lnum' and 'sqnum' from vid header.
>   b. if (pmap[lnum] == -1 || sequence[lnum] < sqnum)
>       pmap[lnum] = eb, sequence[lnum] = sqnum;
> 

I've researched the function ubi_scan() in libscan.c today and
found it's not easy to just add small changes to enable LEB->PEB
mapping table as we want.

There are two methods on enabling this functionality:
1) Add a mapping table in struct ubi_scan_info
2) Add some new parameters for function ubi_scan()

You know each MTD device may contain more than one UBI volume
and each volume should has a private mapping table. So it's not
easy to enable it in struct ubi_scan_info like method 1) because
we don't know the actual counts of volumes.

Further more, @sqnum is not enough for peb determining, we should
consider @on_copy flag for wear-leveling and atomic write. If
@on_copy flag is set, we need to read the whole leb and check CRC
to determine which peb is right. As this is a debugging tool, I
think printing all pebs have same lnum(of course in same volume)
is better. Now, we have to start thinking how to record this
stuff, that's a big problem.


In another way like method 2), we can add a new structure named
ubi_translate, put lnum and vol_id in it and returns a list or an
array contains the set of pebs which has a lnum equal to what we
set in translate struct. Original call like ubiformat set this
structure pointer to NULL and only ubidump use it. But if it's
better than writing a new function to do this stuff separately?
The code in ubi_scan is complicated now. Lnum translation is not
needed for ubiformat and the calculate of ec is not needed for
ubidump.

How about adding a new function to scan the whole MTD device and
just return a list contain all the pebs(mostly 1 or 2) in same
vol_id and same lnum?

in libscan.c

struct ubi_translate_info {
	int lnum,
	int vol_id,
	int find,	// counts of peb we find
	int *array	// dynamic or static(5? in size) array for
			// recording peb num
};

int ubi_translate(struct ubi_translate_info *info /* or **info */)
{
	// full scan the MTD device
	while (...) {
		read(ec);
		
		// determine if it is a valid peb
		...

		read(vid);

		if (vid->vol_id == info->vol_id &&
		    vid->lnum == info->lnum) {
			// add to list @array
			...

			find++;			
		}
	}

	return err; // MTD is bad or something else
}

Just quick thought.

What's your opinion?

By the way, I have to say this separating of UBI and UBIFS makes
it really hard to read data from both sides.

Re: [PATCH RFC v2] UBI: New ioctl() to support ubidump

[Bill Pringlemeir]

On  3 Aug 2014, hujianyang@huawei.com wrote:

> On 2014/8/1 23:35, Bill Pringlemeir wrote:

>> 1. Allocate a 'pmap' array and a temporary 'sequence' array.
>> 2. Initialize 'pmap' to -1.
>> 3a. For each valid 'eb', get 'lnum' and 'sqnum' from vid header.
>> b. if (pmap[lnum] == -1 || sequence[lnum] < sqnum)
>> pmap[lnum] = eb, sequence[lnum] = sqnum;

> I've researched the function ubi_scan() in libscan.c today and
> found it's not easy to just add small changes to enable LEB->PEB
> mapping table as we want.
>
> There are two methods on enabling this functionality:
> 1) Add a mapping table in struct ubi_scan_info
> 2) Add some new parameters for function ubi_scan()
>
> You know each MTD device may contain more than one UBI volume
> and each volume should has a private mapping table. So it's not
> easy to enable it in struct ubi_scan_info like method 1) because
> we don't know the actual counts of volumes.

Yes, I think you are completely correct.  The array only keeps track of
the erase count not the LEB.  Sorry, I was confused about that.

> Further more, @sqnum is not enough for peb determining, we should
> consider @on_copy flag for wear-leveling and atomic write. If
> @on_copy flag is set, we need to read the whole leb and check CRC
> to determine which peb is right. As this is a debugging tool, I
> think printing all pebs have same lnum(of course in same volume)
> is better. Now, we have to start thinking how to record this
> stuff, that's a big problem.

Validating the header CRC is always part of seeing if the block is
valid.  For a real physical device, the unstable bits maybe an issue.
So, it is probably better to examine 'copy_flag'.  Many of these are
abnormal cases, which need to be handled, but the tool would not
normally run through these paths for every block.  Running the CRC on
the data maybe particularly slow.

> In another way like method 2), we can add a new structure named
> ubi_translate, put lnum and vol_id in it and returns a list or an
> array contains the set of pebs which has a lnum equal to what we
> set in translate struct. Original call like ubiformat set this
> structure pointer to NULL and only ubidump use it. But if it's
> better than writing a new function to do this stuff separately?
> The code in ubi_scan is complicated now. Lnum translation is not
> needed for ubiformat and the calculate of ec is not needed for
> ubidump.

I agree it is complex.  It could be decomposed into functions.

> How about adding a new function to scan the whole MTD device and
> just return a list contain all the pebs(mostly 1 or 2) in same
> vol_id and same lnum?
>
> in libscan.c
>
> struct ubi_translate_info {
> 	int lnum,
> 	int vol_id,
> 	int find,	// counts of peb we find
> 	int *array	// dynamic or static(5? in size) array for
> 			// recording peb num
> };

> int ubi_translate(struct ubi_translate_info *info /* or **info */)
> {
> 	// full scan the MTD device
> 	while (...) {
> 		read(ec);
> 		
> 		// determine if it is a valid peb
> 		...
>
> 		read(vid);
>
> 		if (vid->vol_id == info->vol_id &&
> 		    vid->lnum == info->lnum) {
> 			// add to list @array
> 			...
>
> 			find++;			
> 		}
> 	}
>
> 	return err; // MTD is bad or something else
> }

ubi_translate() is more complex.  You need to read the VID header and
possibly data as well.  The ubi_scan() never reads these.  

However, a lot of the erase header checking will be the same.  So
besides all 'si->...=' lines most of ubi_scan() loop body is needed for
the 'ubi_translate()' to validate the erase header.  Then you need the
extra step to read the 'vid'.  I think you could have a function like,

/* ech invalid unless return < MAX_EC */
static uint32_t read_ehdr(struct mtd_dev_info *mtd, int fd, 
		int eb, struct ubi_ec_hdr *ech)
{
  ... 

That returns the 'enum' or erase count.  The 'enum' could be amended
with a 'fatal', which is the current 'goto out_ec' path (or you need
another parameter).  Then, the current ubi_scan() could be,

	for (eb = 0; eb < mtd->eb_cnt; eb++) {
		uint32_t ec;
		ec = read_ehdr(mtd, fd, eb, &ech);
                switch(ec) {
                   ...
                /* Do stuff to 'si->... here. */
			break;
		case EB_FATAL:
			goto out_ec;

And then ubi_translate() could reuse the read_ehdr() to do the
'determine if a valid peb' part.  Ie, whether you should even read the
'vid'.

> By the way, I have to say this separating of UBI and UBIFS makes
> it really hard to read data from both sides.

You mean both layers at once?  Ie, layers == sides?  Or do you mean to
support reading from an 'mtd' or a 'ubi' device?

Regards,
Bill Pringlemeir.