[PATCH RFC] ubi-utils: Add a new utility ubidump

[PATCH RFC] ubi-utils: Add a new utility ubidump

[hujianyang]

Hi all,

I would like to show my work about a new utility named ubidump.
This utility is different from jffs2dump. It can dump a specific
leb to user space(maybe a arrange of lebs in the future).

We have two logic layers in UBIFS and data on the eraseblocks
is separated into two parts: UBI header which include EC header
and VID header, UBIFS filesystem data which is formatted as
different kinds of NODEs.

We don't have a direct way to get both UBI info and UBIFS info
in the past(really?) because these two layers are transparent to
each other. Furthermore, when debugging with UBIFS, I found some
key information is invisible and I need to modify the kernel to
get them. For example, when I was dealing with commit 642f63e
(UBIFS: fix fatal race condition), I can't find a way to dump the
whole LOG area.

I think dynamic debug is a helpful mechanism but in some embedded
environment, kernel messages are stored in a ring buffer and are
wrapped commonly.

So I want to make a new utility to get data on eraseblocks in
user space and analysis filesystem behaviors in an easy way.
This utility can dump both UBI info and UBIFS info and all the
prints are formatted as UBI or UBIFS style.

I just want to make it a lighter utility but I have to copy lots
of ubifs-media codes and debug codes to mtd-utils. UBI headers are
got from a new ioctl(is there other way?) and UBIFS NODEs are got
from a simply read(), then print them by functions copied from
UBI/UBIFS debug.c in kernel.


Usage: ubidump <UBI volume node file name> [-l <lnum>] [-i]
			[--help] [--version]

Example 1: ubidump /dev/ubi0_1 -n 2 - dump leb 2 in volume 1
Example 2: ubidump /dev/ubi0_0 -n 1234 -i - dump leb 1234 with explicit info


-h, --help		print help message
-n, --lnum		logic eraseblock num to dump
-i, --info		show explicit information about NODEs
-H, --header		show only header information
-N, --node		show only NODEs information
-V, --version		print program version


# ./ubidump /dev/ubi0_0 -n 0 -i
lnum 0 is mapped to pnum 2054
Erase counter header dump:
	magic          0x55424923
	version        1
	ec             1
	vid_hdr_offset 256
	data_offset    512
	image_seq      1256176231
	hdr_crc        0x7758fc8b
Volume identifier header dump:
	magic     55424921
	version   1
	vol_type  1
	copy_flag 0
	compat    0
	vol_id    0
	lnum      0
	data_size 0
	used_ebs  0
	data_pad  0
	sqnum     8
	hdr_crc   29b2783a
scan LEB 0:0
look at LEB 0:0 (15872 bytes left)
scanning superblock node at LEB 0:0
	magic          0x6101831
	crc            0x88b345d6
	node_type      6 (superblock node)
	group_type     0 (no node group)
	sqnum          1
	len            4096
	key_hash       0 (R5)
	key_fmt        0 (simple)
	flags          0x2
	big_lpt      1
	space_fixup  0
	min_io_size    512
	leb_size       15872
	leb_cnt        8028
	max_leb_cnt    8028
	max_bud_bytes  5920256
	log_lebs       28
	lpt_lebs       11
	orph_lebs      2
	jhead_cnt      1
	fanout         8
	lsave_cnt      256
	default_compr  1
	rp_size        5242880
	rp_uid         0
	rp_gid         0
	fmt_version    4
	time_gran      1000000000
	UUID           0x105248cUB
look at LEB 0:4096 (11776 bytes left)
hit empty space at LEB 0:4096
stop scanning LEB 0 at offset 15872

[PATCH 1/7] UBI: Add a new ioctl to support ubidump

[hujianyang]

Add a new ioctl to dump EC header and VID header to user space. This
ioctl is called without lock so I think users *should not* run ubidump
when volume is mounted.

Struct ubi_ebdump_req is used to transfer data between user space and
kernel space. Can I find a way to set buffer length from '64' to
UBI_VID_HDR_SIZE and UBI_EC_HDR_SIZE?


Signed-off-by: hujianyang <hujianyang@huawei.com>
---
 drivers/mtd/ubi/cdev.c      | 50 +++++++++++++++++++++++++++++++++++++++++++++
 include/uapi/mtd/ubi-user.h | 15 ++++++++++++++
 2 files changed, 65 insertions(+)

diff --git a/drivers/mtd/ubi/cdev.c b/drivers/mtd/ubi/cdev.c
index 7646220..5b94323 100644
--- a/drivers/mtd/ubi/cdev.c
+++ b/drivers/mtd/ubi/cdev.c
@@ -581,6 +581,56 @@ static long vol_cdev_ioctl(struct file *file, unsigned int cmd,
 		break;
 	}

+	case UBI_IOCEBDUMP:
+	{
+		int pnum;
+		struct ubi_ebdump_req req;
+		struct ubi_ec_hdr *ec_hdr;
+		struct ubi_vid_hdr *vid_hdr;
+
+		err = copy_from_user(&req, argp,
+				     sizeof(struct ubi_ebdump_req));
+		if (err) {
+			err = -EFAULT;
+			break;
+		}
+
+		err = ubi_is_mapped(desc, req.lnum);
+		if (err <= 0)
+			break;
+		pnum = vol->eba_tbl[req.lnum];
+		req.pnum = pnum;
+
+		/*
+		 * No need to check return value of ubi_io_read(),
+		 * we will confirm the date in user space.
+		 */
+		ec_hdr = kzalloc(ubi->ec_hdr_alsize, GFP_NOFS);
+		if (!ec_hdr) {
+			err = -ENOMEM;
+			break;
+		}
+		ubi_io_read_ec_hdr(ubi, pnum, ec_hdr, 0);
+		memcpy(req.ec_hdr, ec_hdr, UBI_EC_HDR_SIZE);
+		kfree(ec_hdr);
+
+		vid_hdr = ubi_zalloc_vid_hdr(ubi, GFP_NOFS);
+		if (!vid_hdr) {
+			err = -ENOMEM;
+			break;
+		}
+		ubi_io_read_vid_hdr(ubi, pnum, vid_hdr, 1);
+		memcpy(req.vid_hdr, vid_hdr, UBI_VID_HDR_SIZE);
+		ubi_free_vid_hdr(ubi, vid_hdr);
+
+		err = copy_to_user(argp, &req,
+				   sizeof(struct ubi_ebdump_req));
+		if (err)
+			err = -EFAULT;
+
+		break;
+	}
+
 	default:
 		err = -ENOTTY;
 		break;
diff --git a/include/uapi/mtd/ubi-user.h b/include/uapi/mtd/ubi-user.h
index 1927b0d..76ff035 100644
--- a/include/uapi/mtd/ubi-user.h
+++ b/include/uapi/mtd/ubi-user.h
@@ -205,6 +205,8 @@
 #define UBI_IOCVOLCRBLK _IOW(UBI_VOL_IOC_MAGIC, 7, struct ubi_blkcreate_req)
 /* Remove the R/O block device */
 #define UBI_IOCVOLRMBLK _IO(UBI_VOL_IOC_MAGIC, 8)
+/* Dump LEB header */
+#define UBI_IOCEBDUMP _IOW(UBI_VOL_IOC_MAGIC, 9, struct ubi_ebdump_req)

 /* Maximum MTD device name length supported by UBI */
 #define MAX_UBI_MTD_NAME_LEN 127
@@ -442,4 +444,17 @@ struct ubi_blkcreate_req {
 	__s8  padding[128];
 }  __packed;

+/**
+ * struct ubi_ebdump_req - a data structure used in dump eraseblock header.
+ * @lnum: logical eraseblock num to dump
+ * @ec_hdr: ec_hdr to set
+ * @vid_hdr: vid_hdr to set
+ */
+struct ubi_ebdump_req {
+	__s32	lnum;
+	__s32	pnum;
+	char	ec_hdr[64];
+	char	vid_hdr[64];
+}  __packed;
+
 #endif /* __UBI_USER_H__ */
-- 
1.8.1.4

[PATCH 2/7] Add new ioctl in userspace

[hujianyang]

Add same ioctl in user space.


Signed-off-by: hujianyang <hujianyang@huawei.com>
---
 include/mtd/ubi-user.h | 15 +++++++++++++++
 ubi-utils/libubi.c     |  8 ++++++++
 2 files changed, 23 insertions(+)

diff --git a/include/mtd/ubi-user.h b/include/mtd/ubi-user.h
index 2b50dad..cc867a2 100644
--- a/include/mtd/ubi-user.h
+++ b/include/mtd/ubi-user.h
@@ -200,6 +200,8 @@
 #define UBI_IOCVOLCRBLK _IOW(UBI_VOL_IOC_MAGIC, 7, struct ubi_blkcreate_req)
 /* Remove the R/O block device */
 #define UBI_IOCVOLRMBLK _IO(UBI_VOL_IOC_MAGIC, 8)
+/* Dump LEB header */
+#define UBI_IOCEBDUMP _IOW(UBI_VOL_IOC_MAGIC, 9, struct ubi_ebdump_req)

 /* Maximum MTD device name length supported by UBI */
 #define MAX_UBI_MTD_NAME_LEN 127
@@ -437,4 +439,17 @@ struct ubi_blkcreate_req {
 	int8_t  padding[128];
 }  __attribute__((packed));

+/**
+ * struct ubi_ebdump_req - a data structure used in dump eraseblock header.
+ * @lnum: logical eraseblock num to dump
+ * @ec_hdr: ec_hdr to set
+ * @vid_hdr: vid_hdr to set
+ */
+struct ubi_ebdump_req {
+	int32_t lnum;
+	int32_t pnum;
+	char	ec_hdr[64];
+	char	vid_hdr[64];
+}  __attribute__((packed));
+
 #endif /* __UBI_USER_H__ */
diff --git a/ubi-utils/libubi.c b/ubi-utils/libubi.c
index 1e08b7d..39bfa1c 100644
--- a/ubi-utils/libubi.c
+++ b/ubi-utils/libubi.c
@@ -1123,6 +1123,14 @@ int ubi_vol_block_remove(int fd)
 	return ioctl(fd, UBI_IOCVOLRMBLK);
 }

+int ubi_leb_dump_header(libubi_t desc, int fd, struct ubi_ebdump_req *request)
+{
+	desc = desc;
+	if (ioctl(fd, UBI_IOCEBDUMP, request))
+		return -1;
+	return 0;
+}
+
 int ubi_update_start(libubi_t desc, int fd, long long bytes)
 {
 	desc = desc;
-- 
1.8.1.4

[PATCH 3/7] Add ubifs-media.h

[hujianyang]

Copy from fs/ubifs/ubifs-media.h but not the whole file, just
what I need(SB_NODE, MST_NODE, LOG_NODE, CH_NODE, PAD_NODE,
REF_NODE).

Could I add "Auther: Artem Bityutskiy" because these files are
based on his work.


Signed-off-by: hujianyang <hujianyang@huawei.com>
---
 include/mtd/ubifs-media.h | 295 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 295 insertions(+)
 create mode 100644 include/mtd/ubifs-media.h

diff --git a/include/mtd/ubifs-media.h b/include/mtd/ubifs-media.h
new file mode 100644
index 0000000..e1767fb
--- /dev/null
+++ b/include/mtd/ubifs-media.h
@@ -0,0 +1,295 @@
+/*
+ * Copyright (c) International Business Machines Corp., 2006
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
+ * the GNU General Public License for more details.
+ *
+ * Author: Hu Jianyang <hujianyang@huawei.com>
+ *
+ * UBIFS library.
+ */
+
+/*
+ * This file defines the layout of UBI headers and all the other UBIiFS
+ * on-flash data structures.
+ */
+
+#ifndef __UBIFS_MEDIA_H__
+#define __UBIFS_MEDIA_H__
+
+#include <asm/byteorder.h>
+
+/* UBIFS node magic number (must not have the padding byte first or last) */
+#define UBIFS_NODE_MAGIC  0x06101831
+
+/* UBIFS padding byte pattern (must not be first or last byte of node magic) */
+#define UBIFS_PADDING_BYTE 0xCE
+
+/* Node sizes (N.B. these are guaranteed to be multiples of 8) */
+#define UBIFS_CH_SZ        sizeof(struct ubifs_ch)
+#define UBIFS_PAD_NODE_SZ  sizeof(struct ubifs_pad_node)
+
+/*
+ * UBIFS node types.
+ *
+ * UBIFS_INO_NODE: inode node
+ * UBIFS_DATA_NODE: data node
+ * UBIFS_DENT_NODE: directory entry node
+ * UBIFS_XENT_NODE: extended attribute node
+ * UBIFS_TRUN_NODE: truncation node
+ * UBIFS_PAD_NODE: padding node
+ * UBIFS_SB_NODE: superblock node
+ * UBIFS_MST_NODE: master node
+ * UBIFS_REF_NODE: LEB reference node
+ * UBIFS_IDX_NODE: index node
+ * UBIFS_CS_NODE: commit start node
+ * UBIFS_ORPH_NODE: orphan node
+ * UBIFS_NODE_TYPES_CNT: count of supported node types
+ *
+ * Note, we index arrays by these numbers, so keep them low and contiguous.
+ * Node type constants for inodes, direntries and so on have to be the same as
+ * corresponding key type constants.
+ */
+enum {
+	UBIFS_INO_NODE,
+	UBIFS_DATA_NODE,
+	UBIFS_DENT_NODE,
+	UBIFS_XENT_NODE,
+	UBIFS_TRUN_NODE,
+	UBIFS_PAD_NODE,
+	UBIFS_SB_NODE,
+	UBIFS_MST_NODE,
+	UBIFS_REF_NODE,
+	UBIFS_IDX_NODE,
+	UBIFS_CS_NODE,
+	UBIFS_ORPH_NODE,
+	UBIFS_NODE_TYPES_CNT,
+};
+
+/*
+ * Supported key hash functions.
+ *
+ * UBIFS_KEY_HASH_R5: R5 hash
+ * UBIFS_KEY_HASH_TEST: test hash which just returns first 4 bytes of the name
+ */
+enum {
+	UBIFS_KEY_HASH_R5,
+	UBIFS_KEY_HASH_TEST,
+};
+
+/*
+ * Supported key formats.
+ *
+ * UBIFS_SIMPLE_KEY_FMT: simple key format
+ */
+enum {
+	UBIFS_SIMPLE_KEY_FMT,
+};
+
+/*
+ * Node group type (used by recovery to recover whole group or none).
+ *
+ * UBIFS_NO_NODE_GROUP: this node is not part of a group
+ * UBIFS_IN_NODE_GROUP: this node is a part of a group
+ * UBIFS_LAST_OF_NODE_GROUP: this node is the last in a group
+ */
+enum {
+	UBIFS_NO_NODE_GROUP = 0,
+	UBIFS_IN_NODE_GROUP,
+	UBIFS_LAST_OF_NODE_GROUP,
+};
+
+/*
+ * Superblock flags.
+ *
+ * UBIFS_FLG_BIGLPT: if "big" LPT model is used if set
+ * UBIFS_FLG_SPACE_FIXUP: first-mount "fixup" of free space within LEBs needed
+ */
+enum {
+	UBIFS_FLG_BIGLPT = 0x02,
+	UBIFS_FLG_SPACE_FIXUP = 0x04,
+};
+
+/**
+ * struct ubifs_ch - common header node.
+ * @magic: UBIFS node magic number (%UBIFS_NODE_MAGIC)
+ * @crc: CRC-32 checksum of the node header
+ * @sqnum: sequence number
+ * @len: full node length
+ * @node_type: node type
+ * @group_type: node group type
+ * @padding: reserved for future, zeroes
+ *
+ * Every UBIFS node starts with this common part. If the node has a key, the
+ * key always goes next.
+ */
+struct ubifs_ch {
+	__le32 magic;
+	__le32 crc;
+	__le64 sqnum;
+	__le32 len;
+	__u8 node_type;
+	__u8 group_type;
+	__u8 padding[2];
+} __attribute__ ((packed));
+
+/**
+ * struct ubifs_pad_node - padding node.
+ * @ch: common header
+ * @pad_len: how many bytes after this node are unused (because padded)
+ * @padding: reserved for future, zeroes
+ */
+struct ubifs_pad_node {
+	struct ubifs_ch ch;
+	__le32 pad_len;
+} __attribute__ ((packed));
+
+/**
+ * struct ubifs_sb_node - superblock node.
+ * @ch: common header
+ * @padding: reserved for future, zeroes
+ * @key_hash: type of hash function used in keys
+ * @key_fmt: format of the key
+ * @flags: file-system flags (%UBIFS_FLG_BIGLPT, etc)
+ * @min_io_size: minimal input/output unit size
+ * @leb_size: logical eraseblock size in bytes
+ * @leb_cnt: count of LEBs used by file-system
+ * @max_leb_cnt: maximum count of LEBs used by file-system
+ * @max_bud_bytes: maximum amount of data stored in buds
+ * @log_lebs: log size in logical eraseblocks
+ * @lpt_lebs: number of LEBs used for lprops table
+ * @orph_lebs: number of LEBs used for recording orphans
+ * @jhead_cnt: count of journal heads
+ * @fanout: tree fanout (max. number of links per indexing node)
+ * @lsave_cnt: number of LEB numbers in LPT's save table
+ * @fmt_version: UBIFS on-flash format version
+ * @default_compr: default compression algorithm (%UBIFS_COMPR_LZO, etc)
+ * @padding1: reserved for future, zeroes
+ * @rp_uid: reserve pool UID
+ * @rp_gid: reserve pool GID
+ * @rp_size: size of the reserved pool in bytes
+ * @padding2: reserved for future, zeroes
+ * @time_gran: time granularity in nanoseconds
+ * @uuid: UUID generated when the file system image was created
+ * @ro_compat_version: UBIFS R/O compatibility version
+ */
+struct ubifs_sb_node {
+	struct ubifs_ch ch;
+	__u8 padding[2];
+	__u8 key_hash;
+	__u8 key_fmt;
+	__le32 flags;
+	__le32 min_io_size;
+	__le32 leb_size;
+	__le32 leb_cnt;
+	__le32 max_leb_cnt;
+	__le64 max_bud_bytes;
+	__le32 log_lebs;
+	__le32 lpt_lebs;
+	__le32 orph_lebs;
+	__le32 jhead_cnt;
+	__le32 fanout;
+	__le32 lsave_cnt;
+	__le32 fmt_version;
+	__le16 default_compr;
+	__u8 padding1[2];
+	__le32 rp_uid;
+	__le32 rp_gid;
+	__le64 rp_size;
+	__le32 time_gran;
+	__u8 uuid[16];
+	__le32 ro_compat_version;
+	__u8 padding2[3968];
+} __attribute__ ((packed));
+
+/**
+ * struct ubifs_mst_node - master node.
+ * @ch: common header
+ * @highest_inum: highest inode number in the committed index
+ * @cmt_no: commit number
+ * @flags: various flags (%UBIFS_MST_DIRTY, etc)
+ * @log_lnum: start of the log
+ * @root_lnum: LEB number of the root indexing node
+ * @root_offs: offset within @root_lnum
+ * @root_len: root indexing node length
+ * @gc_lnum: LEB reserved for garbage collection (%-1 value means the LEB was
+ * not reserved and should be reserved on mount)
+ * @ihead_lnum: LEB number of index head
+ * @ihead_offs: offset of index head
+ * @index_size: size of index on flash
+ * @total_free: total free space in bytes
+ * @total_dirty: total dirty space in bytes
+ * @total_used: total used space in bytes (includes only data LEBs)
+ * @total_dead: total dead space in bytes (includes only data LEBs)
+ * @total_dark: total dark space in bytes (includes only data LEBs)
+ * @lpt_lnum: LEB number of LPT root nnode
+ * @lpt_offs: offset of LPT root nnode
+ * @nhead_lnum: LEB number of LPT head
+ * @nhead_offs: offset of LPT head
+ * @ltab_lnum: LEB number of LPT's own lprops table
+ * @ltab_offs: offset of LPT's own lprops table
+ * @lsave_lnum: LEB number of LPT's save table (big model only)
+ * @lsave_offs: offset of LPT's save table (big model only)
+ * @lscan_lnum: LEB number of last LPT scan
+ * @empty_lebs: number of empty logical eraseblocks
+ * @idx_lebs: number of indexing logical eraseblocks
+ * @leb_cnt: count of LEBs used by file-system
+ * @padding: reserved for future, zeroes
+ */
+struct ubifs_mst_node {
+	struct ubifs_ch ch;
+	__le64 highest_inum;
+	__le64 cmt_no;
+	__le32 flags;
+	__le32 log_lnum;
+	__le32 root_lnum;
+	__le32 root_offs;
+	__le32 root_len;
+	__le32 gc_lnum;
+	__le32 ihead_lnum;
+	__le32 ihead_offs;
+	__le64 index_size;
+	__le64 total_free;
+	__le64 total_dirty;
+	__le64 total_used;
+	__le64 total_dead;
+	__le64 total_dark;
+	__le32 lpt_lnum;
+	__le32 lpt_offs;
+	__le32 nhead_lnum;
+	__le32 nhead_offs;
+	__le32 ltab_lnum;
+	__le32 ltab_offs;
+	__le32 lsave_lnum;
+	__le32 lsave_offs;
+	__le32 lscan_lnum;
+	__le32 empty_lebs;
+	__le32 idx_lebs;
+	__le32 leb_cnt;
+	__u8 padding[344];
+} __attribute__ ((packed));
+
+/**
+ * struct ubifs_ref_node - logical eraseblock reference node.
+ * @ch: common header
+ * @lnum: the referred logical eraseblock number
+ * @offs: start offset in the referred LEB
+ * @jhead: journal head number
+ * @padding: reserved for future, zeroes
+ */
+struct ubifs_ref_node {
+	struct ubifs_ch ch;
+	__le32 lnum;
+	__le32 offs;
+	__le32 jhead;
+	__u8 padding[28];
+} __attribute__ ((packed));
+
+#endif /*!__UBIFS_MEDIA_H__ */
-- 
1.8.1.4

[PATCH 4/7] Add libubifs.h

[hujianyang]

Header file of libubifs.c.

Could I have other ways to get 'ALIGN' and 'min_t' instead
of define them in this file?


Signed-off-by: hujianyang <hujianyang@huawei.com>
---
 ubi-utils/include/libubifs.h | 87 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)
 create mode 100644 ubi-utils/include/libubifs.h

diff --git a/ubi-utils/include/libubifs.h b/ubi-utils/include/libubifs.h
new file mode 100644
index 0000000..94d259f
--- /dev/null
+++ b/ubi-utils/include/libubifs.h
@@ -0,0 +1,87 @@
+/*
+ * Copyright (c) International Business Machines Corp., 2006
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
+ * the GNU General Public License for more details.
+ *
+ * Author: Hu Jianyang <hujianyang@huawei.com>
+ *
+ * UBIFS library.
+ */
+
+#ifndef __LIBUBIFS_H__
+#define __LIBUBIFS_H__
+
+#include <mtd/ubifs-media.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#define ALIGN(x,a) __ALIGN_MASK(x,(typeof(x))(a)-1)
+#define __ALIGN_MASK(x,mask) (((x)+(mask))&~(mask))
+
+#define min_t(t,x,y) ({ \
+	typeof((x)) _x = (x); \
+	typeof((y)) _y = (y); \
+	(_x < _y) ? _x : _y; \
+})
+
+/*
+ * 'ubifs_scan_a_node()' return values.
+ *
+ * SCANNED_GARBAGE:  scanned garbage
+ * SCANNED_EMPTY_SPACE: scanned empty space
+ * SCANNED_A_NODE: scanned a valid node
+ * SCANNED_A_CORRUPT_NODE: scanned a corrupted node
+ * SCANNED_A_BAD_PAD_NODE: scanned a padding node with invalid pad length
+ *
+ * Greater than zero means: 'scanned that number of padding bytes'
+ */
+enum {
+	SCANNED_GARBAGE        = 0,
+	SCANNED_EMPTY_SPACE    = -1,
+	SCANNED_A_NODE         = -2,
+	SCANNED_A_CORRUPT_NODE = -3,
+	SCANNED_A_BAD_PAD_NODE = -4,
+};
+
+void ubifs_dump_node(const void *node);
+
+/**
+ * ubifs_scan_a_node - scan for a node or padding.
+ * @buf: buffer to scan
+ * @size: logical eraseblock size
+ * @len: length of buffer
+ * @lnum: logical eraseblock number
+ * @offs: offset within the logical eraseblock
+ *
+ * This function returns a scanning code to indicate what was scanned.
+ */
+int ubifs_scan_a_node(void *buf, int leb_size, int len, int lnum, int offs);
+
+/**
+ * ubifs_scan - scan a logical eraseblock.
+ * @lnum: logical eraseblock number
+ * @sbuf: scan buffer
+ * @size: size of @buf in bytes
+ * @offs: offset to start at (usually zero)
+ * @detailed: print NODEs detailed info
+ *
+ * This function scans LEB and prints complete information about
+ * its contents.
+ */
+void ubifs_scan(int lnum, void *buf, int leb_size, int offs, int detailed);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /*!__LIBUBIFS_H__ */
-- 
1.8.1.4

[PATCH 5/7] Add libubifs.c

[hujianyang]

These functions are copied from fs/ubifs/scan.c and fs/ubifs/debug.c
with some small changes. We don't have 'ubifs_info *c' in user space
and we don't need to check NODEs, just print what they are.

I think if data is corrupted, kernel will warn it and we can't check
it here because the checking codes are to heavy.

case SCANNED_A_NODE:
We will get the whole detailed info about this node if we set '-i'.

I just dump detailed info with some kinds of NODEs because I think
others are useless for debugging and you can simply add what you need
if it is necessary.


Signed-off-by: hujianyang <hujianyang@huawei.com>
---
 ubi-utils/libubifs.c | 399 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 399 insertions(+)
 create mode 100644 ubi-utils/libubifs.c

diff --git a/ubi-utils/libubifs.c b/ubi-utils/libubifs.c
new file mode 100644
index 0000000..86517fa
--- /dev/null
+++ b/ubi-utils/libubifs.c
@@ -0,0 +1,399 @@
+/*
+ * Copyright (c) International Business Machines Corp., 2006
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
+ * the GNU General Public License for more details.
+ *
+ * Author: Hu Jianyang <hujianyang@huawei.com>
+ *
+ * UBIFS library.
+ */
+
+#define PROGRAM_NAME "libubifs"
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <libubifs.h>
+#include <mtd_swab.h>
+#include "common.h"
+
+static const char *get_key_fmt(int fmt)
+{
+	switch (fmt) {
+	case UBIFS_SIMPLE_KEY_FMT:
+		return "simple";
+	default:
+		return "unknown/invalid format";
+	}
+}
+
+static const char *get_key_hash(int hash)
+{
+	switch (hash) {
+	case UBIFS_KEY_HASH_R5:
+		return "R5";
+	case UBIFS_KEY_HASH_TEST:
+		return "test";
+	default:
+		return "unknown/invalid name hash";
+	}
+}
+
+static const char *node_ntype(int type)
+{
+	switch (type) {
+	case UBIFS_PAD_NODE:
+		return "padding node";
+	case UBIFS_SB_NODE:
+		return "superblock node";
+	case UBIFS_MST_NODE:
+		return "master node";
+	case UBIFS_REF_NODE:
+		return "reference node";
+	case UBIFS_INO_NODE:
+		return "inode node";
+	case UBIFS_DENT_NODE:
+		return "direntry node";
+	case UBIFS_XENT_NODE:
+		return "xentry node";
+	case UBIFS_DATA_NODE:
+		return "data node";
+	case UBIFS_TRUN_NODE:
+		return "truncate node";
+	case UBIFS_IDX_NODE:
+		return "indexing node";
+	case UBIFS_CS_NODE:
+		return "commit start node";
+	case UBIFS_ORPH_NODE:
+		return "orphan node";
+	default:
+		return "unknown node";
+	}
+}
+
+static const char *node_gtype(int type)
+{
+	switch (type) {
+	case UBIFS_NO_NODE_GROUP:
+		return "no node group";
+	case UBIFS_IN_NODE_GROUP:
+		return "in node group";
+	case UBIFS_LAST_OF_NODE_GROUP:
+		return "last of node group";
+	default:
+		return "unknown";
+	}
+}
+
+static void dump_ch(const struct ubifs_ch *ch)
+{
+	printf("\tmagic          %#x\n", le32_to_cpu(ch->magic));
+	printf("\tcrc            %#x\n", le32_to_cpu(ch->crc));
+	printf("\tnode_type      %d (%s)\n", ch->node_type,
+	       node_ntype(ch->node_type));
+	printf("\tgroup_type     %d (%s)\n", ch->group_type,
+	       node_gtype(ch->group_type));
+	printf("\tsqnum          %llu\n",
+	       (unsigned long long)le64_to_cpu(ch->sqnum));
+	printf("\tlen            %u\n", le32_to_cpu(ch->len));
+}
+
+void ubifs_dump_node(const void *node)
+{
+	const struct ubifs_ch *ch = node;
+
+	if (le32_to_cpu(ch->magic) != UBIFS_NODE_MAGIC) {
+		printf("Not a node, first %zu bytes:\n", UBIFS_CH_SZ);
+		return;
+	}
+
+	dump_ch(node);
+
+	switch (ch->node_type) {
+	case UBIFS_PAD_NODE:
+	{
+		const struct ubifs_pad_node *pad = node;
+
+		printf("\tpad_len        %u\n", le32_to_cpu(pad->pad_len));
+		break;
+	}
+	case UBIFS_SB_NODE:
+	{
+		const struct ubifs_sb_node *sup = node;
+		unsigned int sup_flags = le32_to_cpu(sup->flags);
+
+		printf("\tkey_hash       %d (%s)\n",
+		       (int)sup->key_hash, get_key_hash(sup->key_hash));
+		printf("\tkey_fmt        %d (%s)\n",
+		       (int)sup->key_fmt, get_key_fmt(sup->key_fmt));
+		printf("\tflags          %#x\n", sup_flags);
+		printf("\tbig_lpt      %u\n",
+		       !!(sup_flags & UBIFS_FLG_BIGLPT));
+		printf("\tspace_fixup  %u\n",
+		       !!(sup_flags & UBIFS_FLG_SPACE_FIXUP));
+		printf("\tmin_io_size    %u\n", le32_to_cpu(sup->min_io_size));
+		printf("\tleb_size       %u\n", le32_to_cpu(sup->leb_size));
+		printf("\tleb_cnt        %u\n", le32_to_cpu(sup->leb_cnt));
+		printf("\tmax_leb_cnt    %u\n", le32_to_cpu(sup->max_leb_cnt));
+		printf("\tmax_bud_bytes  %llu\n",
+		       (unsigned long long)le64_to_cpu(sup->max_bud_bytes));
+		printf("\tlog_lebs       %u\n", le32_to_cpu(sup->log_lebs));
+		printf("\tlpt_lebs       %u\n", le32_to_cpu(sup->lpt_lebs));
+		printf("\torph_lebs      %u\n", le32_to_cpu(sup->orph_lebs));
+		printf("\tjhead_cnt      %u\n", le32_to_cpu(sup->jhead_cnt));
+		printf("\tfanout         %u\n", le32_to_cpu(sup->fanout));
+		printf("\tlsave_cnt      %u\n", le32_to_cpu(sup->lsave_cnt));
+		printf("\tdefault_compr  %u\n",
+		       (int)le16_to_cpu(sup->default_compr));
+		printf("\trp_size        %llu\n",
+		       (unsigned long long)le64_to_cpu(sup->rp_size));
+		printf("\trp_uid         %u\n", le32_to_cpu(sup->rp_uid));
+		printf("\trp_gid         %u\n", le32_to_cpu(sup->rp_gid));
+		printf("\tfmt_version    %u\n", le32_to_cpu(sup->fmt_version));
+		printf("\ttime_gran      %u\n", le32_to_cpu(sup->time_gran));
+		printf("\tUUID           %pUB\n", sup->uuid);
+		break;
+	}
+	case UBIFS_MST_NODE:
+	{
+		const struct ubifs_mst_node *mst = node;
+
+		printf("\thighest_inum   %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->highest_inum));
+		printf("\tcommit number  %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->cmt_no));
+		printf("\tflags          %#x\n", le32_to_cpu(mst->flags));
+		printf("\tlog_lnum       %u\n", le32_to_cpu(mst->log_lnum));
+		printf("\troot_lnum      %u\n", le32_to_cpu(mst->root_lnum));
+		printf("\troot_offs      %u\n", le32_to_cpu(mst->root_offs));
+		printf("\troot_len       %u\n", le32_to_cpu(mst->root_len));
+		printf("\tgc_lnum        %u\n", le32_to_cpu(mst->gc_lnum));
+		printf("\tihead_lnum     %u\n", le32_to_cpu(mst->ihead_lnum));
+		printf("\tihead_offs     %u\n", le32_to_cpu(mst->ihead_offs));
+		printf("\tindex_size     %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->index_size));
+		printf("\tlpt_lnum       %u\n", le32_to_cpu(mst->lpt_lnum));
+		printf("\tlpt_offs       %u\n", le32_to_cpu(mst->lpt_offs));
+		printf("\tnhead_lnum     %u\n", le32_to_cpu(mst->nhead_lnum));
+		printf("\tnhead_offs     %u\n", le32_to_cpu(mst->nhead_offs));
+		printf("\tltab_lnum      %u\n", le32_to_cpu(mst->ltab_lnum));
+		printf("\tltab_offs      %u\n", le32_to_cpu(mst->ltab_offs));
+		printf("\tlsave_lnum     %u\n", le32_to_cpu(mst->lsave_lnum));
+		printf("\tlsave_offs     %u\n", le32_to_cpu(mst->lsave_offs));
+		printf("\tlscan_lnum     %u\n", le32_to_cpu(mst->lscan_lnum));
+		printf("\tleb_cnt        %u\n", le32_to_cpu(mst->leb_cnt));
+		printf("\tempty_lebs     %u\n", le32_to_cpu(mst->empty_lebs));
+		printf("\tidx_lebs       %u\n", le32_to_cpu(mst->idx_lebs));
+		printf("\ttotal_free     %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->total_free));
+		printf("\ttotal_dirty    %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->total_dirty));
+		printf("\ttotal_used     %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->total_used));
+		printf("\ttotal_dead     %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->total_dead));
+		printf("\ttotal_dark     %llu\n",
+		       (unsigned long long)le64_to_cpu(mst->total_dark));
+		break;
+	}
+	case UBIFS_REF_NODE:
+	{
+		const struct ubifs_ref_node *ref = node;
+
+		printf("\tlnum           %u\n", le32_to_cpu(ref->lnum));
+		printf("\toffs           %u\n", le32_to_cpu(ref->offs));
+		printf("\tjhead          %u\n", le32_to_cpu(ref->jhead));
+		break;
+	}
+	case UBIFS_CS_NODE:
+		break;
+	case UBIFS_INO_NODE:
+	case UBIFS_DENT_NODE:
+	case UBIFS_XENT_NODE:
+	case UBIFS_DATA_NODE:
+	case UBIFS_TRUN_NODE:
+	case UBIFS_IDX_NODE:
+	case UBIFS_ORPH_NODE:
+		printf("cannot dump node, type not support\n");
+	default:
+		printf("node type %d was not recognized\n",
+		       (int)ch->node_type);
+	}
+}
+
+/**
+ * scan_padding_bytes - scan for padding bytes.
+ * @buf: buffer to scan
+ * @len: length of buffer
+ *
+ * This function returns the number of padding bytes on success and
+ * %SCANNED_GARBAGE on failure.
+ */
+static int scan_padding_bytes(void *buf, int len)
+{
+	int pad_len = 0, max_pad_len = min_t(int, UBIFS_PAD_NODE_SZ, len);
+	uint8_t *p = buf;
+
+	printf("not a node\n");
+
+	while (pad_len < max_pad_len && *p++ == UBIFS_PADDING_BYTE)
+		pad_len += 1;
+
+	if (!pad_len || (pad_len & 7))
+		return SCANNED_GARBAGE;
+
+	printf("%d padding bytes\n", pad_len);
+
+	return pad_len;
+}
+
+/**
+ * ubifs_scan_a_node - scan for a node or padding.
+ * @buf: buffer to scan
+ * @size: logical eraseblock size
+ * @len: length of buffer
+ * @lnum: logical eraseblock number
+ * @offs: offset within the logical eraseblock
+ *
+ * This function returns a scanning code to indicate what was scanned.
+ */
+int ubifs_scan_a_node(void *buf, int leb_size, int len, int lnum, int offs)
+{
+	struct ubifs_ch *ch = buf;
+	uint32_t magic;
+
+	magic = le32_to_cpu(ch->magic);
+
+	if (magic == 0xFFFFFFFF) {
+		printf("hit empty space at LEB %d:%d\n", lnum, offs);
+		return SCANNED_EMPTY_SPACE;
+	}
+
+	if (magic != UBIFS_NODE_MAGIC)
+		return scan_padding_bytes(buf, len);
+
+	if (len < UBIFS_CH_SZ)
+		return SCANNED_GARBAGE;
+
+	printf("scanning %s at LEB %d:%d\n",
+	       node_ntype(ch->node_type), lnum, offs);
+
+	/* No ubifs_check_nodei() perform here */
+
+	if (ch->node_type == UBIFS_PAD_NODE) {
+		struct ubifs_pad_node *pad = buf;
+		int pad_len = le32_to_cpu(pad->pad_len);
+		int node_len = le32_to_cpu(ch->len);
+
+		/* Validate the padding node */
+		if (pad_len < 0 ||
+		    offs + node_len + pad_len > leb_size) {
+			printf("bad pad node at LEB %d:%d\n", lnum, offs);
+			ubifs_dump_node(pad);
+			return SCANNED_A_BAD_PAD_NODE;
+		}
+
+		/* Make the node pads to 8-byte boundary */
+		if ((node_len + pad_len) & 7) {
+			printf("bad padding length %d - %d\n",
+			       offs, offs + node_len + pad_len);
+			return SCANNED_A_BAD_PAD_NODE;
+		}
+
+		printf("%d bytes padded at LEB %d:%d, offset now %d\n", pad_len,
+			lnum, offs, ALIGN(offs + node_len + pad_len, 8));
+
+		return node_len + pad_len;
+	}
+
+	return SCANNED_A_NODE;
+}
+
+/**
+ * ubifs_scan - scan a logical eraseblock.
+ * @lnum: logical eraseblock number
+ * @sbuf: scan buffer
+ * @size: size of @buf in bytes
+ * @offs: offset to start at (usually zero)
+ * @detailed: print NODEs detailed info
+ *
+ * This function scans LEB and prints complete information about
+ * its contents.
+ */
+void  ubifs_scan(int lnum, void *sbuf, int leb_size, int offs, int detailed)
+{
+	void *buf = sbuf + offs;
+	int err, len = leb_size - offs;
+
+	printf("scan LEB %d:%d\n", lnum, offs);
+
+	while (len >= 8) {
+		struct ubifs_ch *ch = buf;
+		int node_len, ret;
+
+		printf("look at LEB %d:%d (%d bytes left)\n",
+		       lnum, offs, len);
+
+		ret = ubifs_scan_a_node(buf, leb_size, len, lnum, offs);
+		if (ret > 0) {
+			/* Padding bytes or a valid padding node */
+			offs += ret;
+			buf += ret;
+			len -= ret;
+			continue;
+		}
+
+		if (ret == SCANNED_EMPTY_SPACE)
+			/* Empty space is checked later */
+			break;
+
+		switch (ret) {
+		case SCANNED_GARBAGE:
+			printf("garbage\n");
+			goto corrupted;
+		case SCANNED_A_NODE:
+			if (detailed)
+				ubifs_dump_node(buf);
+			break;
+		case SCANNED_A_CORRUPT_NODE:
+		case SCANNED_A_BAD_PAD_NODE:
+			printf("bad node\n");
+			goto corrupted;
+		default:
+			printf("unknown\n");
+			err = -EINVAL;
+			goto error;
+		}
+
+		node_len = ALIGN(le32_to_cpu(ch->len), 8);
+		offs += node_len;
+		buf += node_len;
+		len -= node_len;
+	}
+
+	for (; len > 4; offs += 4, buf = buf + 4, len -= 4)
+		if (*(uint32_t *)buf != 0xffffffff)
+			break;
+	for (; len; offs++, buf++, len--)
+		if (*(uint8_t *)buf != 0xff) {
+			printf("corrupt empty space at LEB %d:%d\n",
+			       lnum, offs);
+			goto corrupted;
+		}
+
+	printf("stop scanning LEB %d at offset %d\n", lnum, offs);
+	return;
+
+corrupted:
+	printf("corruption at LEB %d:%d\n", lnum, offs);
+	err = -EUCLEAN;
+error:
+	printf("LEB %d scanning failed, error %d\n", lnum, err);
+}
-- 
1.8.1.4

[PATCH 6/7] New utility ubidump

[hujianyang]

This utility is referred to ubiupdatevol.

I think I should move 'dump_header()' in this file to other
place but I don't know where.

This utility don't support dump binary date because I didn't
find a useful helper function like 'print_dump_hex()' in
kernel and kernel may dump the data which is corrupted so I
think no need to dump them here.


Signed-off-by: hujianyang <hujianyang@huawei.com>
---
 ubi-utils/ubidump.c | 287 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 287 insertions(+)
 create mode 100644 ubi-utils/ubidump.c

diff --git a/ubi-utils/ubidump.c b/ubi-utils/ubidump.c
new file mode 100644
index 0000000..70808a5
--- /dev/null
+++ b/ubi-utils/ubidump.c
@@ -0,0 +1,287 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
+ * the GNU General Public License for more details.
+ */
+
+/*
+ * An utility to dump UBI/UBIFS format data in eraseblock
+ *
+ * Author: Hu Jianyang <hujianyang@huawei.com>
+ */
+
+#define PROGRAM_NAME	"ubidump"
+
+#include <stdlib.h>
+#include <getopt.h>
+
+#include <libubi.h>
+#include <libubifs.h>
+#include <libmtd.h>
+#include <mtd_swab.h>
+#include "common.h"
+#include "ubiutils-common.h"
+
+/* The variables below are set by command line arguments */
+struct args {
+	const char *vol;
+	int lnum;
+	int info:1;
+	int header:1;
+	int node:1;
+};
+
+static struct args args =
+{
+	.vol	= NULL,
+	.lnum	= -1,
+	.info	= 0,
+	.header = 0,
+	.node   = 0,
+};
+
+static const char doc[] = PROGRAM_NAME " version " VERSION
+		" - an utility to dump UBI/UBIFS format data in eraseblock";
+
+static const char optionsstr[] =
+"-h, --help		print help message\n"
+"-n, --lnum		logic eraseblock num to dump\n"
+"-i, --info		show explicit information about NODEs\n"
+"-H, --header		show only header information\n"
+"-N, --node		show only NODEs information\n"
+"-V, --version		print program version";
+
+static const char usage[] =
+"Usage: " PROGRAM_NAME " <UBI volume node file name> [-l <lnum>] [-i]\n"
+"\t\t\t[--help] [--version]\n\n"
+"Example 1: " PROGRAM_NAME " /dev/ubi0_1 -n 2 - dump leb 2 in volume 1\n"
+"Example 2: " PROGRAM_NAME " /dev/ubi0_0 -n 1234 -i - dump leb 1234 with explicit info\n";
+
+static const struct option long_options[] = {
+	{ .name = "help",    .has_arg = 0, .flag = NULL, .val = 'h' },
+	{ .name = "lnum",    .has_arg = 1, .flag = NULL, .val = 'n' },
+	{ .name = "info",    .has_arg = 0, .flag = NULL, .val = 'i' },
+	{ .name = "header",  .has_arg = 0, .flag = NULL, .val = 'H' },
+	{ .name = "node",    .has_arg = 0, .flag = NULL, .val = 'N' },
+	{ .name = "version", .has_arg = 0, .flag = NULL, .val = 's' },
+	{ NULL, 0, NULL, 0}
+};
+
+static int parse_opt(int argc, char * const argv[])
+{
+	while (1) {
+		int key, error = 0;
+
+		key = getopt_long(argc, argv, "h?in:HN", long_options, NULL);
+		if (key == -1)
+			break;
+
+		switch (key) {
+		case 'n':
+			args.lnum = simple_strtoul(optarg, &error);
+			if (error || args.lnum < 0)
+				return errmsg("bad lnum: \"%s\"", optarg);
+			break;
+
+		case 'i':
+			args.info = 1;
+			break;
+
+		case 'H':
+			args.header = 1;
+			break;
+
+		case 'N':
+			args.node = 1;
+			break;
+
+		case 'h':
+		case '?':
+			printf("%s\n\n", doc);
+			printf("%s\n\n", usage);
+			printf("%s\n", optionsstr);
+			exit(EXIT_SUCCESS);
+
+		case 'V':
+			common_print_version();
+			exit(EXIT_SUCCESS);
+
+		case ':':
+			return errmsg("parameter is missing");
+
+		default:
+			fprintf(stderr, "Use -h for help\n");
+			return -1;
+		}
+	}
+	if (optind == argc)
+		return errmsg("UBI device name was not specified (use -h for help)");
+	else if (optind != argc - 1)
+		return errmsg("more then one UBI device specified (use -h for help)");
+
+	args.vol = argv[optind];
+
+	if (args.lnum < 0)
+		return errmsg("lnum was not specified (use -h for help)");
+	if (args.header && (args.info || args.node))
+		return errmsg("cannot specify -H with -i or -N");
+
+	return 0;
+}
+
+static void dump_header(struct ubi_ebdump_req *header)
+{
+	struct ubi_ec_hdr *ec_hdr = (struct ubi_ec_hdr *)header->ec_hdr;
+	struct ubi_vid_hdr *vid_hdr = (struct ubi_vid_hdr *)header->vid_hdr;
+
+	printf("lnum %d is mapped to pnum %d\n", header->lnum, header->pnum);
+
+	printf("Erase counter header dump:\n");
+	printf("\tmagic          %#08x\n", be32_to_cpu(ec_hdr->magic));
+	printf("\tversion        %d\n", (int)ec_hdr->version);
+	printf("\tec             %llu\n", (long long)be64_to_cpu(ec_hdr->ec));
+	printf("\tvid_hdr_offset %d\n", be32_to_cpu(ec_hdr->vid_hdr_offset));
+	printf("\tdata_offset    %d\n", be32_to_cpu(ec_hdr->data_offset));
+	printf("\timage_seq      %d\n", be32_to_cpu(ec_hdr->image_seq));
+	printf("\thdr_crc        %#08x\n", be32_to_cpu(ec_hdr->hdr_crc));
+
+	printf("Volume identifier header dump:\n");
+	printf("\tmagic     %08x\n", be32_to_cpu(vid_hdr->magic));
+	printf("\tversion   %d\n",  (int)vid_hdr->version);
+	printf("\tvol_type  %d\n",  (int)vid_hdr->vol_type);
+	printf("\tcopy_flag %d\n",  (int)vid_hdr->copy_flag);
+	printf("\tcompat    %d\n",  (int)vid_hdr->compat);
+	printf("\tvol_id    %d\n",  be32_to_cpu(vid_hdr->vol_id));
+	printf("\tlnum      %d\n",  be32_to_cpu(vid_hdr->lnum));
+	printf("\tdata_size %d\n",  be32_to_cpu(vid_hdr->data_size));
+	printf("\tused_ebs  %d\n",  be32_to_cpu(vid_hdr->used_ebs));
+	printf("\tdata_pad  %d\n",  be32_to_cpu(vid_hdr->data_pad));
+	printf("\tsqnum     %llu\n",
+		(unsigned long long)be64_to_cpu(vid_hdr->sqnum));
+	printf("\thdr_crc   %08x\n", be32_to_cpu(vid_hdr->hdr_crc));
+}
+
+static int dump_leb(libubi_t libubi, int fd, struct ubi_vol_info *vol_info)
+{
+	int ret, leb_size, lnum;
+	char *buf;
+
+	leb_size = vol_info->leb_size;
+	lnum = args.lnum;
+
+	ret = ubi_is_mapped(fd, lnum);
+	if (ret == 0) {
+		errmsg("lnum %d is not mapped", lnum);
+		goto out;
+	} else if (ret < 0) {
+		sys_errmsg("ubi_is_mapped() failed");
+		goto out;
+	}
+
+	/* dump peb header */
+	if (!args.node) {
+		struct ubi_ebdump_req request;
+
+		memset(&request, 0, sizeof(struct ubi_ebdump_req));
+		request.lnum = lnum;
+		request.pnum = -1;
+		ret = ubi_leb_dump_header(libubi, fd, &request);
+		if (ret < 0) {
+			sys_errmsg("cannot dump header of lnum %d", lnum);
+			goto out;
+		} else if (request.pnum == -1) {
+			errmsg("lnum %d is not mapped", lnum);
+			goto out;
+		}
+
+		dump_header(&request);
+	}
+
+	/* dump NODEs */
+	if (!args.header) {
+		off_t offs = lnum * leb_size;
+
+		buf = malloc(leb_size);
+		if (!buf)
+			return errmsg("cannot allocate %d bytes of memory",
+				      leb_size);
+
+		ret = pread(fd, buf, leb_size, offs);
+		if (ret != leb_size) {
+			errmsg("cannot read %d bytes at lnum %d, read %d",
+			       leb_size, lnum, ret);
+			goto out_free;
+		}
+		ubifs_scan(lnum, buf, leb_size, 0, args.info);
+	}
+
+out_free:
+	free(buf);
+out:
+	return ret;
+}
+
+int main(int argc, char * const argv[])
+{
+	int err, fd;
+	libubi_t libubi;
+	struct ubi_vol_info vol_info;
+
+	err = parse_opt(argc, argv);
+	if (err)
+		return -1;
+
+	libubi = libubi_open();
+	if (!libubi) {
+		if (errno == 0)
+			errmsg("UBI is not present in the system");
+		else
+			sys_errmsg("cannot open libubi");
+		goto out_libubi;
+	}
+
+	err = ubi_probe_node(libubi, args.vol);
+	if (err == 1) {
+		errmsg("\"%s\" is an UBI device node, not an UBI volume node",
+		       args.vol);
+		goto out_libubi;
+	} else if (err < 0) {
+		if (errno == ENODEV)
+			errmsg("\"%s\" is not an UBI volume node", args.vol);
+		else
+			sys_errmsg("error while probing \"%s\"", args.vol);
+		goto out_libubi;
+	}
+
+	err = ubi_get_vol_info(libubi, args.vol, &vol_info);
+	if (err) {
+		sys_errmsg("cannot get information about UBI volume \"%s\"",
+			   args.vol);
+		goto out_libubi;
+	}
+
+	fd = open(args.vol, O_RDONLY);
+	if (fd == -1) {
+		sys_errmsg("cannot open UBI volume \"%s\"", args.vol);
+		goto out_libubi;
+	}
+	err = dump_leb(libubi, fd, &vol_info);
+	if (err)
+		goto out_fd;
+
+	close(fd);
+	libubi_close(libubi);
+	return 0;
+
+out_fd:
+	close(fd);
+out_libubi:
+	libubi_close(libubi);
+	return -1;
+}
-- 
1.8.1.4

[PATCH 7/7] Compile support

[hujianyang]

Compile enable.


Signed-off-by: hujianyang <hujianyang@huawei.com>
---
 Makefile | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index eade234..171976c 100644
--- a/Makefile
+++ b/Makefile
@@ -28,7 +28,8 @@ MTD_BINS = \
 	sumtool jffs2reader
 UBI_BINS = \
 	ubiupdatevol ubimkvol ubirmvol ubicrc32 ubinfo ubiattach \
-	ubidetach ubinize ubiformat ubirename mtdinfo ubirsvol ubiblock
+	ubidetach ubinize ubiformat ubirename mtdinfo ubirsvol ubiblock \
+	ubidump

 BINS = $(MTD_BINS)
 BINS += mkfs.ubifs/mkfs.ubifs
@@ -114,11 +115,12 @@ $(call mkdep,mkfs.ubifs/,mkfs.ubifs,,ubi-utils/libubi.a)
 obj-libiniparser.a = libiniparser.o dictionary.o
 obj-libscan.a      = libscan.o
 obj-libubi.a       = libubi.o
+obj-libubifs.a     = libubifs.o
 obj-libubigen.a    = libubigen.o

 obj-mtdinfo   = libubigen.a
 obj-ubinize   = libubigen.a libiniparser.a
 obj-ubiformat = libubigen.a libscan.a

-$(foreach v,libubi.a libubigen.a libiniparser.a libscan.a,$(eval $(call _mkdep,ubi-utils/,$(v))))
-$(foreach v,$(UBI_BINS),$(eval $(call mkdep,ubi-utils/,$(v),libubi.a ubiutils-common.o)))
+$(foreach v,libubi.a libubigen.a libiniparser.a libscan.a libubifs,$(eval $(call _mkdep,ubi-utils/,$(v))))
+$(foreach v,$(UBI_BINS),$(eval $(call mkdep,ubi-utils/,$(v),libubi.a libubifs.a ubiutils-common.o)))
-- 
1.8.1.4

Re: [PATCH 1/7] UBI: Add a new ioctl to support ubidump

[Artem Bityutskiy]

On Mon, 2014-07-07 at 15:17 +0800, hujianyang wrote:
> Add a new ioctl to dump EC header and VID header to user space. This
> ioctl is called without lock so I think users *should not* run ubidump
> when volume is mounted.
> 
> Struct ubi_ebdump_req is used to transfer data between user space and
> kernel space. Can I find a way to set buffer length from '64' to
> UBI_VID_HDR_SIZE and UBI_EC_HDR_SIZE?
> 
> 
> Signed-off-by: hujianyang <hujianyang@huawei.com>

I've never seen an ioctl for retrieving an on-the-media data structure,
do you have any examples? I think the traditional way is that the
user-space reads all the information from the disk itself.

Do you really need to dump UBI headers?

Could you please drop this functionality for now, and provide an ubidump
which only dumps UBIFS data structures for now.

Then we can talk and think about the UBI-level headers dumping.

-- 
Best Regards,
Artem Bityutskiy

Re: [PATCH 6/7] New utility ubidump

[Artem Bityutskiy]

On Mon, 2014-07-07 at 15:26 +0800, hujianyang wrote:
> +"-h, --help		print help message\n"
> +"-n, --lnum		logic eraseblock num to dump\n"

I envision that in the future people may want to specify PEB number
instead of LEB number, and '-n' becomes a confusing name for the option.
Could you please only support '--lnum', and drop '-n' altogether. Then
in the future someone may add '--pnum' support.

> +"-i, --info		show explicit information about NODEs\n"

I do not understand what this means from the description. Would you
please try to describe it better in the help text?

> +"-H, --header		show only header information\n"
> +"-N, --node		show only NODEs information\n"

Same for these.

Thanks!

-- 
Best Regards,
Artem Bityutskiy

Re: [PATCH 1/7] UBI: Add a new ioctl to support ubidump

[hujianyang]

Hi Artem,

Thank you for reviewing these patches.

> 
> Do you really need to dump UBI headers?
>

Yes, because I think ec_counter in ec_hdr and copy_flag, sqnum
in vid_hdr are useful.

Current ->read() can just read data from data_offset where the
leb located. And reading UBI headers by mtd functionality are
very complicated, we don't know pnum either.

Maybe we can try other way instead of this ioctl, by adding an
new function which can read the hole peb specified by lnum to
user space.

Re: [PATCH 6/7] New utility ubidump

[hujianyang]

> I envision that in the future people may want to specify PEB number
> instead of LEB number, and '-n' becomes a confusing name for the option.
> Could you please only support '--lnum', and drop '-n' altogether. Then
> in the future someone may add '--pnum' support.

Yes, you are right. I'll change it in next version.

> 
>> +"-i, --info		show explicit information about NODEs\n"
> 
> I do not understand what this means from the description. Would you
> please try to describe it better in the help text?
> 
>> +"-H, --header		show only header information\n"
>> +"-N, --node		show only NODEs information\n"
> 
> Same for these.
> 
> Thanks!
> 

-H for UBI-level info and -N for UBIFS-level info.
-i means dump UBIFS Nodes by function ubifs_dump_node, not just
scan them.

I will re-describe them.

Thanks.

Re: [PATCH 1/7] UBI: Add a new ioctl to support ubidump

[Artem Bityutskiy]

On Wed, 2014-07-16 at 16:47 +0800, hujianyang wrote:
> Hi Artem,
> 
> Thank you for reviewing these patches.
> 
> > 
> > Do you really need to dump UBI headers?
> >
> 
> Yes, because I think ec_counter in ec_hdr and copy_flag, sqnum
> in vid_hdr are useful.
> 
> Current ->read() can just read data from data_offset where the
> leb located. And reading UBI headers by mtd functionality are
> very complicated, we don't know pnum either.
> 
> Maybe we can try other way instead of this ioctl, by adding an
> new function which can read the hole peb specified by lnum to
> user space.

Yes, this is a lot better approach. But I'd still suggest to start with
a smaller step - ubidump without the UBI headers dumping support. Then
add the UBI headers dumping support as a separate step.

-- 
Best Regards,
Artem Bityutskiy

Re: [PATCH 6/7] New utility ubidump

[Artem Bityutskiy]

On Wed, 2014-07-16 at 16:53 +0800, hujianyang wrote:
> > I envision that in the future people may want to specify PEB number
> > instead of LEB number, and '-n' becomes a confusing name for the option.
> > Could you please only support '--lnum', and drop '-n' altogether. Then
> > in the future someone may add '--pnum' support.
> 
> Yes, you are right. I'll change it in next version.
> 
> > 
> >> +"-i, --info		show explicit information about NODEs\n"
> > 
> > I do not understand what this means from the description. Would you
> > please try to describe it better in the help text?
> > 
> >> +"-H, --header		show only header information\n"
> >> +"-N, --node		show only NODEs information\n"
> > 
> > Same for these.
> > 
> > Thanks!
> > 
> 
> -H for UBI-level info and -N for UBIFS-level info.
> -i means dump UBIFS Nodes by function ubifs_dump_node, not just
> scan them.
> 
> I will re-describe them.

Things like UBI volume table, UBI fast-map stuff are not "headers", so I
am not sure if using word "headers" a good idea. Probably we want
options like --ubifs and --ubi to denote ubi and ubifs-level stuff. The
default would be "everything".

But again, if you start smaller, and upstream a good tool for
UBIFS-level stuff, it will be easier to add UBI stuff separately.

Besides, I have some additional vision, which you do not have to
implement, but which should be taken into account. E.g., ubidump which
does not need UBI/UBIFS drivers, ubidump which can deal with an image
generated with nanddump without "mounting" it, etc. So I was thinking
doing small steps at a time would make it easier for me and for you to
make a tool which has limited functionality today, but which can be
later extended to support more functionality.

-- 
Best Regards,
Artem Bityutskiy

Re: [PATCH 6/7] New utility ubidump

[hujianyang]

> 
> Things like UBI volume table, UBI fast-map stuff are not "headers", so I
> am not sure if using word "headers" a good idea. Probably we want
> options like --ubifs and --ubi to denote ubi and ubifs-level stuff. The
> default would be "everything".
> 
> But again, if you start smaller, and upstream a good tool for
> UBIFS-level stuff, it will be easier to add UBI stuff separately.
> 
> Besides, I have some additional vision, which you do not have to
> implement, but which should be taken into account. E.g., ubidump which
> does not need UBI/UBIFS drivers, ubidump which can deal with an image
> generated with nanddump without "mounting" it, etc. So I was thinking
> doing small steps at a time would make it easier for me and for you to
> make a tool which has limited functionality today, but which can be
> later extended to support more functionality.
> 

It seems you what more than me. I think you are right. I need to
reconsider how to realize these above, not hole of them, but a
good architecture that can be extended easily.

So I think getting data from mtd driver is a good choice and then
run it with an image file. Current UBI functionality has lots of
limit and it is basing on UBI driver. But we need to do more work
in user space (rebuilding volume table and so on) in this way. I
think it's worth.

Give me some time to re-create this utility. I will send it to you
if I finished it. If you get some new ideas, please tell me as soon
as possible.

Thanks~!

Hu

Re: [PATCH 6/7] New utility ubidump

[Artem Bityutskiy]

On Wed, 2014-07-16 at 19:27 +0800, hujianyang wrote:
> > 
> > Things like UBI volume table, UBI fast-map stuff are not "headers", so I
> > am not sure if using word "headers" a good idea. Probably we want
> > options like --ubifs and --ubi to denote ubi and ubifs-level stuff. The
> > default would be "everything".
> > 
> > But again, if you start smaller, and upstream a good tool for
> > UBIFS-level stuff, it will be easier to add UBI stuff separately.
> > 
> > Besides, I have some additional vision, which you do not have to
> > implement, but which should be taken into account. E.g., ubidump which
> > does not need UBI/UBIFS drivers, ubidump which can deal with an image
> > generated with nanddump without "mounting" it, etc. So I was thinking
> > doing small steps at a time would make it easier for me and for you to
> > make a tool which has limited functionality today, but which can be
> > later extended to support more functionality.
> > 
> 
> It seems you what more than me. I think you are right. I need to
> reconsider how to realize these above, not hole of them, but a
> good architecture that can be extended easily.
> 
> So I think getting data from mtd driver is a good choice and then
> run it with an image file. Current UBI functionality has lots of
> limit and it is basing on UBI driver. But we need to do more work
> in user space (rebuilding volume table and so on) in this way. I
> think it's worth.
> 
> Give me some time to re-create this utility. I will send it to you
> if I finished it. If you get some new ideas, please tell me as soon
> as possible.

I am not trying to make you do a lot more than you need.

If you are analyzing an MTD image without "mounting" it, the only way to
find out the LEB->PEB mapping is to do full scan, and basically copy the
UBI driver code. This is a lot of work.

In your case you have the driver, and it is fine to add an ioctl which
provides LEB->PEB mapping. This is a lot simpler.

So I envision that the tool would work like this.

$ ubidump my.img --lnum 5 
  - dump LEB 5, will need to scan the entire image.

$ ubidump /dev/ubiX_Y --lnum 5
  - will just ask the UBI driver to give the PEB number for LEB 5, then
find out the MTD device for this volume (should be possible by checking
the sysfs files), and then reads the MTD device, and gets the UBI-level
information from there.

Something like this, just quick thoughts.

-- 
Best Regards,
Artem Bityutskiy

Re: [PATCH 6/7] New utility ubidump

[Artem Bityutskiy]

On Wed, 2014-07-16 at 19:27 +0800, hujianyang wrote:
> If you get some new ideas, please tell me as soon
> as possible.

The problem is that I do not know if I have more ideas. You may spend a
lot of time on this stuff and then I send one of my ideas may render
your work useless.

This is why I suggest to start small, do just the UBIFS nodes dump,
nothing else. Just forget about UBI. Then let's review that and make
part of mtd-utils. This will clarify the vision, and we may add UBI
support then separately.


-- 
Best Regards,
Artem Bityutskiy

Re: [PATCH 6/7] New utility ubidump

[hujianyang]

> 
> This is why I suggest to start small, do just the UBIFS nodes dump,
> nothing else. Just forget about UBI. Then let's review that and make
> part of mtd-utils. This will clarify the vision, and we may add UBI
> support then separately.
>

So just use simple way to do UBIFS dump and then step by step
make it a strong utility.

It's not too hard. I will first realize an UBIFS dump as you
showed by mtd driver.

Re: [PATCH 6/7] New utility ubidump

[Bill Pringlemeir]

[-- Attachment #1: Type: text/plain, Size: 2369 bytes --]

On 16 Jul 2014, hujianyang at huawei.com wrote:

>> But again, if you start smaller, and upstream a good tool for
>> UBIFS-level stuff, it will be easier to add UBI stuff separately.

>> Besides, I have some additional vision, which you do not have to
>> implement, but which should be taken into account. E.g., ubidump
>> which does not need UBI/UBIFS drivers, ubidump which can deal with an
>> image generated with nanddump without "mounting" it, etc. So I was
>> thinking doing small steps at a time would make it easier for me and
>> for you to make a tool which has limited functionality today, but
>> which can be later extended to support more functionality.

> It seems you what more than me. I think you are right. I need to
> reconsider how to realize these above, not hole of them, but a
> good architecture that can be extended easily.

> So I think getting data from mtd driver is a good choice and then
> run it with an image file. Current UBI functionality has lots of
> limit and it is basing on UBI driver. But we need to do more work
> in user space (rebuilding volume table and so on) in this way. I
> think it's worth.

> Give me some time to re-create this utility. I will send it to you
> if I finished it. If you get some new ideas, please tell me as soon
> as possible.

I started some thing like this.  The source is attached.  As really you
only want 'read only' access, there is a minimal portion of the
UBI/UbiFs code that is needed.  I think that only the 'ubi-media.h' is
needed.  The Linux is high performance and handles read/write with
different fault conditions.

If you write code for read-only/single thread I think it will be much
more simple than the active UBI/UbiFS code in the Linux kernel.  I did
not look at the UbiFS layer as much.  It is far more complex that UBI
imho; of course, I just saw a little of the internal volumes and the
fast map features.  However, fast map itself should not be needed for
this utility.

For your use case of analysis in a running system, I think that reads of
'/dev/mtd0ro', etc can be used.  This way recovery features can also be
developed if we identified some inconsistency; Ie, it is not required to
'mount' the volume before analysis.

The attached source just scans an file image and create an leb/peb
mapping.  I only used the 'ubi-media.h' as documentation.

Fwiw,
Bill Pringlemeir.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Stupid source that processes raw dump of UBI layer. --]
[-- Type: text/x-csrc, Size: 13046 bytes --]

/* -*- mode: c; compile-command: "gcc -Wall -g -o parse_ubi parse_ubi.c"; -*- */

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <endian.h>
#include <string.h>
#include <stdlib.h>
#include <ctype.h>

#define __packed __attribute__((packed))
#include "ubi-media.h"

#define bswap16 be16toh
#define bswap32 be32toh
#define bswap64 be64toh

static int dump_vid = 0;

#define CRCPOLY_LE 0xedb88320
static unsigned int crc32(unsigned int crc, void const *_p, size_t len)
{
    unsigned char const *p = _p;
	int i;
	while (len--) {
		crc ^= *p++;
		for (i = 0; i < 8; i++)
			crc = (crc >> 1) ^ ((crc & 1) ? CRCPOLY_LE : 0);
	}
	return crc;
}

#define ALEN(a) (sizeof(a)/sizeof(a[0]))
static void print_ec(struct ubi_ec_hdr *ec)
{
    if(ec->version != UBI_VERSION || ec->magic != UBI_EC_HDR_MAGIC) {
        printf(" Magic: %x\n", ec->magic);
        printf(" Version: %d\n", (int)ec->version);
        printf(" EC: %llx\n", ec->ec);
        printf(" VID offset: %x\n", ec->vid_hdr_offset);
        printf(" Data offset: %x\n", ec->data_offset);
        printf(" Image seq: %x\n", ec->image_seq);
        exit(-1);
    }
}

static void read_ec(int fd, struct ubi_ec_hdr *ec)
{
    int rval = read(fd, ec,sizeof(*ec));
    if(rval == sizeof(*ec)) {
        unsigned int crc;
        crc = crc32(UBI_CRC32_INIT, ec, UBI_EC_HDR_SIZE_CRC);
        ec->magic = bswap32(ec->magic);
        ec->vid_hdr_offset = bswap32(ec->vid_hdr_offset);
        ec->data_offset = bswap32(ec->data_offset);
        ec->image_seq = bswap32(ec->image_seq);
        ec->hdr_crc = bswap32(ec->hdr_crc);
        ec->ec = bswap64(ec->ec);
        if(crc != ec->hdr_crc)
            printf("EC CRC: %x/%x\n", crc, ec->hdr_crc);
    } else
        memset(ec, 0, sizeof(*ec));
}

static void print_vid(int vid_num, struct ubi_vid_hdr *vid)
{
    if(vid->magic != UBI_VID_HDR_MAGIC)
        printf(" Magic: %x\n", vid->magic);
    if(vid->version != UBI_VERSION)
        printf(" Version: %d\n", (int)vid->version);

    if(!dump_vid) return;
    
    printf("VID %d\n", vid_num);

    /* This is usually the same. */
    if(vid->vol_id >= UBI_INTERNAL_VOL_START)
        printf("Internal vol_id: %d\n", vid->vol_id - UBI_INTERNAL_VOL_START);
    if(vid->vol_type != UBI_VID_DYNAMIC)
        printf(" vol_type: %s\n",
               vid->vol_type == UBI_VID_DYNAMIC ? "dynamic" : "static");
    if(vid->used_ebs)
        printf(" used_ebs: %d\n", vid->used_ebs);
    if(vid->data_pad)
        printf(" data_pad: %d\n", vid->data_pad);
    if((vid->copy_flag != 1 && vid->data_size) ||
       (vid->copy_flag == 0 && vid->data_size))
        printf(" copy_flag: %d\n", (int)vid->copy_flag);

    printf(" lnum: %d\n", vid->lnum);
    if(vid->compat) {
        const char *compat[] = {
            [UBI_COMPAT_DELETE]   = "delete",
            [UBI_COMPAT_RO]       = "ro",
            [UBI_COMPAT_PRESERVE] = "preserve",
            [UBI_COMPAT_REJECT]   = "reject"
        };
        printf(" compat: %s\n", compat[vid->compat]);
    }
    printf(" data_size: %d\n", vid->data_size);
    /* printf(" data_crc: %x\n", vid->data_crc); */
    printf(" hdr_crc: %x\n", vid->hdr_crc);
    printf(" sqnum: %lld\n", vid->sqnum);
}

static int read_vid(int fd, struct ubi_vid_hdr *vid)
{
    int rval = read(fd, vid,sizeof(*vid));
    if(rval == sizeof(*vid)) {
        unsigned int crc;
        crc = crc32(UBI_CRC32_INIT, vid, UBI_EC_HDR_SIZE_CRC);
        vid->magic = bswap32(vid->magic);
        vid->vol_id = bswap32(vid->vol_id);
        vid->lnum = bswap32(vid->lnum);
        vid->data_size = bswap32(vid->data_size);
        vid->used_ebs = bswap32(vid->used_ebs);
        vid->data_pad = bswap32(vid->data_pad);
        vid->data_crc = bswap32(vid->data_crc);
        vid->hdr_crc = bswap32(vid->hdr_crc);
        vid->sqnum = bswap64(vid->sqnum);
        if(crc != vid->hdr_crc && vid->magic == UBI_VID_HDR_MAGIC)
            printf("VID CRC: %x/%x\n", crc, vid->hdr_crc);
#if 0
        /* Apparently data after VID is semi-random. */
        /* Read extra 32 bits and check for all 0xff. */
        {
            off_t cur = lseek(fd, 0, SEEK_CUR);
            if(read(fd, &crc, sizeof(crc)) == sizeof(crc)) {
                if(crc != 0xffffffffUL)
                    printf("Corrupt VID extra @ %d with %x\n", vid->lnum, crc);
            } else {
                printf("Can read VID extra @ %d.\n", vid->lnum);
            }
            lseek(fd, cur, SEEK_SET);
        }
#endif
    } else
        memset(vid, 0, sizeof(*vid));
    return rval;
}

static void print_vtbl(struct ubi_vtbl_record *vtbl)
{
    printf(" Found vtbl [%d] %s\n", vtbl->name_len, vtbl->name);
    printf(" Reserved PEBs: %d\n", vtbl->reserved_pebs);
    printf(" Align: %d\n", vtbl->alignment);
    printf(" Pad: %d\n", vtbl->data_pad);
    if(vtbl->vol_type != UBI_VID_DYNAMIC)
        printf(" vol_type: %s\n",
               vtbl->vol_type == UBI_VID_DYNAMIC ? "dynamic" : "static");
    printf(" Update: %d\n", vtbl->upd_marker);
    printf(" Flags: %d\n", (int)vtbl->flags);
}

static void read_vtbl(int fd, struct ubi_vtbl_record *vtbl)
{
    int rval = read(fd, vtbl, sizeof(*vtbl));
    if(rval == sizeof(*vtbl)) {
        vtbl->reserved_pebs = bswap32(vtbl->reserved_pebs);
        vtbl->alignment = bswap32(vtbl->alignment);
        vtbl->data_pad = bswap32(vtbl->data_pad);
        vtbl->crc = bswap32(vtbl->crc);
        vtbl->name_len = bswap16(vtbl->name_len);
    } else
        memset(vtbl, 0, sizeof(*vtbl));
}

static void print_fm_sb(struct ubi_fm_sb *fm_sb)
{
    int i;
    
    if(fm_sb->magic != UBI_FM_SB_MAGIC)
        printf(" Magic: %x\n", fm_sb->magic);
    if(fm_sb->version != UBI_VERSION)
        printf(" Version: %d\n", (int)fm_sb->version);
    printf(" data_crc: %x\n", fm_sb->data_crc);
    printf(" used_blocks: %x\n", fm_sb->used_blocks);
    for(i = 0; i < fm_sb->used_blocks; i++)
        printf(" block_loc[%d]: %d\n", i, fm_sb->block_loc[i]);
    for(i=0; i < fm_sb->used_blocks; i++)
        printf(" block_ec[%d]: %d\n", i, fm_sb->block_ec[i]);
    printf(" sqnum: %lld\n", fm_sb->sqnum);
}

static void read_fm_sb(int fd, struct ubi_fm_sb *fm_sb)
{
    int rval = read(fd, fm_sb, sizeof(*fm_sb));
    if(rval == sizeof(*fm_sb)) {
        int i;
        fm_sb->magic = bswap32(fm_sb->magic);
        fm_sb->data_crc = bswap32(fm_sb->data_crc);
        fm_sb->used_blocks = bswap32(fm_sb->used_blocks);
        for(i=0; i < UBI_FM_MAX_BLOCKS; i++)
            fm_sb->block_loc[i] = bswap32(fm_sb->block_loc[i]);
        for(i=0; i < UBI_FM_MAX_BLOCKS; i++)
            fm_sb->block_ec[i] = bswap32(fm_sb->block_ec[i]);
        fm_sb->sqnum = bswap64(fm_sb->sqnum);
    } else
        memset(fm_sb, 0, sizeof(*fm_sb));
}

/* Set logical block at physical. */
static int eba_map[1920];
static int pba_map[1920];

static void usage(char *name)
{
    printf("Usage: %s -b [erase block size] -e -v <ubi file> \n", name);
    printf("Where,\n -e  is dump the logic to physical block map.\n");
    printf(" -v  is dump the VID headers.\n");
    printf(" -b [size] sets the erase block size (flash dependent).\n");
  
}

typedef struct fastmap {
    struct ubi_fm_sb        fm_sb;
    struct ubi_fm_hdr       hdr;
    struct ubi_fm_scan_pool pool1;
    struct ubi_fm_scan_pool pool2;
    /* Free, Used, Scrub and Erase */
    struct ubi_fm_ec        ec[0];
    /* ... */
    /* struct ubi_fm_volhdr vol; */
    /* struct ubi_fm_eba eba[0]; */

} fastmap;

int main (int argc, char *argv[])
{
    int fd, i, erase_block = 0, eba_flag = 0;
    int c;
    struct ubi_ec_hdr ec;
    struct ubi_vid_hdr vid;
    int erase_size = 0x20000;
    int leb_size;
    off_t cur_ec = 0;
    int vidless_blocks = 0;

    while ((c = getopt (argc, argv, "hveb:")) != -1)
        switch (c)
        {
            case 'h': /* Help */
                usage(argv[0]);
                goto out;
            case 'b':
                erase_size = atoi(optarg);
                break;
            case 'e':
                eba_flag = 1;
                break;
            case 'v':
                dump_vid = 1;
                break;
            case '?':
                if (optopt == 'b')
                    fprintf (stderr, "Option -%c requires an argument.\n", optopt);
                else if (isprint (optopt))
                    fprintf (stderr, "Unknown option `-%c'.\n", optopt);
                else
                    fprintf (stderr,
                             "Unknown option character `\\x%x'.\n",
                             optopt);
                return 1;
            default:
                goto out;
        }

    if(optind >= argc) {
        usage(argv[0]);
        goto out;
    }

    fd = open(argv[optind], O_RDONLY);
    if(fd < 0) {
        printf("Bad file: %s\n", argv[1]);
        goto out;
    }

    memset(eba_map, -1, sizeof(eba_map));
    memset(pba_map, -1, sizeof(pba_map));
    
    /* Process each 'erase block'. */
    read_ec(fd,&ec);
    while(ec.magic == UBI_EC_HDR_MAGIC) {
        leb_size = erase_size - ec.data_offset;
        print_ec(&ec);

        /* VID present? */
        if(lseek(fd, ec.vid_hdr_offset-sizeof(ec), SEEK_CUR) == -1) {
            printf("Seek error: %s\n", argv[1]);
            goto out;
        }

        if(read_vid(fd,&vid) != sizeof(vid)) {
            printf("File too small: %s\n", argv[1]);
            goto out;
        }
        if(vid.magic == UBI_VID_HDR_MAGIC) {
            print_vid(erase_block, &vid);
            if(vid.vol_id == 3) {
                if(eba_map[vid.lnum] != -1)
                    printf("EBA dup: %d %d\n", eba_map[vid.lnum], erase_block);
                eba_map[vid.lnum] = erase_block;
            }
            pba_map[erase_block] = vid.lnum;

            /* Read volume table. */
            if(vid.vol_id == UBI_INTERNAL_VOL_START) {
                /* Seek to PEB data offset. */
                if(lseek(fd,
                         ec.data_offset - ec.vid_hdr_offset - sizeof(vid),
                         SEEK_CUR) == -1)
                    printf("Seek error: %s\n", argv[1]);
                else {
                    int i;
                    struct ubi_vtbl_record vtbl;
                    for(i = 0; i < UBI_MAX_VOLUMES; i++) {
                        read_vtbl(fd, &vtbl);
                        if(vtbl.reserved_pebs ||
                           vtbl.name_len ||
                           strcmp((char*)vtbl.name, "") != 0) {
                            printf("VTBL %d\n", i);
                            print_vtbl(&vtbl);
                        }
                    }
                }
            } else if(vid.vol_id == UBI_FM_SB_VOLUME_ID) {
                printf("Found Fastmap super block @PEB %d.\n", erase_block);
                if(lseek(fd,
                         ec.data_offset - ec.vid_hdr_offset - sizeof(vid),
                         SEEK_CUR) == -1)
                    printf("Seek error: %s\n", argv[1]);
                else {
                    void *data = alloca(leb_size);
                    struct ubi_fm_sb *fm_sb = data;
                    read_fm_sb(fd, data);
                    print_fm_sb(fm_sb);
                }
            } else if(vid.vol_id == UBI_FM_DATA_VOLUME_ID) {
                printf("Found Fastmap data block @PEB %d.\n", erase_block);
                printf("UNSUPPORTED!!!\n");
            }

        } else if(vid.magic != 0xffffffff){
            printf("VID %d corrupt! %x\n", erase_block, vid.magic);
        } else {
            vidless_blocks++;
        }

        erase_block++;
        cur_ec += erase_size;
        cur_ec = lseek(fd, cur_ec, SEEK_SET);

        /* Process Erase counter. */
        read_ec(fd,&ec);
    }

    printf("Found %d vidless (free) blocks.\n", vidless_blocks);
    if(eba_flag) {
        printf("Logical to physical.\n");
        for(i = 0; i < ALEN(eba_map); i+=8)
            printf("%4d: %4d %4d %4d %4d %4d %4d %4d %4d"
                   " %4d %4d %4d %4d %4d %4d %4d %4d\n", i,
                   eba_map[i],   eba_map[i+1],
                   eba_map[i+2], eba_map[i+3],
                   eba_map[i+4], eba_map[i+5],
                   eba_map[i+6], eba_map[i+7],
                   eba_map[i+8], eba_map[i+9],
                   eba_map[i+10], eba_map[i+11],
                   eba_map[i+12], eba_map[i+13],
                   eba_map[i+14], eba_map[i+15]);
        printf("Physical to logical.\n");
        for(i = 0; i < ALEN(pba_map); i+=8)
            printf("%4d: %4d %4d %4d %4d %4d %4d %4d %4d"
                   " %4d %4d %4d %4d %4d %4d %4d %4d\n", i,
                   pba_map[i],   pba_map[i+1],
                   pba_map[i+2], pba_map[i+3],
                   pba_map[i+4], pba_map[i+5],
                   pba_map[i+6], pba_map[i+7],
                   pba_map[i+8], pba_map[i+9],
                   pba_map[i+10], pba_map[i+11],
                   pba_map[i+12], pba_map[i+13],
                   pba_map[i+14], pba_map[i+15]);
    }
out:
    return 0;
}

Re: [PATCH 6/7] New utility ubidump

[hujianyang]

> 
> I started some thing like this.  The source is attached.  As really you
> only want 'read only' access, there is a minimal portion of the
> UBI/UbiFs code that is needed.  I think that only the 'ubi-media.h' is
> needed.  The Linux is high performance and handles read/write with
> different fault conditions.
> 
> If you write code for read-only/single thread I think it will be much
> more simple than the active UBI/UbiFS code in the Linux kernel.  I did
> not look at the UbiFS layer as much.  It is far more complex that UBI
> imho; of course, I just saw a little of the internal volumes and the
> fast map features.  However, fast map itself should not be needed for
> this utility.
> 
> For your use case of analysis in a running system, I think that reads of
> '/dev/mtd0ro', etc can be used.  This way recovery features can also be
> developed if we identified some inconsistency; Ie, it is not required to
> 'mount' the volume before analysis.
> 
> The attached source just scans an file image and create an leb/peb
> mapping.  I only used the 'ubi-media.h' as documentation.
> 
> Fwiw,
> Bill Pringlemeir.
>

Hi Bill,

I've read your code.  This code shows me a better way to get EC header
and VID header than my 'ioctl' design. Thanks~!

I think my former work started in a wrong way. Using MTD functionality
as yours seems better to develop a new utility we want.

So in the next step, I would like to resend a new patch set which just
read fs data by mtd_read(). I think a new ioctl is needed to translate
specified eraseblock num from lnum to pnum. I will not care about volume
table or fastmap this time. This patch set will not support dumping
image file(something like -f) either. As Artem said, we should do this
step by step.

Thanks for your advice. I will resend this utility soon and Cc to you.


Hu

Re: [PATCH 6/7] New utility ubidump

[Bill Pringlemeir]

On 22 Jul 2014, hujianyang@huawei.com wrote:

> I've read your code.  This code shows me a better way to get EC header
> and VID header than my 'ioctl' design. Thanks~!

> I think my former work started in a wrong way. Using MTD functionality
> as yours seems better to develop a new utility we want.

> So in the next step, I would like to resend a new patch set which just
> read fs data by mtd_read(). I think a new ioctl is needed to translate
> specified eraseblock num from lnum to pnum.

The code I sent is not very clear on this.  However, the variables
'eba_map' and 'pba_map' are static arrays that do this mapping.
Basically, the algorithm is,

  for each erase block,
    read EC header.
    if(EC good) 
       read VID header
       if(VID good)
         place_in_map().

The structure of the base UBI is very simple.  An issue the code I sent
doesn't handle is when two headers map to the same 'logical block'.  The
'sqnum' should resolve this.

   place_in_map()
     if(LEB empty)
      map PEB.
     else if(LEB sqnum < new sqnum) /* care of wrapping or large change? */
      map new PEB to LEB.

For this utility, we might wish to keep the older PEB around in case the
current version is corrupt at a higher (UbiFS) layer or maybe not.
However, writing our own layer gives this flexibility and we don't have
to add code to the Linux MTD layers.  More importantly, the utility will
not need (a compatible) Linux MTD layer on the host machine to check the
UBI/UbiFs dump, if we can do this without an 'ioctl()'.

Thanks for attempting this.  I think many people realize the value of
such a tool.

Regards,
Bill Pringlemeir.

Re: [PATCH 6/7] New utility ubidump

[Artem Bityutskiy]

On Tue, 2014-07-22 at 16:15 +0800, hujianyang wrote:
> I've read your code.  This code shows me a better way to get EC header
> and VID header than my 'ioctl' design. Thanks~!
> 
> I think my former work started in a wrong way. Using MTD functionality
> as yours seems better to develop a new utility we want.
> 
> So in the next step, I would like to resend a new patch set which just
> read fs data by mtd_read(). I think a new ioctl is needed to translate
> specified eraseblock num from lnum to pnum. I will not care about volume
> table or fastmap this time. This patch set will not support dumping
> image file(something like -f) either. As Artem said, we should do this
> step by step.
> 
> Thanks for your advice. I will resend this utility soon and Cc to you.

If possible, please, try to implement few things at a time. Try to make
your initial submission smaller, and then add more stuff on top. This
way it is going to be easier to merge the work. Thanks!

-- 
Best Regards,
Artem Bityutskiy

Re: [PATCH 6/7] New utility ubidump

[hujianyang]

> 
> If possible, please, try to implement few things at a time. Try to make
> your initial submission smaller, and then add more stuff on top. This
> way it is going to be easier to merge the work. Thanks!
>

I've resend the v2 patches. I spend a short vacation last week
and was busy with other stuff when I came back to work.

I think we can just apply binary data dump, ignore ubif/ubifs
format data at first to make initial submission smaller. How
do you feel about this?

just like:
ubidump /dev/ubiX_Y --lnum n

output binary/hex data.