Util-Linux Archive on lore.kernel.org
 help / color / Atom feed
* Re: [PATCH v2] fiemap : add fiemap misc tool
       [not found] <1554883940-19963-1-git-send-email-pizhenwei@bytedance.com>
@ 2019-04-10  9:59 ` Karel Zak
  2019-04-10 10:11   ` [External Email] " zhenwei pi
  2019-04-10 12:12   ` Lukas Czerner
  0 siblings, 2 replies; 3+ messages in thread
From: Karel Zak @ 2019-04-10  9:59 UTC (permalink / raw)
  To: zhenwei pi; +Cc: Theodore Ts'o, linux-ext4, util-linux

On Wed, Apr 10, 2019 at 04:12:20PM +0800, zhenwei pi wrote:
> Add fiemap to dump file extent mappings. Typically we can recognize a
> file is sparse or not.

It's good idea to export FIEMAP to command line, but it seems we
already have xfs_io and filefrag.

The command filefrag seems pretty usable and with "-e" it provides
exactly the same output like your "fiemap" tool.

The problem I see is that filefrag is in e2fsprogs package. It would
be better to have it in some more generic package and FS independent
package.

Maybe we can do another move from e2fsprogs to util-linux. Ted, your
opinion?

    Karel


# filefrag -e /var/log/wtmp
Filesystem type is: ef53
File size of /var/log/wtmp is 9032064 (2206 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..       0:   11567426..  11567426:      1:            
   1:        1..     190:   11730944..  11731133:    190:   11567427:
   2:      191..     380:   11732109..  11732298:    190:   11731134:
   3:      381..     570:   11731883..  11732072:    190:   11732299:
   4:      571..     633:   11731501..  11731563:     63:   11732073:
   5:      634..     761:   11731584..  11731711:    128:   11731564:
   6:      762..     763:   11731198..  11731199:      2:   11731712:
   7:      764..     764:   11732073..  11732073:      1:   11731200:
   8:      765..     955:   11731200..  11731390:    191:   11732074:
   9:      956..    1145:   11732705..  11732894:    190:   11731391:
  10:     1146..    1254:   11733137..  11733245:    109:   11732895:
  11:     1255..    1444:   11732895..  11733084:    190:   11733246:
  12:     1445..    1824:   11733246..  11733625:    380:   11733085:
  13:     1825..    2014:   11737497..  11737686:    190:   11733626:
  14:     2015..    2205:   11896890..  11897080:    191:   11737687: last,eof
/var/log/wtmp: 15 extents found


> 
> For example:
>  #./fiemap /var/log/syslog /var/log/syslog.2.gz
> File /var/log/syslog has 19 extent(s):
>   	Logical          Physical         Length           Flag
> 0:	0000000000000000 00000010d63a8000 0000000000001000 0000
> 1:	0000000000001000 00000010d63c4000 0000000000001000 0000
> 2:	0000000000002000 00000010d63d6000 0000000000001000 0000
> 3:	0000000000003000 00000010d723e000 0000000000001000 0000
> 4:	0000000000004000 000000076c377000 0000000000001000 0000
> 5:	0000000000005000 00000010d550f000 0000000000001000 0000
> 6:	0000000000006000 00000010d7290000 0000000000001000 0000
> 7:	0000000000007000 00000010d57c9000 0000000000001000 0000
> 8:	0000000000008000 00000010d57f5000 0000000000001000 0000
> 9:	0000000000009000 000000076d6b3000 0000000000001000 0000
> 10:	000000000000a000 00000010d558d000 0000000000001000 0000
> 11:	000000000000b000 00000010d77ff000 0000000000001000 0000
> 12:	000000000000c000 00000010d67fb000 0000000000001000 0000
> 13:	000000000000d000 00000010d73fc000 0000000000001000 0000
> 14:	000000000000e000 00000005d2bb2000 0000000000001000 0000
> 15:	000000000000f000 00000007705a1000 0000000000001000 0000
> 16:	0000000000010000 0000000c8bf10000 0000000000010000 0000
> 17:	0000000000020000 0000001070020000 0000000000060000 0000
> 18:	0000000000080000 00000011aa580000 0000000000014000 0001
> 
> File /var/log/syslog.2.gz has 1 extent(s):
>   	Logical          Physical         Length           Flag
> 0:	0000000000000000 0000000995c40000 0000000000023000 0001
> 
> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> ---
>  AUTHORS                  |   1 +
>  configure.ac             |   3 +
>  misc-utils/Makemodule.am |   5 ++
>  misc-utils/fiemap.c      | 155 +++++++++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 164 insertions(+)
>  create mode 100644 misc-utils/fiemap.c
> 
> diff --git a/AUTHORS b/AUTHORS
> index d7dbbd6..a0cecf7 100644
> --- a/AUTHORS
> +++ b/AUTHORS
> @@ -20,6 +20,7 @@ AUTHORS (merged projects & commands):
>        fallocate:       Eric Sandeen <sandeen@redhat.com>
>                         Karel Zak <kzak@redhat.com>
>                         Matěj Cepl <mcepl@redhat.com>
> +      fiemap:          zhenwei pi <pizhenwei@bytedance.com>
>        fincore:         Masatake YAMATO <yamato@redhat.com>
>        findmnt:         Karel Zak <kzak@redhat.com>
>        flock:           H. Peter Anvin <hpa@zytor.com>
> diff --git a/configure.ac b/configure.ac
> index bbf07db..ccb1a12 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -1769,6 +1769,9 @@ UL_REQUIRES_LINUX([fincore])
>  UL_REQUIRES_BUILD([fincore], [libsmartcols])
>  AM_CONDITIONAL([BUILD_FINCORE], [test "x$build_fincore" = xyes])
>  
> +UL_BUILD_INIT([fiemap], [yes])
> +AM_CONDITIONAL([BUILD_FIEMAP], [test "x$build_fiemap" = xyes])
> +
>  UL_BUILD_INIT([fsfreeze], [check])
>  UL_REQUIRES_LINUX([fsfreeze])
>  AM_CONDITIONAL([BUILD_FSFREEZE], [test "x$build_fsfreeze" = xyes])
> diff --git a/misc-utils/Makemodule.am b/misc-utils/Makemodule.am
> index f56a819..8bdc6fa 100644
> --- a/misc-utils/Makemodule.am
> +++ b/misc-utils/Makemodule.am
> @@ -228,3 +228,8 @@ hardlink_CFLAGS += $(PCRE_CFLAGS)
>  endif
>  dist_man_MANS += misc-utils/hardlink.1
>  endif
> +
> +if BUILD_FIEMAP
> +usrbin_exec_PROGRAMS += fiemap
> +fiemap_SOURCES = misc-utils/fiemap.c
> +endif
> diff --git a/misc-utils/fiemap.c b/misc-utils/fiemap.c
> new file mode 100644
> index 0000000..b77eb14
> --- /dev/null
> +++ b/misc-utils/fiemap.c
> @@ -0,0 +1,155 @@
> +/*
> + * Copyright (C) 2019 zhenwei pi <pizhenwei@bytedance.com>
> + *
> + * This file may be redistributed under the terms of the GNU Public
> + * License.
> + */
> +#include <stdlib.h>
> +#include <stdio.h>
> +#include <string.h>
> +#include <unistd.h>
> +#include <fcntl.h>
> +#include <getopt.h>
> +#include <sys/ioctl.h>
> +#include <linux/fs.h>
> +
> +#include "c.h"
> +#include "nls.h"
> +#include "closestream.h"
> +
> +#ifdef FS_IOC_FIEMAP
> +#include <linux/fiemap.h>
> +
> +static struct fiemap *read_fiemap(int fd)
> +{
> +	struct fiemap *fiemap;
> +	int extents_size;
> +
> +	fiemap = (struct fiemap *)malloc(sizeof(struct fiemap));
> +	if (fiemap == NULL)
> +		err(EXIT_FAILURE, _("malloc"));
> +
> +	memset(fiemap, 0, sizeof(struct fiemap));
> +
> +	fiemap->fm_start = 0;
> +	fiemap->fm_length = ~0;
> +	fiemap->fm_flags = 0;
> +	fiemap->fm_extent_count = 0;
> +	fiemap->fm_mapped_extents = 0;
> +
> +	/* count how many extents there are */
> +	if (ioctl(fd, FS_IOC_FIEMAP, fiemap) < 0)
> +		err(EXIT_FAILURE, _("fiemap ioctl() failed"));
> +
> +	/* read in the extents */
> +	extents_size = sizeof(struct fiemap_extent) *
> +		(fiemap->fm_mapped_extents);
> +
> +	/* resize fiemaps for all extents */
> +	fiemap = (struct fiemap *)realloc(fiemap, sizeof(struct fiemap) +
> +			extents_size);
> +	if (fiemap == NULL)
> +		err(EXIT_FAILURE, _("realloc for extents memory"));
> +
> +	memset(fiemap->fm_extents, 0, extents_size);
> +	fiemap->fm_extent_count = fiemap->fm_mapped_extents;
> +	fiemap->fm_mapped_extents = 0;
> +
> +	if (ioctl(fd, FS_IOC_FIEMAP, fiemap) < 0) {
> +		err(EXIT_FAILURE, _("ioctl() FS_IOC_FIEMAP failed"));
> +		return NULL;
> +	}
> +
> +	return fiemap;
> +}
> +
> +static void show_fiemap(struct fiemap *fiemap, char *filename)
> +{
> +	unsigned int i = 0;
> +
> +	printf("File %s has %d extent(s):\n", filename,
> +			fiemap->fm_mapped_extents);
> +	printf("#\tLogical          Physical         Length           Flag\n");
> +	for (i = 0; i < fiemap->fm_mapped_extents; i++) {
> +		printf("%d:\t%-16.16llx %-16.16llx %-16.16llx %-4.4x\n", i,
> +				fiemap->fm_extents[i].fe_logical,
> +				fiemap->fm_extents[i].fe_physical,
> +				fiemap->fm_extents[i].fe_length,
> +				fiemap->fm_extents[i].fe_flags);
> +	}
> +
> +	printf("\n");
> +}
> +
> +static void __attribute__((__noreturn__)) usage(void)
> +{
> +	FILE *out = stdout;
> +
> +	fputs(USAGE_HEADER, out);
> +	fprintf(out, _(" %s <file>...\n"), program_invocation_short_name);
> +
> +	fputs(USAGE_SEPARATOR, out);
> +	exit(EXIT_SUCCESS);
> +}
> +
> +int main(int argc, char **argv)
> +{
> +	int c;
> +	static const struct option longopts[] = {
> +		{ "version",    no_argument, NULL, 'V' },
> +		{ "help",       no_argument, NULL, 'h' },
> +		{ NULL, 0, NULL, 0 },
> +	};
> +
> +	setlocale(LC_ALL, "");
> +	bindtextdomain(PACKAGE, LOCALEDIR);
> +	textdomain(PACKAGE);
> +	atexit(close_stdout);
> +
> +	while ((c = getopt_long (argc, argv, "Vh", longopts, NULL)) != -1) {
> +		switch (c) {
> +		case 'V':
> +			printf(UTIL_LINUX_VERSION);
> +			return EXIT_SUCCESS;
> +
> +		case 'h':
> +			usage();
> +
> +		default:
> +			errtryhelp(EXIT_FAILURE);
> +		}
> +	}
> +
> +	if (optind == argc) {
> +		warnx(_("no file specified"));
> +		errtryhelp(EXIT_FAILURE);
> +	}
> +
> +	for ( ; optind < argc; optind++) {
> +		int fd = 0;
> +
> +		fd = open(argv[optind], O_RDONLY);
> +		if (fd < 0) {
> +			err(EXIT_FAILURE, _("open file failed"));
> +		} else {
> +			struct fiemap *fiemap = NULL;
> +
> +			fiemap = read_fiemap(fd);
> +			if (fiemap != NULL) {
> +				show_fiemap(fiemap, argv[optind]);
> +				free(fiemap);
> +			}
> +			close(fd);
> +		}
> +	}
> +
> +	return 0;
> +}
> +#else
> +int main(int argc, char **argv)
> +{
> +	err(EXIT_FAILURE, _("FS_IOC_FIEMAP not supported"));
> +
> +	return 0;
> +}
> +#endif
> -- 
> 2.7.4
> 

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [External Email] Re: [PATCH v2] fiemap : add fiemap misc tool
  2019-04-10  9:59 ` [PATCH v2] fiemap : add fiemap misc tool Karel Zak
@ 2019-04-10 10:11   ` " zhenwei pi
  2019-04-10 12:12   ` Lukas Czerner
  1 sibling, 0 replies; 3+ messages in thread
From: zhenwei pi @ 2019-04-10 10:11 UTC (permalink / raw)
  To: Karel Zak; +Cc: Theodore Ts'o, linux-ext4, util-linux

On 04/10/2019 05:59 PM, Karel Zak wrote:

> On Wed, Apr 10, 2019 at 04:12:20PM +0800, zhenwei pi wrote:
>> Add fiemap to dump file extent mappings. Typically we can recognize a
>> file is sparse or not.
> It's good idea to export FIEMAP to command line, but it seems we
> already have xfs_io and filefrag.
>
> The command filefrag seems pretty usable and with "-e" it provides
> exactly the same output like your "fiemap" tool.
>
> The problem I see is that filefrag is in e2fsprogs package. It would
> be better to have it in some more generic package and FS independent
> package.
>
> Maybe we can do another move from e2fsprogs to util-linux. Ted, your
> opinion?
>
>      Karel

I did't know "filefrag -e". There is no need to re-write the same tool.
Thanks.

	zhenwei pi

>
> # filefrag -e /var/log/wtmp
> Filesystem type is: ef53
> File size of /var/log/wtmp is 9032064 (2206 blocks of 4096 bytes)
>   ext:     logical_offset:        physical_offset: length:   expected: flags:
>     0:        0..       0:   11567426..  11567426:      1:
>     1:        1..     190:   11730944..  11731133:    190:   11567427:
>     2:      191..     380:   11732109..  11732298:    190:   11731134:
>     3:      381..     570:   11731883..  11732072:    190:   11732299:
>     4:      571..     633:   11731501..  11731563:     63:   11732073:
>     5:      634..     761:   11731584..  11731711:    128:   11731564:
>     6:      762..     763:   11731198..  11731199:      2:   11731712:
>     7:      764..     764:   11732073..  11732073:      1:   11731200:
>     8:      765..     955:   11731200..  11731390:    191:   11732074:
>     9:      956..    1145:   11732705..  11732894:    190:   11731391:
>    10:     1146..    1254:   11733137..  11733245:    109:   11732895:
>    11:     1255..    1444:   11732895..  11733084:    190:   11733246:
>    12:     1445..    1824:   11733246..  11733625:    380:   11733085:
>    13:     1825..    2014:   11737497..  11737686:    190:   11733626:
>    14:     2015..    2205:   11896890..  11897080:    191:   11737687: last,eof
> /var/log/wtmp: 15 extents found
>
>
>> For example:
>>   #./fiemap /var/log/syslog /var/log/syslog.2.gz
>> File /var/log/syslog has 19 extent(s):
>>    	Logical          Physical         Length           Flag
>> 0:	0000000000000000 00000010d63a8000 0000000000001000 0000
>> 1:	0000000000001000 00000010d63c4000 0000000000001000 0000
>> 2:	0000000000002000 00000010d63d6000 0000000000001000 0000
>> 3:	0000000000003000 00000010d723e000 0000000000001000 0000
>> 4:	0000000000004000 000000076c377000 0000000000001000 0000
>> 5:	0000000000005000 00000010d550f000 0000000000001000 0000
>> 6:	0000000000006000 00000010d7290000 0000000000001000 0000
>> 7:	0000000000007000 00000010d57c9000 0000000000001000 0000
>> 8:	0000000000008000 00000010d57f5000 0000000000001000 0000
>> 9:	0000000000009000 000000076d6b3000 0000000000001000 0000
>> 10:	000000000000a000 00000010d558d000 0000000000001000 0000
>> 11:	000000000000b000 00000010d77ff000 0000000000001000 0000
>> 12:	000000000000c000 00000010d67fb000 0000000000001000 0000
>> 13:	000000000000d000 00000010d73fc000 0000000000001000 0000
>> 14:	000000000000e000 00000005d2bb2000 0000000000001000 0000
>> 15:	000000000000f000 00000007705a1000 0000000000001000 0000
>> 16:	0000000000010000 0000000c8bf10000 0000000000010000 0000
>> 17:	0000000000020000 0000001070020000 0000000000060000 0000
>> 18:	0000000000080000 00000011aa580000 0000000000014000 0001
>>
>> File /var/log/syslog.2.gz has 1 extent(s):
>>    	Logical          Physical         Length           Flag
>> 0:	0000000000000000 0000000995c40000 0000000000023000 0001
>>
>> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
>> ---
>>   AUTHORS                  |   1 +
>>   configure.ac             |   3 +
>>   misc-utils/Makemodule.am |   5 ++
>>   misc-utils/fiemap.c      | 155 +++++++++++++++++++++++++++++++++++++++++++++++
>>   4 files changed, 164 insertions(+)
>>   create mode 100644 misc-utils/fiemap.c
>>
>> diff --git a/AUTHORS b/AUTHORS
>> index d7dbbd6..a0cecf7 100644
>> --- a/AUTHORS
>> +++ b/AUTHORS
>> @@ -20,6 +20,7 @@ AUTHORS (merged projects & commands):
>>         fallocate:       Eric Sandeen <sandeen@redhat.com>
>>                          Karel Zak <kzak@redhat.com>
>>                          Matěj Cepl <mcepl@redhat.com>
>> +      fiemap:          zhenwei pi <pizhenwei@bytedance.com>
>>         fincore:         Masatake YAMATO <yamato@redhat.com>
>>         findmnt:         Karel Zak <kzak@redhat.com>
>>         flock:           H. Peter Anvin <hpa@zytor.com>
>> diff --git a/configure.ac b/configure.ac
>> index bbf07db..ccb1a12 100644
>> --- a/configure.ac
>> +++ b/configure.ac
>> @@ -1769,6 +1769,9 @@ UL_REQUIRES_LINUX([fincore])
>>   UL_REQUIRES_BUILD([fincore], [libsmartcols])
>>   AM_CONDITIONAL([BUILD_FINCORE], [test "x$build_fincore" = xyes])
>>   
>> +UL_BUILD_INIT([fiemap], [yes])
>> +AM_CONDITIONAL([BUILD_FIEMAP], [test "x$build_fiemap" = xyes])
>> +
>>   UL_BUILD_INIT([fsfreeze], [check])
>>   UL_REQUIRES_LINUX([fsfreeze])
>>   AM_CONDITIONAL([BUILD_FSFREEZE], [test "x$build_fsfreeze" = xyes])
>> diff --git a/misc-utils/Makemodule.am b/misc-utils/Makemodule.am
>> index f56a819..8bdc6fa 100644
>> --- a/misc-utils/Makemodule.am
>> +++ b/misc-utils/Makemodule.am
>> @@ -228,3 +228,8 @@ hardlink_CFLAGS += $(PCRE_CFLAGS)
>>   endif
>>   dist_man_MANS += misc-utils/hardlink.1
>>   endif
>> +
>> +if BUILD_FIEMAP
>> +usrbin_exec_PROGRAMS += fiemap
>> +fiemap_SOURCES = misc-utils/fiemap.c
>> +endif
>> diff --git a/misc-utils/fiemap.c b/misc-utils/fiemap.c
>> new file mode 100644
>> index 0000000..b77eb14
>> --- /dev/null
>> +++ b/misc-utils/fiemap.c
>> @@ -0,0 +1,155 @@
>> +/*
>> + * Copyright (C) 2019 zhenwei pi <pizhenwei@bytedance.com>
>> + *
>> + * This file may be redistributed under the terms of the GNU Public
>> + * License.
>> + */
>> +#include <stdlib.h>
>> +#include <stdio.h>
>> +#include <string.h>
>> +#include <unistd.h>
>> +#include <fcntl.h>
>> +#include <getopt.h>
>> +#include <sys/ioctl.h>
>> +#include <linux/fs.h>
>> +
>> +#include "c.h"
>> +#include "nls.h"
>> +#include "closestream.h"
>> +
>> +#ifdef FS_IOC_FIEMAP
>> +#include <linux/fiemap.h>
>> +
>> +static struct fiemap *read_fiemap(int fd)
>> +{
>> +	struct fiemap *fiemap;
>> +	int extents_size;
>> +
>> +	fiemap = (struct fiemap *)malloc(sizeof(struct fiemap));
>> +	if (fiemap == NULL)
>> +		err(EXIT_FAILURE, _("malloc"));
>> +
>> +	memset(fiemap, 0, sizeof(struct fiemap));
>> +
>> +	fiemap->fm_start = 0;
>> +	fiemap->fm_length = ~0;
>> +	fiemap->fm_flags = 0;
>> +	fiemap->fm_extent_count = 0;
>> +	fiemap->fm_mapped_extents = 0;
>> +
>> +	/* count how many extents there are */
>> +	if (ioctl(fd, FS_IOC_FIEMAP, fiemap) < 0)
>> +		err(EXIT_FAILURE, _("fiemap ioctl() failed"));
>> +
>> +	/* read in the extents */
>> +	extents_size = sizeof(struct fiemap_extent) *
>> +		(fiemap->fm_mapped_extents);
>> +
>> +	/* resize fiemaps for all extents */
>> +	fiemap = (struct fiemap *)realloc(fiemap, sizeof(struct fiemap) +
>> +			extents_size);
>> +	if (fiemap == NULL)
>> +		err(EXIT_FAILURE, _("realloc for extents memory"));
>> +
>> +	memset(fiemap->fm_extents, 0, extents_size);
>> +	fiemap->fm_extent_count = fiemap->fm_mapped_extents;
>> +	fiemap->fm_mapped_extents = 0;
>> +
>> +	if (ioctl(fd, FS_IOC_FIEMAP, fiemap) < 0) {
>> +		err(EXIT_FAILURE, _("ioctl() FS_IOC_FIEMAP failed"));
>> +		return NULL;
>> +	}
>> +
>> +	return fiemap;
>> +}
>> +
>> +static void show_fiemap(struct fiemap *fiemap, char *filename)
>> +{
>> +	unsigned int i = 0;
>> +
>> +	printf("File %s has %d extent(s):\n", filename,
>> +			fiemap->fm_mapped_extents);
>> +	printf("#\tLogical          Physical         Length           Flag\n");
>> +	for (i = 0; i < fiemap->fm_mapped_extents; i++) {
>> +		printf("%d:\t%-16.16llx %-16.16llx %-16.16llx %-4.4x\n", i,
>> +				fiemap->fm_extents[i].fe_logical,
>> +				fiemap->fm_extents[i].fe_physical,
>> +				fiemap->fm_extents[i].fe_length,
>> +				fiemap->fm_extents[i].fe_flags);
>> +	}
>> +
>> +	printf("\n");
>> +}
>> +
>> +static void __attribute__((__noreturn__)) usage(void)
>> +{
>> +	FILE *out = stdout;
>> +
>> +	fputs(USAGE_HEADER, out);
>> +	fprintf(out, _(" %s <file>...\n"), program_invocation_short_name);
>> +
>> +	fputs(USAGE_SEPARATOR, out);
>> +	exit(EXIT_SUCCESS);
>> +}
>> +
>> +int main(int argc, char **argv)
>> +{
>> +	int c;
>> +	static const struct option longopts[] = {
>> +		{ "version",    no_argument, NULL, 'V' },
>> +		{ "help",       no_argument, NULL, 'h' },
>> +		{ NULL, 0, NULL, 0 },
>> +	};
>> +
>> +	setlocale(LC_ALL, "");
>> +	bindtextdomain(PACKAGE, LOCALEDIR);
>> +	textdomain(PACKAGE);
>> +	atexit(close_stdout);
>> +
>> +	while ((c = getopt_long (argc, argv, "Vh", longopts, NULL)) != -1) {
>> +		switch (c) {
>> +		case 'V':
>> +			printf(UTIL_LINUX_VERSION);
>> +			return EXIT_SUCCESS;
>> +
>> +		case 'h':
>> +			usage();
>> +
>> +		default:
>> +			errtryhelp(EXIT_FAILURE);
>> +		}
>> +	}
>> +
>> +	if (optind == argc) {
>> +		warnx(_("no file specified"));
>> +		errtryhelp(EXIT_FAILURE);
>> +	}
>> +
>> +	for ( ; optind < argc; optind++) {
>> +		int fd = 0;
>> +
>> +		fd = open(argv[optind], O_RDONLY);
>> +		if (fd < 0) {
>> +			err(EXIT_FAILURE, _("open file failed"));
>> +		} else {
>> +			struct fiemap *fiemap = NULL;
>> +
>> +			fiemap = read_fiemap(fd);
>> +			if (fiemap != NULL) {
>> +				show_fiemap(fiemap, argv[optind]);
>> +				free(fiemap);
>> +			}
>> +			close(fd);
>> +		}
>> +	}
>> +
>> +	return 0;
>> +}
>> +#else
>> +int main(int argc, char **argv)
>> +{
>> +	err(EXIT_FAILURE, _("FS_IOC_FIEMAP not supported"));
>> +
>> +	return 0;
>> +}
>> +#endif
>> -- 
>> 2.7.4
>>

-- 
Thanks and Best Regards,
zhenwei pi


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH v2] fiemap : add fiemap misc tool
  2019-04-10  9:59 ` [PATCH v2] fiemap : add fiemap misc tool Karel Zak
  2019-04-10 10:11   ` [External Email] " zhenwei pi
@ 2019-04-10 12:12   ` Lukas Czerner
  1 sibling, 0 replies; 3+ messages in thread
From: Lukas Czerner @ 2019-04-10 12:12 UTC (permalink / raw)
  To: Karel Zak; +Cc: zhenwei pi, Theodore Ts'o, linux-ext4, util-linux

On Wed, Apr 10, 2019 at 11:59:47AM +0200, Karel Zak wrote:
> On Wed, Apr 10, 2019 at 04:12:20PM +0800, zhenwei pi wrote:
> > Add fiemap to dump file extent mappings. Typically we can recognize a
> > file is sparse or not.
> 
> It's good idea to export FIEMAP to command line, but it seems we
> already have xfs_io and filefrag.
> 
> The command filefrag seems pretty usable and with "-e" it provides
> exactly the same output like your "fiemap" tool.
> 
> The problem I see is that filefrag is in e2fsprogs package. It would
> be better to have it in some more generic package and FS independent
> package.
> 
> Maybe we can do another move from e2fsprogs to util-linux. Ted, your
> opinion?

Looking at the filefrag code the only ext specific part I can identify
is checking for non-extent based inode and changing the output slightly
as well as taking direct/indirect mapping block layout into consideration
in case FIBMAP is used.

Other than that it's purely generic and non ext[234] specific code that
can easily live in util-linux and personally I do not think I'd miss it :)

-Lukas

> 
>     Karel
> 
> 
> # filefrag -e /var/log/wtmp
> Filesystem type is: ef53
> File size of /var/log/wtmp is 9032064 (2206 blocks of 4096 bytes)
>  ext:     logical_offset:        physical_offset: length:   expected: flags:
>    0:        0..       0:   11567426..  11567426:      1:            
>    1:        1..     190:   11730944..  11731133:    190:   11567427:
>    2:      191..     380:   11732109..  11732298:    190:   11731134:
>    3:      381..     570:   11731883..  11732072:    190:   11732299:
>    4:      571..     633:   11731501..  11731563:     63:   11732073:
>    5:      634..     761:   11731584..  11731711:    128:   11731564:
>    6:      762..     763:   11731198..  11731199:      2:   11731712:
>    7:      764..     764:   11732073..  11732073:      1:   11731200:
>    8:      765..     955:   11731200..  11731390:    191:   11732074:
>    9:      956..    1145:   11732705..  11732894:    190:   11731391:
>   10:     1146..    1254:   11733137..  11733245:    109:   11732895:
>   11:     1255..    1444:   11732895..  11733084:    190:   11733246:
>   12:     1445..    1824:   11733246..  11733625:    380:   11733085:
>   13:     1825..    2014:   11737497..  11737686:    190:   11733626:
>   14:     2015..    2205:   11896890..  11897080:    191:   11737687: last,eof
> /var/log/wtmp: 15 extents found
> 
> 
> > 
> > For example:
> >  #./fiemap /var/log/syslog /var/log/syslog.2.gz
> > File /var/log/syslog has 19 extent(s):
> >   	Logical          Physical         Length           Flag
> > 0:	0000000000000000 00000010d63a8000 0000000000001000 0000
> > 1:	0000000000001000 00000010d63c4000 0000000000001000 0000
> > 2:	0000000000002000 00000010d63d6000 0000000000001000 0000
> > 3:	0000000000003000 00000010d723e000 0000000000001000 0000
> > 4:	0000000000004000 000000076c377000 0000000000001000 0000
> > 5:	0000000000005000 00000010d550f000 0000000000001000 0000
> > 6:	0000000000006000 00000010d7290000 0000000000001000 0000
> > 7:	0000000000007000 00000010d57c9000 0000000000001000 0000
> > 8:	0000000000008000 00000010d57f5000 0000000000001000 0000
> > 9:	0000000000009000 000000076d6b3000 0000000000001000 0000
> > 10:	000000000000a000 00000010d558d000 0000000000001000 0000
> > 11:	000000000000b000 00000010d77ff000 0000000000001000 0000
> > 12:	000000000000c000 00000010d67fb000 0000000000001000 0000
> > 13:	000000000000d000 00000010d73fc000 0000000000001000 0000
> > 14:	000000000000e000 00000005d2bb2000 0000000000001000 0000
> > 15:	000000000000f000 00000007705a1000 0000000000001000 0000
> > 16:	0000000000010000 0000000c8bf10000 0000000000010000 0000
> > 17:	0000000000020000 0000001070020000 0000000000060000 0000
> > 18:	0000000000080000 00000011aa580000 0000000000014000 0001
> > 
> > File /var/log/syslog.2.gz has 1 extent(s):
> >   	Logical          Physical         Length           Flag
> > 0:	0000000000000000 0000000995c40000 0000000000023000 0001
> > 
> > Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> > ---
> >  AUTHORS                  |   1 +
> >  configure.ac             |   3 +
> >  misc-utils/Makemodule.am |   5 ++
> >  misc-utils/fiemap.c      | 155 +++++++++++++++++++++++++++++++++++++++++++++++
> >  4 files changed, 164 insertions(+)
> >  create mode 100644 misc-utils/fiemap.c
> > 
> > diff --git a/AUTHORS b/AUTHORS
> > index d7dbbd6..a0cecf7 100644
> > --- a/AUTHORS
> > +++ b/AUTHORS
> > @@ -20,6 +20,7 @@ AUTHORS (merged projects & commands):
> >        fallocate:       Eric Sandeen <sandeen@redhat.com>
> >                         Karel Zak <kzak@redhat.com>
> >                         Matěj Cepl <mcepl@redhat.com>
> > +      fiemap:          zhenwei pi <pizhenwei@bytedance.com>
> >        fincore:         Masatake YAMATO <yamato@redhat.com>
> >        findmnt:         Karel Zak <kzak@redhat.com>
> >        flock:           H. Peter Anvin <hpa@zytor.com>
> > diff --git a/configure.ac b/configure.ac
> > index bbf07db..ccb1a12 100644
> > --- a/configure.ac
> > +++ b/configure.ac
> > @@ -1769,6 +1769,9 @@ UL_REQUIRES_LINUX([fincore])
> >  UL_REQUIRES_BUILD([fincore], [libsmartcols])
> >  AM_CONDITIONAL([BUILD_FINCORE], [test "x$build_fincore" = xyes])
> >  
> > +UL_BUILD_INIT([fiemap], [yes])
> > +AM_CONDITIONAL([BUILD_FIEMAP], [test "x$build_fiemap" = xyes])
> > +
> >  UL_BUILD_INIT([fsfreeze], [check])
> >  UL_REQUIRES_LINUX([fsfreeze])
> >  AM_CONDITIONAL([BUILD_FSFREEZE], [test "x$build_fsfreeze" = xyes])
> > diff --git a/misc-utils/Makemodule.am b/misc-utils/Makemodule.am
> > index f56a819..8bdc6fa 100644
> > --- a/misc-utils/Makemodule.am
> > +++ b/misc-utils/Makemodule.am
> > @@ -228,3 +228,8 @@ hardlink_CFLAGS += $(PCRE_CFLAGS)
> >  endif
> >  dist_man_MANS += misc-utils/hardlink.1
> >  endif
> > +
> > +if BUILD_FIEMAP
> > +usrbin_exec_PROGRAMS += fiemap
> > +fiemap_SOURCES = misc-utils/fiemap.c
> > +endif
> > diff --git a/misc-utils/fiemap.c b/misc-utils/fiemap.c
> > new file mode 100644
> > index 0000000..b77eb14
> > --- /dev/null
> > +++ b/misc-utils/fiemap.c
> > @@ -0,0 +1,155 @@
> > +/*
> > + * Copyright (C) 2019 zhenwei pi <pizhenwei@bytedance.com>
> > + *
> > + * This file may be redistributed under the terms of the GNU Public
> > + * License.
> > + */
> > +#include <stdlib.h>
> > +#include <stdio.h>
> > +#include <string.h>
> > +#include <unistd.h>
> > +#include <fcntl.h>
> > +#include <getopt.h>
> > +#include <sys/ioctl.h>
> > +#include <linux/fs.h>
> > +
> > +#include "c.h"
> > +#include "nls.h"
> > +#include "closestream.h"
> > +
> > +#ifdef FS_IOC_FIEMAP
> > +#include <linux/fiemap.h>
> > +
> > +static struct fiemap *read_fiemap(int fd)
> > +{
> > +	struct fiemap *fiemap;
> > +	int extents_size;
> > +
> > +	fiemap = (struct fiemap *)malloc(sizeof(struct fiemap));
> > +	if (fiemap == NULL)
> > +		err(EXIT_FAILURE, _("malloc"));
> > +
> > +	memset(fiemap, 0, sizeof(struct fiemap));
> > +
> > +	fiemap->fm_start = 0;
> > +	fiemap->fm_length = ~0;
> > +	fiemap->fm_flags = 0;
> > +	fiemap->fm_extent_count = 0;
> > +	fiemap->fm_mapped_extents = 0;
> > +
> > +	/* count how many extents there are */
> > +	if (ioctl(fd, FS_IOC_FIEMAP, fiemap) < 0)
> > +		err(EXIT_FAILURE, _("fiemap ioctl() failed"));
> > +
> > +	/* read in the extents */
> > +	extents_size = sizeof(struct fiemap_extent) *
> > +		(fiemap->fm_mapped_extents);
> > +
> > +	/* resize fiemaps for all extents */
> > +	fiemap = (struct fiemap *)realloc(fiemap, sizeof(struct fiemap) +
> > +			extents_size);
> > +	if (fiemap == NULL)
> > +		err(EXIT_FAILURE, _("realloc for extents memory"));
> > +
> > +	memset(fiemap->fm_extents, 0, extents_size);
> > +	fiemap->fm_extent_count = fiemap->fm_mapped_extents;
> > +	fiemap->fm_mapped_extents = 0;
> > +
> > +	if (ioctl(fd, FS_IOC_FIEMAP, fiemap) < 0) {
> > +		err(EXIT_FAILURE, _("ioctl() FS_IOC_FIEMAP failed"));
> > +		return NULL;
> > +	}
> > +
> > +	return fiemap;
> > +}
> > +
> > +static void show_fiemap(struct fiemap *fiemap, char *filename)
> > +{
> > +	unsigned int i = 0;
> > +
> > +	printf("File %s has %d extent(s):\n", filename,
> > +			fiemap->fm_mapped_extents);
> > +	printf("#\tLogical          Physical         Length           Flag\n");
> > +	for (i = 0; i < fiemap->fm_mapped_extents; i++) {
> > +		printf("%d:\t%-16.16llx %-16.16llx %-16.16llx %-4.4x\n", i,
> > +				fiemap->fm_extents[i].fe_logical,
> > +				fiemap->fm_extents[i].fe_physical,
> > +				fiemap->fm_extents[i].fe_length,
> > +				fiemap->fm_extents[i].fe_flags);
> > +	}
> > +
> > +	printf("\n");
> > +}
> > +
> > +static void __attribute__((__noreturn__)) usage(void)
> > +{
> > +	FILE *out = stdout;
> > +
> > +	fputs(USAGE_HEADER, out);
> > +	fprintf(out, _(" %s <file>...\n"), program_invocation_short_name);
> > +
> > +	fputs(USAGE_SEPARATOR, out);
> > +	exit(EXIT_SUCCESS);
> > +}
> > +
> > +int main(int argc, char **argv)
> > +{
> > +	int c;
> > +	static const struct option longopts[] = {
> > +		{ "version",    no_argument, NULL, 'V' },
> > +		{ "help",       no_argument, NULL, 'h' },
> > +		{ NULL, 0, NULL, 0 },
> > +	};
> > +
> > +	setlocale(LC_ALL, "");
> > +	bindtextdomain(PACKAGE, LOCALEDIR);
> > +	textdomain(PACKAGE);
> > +	atexit(close_stdout);
> > +
> > +	while ((c = getopt_long (argc, argv, "Vh", longopts, NULL)) != -1) {
> > +		switch (c) {
> > +		case 'V':
> > +			printf(UTIL_LINUX_VERSION);
> > +			return EXIT_SUCCESS;
> > +
> > +		case 'h':
> > +			usage();
> > +
> > +		default:
> > +			errtryhelp(EXIT_FAILURE);
> > +		}
> > +	}
> > +
> > +	if (optind == argc) {
> > +		warnx(_("no file specified"));
> > +		errtryhelp(EXIT_FAILURE);
> > +	}
> > +
> > +	for ( ; optind < argc; optind++) {
> > +		int fd = 0;
> > +
> > +		fd = open(argv[optind], O_RDONLY);
> > +		if (fd < 0) {
> > +			err(EXIT_FAILURE, _("open file failed"));
> > +		} else {
> > +			struct fiemap *fiemap = NULL;
> > +
> > +			fiemap = read_fiemap(fd);
> > +			if (fiemap != NULL) {
> > +				show_fiemap(fiemap, argv[optind]);
> > +				free(fiemap);
> > +			}
> > +			close(fd);
> > +		}
> > +	}
> > +
> > +	return 0;
> > +}
> > +#else
> > +int main(int argc, char **argv)
> > +{
> > +	err(EXIT_FAILURE, _("FS_IOC_FIEMAP not supported"));
> > +
> > +	return 0;
> > +}
> > +#endif
> > -- 
> > 2.7.4
> > 
> 
> -- 
>  Karel Zak  <kzak@redhat.com>
>  http://karelzak.blogspot.com

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, back to index

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <1554883940-19963-1-git-send-email-pizhenwei@bytedance.com>
2019-04-10  9:59 ` [PATCH v2] fiemap : add fiemap misc tool Karel Zak
2019-04-10 10:11   ` [External Email] " zhenwei pi
2019-04-10 12:12   ` Lukas Czerner

Util-Linux Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/util-linux/0 util-linux/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 util-linux util-linux/ https://lore.kernel.org/util-linux \
		util-linux@vger.kernel.org util-linux@archiver.kernel.org
	public-inbox-index util-linux


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.util-linux


AGPL code for this site: git clone https://public-inbox.org/ public-inbox