All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH][BTRFS-PROGS] Enhance btrfs fi df
@ 2012-11-02 10:15 Goffredo Baroncelli
  2012-11-02 10:15 ` [PATCH 1/8] Enhance the command btrfs filesystem df Goffredo Baroncelli
                   ` (8 more replies)
  0 siblings, 9 replies; 24+ messages in thread
From: Goffredo Baroncelli @ 2012-11-02 10:15 UTC (permalink / raw)
  To: linux-btrfs
  Cc: Hugo Mills, Michael Kjörling, Martin Steigerwald, cwillu,
	Chris Murphy

Hi all, on the basis of the discussion in the thread 
'[RFC] New attempt to a better "btrfs fi df"', I prepared the following
set of patches.
These patches update the btrfs fi df command and add two new commands:
- btrfs filesystem disk-usage <path>
- btrfs device disk-usage <path>

The command "btrfs filesystem df" now shows only the disk usage/available.

$ btrfs filesystem df /mnt/btrfs1/
Disk size:		 109.00GB
Disk allocated:		   5.90GB
Disk unallocated:	 103.10GB
Used:			 284.00KB
Free (Estimated):	  63.00GB	(Max: 106.51GB, min: 54.96GB)
Data to disk ratio:	     58 %

This was the "Summary" section of the my first attempt to enhance this
command.
The "Free (Estimated)" try to give an estimation of the free space
on the basis of the chunks usage.

The other two commands show the relation between the chunks and the disks.

$ sudo ./btrfs filesystem disk-usage /mnt/btrfs1/
Data,Single: Size:8.00MB, Used:0.00
   /dev/vdb	    8.00MB

Data,RAID0: Size:921.75MB, Used:256.00KB
   /dev/vdb	  307.25MB
   /dev/vdc	  307.25MB
   /dev/vdd	  307.25MB

Metadata,Single: Size:8.00MB, Used:0.00
   /dev/vdb	    8.00MB

Metadata,RAID1: Size:2.45GB, Used:24.00KB
   /dev/vdb	    1.45GB
   /dev/vdc	    1.00GB
   /dev/vdd	  460.94MB
   /dev/vdi	    2.00GB

System,Single: Size:4.00MB, Used:0.00
   /dev/vdb	    4.00MB

System,RAID1: Size:40.00MB, Used:4.00KB
   /dev/vdc	   40.00MB
   /dev/vdd	   40.00MB

Unallocated:
   /dev/vdb	    1.23GB
   /dev/vdc	    1.66GB
   /dev/vdd	    2.21GB
   /dev/vdi	   98.00GB

or in tabular format

$ sudo ./btrfs filesystem disk-usage -t /mnt/btrfs1/
         Data   Data     Metadata Metadata System System             
         Single RAID0    Single   RAID1    Single RAID1   Unallocated
                                                                     
/dev/vdb 8.00MB 307.25MB   8.00MB   1.45GB 4.00MB       -      1.23GB
/dev/vdc      - 307.25MB        -   1.00GB      - 40.00MB      1.66GB
/dev/vdd      - 307.25MB        - 460.94MB      - 40.00MB      2.21GB
/dev/vdi      -        -        -   2.00GB      -       -     98.00GB
         ====== ======== ======== ======== ====== ======= ===========
Total    8.00MB 921.75MB   8.00MB   2.45GB 4.00MB 40.00MB    103.10GB
Used       0.00 256.00KB     0.00  24.00KB   0.00  4.00KB


These are the most complete output, where it is possible to know which
disk a chunk uses and the usage of every chunk.

Finally the last command show which chunks a disk hosts:

$ sudo ./btrfs device disk-usage /mnt/btrfs1/
/dev/vdb	    3.00GB
   Data,Single:              8.00MB
   Data,RAID0:             307.25MB
   Metadata,Single:          8.00MB
   Metadata,RAID1:           1.45GB
   System,Single:            4.00MB
   Unallocated:              1.23GB

/dev/vdc	    3.00GB
   Data,RAID0:             307.25MB
   Metadata,RAID1:           1.00GB
   System,RAID1:            40.00MB
   Unallocated:              1.66GB

/dev/vdd	    3.00GB
   Data,RAID0:             307.25MB
   Metadata,RAID1:         460.94MB
   System,RAID1:            40.00MB
   Unallocated:              2.21GB

/dev/vdi	  100.00GB
   Metadata,RAID1:           2.00GB
   Unallocated:             98.00GB

More or less are the same information above, only grouped by disk.
Unfortunately I don't have any information on the chunk usage per disk basis.

Comments are welcome.
The code is pullable from
        http://cassiopea.homelinux.net/git/btrfs-progs-unstable.git
branch
        info-cmd

--
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5


^ permalink raw reply	[flat|nested] 24+ messages in thread

* [PATCH 1/8] Enhance the command btrfs filesystem df.
  2012-11-02 10:15 [PATCH][BTRFS-PROGS] Enhance btrfs fi df Goffredo Baroncelli
@ 2012-11-02 10:15 ` Goffredo Baroncelli
  2012-11-02 10:15 ` [PATCH 2/8] Create the man page entry for the command btrfs fi df Goffredo Baroncelli
                   ` (7 subsequent siblings)
  8 siblings, 0 replies; 24+ messages in thread
From: Goffredo Baroncelli @ 2012-11-02 10:15 UTC (permalink / raw)
  To: linux-btrfs
  Cc: Hugo Mills, Michael Kjörling, Martin Steigerwald, cwillu,
	Chris Murphy, Goffredo Baroncelli

From: Goffredo Baroncelli <kreijack@inwind.it>

Enhance the command "btrfs filesystem df" to show space usage information
for a mount point(s). It shows also an estimation of the space available,
on the basis of the current one used.

Signed-off-by: Goffredo Baroncelli <kreijack@inwind.it>
---
 Makefile             |    2 +-
 cmds-fi-disk_usage.c |  334 ++++++++++++++++++++++++++++++++++++++++++++++++++
 cmds-fi-disk_usage.h |   25 ++++
 cmds-filesystem.c    |  125 +------------------
 ctree.h              |   11 ++
 utils.c              |   13 ++
 utils.h              |    4 +
 7 files changed, 390 insertions(+), 124 deletions(-)
 create mode 100644 cmds-fi-disk_usage.c
 create mode 100644 cmds-fi-disk_usage.h

diff --git a/Makefile b/Makefile
index 4894903..4a9b6e0 100644
--- a/Makefile
+++ b/Makefile
@@ -8,7 +8,7 @@ objects = ctree.o disk-io.o radix-tree.o extent-tree.o print-tree.o \
 	  send-stream.o send-utils.o qgroup.o
 cmds_objects = cmds-subvolume.o cmds-filesystem.o cmds-device.o cmds-scrub.o \
 	       cmds-inspect.o cmds-balance.o cmds-send.o cmds-receive.o \
-	       cmds-quota.o cmds-qgroup.o
+	       cmds-quota.o cmds-qgroup.o cmds-fi-disk_usage.o
 
 CHECKFLAGS= -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ -Wbitwise \
 	    -Wuninitialized -Wshadow -Wundef
diff --git a/cmds-fi-disk_usage.c b/cmds-fi-disk_usage.c
new file mode 100644
index 0000000..9131c47
--- /dev/null
+++ b/cmds-fi-disk_usage.c
@@ -0,0 +1,334 @@
+/*
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/ioctl.h>
+#include <errno.h>
+
+#include "utils.h"
+#include "kerncompat.h"
+#include "ctree.h"
+
+#include "commands.h"
+
+#include "version.h"
+
+#define DF_HUMAN_UNIT		(1<<0)
+
+/* to store the information about the chunk */
+struct chunk_info {
+	u64	type;
+	u64	size;
+	u64	devid;
+	int	processed:1;
+};
+
+struct disk_info {
+	u64	devid;
+	char	path[BTRFS_DEVICE_PATH_NAME_MAX];
+	u64	size;
+};
+
+/* to store the tmp strings */
+static void **strings_to_free;
+static int count_string_to_free;
+
+static void add_strings_to_free(char *s)
+{
+	int  size;
+
+	size = sizeof(void *) * ++count_string_to_free;
+	strings_to_free = realloc(strings_to_free, size);
+
+	/* if we don't have enough memory, we have more serius
+	   problem than that a wrong handling of not enough memory */
+	if (!strings_to_free) {
+		fprintf(stderr, "add_string_to_free(): Not enough memory\n");
+		strings_to_free = 0;
+		count_string_to_free = 0;
+	}
+
+	strings_to_free[count_string_to_free-1] = s;
+}
+
+static void free_strings_to_free()
+{
+	int	i;
+	for (i = 0 ; i < count_string_to_free ; i++)
+		free(strings_to_free[i]);
+
+	free(strings_to_free);
+
+	strings_to_free = 0;
+	count_string_to_free = 0;
+}
+
+static char *df_pretty_sizes(u64 size, int mode)
+{
+	char *s;
+
+	if (mode & DF_HUMAN_UNIT) {
+		s = pretty_sizes(size);
+		if (!s)
+			return NULL;
+	} else {
+		s = malloc(20);
+		if (!s)
+			return NULL;
+		sprintf(s, "%llu", size);
+	}
+
+	add_strings_to_free(s);
+	return s;
+}
+
+static int cmp_chunk_block_group(u64 f1, u64 f2)
+{
+
+	u64 mask;
+
+	if ((f1 & BTRFS_BLOCK_GROUP_TYPE_MASK) ==
+		(f2 & BTRFS_BLOCK_GROUP_TYPE_MASK))
+			mask = BTRFS_BLOCK_GROUP_PROFILE_MASK;
+	else if (f2 & BTRFS_BLOCK_GROUP_SYSTEM)
+		return -1;
+	else if (f1 & BTRFS_BLOCK_GROUP_SYSTEM)
+		return +1;
+	else
+			mask = BTRFS_BLOCK_GROUP_TYPE_MASK;
+
+	if ((f1 & mask) > (f2 & mask))
+		return +1;
+	else if ((f1 & mask) < (f2 & mask))
+		return -1;
+	else
+		return 0;
+}
+
+static int cmp_btrfs_ioctl_space_info(const void *a, const void *b)
+{
+	return cmp_chunk_block_group(
+		((struct btrfs_ioctl_space_info *)a)->flags,
+		((struct btrfs_ioctl_space_info *)b)->flags);
+}
+
+static struct btrfs_ioctl_space_args *load_space_info(int fd, char *path)
+{
+	struct btrfs_ioctl_space_args *sargs = 0, *sargs_orig = 0;
+	int e, ret, count;
+
+	sargs_orig = sargs = malloc(sizeof(struct btrfs_ioctl_space_args));
+	if (!sargs) {
+		fprintf(stderr, "ERROR: not enough memory\n");
+		return NULL;
+	}
+
+	sargs->space_slots = 0;
+	sargs->total_spaces = 0;
+
+	ret = ioctl(fd, BTRFS_IOC_SPACE_INFO, sargs);
+	e = errno;
+	if (ret) {
+		fprintf(stderr,
+			"ERROR: couldn't get space info on '%s' - %s\n",
+			path, strerror(e));
+		free(sargs);
+		return NULL;
+	}
+	if (!sargs->total_spaces) {
+		free(sargs);
+		printf("No chunks found\n");
+		return NULL;
+	}
+
+	count = sargs->total_spaces;
+
+	sargs = realloc(sargs, sizeof(struct btrfs_ioctl_space_args) +
+			(count * sizeof(struct btrfs_ioctl_space_info)));
+	if (!sargs) {
+		free(sargs_orig);
+		fprintf(stderr, "ERROR: not enough memory\n");
+		return NULL;
+	}
+
+	sargs->space_slots = count;
+	sargs->total_spaces = 0;
+
+	ret = ioctl(fd, BTRFS_IOC_SPACE_INFO, sargs);
+	e = errno;
+
+	if (ret) {
+		fprintf(stderr,
+			"ERROR: couldn't get space info on '%s' - %s\n",
+			path, strerror(e));
+		free(sargs);
+		return NULL;
+	}
+
+	qsort(&(sargs->spaces), count, sizeof(struct btrfs_ioctl_space_info),
+		cmp_btrfs_ioctl_space_info);
+
+	return sargs;
+}
+
+static int _cmd_disk_free(int fd, char *path, int mode)
+{
+	struct btrfs_ioctl_space_args *sargs = 0;
+	int i;
+	int ret = 0;
+	int e, width;
+	u64 total_disk;		/* filesystem size == sum of
+				   disks sizes */
+	u64 total_chunks;	/* sum of chunks sizes on disk(s) */
+	u64 total_used;		/* logical space used */
+	u64 total_free;		/* logical space un-used */
+	double K;
+
+	if ((sargs = load_space_info(fd, path)) == NULL) {
+		ret = -1;
+		goto exit;
+	}
+
+	total_disk = disk_size(path);
+	e = errno;
+	if (total_disk == 0) {
+		fprintf(stderr,
+			"ERROR: couldn't get space info on '%s' - %s\n",
+			path, strerror(e));
+
+		ret = 19;
+		goto exit;
+	}
+
+	total_chunks = total_used = total_free = 0;
+
+	for (i = 0; i < sargs->total_spaces; i++) {
+		int  ratio = 1;
+		u64  allocated;
+
+		u64 flags = sargs->spaces[i].flags;
+
+		if (flags & BTRFS_BLOCK_GROUP_RAID0)
+			ratio = 1;
+		else if (flags & BTRFS_BLOCK_GROUP_RAID1)
+			ratio = 2;
+		else if (flags & BTRFS_BLOCK_GROUP_DUP)
+			ratio = 2;
+		else if (flags & BTRFS_BLOCK_GROUP_RAID10)
+			ratio = 2;
+		else
+			ratio = 1;
+
+		allocated = sargs->spaces[i].total_bytes * ratio;
+
+		total_chunks += allocated;
+		total_used += sargs->spaces[i].used_bytes;
+		total_free += (sargs->spaces[i].total_bytes -
+					sargs->spaces[i].used_bytes);
+
+	}
+	K = ((double)total_used + (double)total_free) /	(double)total_chunks;
+
+	if (mode & DF_HUMAN_UNIT)
+		width = 9;
+	else
+		width = 18;
+
+	printf("Disk size:\t\t%*s\n", width,
+		df_pretty_sizes(total_disk, mode));
+	printf("Disk allocated:\t\t%*s\n", width,
+		df_pretty_sizes(total_chunks, mode));
+	printf("Disk unallocated:\t%*s\n", width,
+		df_pretty_sizes(total_disk-total_chunks, mode));
+	printf("Used:\t\t\t%*s\n", width,
+		df_pretty_sizes(total_used, mode));
+	printf("Free (Estimated):\t%*s\t(Max: %s, min: %s)\n",
+		width,
+		df_pretty_sizes((u64)(K*total_disk-total_used), mode),
+		df_pretty_sizes(total_disk-total_chunks+total_free, mode),
+		df_pretty_sizes((total_disk-total_chunks)/2+total_free, mode));
+	printf("Data to disk ratio:\t%*.0f %%\n",
+		width-2, K*100);
+
+exit:
+
+	free_strings_to_free();
+	if (sargs)
+		free(sargs);
+
+	return ret;
+}
+
+const char * const cmd_filesystem_df_usage[] = {
+	"btrfs filesystem df [-k] <path> [<path>..]",
+	"Show space usage information for a mount point(s).",
+	"",
+	"-b\tSet byte as unit",
+	NULL
+};
+
+int cmd_filesystem_df(int argc, char **argv)
+{
+
+	int	flags = DF_HUMAN_UNIT;
+	int	i, more_than_one = 0;
+
+	optind = 1;
+	while (1) {
+		char	c = getopt(argc, argv, "b");
+		if (c < 0)
+			break;
+
+		switch (c) {
+		case 'b':
+			flags &= ~DF_HUMAN_UNIT;
+			break;
+		default:
+			usage(cmd_filesystem_df_usage);
+		}
+	}
+
+	if (check_argc_min(argc - optind, 1)) {
+		usage(cmd_filesystem_df_usage);
+		return 21;
+	}
+
+	for (i = optind; i < argc ; i++) {
+		int r, fd;
+		if (more_than_one)
+			printf("\n");
+
+		fd = open_file_or_dir(argv[i]);
+		if (fd < 0) {
+			fprintf(stderr, "ERROR: can't access to '%s'\n",
+				argv[1]);
+			return 12;
+		}
+		r = _cmd_disk_free(fd, argv[i], flags);
+		close(fd);
+
+		if (r)
+			return r;
+		more_than_one = 1;
+
+	}
+
+	return 0;
+}
+
diff --git a/cmds-fi-disk_usage.h b/cmds-fi-disk_usage.h
new file mode 100644
index 0000000..9f68bb3
--- /dev/null
+++ b/cmds-fi-disk_usage.h
@@ -0,0 +1,25 @@
+/*
+ * Copyright (C) 2007 Oracle.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ */
+
+#ifndef __CMDS_FI_DISK_USAGE__
+#define __CMDS_FI_DISK_USAGE__
+
+extern const char * const cmd_filesystem_df_usage[];
+int cmd_filesystem_df(int argc, char **argv);
+
+#endif
diff --git a/cmds-filesystem.c b/cmds-filesystem.c
index 9c43d35..1b915e4 100644
--- a/cmds-filesystem.c
+++ b/cmds-filesystem.c
@@ -33,134 +33,13 @@
 
 #include "commands.h"
 #include "btrfslabel.h"
+#include "cmds-fi-disk_usage.h"
 
 static const char * const filesystem_cmd_group_usage[] = {
 	"btrfs filesystem [<group>] <command> [<args>]",
 	NULL
 };
 
-static const char * const cmd_df_usage[] = {
-	"btrfs filesystem df <path>",
-	"Show space usage information for a mount point",
-	NULL
-};
-
-static int cmd_df(int argc, char **argv)
-{
-	struct btrfs_ioctl_space_args *sargs, *sargs_orig;
-	u64 count = 0, i;
-	int ret;
-	int fd;
-	int e;
-	char *path;
-
-	if (check_argc_exact(argc, 2))
-		usage(cmd_df_usage);
-
-	path = argv[1];
-
-	fd = open_file_or_dir(path);
-	if (fd < 0) {
-		fprintf(stderr, "ERROR: can't access to '%s'\n", path);
-		return 12;
-	}
-
-	sargs_orig = sargs = malloc(sizeof(struct btrfs_ioctl_space_args));
-	if (!sargs)
-		return -ENOMEM;
-
-	sargs->space_slots = 0;
-	sargs->total_spaces = 0;
-
-	ret = ioctl(fd, BTRFS_IOC_SPACE_INFO, sargs);
-	e = errno;
-	if (ret) {
-		fprintf(stderr, "ERROR: couldn't get space info on '%s' - %s\n",
-			path, strerror(e));
-		close(fd);
-		free(sargs);
-		return ret;
-	}
-	if (!sargs->total_spaces) {
-		close(fd);
-		free(sargs);
-		return 0;
-	}
-
-	count = sargs->total_spaces;
-
-	sargs = realloc(sargs, sizeof(struct btrfs_ioctl_space_args) +
-			(count * sizeof(struct btrfs_ioctl_space_info)));
-	if (!sargs) {
-		close(fd);
-		free(sargs_orig);
-		return -ENOMEM;
-	}
-
-	sargs->space_slots = count;
-	sargs->total_spaces = 0;
-
-	ret = ioctl(fd, BTRFS_IOC_SPACE_INFO, sargs);
-	e = errno;
-	if (ret) {
-		fprintf(stderr, "ERROR: couldn't get space info on '%s' - %s\n",
-			path, strerror(e));
-		close(fd);
-		free(sargs);
-		return ret;
-	}
-
-	for (i = 0; i < sargs->total_spaces; i++) {
-		char description[80];
-		char *total_bytes;
-		char *used_bytes;
-		int written = 0;
-		u64 flags = sargs->spaces[i].flags;
-
-		memset(description, 0, 80);
-
-		if (flags & BTRFS_BLOCK_GROUP_DATA) {
-			if (flags & BTRFS_BLOCK_GROUP_METADATA) {
-				snprintf(description, 14, "%s",
-					 "Data+Metadata");
-				written += 13;
-			} else {
-				snprintf(description, 5, "%s", "Data");
-				written += 4;
-			}
-		} else if (flags & BTRFS_BLOCK_GROUP_SYSTEM) {
-			snprintf(description, 7, "%s", "System");
-			written += 6;
-		} else if (flags & BTRFS_BLOCK_GROUP_METADATA) {
-			snprintf(description, 9, "%s", "Metadata");
-			written += 8;
-		}
-
-		if (flags & BTRFS_BLOCK_GROUP_RAID0) {
-			snprintf(description+written, 8, "%s", ", RAID0");
-			written += 7;
-		} else if (flags & BTRFS_BLOCK_GROUP_RAID1) {
-			snprintf(description+written, 8, "%s", ", RAID1");
-			written += 7;
-		} else if (flags & BTRFS_BLOCK_GROUP_DUP) {
-			snprintf(description+written, 6, "%s", ", DUP");
-			written += 5;
-		} else if (flags & BTRFS_BLOCK_GROUP_RAID10) {
-			snprintf(description+written, 9, "%s", ", RAID10");
-			written += 8;
-		}
-
-		total_bytes = pretty_sizes(sargs->spaces[i].total_bytes);
-		used_bytes = pretty_sizes(sargs->spaces[i].used_bytes);
-		printf("%s: total=%s, used=%s\n", description, total_bytes,
-		       used_bytes);
-	}
-	close(fd);
-	free(sargs);
-
-	return 0;
-}
-
 static int uuid_search(struct btrfs_fs_devices *fs_devices, char *search)
 {
 	char uuidbuf[37];
@@ -537,7 +416,7 @@ static int cmd_label(int argc, char **argv)
 
 const struct cmd_group filesystem_cmd_group = {
 	filesystem_cmd_group_usage, NULL, {
-		{ "df", cmd_df, cmd_df_usage, NULL, 0 },
+		{ "df", cmd_filesystem_df, cmd_filesystem_df_usage, NULL, 0 },
 		{ "show", cmd_show, cmd_show_usage, NULL, 0 },
 		{ "sync", cmd_sync, cmd_sync_usage, NULL, 0 },
 		{ "defragment", cmd_defrag, cmd_defrag_usage, NULL, 0 },
diff --git a/ctree.h b/ctree.h
index 293b24f..7c7d077 100644
--- a/ctree.h
+++ b/ctree.h
@@ -780,6 +780,17 @@ struct btrfs_csum_item {
 #define BTRFS_BLOCK_GROUP_DUP		(1ULL << 5)
 #define BTRFS_BLOCK_GROUP_RAID10	(1ULL << 6)
 #define BTRFS_BLOCK_GROUP_RESERVED	BTRFS_AVAIL_ALLOC_BIT_SINGLE
+#define BTRFS_NR_RAID_TYPES             5
+
+#define BTRFS_BLOCK_GROUP_TYPE_MASK     (BTRFS_BLOCK_GROUP_DATA |    \
+					 BTRFS_BLOCK_GROUP_SYSTEM |  \
+					 BTRFS_BLOCK_GROUP_METADATA)
+
+#define BTRFS_BLOCK_GROUP_PROFILE_MASK  (BTRFS_BLOCK_GROUP_RAID0 |   \
+					 BTRFS_BLOCK_GROUP_RAID1 |   \
+					 BTRFS_BLOCK_GROUP_DUP |     \
+					 BTRFS_BLOCK_GROUP_RAID10)
+
 
 /* used in struct btrfs_balance_args fields */
 #define BTRFS_AVAIL_ALLOC_BIT_SINGLE	(1ULL << 48)
diff --git a/utils.c b/utils.c
index 205e667..fba11e0 100644
--- a/utils.c
+++ b/utils.c
@@ -35,6 +35,8 @@
 #include <linux/major.h>
 #include <linux/kdev_t.h>
 #include <limits.h>
+#include <sys/vfs.h>
+
 #include "kerncompat.h"
 #include "radix-tree.h"
 #include "ctree.h"
@@ -1220,3 +1222,14 @@ scan_again:
 	return 0;
 }
 
+u64 disk_size(char *path)
+{
+	struct statfs	sfs;
+
+	if (statfs(path, &sfs) < 0)
+		return 0;
+	else
+		return sfs.f_bsize * sfs.f_blocks;
+
+}
+
diff --git a/utils.h b/utils.h
index 3a0368b..a82b81c 100644
--- a/utils.h
+++ b/utils.h
@@ -19,6 +19,9 @@
 #ifndef __UTILS__
 #define __UTILS__
 
+#include "kerncompat.h"
+#include "ctree.h"
+
 #define BTRFS_MKFS_SYSTEM_GROUP_SIZE (4 * 1024 * 1024)
 
 int make_btrfs(int fd, const char *device, const char *label,
@@ -46,4 +49,5 @@ int check_label(char *input);
 int get_mountpt(char *dev, char *mntpt, size_t size);
 
 int btrfs_scan_block_devices(int run_ioctl);
+u64 disk_size(char *path);
 #endif
-- 
1.7.10.4


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 2/8] Create the man page entry for the command btrfs fi df
  2012-11-02 10:15 [PATCH][BTRFS-PROGS] Enhance btrfs fi df Goffredo Baroncelli
  2012-11-02 10:15 ` [PATCH 1/8] Enhance the command btrfs filesystem df Goffredo Baroncelli
@ 2012-11-02 10:15 ` Goffredo Baroncelli
  2012-11-02 10:15 ` [PATCH 3/8] Move open_file_or_dir() in utils.c Goffredo Baroncelli
                   ` (6 subsequent siblings)
  8 siblings, 0 replies; 24+ messages in thread
From: Goffredo Baroncelli @ 2012-11-02 10:15 UTC (permalink / raw)
  To: linux-btrfs
  Cc: Hugo Mills, Michael Kjörling, Martin Steigerwald, cwillu,
	Chris Murphy, Goffredo Baroncelli

From: Goffredo Baroncelli <kreijack@inwind.it>

Signed-off-by: Goffredo Baroncelli <kreijack@inwind.it>
---
 man/btrfs.8.in |   49 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/man/btrfs.8.in b/man/btrfs.8.in
index 9222580..27d0160 100644
--- a/man/btrfs.8.in
+++ b/man/btrfs.8.in
@@ -27,6 +27,8 @@ btrfs \- control a btrfs filesystem
 .PP
 \fBbtrfs\fP \fBfilesystem label\fP\fI <dev> [newlabel]\fP
 .PP
+\fBbtrfs\fP \fBfilesystem df\fP\fI [-b] \fIpath [path..]\fR\fP
+.PP
 \fBbtrfs\fP \fBsubvolume find-new\fP\fI <subvolume> <last_gen>\fP
 .PP
 \fBbtrfs\fP \fBfilesystem balance\fP\fI <path> \fP
@@ -242,6 +244,53 @@ NOTE: Currently there are the following limitations:
 - the filesystem should not have more than one device.
 .TP
 
+\fBfilesystem df\fP [-b] \fIpath [path..]\fR
+
+Show space usage information for a mount point.
+
+\fB-b\fP Set byte as unit
+
+The command \fBbtrfs filesystem df\fP is used to query how many space on the 
+disk(s) are used and an estimation of the free
+space of the filesystem.
+The output of the command \fBbtrfs filesystem df\fP shows:
+
+.RS
+.IP \fBDisk\ size\fP
+the total size of the disks which compose the filesystem.
+
+.IP \fBDisk\ allocated\fP
+the size of the area of the disks used by the chunks.
+
+.IP \fBDisk\ unallocated\fP 
+the size of the area of the disks which is free (i.e.
+the differences of the values above).
+
+.IP \fBUsed\fP
+the portion of the logical space used by the file and metadata.
+
+.IP \fBFree\ (estimated)\fP
+the estimated free space available: i.e. how many space can be used
+by the user. The evaluation 
+cannot be rigorous because it depends by the allocation policy (DUP, Single,
+RAID1...) of the metadata and data chunks. If every chunk is stored as
+"Single" the sum of the \fBfree (estimated)\fP space and the \fBused\fP 
+space  is equal to the \fBdisk size\fP.
+Otherwise if all the chunk are mirrored (raid1 or raid10) or duplicated
+the sum of the \fBfree (estimated)\fP space and the \fBused\fP space is
+half of the \fBdisk size\fP. Normally the \fBfree (estimated)\fP is between
+these two limits.
+
+.IP \fBData\ to\ disk\ ratio\fP
+the ratio betwen the \fBlogical size\fP (i.e. the space available by
+the chunks) and the \fBdisk allocated\fP (by the chunks). Normally it is 
+lower than 100% because the metadata is duplicated for security reasons.
+If all the data and metadata are duplicated (or have a profile like RAID1)
+the \fBData\ to\ disk\ ratio\fP could be 50%.
+
+.RE
+.TP
+
 \fBfilesystem show\fR [--all-devices|<uuid>|<label>]\fR
 Show the btrfs filesystem with some additional info. If no \fIUUID\fP or 
 \fIlabel\fP is passed, \fBbtrfs\fR show info of all the btrfs filesystem.
-- 
1.7.10.4


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 3/8] Move open_file_or_dir() in utils.c
  2012-11-02 10:15 [PATCH][BTRFS-PROGS] Enhance btrfs fi df Goffredo Baroncelli
  2012-11-02 10:15 ` [PATCH 1/8] Enhance the command btrfs filesystem df Goffredo Baroncelli
  2012-11-02 10:15 ` [PATCH 2/8] Create the man page entry for the command btrfs fi df Goffredo Baroncelli
@ 2012-11-02 10:15 ` Goffredo Baroncelli
  2012-11-02 10:15 ` [PATCH 4/8] Move scrub_fs_info() and scrub_dev_info() " Goffredo Baroncelli
                   ` (5 subsequent siblings)
  8 siblings, 0 replies; 24+ messages in thread
From: Goffredo Baroncelli @ 2012-11-02 10:15 UTC (permalink / raw)
  To: linux-btrfs
  Cc: Hugo Mills, Michael Kjörling, Martin Steigerwald, cwillu,
	Chris Murphy, Goffredo Baroncelli

From: Goffredo Baroncelli <kreijack@inwind.it>

Due to the moving of the scrub_fs_info() and scrub_device_info() in utils.h
the function open_file_or_dir() is moved in utils.c too, in order to avoid
the including of "commands.h".
Moreover in btrfsctl.c the function open_file_or_dir() is renamed as
btrfsctl_open_file_or_dir() in order to avoid collision name because these
two functions are a bit different.
Since open_file_or_dir() makes use of the function dirfd(3), the required
XOPEN version was raised from 6 to 7.

Signed-off-by: Goffredo Baroncelli <kreijack@inwind.it>
---
 Makefile         |    4 ++--
 btrfsctl.c       |    6 +++---
 cmds-balance.c   |    1 +
 cmds-inspect.c   |    1 +
 cmds-qgroup.c    |    1 +
 cmds-quota.c     |    1 +
 cmds-subvolume.c |    1 +
 commands.h       |    3 ---
 common.c         |   46 ----------------------------------------------
 utils.c          |   29 +++++++++++++++++++++++++++--
 utils.h          |    1 +
 11 files changed, 38 insertions(+), 56 deletions(-)
 delete mode 100644 common.c

diff --git a/Makefile b/Makefile
index 4a9b6e0..9a45a84 100644
--- a/Makefile
+++ b/Makefile
@@ -41,8 +41,8 @@ all: version $(progs) manpages
 version:
 	bash version.sh
 
-btrfs: $(objects) btrfs.o help.o common.o $(cmds_objects)
-	$(CC) $(CFLAGS) -o btrfs btrfs.o help.o common.o $(cmds_objects) \
+btrfs: $(objects) btrfs.o help.o $(cmds_objects)
+	$(CC) $(CFLAGS) -o btrfs btrfs.o help.o $(cmds_objects) \
 		$(objects) $(LDFLAGS) $(LIBS) -lpthread
 
 calc-size: $(objects) calc-size.o
diff --git a/btrfsctl.c b/btrfsctl.c
index 518684c..c1981ba 100644
--- a/btrfsctl.c
+++ b/btrfsctl.c
@@ -63,7 +63,7 @@ static void print_usage(void)
 	exit(1);
 }
 
-static int open_file_or_dir(const char *fname)
+static int btrfsctl_open_file_or_dir(const char *fname)
 {
 	int ret;
 	struct stat st;
@@ -128,7 +128,7 @@ int main(int ac, char **av)
 			snap_location = strdup(fullpath);
 			snap_location = dirname(snap_location);
 
-			snap_fd = open_file_or_dir(snap_location);
+			snap_fd = btrfsctl_open_file_or_dir(snap_location);
 
 			name = strdup(fullpath);
 			name = basename(name);
@@ -238,7 +238,7 @@ int main(int ac, char **av)
 		}
 		name = fname;
 	 } else {
-		fd = open_file_or_dir(fname);
+		fd = btrfsctl_open_file_or_dir(fname);
 	 }
 
 	if (name) {
diff --git a/cmds-balance.c b/cmds-balance.c
index 38a7426..5793b5c 100644
--- a/cmds-balance.c
+++ b/cmds-balance.c
@@ -26,6 +26,7 @@
 #include "ctree.h"
 #include "ioctl.h"
 #include "volumes.h"
+#include "utils.h"
 
 #include "commands.h"
 
diff --git a/cmds-inspect.c b/cmds-inspect.c
index edabff5..043b65b 100644
--- a/cmds-inspect.c
+++ b/cmds-inspect.c
@@ -21,6 +21,7 @@
 #include <errno.h>
 
 #include "kerncompat.h"
+#include "utils.h"
 #include "ioctl.h"
 
 #include "commands.h"
diff --git a/cmds-qgroup.c b/cmds-qgroup.c
index 1525c11..e9f3143 100644
--- a/cmds-qgroup.c
+++ b/cmds-qgroup.c
@@ -22,6 +22,7 @@
 
 #include "ctree.h"
 #include "ioctl.h"
+#include "utils.h"
 
 #include "commands.h"
 
diff --git a/cmds-quota.c b/cmds-quota.c
index cf9ad97..ca9503a 100644
--- a/cmds-quota.c
+++ b/cmds-quota.c
@@ -21,6 +21,7 @@
 
 #include "ctree.h"
 #include "ioctl.h"
+#include "utils.h"
 
 #include "commands.h"
 
diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index ac39f7b..e3cdb1e 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -32,6 +32,7 @@
 #include "ctree.h"
 #include "commands.h"
 #include "btrfs-list.h"
+#include "utils.h"
 
 static const char * const subvolume_cmd_group_usage[] = {
 	"btrfs subvolume <command> <args>",
diff --git a/commands.h b/commands.h
index bb6d2dd..8114a73 100644
--- a/commands.h
+++ b/commands.h
@@ -79,9 +79,6 @@ void help_ambiguous_token(const char *arg, const struct cmd_group *grp);
 
 void help_command_group(const struct cmd_group *grp, int argc, char **argv);
 
-/* common.c */
-int open_file_or_dir(const char *fname);
-
 extern const struct cmd_group subvolume_cmd_group;
 extern const struct cmd_group filesystem_cmd_group;
 extern const struct cmd_group balance_cmd_group;
diff --git a/common.c b/common.c
deleted file mode 100644
index 03f6570..0000000
--- a/common.c
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public
- * License v2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- * General Public License for more details.
- *
- * You should have received a copy of the GNU General Public
- * License along with this program; if not, write to the
- * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
- * Boston, MA 021110-1307, USA.
- */
-
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <dirent.h>
-#include <fcntl.h>
-
-int open_file_or_dir(const char *fname)
-{
-	int ret;
-	struct stat st;
-	DIR *dirstream;
-	int fd;
-
-	ret = stat(fname, &st);
-	if (ret < 0) {
-		return -1;
-	}
-	if (S_ISDIR(st.st_mode)) {
-		dirstream = opendir(fname);
-		if (!dirstream) {
-			return -2;
-		}
-		fd = dirfd(dirstream);
-	} else {
-		fd = open(fname, O_RDWR);
-	}
-	if (fd < 0) {
-		return -3;
-	}
-	return fd;
-}
diff --git a/utils.c b/utils.c
index fba11e0..f29745f 100644
--- a/utils.c
+++ b/utils.c
@@ -16,8 +16,9 @@
  * Boston, MA 021110-1307, USA.
  */
 
-#define _XOPEN_SOURCE 600
-#define __USE_XOPEN2K
+#define _XOPEN_SOURCE 700
+#define __USE_XOPEN2K8
+#define __XOPEN2K8
 #include <stdio.h>
 #include <stdlib.h>
 #ifndef __CHECKER__
@@ -1117,6 +1118,30 @@ char *pretty_sizes(u64 size)
 	return pretty;
 }
 
+int open_file_or_dir(const char *fname)
+{
+	int ret;
+	struct stat st;
+	DIR *dirstream;
+	int fd;
+
+	ret = stat(fname, &st);
+	if (ret < 0)
+		return -1;
+	if (S_ISDIR(st.st_mode)) {
+		dirstream = opendir(fname);
+		if (!dirstream)
+			return -2;
+		fd = dirfd(dirstream);
+	} else {
+		fd = open(fname, O_RDWR);
+	}
+	if (fd < 0)
+		return -3;
+
+	return fd;
+}
+
 /*
  * Checks to make sure that the label matches our requirements.
  * Returns:
diff --git a/utils.h b/utils.h
index a82b81c..f0100c2 100644
--- a/utils.h
+++ b/utils.h
@@ -45,6 +45,7 @@ int check_mounted_where(int fd, const char *file, char *where, int size,
 int btrfs_device_already_in_root(struct btrfs_root *root, int fd,
 				 int super_offset);
 char *pretty_sizes(u64 size);
+int open_file_or_dir(const char *fname);
 int check_label(char *input);
 int get_mountpt(char *dev, char *mntpt, size_t size);
 
-- 
1.7.10.4


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 4/8] Move scrub_fs_info() and scrub_dev_info() in utils.c
  2012-11-02 10:15 [PATCH][BTRFS-PROGS] Enhance btrfs fi df Goffredo Baroncelli
                   ` (2 preceding siblings ...)
  2012-11-02 10:15 ` [PATCH 3/8] Move open_file_or_dir() in utils.c Goffredo Baroncelli
@ 2012-11-02 10:15 ` Goffredo Baroncelli
  2012-11-02 10:15 ` [PATCH 5/8] Add command btrfs filesystem disk-usage Goffredo Baroncelli
                   ` (4 subsequent siblings)
  8 siblings, 0 replies; 24+ messages in thread
From: Goffredo Baroncelli @ 2012-11-02 10:15 UTC (permalink / raw)
  To: linux-btrfs
  Cc: Hugo Mills, Michael Kjörling, Martin Steigerwald, cwillu,
	Chris Murphy, Goffredo Baroncelli

From: Goffredo Baroncelli <kreijack@inwind.it>

Move scrub_fs_info() and scrub_dev_info() in utils.c because these
will be used by the command "btrfs filesystem disk-usage" and
"btrfs device disk-usage".

Signed-off-by: Goffredo Baroncelli <kreijack@inwind.it>
---
 cmds-scrub.c |   87 ++--------------------------------------------------------
 utils.c      |   83 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 utils.h      |    5 ++++
 3 files changed, 90 insertions(+), 85 deletions(-)

diff --git a/cmds-scrub.c b/cmds-scrub.c
index 24be20f..8a5bf83 100644
--- a/cmds-scrub.c
+++ b/cmds-scrub.c
@@ -967,89 +967,6 @@ static struct scrub_file_record *last_dev_scrub(
 	return NULL;
 }
 
-static int scrub_device_info(int fd, u64 devid,
-			     struct btrfs_ioctl_dev_info_args *di_args)
-{
-	int ret;
-
-	di_args->devid = devid;
-	memset(&di_args->uuid, '\0', sizeof(di_args->uuid));
-
-	ret = ioctl(fd, BTRFS_IOC_DEV_INFO, di_args);
-	return ret ? -errno : 0;
-}
-
-static int scrub_fs_info(char *path,
-				struct btrfs_ioctl_fs_info_args *fi_args,
-				struct btrfs_ioctl_dev_info_args **di_ret)
-{
-	int ret = 0;
-	int ndevs = 0;
-	int i = 1;
-	int fd;
-	struct btrfs_fs_devices *fs_devices_mnt = NULL;
-	struct btrfs_ioctl_dev_info_args *di_args;
-	char mp[BTRFS_PATH_NAME_MAX + 1];
-
-	memset(fi_args, 0, sizeof(*fi_args));
-
-	fd  = open_file_or_dir(path);
-	if (fd < 0) {
-	       fprintf(stderr, "ERROR: can't access to '%s'\n", path);
-	       return -1;
-	}
-
-	ret = ioctl(fd, BTRFS_IOC_FS_INFO, fi_args);
-	if (ret && errno == EINVAL) {
-		/* path is no mounted btrfs. try if it's a device */
-		ret = check_mounted_where(fd, path, mp, sizeof(mp),
-						&fs_devices_mnt);
-		if (!ret)
-			return -EINVAL;
-		if (ret < 0)
-			return ret;
-		fi_args->num_devices = 1;
-		fi_args->max_id = fs_devices_mnt->latest_devid;
-		i = fs_devices_mnt->latest_devid;
-		memcpy(fi_args->fsid, fs_devices_mnt->fsid, BTRFS_FSID_SIZE);
-		close(fd);
-		fd = open_file_or_dir(mp);
-		if (fd < 0)
-			return -errno;
-	} else if (ret) {
-		close(fd);
-		return -errno;
-	}
-
-	if (!fi_args->num_devices) {
-		close(fd);
-		return 0;
-	}
-
-	di_args = *di_ret = malloc(fi_args->num_devices * sizeof(*di_args));
-	if (!di_args) {
-		close(fd);
-		return -errno;
-	}
-
-	for (; i <= fi_args->max_id; ++i) {
-		BUG_ON(ndevs >= fi_args->num_devices);
-		ret = scrub_device_info(fd, i, &di_args[ndevs]);
-		if (ret == -ENODEV)
-			continue;
-		if (ret) {
-			close(fd);
-			return ret;
-		}
-		++ndevs;
-	}
-
-	BUG_ON(ndevs == 0);
-
-	close(fd);
-	return 0;
-}
-
 int mkdir_p(char *path)
 {
 	int i;
@@ -1170,7 +1087,7 @@ static int scrub_start(int argc, char **argv, int resume)
 		return 12;
 	}
 
-	ret = scrub_fs_info(path, &fi_args, &di_args);
+	ret = get_fs_info(path, &fi_args, &di_args);
 	if (ret) {
 		ERR(!do_quiet, "ERROR: getting dev info for scrub failed: "
 		    "%s\n", strerror(-ret));
@@ -1629,7 +1546,7 @@ static int cmd_scrub_status(int argc, char **argv)
 
 	path = argv[optind];
 
-	ret = scrub_fs_info(path, &fi_args, &di_args);
+	ret = get_fs_info(path, &fi_args, &di_args);
 	if (ret) {
 		fprintf(stderr, "ERROR: getting dev info for scrub failed: "
 				"%s\n", strerror(-ret));
diff --git a/utils.c b/utils.c
index f29745f..023fbca 100644
--- a/utils.c
+++ b/utils.c
@@ -1142,6 +1142,89 @@ int open_file_or_dir(const char *fname)
 	return fd;
 }
 
+int get_device_info(int fd, u64 devid,
+		     struct btrfs_ioctl_dev_info_args *di_args)
+{
+	int ret;
+
+	di_args->devid = devid;
+	memset(&di_args->uuid, '\0', sizeof(di_args->uuid));
+
+	ret = ioctl(fd, BTRFS_IOC_DEV_INFO, di_args);
+	return ret ? -errno : 0;
+}
+
+int get_fs_info(char *path,
+		struct btrfs_ioctl_fs_info_args *fi_args,
+		struct btrfs_ioctl_dev_info_args **di_ret)
+{
+	int ret = 0;
+	int ndevs = 0;
+	int i = 1;
+	int fd;
+	struct btrfs_fs_devices *fs_devices_mnt = NULL;
+	struct btrfs_ioctl_dev_info_args *di_args;
+	char mp[BTRFS_PATH_NAME_MAX + 1];
+
+	memset(fi_args, 0, sizeof(*fi_args));
+
+	fd  = open_file_or_dir(path);
+	if (fd < 0) {
+		fprintf(stderr, "ERROR: can't access to '%s'\n", path);
+		return -1;
+	}
+
+	ret = ioctl(fd, BTRFS_IOC_FS_INFO, fi_args);
+	if (ret && errno == EINVAL) {
+		/* path is no mounted btrfs. try if it's a device */
+		ret = check_mounted_where(fd, path, mp, sizeof(mp),
+						&fs_devices_mnt);
+		if (!ret)
+			return -EINVAL;
+		if (ret < 0)
+			return ret;
+		fi_args->num_devices = 1;
+		fi_args->max_id = fs_devices_mnt->latest_devid;
+		i = fs_devices_mnt->latest_devid;
+		memcpy(fi_args->fsid, fs_devices_mnt->fsid, BTRFS_FSID_SIZE);
+		close(fd);
+		fd = open_file_or_dir(mp);
+		if (fd < 0)
+			return -errno;
+	} else if (ret) {
+		close(fd);
+		return -errno;
+	}
+
+	if (!fi_args->num_devices) {
+		close(fd);
+		return 0;
+	}
+
+	di_args = *di_ret = malloc(fi_args->num_devices * sizeof(*di_args));
+	if (!di_args) {
+		close(fd);
+		return -errno;
+	}
+
+	for (; i <= fi_args->max_id; ++i) {
+		BUG_ON(ndevs >= fi_args->num_devices);
+		ret = get_device_info(fd, i, &di_args[ndevs]);
+		if (ret == -ENODEV)
+			continue;
+		if (ret) {
+			close(fd);
+			return ret;
+		}
+		++ndevs;
+	}
+
+	BUG_ON(ndevs == 0);
+
+	close(fd);
+	return 0;
+}
+
 /*
  * Checks to make sure that the label matches our requirements.
  * Returns:
diff --git a/utils.h b/utils.h
index f0100c2..34a814d 100644
--- a/utils.h
+++ b/utils.h
@@ -46,6 +46,11 @@ int btrfs_device_already_in_root(struct btrfs_root *root, int fd,
 				 int super_offset);
 char *pretty_sizes(u64 size);
 int open_file_or_dir(const char *fname);
+int get_device_info(int fd, u64 devid,
+			struct btrfs_ioctl_dev_info_args *di_args);
+int get_fs_info(char *path,
+		struct btrfs_ioctl_fs_info_args *fi_args,
+		struct btrfs_ioctl_dev_info_args **di_ret);
 int check_label(char *input);
 int get_mountpt(char *dev, char *mntpt, size_t size);
 
-- 
1.7.10.4


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 5/8] Add command btrfs filesystem disk-usage
  2012-11-02 10:15 [PATCH][BTRFS-PROGS] Enhance btrfs fi df Goffredo Baroncelli
                   ` (3 preceding siblings ...)
  2012-11-02 10:15 ` [PATCH 4/8] Move scrub_fs_info() and scrub_dev_info() " Goffredo Baroncelli
@ 2012-11-02 10:15 ` Goffredo Baroncelli
  2012-11-02 10:15 ` [PATCH 6/8] Create entry in man page for " Goffredo Baroncelli
                   ` (3 subsequent siblings)
  8 siblings, 0 replies; 24+ messages in thread
From: Goffredo Baroncelli @ 2012-11-02 10:15 UTC (permalink / raw)
  To: linux-btrfs
  Cc: Hugo Mills, Michael Kjörling, Martin Steigerwald, cwillu,
	Chris Murphy, Goffredo Baroncelli

From: Goffredo Baroncelli <kreijack@inwind.it>

Signed-off-by: Goffredo Baroncelli <kreijack@inwind.it>
---
 cmds-fi-disk_usage.c |  662 +++++++++++++++++++++++++++++++++++++++++++++++++-
 cmds-fi-disk_usage.h |    2 +
 cmds-filesystem.c    |    2 +
 utils.c              |   15 ++
 utils.h              |    1 +
 5 files changed, 680 insertions(+), 2 deletions(-)

diff --git a/cmds-fi-disk_usage.c b/cmds-fi-disk_usage.c
index 9131c47..4bec167 100644
--- a/cmds-fi-disk_usage.c
+++ b/cmds-fi-disk_usage.c
@@ -20,6 +20,7 @@
 #include <unistd.h>
 #include <sys/ioctl.h>
 #include <errno.h>
+#include <stdarg.h>
 
 #include "utils.h"
 #include "kerncompat.h"
@@ -31,14 +32,14 @@
 
 #define DF_HUMAN_UNIT		(1<<0)
 
-/* to store the information about the chunk */
+/* to store the information about the chunks */
 struct chunk_info {
 	u64	type;
 	u64	size;
 	u64	devid;
-	int	processed:1;
 };
 
+/* to store information about the disks */
 struct disk_info {
 	u64	devid;
 	char	path[BTRFS_DEVICE_PATH_NAME_MAX];
@@ -79,6 +80,95 @@ static void free_strings_to_free()
 	count_string_to_free = 0;
 }
 
+static int cmd_info_add_info(struct chunk_info **info_ptr,
+			int *info_count,
+			struct btrfs_chunk *chunk)
+{
+
+	u64	type = btrfs_stack_chunk_type(chunk);
+	u64	size = btrfs_stack_chunk_length(chunk);
+	int	num_stripes = btrfs_stack_chunk_num_stripes(chunk);
+	int	sub_stripes = btrfs_stack_chunk_sub_stripes(chunk);
+	int	j;
+
+	for (j = 0 ; j < num_stripes ; j++) {
+		int i;
+		struct chunk_info *p = 0;
+		struct btrfs_stripe *stripe;
+		u64    devid;
+
+		stripe = btrfs_stripe_nr(chunk, j);
+		devid = btrfs_stack_stripe_devid(stripe);
+
+		for (i = 0 ; i < *info_count ; i++)
+			if ((*info_ptr)[i].type == type &&
+			    (*info_ptr)[i].devid == devid) {
+				p = (*info_ptr) + i;
+				break;
+			}
+
+		if (!p) {
+			int size = sizeof(struct btrfs_chunk) * (*info_count+1);
+			struct chunk_info *res = realloc(*info_ptr, size);
+
+			if (!res) {
+				fprintf(stderr, "ERROR: not enough memory\n");
+				return -1;
+			}
+
+			*info_ptr = res;
+			p = res + *info_count;
+			(*info_count)++;
+
+			p->devid = devid;
+			p->type = type;
+			p->size = 0;
+		}
+
+		if (type & (BTRFS_BLOCK_GROUP_RAID1 | BTRFS_BLOCK_GROUP_DUP))
+			p->size += size;
+		else if (type & BTRFS_BLOCK_GROUP_RAID10)
+			p->size += size / (num_stripes / sub_stripes);
+		else
+			p->size += size / num_stripes;
+
+	}
+
+	return 0;
+
+}
+
+static void btrfs_flags2description(u64 flags, char **description)
+{
+	if (flags & BTRFS_BLOCK_GROUP_DATA) {
+		if (flags & BTRFS_BLOCK_GROUP_METADATA)
+			*description = "Data+Metadata";
+		else
+			*description = "Data";
+	} else if (flags & BTRFS_BLOCK_GROUP_SYSTEM) {
+		*description = "System";
+	} else if (flags & BTRFS_BLOCK_GROUP_METADATA) {
+		*description = "Metadata";
+	} else {
+		*description = "Unknown";
+	}
+}
+
+static void btrfs_flags2profile(u64 flags, char **profile)
+{
+	if (flags & BTRFS_BLOCK_GROUP_RAID0) {
+		*profile = "RAID0";
+	} else if (flags & BTRFS_BLOCK_GROUP_RAID1) {
+		*profile = "RAID1";
+	} else if (flags & BTRFS_BLOCK_GROUP_DUP) {
+		*profile = "DUP";
+	} else if (flags & BTRFS_BLOCK_GROUP_RAID10) {
+		*profile = "RAID10";
+	} else {
+		*profile = "Single";
+	}
+}
+
 static char *df_pretty_sizes(u64 size, int mode)
 {
 	char *s;
@@ -332,3 +422,571 @@ int cmd_filesystem_df(int argc, char **argv)
 	return 0;
 }
 
+static int cmp_chunk_info(const void *a, const void *b)
+{
+	return cmp_chunk_block_group(
+		((struct chunk_info *)a)->type,
+		((struct chunk_info *)b)->type);
+}
+
+static int load_chunk_info(int fd,
+			  struct chunk_info **info_ptr,
+			  int *info_count)
+{
+
+	int ret;
+	struct btrfs_ioctl_search_args args;
+	struct btrfs_ioctl_search_key *sk = &args.key;
+	struct btrfs_ioctl_search_header *sh;
+	unsigned long off = 0;
+	int i, e;
+
+
+	memset(&args, 0, sizeof(args));
+
+	/*
+	 * there may be more than one ROOT_ITEM key if there are
+	 * snapshots pending deletion, we have to loop through
+	 * them.
+	 */
+
+
+	sk->tree_id = BTRFS_CHUNK_TREE_OBJECTID;
+
+	sk->min_objectid = 0;
+	sk->max_objectid = (u64)-1;
+	sk->max_type = 0;
+	sk->min_type = (u8)-1;
+	sk->min_offset = 0;
+	sk->max_offset = (u64)-1;
+	sk->min_transid = 0;
+	sk->max_transid = (u64)-1;
+	sk->nr_items = 4096;
+
+	while (1) {
+		ret = ioctl(fd, BTRFS_IOC_TREE_SEARCH, &args);
+		e = errno;
+		if (ret < 0) {
+			fprintf(stderr,
+				"ERROR: can't perform the search - %s\n",
+				strerror(e));
+			return 0;
+		}
+		/* the ioctl returns the number of item it found in nr_items */
+
+		if (sk->nr_items == 0)
+			break;
+
+		off = 0;
+		for (i = 0; i < sk->nr_items; i++) {
+			struct btrfs_chunk *item;
+			sh = (struct btrfs_ioctl_search_header *)(args.buf +
+								  off);
+
+			off += sizeof(*sh);
+			item = (struct btrfs_chunk *)(args.buf + off);
+
+			if (cmd_info_add_info(info_ptr, info_count, item)) {
+				*info_ptr = 0;
+				free(*info_ptr);
+				return 100;
+			}
+
+			off += sh->len;
+
+			sk->min_objectid = sh->objectid;
+			sk->min_type = sh->type;
+			sk->min_offset = sh->offset+1;
+
+		}
+		if (!sk->min_offset)	/* overflow */
+			sk->min_type++;
+		else
+			continue;
+
+		if (!sk->min_type)
+			sk->min_objectid++;
+		 else
+			continue;
+
+		if (!sk->min_objectid)
+			break;
+	}
+
+	qsort(*info_ptr, *info_count, sizeof(struct chunk_info),
+		cmp_chunk_info);
+
+	return 0;
+
+}
+
+static int cmp_disk_info(const void *a, const void *b)
+{
+	return strcmp(((struct disk_info *)a)->path,
+			((struct disk_info *)b)->path);
+}
+
+static int load_disks_info(int fd,
+			   struct disk_info **disks_info_ptr,
+			   int *disks_info_count)
+{
+
+	int ret, i, ndevs;
+	struct btrfs_ioctl_fs_info_args fi_args;
+	struct btrfs_ioctl_dev_info_args dev_info;
+	struct disk_info *info;
+
+	*disks_info_count = 0;
+	*disks_info_ptr = 0;
+
+	ret = ioctl(fd, BTRFS_IOC_FS_INFO, &fi_args);
+	if (ret < 0) {
+		fprintf(stderr, "ERROR: cannot get filesystem info\n");
+		return -1;
+	}
+
+	info = malloc(sizeof(struct disk_info) * fi_args.num_devices);
+	if (!info) {
+		fprintf(stderr, "ERROR: not enough memory\n");
+		return -1;
+	}
+
+	for (i = 0, ndevs = 0 ; i <= fi_args.max_id ; i++) {
+
+		BUG_ON(ndevs >= fi_args.num_devices);
+		ret = get_device_info(fd, i, &dev_info);
+
+		if (ret == -ENODEV)
+			continue;
+		if (ret) {
+			fprintf(stderr,
+			    "ERROR: cannot get info about device devid=%d\n",
+			    i);
+			free(info);
+			return -1;
+		}
+
+		info[ndevs].devid = dev_info.devid;
+		strcpy(info[ndevs].path, (char *)dev_info.path);
+		info[ndevs].size = get_partition_size((char *)dev_info.path);
+		++ndevs;
+	}
+
+	BUG_ON(ndevs != fi_args.num_devices);
+	qsort(info, fi_args.num_devices,
+		sizeof(struct disk_info), cmp_disk_info);
+
+	*disks_info_count = fi_args.num_devices;
+	*disks_info_ptr = info;
+
+	return 0;
+
+}
+
+static void print_unused(struct chunk_info *info_ptr,
+			  int info_count,
+			  struct disk_info *disks_info_ptr,
+			  int disks_info_count,
+			  int mode)
+{
+	int i;
+	for (i = 0 ; i < disks_info_count ; i++) {
+
+		int	j;
+		u64	total = 0;
+		char	*s;
+
+		for (j = 0 ; j < info_count ; j++)
+			if (info_ptr[j].devid == disks_info_ptr[i].devid)
+				total += info_ptr[j].size;
+
+		s = df_pretty_sizes(disks_info_ptr[i].size - total, mode);
+		printf("   %s\t%10s\n", disks_info_ptr[i].path, s);
+
+	}
+
+}
+
+static void print_chunk_disks(u64 chunk_type,
+				struct chunk_info *chunks_info_ptr,
+				int chunks_info_count,
+				struct disk_info *disks_info_ptr,
+				int disks_info_count,
+				int mode)
+{
+	int i;
+	for (i = 0 ; i < disks_info_count ; i++) {
+
+		int	j;
+		u64	total = 0;
+		char	*s;
+
+		for (j = 0 ; j < chunks_info_count ; j++) {
+			if (chunks_info_ptr[j].type != chunk_type)
+				continue;
+			if (chunks_info_ptr[j].devid != disks_info_ptr[i].devid)
+				continue;
+
+			total += chunks_info_ptr[j].size;
+		}
+
+		if (total > 0) {
+			s = df_pretty_sizes(total, mode);
+			printf("   %s\t%10s\n", disks_info_ptr[i].path, s);
+		}
+	}
+}
+
+static char **create_table(int columns, int rows)
+{
+	char **p = calloc(rows * columns, sizeof(char *));
+	if (p)
+		add_strings_to_free((char *)p);
+	return p;
+}
+
+/*
+ * If fmt  starts with '<', the text is left aligned; if fmt starts with
+ * '>' the text is right aligned. If fmt is equal to '=' the text will
+ * be replaced by a '=====' dimensioned in the basis of the column width
+ */
+static char *vprintf_table(char **p, int num_cols, int column, int row,
+			  char *fmt, va_list ap)
+{
+	int idx =  num_cols*row+column;
+	char *msg = calloc(100, sizeof(char));
+
+	if (!msg)
+		return NULL;
+
+	add_strings_to_free(msg);
+	p[idx] = msg;
+	vsnprintf(msg, 99, fmt, ap);
+
+	return msg;
+}
+
+static char *printf_table(char **p, int num_cols, int column, int row,
+			  char *fmt, ...)
+{
+	va_list ap;
+	char *ret;
+
+	va_start(ap, fmt);
+	ret = vprintf_table(p, num_cols, column, row, fmt, ap);
+	va_end(ap);
+
+	return ret;
+}
+
+static void dump_table(char **p, int ncols, int nrows)
+{
+	int	sizes[ncols];
+	int	i, j;
+
+	for (i = 0 ; i < ncols ; i++) {
+		sizes[i] = 0;
+		for (j = 0 ; j < nrows ; j++) {
+			int idx = i + j*ncols;
+			int s;
+
+			if (!p[idx])
+				continue;
+
+			s = strlen(p[idx]) - 1;
+			if (s < 1 || p[idx][0] == '=')
+				continue;
+
+			if (s > sizes[i])
+				sizes[i] = s;
+		}
+	}
+
+
+	for (j = 0 ; j < nrows ; j++) {
+		for (i = 0 ; i < ncols ; i++) {
+
+			int idx = i + j*ncols;
+
+			if (!p[idx] || !strlen(p[idx])) {
+				printf("%*s", sizes[i], "");
+			} else if (p[idx][0] == '=') {
+				int k;
+				for (k = 0 ; k < sizes[i] ; k++)
+					putchar('=');
+			} else {
+				printf("%*s",
+					p[idx][0] == '<' ? -sizes[i] : sizes[i],
+					p[idx]+1);
+			}
+			if (i != (ncols - 1))
+				putchar(' ');
+		}
+		putchar('\n');
+	}
+
+}
+
+
+static void _cmd_filesystem_disk_usage_tabular(int mode,
+					struct btrfs_ioctl_space_args *sargs,
+					struct chunk_info *chunks_info_ptr,
+					int chunks_info_count,
+					struct disk_info *disks_info_ptr,
+					int disks_info_count)
+{
+	int i;
+	u64 total_unused = 0;
+	char **matrix = 0;
+	int  ncols, nrows;
+
+	ncols = sargs->total_spaces + 2;
+	nrows = 2 + 1 + disks_info_count + 1 + 2;
+
+	matrix = create_table(ncols, nrows);
+	if (!matrix) {
+		fprintf(stderr, "ERROR: not enough memory\n");
+		return;
+	}
+
+	/* header */
+	for (i = 0; i < sargs->total_spaces; i++) {
+		char *description;
+
+		u64 flags = sargs->spaces[i].flags;
+		btrfs_flags2description(flags, &description);
+
+		printf_table(matrix, ncols, 1+i, 0, "<%s", description);
+	}
+
+	for (i = 0; i < sargs->total_spaces; i++) {
+		char *r_mode;
+
+		u64 flags = sargs->spaces[i].flags;
+		btrfs_flags2profile(flags, &r_mode);
+
+		printf_table(matrix, ncols, 1+i, 1, "<%s", r_mode);
+	}
+
+	printf_table(matrix, ncols, 1+sargs->total_spaces, 1, "<Unallocated");
+
+	/* body */
+	for (i = 0 ; i < disks_info_count ; i++) {
+		int k, col;
+		char *p;
+
+		u64  total_allocated = 0, unused;
+
+		p = strrchr(disks_info_ptr[i].path, '/');
+		if (!p)
+			p = disks_info_ptr[i].path;
+		else
+			p++;
+
+		printf_table(matrix, ncols, 0, i+3, "<%s",
+				disks_info_ptr[i].path);
+
+		for (col = 1, k = 0 ; k < sargs->total_spaces ; k++)  {
+			u64	flags = sargs->spaces[k].flags;
+			int	j;
+
+			for (j = 0 ; j < chunks_info_count ; j++) {
+				u64 size = chunks_info_ptr[j].size;
+
+				if (chunks_info_ptr[j].type != flags ||
+				    chunks_info_ptr[j].devid !=
+					disks_info_ptr[i].devid)
+						continue;
+
+				printf_table(matrix, ncols, col, i+3,
+					">%s", df_pretty_sizes(size, mode));
+				total_allocated += size;
+				col++;
+				break;
+
+			}
+			if (j == chunks_info_count) {
+				printf_table(matrix, ncols, col, i+3, ">-");
+				col++;
+			}
+		}
+
+		unused = get_partition_size(disks_info_ptr[i].path) -
+				total_allocated;
+
+		printf_table(matrix, ncols, sargs->total_spaces + 1, i + 3,
+			       ">%s", df_pretty_sizes(unused, mode));
+		total_unused += unused;
+
+	}
+
+	for (i = 0; i <= sargs->total_spaces; i++)
+		printf_table(matrix, ncols, i + 1, disks_info_count + 3, "=");
+
+
+	/* footer */
+	printf_table(matrix, ncols, 0, disks_info_count + 4, "<Total");
+	for (i = 0; i < sargs->total_spaces; i++)
+		printf_table(matrix, ncols, 1 + i, disks_info_count + 4,
+			">%s",
+			df_pretty_sizes(sargs->spaces[i].total_bytes, mode));
+
+	printf_table(matrix, ncols, sargs->total_spaces+1, disks_info_count+4,
+		">%s", df_pretty_sizes(total_unused, mode));
+
+	printf_table(matrix, ncols, 0, disks_info_count+5, "<Used");
+	for (i = 0; i < sargs->total_spaces; i++)
+		printf_table(matrix, ncols, 1+i, disks_info_count+5,
+			">%s",
+			df_pretty_sizes(sargs->spaces[i].used_bytes, mode));
+
+
+	dump_table(matrix, ncols, nrows);
+
+}
+
+static void _cmd_filesystem_disk_usage_linear(int mode,
+					struct btrfs_ioctl_space_args *sargs,
+					struct chunk_info *info_ptr,
+					int info_count,
+					struct disk_info *disks_info_ptr,
+					int disks_info_count)
+{
+	int i;
+
+	for (i = 0; i < sargs->total_spaces; i++) {
+		char *description;
+		char *r_mode;
+
+		u64 flags = sargs->spaces[i].flags;
+		btrfs_flags2description(flags, &description);
+		btrfs_flags2profile(flags, &r_mode);
+
+		printf("%s,%s: Size:%s, Used:%s\n",
+			description,
+			r_mode,
+			df_pretty_sizes(sargs->spaces[i].total_bytes ,
+			    mode),
+			df_pretty_sizes(sargs->spaces[i].used_bytes,
+					mode));
+
+		print_chunk_disks(flags, info_ptr, info_count,
+				disks_info_ptr, disks_info_count,
+				mode);
+		printf("\n");
+
+	}
+
+	printf("Unallocated:\n");
+	print_unused(info_ptr, info_count,
+			disks_info_ptr, disks_info_count,
+			mode);
+
+
+
+}
+
+static int _cmd_filesystem_disk_usage(int fd, char *path, int mode, int tabular)
+{
+	struct btrfs_ioctl_space_args *sargs = 0;
+	int info_count = 0;
+	struct chunk_info *info_ptr = 0;
+	struct disk_info *disks_info_ptr = 0;
+	int disks_info_count = 0;
+	int ret = 0;
+
+	if (load_chunk_info(fd, &info_ptr, &info_count) ||
+	    load_disks_info(fd, &disks_info_ptr, &disks_info_count)) {
+		ret = -1;
+		goto exit;
+	}
+
+	if ((sargs = load_space_info(fd, path)) == NULL) {
+		ret = -1;
+		goto exit;
+	}
+
+	if (tabular)
+		_cmd_filesystem_disk_usage_tabular(mode, sargs,
+					info_ptr, info_count,
+					disks_info_ptr, disks_info_count);
+	else
+		_cmd_filesystem_disk_usage_linear(mode, sargs,
+					info_ptr, info_count,
+					disks_info_ptr, disks_info_count);
+
+exit:
+
+	free_strings_to_free();
+	if (sargs)
+		free(sargs);
+	if (disks_info_ptr)
+		free(disks_info_ptr);
+	if (info_ptr)
+		free(info_ptr);
+
+	return ret;
+}
+
+const char * const cmd_filesystem_disk_usage_usage[] = {
+	"btrfs filesystem disk-usage [-b][-t] <path> [<path>..]",
+	"Show in which disk the chunks are allocated.",
+	"",
+	"-b\tSet byte as unit",
+	"-t\tShow data in tabular format",
+	NULL
+};
+
+int cmd_filesystem_disk_usage(int argc, char **argv)
+{
+
+	int	flags =	DF_HUMAN_UNIT;
+	int	i, more_than_one = 0;
+	int	tabular = 0;
+
+	optind = 1;
+	while (1) {
+		char	c = getopt(argc, argv, "bt");
+		if (c < 0)
+			break;
+		switch (c) {
+		case 'b':
+			flags &= ~DF_HUMAN_UNIT;
+			break;
+		case 't':
+			tabular = 1;
+			break;
+		default:
+			usage(cmd_filesystem_disk_usage_usage);
+		}
+	}
+
+	if (check_argc_min(argc - optind, 1)) {
+		usage(cmd_filesystem_disk_usage_usage);
+		return 21;
+	}
+
+	for (i = optind; i < argc ; i++) {
+		int r, fd;
+		if (more_than_one)
+			printf("\n");
+
+		fd = open_file_or_dir(argv[i]);
+		if (fd < 0) {
+			fprintf(stderr, "ERROR: can't access to '%s'\n",
+				argv[1]);
+			return 12;
+		}
+		r = _cmd_filesystem_disk_usage(fd, argv[i], flags, tabular);
+		close(fd);
+
+		if (r)
+			return r;
+		more_than_one = 1;
+
+	}
+
+	return 0;
+}
+
+
diff --git a/cmds-fi-disk_usage.h b/cmds-fi-disk_usage.h
index 9f68bb3..ae11570 100644
--- a/cmds-fi-disk_usage.h
+++ b/cmds-fi-disk_usage.h
@@ -21,5 +21,7 @@
 
 extern const char * const cmd_filesystem_df_usage[];
 int cmd_filesystem_df(int argc, char **argv);
+extern const char * const cmd_filesystem_disk_usage_usage[];
+int cmd_filesystem_disk_usage(int argc, char **argv);
 
 #endif
diff --git a/cmds-filesystem.c b/cmds-filesystem.c
index 1b915e4..7a833b4 100644
--- a/cmds-filesystem.c
+++ b/cmds-filesystem.c
@@ -423,6 +423,8 @@ const struct cmd_group filesystem_cmd_group = {
 		{ "balance", cmd_balance, NULL, &balance_cmd_group, 1 },
 		{ "resize", cmd_resize, cmd_resize_usage, NULL, 0 },
 		{ "label", cmd_label, cmd_label_usage, NULL, 0 },
+		{ "disk-usage", cmd_filesystem_disk_usage,
+			cmd_filesystem_disk_usage_usage, NULL, 0 },
 		{ 0, 0, 0, 0, 0 },
 	}
 };
diff --git a/utils.c b/utils.c
index 023fbca..2b12890 100644
--- a/utils.c
+++ b/utils.c
@@ -1341,3 +1341,18 @@ u64 disk_size(char *path)
 
 }
 
+u64 get_partition_size(char *dev)
+{
+	u64 result;
+	int fd = open(dev, O_RDONLY);
+
+	if (fd < 0)
+		return 0;
+	if (ioctl(fd, BLKGETSIZE64, &result) < 0) {
+		close(fd);
+		return 0;
+	}
+	close(fd);
+
+	return result;
+}
diff --git a/utils.h b/utils.h
index 34a814d..e1caaae 100644
--- a/utils.h
+++ b/utils.h
@@ -56,4 +56,5 @@ int get_mountpt(char *dev, char *mntpt, size_t size);
 
 int btrfs_scan_block_devices(int run_ioctl);
 u64 disk_size(char *path);
+u64 get_partition_size(char *dev);
 #endif
-- 
1.7.10.4


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 6/8] Create entry in man page for btrfs filesystem disk-usage
  2012-11-02 10:15 [PATCH][BTRFS-PROGS] Enhance btrfs fi df Goffredo Baroncelli
                   ` (4 preceding siblings ...)
  2012-11-02 10:15 ` [PATCH 5/8] Add command btrfs filesystem disk-usage Goffredo Baroncelli
@ 2012-11-02 10:15 ` Goffredo Baroncelli
  2012-11-02 10:15 ` [PATCH 7/8] Add btrfs device disk-usage command Goffredo Baroncelli
                   ` (2 subsequent siblings)
  8 siblings, 0 replies; 24+ messages in thread
From: Goffredo Baroncelli @ 2012-11-02 10:15 UTC (permalink / raw)
  To: linux-btrfs
  Cc: Hugo Mills, Michael Kjörling, Martin Steigerwald, cwillu,
	Chris Murphy, Goffredo Baroncelli

From: Goffredo Baroncelli <kreijack@inwind.it>

Signed-off-by: Goffredo Baroncelli <kreijack@inwind.it>
---
 man/btrfs.8.in |   13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/man/btrfs.8.in b/man/btrfs.8.in
index 27d0160..eabf0a8 100644
--- a/man/btrfs.8.in
+++ b/man/btrfs.8.in
@@ -25,6 +25,9 @@ btrfs \- control a btrfs filesystem
 .PP
 \fBbtrfs\fP \fBfilesystem resize\fP\fI [devid:][+/\-]<size>[gkm]|[devid:]max <filesystem>\fP
 .PP
+\fBbtrfs\fP \fBfilesystem filesystem disk-usage [-t][-b]\fP\fI <path> 
+[path..]\fP
+.PP
 \fBbtrfs\fP \fBfilesystem label\fP\fI <dev> [newlabel]\fP
 .PP
 \fBbtrfs\fP \fBfilesystem df\fP\fI [-b] \fIpath [path..]\fR\fP
@@ -227,6 +230,16 @@ it with the new desired size.  When recreating the partition make sure to use
 the same starting disk cylinder as before.
 .TP
 
+\fBfilesystem disk-usage\fP [-t][-b] \fIpath [path..]\fR
+
+Show in which disk the chunks are allocated.
+
+\fB-b\fP Set byte as unit
+
+\fB-t\fP Show data in tabular format
+
+.TP
+
 \fBfilesystem label\fP\fI <dev> [newlabel]\fP
 Show or update the label of a filesystem. \fI<dev>\fR is used to identify the
 filesystem. 
-- 
1.7.10.4


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 7/8] Add btrfs device disk-usage command
  2012-11-02 10:15 [PATCH][BTRFS-PROGS] Enhance btrfs fi df Goffredo Baroncelli
                   ` (5 preceding siblings ...)
  2012-11-02 10:15 ` [PATCH 6/8] Create entry in man page for " Goffredo Baroncelli
@ 2012-11-02 10:15 ` Goffredo Baroncelli
  2012-11-02 10:15 ` [PATCH 8/8] Create a new entry in btrfs man page for btrfs device disk-usage Goffredo Baroncelli
  2012-11-02 11:18 ` [PATCH][BTRFS-PROGS] Enhance btrfs fi df Martin Steigerwald
  8 siblings, 0 replies; 24+ messages in thread
From: Goffredo Baroncelli @ 2012-11-02 10:15 UTC (permalink / raw)
  To: linux-btrfs
  Cc: Hugo Mills, Michael Kjörling, Martin Steigerwald, cwillu,
	Chris Murphy, Goffredo Baroncelli

From: Goffredo Baroncelli <kreijack@inwind.it>

Signed-off-by: Goffredo Baroncelli <kreijack@inwind.it>
---
 cmds-device.c        |    3 ++
 cmds-fi-disk_usage.c |  140 ++++++++++++++++++++++++++++++++++++++++++++++++++
 cmds-fi-disk_usage.h |    4 ++
 3 files changed, 147 insertions(+)

diff --git a/cmds-device.c b/cmds-device.c
index 4787aca..873c39c 100644
--- a/cmds-device.c
+++ b/cmds-device.c
@@ -27,6 +27,7 @@
 #include "ctree.h"
 #include "ioctl.h"
 #include "utils.h"
+#include "cmds-fi-disk_usage.h"
 
 #include "commands.h"
 
@@ -255,6 +256,8 @@ const struct cmd_group device_cmd_group = {
 		{ "add", cmd_add_dev, cmd_add_dev_usage, NULL, 0 },
 		{ "delete", cmd_rm_dev, cmd_rm_dev_usage, NULL, 0 },
 		{ "scan", cmd_scan_dev, cmd_scan_dev_usage, NULL, 0 },
+		{ "disk-usage", cmd_device_disk_usage,
+			cmd_device_disk_usage_usage, NULL, 0 },
 		{ 0, 0, 0, 0, 0 }
 	}
 };
diff --git a/cmds-fi-disk_usage.c b/cmds-fi-disk_usage.c
index 4bec167..0c8af9d 100644
--- a/cmds-fi-disk_usage.c
+++ b/cmds-fi-disk_usage.c
@@ -989,4 +989,144 @@ int cmd_filesystem_disk_usage(int argc, char **argv)
 	return 0;
 }
 
+static void print_disk_chunks(int fd,
+				u64 devid,
+				u64 total_size,
+				struct chunk_info *chunks_info_ptr,
+				int chunks_info_count,
+				int mode)
+{
+	int i;
+	u64 allocated = 0;
+	char *s;
+
+	for (i = 0 ; i < chunks_info_count ; i++) {
+		char *description = "";
+		char *r_mode;
+		u64 flags;
+
+		if (chunks_info_ptr[i].devid != devid)
+			continue;
+
+		flags = chunks_info_ptr[i].type;
+
+		btrfs_flags2description(flags, &description);
+		btrfs_flags2profile(flags, &r_mode);
+
+		s = df_pretty_sizes(chunks_info_ptr[i].size, mode);
+		printf("   %s,%s:%*s%10s\n",
+			description,
+			r_mode,
+			(int)(20 - strlen(description) - strlen(r_mode)), "",
+			s);
+
+		allocated += chunks_info_ptr[i].size;
+
+	}
+	s = df_pretty_sizes(total_size - allocated, mode);
+	printf("   Unallocated: %*s%10s\n",
+		(int)(20 - strlen("Unallocated")), "",
+		s);
+
+}
+
+static int _cmd_device_disk_usage(int fd, char *path, int mode)
+{
+	int i;
+	int ret = 0;
+	int info_count = 0;
+	struct chunk_info *info_ptr = 0;
+	struct disk_info *disks_info_ptr = 0;
+	int disks_info_count = 0;
+
+	if (load_chunk_info(fd, &info_ptr, &info_count) ||
+	    load_disks_info(fd, &disks_info_ptr, &disks_info_count)) {
+		ret = -1;
+		goto exit;
+	}
+
+	for (i = 0 ; i < disks_info_count ; i++) {
+		char *s;
+
+		s = df_pretty_sizes(disks_info_ptr[i].size, mode);
+		printf("%s\t%10s\n", disks_info_ptr[i].path, s);
+
+		print_disk_chunks(fd, disks_info_ptr[i].devid,
+				disks_info_ptr[i].size,
+				info_ptr, info_count,
+				mode);
+		printf("\n");
+
+	}
+
+
+exit:
+
+	free_strings_to_free();
+	if (disks_info_ptr)
+		free(disks_info_ptr);
+	if (info_ptr)
+		free(info_ptr);
+
+	return ret;
+}
+
+const char * const cmd_device_disk_usage_usage[] = {
+	"btrfs device disk-usage [-b] <path> [<path>..]",
+	"Show which chunks are in a device.",
+	"",
+	"-b\tSet byte as unit",
+	NULL
+};
+
+int cmd_device_disk_usage(int argc, char **argv)
+{
+
+	int	flags = DF_HUMAN_UNIT;
+	int	i, more_than_one = 0;
+
+	optind = 1;
+	while (1) {
+		char	c = getopt(argc, argv, "b");
+
+		if (c < 0)
+			break;
+
+		switch (c) {
+		case 'b':
+			flags &= ~DF_HUMAN_UNIT;
+			break;
+		default:
+			usage(cmd_device_disk_usage_usage);
+		}
+	}
+
+	if (check_argc_min(argc - optind, 1)) {
+		usage(cmd_device_disk_usage_usage);
+		return 21;
+	}
+
+	for (i = optind; i < argc ; i++) {
+		int r, fd;
+		if (more_than_one)
+			printf("\n");
+
+		fd = open_file_or_dir(argv[i]);
+		if (fd < 0) {
+			fprintf(stderr, "ERROR: can't access to '%s'\n",
+				argv[1]);
+			return 12;
+		}
+		r = _cmd_device_disk_usage(fd, argv[i], flags);
+		close(fd);
+
+		if (r)
+			return r;
+		more_than_one = 1;
+
+	}
+
+	return 0;
+}
+
 
diff --git a/cmds-fi-disk_usage.h b/cmds-fi-disk_usage.h
index ae11570..c315004 100644
--- a/cmds-fi-disk_usage.h
+++ b/cmds-fi-disk_usage.h
@@ -21,7 +21,11 @@
 
 extern const char * const cmd_filesystem_df_usage[];
 int cmd_filesystem_df(int argc, char **argv);
+
 extern const char * const cmd_filesystem_disk_usage_usage[];
 int cmd_filesystem_disk_usage(int argc, char **argv);
 
+extern const char * const cmd_device_disk_usage_usage[];
+int cmd_device_disk_usage(int argc, char **argv);
+
 #endif
-- 
1.7.10.4


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 8/8] Create a new entry in btrfs man page for btrfs device disk-usage.
  2012-11-02 10:15 [PATCH][BTRFS-PROGS] Enhance btrfs fi df Goffredo Baroncelli
                   ` (6 preceding siblings ...)
  2012-11-02 10:15 ` [PATCH 7/8] Add btrfs device disk-usage command Goffredo Baroncelli
@ 2012-11-02 10:15 ` Goffredo Baroncelli
  2012-11-02 11:18 ` [PATCH][BTRFS-PROGS] Enhance btrfs fi df Martin Steigerwald
  8 siblings, 0 replies; 24+ messages in thread
From: Goffredo Baroncelli @ 2012-11-02 10:15 UTC (permalink / raw)
  To: linux-btrfs
  Cc: Hugo Mills, Michael Kjörling, Martin Steigerwald, cwillu,
	Chris Murphy, Goffredo Baroncelli

From: Goffredo Baroncelli <kreijack@inwind.it>

Signed-off-by: Goffredo Baroncelli <kreijack@inwind.it>
---
 man/btrfs.8.in |    8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/man/btrfs.8.in b/man/btrfs.8.in
index eabf0a8..a411ff8 100644
--- a/man/btrfs.8.in
+++ b/man/btrfs.8.in
@@ -44,6 +44,8 @@ btrfs \- control a btrfs filesystem
 .PP
 \fBbtrfs\fP \fBdevice delete\fP\fI <device> [<device>...] <path> \fP
 .PP
+\fBbtrfs\fP \fBdevice disk-usage\fP\fI [-b] <path> [<path>...] \fP
+.PP
 \fBbtrfs\fP \fBscrub start\fP [-Bdqru] {\fI<path>\fP|\fI<device>\fP}
 .PP
 \fBbtrfs\fP \fBscrub cancel\fP {\fI<path>\fP|\fI<device>\fP}
@@ -324,6 +326,12 @@ Add device(s) to the filesystem identified by \fI<path>\fR.
 Remove device(s) from a filesystem identified by \fI<path>\fR.
 .TP
 
+\fBdevice disk-usage\fR\fI [-b] <path> [<path>..] <path>\fR
+Show which chunks are in a device.
+
+\fB-b\fP set byte as unit.
+.TP
+
 \fBdevice scan\fR \fI[--all-devices|<device> [<device>...]\fR
 If one or more devices are passed, these are scanned for a btrfs filesystem. 
 If no devices are passed, \fBbtrfs\fR scans all the block devices listed
-- 
1.7.10.4


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-02 10:15 [PATCH][BTRFS-PROGS] Enhance btrfs fi df Goffredo Baroncelli
                   ` (7 preceding siblings ...)
  2012-11-02 10:15 ` [PATCH 8/8] Create a new entry in btrfs man page for btrfs device disk-usage Goffredo Baroncelli
@ 2012-11-02 11:18 ` Martin Steigerwald
  2012-11-02 12:02   ` Goffredo Baroncelli
  8 siblings, 1 reply; 24+ messages in thread
From: Martin Steigerwald @ 2012-11-02 11:18 UTC (permalink / raw)
  To: Goffredo Baroncelli
  Cc: linux-btrfs, Hugo Mills, Michael Kjörling, cwillu, Chris Murphy

Am Freitag, 2. November 2012 schrieb Goffredo Baroncelli:
> Hi all, on the basis of the discussion in the thread
> '[RFC] New attempt to a better "btrfs fi df"', I prepared the following
> set of patches.
> These patches update the btrfs fi df command and add two new commands:
> - btrfs filesystem disk-usage <path>
> - btrfs device disk-usage <path>
> 
> The command "btrfs filesystem df" now shows only the disk
> usage/available.
> 
> $ btrfs filesystem df /mnt/btrfs1/
> Disk size:		 109.00GB
> Disk allocated:		   5.90GB
> Disk unallocated:	 103.10GB
> Used:			 284.00KB
> Free (Estimated):	  63.00GB	(Max: 106.51GB, min: 54.96GB)
> Data to disk ratio:	     58 %

This is coming along nicely.

Tested-By: Martin Steigerwald <martin@lichtvoll.de>

I can test on some other boxes next week, if you want to.

I just wonder about one thing:


merkaba:[…]/btrfs-progs-unstable> ./btrfs fi df /
Disk size:                18.62GB
Disk allocated:           18.62GB
Disk unallocated:            0.00
Used:                     11.26GB
Free (Estimated):          5.61GB       (Max: 5.61GB, min: 5.61GB)
Data to disk ratio:          91 %


merkaba:[…]/btrfs-progs-unstable> ./btrfs filesystem disk-usage /
Data,Single: Size:15.10GB, Used:10.65GB
   /dev/dm-0       15.10GB

Metadata,Single: Size:8.00MB, Used:0.00
   /dev/dm-0        8.00MB

Metadata,DUP: Size:1.75GB, Used:627.84MB
   /dev/dm-0        3.50GB

System,Single: Size:4.00MB, Used:0.00
   /dev/dm-0        4.00MB

System,DUP: Size:8.00MB, Used:4.00KB
   /dev/dm-0       16.00MB

Unallocated:
   /dev/dm-0          0.00


merkaba:[…]/btrfs-progs-unstable> ./btrfs filesystem disk-usage -t /
          Data    Metadata Metadata System System             
          Single  Single   DUP      Single DUP     Unallocated
                                                              
/dev/dm-0 15.10GB   8.00MB   3.50GB 4.00MB 16.00MB        0.00
          ======= ======== ======== ====== ======= ===========
Total     15.10GB   8.00MB   1.75GB 4.00MB  8.00MB        0.00
Used      10.65GB     0.00 627.84MB   0.00  4.00KB            


Metadata, DUP is displayed as 3,50GB on the device level and as 1,75GB
in total. I understand the logic behind this, but this could be a bit
confusing.

But it makes sense: Showing real allocation on device level makes sense,
cause thats what really allocated on disk. Total makes some sense, cause
thats what is being used from the tree by BTRFS.

It still looks confusing at first…

Maybe two sizes: One total with dup / raid1 / raid10 being accounted for
and one without?

Well maybe just leave as is for now. This output is for experienced
users.


merkaba:[…]/btrfs-progs-unstable> ./btrfs device disk-usage /   
/dev/dm-0          18.62GB
   Data,Single:             15.10GB
   Metadata,Single:          8.00MB
   Metadata,DUP:             3.50GB
   System,Single:            4.00MB
   System,DUP:              16.00MB
   Unallocated:                0.00


This is a nice view on the disk. I know its fully allocated by BTRFS,
and in order to make more free space for the data tree for example I´d
need to look at the tree usage and then if it makes sense do a balance
operation.

Well in that case, I plan to migrate metadata and system to single. And
remove the dup trees then.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-02 11:18 ` [PATCH][BTRFS-PROGS] Enhance btrfs fi df Martin Steigerwald
@ 2012-11-02 12:02   ` Goffredo Baroncelli
  2012-11-02 19:05     ` Gabriel
  0 siblings, 1 reply; 24+ messages in thread
From: Goffredo Baroncelli @ 2012-11-02 12:02 UTC (permalink / raw)
  To: Martin Steigerwald
  Cc: linux-btrfs, Hugo Mills, Michael Kjörling, cwillu, Chris Murphy

On 2012-11-02 12:18, Martin Steigerwald wrote:
> I can test on some other boxes next week, if you want to.

Yes please,

> 
> I just wonder about one thing:
> 
> 
> merkaba:[…]/btrfs-progs-unstable> ./btrfs fi df /
> Disk size:                18.62GB
> Disk allocated:           18.62GB
> Disk unallocated:            0.00
> Used:                     11.26GB
> Free (Estimated):          5.61GB       (Max: 5.61GB, min: 5.61GB)
> Data to disk ratio:          91 %
> 
> 
> merkaba:[…]/btrfs-progs-unstable> ./btrfs filesystem disk-usage /
> Data,Single: Size:15.10GB, Used:10.65GB
>    /dev/dm-0       15.10GB
> 
> Metadata,Single: Size:8.00MB, Used:0.00
>    /dev/dm-0        8.00MB
> 
> Metadata,DUP: Size:1.75GB, Used:627.84MB
>    /dev/dm-0        3.50GB
> 
> System,Single: Size:4.00MB, Used:0.00
>    /dev/dm-0        4.00MB
> 
> System,DUP: Size:8.00MB, Used:4.00KB
>    /dev/dm-0       16.00MB
> 
> Unallocated:
>    /dev/dm-0          0.00
> 
> 
> merkaba:[…]/btrfs-progs-unstable> ./btrfs filesystem disk-usage -t /
>           Data    Metadata Metadata System System             
>           Single  Single   DUP      Single DUP     Unallocated
>                                                               
> /dev/dm-0 15.10GB   8.00MB   3.50GB 4.00MB 16.00MB        0.00
>           ======= ======== ======== ====== ======= ===========
> Total     15.10GB   8.00MB   1.75GB 4.00MB  8.00MB        0.00
> Used      10.65GB     0.00 627.84MB   0.00  4.00KB            
> 
> 
> Metadata, DUP is displayed as 3,50GB on the device level and as 1,75GB
> in total. I understand the logic behind this, but this could be a bit
> confusing.
> 
> But it makes sense: Showing real allocation on device level makes sense,
> cause thats what really allocated on disk. Total makes some sense, cause
> thats what is being used from the tree by BTRFS.

Yes, me too. At the first I was confused when you noticed this
discrepancy. So I have to admit that it is not so obvious to understand.
However we didn't find any way to make it more clear...

> It still looks confusing at first…
We could use "Chunk(s) capacity" instead of total/size ? I would like an
opinion from a "english people" point of view..

GB


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-02 12:02   ` Goffredo Baroncelli
@ 2012-11-02 19:05     ` Gabriel
  2012-11-02 19:31       ` Goffredo Baroncelli
                         ` (2 more replies)
  0 siblings, 3 replies; 24+ messages in thread
From: Gabriel @ 2012-11-02 19:05 UTC (permalink / raw)
  To: linux-btrfs

On Fri, 02 Nov 2012 13:02:32 +0100, Goffredo Baroncelli wrote:
> On 2012-11-02 12:18, Martin Steigerwald wrote:
>> Metadata, DUP is displayed as 3,50GB on the device level and as 1,75GB
>> in total. I understand the logic behind this, but this could be a bit
>> confusing.
>> 
>> But it makes sense: Showing real allocation on device level makes
>> sense,
>> cause thats what really allocated on disk. Total makes some sense,
>> cause thats what is being used from the tree by BTRFS.
> 
> Yes, me too. At the first I was confused when you noticed this
> discrepancy. So I have to admit that it is not so obvious to understand.
> However we didn't find any way to make it more clear...
> 
>> It still looks confusing at first…
> We could use "Chunk(s) capacity" instead of total/size ? I would like an
> opinion from a "english people" point of view..

This is easy to fix, here's a mockup:

Metadata,DUP: Size: 1.75GB ×2, Used: 627.84MB ×2
   /dev/dm-0        3.50GB

           Data   Metadata Metadata    System System              
           Single Single   DUP         Single DUP         Unallocated
                                                               
/dev/dm-16 1.31TB   8.00MB  56.00GB    4.00MB  16.00MB           0.00
           ====== ======== =========== ====== =========== ===========
Total      1.31TB   8.00MB  28.00GB ×2 4.00MB   8.00MB ×2        0.00
Used       1.31TB     0.00   5.65GB ×2   0.00 152.00KB ×2            

Also, I don't know if you could use libblkid, but it finds more 
descriptive names than dm-NN (thanks to some smart sorting logic).



^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-02 19:05     ` Gabriel
@ 2012-11-02 19:31       ` Goffredo Baroncelli
  2012-11-02 20:40         ` Gabriel
  2012-11-02 22:06       ` Hugo Mills
  2012-11-03 12:11       ` Goffredo Baroncelli
  2 siblings, 1 reply; 24+ messages in thread
From: Goffredo Baroncelli @ 2012-11-02 19:31 UTC (permalink / raw)
  To: Gabriel; +Cc: linux-btrfs

On 11/02/2012 08:05 PM, Gabriel wrote:
> On Fri, 02 Nov 2012 13:02:32 +0100, Goffredo Baroncelli wrote:
>> On 2012-11-02 12:18, Martin Steigerwald wrote:
>>> Metadata, DUP is displayed as 3,50GB on the device level and as 1,75GB
>>> in total. I understand the logic behind this, but this could be a bit
>>> confusing.
>>>
>>> But it makes sense: Showing real allocation on device level makes
>>> sense,
>>> cause thats what really allocated on disk. Total makes some sense,
>>> cause thats what is being used from the tree by BTRFS.
>>
>> Yes, me too. At the first I was confused when you noticed this
>> discrepancy. So I have to admit that it is not so obvious to understand.
>> However we didn't find any way to make it more clear...
>>
>>> It still looks confusing at first…
>> We could use "Chunk(s) capacity" instead of total/size ? I would like an
>> opinion from a "english people" point of view..
> 
> This is easy to fix, here's a mockup:
> 
> Metadata,DUP: Size: 1.75GB ×2, Used: 627.84MB ×2
>    /dev/dm-0        3.50GB
> 
>            Data   Metadata Metadata    System System              
>            Single Single   DUP         Single DUP         Unallocated
>                                                                
> /dev/dm-16 1.31TB   8.00MB  56.00GB    4.00MB  16.00MB           0.00
>            ====== ======== =========== ====== =========== ===========
> Total      1.31TB   8.00MB  28.00GB ×2 4.00MB   8.00MB ×2        0.00
> Used       1.31TB     0.00   5.65GB ×2   0.00 152.00KB ×2     

Nice idea. Even tough I like the opposite:


           Data   Metadata Metadata    System System
           Single Single   DUP         Single DUP         Unallocated

/dev/dm-16 1.31TB   8.00MB  28.00GB x2 4.00MB   8.00MB x2        0.00
           ====== ======== =========== ====== =========== ===========
Total      1.31TB   8.00MB  28.00GB    4.00MB   8.00MB           0.00
Used       1.31TB     0.00   5.65GB      0.00 152.00KB


However how your solution will became when RAID5/RAID6 will arrive ? mmm
may be the solution is simpler: the "x2" factor is applied only to DUP
profile. The other profiles span different disks.

As another option, we can add a field/line which reports the RAID factor:


Metadata,DUP: Size: 1.75GB, Used: 627.84MB, Raid factor: 2x
   /dev/dm-0        3.50GB


            Data   Metadata Metadata   System System
            Single Single   DUP        Single DUP    Unallocated

/dev/dm-16  1.31TB   8.00MB  56.00GB 4.00MB  16.00MB        0.00
            ====== ======== ======== ====== ======== ===========
Raid factor      -        -       x2      -       x2           -
Total       1.31TB   8.00MB  28.00GB 4.00MB   8.00MB        0.00
Used        1.31TB     0.00   5.65GB   0.00 152.00KB





> 
> Also, I don't know if you could use libblkid, but it finds more 
> descriptive names than dm-NN (thanks to some smart sorting logic).

I don't think that it would be impossible to use libblkid, however it
would be difficult to find spaces for longer device name

> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-02 19:31       ` Goffredo Baroncelli
@ 2012-11-02 20:40         ` Gabriel
  2012-11-02 21:46           ` Michael Kjörling
  0 siblings, 1 reply; 24+ messages in thread
From: Gabriel @ 2012-11-02 20:40 UTC (permalink / raw)
  To: linux-btrfs

On Fri, 02 Nov 2012 20:31:56 +0100, Goffredo Baroncelli wrote:

> On 11/02/2012 08:05 PM, Gabriel wrote:
>> On Fri, 02 Nov 2012 13:02:32 +0100, Goffredo Baroncelli wrote:
>>> On 2012-11-02 12:18, Martin Steigerwald wrote:
>>>> Metadata, DUP is displayed as 3,50GB on the device level and as
>>>> 1,75GB in total. I understand the logic behind this, but this could
>>>> be a bit confusing.
>>>>
>>>> But it makes sense: Showing real allocation on device level makes
>>>> sense,
>>>> cause thats what really allocated on disk. Total makes some sense,
>>>> cause thats what is being used from the tree by BTRFS.
>>>
>>> Yes, me too. At the first I was confused when you noticed this
>>> discrepancy. So I have to admit that it is not so obvious to
>>> understand.
>>> However we didn't find any way to make it more clear...
>>>
>>>> It still looks confusing at first…
>>> We could use "Chunk(s) capacity" instead of total/size ? I would like
>>> an opinion from a "english people" point of view..
>> 
>> This is easy to fix, here's a mockup:
>> 
>> Metadata,DUP: Size: 1.75GB ×2, Used: 627.84MB ×2
>>    /dev/dm-0        3.50GB
>> 
>>            Data   Metadata Metadata    System System Single Single  
>>            DUP         Single DUP         Unallocated
>>                                                                
>> /dev/dm-16 1.31TB   8.00MB  56.00GB    4.00MB  16.00MB           0.00
>>            ====== ======== =========== ====== =========== ===========
>> Total      1.31TB   8.00MB  28.00GB ×2 4.00MB   8.00MB ×2        0.00
>> Used       1.31TB     0.00   5.65GB ×2   0.00 152.00KB ×2
> 
> Nice idea. Even tough I like the opposite:
> 
> 
>            Data   Metadata Metadata    System System Single Single   DUP
>                    Single DUP         Unallocated
> 
> /dev/dm-16 1.31TB   8.00MB  28.00GB x2 4.00MB   8.00MB x2        0.00
>            ====== ======== =========== ====== =========== ===========
> Total      1.31TB   8.00MB  28.00GB    4.00MB   8.00MB           0.00
> Used       1.31TB     0.00   5.65GB      0.00 152.00KB
> 
> 
> However how your solution will became when RAID5/RAID6 will arrive ? mmm
> may be the solution is simpler: the "x2" factor is applied only to DUP
> profile. The other profiles span different disks.

That problem solved itself :)

> As another option, we can add a field/line which reports the RAID
> factor:
> 
> Metadata,DUP: Size: 1.75GB, Used: 627.84MB, Raid factor: 2x
>    /dev/dm-0        3.50GB
> 
> 
>             Data   Metadata Metadata   System System Single Single   DUP
>                    Single DUP    Unallocated
> 
> /dev/dm-16  1.31TB   8.00MB  56.00GB 4.00MB  16.00MB        0.00
>             ====== ======== ======== ====== ======== ===========
> Raid factor      -        -       x2      -       x2           -
> Total       1.31TB   8.00MB  28.00GB 4.00MB   8.00MB        0.00 Used   
>     1.31TB     0.00   5.65GB   0.00 152.00KB

All fine options. Though if you remove the ×2 on the totals line,
you should compute it instead (it looks like a tally, both sides
of the == line should be equal).

Now that I've started bikeshedding, here is something that I would
find pretty much ideal:

            Data    Metadata           System             Unallocated              

VolGroup/Btrfs
  Reserved   1.31TB 8.00MB + 2×28.00MB 16.00MB + 2×4.00MB           -
  Used       1.31TB          2× 5.65GB         2×152.00KB           -
            ======= ================== ================== ===========
Total
  Reserved   1.31TB            56.00GB            24.00MB           -
  Used       1.31TB            11.30GB           304.00KB           -
  Free      12.34GB            44.70GB            23.70MB           -



>> Also, I don't know if you could use libblkid, but it finds more
>> descriptive names than dm-NN (thanks to some smart sorting logic).
> 
> I don't think that it would be impossible to use libblkid, however
> it would be difficult to find spaces for longer device name

I suggest cutting out the /dev and putting a line break after the
name. The extra info makes it more human-friendly, and the line
break may complicate machine parsing but the non-tabular format is
better at that anyway.


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-02 20:40         ` Gabriel
@ 2012-11-02 21:46           ` Michael Kjörling
  2012-11-02 23:34             ` Gabriel
  0 siblings, 1 reply; 24+ messages in thread
From: Michael Kjörling @ 2012-11-02 21:46 UTC (permalink / raw)
  To: Gabriel; +Cc: linux-btrfs

On 2 Nov 2012 20:40 +0000, from g2p.code@gmail.com (Gabriel):
> Now that I've started bikeshedding, here is something that I would
> find pretty much ideal:
> 
>             Data    Metadata           System             Unallocated              
> 
> VolGroup/Btrfs
>   Reserved   1.31TB 8.00MB + 2×28.00MB 16.00MB + 2×4.00MB           -
>   Used       1.31TB          2× 5.65GB         2×152.00KB           -
>             ======= ================== ================== ===========
> Total
>   Reserved   1.31TB            56.00GB            24.00MB           -
>   Used       1.31TB            11.30GB           304.00KB           -
>   Free      12.34GB            44.70GB            23.70MB           -

If we can take such liberties, then why bother with the 2× at all?

Also, I think the "B" can go, since it's implied by talking about
storage capacities. A lot of tools do this already; look at GNU "df -h"
and "ls -lh" for just two examples. That gives you a few extra columns
which can be used to make the table column spacing a little bigger even
in an 80-column terminal.

I'm _guessing_ that you meant for metadata reserved to be 2 × 28 GB and
not 2 × 28 MB, because otherwise the numbers really don't add up.

              Data    Metadata        System            Unallocated

VolGroup/Btrfs
  Reserved      1.31T  8.00M + 28.00G  16.00M +   4.00M            -
   ResRedun         -          28.00G             4.00M            -
  Used          1.31T           5.65G           152.00K            -
   UseRedun         -           5.65G           152.00K            -
              =======  ==============  ================  ===========
Total
  Reserved      1.31T          56.01G            24.00M            -
  Used          1.31T          11.30G           304.00K            -
  Free         12.34G          44.71G            23.70M            -

This way, the numbers should add up nicely. ("Redun" for "redundancy" or
something like that.) 8M + 28G + 28G = 56.01G, 5.65G + 5.65G = 11.30G,
56.01G - 11.30G = 44.71G. I'm not sure you couldn't even work 8.00M +
28.00G into a single 28.01G entry at Reserved/Metadata, with
ResRedun/Metadata 28.00G. That would require some care when the units
are different enough that the difference doesn't show up in the numbers,
though, since then there is nothing to indicate that parts of the
metadata is not stored in a redundant fashion.

If some redundancy scheme (RAID 5?) uses an oddball factor, that can
still easily be expressed in a view like the above simply by displaying
the user data and redundancy data separately, in exactly the same way.

And personally, I feel that a summary view like this, for Data, if an
exact number cannot be calculated, should display the _minimum amount of
available free space_, with "free space" being _usable by user files_.
If I start copying a 12.0GB file onto the file system exemplified above,
I most assuredly _don't_ want to get a report of "device full" after 10
GB! ("You mating female dog, you told me I had 12.3 GB free, wrote 10 GB
and now you're saying there's NO free space?! To hell with this, I'm
switching to Windows!") That also saves this tool from having to take
into account possible compression ratios for when file system level
compression is enabled, savings from possible deduplication of data, etc
etc. Of course it also means that the amount of free space may shrink by
less than the size of the added data, but hey, that's a nice bonus if
your disk grows bigger as you add more data to it. :-)


> I suggest cutting out the /dev and putting a line break after the
> name. The extra info makes it more human-friendly, and the line
> break may complicate machine parsing but the non-tabular format is
> better at that anyway.

That might work well for anything under /dev, but what about things that
aren't? And I stand by my earlier position that the tabular data
shouldn't be machine-parsed anyway. As you say, the non-tabular format
is better for that.

-- 
Michael Kjörling • http://michael.kjorling.se • michael@kjorling.se
                “People who think they know everything really annoy
                those of us who know we don’t.” (Bjarne Stroustrup)

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-02 19:05     ` Gabriel
  2012-11-02 19:31       ` Goffredo Baroncelli
@ 2012-11-02 22:06       ` Hugo Mills
  2012-11-02 23:23         ` Gabriel
  2012-11-03 12:35         ` Goffredo Baroncelli
  2012-11-03 12:11       ` Goffredo Baroncelli
  2 siblings, 2 replies; 24+ messages in thread
From: Hugo Mills @ 2012-11-02 22:06 UTC (permalink / raw)
  To: Gabriel; +Cc: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 2818 bytes --]

On Fri, Nov 02, 2012 at 07:05:37PM +0000, Gabriel wrote:
> On Fri, 02 Nov 2012 13:02:32 +0100, Goffredo Baroncelli wrote:
> > On 2012-11-02 12:18, Martin Steigerwald wrote:
> >> Metadata, DUP is displayed as 3,50GB on the device level and as 1,75GB
> >> in total. I understand the logic behind this, but this could be a bit
> >> confusing.
> >> 
> >> But it makes sense: Showing real allocation on device level makes
> >> sense,
> >> cause thats what really allocated on disk. Total makes some sense,
> >> cause thats what is being used from the tree by BTRFS.
> > 
> > Yes, me too. At the first I was confused when you noticed this
> > discrepancy. So I have to admit that it is not so obvious to understand.
> > However we didn't find any way to make it more clear...
> > 
> >> It still looks confusing at first…
> > We could use "Chunk(s) capacity" instead of total/size ? I would like an
> > opinion from a "english people" point of view..
> 
> This is easy to fix, here's a mockup:
> 
> Metadata,DUP: Size: 1.75GB ×2, Used: 627.84MB ×2
>    /dev/dm-0        3.50GB

   I've not considered the full semantics of all this yet -- I'll try
to do that tomorrow. However, I note that the "×2" here could become
non-integer with the RAID-5/6 code (which is due Real Soon Now). In
the first RAID-5/6 code drop, it won't even be simple to calculate
where there are different-sized devices in the filesystem. Putting an
exact figure on that number is potentially going to be awkward. I
think we're going to need kernel help for working out what that number
should be, in the general case.

   Again, I'm raising minor points based on future capabilities, but I
feel it's worth considering them at this stage, even if the correct
answer is "yes, we'll do this now, and deal with any other problems
later".

   Hugo.

>            Data   Metadata Metadata    System System              
>            Single Single   DUP         Single DUP         Unallocated
>                                                                
> /dev/dm-16 1.31TB   8.00MB  56.00GB    4.00MB  16.00MB           0.00
>            ====== ======== =========== ====== =========== ===========
> Total      1.31TB   8.00MB  28.00GB ×2 4.00MB   8.00MB ×2        0.00
> Used       1.31TB     0.00   5.65GB ×2   0.00 152.00KB ×2            
> 
> Also, I don't know if you could use libblkid, but it finds more 
> descriptive names than dm-NN (thanks to some smart sorting logic).
> 
> 

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
  --- My doctor tells me that I have a malformed public-duty gland, ---  
                and a natural deficiency in moral fibre.                 

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-02 22:06       ` Hugo Mills
@ 2012-11-02 23:23         ` Gabriel
  2012-11-02 23:44           ` Hugo Mills
  2012-11-03 12:35         ` Goffredo Baroncelli
  1 sibling, 1 reply; 24+ messages in thread
From: Gabriel @ 2012-11-02 23:23 UTC (permalink / raw)
  To: linux-btrfs

On Fri, 02 Nov 2012 22:06:04 +0000, Hugo Mills wrote:

> On Fri, Nov 02, 2012 at 07:05:37PM +0000, Gabriel wrote:
>> On Fri, 02 Nov 2012 13:02:32 +0100, Goffredo Baroncelli wrote:
>> > On 2012-11-02 12:18, Martin Steigerwald wrote:
>> >> Metadata, DUP is displayed as 3,50GB on the device level and as 1,75GB
>> >> in total. I understand the logic behind this, but this could be a bit
>> >> confusing.
>> >> 
>> >> But it makes sense: Showing real allocation on device level makes
>> >> sense,
>> >> cause thats what really allocated on disk. Total makes some sense,
>> >> cause thats what is being used from the tree by BTRFS.
>> > 
>> > Yes, me too. At the first I was confused when you noticed this
>> > discrepancy. So I have to admit that it is not so obvious to understand.
>> > However we didn't find any way to make it more clear...
>> > 
>> >> It still looks confusing at first…
>> > We could use "Chunk(s) capacity" instead of total/size ? I would like an
>> > opinion from a "english people" point of view..
>> 
>> This is easy to fix, here's a mockup:
>> 
>> Metadata,DUP: Size: 1.75GB ×2, Used: 627.84MB ×2
>>    /dev/dm-0        3.50GB
> 
>    I've not considered the full semantics of all this yet -- I'll try
> to do that tomorrow. However, I note that the "×2" here could become
> non-integer with the RAID-5/6 code (which is due Real Soon Now). In
> the first RAID-5/6 code drop, it won't even be simple to calculate
> where there are different-sized devices in the filesystem. Putting an
> exact figure on that number is potentially going to be awkward. I
> think we're going to need kernel help for working out what that number
> should be, in the general case.

DUP can be nested below a device because it represents same-device
redundancy (purpose: survive smudges but not device failure).

On the other hand raid levels should occupy the same space on all
linked devices (a necessary consequence of the guarantee that RAID5
can survive the loss of any device and RAID6 any two devices).

The two probably won't need to be represented at the same time
except during a reshape, because I imagine DUP gets converted to
RAID (1 or 5) as soon as the second device is added.

A 1→2 reshape would look a bit like this (doing only the data column
and skipping totals):

InitialDevice
  Reserved           1.21TB
  Used               1.21TB
RAID1(InitialDevice, SecondDevice)
  Reserved   1.31TB + 100GB
  Used             2× 100GB

RAID5, RAID6: same with fractions, n+1⁄n and n+2⁄n.

>    Again, I'm raising minor points based on future capabilities, but I
> feel it's worth considering them at this stage, even if the correct
> answer is "yes, we'll do this now, and deal with any other problems
> later".
> 
>    Hugo.
> 
>>            Data   Metadata Metadata    System System              
>>            Single Single   DUP         Single DUP         Unallocated
>>                                                                
>> /dev/dm-16 1.31TB   8.00MB  56.00GB    4.00MB  16.00MB           0.00
>>            ====== ======== =========== ====== =========== ===========
>> Total      1.31TB   8.00MB  28.00GB ×2 4.00MB   8.00MB ×2        0.00
>> Used       1.31TB     0.00   5.65GB ×2   0.00 152.00KB ×2            
>> 
>> Also, I don't know if you could use libblkid, but it finds more 
>> descriptive names than dm-NN (thanks to some smart sorting logic).
>> 
>>



^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-02 21:46           ` Michael Kjörling
@ 2012-11-02 23:34             ` Gabriel
  0 siblings, 0 replies; 24+ messages in thread
From: Gabriel @ 2012-11-02 23:34 UTC (permalink / raw)
  To: linux-btrfs

On Fri, 02 Nov 2012 21:46:35 +0000, Michael Kjörling wrote:
> On 2 Nov 2012 20:40 +0000, from g2p.code@gmail.com (Gabriel):
>> Now that I've started bikeshedding, here is something that I would
>> find pretty much ideal:
>> 
>>             Data    Metadata           System             Unallocated              
>> 
>> VolGroup/Btrfs
>>   Reserved   1.31TB 8.00MB + 2×28.00GB 16.00MB + 2×4.00MB           -
>>   Used       1.31TB          2× 5.65GB         2×152.00KB           
>>             ======= ================== ================== ===========
>> Total
>>   Reserved   1.31TB            56.00GB            24.00MB           -
>>   Used       1.31TB            11.30GB           304.00KB           
>>   Free      12.34GB            44.70GB            23.70MB           -
> 
> If we can take such liberties, then why bother with the 2× at all?

It does save a line.

> Also, I think the "B" can go, since it's implied by talking about
> storage capacities. A lot of tools do this already; look at GNU "df -h"
> and "ls -lh" for just two examples. That gives you a few extra columns
> which can be used to make the table column spacing a little bigger even
> in an 80-column terminal.

Good idea.

> I'm _guessing_ that you meant for metadata reserved to be 2 × 28 GB and
> not 2 × 28 MB, because otherwise the numbers really don't add up.

Feh, that's just a typo from when I swapped the 8.00M to the left.

>               Data    Metadata        System            Unallocated
> 
> VolGroup/Btrfs
>   Reserved      1.31T  8.00M + 28.00G  16.00M +   4.00M            -
>    ResRedun         -          28.00G             4.00M            -
>   Used          1.31T           5.65G           152.00K            -
>    UseRedun         -           5.65G           152.00K            -
>               =======  ==============  ================  ===========
> Total
>   Reserved      1.31T          56.01G            24.00M            -
>   Used          1.31T          11.30G           304.00K            -
>   Free         12.34G          44.71G            23.70M            -
> 
> This way, the numbers should add up nicely. ("Redun" for "redundancy" or
> something like that.) 8M + 28G + 28G = 56.01G, 5.65G + 5.65G = 11.30G,
> 56.01G - 11.30G = 44.71G. I'm not sure you couldn't even work 8.00M +
> 28.00G into a single 28.01G entry at Reserved/Metadata, with
> ResRedun/Metadata 28.00G. That would require some care when the units
> are different enough that the difference doesn't show up in the numbers,
> though, since then there is nothing to indicate that parts of the
> metadata is not stored in a redundant fashion.

I tried to work out DUP vs RAID redundancy in my message to Hugo.

> If some redundancy scheme (RAID 5?) uses an oddball factor, that can
> still easily be expressed in a view like the above simply by displaying
> the user data and redundancy data separately, in exactly the same way.
> 
> And personally, I feel that a summary view like this, for Data, if an
> exact number cannot be calculated, should display the _minimum amount of
> available free space_, with "free space" being _usable by user files_.
> If I start copying a 12.0GB file onto the file system exemplified above,
> I most assuredly _don't_ want to get a report of "device full" after 10
> GB! ("You mating female dog, you told me I had 12.3 GB free, wrote 10 GB
> and now you're saying there's NO free space?! To hell with this, I'm
> switching to Windows!") That also saves this tool from having to take
> into account possible compression ratios for when file system level
> compression is enabled, savings from possible deduplication of data, etc
> etc. Of course it also means that the amount of free space may shrink by
> less than the size of the added data, but hey, that's a nice bonus if
> your disk grows bigger as you add more data to it. :-)

I think we can guarantee minimum amounts of free space, as long as
data/metadata/system are segregated properly?
OK, reshapes complicate this. For those we could to take the worst
case between now and the completed reshape.
Or maybe add a second tally:

devices
===
total
 reserved
 used
 free
===
anticipated (reshaped 8% eta 3:12)
 reserved
 used
 free

>> I suggest cutting out the /dev and putting a line break after the
>> name. The extra info makes it more human-friendly, and the line
>> break may complicate machine parsing but the non-tabular format is
>> better at that anyway.
> 
> That might work well for anything under /dev, but what about things that
> aren't?

Absolute path for those, assuming it ever happens.

> And I stand by my earlier position that the tabular data
> shouldn't be machine-parsed anyway. As you say, the non-tabular format
> is better for that.


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-02 23:23         ` Gabriel
@ 2012-11-02 23:44           ` Hugo Mills
  2012-11-03  0:14             ` Gabriel
  2012-11-03 12:28             ` Goffredo Baroncelli
  0 siblings, 2 replies; 24+ messages in thread
From: Hugo Mills @ 2012-11-02 23:44 UTC (permalink / raw)
  To: Gabriel; +Cc: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 4684 bytes --]

On Fri, Nov 02, 2012 at 11:23:14PM +0000, Gabriel wrote:
> On Fri, 02 Nov 2012 22:06:04 +0000, Hugo Mills wrote:
> 
> > On Fri, Nov 02, 2012 at 07:05:37PM +0000, Gabriel wrote:
> >> On Fri, 02 Nov 2012 13:02:32 +0100, Goffredo Baroncelli wrote:
> >> > On 2012-11-02 12:18, Martin Steigerwald wrote:
> >> >> Metadata, DUP is displayed as 3,50GB on the device level and as 1,75GB
> >> >> in total. I understand the logic behind this, but this could be a bit
> >> >> confusing.
> >> >> 
> >> >> But it makes sense: Showing real allocation on device level makes
> >> >> sense,
> >> >> cause thats what really allocated on disk. Total makes some sense,
> >> >> cause thats what is being used from the tree by BTRFS.
> >> > 
> >> > Yes, me too. At the first I was confused when you noticed this
> >> > discrepancy. So I have to admit that it is not so obvious to understand.
> >> > However we didn't find any way to make it more clear...
> >> > 
> >> >> It still looks confusing at first…
> >> > We could use "Chunk(s) capacity" instead of total/size ? I would like an
> >> > opinion from a "english people" point of view..
> >> 
> >> This is easy to fix, here's a mockup:
> >> 
> >> Metadata,DUP: Size: 1.75GB ×2, Used: 627.84MB ×2
> >>    /dev/dm-0        3.50GB
> > 
> >    I've not considered the full semantics of all this yet -- I'll try
> > to do that tomorrow. However, I note that the "×2" here could become
> > non-integer with the RAID-5/6 code (which is due Real Soon Now). In
> > the first RAID-5/6 code drop, it won't even be simple to calculate
> > where there are different-sized devices in the filesystem. Putting an
> > exact figure on that number is potentially going to be awkward. I
> > think we're going to need kernel help for working out what that number
> > should be, in the general case.
> 
> DUP can be nested below a device because it represents same-device
> redundancy (purpose: survive smudges but not device failure).
> 
> On the other hand raid levels should occupy the same space on all
> linked devices (a necessary consequence of the guarantee that RAID5
> can survive the loss of any device and RAID6 any two devices).

   No, the multiplier here is variable. Consider:

1 MiB stored in RAID-5 across 3 devices takes up 1.5 MiB -- multiplier ×1.5
   (1 MiB over 2 devices is 512 KiB, plus an additional 512 KiB for parity)
1 MiB stored in RAID-5 across 6 devices takes up 1.2 MiB -- multipler ×1.2
   (1 MiB over 5 devices is 204.8 KiB, plus an additional 204.8 KiB for parity)

   With the (initial) proposed implementation of RAID-5, the
stripe-width (i.e. the number of devices used for any given chunk
allocation) will be *as many as can be allocated*. Chris confirmed
this today on IRC. So if I have a disk array of 2T, 2T, 2T, 1T, 1T,
1T, then the first 1T of allocation will stripe across 6 devices,
giving me 5 data+1 parity, or a multiplier of ×1.2. As soon as the
smaller devices are full, the stripe width will drop to 3 devices, and
we'll be using 2 data+1 parity allocation, or a multiplier of ×1.5 for
any subsequent chunks. So, as more data over the first 5T is stored,
the multiplier steadily decreases, until we fill the FS, and we get a
multiplier of ×1.35 overall. This gets more complicated if you have
devices of many different sizes. (Imagine 6 disks with sizes 500G, 1T,
1.5T, 2T, 3T, 3T).

   We probably can work out the current RAID overhead and feed it back
sensibly, but it's (a) not constant as the allocation of the chunks
increases, and (b) not trivial to compute.

> The two probably won't need to be represented at the same time
> except during a reshape, because I imagine DUP gets converted to
> RAID (1 or 5) as soon as the second device is added.
> 
> A 1→2 reshape would look a bit like this (doing only the data column
> and skipping totals):
> 
> InitialDevice
>   Reserved           1.21TB
>   Used               1.21TB
> RAID1(InitialDevice, SecondDevice)
>   Reserved   1.31TB + 100GB
>   Used             2× 100GB
> 
> RAID5, RAID6: same with fractions, n+1⁄n and n+2⁄n.

   Except that n isn't guaranteed to be constant. That was pretty much
my only point. Don't assume that it will be (or at the very least, be
aware that you are assuming it is, and be prepared for inconsistencies).

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
        --- Well, sir, the floor is yours.  But remember, the ---        
                              roof is ours!                              

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-02 23:44           ` Hugo Mills
@ 2012-11-03  0:14             ` Gabriel
  2012-11-03 12:28             ` Goffredo Baroncelli
  1 sibling, 0 replies; 24+ messages in thread
From: Gabriel @ 2012-11-03  0:14 UTC (permalink / raw)
  To: linux-btrfs

On Fri, 02 Nov 2012 23:44:19 +0000, Hugo Mills wrote:
> On Fri, Nov 02, 2012 at 11:23:14PM +0000, Gabriel wrote:
>> On Fri, 02 Nov 2012 22:06:04 +0000, Hugo Mills wrote:
>> >    I've not considered the full semantics of all this yet -- I'll try
>> > to do that tomorrow. However, I note that the "×2" here could become
>> > non-integer with the RAID-5/6 code (which is due Real Soon Now). In
>> > the first RAID-5/6 code drop, it won't even be simple to calculate
>> > where there are different-sized devices in the filesystem. Putting an
>> > exact figure on that number is potentially going to be awkward. I
>> > think we're going to need kernel help for working out what that number
>> > should be, in the general case.
>> 
>> DUP can be nested below a device because it represents same-device
>> redundancy (purpose: survive smudges but not device failure).
>> 
>> On the other hand raid levels should occupy the same space on all
>> linked devices (a necessary consequence of the guarantee that RAID5
>> can survive the loss of any device and RAID6 any two devices).
> 
>    No, the multiplier here is variable. Consider:
> 
> 1 MiB stored in RAID-5 across 3 devices takes up 1.5 MiB -- multiplier ×1.5
>    (1 MiB over 2 devices is 512 KiB, plus an additional 512 KiB for parity)
> 1 MiB stored in RAID-5 across 6 devices takes up 1.2 MiB -- multipler ×1.2
>    (1 MiB over 5 devices is 204.8 KiB, plus an additional 204.8 KiB for parity)
> 
>    With the (initial) proposed implementation of RAID-5, the
> stripe-width (i.e. the number of devices used for any given chunk
> allocation) will be *as many as can be allocated*. Chris confirmed
> this today on IRC. So if I have a disk array of 2T, 2T, 2T, 1T, 1T,
> 1T, then the first 1T of allocation will stripe across 6 devices,
> giving me 5 data+1 parity, or a multiplier of ×1.2. As soon as the
> smaller devices are full, the stripe width will drop to 3 devices, and
> we'll be using 2 data+1 parity allocation, or a multiplier of ×1.5 for
> any subsequent chunks. So, as more data over the first 5T is stored,
> the multiplier steadily decreases, until we fill the FS, and we get a
> multiplier of ×1.35 overall. This gets more complicated if you have
> devices of many different sizes. (Imagine 6 disks with sizes 500G, 1T,
> 1.5T, 2T, 3T, 3T).
> 
>    We probably can work out the current RAID overhead and feed it back
> sensibly, but it's (a) not constant as the allocation of the chunks
> increases, and (b) not trivial to compute.

All right, your example does illustrate things better. And I had no
idea about the implementation, but the as-many-stripes-as-possible
logic does make sense.

That doesn't break the sketch I made; I used RAIDn(device list)
as the block heading.

Your first example becomes:
RAID5(disk[1-6]), up to 6⁄5×5T.

Once that is filled we add a second block:
RAID5(disk[1-6])
  (the usual grid: free, reserved; data metadata system)
RAID5(disk[1-3]), 3⁄2×2T more.
  (the usual grid)

For proper reporting of free space we either need the kernel
to reserve all the blocks and tell us about them, or just some
info about the kernel's policy.
RAID5 with maximum stripes and no reduced redundancy is enough
info to compute the rest in userspace. Though the block approach
will be more reliable if the kernel has to make complicated policy
decisions, like the choice to reshape after device failure.

>> The two probably won't need to be represented at the same time
>> except during a reshape, because I imagine DUP gets converted to
>> RAID (1 or 5) as soon as the second device is added.
>> 
>> A 1→2 reshape would look a bit like this (doing only the data column
>> and skipping totals):
>> 
>> InitialDevice
>>   Reserved           1.21TB
>>   Used               1.21TB
>> RAID1(InitialDevice, SecondDevice)
>>   Reserved   1.31TB + 100GB
>>   Used             2× 100GB
>> 
>> RAID5, RAID6: same with fractions, n+1⁄n and n+2⁄n.
> 
>    Except that n isn't guaranteed to be constant. That was pretty much
> my only point. Don't assume that it will be (or at the very least, be
> aware that you are assuming it is, and be prepared for inconsistencies).
> 
>    Hugo.



^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-02 19:05     ` Gabriel
  2012-11-02 19:31       ` Goffredo Baroncelli
  2012-11-02 22:06       ` Hugo Mills
@ 2012-11-03 12:11       ` Goffredo Baroncelli
  2 siblings, 0 replies; 24+ messages in thread
From: Goffredo Baroncelli @ 2012-11-03 12:11 UTC (permalink / raw)
  To: Gabriel; +Cc: linux-btrfs

On 11/02/2012 08:05 PM, Gabriel wrote:
> On Fri, 02 Nov 2012 13:02:32 +0100, Goffredo Baroncelli wrote:
>> On 2012-11-02 12:18, Martin Steigerwald wrote:
[...]
>> We could use "Chunk(s) capacity" instead of total/size ? I would like an
>> opinion from a "english people" point of view..
> 
> This is easy to fix, here's a mockup:
> 
> Metadata,DUP: Size: 1.75GB ×2, Used: 627.84MB ×2
>    /dev/dm-0        3.50GB
> 
>            Data   Metadata Metadata    System System              
>            Single Single   DUP         Single DUP         Unallocated
>                                                                
> /dev/dm-16 1.31TB   8.00MB  56.00GB    4.00MB  16.00MB           0.00
>            ====== ======== =========== ====== =========== ===========
> Total      1.31TB   8.00MB  28.00GB ×2 4.00MB   8.00MB ×2        0.00
> Used       1.31TB     0.00   5.65GB ×2   0.00 152.00KB ×2            

I want to point out that we faced a lot of difficult to show that a
chunk has a capacity, and the space stored on the disk(s) is greater.
This leaded to show both the chunk (in term of Type and Profile [*]) and
the disk used.
The only thing that it will be a bit unclear is when the Profile is DUP,
because the disk usage is double of the space available.
For this reasons I am considering to put "2x" on the line related to the
disks.
Putting a "x2" In the total increase the confusion.

GB




[*] For type I means Data, Metadata, System and profile I means
DUP,Raid1/0/10/5/6,Single...


> 
> Also, I don't know if you could use libblkid, but it finds more 
> descriptive names than dm-NN (thanks to some smart sorting logic).
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-02 23:44           ` Hugo Mills
  2012-11-03  0:14             ` Gabriel
@ 2012-11-03 12:28             ` Goffredo Baroncelli
  1 sibling, 0 replies; 24+ messages in thread
From: Goffredo Baroncelli @ 2012-11-03 12:28 UTC (permalink / raw)
  To: Hugo Mills, Gabriel, linux-btrfs

On 11/03/2012 12:44 AM, Hugo Mills wrote:
> 1 MiB stored in RAID-5 across 3 devices takes up 1.5 MiB -- multiplier ×1.5
>    (1 MiB over 2 devices is 512 KiB, plus an additional 512 KiB for parity)
> 1 MiB stored in RAID-5 across 6 devices takes up 1.2 MiB -- multipler ×1.2
>    (1 MiB over 5 devices is 204.8 KiB, plus an additional 204.8 KiB for parity)
> 
>    With the (initial) proposed implementation of RAID-5, the
> stripe-width (i.e. the number of devices used for any given chunk
> allocation) will be *as many as can be allocated*. Chris confirmed
> this today on IRC. So if I have a disk array of 2T, 2T, 2T, 1T, 1T,
> 1T, then the first 1T of allocation will stripe across 6 devices,


Interesting.
Let me simulate a possible output


> ./btrfs filesystem disk-usage -t /
          Data    Metadata Metadata    System System
          RAID5   Single   DUP         Single DUP     Unallocated

/dev/dm-0  1.50TB   8.00MB           - 4.00MB 16.00MB    500.00MB
/dev/dm-1  1.50TB        -           - 4.00MB 16.00MB    500.00MB
/dev/dm-2  1.50TB        -           - 4.00MB       -    500.00MB
/dev/dm-3  1.00TB        - 2x 100.00MB 4.00MB       -    300.00MB
/dev/dm-4  1.00TB        - 2x 100.00MB 4.00MB       -    300.00MB
          ======= ======== =========== ====== ======= ===========
Total      5.00TB   8.00MB    200.00MB 4.00MB  8.00MB      2.10TB
Used      10.65GB     0.00     50.00MB   0.00  4.00KB


Would be it clear ? And what if we move the Total/used below the header ?

> ./btrfs filesystem disk-usage -t /
          Data    Metadata Metadata    System System
          RAID5   Single   DUP         Single DUP     Unallocated

Total      5.00TB   8.00MB    200.00MB 4.00MB  8.00MB      2.10TB
Used      10.65GB     0.00     50.00MB   0.00  4.00KB
          ======= ======== =========== ====== ======= ===========
/dev/dm-0  1.50TB   8.00MB           - 4.00MB 16.00MB    500.00MB
/dev/dm-1  1.50TB        -           - 4.00MB 16.00MB    500.00MB
/dev/dm-2  1.50TB        -           - 4.00MB       -    500.00MB
/dev/dm-3  1.00TB        - 2x 100.00MB 4.00MB       -    300.00MB
/dev/dm-4  1.00TB        - 2x 100.00MB 4.00MB       -    300.00MB




GB

P.S:
The raid5 is composed by (4+1) x 1T and (2+1) x 0.5TB, supposing the
disks as 2x 1.5TB and 3x2Tb.


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-02 22:06       ` Hugo Mills
  2012-11-02 23:23         ` Gabriel
@ 2012-11-03 12:35         ` Goffredo Baroncelli
  2012-11-03 22:04           ` cwillu
  1 sibling, 1 reply; 24+ messages in thread
From: Goffredo Baroncelli @ 2012-11-03 12:35 UTC (permalink / raw)
  To: Hugo Mills, Gabriel, linux-btrfs

On 11/02/2012 11:06 PM, Hugo Mills wrote:
> non-integer with the RAID-5/6 code (which is due Real Soon Now).
Hi Hugo,

do you have more information about raid ? When it will land on the btrfs
earth ? :-)

-- 
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH][BTRFS-PROGS] Enhance btrfs fi df
  2012-11-03 12:35         ` Goffredo Baroncelli
@ 2012-11-03 22:04           ` cwillu
  0 siblings, 0 replies; 24+ messages in thread
From: cwillu @ 2012-11-03 22:04 UTC (permalink / raw)
  To: kreijack; +Cc: Hugo Mills, Gabriel, linux-btrfs

> do you have more information about raid ? When it will land on the btrfs
> earth ? :-)

An unnamed source recently said "today I'm fixing parity rebuild in
the middle of a read/modify/write. its one of my last blockers", at
which point several gags about progress meters were made.

^ permalink raw reply	[flat|nested] 24+ messages in thread

end of thread, other threads:[~2012-11-03 22:04 UTC | newest]

Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-11-02 10:15 [PATCH][BTRFS-PROGS] Enhance btrfs fi df Goffredo Baroncelli
2012-11-02 10:15 ` [PATCH 1/8] Enhance the command btrfs filesystem df Goffredo Baroncelli
2012-11-02 10:15 ` [PATCH 2/8] Create the man page entry for the command btrfs fi df Goffredo Baroncelli
2012-11-02 10:15 ` [PATCH 3/8] Move open_file_or_dir() in utils.c Goffredo Baroncelli
2012-11-02 10:15 ` [PATCH 4/8] Move scrub_fs_info() and scrub_dev_info() " Goffredo Baroncelli
2012-11-02 10:15 ` [PATCH 5/8] Add command btrfs filesystem disk-usage Goffredo Baroncelli
2012-11-02 10:15 ` [PATCH 6/8] Create entry in man page for " Goffredo Baroncelli
2012-11-02 10:15 ` [PATCH 7/8] Add btrfs device disk-usage command Goffredo Baroncelli
2012-11-02 10:15 ` [PATCH 8/8] Create a new entry in btrfs man page for btrfs device disk-usage Goffredo Baroncelli
2012-11-02 11:18 ` [PATCH][BTRFS-PROGS] Enhance btrfs fi df Martin Steigerwald
2012-11-02 12:02   ` Goffredo Baroncelli
2012-11-02 19:05     ` Gabriel
2012-11-02 19:31       ` Goffredo Baroncelli
2012-11-02 20:40         ` Gabriel
2012-11-02 21:46           ` Michael Kjörling
2012-11-02 23:34             ` Gabriel
2012-11-02 22:06       ` Hugo Mills
2012-11-02 23:23         ` Gabriel
2012-11-02 23:44           ` Hugo Mills
2012-11-03  0:14             ` Gabriel
2012-11-03 12:28             ` Goffredo Baroncelli
2012-11-03 12:35         ` Goffredo Baroncelli
2012-11-03 22:04           ` cwillu
2012-11-03 12:11       ` Goffredo Baroncelli

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.