[PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
be compatible w any set of older/newer kernels.

So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
skinny-metadata even if the running kernel does not supports it, and
so the mount fails on the running.

Here in this set of patches will make sure the progs understands the
kernel supported features.

So in this patch, checks if sysfs tells whether the feature is
supported if not, then it will relay on static kernel version which
provided that feature (skinny-metadata here in this example), next
if for some reason the running kernel does not provide the kernel
version, then it will fall back to the original method to enable
the feature with a hope that kernel will support it.

Also the last patch adds a warning when we fail to read either
sysfs features or the running kernel version.

With this I hope all the concerns from the review comments are
addressed.


Anand Jain (5):
  btrfs-progs: introduce framework to check kernel supported features
  btrfs-progs: add framework to check features supported by sysfs
  btrfs-progs: kernel based default features for mkfs
  btrfs-progs: kernel based default features for btrfs-convert
  btrfs-progs: add warning when we fail to read sysfs or kernel version

 btrfs-convert.c |  18 ++++++-
 mkfs.c          |  22 ++++++++-
 utils.c         | 146 +++++++++++++++++++++++++++++++++++++++++++++++++++-----
 utils.h         |   2 +
 4 files changed, 173 insertions(+), 15 deletions(-)

-- 
2.6.2

[PATCH v2 1/5] btrfs-progs: introduce framework to check kernel supported features

[Anand Jain]

In the newer kernel, supported kernel features can be known from
  /sys/fs/btrfs/features
however this interface was introduced only after 3.14, and most the
incompatible FS features were introduce before 3.14.

This patch proposes to maintain kernel version against the feature list,
and so that will be the minimum kernel version needed to use the feature.

Further, for features supported later than 3.14 this list can still be
updated, so it serves as a repository which can be displayed for easy
reference.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
v2: Check for condition that what happens when we fail to read kernel
    version. Now the code will fail back to use the default as set by
    the progs.

 utils.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----
 utils.h |  1 +
 2 files changed, 76 insertions(+), 5 deletions(-)

diff --git a/utils.c b/utils.c
index b754686..24042e5 100644
--- a/utils.c
+++ b/utils.c
@@ -32,10 +32,12 @@
 #include <linux/loop.h>
 #include <linux/major.h>
 #include <linux/kdev_t.h>
+#include <linux/version.h>
 #include <limits.h>
 #include <blkid/blkid.h>
 #include <sys/vfs.h>
 #include <sys/statfs.h>
+#include <sys/utsname.h>
 #include <linux/magic.h>
 
 #include "kerncompat.h"
@@ -567,21 +569,28 @@ out:
 	return ret;
 }
 
+/*
+ * min_ker_ver: update with minimum kernel version at which the feature
+ * was integrated into the mainline. For the transit period, that is
+ * feature not yet in mainline but in mailing list and for testing,
+ * please use "0.0" to indicate the same.
+ */
 static const struct btrfs_fs_feature {
 	const char *name;
 	u64 flag;
 	const char *desc;
+	const char *min_ker_ver;
 } mkfs_features[] = {
 	{ "mixed-bg", BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS,
-		"mixed data and metadata block groups" },
+		"mixed data and metadata block groups", "2.7.31"},
 	{ "extref", BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF,
-		"increased hardlink limit per file to 65536" },
+		"increased hardlink limit per file to 65536", "3.7"},
 	{ "raid56", BTRFS_FEATURE_INCOMPAT_RAID56,
-		"raid56 extended format" },
+		"raid56 extended format", "3.9"},
 	{ "skinny-metadata", BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA,
-		"reduced-size metadata extent refs" },
+		"reduced-size metadata extent refs", "3.10"},
 	{ "no-holes", BTRFS_FEATURE_INCOMPAT_NO_HOLES,
-		"no explicit hole extents for files" },
+		"no explicit hole extents for files", "3.14"},
 	/* Keep this one last */
 	{ "list-all", BTRFS_FEATURE_LIST_ALL, NULL }
 };
@@ -3077,3 +3086,64 @@ unsigned int get_unit_mode_from_arg(int *argc, char *argv[], int df_mode)
 
 	return unit_mode;
 }
+
+static int version_to_code(char *v)
+{
+	int i = 0;
+	char *b[3] = {NULL};
+	char *save_b = NULL;
+
+	for (b[i] = strtok_r(v, ".", &save_b);
+		b[i] != NULL;
+		b[i] = strtok_r(NULL, ".", &save_b))
+		i++;
+
+	if (b[2] == NULL)
+		return KERNEL_VERSION(atoi(b[0]), atoi(b[1]), 0);
+	else
+		return KERNEL_VERSION(atoi(b[0]), atoi(b[1]), atoi(b[2]));
+
+}
+
+static int get_kernel_code()
+{
+	int ret;
+	struct utsname utsbuf;
+	char *version;
+
+	ret = uname(&utsbuf);
+	if (ret)
+		return -ret;
+
+	if (!strlen(utsbuf.release))
+		return -EINVAL;
+
+	version = strtok(utsbuf.release, "-");
+
+	return version_to_code(version);
+}
+
+u64 btrfs_features_allowed_by_kernel(void)
+{
+	int i;
+	int local_kernel_code = get_kernel_code();
+	u64 features = 0;
+
+	/*
+	 * When system did not provide the kernel version then just
+	 * return 0, the caller has to depend on the intelligence as
+	 * per btrfs-progs version
+	 */
+	if (local_kernel_code <= 0)
+		return 0;
+
+	for (i = 0; i < ARRAY_SIZE(mkfs_features) - 1; i++) {
+		char *ver = strdup(mkfs_features[i].min_ker_ver);
+
+		if (local_kernel_code >= version_to_code(ver))
+			features |= mkfs_features[i].flag;
+
+		free(ver);
+	}
+	return (features);
+}
diff --git a/utils.h b/utils.h
index 192f3d1..9044643 100644
--- a/utils.h
+++ b/utils.h
@@ -104,6 +104,7 @@ void btrfs_list_all_fs_features(u64 mask_disallowed);
 char* btrfs_parse_fs_features(char *namelist, u64 *flags);
 void btrfs_process_fs_features(u64 flags);
 void btrfs_parse_features_to_string(char *buf, u64 flags);
+u64 btrfs_features_allowed_by_kernel(void);
 
 struct btrfs_mkfs_config {
 	char *label;
-- 
2.6.2

[PATCH v2 2/5] btrfs-progs: add framework to check features supported by sysfs

[Anand Jain]

This adds a framework to check the /sys/fs/btrfs/features for the list
of supported features by the btrfs kernel. Which I hope by using it the
mkfs and btrfs-convert could tune to set features as supported by the
running kernel.


Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 utils.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------
 utils.h |  1 +
 2 files changed, 59 insertions(+), 8 deletions(-)

diff --git a/utils.c b/utils.c
index 24042e5..48a1989 100644
--- a/utils.c
+++ b/utils.c
@@ -577,22 +577,23 @@ out:
  */
 static const struct btrfs_fs_feature {
 	const char *name;
+	const char *name_ker;
 	u64 flag;
 	const char *desc;
 	const char *min_ker_ver;
 } mkfs_features[] = {
-	{ "mixed-bg", BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS,
+	{ "mixed-bg", "mixed_groups", BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS,
 		"mixed data and metadata block groups", "2.7.31"},
-	{ "extref", BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF,
+	{ "extref", "extended_iref", BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF,
 		"increased hardlink limit per file to 65536", "3.7"},
-	{ "raid56", BTRFS_FEATURE_INCOMPAT_RAID56,
+	{ "raid56", "raid56", BTRFS_FEATURE_INCOMPAT_RAID56,
 		"raid56 extended format", "3.9"},
-	{ "skinny-metadata", BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA,
+	{ "skinny-metadata", "skinny_metadata", BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA,
 		"reduced-size metadata extent refs", "3.10"},
-	{ "no-holes", BTRFS_FEATURE_INCOMPAT_NO_HOLES,
+	{ "no-holes", "no_holes", BTRFS_FEATURE_INCOMPAT_NO_HOLES,
 		"no explicit hole extents for files", "3.14"},
 	/* Keep this one last */
-	{ "list-all", BTRFS_FEATURE_LIST_ALL, NULL }
+	{ "list-all", "", BTRFS_FEATURE_LIST_ALL, NULL }
 };
 
 static int parse_one_fs_feature(const char *name, u64 *flags)
@@ -602,10 +603,12 @@ static int parse_one_fs_feature(const char *name, u64 *flags)
 
 	for (i = 0; i < ARRAY_SIZE(mkfs_features); i++) {
 		if (name[0] == '^' &&
-			!strcmp(mkfs_features[i].name, name + 1)) {
+			(!strcmp(mkfs_features[i].name, name + 1) ||
+			!strcmp(mkfs_features[i].name_ker, name + 1))) {
 			*flags &= ~ mkfs_features[i].flag;
 			found = 1;
-		} else if (!strcmp(mkfs_features[i].name, name)) {
+		} else if (!strcmp(mkfs_features[i].name, name) ||
+			!strcmp(mkfs_features[i].name_ker, name)) {
 			*flags |= mkfs_features[i].flag;
 			found = 1;
 		}
@@ -3147,3 +3150,50 @@ u64 btrfs_features_allowed_by_kernel(void)
 	}
 	return (features);
 }
+
+int check_or_load_btrfs_ko()
+{
+	int fd;
+
+	/*
+	 * open will load btrfs kernel module if its not loaded,
+	 * and if the kernel has CONFIG auto load set?
+	 */
+	fd = open("/dev/btrfs-control", O_RDONLY);
+	if (fd < 0)
+		return -errno;
+
+	close(fd);
+	return 0;
+}
+
+int btrfs_features_allowed_by_sysfs(u64 *features)
+{
+	int ret;
+	DIR *dir;
+	struct dirent *ent;
+
+	ret = check_or_load_btrfs_ko();
+	if (ret) {
+		/* returns, -errno */
+		return ret;
+	}
+
+	dir = opendir("/sys/fs/btrfs/features");
+	if (!dir) {
+		/*
+		 * An old kernel which does not support sysfs/features
+		 */
+		return -errno;
+	}
+
+	*features = 0;
+	while((ent = readdir(dir)) != NULL) {
+		if (!strcmp(".", ent->d_name) ||
+				!strcmp("..", ent->d_name))
+			continue;
+		parse_one_fs_feature(ent->d_name, features);
+	}
+	closedir(dir);
+	return 0;
+}
diff --git a/utils.h b/utils.h
index 9044643..af0aa31 100644
--- a/utils.h
+++ b/utils.h
@@ -105,6 +105,7 @@ char* btrfs_parse_fs_features(char *namelist, u64 *flags);
 void btrfs_process_fs_features(u64 flags);
 void btrfs_parse_features_to_string(char *buf, u64 flags);
 u64 btrfs_features_allowed_by_kernel(void);
+int btrfs_features_allowed_by_sysfs(u64 *features);
 
 struct btrfs_mkfs_config {
 	char *label;
-- 
2.6.2

[PATCH v2 3/5] btrfs-progs: kernel based default features for mkfs

[Anand Jain]

Mkfs from latest btrfs-progs will enable latest default features,
and if the kernel is down-rev and does not support a latest default
feature then mount fails, as expected.

This patch disables default features based on the running kernel.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
v2: Check if sysfs tells what features are supported

 mkfs.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/mkfs.c b/mkfs.c
index a5802f7..4f46ad9 100644
--- a/mkfs.c
+++ b/mkfs.c
@@ -1357,10 +1357,24 @@ int main(int ac, char **av)
 	int dev_cnt = 0;
 	int saved_optind;
 	char fs_uuid[BTRFS_UUID_UNPARSED_SIZE] = { 0 };
-	u64 features = BTRFS_MKFS_DEFAULT_FEATURES;
+	u64 features;
 	struct mkfs_allocation allocation = { 0 };
 	struct btrfs_mkfs_config mkfs_cfg;
 
+	/*
+	 * If kernel could tell supported features by sysfs then use it,
+	 * if not, then set it by static kernel version, if the this also
+	 * fails then default to the progs default, which is set as per
+	 * BTRFS_MKFS_DEFAULT_FEATURES
+	 */
+	if (btrfs_features_allowed_by_sysfs(&features))
+		features = btrfs_features_allowed_by_kernel();
+
+	if (features)
+		features &= BTRFS_MKFS_DEFAULT_FEATURES;
+	else
+		features = BTRFS_MKFS_DEFAULT_FEATURES;
+
 	while(1) {
 		int c;
 		static const struct option long_options[] = {
-- 
2.6.2

[PATCH v2 4/5] btrfs-progs: kernel based default features for btrfs-convert

[Anand Jain]

btrfs-convert convert FS with latest default features enabled, and
if the kernel is down-rev and does not support a latest feature then
mount fails, as expected.

This patch disables default features based on the running kernel.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
v2: Check if sysfs tells what features are supported

 btrfs-convert.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 73ac584..52ea12a 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -2896,7 +2896,15 @@ int main(int argc, char *argv[])
 	int progress = 1;
 	char *file;
 	char fslabel[BTRFS_LABEL_SIZE];
-	u64 features = BTRFS_MKFS_DEFAULT_FEATURES;
+	u64 features;
+
+	if (btrfs_features_allowed_by_sysfs(&features))
+		features = btrfs_features_allowed_by_kernel();
+
+	if (features)
+		features &= BTRFS_MKFS_DEFAULT_FEATURES;
+	else
+		features = BTRFS_MKFS_DEFAULT_FEATURES;
 
 	while(1) {
 		enum { GETOPT_VAL_NO_PROGRESS = 256 };
-- 
2.6.2

[PATCH 5/5] btrfs-progs: add warning when we fail to read sysfs or version

[Anand Jain]

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 btrfs-convert.c | 10 +++++++++-
 mkfs.c          |  8 +++++++-
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 52ea12a..b0a998b 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -2898,14 +2898,22 @@ int main(int argc, char *argv[])
 	char fslabel[BTRFS_LABEL_SIZE];
 	u64 features;
 
-	if (btrfs_features_allowed_by_sysfs(&features))
+	if (btrfs_features_allowed_by_sysfs(&features)) {
+		fprintf(stderr,
+		"Warning: Failed/Unsupported to obtain feature list from sysfs\n");
+
 		features = btrfs_features_allowed_by_kernel();
+		if (!features)
+			fprintf(stderr,
+		"Warning: Failed/Unsupported to get running kernel version\n");
+	}
 
 	if (features)
 		features &= BTRFS_MKFS_DEFAULT_FEATURES;
 	else
 		features = BTRFS_MKFS_DEFAULT_FEATURES;
 
+
 	while(1) {
 		enum { GETOPT_VAL_NO_PROGRESS = 256 };
 		static const struct option long_options[] = {
diff --git a/mkfs.c b/mkfs.c
index 4f46ad9..6cb998b 100644
--- a/mkfs.c
+++ b/mkfs.c
@@ -1367,8 +1367,14 @@ int main(int ac, char **av)
 	 * fails then default to the progs default, which is set as per
 	 * BTRFS_MKFS_DEFAULT_FEATURES
 	 */
-	if (btrfs_features_allowed_by_sysfs(&features))
+	if (btrfs_features_allowed_by_sysfs(&features)) {
+		fprintf(stderr,
+		"Warning: Failed/Unsupported to obtain feature list from sysfs\n");
 		features = btrfs_features_allowed_by_kernel();
+		if (!features)
+			fprintf(stderr,
+		"Warning: Failed/Unsupported to get running kernel version\n");
+	}
 
 	if (features)
 		features &= BTRFS_MKFS_DEFAULT_FEATURES;
-- 
2.6.2

Re: [PATCH v2 3/5] btrfs-progs: kernel based default features for mkfs

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 895 bytes --]

Hey.

On Mon, 2015-11-23 at 20:56 +0800, Anand Jain wrote:
> This patch disables default features based on the running kernel.
Not sure if that's very realistic in practise (most people will have
some distro, whose btrfsprogs version probably matches the kernel), but
purely from the end-user PoV:

I would find it useful if btrfs gives a warning if it creates a
filesystem which (because unsupported in the current kernel) lacks
features which are considered default by then.


AFAIU, really "clonding" (I mean including all snapshots, subvols,
etc.) a btrfs is not possible right now (or is it?), so a btrfs is
something that rather should stay longer (as one cannot easily
copy&swap it to/with a new one)... so for me as an end-user, it may be
easier to switch to a newer kernel, in order to get a feature which is
default, than to migrate the fs later.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: [PATCH v2 3/5] btrfs-progs: kernel based default features for mkfs

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1688 bytes --]

On 2015-11-23 10:57, Christoph Anton Mitterer wrote:
> Hey.
>
> On Mon, 2015-11-23 at 20:56 +0800, Anand Jain wrote:
>> This patch disables default features based on the running kernel.
> Not sure if that's very realistic in practise (most people will have
> some distro, whose btrfsprogs version probably matches the kernel), but
> purely from the end-user PoV:
>
> I would find it useful if btrfs gives a warning if it creates a
> filesystem which (because unsupported in the current kernel) lacks
> features which are considered default by then.
It should give a warning if the user requests a feature that is 
unsupported by the kernel it's being run on, but it should not by 
default try to enable something that isn't supported by the kernel it's 
running on.
>
>
> AFAIU, really "clonding" (I mean including all snapshots, subvols,
> etc.) a btrfs is not possible right now (or is it?), so a btrfs is
> something that rather should stay longer (as one cannot easily
> copy&swap it to/with a new one)... so for me as an end-user, it may be
> easier to switch to a newer kernel, in order to get a feature which is
> default, than to migrate the fs later.
It is actually possible to clone a btrfs filesystem, just not in a way 
that people used to stuff like ext4 would recognize.  In essence, you 
need to take the FS mostly off-line, force all subvolumes to read-only, 
then use send-receive to transfer things, and finally make the 
subvolumes writable again.  I've been considering doing a script to do 
this automatically, but have never gotten around to it as it's not 
something that is quick to code, and it's not something I do very often.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: [PATCH v2 3/5] btrfs-progs: kernel based default features for mkfs

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1576 bytes --]

On Mon, 2015-11-23 at 11:05 -0500, Austin S Hemmelgarn wrote:
> I would find it useful if btrfs gives a warning if it creates a
> > filesystem which (because unsupported in the current kernel) lacks
> > features which are considered default by then.
> It should give a warning if the user requests a feature that is 
> unsupported by the kernel it's being run on, but it should not by 
> default try to enable something that isn't supported by the kernel
> it's 
> running on.
Well that as well, and of course it shouldn't try to enable a feature
that wouldn't work, but what I meant was, e.g. if I create a fs with
btrfs-progs 4.3 (where skinny-extents are default) but on such an old
kernel where this isn't supported yet,... it should tell me "Normally
I'd create the fs with skinny-extents, but I don't as your kernel is
too old".


> It is actually possible to clone a btrfs filesystem, just not in a
> way 
> that people used to stuff like ext4 would recognize.  In essence, you
> need to take the FS mostly off-line, force all subvolumes to read-
> only, 
> then use send-receive to transfer things, and finally make the 
> subvolumes writable again.  I've been considering doing a script to
> do 
> this automatically, but have never gotten around to it as it's not 
> something that is quick to code, and it's not something I do very
> often.
And that would also keep all ref-links, etc.? I.e. the copied fs
wouldn't eat up much more space than the original?
Well than such script should be part of btrfs-progs :-)

Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: [PATCH v2 3/5] btrfs-progs: kernel based default features for mkfs

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2109 bytes --]

On 2015-11-23 11:14, Christoph Anton Mitterer wrote:
> On Mon, 2015-11-23 at 11:05 -0500, Austin S Hemmelgarn wrote:
>> I would find it useful if btrfs gives a warning if it creates a
>>> filesystem which (because unsupported in the current kernel) lacks
>>> features which are considered default by then.
>> It should give a warning if the user requests a feature that is
>> unsupported by the kernel it's being run on, but it should not by
>> default try to enable something that isn't supported by the kernel
>> it's
>> running on.
> Well that as well, and of course it shouldn't try to enable a feature
> that wouldn't work, but what I meant was, e.g. if I create a fs with
> btrfs-progs 4.3 (where skinny-extents are default) but on such an old
> kernel where this isn't supported yet,... it should tell me "Normally
> I'd create the fs with skinny-extents, but I don't as your kernel is
> too old".
Ah, I misunderstood your meaning, that would be nice to have.
>> It is actually possible to clone a btrfs filesystem, just not in a
>> way
>> that people used to stuff like ext4 would recognize.  In essence, you
>> need to take the FS mostly off-line, force all subvolumes to read-
>> only,
>> then use send-receive to transfer things, and finally make the
>> subvolumes writable again.  I've been considering doing a script to
>> do
>> this automatically, but have never gotten around to it as it's not
>> something that is quick to code, and it's not something I do very
>> often.
> And that would also keep all ref-links, etc.? I.e. the copied fs
> wouldn't eat up much more space than the original?
> Well than such script should be part of btrfs-progs :-)
There's no way to reliably preserve _all_ ref-links, but between 
snapshots and the subvolumes they are snapshots of, it should, assuming 
that you either send them all at once (which would take a long time 
probably), or that you do proper incremental transfers.  The other 
disadvantage is that it may not (depending on the features of course) 
cause any new features to be used except on new files.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[David Sterba]

On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
> Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
> be compatible w any set of older/newer kernels.
> 
> So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
> skinny-metadata even if the running kernel does not supports it, and
> so the mount fails on the running.

So the default behaviour of mkfs will try to best guess the feature set
of currently running kernel. I think this is is the most common scenario
and justifies the change in default behaviours.

For the other cases I'd like to introduce some human-readable shortcuts
to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
options supported by the unpatched mainline kernel of version 3.2. This
would be present for all version, regardless if there was a change in the
options or not.

Similarly for convenience, add 'running' that would pick the options
from running kernel but will be explicit.

A remaining option should override the 'running' behaviour and pick the
latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
name is yet to be determined.

> Here in this set of patches will make sure the progs understands the
> kernel supported features.
> 
> So in this patch, checks if sysfs tells whether the feature is
> supported if not, then it will relay on static kernel version which
> provided that feature (skinny-metadata here in this example), next
> if for some reason the running kernel does not provide the kernel
> version, then it will fall back to the original method to enable
> the feature with a hope that kernel will support it.
> 
> Also the last patch adds a warning when we fail to read either
> sysfs features or the running kernel version.

Your patchset is a good start, the additional options I've described can
be added on top of that. We might need to switch the version
representation from string to KERNEL_VERSION but that's an
implementation detail.

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2283 bytes --]

On 2015-11-23 12:56, David Sterba wrote:
> On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
>> Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
>> be compatible w any set of older/newer kernels.
>>
>> So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
>> skinny-metadata even if the running kernel does not supports it, and
>> so the mount fails on the running.
>
> So the default behaviour of mkfs will try to best guess the feature set
> of currently running kernel. I think this is is the most common scenario
> and justifies the change in default behaviours.
I feel that Christoph's suggestion in the other sub-thread to have it 
spit out a notice that it disabled something because of the kernel it's 
running on is worth adding also.  We should probably also spit out a 
warning if the user asks for a feature that isn't supported on the 
current kernel (but still let them create the filesystem regardless).
>
> For the other cases I'd like to introduce some human-readable shortcuts
> to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
> options supported by the unpatched mainline kernel of version 3.2. This
> would be present for all version, regardless if there was a change in the
> options or not.
>
> Similarly for convenience, add 'running' that would pick the options
> from running kernel but will be explicit.
Is the intent to enable stuff that the devs consider stable that's 
supported by the running kernel, or all the features supported by the 
running kernel?  It's probably best to use the first as the defaults, 
and then have an option to pull in everything the running kernel 
supports (possibly name that option something like 'running-all').
>
> A remaining option should override the 'running' behaviour and pick the
> latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
> name is yet to be determined.
Maybe something like 'recommended' or 'suggested'?

It might also be nice to have an option to tell it to turn on everything 
the tools support (possibly call that one something like 
'max-features'), though this is probably less useful due to the fact 
that most mkfs features in BTRFS are incompat features.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Duncan]

Austin S Hemmelgarn posted on Mon, 23 Nov 2015 15:14:44 -0500 as
excerpted:

>> A remaining option should override the 'running' behaviour and pick the
>> latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
>> name is yet to be determined.

> Maybe something like 'recommended' or 'suggested'?

Just keep it simple.  If it enables the latest defaults, just call it 
"latest".  =:^)

Better yet, match compat-3.2, etc, with compat-running and compat-latest, 
making it crystal clear they're parallel options, as well as labeling 
them all compat-*, making it crystal clear what the intended 
functionality is.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

Thanks for comments.


Distro would also want to use the latest btrfs-progs on older kernel
since it will have latest fsck/send/receive fixes, better UI
and updated man pages.

btrfs-progs which claim backward kernel compatible and it shouldn't
fail on the below cmd when the btrfs-progs is upgraded.
    mkfs.btrfs /dev/sda && mount /dev/sda /btrfs
but it does in some test cases.


A warning is unnecessary IMO. Imagine user who upgrade progs for
better doc/UI and have no intention to upgrade the kernel, gets a 
Warning!. If user does not upgrade kernel its a fair assumption
that they don't need/not-looking for latest kernel features. (What
did I miss ?).


Next.
For users looking to have a disk-layout which is compatible with
older kernels (and may not be a running kernel), then with the
current patch set its quite possible to do something like below,

mkfs.btrfs -O as-per-kernel=3.2
mkfs.btrfs -O as-per-kernel=4.0
mkfs.btrfs -O as-per-kernel=x.x (anything)

And only those features that are supported until version x.x
(mainline) will be enabled by default unless user want to over
default totally by using -O <feature>.

Thanks, Anand



Anand Jain wrote:
> Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
> be compatible w any set of older/newer kernels.
>
> So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
> skinny-metadata even if the running kernel does not supports it, and
> so the mount fails on the running.
>
> Here in this set of patches will make sure the progs understands the
> kernel supported features.
>
> So in this patch, checks if sysfs tells whether the feature is
> supported if not, then it will relay on static kernel version which
> provided that feature (skinny-metadata here in this example), next
> if for some reason the running kernel does not provide the kernel
> version, then it will fall back to the original method to enable
> the feature with a hope that kernel will support it.
>
> Also the last patch adds a warning when we fail to read either
> sysfs features or the running kernel version.
>
> With this I hope all the concerns from the review comments are
> addressed.
>
>
> Anand Jain (5):
>    btrfs-progs: introduce framework to check kernel supported features
>    btrfs-progs: add framework to check features supported by sysfs
>    btrfs-progs: kernel based default features for mkfs
>    btrfs-progs: kernel based default features for btrfs-convert
>    btrfs-progs: add warning when we fail to read sysfs or kernel version
>
>   btrfs-convert.c |  18 ++++++-
>   mkfs.c          |  22 ++++++++-
>   utils.c         | 146 +++++++++++++++++++++++++++++++++++++++++++++++++++-----
>   utils.h         |   2 +
>   4 files changed, 173 insertions(+), 15 deletions(-)
>

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

David Sterba wrote:
> On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
>> Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
>> be compatible w any set of older/newer kernels.
>>
>> So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
>> skinny-metadata even if the running kernel does not supports it, and
>> so the mount fails on the running.
>
> So the default behaviour of mkfs will try to best guess the feature set
> of currently running kernel. I think this is is the most common scenario
> and justifies the change in default behaviours.
>
> For the other cases I'd like to introduce some human-readable shortcuts
> to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
> options supported by the unpatched mainline kernel of version 3.2.

  This is a nice idea. I am planning. How about 'as-per-kernel=x.x'
  instead of compat-3.2.

  Also looks like it better to list the feature and version mapping
  as btrfs-progs already knows it at this patchset.


> This
> would be present for all version, regardless if there was a change in the
> options or not.

   Hmm.. I didn't quite get that.

> Similarly for convenience, add 'running' that would pick the options
> from running kernel but will be explicit.
>
> A remaining option should override the 'running' behaviour and pick the
> latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
> name is yet to be determined.
>
>> Here in this set of patches will make sure the progs understands the
>> kernel supported features.
>>
>> So in this patch, checks if sysfs tells whether the feature is
>> supported if not, then it will relay on static kernel version which
>> provided that feature (skinny-metadata here in this example), next
>> if for some reason the running kernel does not provide the kernel
>> version, then it will fall back to the original method to enable
>> the feature with a hope that kernel will support it.
>>
>> Also the last patch adds a warning when we fail to read either
>> sysfs features or the running kernel version.
>
> Your patchset is a good start, the additional options I've described can
> be added on top of that.

  Yes.

Thanks, Anand



> We might need to switch the version
> representation from string to KERNEL_VERSION but that's an
> implementation detail.



> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH v2 1/5] btrfs-progs: introduce framework to check kernel supported features

[Mike Fleetwood]

On 23 November 2015 at 12:56, Anand Jain <anand.jain@oracle.com> wrote:
> In the newer kernel, supported kernel features can be known from
>   /sys/fs/btrfs/features
> however this interface was introduced only after 3.14, and most the
> incompatible FS features were introduce before 3.14.
>
> This patch proposes to maintain kernel version against the feature list,
> and so that will be the minimum kernel version needed to use the feature.
>
> Further, for features supported later than 3.14 this list can still be
> updated, so it serves as a repository which can be displayed for easy
> reference.
>
> Signed-off-by: Anand Jain <anand.jain@oracle.com>
> ---
> v2: Check for condition that what happens when we fail to read kernel
>     version. Now the code will fail back to use the default as set by
>     the progs.
>
>  utils.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----
>  utils.h |  1 +
>  2 files changed, 76 insertions(+), 5 deletions(-)
>
> diff --git a/utils.c b/utils.c
> index b754686..24042e5 100644
> --- a/utils.c
> +++ b/utils.c
> @@ -32,10 +32,12 @@
>  #include <linux/loop.h>
>  #include <linux/major.h>
>  #include <linux/kdev_t.h>
> +#include <linux/version.h>
>  #include <limits.h>
>  #include <blkid/blkid.h>
>  #include <sys/vfs.h>
>  #include <sys/statfs.h>
> +#include <sys/utsname.h>
>  #include <linux/magic.h>
>
>  #include "kerncompat.h"
> @@ -567,21 +569,28 @@ out:
>         return ret;
>  }
>
> +/*
> + * min_ker_ver: update with minimum kernel version at which the feature
> + * was integrated into the mainline. For the transit period, that is
> + * feature not yet in mainline but in mailing list and for testing,
> + * please use "0.0" to indicate the same.
> + */
>  static const struct btrfs_fs_feature {
>         const char *name;
>         u64 flag;
>         const char *desc;
> +       const char *min_ker_ver;
>  } mkfs_features[] = {
>         { "mixed-bg", BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS,
> -               "mixed data and metadata block groups" },
> +               "mixed data and metadata block groups", "2.7.31"},
I think you mean 2.6.37 here.
67377734fd24c3 "Btrfs: add support for mixed data+metadata block groups"

Thanks,
Mike

>         { "extref", BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF,
> -               "increased hardlink limit per file to 65536" },
> +               "increased hardlink limit per file to 65536", "3.7"},
>         { "raid56", BTRFS_FEATURE_INCOMPAT_RAID56,
> -               "raid56 extended format" },
> +               "raid56 extended format", "3.9"},
>         { "skinny-metadata", BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA,
> -               "reduced-size metadata extent refs" },
> +               "reduced-size metadata extent refs", "3.10"},
>         { "no-holes", BTRFS_FEATURE_INCOMPAT_NO_HOLES,
> -               "no explicit hole extents for files" },
> +               "no explicit hole extents for files", "3.14"},
>         /* Keep this one last */
>         { "list-all", BTRFS_FEATURE_LIST_ALL, NULL }
>  };
> @@ -3077,3 +3086,64 @@ unsigned int get_unit_mode_from_arg(int *argc, char *argv[], int df_mode)
>
>         return unit_mode;
>  }
> +
> +static int version_to_code(char *v)
> +{
> +       int i = 0;
> +       char *b[3] = {NULL};
> +       char *save_b = NULL;
> +
> +       for (b[i] = strtok_r(v, ".", &save_b);
> +               b[i] != NULL;
> +               b[i] = strtok_r(NULL, ".", &save_b))
> +               i++;
> +
> +       if (b[2] == NULL)
> +               return KERNEL_VERSION(atoi(b[0]), atoi(b[1]), 0);
> +       else
> +               return KERNEL_VERSION(atoi(b[0]), atoi(b[1]), atoi(b[2]));
> +
> +}
> +
> +static int get_kernel_code()
> +{
> +       int ret;
> +       struct utsname utsbuf;
> +       char *version;
> +
> +       ret = uname(&utsbuf);
> +       if (ret)
> +               return -ret;
> +
> +       if (!strlen(utsbuf.release))
> +               return -EINVAL;
> +
> +       version = strtok(utsbuf.release, "-");
> +
> +       return version_to_code(version);
> +}
> +
> +u64 btrfs_features_allowed_by_kernel(void)
> +{
> +       int i;
> +       int local_kernel_code = get_kernel_code();
> +       u64 features = 0;
> +
> +       /*
> +        * When system did not provide the kernel version then just
> +        * return 0, the caller has to depend on the intelligence as
> +        * per btrfs-progs version
> +        */
> +       if (local_kernel_code <= 0)
> +               return 0;
> +
> +       for (i = 0; i < ARRAY_SIZE(mkfs_features) - 1; i++) {
> +               char *ver = strdup(mkfs_features[i].min_ker_ver);
> +
> +               if (local_kernel_code >= version_to_code(ver))
> +                       features |= mkfs_features[i].flag;
> +
> +               free(ver);
> +       }
> +       return (features);
> +}
> diff --git a/utils.h b/utils.h
> index 192f3d1..9044643 100644
> --- a/utils.h
> +++ b/utils.h
> @@ -104,6 +104,7 @@ void btrfs_list_all_fs_features(u64 mask_disallowed);
>  char* btrfs_parse_fs_features(char *namelist, u64 *flags);
>  void btrfs_process_fs_features(u64 flags);
>  void btrfs_parse_features_to_string(char *buf, u64 flags);
> +u64 btrfs_features_allowed_by_kernel(void);
>
>  struct btrfs_mkfs_config {
>         char *label;
> --
> 2.6.2

Re: [PATCH v2 1/5] btrfs-progs: introduce framework to check kernel supported features

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2750 bytes --]

On 2015-11-24 09:39, Mike Fleetwood wrote:
> On 23 November 2015 at 12:56, Anand Jain <anand.jain@oracle.com> wrote:
>> In the newer kernel, supported kernel features can be known from
>>    /sys/fs/btrfs/features
>> however this interface was introduced only after 3.14, and most the
>> incompatible FS features were introduce before 3.14.
>>
>> This patch proposes to maintain kernel version against the feature list,
>> and so that will be the minimum kernel version needed to use the feature.
>>
>> Further, for features supported later than 3.14 this list can still be
>> updated, so it serves as a repository which can be displayed for easy
>> reference.
>>
>> Signed-off-by: Anand Jain <anand.jain@oracle.com>
>> ---
>> v2: Check for condition that what happens when we fail to read kernel
>>      version. Now the code will fail back to use the default as set by
>>      the progs.
>>
>>   utils.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----
>>   utils.h |  1 +
>>   2 files changed, 76 insertions(+), 5 deletions(-)
>>
>> diff --git a/utils.c b/utils.c
>> index b754686..24042e5 100644
>> --- a/utils.c
>> +++ b/utils.c
>> @@ -32,10 +32,12 @@
>>   #include <linux/loop.h>
>>   #include <linux/major.h>
>>   #include <linux/kdev_t.h>
>> +#include <linux/version.h>
>>   #include <limits.h>
>>   #include <blkid/blkid.h>
>>   #include <sys/vfs.h>
>>   #include <sys/statfs.h>
>> +#include <sys/utsname.h>
>>   #include <linux/magic.h>
>>
>>   #include "kerncompat.h"
>> @@ -567,21 +569,28 @@ out:
>>          return ret;
>>   }
>>
>> +/*
>> + * min_ker_ver: update with minimum kernel version at which the feature
>> + * was integrated into the mainline. For the transit period, that is
>> + * feature not yet in mainline but in mailing list and for testing,
>> + * please use "0.0" to indicate the same.
>> + */
>>   static const struct btrfs_fs_feature {
>>          const char *name;
>>          u64 flag;
>>          const char *desc;
>> +       const char *min_ker_ver;
>>   } mkfs_features[] = {
>>          { "mixed-bg", BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS,
>> -               "mixed data and metadata block groups" },
>> +               "mixed data and metadata block groups", "2.7.31"},
> I think you mean 2.6.37 here.
> 67377734fd24c3 "Btrfs: add support for mixed data+metadata block groups"
This brings up a rather important question:
Should compat-X.Y mean features that were considered usable in that 
version, or everything that version offered?  I understand wanting 
consistency with the kernel versions, but we shouldn't be creating 
filesystems that we know will break on the specified kernel even if it 
is mountable on it.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

[PATCH v3 1/5] btrfs-progs: introduce framework to check kernel supported features

[Anand Jain]

In the newer kernel, supported kernel features can be known from
  /sys/fs/btrfs/features
however this interface was introduced only after 3.14, and most the
incompatible FS features were introduce before 3.14.

This patch proposes to maintain kernel version against the feature list,
and so that will be the minimum kernel version needed to use the feature.

Further, for features supported later than 3.14 this list can still be
updated, so it serves as a repository which can be displayed for easy
reference.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
v3: Mike pointed out that mixed-bg was from version 2.6.37, update it
v2: Check for condition that what happens when we fail to read kernel
    version. Now the code will fail back to use the default as set by
    the progs.

 utils.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----
 utils.h |  1 +
 2 files changed, 76 insertions(+), 5 deletions(-)

diff --git a/utils.c b/utils.c
index b754686..cc0bdfb 100644
--- a/utils.c
+++ b/utils.c
@@ -32,10 +32,12 @@
 #include <linux/loop.h>
 #include <linux/major.h>
 #include <linux/kdev_t.h>
+#include <linux/version.h>
 #include <limits.h>
 #include <blkid/blkid.h>
 #include <sys/vfs.h>
 #include <sys/statfs.h>
+#include <sys/utsname.h>
 #include <linux/magic.h>
 
 #include "kerncompat.h"
@@ -567,21 +569,28 @@ out:
 	return ret;
 }
 
+/*
+ * min_ker_ver: update with minimum kernel version at which the feature
+ * was integrated into the mainline. For the transit period, that is
+ * feature not yet in mainline but in mailing list and for testing,
+ * please use "0.0" to indicate the same.
+ */
 static const struct btrfs_fs_feature {
 	const char *name;
 	u64 flag;
 	const char *desc;
+	const char *min_ker_ver;
 } mkfs_features[] = {
 	{ "mixed-bg", BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS,
-		"mixed data and metadata block groups" },
+		"mixed data and metadata block groups", "2.7.37"},
 	{ "extref", BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF,
-		"increased hardlink limit per file to 65536" },
+		"increased hardlink limit per file to 65536", "3.7"},
 	{ "raid56", BTRFS_FEATURE_INCOMPAT_RAID56,
-		"raid56 extended format" },
+		"raid56 extended format", "3.9"},
 	{ "skinny-metadata", BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA,
-		"reduced-size metadata extent refs" },
+		"reduced-size metadata extent refs", "3.10"},
 	{ "no-holes", BTRFS_FEATURE_INCOMPAT_NO_HOLES,
-		"no explicit hole extents for files" },
+		"no explicit hole extents for files", "3.14"},
 	/* Keep this one last */
 	{ "list-all", BTRFS_FEATURE_LIST_ALL, NULL }
 };
@@ -3077,3 +3086,64 @@ unsigned int get_unit_mode_from_arg(int *argc, char *argv[], int df_mode)
 
 	return unit_mode;
 }
+
+static int version_to_code(char *v)
+{
+	int i = 0;
+	char *b[3] = {NULL};
+	char *save_b = NULL;
+
+	for (b[i] = strtok_r(v, ".", &save_b);
+		b[i] != NULL;
+		b[i] = strtok_r(NULL, ".", &save_b))
+		i++;
+
+	if (b[2] == NULL)
+		return KERNEL_VERSION(atoi(b[0]), atoi(b[1]), 0);
+	else
+		return KERNEL_VERSION(atoi(b[0]), atoi(b[1]), atoi(b[2]));
+
+}
+
+static int get_kernel_code()
+{
+	int ret;
+	struct utsname utsbuf;
+	char *version;
+
+	ret = uname(&utsbuf);
+	if (ret)
+		return -ret;
+
+	if (!strlen(utsbuf.release))
+		return -EINVAL;
+
+	version = strtok(utsbuf.release, "-");
+
+	return version_to_code(version);
+}
+
+u64 btrfs_features_allowed_by_kernel(void)
+{
+	int i;
+	int local_kernel_code = get_kernel_code();
+	u64 features = 0;
+
+	/*
+	 * When system did not provide the kernel version then just
+	 * return 0, the caller has to depend on the intelligence as
+	 * per btrfs-progs version
+	 */
+	if (local_kernel_code <= 0)
+		return 0;
+
+	for (i = 0; i < ARRAY_SIZE(mkfs_features) - 1; i++) {
+		char *ver = strdup(mkfs_features[i].min_ker_ver);
+
+		if (local_kernel_code >= version_to_code(ver))
+			features |= mkfs_features[i].flag;
+
+		free(ver);
+	}
+	return (features);
+}
diff --git a/utils.h b/utils.h
index 192f3d1..9044643 100644
--- a/utils.h
+++ b/utils.h
@@ -104,6 +104,7 @@ void btrfs_list_all_fs_features(u64 mask_disallowed);
 char* btrfs_parse_fs_features(char *namelist, u64 *flags);
 void btrfs_process_fs_features(u64 flags);
 void btrfs_parse_features_to_string(char *buf, u64 flags);
+u64 btrfs_features_allowed_by_kernel(void);
 
 struct btrfs_mkfs_config {
 	char *label;
-- 
2.6.2

Re: [PATCH v2 1/5] btrfs-progs: introduce framework to check kernel supported features

[David Sterba]

On Tue, Nov 24, 2015 at 03:21:19PM -0500, Austin S Hemmelgarn wrote:
> > I think you mean 2.6.37 here.
> > 67377734fd24c3 "Btrfs: add support for mixed data+metadata block groups"
> This brings up a rather important question:
> Should compat-X.Y mean features that were considered usable in that 
> version, or everything that version offered?  I understand wanting 
> consistency with the kernel versions, but we shouldn't be creating 
> filesystems that we know will break on the specified kernel even if it 
> is mountable on it.

IMO compat refers to the compatibility feature bits so it's whether the
filesystem is mountable on a given version. Usability can be subjective.
I assume the kernel versions in wide use match some of the long term
branches. If it's k.org, we can submit the fixes and distros update
their long term branches.

A table of "is the feature usable" would be interesting but I think it's
for wiki.

Re: [PATCH v2 1/5] btrfs-progs: introduce framework to check kernel supported features

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1840 bytes --]

On 2015-11-26 12:38, David Sterba wrote:
> On Tue, Nov 24, 2015 at 03:21:19PM -0500, Austin S Hemmelgarn wrote:
>>> I think you mean 2.6.37 here.
>>> 67377734fd24c3 "Btrfs: add support for mixed data+metadata block groups"
>> This brings up a rather important question:
>> Should compat-X.Y mean features that were considered usable in that
>> version, or everything that version offered?  I understand wanting
>> consistency with the kernel versions, but we shouldn't be creating
>> filesystems that we know will break on the specified kernel even if it
>> is mountable on it.
>
> IMO compat refers to the compatibility feature bits so it's whether the
> filesystem is mountable on a given version. Usability can be subjective.
> I assume the kernel versions in wide use match some of the long term
> branches. If it's k.org, we can submit the fixes and distros update
> their long term branches.
My point was that if we know that there's a significant chance of either 
data corruption or a kernel crash when using a given feature with a 
given kernel, we should not turn on that feature for that kernel.  For 
every other project I've ever seen, compatible means that you can be 
fairly certain that it won't crash, and won't destroy data.  There is no 
excuse for knowingly making filesystems that will break the system.

Also, expecting the distros to keep up with development given the pace 
at which BTRFS is moving is not all that realistic.  Secondarily, at 
least Ubuntu has a habit of picking kernel versions that aren't marked 
LTS by kernel.org.
>
> A table of "is the feature usable" would be interesting but I think it's
> for wiki.
I'd say it would be a lot more helpful in the manpage than in the wiki 
(if you're reprovisioning a system, you don't necessarily have access to 
the wiki).


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Liu Bo]

On Mon, Nov 23, 2015 at 06:56:09PM +0100, David Sterba wrote:
> On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
> > Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
> > be compatible w any set of older/newer kernels.
> > 
> > So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
> > skinny-metadata even if the running kernel does not supports it, and
> > so the mount fails on the running.
> 
> So the default behaviour of mkfs will try to best guess the feature set
> of currently running kernel. I think this is is the most common scenario
> and justifies the change in default behaviours.
> 
> For the other cases I'd like to introduce some human-readable shortcuts
> to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
> options supported by the unpatched mainline kernel of version 3.2. This
> would be present for all version, regardless if there was a change in the
> options or not.
> 
> Similarly for convenience, add 'running' that would pick the options
> from running kernel but will be explicit.
> 
> A remaining option should override the 'running' behaviour and pick the
> latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
> name is yet to be determined.
> 
> > Here in this set of patches will make sure the progs understands the
> > kernel supported features.
> > 
> > So in this patch, checks if sysfs tells whether the feature is
> > supported if not, then it will relay on static kernel version which
> > provided that feature (skinny-metadata here in this example), next
> > if for some reason the running kernel does not provide the kernel
> > version, then it will fall back to the original method to enable
> > the feature with a hope that kernel will support it.
> > 
> > Also the last patch adds a warning when we fail to read either
> > sysfs features or the running kernel version.
> 
> Your patchset is a good start, the additional options I've described can
> be added on top of that. We might need to switch the version
> representation from string to KERNEL_VERSION but that's an
> implementation detail.

Depending on sysfs is stable but depending on kernel version may be not,
we may have a distro kernel which backports some incompat features from
upstream, then we have to decide based on sysfs interface.

However, this brings another problems, for very old kernels, they don't
have sysfs, do you have any suggestions for that?

Thanks,

-liubo

> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Qu Wenruo]

Liu Bo wrote on 2015/12/03 17:44 -0800:
> On Mon, Nov 23, 2015 at 06:56:09PM +0100, David Sterba wrote:
>> On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
>>> Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
>>> be compatible w any set of older/newer kernels.
>>>
>>> So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
>>> skinny-metadata even if the running kernel does not supports it, and
>>> so the mount fails on the running.
>>
>> So the default behaviour of mkfs will try to best guess the feature set
>> of currently running kernel. I think this is is the most common scenario
>> and justifies the change in default behaviours.
>>
>> For the other cases I'd like to introduce some human-readable shortcuts
>> to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
>> options supported by the unpatched mainline kernel of version 3.2. This
>> would be present for all version, regardless if there was a change in the
>> options or not.
>>
>> Similarly for convenience, add 'running' that would pick the options
>> from running kernel but will be explicit.
>>
>> A remaining option should override the 'running' behaviour and pick the
>> latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
>> name is yet to be determined.
>>
>>> Here in this set of patches will make sure the progs understands the
>>> kernel supported features.
>>>
>>> So in this patch, checks if sysfs tells whether the feature is
>>> supported if not, then it will relay on static kernel version which
>>> provided that feature (skinny-metadata here in this example), next
>>> if for some reason the running kernel does not provide the kernel
>>> version, then it will fall back to the original method to enable
>>> the feature with a hope that kernel will support it.
>>>
>>> Also the last patch adds a warning when we fail to read either
>>> sysfs features or the running kernel version.
>>
>> Your patchset is a good start, the additional options I've described can
>> be added on top of that. We might need to switch the version
>> representation from string to KERNEL_VERSION but that's an
>> implementation detail.
>
> Depending on sysfs is stable but depending on kernel version may be not,
> we may have a distro kernel which backports some incompat features from
> upstream, then we have to decide based on sysfs interface.

+1.

Although sysfs does not always show up even for supported kernel, e.g 
btrfs modules is not loaded after boot.
So we need to consider twice before choosing a fallback method.

>
> However, this brings another problems, for very old kernels, they don't
> have sysfs, do you have any suggestions for that?

Other fs, like xfs/ext* doesn't even have sysfs feature interface, only 
release announcement mentioning default behavior change.
And I don't see many users complaining about it.

Here is the example of xfsprogs changed its default feature recently:
In 10th, June, 2015, xfsprogs v3.2.3 is released, with new default 
feature of enabling CRC for fs.
The first supported kernel is 3.15, which is release in 8th Jun, 2014.
Almost one year ago.

On the other hand, the sysfs feature is introduced at the end of year 2013.
It's already over 2 years.

So just forgot the extra minor case of super old kernel would be good 
enough.

Thanks,
Qu

>
> Thanks,
>
> -liubo
>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Liu Bo]

On Fri, Dec 04, 2015 at 10:08:35AM +0800, Qu Wenruo wrote:
> 
> 
> Liu Bo wrote on 2015/12/03 17:44 -0800:
> >On Mon, Nov 23, 2015 at 06:56:09PM +0100, David Sterba wrote:
> >>On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
> >>>Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
> >>>be compatible w any set of older/newer kernels.
> >>>
> >>>So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
> >>>skinny-metadata even if the running kernel does not supports it, and
> >>>so the mount fails on the running.
> >>
> >>So the default behaviour of mkfs will try to best guess the feature set
> >>of currently running kernel. I think this is is the most common scenario
> >>and justifies the change in default behaviours.
> >>
> >>For the other cases I'd like to introduce some human-readable shortcuts
> >>to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
> >>options supported by the unpatched mainline kernel of version 3.2. This
> >>would be present for all version, regardless if there was a change in the
> >>options or not.
> >>
> >>Similarly for convenience, add 'running' that would pick the options
> >>from running kernel but will be explicit.
> >>
> >>A remaining option should override the 'running' behaviour and pick the
> >>latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
> >>name is yet to be determined.
> >>
> >>>Here in this set of patches will make sure the progs understands the
> >>>kernel supported features.
> >>>
> >>>So in this patch, checks if sysfs tells whether the feature is
> >>>supported if not, then it will relay on static kernel version which
> >>>provided that feature (skinny-metadata here in this example), next
> >>>if for some reason the running kernel does not provide the kernel
> >>>version, then it will fall back to the original method to enable
> >>>the feature with a hope that kernel will support it.
> >>>
> >>>Also the last patch adds a warning when we fail to read either
> >>>sysfs features or the running kernel version.
> >>
> >>Your patchset is a good start, the additional options I've described can
> >>be added on top of that. We might need to switch the version
> >>representation from string to KERNEL_VERSION but that's an
> >>implementation detail.
> >
> >Depending on sysfs is stable but depending on kernel version may be not,
> >we may have a distro kernel which backports some incompat features from
> >upstream, then we have to decide based on sysfs interface.
> 
> +1.
> 
> Although sysfs does not always show up even for supported kernel, e.g btrfs
> modules is not loaded after boot.
> So we need to consider twice before choosing a fallback method.
> 
> >
> >However, this brings another problems, for very old kernels, they don't
> >have sysfs, do you have any suggestions for that?
> 
> Other fs, like xfs/ext* doesn't even have sysfs feature interface, only
> release announcement mentioning default behavior change.
> And I don't see many users complaining about it.
> 
> Here is the example of xfsprogs changed its default feature recently:
> In 10th, June, 2015, xfsprogs v3.2.3 is released, with new default feature
> of enabling CRC for fs.
> The first supported kernel is 3.15, which is release in 8th Jun, 2014.
> Almost one year ago.

It's the same thing, if you use a earlier version(before v5) xfs and a
v5 xfsprogs, you are not going to mount it.

> 
> On the other hand, the sysfs feature is introduced at the end of year 2013.
> It's already over 2 years.
> 
> So just forgot the extra minor case of super old kernel would be good
> enough.

Sorry we're not able to do that since most users won't keep up upgrading their
kernels to the latest one, instead they use the stable one they think.

The fact is that btrfs has way more incompatible features than either ext4 or xfs,
and no complain on ext4/xfs from them won't solve our btrfs issue anyway.

The problem is much more serious for enterprise users which are sort of
conservative, they would backport what they need, if they use
btrfs they will experience the painful things.

There're plenty of fixes for progs code, people needs stabler recovery
tools rather than new features they may not use.

So we'd like to have a univeral progs code for old kernels.

Thanks,

-liubo

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Qu Wenruo]

Liu Bo wrote on 2015/12/03 18:53 -0800:
> On Fri, Dec 04, 2015 at 10:08:35AM +0800, Qu Wenruo wrote:
>>
>>
>> Liu Bo wrote on 2015/12/03 17:44 -0800:
>>> On Mon, Nov 23, 2015 at 06:56:09PM +0100, David Sterba wrote:
>>>> On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
>>>>> Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
>>>>> be compatible w any set of older/newer kernels.
>>>>>
>>>>> So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
>>>>> skinny-metadata even if the running kernel does not supports it, and
>>>>> so the mount fails on the running.
>>>>
>>>> So the default behaviour of mkfs will try to best guess the feature set
>>>> of currently running kernel. I think this is is the most common scenario
>>>> and justifies the change in default behaviours.
>>>>
>>>> For the other cases I'd like to introduce some human-readable shortcuts
>>>> to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
>>>> options supported by the unpatched mainline kernel of version 3.2. This
>>>> would be present for all version, regardless if there was a change in the
>>>> options or not.
>>>>
>>>> Similarly for convenience, add 'running' that would pick the options
>>> >from running kernel but will be explicit.
>>>>
>>>> A remaining option should override the 'running' behaviour and pick the
>>>> latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
>>>> name is yet to be determined.
>>>>
>>>>> Here in this set of patches will make sure the progs understands the
>>>>> kernel supported features.
>>>>>
>>>>> So in this patch, checks if sysfs tells whether the feature is
>>>>> supported if not, then it will relay on static kernel version which
>>>>> provided that feature (skinny-metadata here in this example), next
>>>>> if for some reason the running kernel does not provide the kernel
>>>>> version, then it will fall back to the original method to enable
>>>>> the feature with a hope that kernel will support it.
>>>>>
>>>>> Also the last patch adds a warning when we fail to read either
>>>>> sysfs features or the running kernel version.
>>>>
>>>> Your patchset is a good start, the additional options I've described can
>>>> be added on top of that. We might need to switch the version
>>>> representation from string to KERNEL_VERSION but that's an
>>>> implementation detail.
>>>
>>> Depending on sysfs is stable but depending on kernel version may be not,
>>> we may have a distro kernel which backports some incompat features from
>>> upstream, then we have to decide based on sysfs interface.
>>
>> +1.
>>
>> Although sysfs does not always show up even for supported kernel, e.g btrfs
>> modules is not loaded after boot.
>> So we need to consider twice before choosing a fallback method.
>>
>>>
>>> However, this brings another problems, for very old kernels, they don't
>>> have sysfs, do you have any suggestions for that?
>>
>> Other fs, like xfs/ext* doesn't even have sysfs feature interface, only
>> release announcement mentioning default behavior change.
>> And I don't see many users complaining about it.
>>
>> Here is the example of xfsprogs changed its default feature recently:
>> In 10th, June, 2015, xfsprogs v3.2.3 is released, with new default feature
>> of enabling CRC for fs.
>> The first supported kernel is 3.15, which is release in 8th Jun, 2014.
>> Almost one year ago.
>
> It's the same thing, if you use a earlier version(before v5) xfs and a
> v5 xfsprogs, you are not going to mount it.
>
>>
>> On the other hand, the sysfs feature is introduced at the end of year 2013.
>> It's already over 2 years.
>>
>> So just forgot the extra minor case of super old kernel would be good
>> enough.
>
> Sorry we're not able to do that since most users won't keep up upgrading their
> kernels to the latest one, instead they use the stable one they think.
>
> The fact is that btrfs has way more incompatible features than either ext4 or xfs,
> and no complain on ext4/xfs from them won't solve our btrfs issue anyway.
>
> The problem is much more serious for enterprise users which are sort of
> conservative, they would backport what they need, if they use
> btrfs they will experience the painful things.

Only if enterprise really think btrfs is stable enough.
For this point, xfs is considered more stable than btrfs, but v5 xfs 
recent change doesn't introduce such facility to do that compatibility 
check in xfsprogs.

>
> There're plenty of fixes for progs code, people needs stabler recovery
> tools rather than new features they may not use.
>
> So we'd like to have a univeral progs code for old kernels.

Overall, btrfs is considered as a fast-moving and not that stable fs (at 
least not as stable as ext4/xfs).
And users are always encourages to use latest kernel for these reason.

Shouldn't we do such thing when btrfs is stable enough?

Thanks,
Qu

>
> Thanks,
>
> -liubo
>
>

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[David Sterba]

On Fri, Dec 04, 2015 at 10:08:35AM +0800, Qu Wenruo wrote:
> Liu Bo wrote on 2015/12/03 17:44 -0800:
> > On Mon, Nov 23, 2015 at 06:56:09PM +0100, David Sterba wrote:
> >> On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
> >>> Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
> >>> be compatible w any set of older/newer kernels.
> >>>
> >>> So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
> >>> skinny-metadata even if the running kernel does not supports it, and
> >>> so the mount fails on the running.
> >>
> >> So the default behaviour of mkfs will try to best guess the feature set
> >> of currently running kernel. I think this is is the most common scenario
> >> and justifies the change in default behaviours.
> >>
> >> For the other cases I'd like to introduce some human-readable shortcuts
> >> to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
> >> options supported by the unpatched mainline kernel of version 3.2. This
> >> would be present for all version, regardless if there was a change in the
> >> options or not.
> >>
> >> Similarly for convenience, add 'running' that would pick the options
> >> from running kernel but will be explicit.
> >>
> >> A remaining option should override the 'running' behaviour and pick the
> >> latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
> >> name is yet to be determined.
> >>
> >>> Here in this set of patches will make sure the progs understands the
> >>> kernel supported features.
> >>>
> >>> So in this patch, checks if sysfs tells whether the feature is
> >>> supported if not, then it will relay on static kernel version which
> >>> provided that feature (skinny-metadata here in this example), next
> >>> if for some reason the running kernel does not provide the kernel
> >>> version, then it will fall back to the original method to enable
> >>> the feature with a hope that kernel will support it.
> >>>
> >>> Also the last patch adds a warning when we fail to read either
> >>> sysfs features or the running kernel version.
> >>
> >> Your patchset is a good start, the additional options I've described can
> >> be added on top of that. We might need to switch the version
> >> representation from string to KERNEL_VERSION but that's an
> >> implementation detail.
> >
> > Depending on sysfs is stable but depending on kernel version may be not,
> > we may have a distro kernel which backports some incompat features from
> > upstream, then we have to decide based on sysfs interface.
> 
> +1.
> 
> Although sysfs does not always show up even for supported kernel, e.g 
> btrfs modules is not loaded after boot.
> So we need to consider twice before choosing a fallback method.

There are several factors that we have to take into account for the
default behaviour and fallback. I'm close to a final proposal yet missed
the possibility of unloaded module that would remove the access to
sysfs, as you point out.

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Liu Bo]

On Fri, Dec 04, 2015 at 11:57:55AM +0800, Qu Wenruo wrote:
> 
> 
> Liu Bo wrote on 2015/12/03 18:53 -0800:
> >On Fri, Dec 04, 2015 at 10:08:35AM +0800, Qu Wenruo wrote:
> >>
> >>
> >>Liu Bo wrote on 2015/12/03 17:44 -0800:
> >>>On Mon, Nov 23, 2015 at 06:56:09PM +0100, David Sterba wrote:
> >>>>On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
> >>>>>Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
> >>>>>be compatible w any set of older/newer kernels.
> >>>>>
> >>>>>So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
> >>>>>skinny-metadata even if the running kernel does not supports it, and
> >>>>>so the mount fails on the running.
> >>>>
> >>>>So the default behaviour of mkfs will try to best guess the feature set
> >>>>of currently running kernel. I think this is is the most common scenario
> >>>>and justifies the change in default behaviours.
> >>>>
> >>>>For the other cases I'd like to introduce some human-readable shortcuts
> >>>>to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
> >>>>options supported by the unpatched mainline kernel of version 3.2. This
> >>>>would be present for all version, regardless if there was a change in the
> >>>>options or not.
> >>>>
> >>>>Similarly for convenience, add 'running' that would pick the options
> >>>>from running kernel but will be explicit.
> >>>>
> >>>>A remaining option should override the 'running' behaviour and pick the
> >>>>latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
> >>>>name is yet to be determined.
> >>>>
> >>>>>Here in this set of patches will make sure the progs understands the
> >>>>>kernel supported features.
> >>>>>
> >>>>>So in this patch, checks if sysfs tells whether the feature is
> >>>>>supported if not, then it will relay on static kernel version which
> >>>>>provided that feature (skinny-metadata here in this example), next
> >>>>>if for some reason the running kernel does not provide the kernel
> >>>>>version, then it will fall back to the original method to enable
> >>>>>the feature with a hope that kernel will support it.
> >>>>>
> >>>>>Also the last patch adds a warning when we fail to read either
> >>>>>sysfs features or the running kernel version.
> >>>>
> >>>>Your patchset is a good start, the additional options I've described can
> >>>>be added on top of that. We might need to switch the version
> >>>>representation from string to KERNEL_VERSION but that's an
> >>>>implementation detail.
> >>>
> >>>Depending on sysfs is stable but depending on kernel version may be not,
> >>>we may have a distro kernel which backports some incompat features from
> >>>upstream, then we have to decide based on sysfs interface.
> >>
> >>+1.
> >>
> >>Although sysfs does not always show up even for supported kernel, e.g btrfs
> >>modules is not loaded after boot.
> >>So we need to consider twice before choosing a fallback method.
> >>
> >>>
> >>>However, this brings another problems, for very old kernels, they don't
> >>>have sysfs, do you have any suggestions for that?
> >>
> >>Other fs, like xfs/ext* doesn't even have sysfs feature interface, only
> >>release announcement mentioning default behavior change.
> >>And I don't see many users complaining about it.
> >>
> >>Here is the example of xfsprogs changed its default feature recently:
> >>In 10th, June, 2015, xfsprogs v3.2.3 is released, with new default feature
> >>of enabling CRC for fs.
> >>The first supported kernel is 3.15, which is release in 8th Jun, 2014.
> >>Almost one year ago.
> >
> >It's the same thing, if you use a earlier version(before v5) xfs and a
> >v5 xfsprogs, you are not going to mount it.
> >
> >>
> >>On the other hand, the sysfs feature is introduced at the end of year 2013.
> >>It's already over 2 years.
> >>
> >>So just forgot the extra minor case of super old kernel would be good
> >>enough.
> >
> >Sorry we're not able to do that since most users won't keep up upgrading their
> >kernels to the latest one, instead they use the stable one they think.
> >
> >The fact is that btrfs has way more incompatible features than either ext4 or xfs,
> >and no complain on ext4/xfs from them won't solve our btrfs issue anyway.
> >
> >The problem is much more serious for enterprise users which are sort of
> >conservative, they would backport what they need, if they use
> >btrfs they will experience the painful things.
> 
> Only if enterprise really think btrfs is stable enough.
> For this point, xfs is considered more stable than btrfs, but v5 xfs recent
> change doesn't introduce such facility to do that compatibility check in
> xfsprogs.

Xfs on kernel side obviously refuses to mount if you create an incompatible
feature with a recent xfsprogs but try to mount it with older kernel.

STATIC int
xfs_mount_validate_sb()
{
	...
	if (xfs_sb_has_incompat_feature(sbp, XFS_SB_FEAT_INCOMPAT_UNKNOWN)) {         
		xfs_warn(mp,                                             
		"Superblock has unknown incompatible features (0x%x) enabled.",                  
                                 (sbp->sb_features_incompat & XFS_SB_FEAT_INCOMPAT_UNKNOWN));  
		xfs_warn(mp,
		"Filesystem can not be safely mounted by this kernel.");                         
		return -EINVAL;                                          
	}                                                                
	...
}


And this did happen:
http://comments.gmane.org/gmane.comp.file-systems.xfs.general/69514

> 
> >
> >There're plenty of fixes for progs code, people needs stabler recovery
> >tools rather than new features they may not use.
> >
> >So we'd like to have a univeral progs code for old kernels.
> 
> Overall, btrfs is considered as a fast-moving and not that stable fs (at
> least not as stable as ext4/xfs).
> And users are always encourages to use latest kernel for these reason.
> 
> Shouldn't we do such thing when btrfs is stable enough?

That's not true, we use a stable kernel instead, which has a stable btrfs,
it's not latest kernel but stable kernel that is more suitable for production use.

We offer btrfs support as the customers requested, I asked for this
because our customer needs btrfs-progs to be wiser on this feature selecting stuff.

Thanks,

-liubo

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

David,

> the possibility of unloaded module that would remove the access to
> sysfs, as you point out.

  Kindly note, the patch below made /dev/btrfs-control a static node,

-----
commit 578454ff7eab61d13a26b568f99a89a2c9edc881
Author: Kay Sievers <kay.sievers@vrfy.org>
Date: Thu May 20 18:07:20 2010 +0200

driver core: add devname module aliases to allow module on-demand 
auto-loading
------

  And here the function, check_or_load_btrfs_ko(), in the PATCH v2 2/5,
  will take care of this problem.

--------
+
+int check_or_load_btrfs_ko()
+{
+	int fd;
+
+	/*
+	 * open will load btrfs kernel module if its not loaded,
+	 * and if the kernel has CONFIG auto load set?
+	 */
+	fd = open("/dev/btrfs-control", O_RDONLY);
+	if (fd < 0)
+		return -errno;
+
+	close(fd);
+	return 0;
+}
+
--------

  Since now static minor number for /dev/btrfs-control is mapped to
  the btrfs kernel module, it will ensure btrfs is loaded when
  /dev/btrfs-control is accessed.

  Further, /dev/btrfs-control node is created by udevd, by reading
  the modules.devname which is either supplied/updated by the distro
  or compilation.

  For systems without udev, IMO should run mknod ..btrfs-control
  in their install script which I guess is a must.

--------
# ls -li /dev/btrfs-control
7338 crw-rw---- 1 root disk 10, 234 Dec  5 10:45 /dev/btrfs-control

# cat modules.devname | egrep btrfs
btrfs btrfs-control c10:234

# cat ./include/linux/miscdevice.h | egrep BTRFS
#define BTRFS_MINOR 234
--------

  So IMO this is not a real problem.

Thanks, Anand

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

Hi David,

  This patch isn't integrated, any idea why ? Just a note, if it matters,
  this set has already been integrated into Oracle Linux.

Thanks, Anand


On 11/23/15 20:56, Anand Jain wrote:
> Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
> be compatible w any set of older/newer kernels.
>
> So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
> skinny-metadata even if the running kernel does not supports it, and
> so the mount fails on the running.
>
> Here in this set of patches will make sure the progs understands the
> kernel supported features.
>
> So in this patch, checks if sysfs tells whether the feature is
> supported if not, then it will relay on static kernel version which
> provided that feature (skinny-metadata here in this example), next
> if for some reason the running kernel does not provide the kernel
> version, then it will fall back to the original method to enable
> the feature with a hope that kernel will support it.
>
> Also the last patch adds a warning when we fail to read either
> sysfs features or the running kernel version.
>
> With this I hope all the concerns from the review comments are
> addressed.
>
>
> Anand Jain (5):
>   btrfs-progs: introduce framework to check kernel supported features
>   btrfs-progs: add framework to check features supported by sysfs
>   btrfs-progs: kernel based default features for mkfs
>   btrfs-progs: kernel based default features for btrfs-convert
>   btrfs-progs: add warning when we fail to read sysfs or kernel version
>
>  btrfs-convert.c |  18 ++++++-
>  mkfs.c          |  22 ++++++++-
>  utils.c         | 146 +++++++++++++++++++++++++++++++++++++++++++++++++++-----
>  utils.h         |   2 +
>  4 files changed, 173 insertions(+), 15 deletions(-)
>

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[David Sterba]

On Tue, Nov 08, 2016 at 09:14:39PM +0800, Anand Jain wrote:
>   This patch isn't integrated, any idea why ?

Because it does not cover all usecases. I've committed the preparatory
bits from branch where I've been working on that feature so we can
actually get to the hard parts.

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

(sorry for the delay due to my vacation).

On 11/14/16 20:13, David Sterba wrote:
> On Tue, Nov 08, 2016 at 09:14:39PM +0800, Anand Jain wrote:
>>   This patch isn't integrated, any idea why ?
>
> Because it does not cover all usecases.

  David,

    You have stated one of the use case here
     https://www.spinics.net/lists/linux-btrfs/msg49391.html

    which I have it implemented in these patches
      [PATCH 5/7] btrfs-progs: introduce framework version to features
      [PATCH 6/7] btrfs-progs: add -O comp= option for mkfs.btrfs
      [PATCH 7/7] btrfs-progs: add -O comp= option for btrfs-convert

    was their more ?

> I've committed the preparatory
> bits from branch where I've been working on that feature so we can
> actually get to the hard parts.

  these patch set is a year old as well. are you planning to drop
  these patches ? the clarity is missing. IMO.

Thanks, Anand

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[David Sterba]

On Tue, Nov 22, 2016 at 04:54:58PM +0800, Anand Jain wrote:
> On 11/14/16 20:13, David Sterba wrote:
> > On Tue, Nov 08, 2016 at 09:14:39PM +0800, Anand Jain wrote:
> >>   This patch isn't integrated, any idea why ?
> >
> > Because it does not cover all usecases.
> 
>     You have stated one of the use case here
>      https://www.spinics.net/lists/linux-btrfs/msg49391.html
> 
>     which I have it implemented in these patches
>       [PATCH 5/7] btrfs-progs: introduce framework version to features
>       [PATCH 6/7] btrfs-progs: add -O comp= option for mkfs.btrfs
>       [PATCH 7/7] btrfs-progs: add -O comp= option for btrfs-convert
> 
>     was their more ?

Yeah, it turned out there's more.

> > I've committed the preparatory
> > bits from branch where I've been working on that feature so we can
> > actually get to the hard parts.
> 
>   these patch set is a year old as well. are you planning to drop
>   these patches ? the clarity is missing. IMO.

Preparatory bits are in patches:

  btrfs-progs: mkfs: add names of matching sysfs feature names
  btrfs-progs: mkfs: enhance feature table
  btrfs-progs: mkfs: extend mkfs features with compat, safe and default versions

the core work is not yet done.

The original usecase has been extended to allow for selecting features
that are compatible, ie. can be mounted, safe, ie. are ok wrt The
Status page, and default, ie. automatically selected by mkfs. Additionally to
that, a system-wide preset from a config file in /etc.

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

On 11/22/16 21:16, David Sterba wrote:
> On Tue, Nov 22, 2016 at 04:54:58PM +0800, Anand Jain wrote:
>> On 11/14/16 20:13, David Sterba wrote:
>>> On Tue, Nov 08, 2016 at 09:14:39PM +0800, Anand Jain wrote:
>>>>   This patch isn't integrated, any idea why ?
>>>
>>> Because it does not cover all usecases.
>>
>>     You have stated one of the use case here
>>      https://www.spinics.net/lists/linux-btrfs/msg49391.html
>>
>>     which I have it implemented in these patches
>>       [PATCH 5/7] btrfs-progs: introduce framework version to features
>>       [PATCH 6/7] btrfs-progs: add -O comp= option for mkfs.btrfs
>>       [PATCH 7/7] btrfs-progs: add -O comp= option for btrfs-convert
>>
>>     was their more ?
>
> Yeah, it turned out there's more.
>
>>> I've committed the preparatory
>>> bits from branch where I've been working on that feature so we can
>>> actually get to the hard parts.
>>
>>   these patch set is a year old as well. are you planning to drop
>>   these patches ? the clarity is missing. IMO.
>
> Preparatory bits are in patches:
>
>   btrfs-progs: mkfs: add names of matching sysfs feature names
>   btrfs-progs: mkfs: enhance feature table
>   btrfs-progs: mkfs: extend mkfs features with compat, safe and default versions

  I don't seen these patches in the ML. Looks like it was integrated
  in 4.8.3.

> the core work is not yet done.
>
> The original usecase has been extended to allow for selecting features
> that are compatible, ie. can be mounted, safe, ie. are ok wrt The
> Status page, and default, ie. automatically selected by mkfs.

> Additionally to
> that, a system-wide preset from a config file in /etc.


  Certainly these patches could have been on (cleanup/enhancement)
  top of my patches which was sent a year back.



Thanks. Anand

> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[David Sterba]

On Wed, Nov 23, 2016 at 11:00:49AM +0800, Anand Jain wrote:
> > Preparatory bits are in patches:
> >
> >   btrfs-progs: mkfs: add names of matching sysfs feature names
> >   btrfs-progs: mkfs: enhance feature table
> >   btrfs-progs: mkfs: extend mkfs features with compat, safe and default versions
> 
>   I don't seen these patches in the ML. Looks like it was integrated
>   in 4.8.3.

Yeah the patches and many other from me haven't been sent to the
mailinglist, unless I need input regarding usecase. Sending everything
would increase work for me and most of the patches are boring cleanups
anyway.

> > the core work is not yet done.
> >
> > The original usecase has been extended to allow for selecting features
> > that are compatible, ie. can be mounted, safe, ie. are ok wrt The
> > Status page, and default, ie. automatically selected by mkfs.
> 
> > Additionally to
> > that, a system-wide preset from a config file in /etc.
> 
> 
>   Certainly these patches could have been on (cleanup/enhancement)
>   top of my patches which was sent a year back.

Well, back then I started on top of your patches and found out that I'd
have to rework too much code so it was easier to start from scratch.