* [PATCH] btrfs-progs: btrfs-convert: Add larger device support
@ 2017-04-27 10:05 Lakshmipathi.G
[not found] ` <3907c93a-c525-5874-8d14-594f66383261@cn.fujitsu.com>
2017-05-02 15:06 ` David Sterba
0 siblings, 2 replies; 9+ messages in thread
From: Lakshmipathi.G @ 2017-04-27 10:05 UTC (permalink / raw)
To: quwenruo, linux-btrfs, dsterba
Bug: Fail to convert 22TB EXT4 to BTRFS
https://bugzilla.kernel.org/show_bug.cgi?id=194795
Signed-off-by: Lakshmipathi.G <Lakshmipathi.G@giis.co.in>
---
convert/source-ext2.c | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/convert/source-ext2.c b/convert/source-ext2.c
index 1b0576b..275cb89 100644
--- a/convert/source-ext2.c
+++ b/convert/source-ext2.c
@@ -34,8 +34,9 @@ static int ext2_open_fs(struct btrfs_convert_context *cctx, const char *name)
ext2_filsys ext2_fs;
ext2_ino_t ino;
u32 ro_feature;
+ int open_flag = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS;
- ret = ext2fs_open(name, 0, 0, 0, unix_io_manager, &ext2_fs);
+ ret = ext2fs_open(name, open_flag, 0, 0, unix_io_manager, &ext2_fs);
if (ret) {
fprintf(stderr, "ext2fs_open: %s\n", error_message(ret));
return -1;
@@ -148,7 +149,7 @@ static int ext2_read_used_space(struct btrfs_convert_context *cctx)
return -ENOMEM;
for (i = 0; i < fs->group_desc_count; i++) {
- ret = ext2fs_get_block_bitmap_range(fs->block_map, blk_itr,
+ ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr,
block_nbytes * 8, block_bitmap);
if (ret) {
error("fail to get bitmap from ext2, %s",
@@ -353,7 +354,7 @@ static int ext2_create_symlink(struct btrfs_trans_handle *trans,
int ret;
char *pathname;
u64 inode_size = btrfs_stack_inode_size(btrfs_inode);
- if (ext2fs_inode_data_blocks(ext2_fs, ext2_inode)) {
+ if (ext2fs_inode_data_blocks2(ext2_fs, ext2_inode)) {
btrfs_set_stack_inode_size(btrfs_inode, inode_size + 1);
ret = ext2_create_file_extents(trans, root, objectid,
btrfs_inode, ext2_fs, ext2_ino,
@@ -627,9 +628,9 @@ static int ext2_copy_extended_attrs(struct btrfs_trans_handle *trans,
ret = -ENOMEM;
goto out;
}
- err = ext2fs_read_ext_attr(ext2_fs, ext2_inode->i_file_acl, buffer);
+ err = ext2fs_read_ext_attr2(ext2_fs, ext2_inode->i_file_acl, buffer);
if (err) {
- fprintf(stderr, "ext2fs_read_ext_attr: %s\n",
+ fprintf(stderr, "ext2fs_read_ext_attr2: %s\n",
error_message(err));
ret = -1;
goto out;
--
2.7.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support
[not found] ` <016db1e7-c9d5-a8ec-1d64-cbd08f14503d@cn.fujitsu.com>
@ 2017-04-28 5:55 ` Lakshmipathi.G
0 siblings, 0 replies; 9+ messages in thread
From: Lakshmipathi.G @ 2017-04-28 5:55 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs, dsterba
Hi Qu,
Seems like its a known issue with older bitmaps.
http://linux-ext4.vger.kernel.narkive.com/wkNbJe0b/patch-debugfs-open-with-ext2-flag-64bits
Cheers.
Lakshmipathi.G
On Fri, Apr 28, 2017 at 09:48:10AM +0800, Qu Wenruo wrote:
>
>
> At 04/28/2017 09:27 AM, Qu Wenruo wrote:
> >At 04/27/2017 06:05 PM, Lakshmipathi.G wrote:
> >>Bug: Fail to convert 22TB EXT4 to BTRFS
> >>https://bugzilla.kernel.org/show_bug.cgi?id=194795
> >
> >Thanks for fixing it.
> >
> >While testing manually using LVM thin provision, although the convert
> >itself works, the converted image has something wrong on its size.
> >
> >The converted image is only 4T sized, not the correct size (20T) of my
> >original setup.
> >
> >This may not be the problem of your patch.
> >I'll double check the original code to ensure it works.
>
> It's the method we get blocks count wrong.
>
> We should do it just as dumpe2fs does:
> ------
> static __u64 e2p_blocks_count(struct ext2_super_block *super)
> {
> return super->s_blocks_count |
> (ext2fs_has_feature_64bit(super) ?
> (__u64) super->s_blocks_count_hi << 32 : 0);
> }
> ------
>
> Current super->s_blocks_count can only handle U32_MAX, higher bits are in
> super->s_blocks_count_hi.
>
> Thanks,
> Qu
>
> >
> >Thanks,
> >Qu
> >
> >>
> >>Signed-off-by: Lakshmipathi.G <Lakshmipathi.G@giis.co.in>
> >>---
> >> convert/source-ext2.c | 11 ++++++-----
> >> 1 file changed, 6 insertions(+), 5 deletions(-)
> >>
> >>diff --git a/convert/source-ext2.c b/convert/source-ext2.c
> >>index 1b0576b..275cb89 100644
> >>--- a/convert/source-ext2.c
> >>+++ b/convert/source-ext2.c
> >>@@ -34,8 +34,9 @@ static int ext2_open_fs(struct btrfs_convert_context
> >>*cctx, const char *name)
> >> ext2_filsys ext2_fs;
> >> ext2_ino_t ino;
> >> u32 ro_feature;
> >>+ int open_flag = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS;
> >>- ret = ext2fs_open(name, 0, 0, 0, unix_io_manager, &ext2_fs);
> >>+ ret = ext2fs_open(name, open_flag, 0, 0, unix_io_manager, &ext2_fs);
> >> if (ret) {
> >> fprintf(stderr, "ext2fs_open: %s\n", error_message(ret));
> >> return -1;
> >>@@ -148,7 +149,7 @@ static int ext2_read_used_space(struct
> >>btrfs_convert_context *cctx)
> >> return -ENOMEM;
> >> for (i = 0; i < fs->group_desc_count; i++) {
> >>- ret = ext2fs_get_block_bitmap_range(fs->block_map, blk_itr,
> >>+ ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr,
> >> block_nbytes * 8, block_bitmap);
> >> if (ret) {
> >> error("fail to get bitmap from ext2, %s",
> >>@@ -353,7 +354,7 @@ static int ext2_create_symlink(struct
> >>btrfs_trans_handle *trans,
> >> int ret;
> >> char *pathname;
> >> u64 inode_size = btrfs_stack_inode_size(btrfs_inode);
> >>- if (ext2fs_inode_data_blocks(ext2_fs, ext2_inode)) {
> >>+ if (ext2fs_inode_data_blocks2(ext2_fs, ext2_inode)) {
> >> btrfs_set_stack_inode_size(btrfs_inode, inode_size + 1);
> >> ret = ext2_create_file_extents(trans, root, objectid,
> >> btrfs_inode, ext2_fs, ext2_ino,
> >>@@ -627,9 +628,9 @@ static int ext2_copy_extended_attrs(struct
> >>btrfs_trans_handle *trans,
> >> ret = -ENOMEM;
> >> goto out;
> >> }
> >>- err = ext2fs_read_ext_attr(ext2_fs, ext2_inode->i_file_acl, buffer);
> >>+ err = ext2fs_read_ext_attr2(ext2_fs, ext2_inode->i_file_acl,
> >>buffer);
> >> if (err) {
> >>- fprintf(stderr, "ext2fs_read_ext_attr: %s\n",
> >>+ fprintf(stderr, "ext2fs_read_ext_attr2: %s\n",
> >> error_message(err));
> >> ret = -1;
> >> goto out;
> >>
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support
2017-04-27 10:05 [PATCH] btrfs-progs: btrfs-convert: Add larger device support Lakshmipathi.G
[not found] ` <3907c93a-c525-5874-8d14-594f66383261@cn.fujitsu.com>
@ 2017-05-02 15:06 ` David Sterba
2017-05-02 15:28 ` Lakshmipathi.G
2017-05-02 15:35 ` David Sterba
1 sibling, 2 replies; 9+ messages in thread
From: David Sterba @ 2017-05-02 15:06 UTC (permalink / raw)
To: Lakshmipathi.G; +Cc: quwenruo, linux-btrfs, dsterba
On Thu, Apr 27, 2017 at 03:35:34PM +0530, Lakshmipathi.G wrote:
> Bug: Fail to convert 22TB EXT4 to BTRFS
> https://bugzilla.kernel.org/show_bug.cgi?id=194795
Thanks for working on the bug, the proposed fix looks good to me judging
by the use of extended API. This would need a test though. I'm trying
this on a sparse file, 3TB occupies 50G and is slow to create.
Conversion works and hasn't finished yet, so this is not suitable for
tests run by default.
Please write a more detailed changelog, a reference to bugzilla is not
really sufficient.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support
2017-05-02 15:06 ` David Sterba
@ 2017-05-02 15:28 ` Lakshmipathi.G
2017-05-02 15:35 ` David Sterba
1 sibling, 0 replies; 9+ messages in thread
From: Lakshmipathi.G @ 2017-05-02 15:28 UTC (permalink / raw)
To: dsterba, quwenruo, linux-btrfs
On Tue, May 02, 2017 at 05:06:01PM +0200, David Sterba wrote:
> On Thu, Apr 27, 2017 at 03:35:34PM +0530, Lakshmipathi.G wrote:
> > Bug: Fail to convert 22TB EXT4 to BTRFS
> > https://bugzilla.kernel.org/show_bug.cgi?id=194795
>
> Thanks for working on the bug, the proposed fix looks good to me judging
> by the use of extended API. This would need a test though. I'm trying
> this on a sparse file, 3TB occupies 50G and is slow to create.
> Conversion works and hasn't finished yet, so this is not suitable for
> tests run by default.
>
> Please write a more detailed changelog, a reference to bugzilla is not
> really sufficient.
Sure, will send a patch with better commit description about the bug and
proposed fix.
Cheers.
Lakshmipathi.G
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support
2017-05-02 15:06 ` David Sterba
2017-05-02 15:28 ` Lakshmipathi.G
@ 2017-05-02 15:35 ` David Sterba
2017-05-03 16:38 ` Lakshmipathi.G
1 sibling, 1 reply; 9+ messages in thread
From: David Sterba @ 2017-05-02 15:35 UTC (permalink / raw)
To: dsterba, Lakshmipathi.G, quwenruo, linux-btrfs
On Tue, May 02, 2017 at 05:06:01PM +0200, David Sterba wrote:
> On Thu, Apr 27, 2017 at 03:35:34PM +0530, Lakshmipathi.G wrote:
> > Bug: Fail to convert 22TB EXT4 to BTRFS
> > https://bugzilla.kernel.org/show_bug.cgi?id=194795
>
> Thanks for working on the bug, the proposed fix looks good to me judging
> by the use of extended API. This would need a test though. I'm trying
> this on a sparse file, 3TB occupies 50G and is slow to create.
> Conversion works and hasn't finished yet, so this is not suitable for
> tests run by default.
So it took a few more minutes. I'm testing it again, not on NFS, and
now the creation of empty filesystem took just a few seconds and empty
file is ~160MB. This sounds much better. So please write a test, using
something like
truncate -s3T image
mkfs.ext4 image
mount && write some data
convert && rollback
Thanks. Later we might need to add some mkfs.ext4 option coverage.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support
2017-05-02 15:35 ` David Sterba
@ 2017-05-03 16:38 ` Lakshmipathi.G
2017-05-09 14:16 ` Lakshmipathi.G
0 siblings, 1 reply; 9+ messages in thread
From: Lakshmipathi.G @ 2017-05-03 16:38 UTC (permalink / raw)
To: dsterba, quwenruo, linux-btrfs
> file is ~160MB. This sounds much better. So please write a test, using
> something like
>
> truncate -s3T image
> mkfs.ext4 image
> mount && write some data
> convert && rollback
>
> Thanks. Later we might need to add some mkfs.ext4 option coverage.
Sure, will send the above test script in upcoming days, then look
into mkfs.ext4 options. thanks.
Cheers.
Lakshmipathi.G
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support
2017-05-03 16:38 ` Lakshmipathi.G
@ 2017-05-09 14:16 ` Lakshmipathi.G
2017-05-09 15:41 ` David Sterba
0 siblings, 1 reply; 9+ messages in thread
From: Lakshmipathi.G @ 2017-05-09 14:16 UTC (permalink / raw)
To: dsterba, quwenruo, linux-btrfs
created a test script but it fail to detect mk2fs. Test script 009
also produces:
# ./convert-tests/009-common-inode-flags/test.sh
[TEST/conv] common inode flags test, btrfs defaults
failed: mke2fs -t ext4 -b 4096 -F
/home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.img
Seems like run_check fails to detect mke2fs command. Something wrong
with my setup or issue with run_check?
--
----
Cheers,
Lakshmipathi.G
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support
2017-05-09 14:16 ` Lakshmipathi.G
@ 2017-05-09 15:41 ` David Sterba
2017-05-09 17:15 ` Lakshmipathi.G
0 siblings, 1 reply; 9+ messages in thread
From: David Sterba @ 2017-05-09 15:41 UTC (permalink / raw)
To: Lakshmipathi.G; +Cc: quwenruo, linux-btrfs
On Tue, May 09, 2017 at 07:46:08PM +0530, Lakshmipathi.G wrote:
> created a test script but it fail to detect mk2fs. Test script 009
> also produces:
>
> # ./convert-tests/009-common-inode-flags/test.sh
> [TEST/conv] common inode flags test, btrfs defaults
> failed: mke2fs -t ext4 -b 4096 -F
> /home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.img
>
> Seems like run_check fails to detect mke2fs command. Something wrong
> with my setup or issue with run_check?
Does the mke2fs command exist on your system? We maybe want to use
mkfs.ext4 directly.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support
2017-05-09 15:41 ` David Sterba
@ 2017-05-09 17:15 ` Lakshmipathi.G
0 siblings, 0 replies; 9+ messages in thread
From: Lakshmipathi.G @ 2017-05-09 17:15 UTC (permalink / raw)
To: dsterba, Lakshmipathi.G, quwenruo, linux-btrfs
>
> Does the mke2fs command exist on your system? We maybe want to use
> mkfs.ext4 directly.
Yes, mke2fs is there. (I'll check using mkfs.ext4 directly)
# ./convert-tests/009-common-inode-flags/test.sh
[TEST/conv] common inode flags test, btrfs defaults
failed: mke2fs -t ext4 -b 4096 -F
/home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.img
After failure, I can run failed command manually.
# mke2fs -t ext4 -b 4096 -F
/home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.imgmke2fs 1.42.13
(17-May-2015)
Discarding device blocks: done
Creating filesystem with 131072 4k blocks and 32768 inodes
Filesystem UUID: 659dbb16-ea5e-48e0-8b1a-30474f54c282
Superblock backups stored on blocks:
32768, 98304
Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
--
----
Cheers,
Lakshmipathi.G
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2017-05-09 17:16 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-04-27 10:05 [PATCH] btrfs-progs: btrfs-convert: Add larger device support Lakshmipathi.G
[not found] ` <3907c93a-c525-5874-8d14-594f66383261@cn.fujitsu.com>
[not found] ` <016db1e7-c9d5-a8ec-1d64-cbd08f14503d@cn.fujitsu.com>
2017-04-28 5:55 ` Lakshmipathi.G
2017-05-02 15:06 ` David Sterba
2017-05-02 15:28 ` Lakshmipathi.G
2017-05-02 15:35 ` David Sterba
2017-05-03 16:38 ` Lakshmipathi.G
2017-05-09 14:16 ` Lakshmipathi.G
2017-05-09 15:41 ` David Sterba
2017-05-09 17:15 ` Lakshmipathi.G
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.