btrfs-convert aborts with an assert

btrfs-convert aborts with an assert

[Torbjörn Jansson]

Hello

i tried to do a btrfs-convert of a ext4 filesystem and after a short while 
after starting it i was greeted with:

# btrfs-convert /dev/xxxx
btrfs-convert from btrfs-progs v5.16.2

convert/main.c:1162: do_convert: Assertion `cctx.total_bytes != 0` failed, value 0
btrfs-convert(+0xffb0)[0x557defdabfb0]
btrfs-convert(main+0x6c5)[0x557defdaa125]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f66e1f8bd0a]
btrfs-convert(_start+0x2a)[0x557defdab52a]
Aborted

Any idea whats going on?
Is it a known bug?
Is the btrfs-progs that come with my dist too old?
FYI the ext4 filesystem is a bit large ~10tb of used data on it.

I assume the convert didn't even start in this case and nothing was modified on 
the ext4 filesystem, correct? or?

Re: btrfs-convert aborts with an assert

[Nikolay Borisov]

On 6.06.22 г. 17:03 ч., Torbjörn Jansson wrote:
> Hello
> 
> i tried to do a btrfs-convert of a ext4 filesystem and after a short 
> while after starting it i was greeted with:
> 
> # btrfs-convert /dev/xxxx
> btrfs-convert from btrfs-progs v5.16.2
> 
> convert/main.c:1162: do_convert: Assertion `cctx.total_bytes != 0` 
> failed, value 0
> btrfs-convert(+0xffb0)[0x557defdabfb0]
> btrfs-convert(main+0x6c5)[0x557defdaa125]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f66e1f8bd0a]
> btrfs-convert(_start+0x2a)[0x557defdab52a]
> Aborted
> 
> Any idea whats going on?
> Is it a known bug?
> Is the btrfs-progs that come with my dist too old?
> FYI the ext4 filesystem is a bit large ~10tb of used data on it.
> 
> I assume the convert didn't even start in this case and nothing was 
> modified on the ext4 filesystem, correct? or?
> 

This suggests that btrfs progs is unable to properly read/parse the 
superblock of the ext4 filesystem. So you are right that convert hasn't 
even started in this case so your ext4 is intact.

However, that's strange since for ext* related queries we do use libext2fs.

Re: btrfs-convert aborts with an assert

[Nikolay Borisov]

On 6.06.22 г. 17:03 ч., Torbjörn Jansson wrote:
> Hello
> 
> i tried to do a btrfs-convert of a ext4 filesystem and after a short 
> while after starting it i was greeted with:
> 
> # btrfs-convert /dev/xxxx
> btrfs-convert from btrfs-progs v5.16.2
> 
> convert/main.c:1162: do_convert: Assertion `cctx.total_bytes != 0` 
> failed, value 0
> btrfs-convert(+0xffb0)[0x557defdabfb0]
> btrfs-convert(main+0x6c5)[0x557defdaa125]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f66e1f8bd0a]
> btrfs-convert(_start+0x2a)[0x557defdab52a]
> Aborted
> 
> Any idea whats going on?
> Is it a known bug?
> Is the btrfs-progs that come with my dist too old?
> FYI the ext4 filesystem is a bit large ~10tb of used data on it.
> 
> I assume the convert didn't even start in this case and nothing was 
> modified on the ext4 filesystem, correct? or?
> 

Care too run the following command and share the output:

echo "show_super_stats -h" | debugfs -f /dev/stdin /dev/loop0

change /dev/loop0 to wherever is your ext4 filesyste, however debugfs 
require the fs to be unmounted.

Re: btrfs-convert aborts with an assert

[David Sterba]

On Mon, Jun 06, 2022 at 04:03:33PM +0200, Torbjörn Jansson wrote:
> Hello
> 
> i tried to do a btrfs-convert of a ext4 filesystem and after a short while 
> after starting it i was greeted with:
> 
> # btrfs-convert /dev/xxxx
> btrfs-convert from btrfs-progs v5.16.2
> 
> convert/main.c:1162: do_convert: Assertion `cctx.total_bytes != 0` failed, value 0
> btrfs-convert(+0xffb0)[0x557defdabfb0]
> btrfs-convert(main+0x6c5)[0x557defdaa125]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f66e1f8bd0a]
> btrfs-convert(_start+0x2a)[0x557defdab52a]
> Aborted

Please open an issue on github, thanks.

> Any idea whats going on?
> Is it a known bug?
> Is the btrfs-progs that come with my dist too old?

Two versions back is not that old, but there are bugfixes in each
release, so it's better to stay up to date.

> FYI the ext4 filesystem is a bit large ~10tb of used data on it.

This looks similar to another issue that caused weird checksum errors
(but no problems during conversion).
https://github.com/kdave/btrfs-progs/issues/349 the common point here is
thatt it's 2TB and a multiple in your case.

> I assume the convert didn't even start in this case and nothing was modified on 
> the ext4 filesystem, correct? or?

Yeah, until the conversion is completely finished without errors, the
old ext4 is left intact and the new partial btrfs structures are not
recognized as a valid filesystem.

Re: btrfs-convert aborts with an assert

[Torbjörn Jansson]

On 2022-06-06 16:31, Nikolay Borisov wrote:
> 
> 
> On 6.06.22 г. 17:03 ч., Torbjörn Jansson wrote:
>> Hello
>>
>> i tried to do a btrfs-convert of a ext4 filesystem and after a short while 
>> after starting it i was greeted with:
>>
>> # btrfs-convert /dev/xxxx
>> btrfs-convert from btrfs-progs v5.16.2
>>
>> convert/main.c:1162: do_convert: Assertion `cctx.total_bytes != 0` failed, 
>> value 0
>> btrfs-convert(+0xffb0)[0x557defdabfb0]
>> btrfs-convert(main+0x6c5)[0x557defdaa125]
>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f66e1f8bd0a]
>> btrfs-convert(_start+0x2a)[0x557defdab52a]
>> Aborted
>>
>> Any idea whats going on?
>> Is it a known bug?
>> Is the btrfs-progs that come with my dist too old?
>> FYI the ext4 filesystem is a bit large ~10tb of used data on it.
>>
>> I assume the convert didn't even start in this case and nothing was modified 
>> on the ext4 filesystem, correct? or?
>>
> 
> Care too run the following command and share the output:
> 
> echo "show_super_stats -h" | debugfs -f /dev/stdin /dev/loop0
> 
> change /dev/loop0 to wherever is your ext4 filesyste, however debugfs require 
> the fs to be unmounted.
> 

See output below.

one more thing, when i ran fsck.ext4 -f on the device before trying to run 
btrfs-convert it said things like:
Inode 61475702 extent tree (at level 1) could be shorter.  Optimize<y>? yes
Inode 61477092 extent tree (at level 2) could be narrower.  Optimize<y>? yes
and then a bunch more, all aditonal looked similar and said narrower.


Filesystem volume name:   <none>
Last mounted on:          /mnt/data
Filesystem UUID:          78b93577-0bb5-41a1-89d7-3027cf0b9bc2
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr dir_index filetype meta_bg 
extent 64bit flex_bg casefold sparse_super large_file huge_file dir_nlink 
extra_isize metadata_csum
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              536870912
Block count:              4294967296
Reserved block count:     0
Overhead clusters:        12982806
Free blocks:              2134612618
Free inodes:              536845192
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         4096
Inode blocks per group:   256
First meta block group:   1792
Flex block group size:    16
Filesystem created:       Fri Aug 13 17:27:43 2021
Last mount time:          Sat May 21 13:56:20 2022
Last write time:          Mon Jun  6 15:33:45 2022
Mount count:              0
Maximum mount count:      -1
Last checked:             Mon Jun  6 15:33:45 2022
Check interval:           0 (<none>)
Lifetime writes:          93 TB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      79d63f5e-1fd3-443f-ba05-1a90b7358160
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0xb53baa13
Character encoding:       utf8-12.1
Directories:              138

Re: btrfs-convert aborts with an assert

[Nikolay Borisov]

On 6.06.22 г. 18:00 ч., Torbjörn Jansson wrote:
> On 2022-06-06 16:31, Nikolay Borisov wrote:
>>
>>
>> On 6.06.22 г. 17:03 ч., Torbjörn Jansson wrote:
>>> Hello
>>>
>>> i tried to do a btrfs-convert of a ext4 filesystem and after a short 
>>> while after starting it i was greeted with:
>>>
>>> # btrfs-convert /dev/xxxx
>>> btrfs-convert from btrfs-progs v5.16.2
>>>
>>> convert/main.c:1162: do_convert: Assertion `cctx.total_bytes != 0` 
>>> failed, value 0
>>> btrfs-convert(+0xffb0)[0x557defdabfb0]
>>> btrfs-convert(main+0x6c5)[0x557defdaa125]
>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f66e1f8bd0a]
>>> btrfs-convert(_start+0x2a)[0x557defdab52a]
>>> Aborted
>>>
>>> Any idea whats going on?
>>> Is it a known bug?
>>> Is the btrfs-progs that come with my dist too old?
>>> FYI the ext4 filesystem is a bit large ~10tb of used data on it.
>>>
>>> I assume the convert didn't even start in this case and nothing was 
>>> modified on the ext4 filesystem, correct? or?
>>>
>>
>> Care too run the following command and share the output:
>>
>> echo "show_super_stats -h" | debugfs -f /dev/stdin /dev/loop0
>>
>> change /dev/loop0 to wherever is your ext4 filesyste, however debugfs 
>> require the fs to be unmounted.
>>
> 
> See output below.
> 
> one more thing, when i ran fsck.ext4 -f on the device before trying to 
> run btrfs-convert it said things like:
> Inode 61475702 extent tree (at level 1) could be shorter.  Optimize<y>? yes
> Inode 61477092 extent tree (at level 2) could be narrower.  Optimize<y>? 
> yes
> and then a bunch more, all aditonal looked similar and said narrower.
> 
> 
> Filesystem volume name:   <none>
> Last mounted on:          /mnt/data
> Filesystem UUID:          78b93577-0bb5-41a1-89d7-3027cf0b9bc2
> Filesystem magic number:  0xEF53
> Filesystem revision #:    1 (dynamic)
> Filesystem features:      has_journal ext_attr dir_index filetype 
> meta_bg extent 64bit flex_bg casefold sparse_super large_file huge_file 
> dir_nlink extra_isize metadata_csum
> Filesystem flags:         signed_directory_hash
> Default mount options:    user_xattr acl
> Filesystem state:         clean
> Errors behavior:          Continue
> Filesystem OS type:       Linux
> Inode count:              536870912
> Block count:              4294967296
> Reserved block count:     0
> Overhead clusters:        12982806
> Free blocks:              2134612618
> Free inodes:              536845192
> First block:              0
> Block size:               4096
> Fragment size:            4096
> Group descriptor size:    64
> Blocks per group:         32768
> Fragments per group:      32768
> Inodes per group:         4096
> Inode blocks per group:   256
> First meta block group:   1792
> Flex block group size:    16
> Filesystem created:       Fri Aug 13 17:27:43 2021
> Last mount time:          Sat May 21 13:56:20 2022
> Last write time:          Mon Jun  6 15:33:45 2022
> Mount count:              0
> Maximum mount count:      -1
> Last checked:             Mon Jun  6 15:33:45 2022
> Check interval:           0 (<none>)
> Lifetime writes:          93 TB
> Reserved blocks uid:      0 (user root)
> Reserved blocks gid:      0 (group root)
> First inode:              11
> Inode size:               256
> Required extra isize:     32
> Desired extra isize:      32
> Journal inode:            8
> Default directory hash:   half_md4
> Directory Hash Seed:      79d63f5e-1fd3-443f-ba05-1a90b7358160
> Journal backup:           inode blocks
> Checksum type:            crc32c
> Checksum:                 0xb53baa13
> Character encoding:       utf8-12.1
> Directories:              138
> 

So indeed it seems like you've hit a bug with the way progs interacts 
with the information provided by e2fsprogs. I will work on a fix and 
will post it to the mailing list probably tomorrow. In the mean time it 
would be best to open a github issue to track progress there as well.

Re: btrfs-convert aborts with an assert

[Torbjörn Jansson]

On 2022-06-06 17:21, Nikolay Borisov wrote:
> 
> 
> On 6.06.22 г. 18:00 ч., Torbjörn Jansson wrote:
>> On 2022-06-06 16:31, Nikolay Borisov wrote:
>>>
>>>
>>> On 6.06.22 г. 17:03 ч., Torbjörn Jansson wrote:
>>>> Hello
>>>>
>>>> i tried to do a btrfs-convert of a ext4 filesystem and after a short while 
>>>> after starting it i was greeted with:
>>>>
>>>> # btrfs-convert /dev/xxxx
>>>> btrfs-convert from btrfs-progs v5.16.2
>>>>
>>>> convert/main.c:1162: do_convert: Assertion `cctx.total_bytes != 0` failed, 
>>>> value 0
>>>> btrfs-convert(+0xffb0)[0x557defdabfb0]
>>>> btrfs-convert(main+0x6c5)[0x557defdaa125]
>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f66e1f8bd0a]
>>>> btrfs-convert(_start+0x2a)[0x557defdab52a]
>>>> Aborted
>>>>
>>>> Any idea whats going on?
>>>> Is it a known bug?
>>>> Is the btrfs-progs that come with my dist too old?
>>>> FYI the ext4 filesystem is a bit large ~10tb of used data on it.
>>>>
>>>> I assume the convert didn't even start in this case and nothing was 
>>>> modified on the ext4 filesystem, correct? or?
>>>>
>>>
>>> Care too run the following command and share the output:
>>>
>>> echo "show_super_stats -h" | debugfs -f /dev/stdin /dev/loop0
>>>
>>> change /dev/loop0 to wherever is your ext4 filesyste, however debugfs 
>>> require the fs to be unmounted.
>>>
>>
>> See output below.
>>
>> one more thing, when i ran fsck.ext4 -f on the device before trying to run 
>> btrfs-convert it said things like:
>> Inode 61475702 extent tree (at level 1) could be shorter.  Optimize<y>? yes
>> Inode 61477092 extent tree (at level 2) could be narrower.  Optimize<y>? yes
>> and then a bunch more, all aditonal looked similar and said narrower.
>>
>>
>> Filesystem volume name:   <none>
>> Last mounted on:          /mnt/data
>> Filesystem UUID:          78b93577-0bb5-41a1-89d7-3027cf0b9bc2
>> Filesystem magic number:  0xEF53
>> Filesystem revision #:    1 (dynamic)
>> Filesystem features:      has_journal ext_attr dir_index filetype meta_bg 
>> extent 64bit flex_bg casefold sparse_super large_file huge_file dir_nlink 
>> extra_isize metadata_csum
>> Filesystem flags:         signed_directory_hash
>> Default mount options:    user_xattr acl
>> Filesystem state:         clean
>> Errors behavior:          Continue
>> Filesystem OS type:       Linux
>> Inode count:              536870912
>> Block count:              4294967296
>> Reserved block count:     0
>> Overhead clusters:        12982806
>> Free blocks:              2134612618
>> Free inodes:              536845192
>> First block:              0
>> Block size:               4096
>> Fragment size:            4096
>> Group descriptor size:    64
>> Blocks per group:         32768
>> Fragments per group:      32768
>> Inodes per group:         4096
>> Inode blocks per group:   256
>> First meta block group:   1792
>> Flex block group size:    16
>> Filesystem created:       Fri Aug 13 17:27:43 2021
>> Last mount time:          Sat May 21 13:56:20 2022
>> Last write time:          Mon Jun  6 15:33:45 2022
>> Mount count:              0
>> Maximum mount count:      -1
>> Last checked:             Mon Jun  6 15:33:45 2022
>> Check interval:           0 (<none>)
>> Lifetime writes:          93 TB
>> Reserved blocks uid:      0 (user root)
>> Reserved blocks gid:      0 (group root)
>> First inode:              11
>> Inode size:               256
>> Required extra isize:     32
>> Desired extra isize:      32
>> Journal inode:            8
>> Default directory hash:   half_md4
>> Directory Hash Seed:      79d63f5e-1fd3-443f-ba05-1a90b7358160
>> Journal backup:           inode blocks
>> Checksum type:            crc32c
>> Checksum:                 0xb53baa13
>> Character encoding:       utf8-12.1
>> Directories:              138
>>
> 
> So indeed it seems like you've hit a bug with the way progs interacts with the 
> information provided by e2fsprogs. I will work on a fix and will post it to the 
> mailing list probably tomorrow. In the mean time it would be best to open a 
> github issue to track progress there as well.
> 

Thanks.
i created a bugreport: https://bugzilla.kernel.org/show_bug.cgi?id=216086

i will probably try to do what i wanted in a different way due to this bug so i 
don't have to rely on btrfs-convert.
i was going to convert from md (raid1) + lvm + ext4 to plain btrfs with raid1
but now i think i will split the md mirror first, create a new btrfs filesystem 
and them copy over the data myself.

Re: btrfs-convert aborts with an assert

[Chris Murphy]

On Mon, Jun 6, 2022 at 12:51 PM Torbjörn Jansson <torbjorn@jansson.tech> wrote:

> Filesystem features:      has_journal ext_attr dir_index filetype meta_bg
> extent 64bit flex_bg casefold sparse_super large_file huge_file dir_nlink
> extra_isize metadata_csum

^^casefold

I don't think this is related to the reported problem, but it occurs
to me maybe btrfs-progs should check for this feature and warn? The
feature being set means there could be behavioral expectations that
Btrfs can't currently provide since it doesn't support casefolding.


-- 
Chris Murphy

Re: btrfs-convert aborts with an assert

[Torbjörn Jansson]

On 2022-06-07 20:24, Chris Murphy wrote:
> On Mon, Jun 6, 2022 at 12:51 PM Torbjörn Jansson <torbjorn@jansson.tech> wrote:
> 
>> Filesystem features:      has_journal ext_attr dir_index filetype meta_bg
>> extent 64bit flex_bg casefold sparse_super large_file huge_file dir_nlink
>> extra_isize metadata_csum
> 
> ^^casefold
> 
> I don't think this is related to the reported problem, but it occurs
> to me maybe btrfs-progs should check for this feature and warn? The
> feature being set means there could be behavioral expectations that
> Btrfs can't currently provide since it doesn't support casefolding.
> 

In my case i planed on using it but since, at the time, the ext4 tools for my 
dist did not support changing the flag properly i never started to use it.

i'm not a dev and don't know all the details about this but i think at least 
there should not be possible to end up with duplicate filenames as a result of 
this.