U-Boot btrfs implementation - will there by syncing?

U-Boot btrfs implementation - will there by syncing?

[Marek Behún]

Hello Qu and others,

one of the points of Qu's reimplementation of U-Boot's btrfs code that
made it somehow synced with btrfs-progs, was that the old
implementation was, quote

  pretty different from btrfs-progs nor kernel, making it pretty hard to
  sync code between different projects

Another point was that afterwards Qu wanted to bring some of the missing
btrfs features into U-Boot, since they should be much easier to
implement after the reimplementation.

I was looking at current btrfs-progs, and noticed that there were many
changes since then, but only one or two were ever synced to U-Boot.

I would like to know if Qu, or anyone else, is planning to do this
code-syncing, and maybe implement some of the missing features?

The reason I am asking this is that in the meantime I've reached the
opinion that the idea of code-syncing in this sense almost never really
works. It is usually proposed by one person who ports the code from
another project, and then keeps it synced for a time, but then stops
doing it because they have too much other work, or change employer, or
another reason.

It happened multiple times. One example is U-Boot's Hush shell, which
was imported from busybox, with the intention to keep it in sync, but
the reality is that U-Boot's implementation is now years (decade?) old,
and there were so many ad-hoc fixes done in U-Boot that currently the
only way to sync is basically to change the whole code (which will also
cause issues with existing user scripts, but maybe it should be done
anyway).

Another reason why I am writing this is that I think that with the
amount of work Qu put into that reimplementation last year, he could
have instead implemented some of the new features in the old
implementation and spend a similar amount of time doing that, and the
result would be those new features in U-Boot for over a year by now.

Also, I've come to the opinion that maybe two different and maintained
implementations are a good thing for such a project as a filesystem,
similar to how multiple implementations of C/C++ compiler are a good
thing.

For the last few months I was on-and-off thinking about whether it
would make sense to revert to the original implementation and then
implement some of the missing features. But it would not make sense if
Qu, or anyone else, is planning to do that code syncing and bringing of
new features from btrfs-progs, so that's why I am asking.

Please give your opinions, people.

Thanks.

Marek

Re: U-Boot btrfs implementation - will there by syncing?

[Qu Wenruo]

On 2021/12/17 07:05, Marek Behún wrote:
> Hello Qu and others,
>
> one of the points of Qu's reimplementation of U-Boot's btrfs code that
> made it somehow synced with btrfs-progs, was that the old
> implementation was, quote
>
>    pretty different from btrfs-progs nor kernel, making it pretty hard to
>    sync code between different projects
>
> Another point was that afterwards Qu wanted to bring some of the missing
> btrfs features into U-Boot, since they should be much easier to
> implement after the reimplementation.
>
> I was looking at current btrfs-progs, and noticed that there were many
> changes since then, but only one or two were ever synced to U-Boot.

Yep, those changes are mostly preparation for the incoming extent-tree
v2 feature, they bring no real behavior change for now.

And I'm planning to cross-port them just to make later cross-port easier.

Thanks for the reminder.

>
> I would like to know if Qu, or anyone else, is planning to do this
> code-syncing, and maybe implement some of the missing features?
>
> The reason I am asking this is that in the meantime I've reached the
> opinion that the idea of code-syncing in this sense almost never really
> works. It is usually proposed by one person who ports the code from
> another project, and then keeps it synced for a time, but then stops
> doing it because they have too much other work, or change employer, or
> another reason.
>
> It happened multiple times. One example is U-Boot's Hush shell, which
> was imported from busybox, with the intention to keep it in sync, but
> the reality is that U-Boot's implementation is now years (decade?) old,
> and there were so many ad-hoc fixes done in U-Boot that currently the
> only way to sync is basically to change the whole code (which will also
> cause issues with existing user scripts, but maybe it should be done
> anyway).
>
> Another reason why I am writing this is that I think that with the
> amount of work Qu put into that reimplementation last year, he could
> have instead implemented some of the new features in the old
> implementation and spend a similar amount of time doing that, and the
> result would be those new features in U-Boot for over a year by now.

I doubt that, the original interface is completely different than the
interface we have in kernel/progs.

That means, if one wants to cross port the recent extent-tree v2
changes, it will be a hell.

So in the long run, it's still beneficial, even they get de-synced somehow.

>
> Also, I've come to the opinion that maybe two different and maintained
> implementations are a good thing for such a project as a filesystem,
> similar to how multiple implementations of C/C++ compiler are a good
> thing.
>
> For the last few months I was on-and-off thinking about whether it
> would make sense to revert to the original implementation and then
> implement some of the missing features. But it would not make sense if
> Qu, or anyone else, is planning to do that code syncing and bringing of
> new features from btrfs-progs, so that's why I am asking.

The worst example for different implementations are another bootloader,
GRUB.

Its btrfs implementation has way less features, no sanity checks, and is
super easy to craft an image to cause infinite loop for GRUB.
And it even can not handle the new default mkfs features (NO_HOLES,
which is there for long long time).

Thus I even re-implemented a new license compatible project from
scratch, btrfs-fuse, to provide a basis for later cross-port.
(That only has read-only support, and is mostly MIT licesnsed, the
interface is almost the same as progs/kernel).

It may be possible for simpler filesystems, but for complex filesysmtes
like btrfs, a completely different implementation with completely
different interfaces is really not an ideal situation.

I don't really want to do it again and again, using different (and
sometimes very crappy) interfaces to do the same work.

Thanks,
Qu

>
> Please give your opinions, people.
>
> Thanks.
>
> Marek

Re: U-Boot btrfs implementation - will there by syncing?

[Qu Wenruo]

On 2021/12/17 07:45, Qu Wenruo wrote:
>
>
> On 2021/12/17 07:05, Marek Behún wrote:
>> Hello Qu and others,
>>
>> one of the points of Qu's reimplementation of U-Boot's btrfs code that
>> made it somehow synced with btrfs-progs, was that the old
>> implementation was, quote
>>
>>    pretty different from btrfs-progs nor kernel, making it pretty hard to
>>    sync code between different projects
>>
>> Another point was that afterwards Qu wanted to bring some of the missing
>> btrfs features into U-Boot, since they should be much easier to
>> implement after the reimplementation.
>>
>> I was looking at current btrfs-progs, and noticed that there were many
>> changes since then, but only one or two were ever synced to U-Boot.
>
> Yep, those changes are mostly preparation for the incoming extent-tree
> v2 feature, they bring no real behavior change for now.
>
> And I'm planning to cross-port them just to make later cross-port easier.

Forgot to mention that, a lot of btrfs-progs changes are not to the core
implementation, but ioctls/user-interafaces/check/image tools, thus you
may need to do more filtering to figure out what's really needed to be
backported.

But I may not cover all changes, so if you find some aspects missing,
feel free to point it out, or even better, try to crossport if possible.

>
> Thanks for the reminder.
>
>>
>> I would like to know if Qu, or anyone else, is planning to do this
>> code-syncing, and maybe implement some of the missing features?

Some missing features, are blocked by lacking of interfaces and knoledge.

Currently the missing hash (BLAKE2) needs BLAKE2 support in U-boot,
which is still not yet in U-boot.

If really needed, I may crossport BLAKE2B to Uboot then implement the
support in btrfs.


The similar case is for multi-device support, I'm not familiar enough on
how to iterate through all block devices in Uboot.

If there is already such interfaces or someone could provide some help,
I'm pretty happy to implement it in Uboot.

>>
>> The reason I am asking this is that in the meantime I've reached the
>> opinion that the idea of code-syncing in this sense almost never really
>> works. It is usually proposed by one person who ports the code from
>> another project, and then keeps it synced for a time, but then stops
>> doing it because they have too much other work, or change employer, or
>> another reason.

Personally speaking, my current employer (SUSE) is super awesome, and
there is no plan to move to another employer.

Furthermore, SUSE is a super strong supporter for btrfs, thus
maintaining a common base to provide better btrfs support is definitely
benefiting SUSE.

Thus you could count on such long term support, and personally speaking
I'm pretty happy to work on btrfs, even not paid for.

So the long term support, at least from myself, should be something you
can rely on.

Although there are indeed cases where I'm buried by other btrfs work,
thus such reminder is really helpful.

Thanks,
Qu

>>
>> It happened multiple times. One example is U-Boot's Hush shell, which
>> was imported from busybox, with the intention to keep it in sync, but
>> the reality is that U-Boot's implementation is now years (decade?) old,
>> and there were so many ad-hoc fixes done in U-Boot that currently the
>> only way to sync is basically to change the whole code (which will also
>> cause issues with existing user scripts, but maybe it should be done
>> anyway).
>>
>> Another reason why I am writing this is that I think that with the
>> amount of work Qu put into that reimplementation last year, he could
>> have instead implemented some of the new features in the old
>> implementation and spend a similar amount of time doing that, and the
>> result would be those new features in U-Boot for over a year by now.
>
> I doubt that, the original interface is completely different than the
> interface we have in kernel/progs.
>
> That means, if one wants to cross port the recent extent-tree v2
> changes, it will be a hell.
>
> So in the long run, it's still beneficial, even they get de-synced somehow.
>
>>
>> Also, I've come to the opinion that maybe two different and maintained
>> implementations are a good thing for such a project as a filesystem,
>> similar to how multiple implementations of C/C++ compiler are a good
>> thing.
>>
>> For the last few months I was on-and-off thinking about whether it
>> would make sense to revert to the original implementation and then
>> implement some of the missing features. But it would not make sense if
>> Qu, or anyone else, is planning to do that code syncing and bringing of
>> new features from btrfs-progs, so that's why I am asking.
>
> The worst example for different implementations are another bootloader,
> GRUB.
>
> Its btrfs implementation has way less features, no sanity checks, and is
> super easy to craft an image to cause infinite loop for GRUB.
> And it even can not handle the new default mkfs features (NO_HOLES,
> which is there for long long time).
>
> Thus I even re-implemented a new license compatible project from
> scratch, btrfs-fuse, to provide a basis for later cross-port.
> (That only has read-only support, and is mostly MIT licesnsed, the
> interface is almost the same as progs/kernel).
>
> It may be possible for simpler filesystems, but for complex filesysmtes
> like btrfs, a completely different implementation with completely
> different interfaces is really not an ideal situation.
>
> I don't really want to do it again and again, using different (and
> sometimes very crappy) interfaces to do the same work.
>
> Thanks,
> Qu
>
>>
>> Please give your opinions, people.
>>
>> Thanks.
>>
>> Marek