state of drbd in mainline

state of drbd in mainline

[Christoph Hellwig]

Hi all,

while I've been reworking large parts of the block layer over the
last months I've basically never gotten drbd reviews, and in general
drbd seems to be completely maintained upstream.  Do we even have users
for it?  Is it time to drop drbd from the kernel tree?

Thanks,
	Christoph

Re: state of drbd in mainline

[Philipp Reisner]

Hi Christoph,

what do you expect for the DRBD changes? Usually, they fall into the
category: yes, obviously. When you are changing the block layer in
this way, then that is necessary for the in-tree DRBD code.

Regarding users: Yes, there are users of the in-tree DRBD code. Very
big corporations. And, yes, we see pressure building up that we get
our act together and re-sync the in-tree DRBD with the out-of tree
DRBD.

best regards,
 Phil


On Tue, Mar 29, 2022 at 9:06 AM Christoph Hellwig <hch@lst.de> wrote:
>
> Hi all,
>
> while I've been reworking large parts of the block layer over the
> last months I've basically never gotten drbd reviews, and in general
> drbd seems to be completely maintained upstream.  Do we even have users
> for it?  Is it time to drop drbd from the kernel tree?
>
> Thanks,
>         Christoph

Re: state of drbd in mainline

[Christoph Hellwig]

On Tue, Mar 29, 2022 at 09:22:26AM +0200, Philipp Reisner wrote:
> Hi Christoph,
> 
> what do you expect for the DRBD changes? Usually, they fall into the
> category: yes, obviously. When you are changing the block layer in this
> way, then that is necessary for the in-tree DRBD code.
> 
> Regarding users: Yes, there are users of the in-tree DRBD code. Very big
> corporations. And, yes, we see pressure building up that we get our act
> together and re-sync the in-tree DRBD with the out-of tree DRBD.

The complete lack of bug reports and maintainer interaction usually
suggests low to no use and heavy bitrot.  If that is not the case
here that's fine, just asking..

Re: state of drbd in mainline

[Philipp Reisner]

> The complete lack of bug reports and maintainer interaction usually
> suggests low to no use and heavy bitrot.  If that is not the case
> here that's fine, just asking..
>

FYI, feature-wise the in-tree DRBD is frozen. Maintenance only. Users,
seeking advanced
features switch to the out-of-tree DRBD and use other means of communication:
Github issues, linbit's community slack channel, last not least
linbit's ticketing system.

That is why you see virtually no activity.

Re: state of drbd in mainline

[Jens Axboe]

On 3/29/22 1:50 AM, Philipp Reisner wrote:
>> The complete lack of bug reports and maintainer interaction usually
>> suggests low to no use and heavy bitrot.  If that is not the case
>> here that's fine, just asking..
>>
> 
> FYI, feature-wise the in-tree DRBD is frozen. Maintenance only. Users,
> seeking advanced
> features switch to the out-of-tree DRBD and use other means of communication:
> Github issues, linbit's community slack channel, last not least
> linbit's ticketing system.
> 
> That is why you see virtually no activity.

I've been thinking the same thing. Even if most users are out-of-tree, you'd
still expect to see at least some reports on the list. A few followup
questions:

- Why aren't the drbd maintainers responding to posted patches? They seem
  to simply be ignored, and I'm left to pickup the trivial ones that look
  fine to me. In-kernel drbd appears largely unmaintained, and has been for
  years.

- Even if out-of-band communication is used for in-kernel users of drbd,
  that doesn't result in any patches or fixes that should go upstream?

- If there's zero activity for in-kernel drbd, all users are using the
  out-of-tree version?

As far as I can tell, drbd upstream is stone cold dead, and has been for
years. Why shouldn't it just get removed? Is it just bait to get people
to use an out-of-tree version?

-- 
Jens Axboe

Re: state of drbd in mainline

[Philipp Reisner]

Jens, my intention is to keep it in-tree, and at some point update it.
Regarding your questions:

[...]

> - Why aren't the drbd maintainers responding to posted patches? They seem
>   to simply be ignored, and I'm left to pickup the trivial ones that look
>   fine to me. In-kernel drbd appears largely unmaintained, and has been for
>   years.

The team here has grown, we are busy. Since you started to pick up the
trivial patches yourself, I thought it is not necessary that I collect them and
send a pull request in merge-window time.

> - Even if out-of-band communication is used for in-kernel users of drbd,
>   that doesn't result in any patches or fixes that should go upstream?

This one:
https://patchwork.kernel.org/project/linux-block/patch/20210426163032.3454129-1-christoph.boehmwalder@linbit.com/
(relevant to users that have DRBD on top of md raid)

> - If there's zero activity for in-kernel drbd, all users are using the
>   out-of-tree version?

There are users of the in-tree version, some with huge fleets.
Some do not need the newer out-of-tree DRBD, and the in-tree version is a
lot easier to compile. You need coccinelle for the out-of-tree version,
and that can already be a hindering barrier for some.

> As far as I can tell, drbd upstream is stone cold dead, and has been for
> years. Why shouldn't it just get removed?

Because there are users.

> Is it just bait to get people to use an out-of-tree version?

No.

> --
> Jens Axboe

Re: state of drbd in mainline

[Jens Axboe]

On 3/29/22 9:44 AM, Philipp Reisner wrote:
> Jens, my intention is to keep it in-tree, and at some point update it.
> Regarding your questions:

That'd be great, but it's been years since there was any significant
updates to the in-kernel drbd... I would strongly suggest that the
in-kernel be brought closer to what people are mostly running, as it
stands it's basically unmaintained.

>> - Why aren't the drbd maintainers responding to posted patches? They seem
>>   to simply be ignored, and I'm left to pickup the trivial ones that look
>>   fine to me. In-kernel drbd appears largely unmaintained, and has been for
>>   years.
> 
> The team here has grown, we are busy. Since you started to pick up the
> trivial patches yourself, I thought it is not necessary that I collect
> them and send a pull request in merge-window time.

That's just one part of it, the fact that the out-of-tree drbd has been
drifting further and further away from the in-kernel one is a big
problem. For trivial patches, I have no issue picking them up. But
silence on your side is not very helpful. Please review and ack patches.
If I see it acked, then I can easily pick it up.

>> - Even if out-of-band communication is used for in-kernel users of drbd,
>>   that doesn't result in any patches or fixes that should go upstream?
> 
> This one:
> https://patchwork.kernel.org/project/linux-block/patch/20210426163032.3454129-1-christoph.boehmwalder@linbit.com/
> (relevant to users that have DRBD on top of md raid)

That's a single patch, from a year ago. Not a strong counter point,
looks more like it's proving my case...

>> - If there's zero activity for in-kernel drbd, all users are using the
>>   out-of-tree version?
> 
> There are users of the in-tree version, some with huge fleets.
> Some do not need the newer out-of-tree DRBD, and the in-tree version is a
> lot easier to compile. You need coccinelle for the out-of-tree version,
> and that can already be a hindering barrier for some.

The main discrepancy here is that there are apparently huge numbers of
in-tree users, yet no fixes or patches at all making it to mainline.
Either drbd is bug free and doesn't require any fixes at all, which I
very much would doubt, or fixes aren't being sent upstream.

-- 
Jens Axboe

Re: state of drbd in mainline

[Philipp Reisner]

> > Jens, my intention is to keep it in-tree, and at some point update it.
> > Regarding your questions:
>
> That'd be great, but it's been years since there was any significant
> updates to the in-kernel drbd... I would strongly suggest that the
> in-kernel be brought closer to what people are mostly running, as it
> stands it's basically unmaintained.

The changes we worked on over many Years in the more recent drbd-9.x
branches are just too fundamental to do them in small chunks, we could
upstream bit by bit.  We need to get that reviewed in a big series.  If I
started to dump them on linux-block right away, nobody would look at it
seriously, since it would be too much.  I intend to get people from red
hat/suse assigned to do such a review. Then we will do that on linux-block,
so that everyone who cares sees what happens.

[...]
> The main discrepancy here is that there are apparently huge numbers of
> in-tree users, yet no fixes or patches at all making it to mainline.
> Either drbd is bug free and doesn't require any fixes at all, which I
> very much would doubt, or fixes aren't being sent upstream.

It is the broad consent among the users of the drbd-8.4 branch (that is what
is in-tree), is that it works for its purpose. It is for sure not bug-free,
but people are not running into bugs anymore. So, call it free of relevant
bugs, if you want.  No new features go into that branch, on purpose. To keep
it that way.

Have a look at that one real bug-fix that was identified in the last Year.
https://patchwork.kernel.org/project/linux-block/patch/20210426163032.3454129-1-christoph.boehmwalder@linbit.com/

When do you want to have that reposted to you?
right now? Just before the next merge window opens?

Re: state of drbd in mainline

[Mike Snitzer]

On Wed, Mar 30 2022 at 11:23P -0400,
Philipp Reisner <philipp.reisner@linbit.com> wrote:

> > > Jens, my intention is to keep it in-tree, and at some point update it.
> > > Regarding your questions:
> >
> > That'd be great, but it's been years since there was any significant
> > updates to the in-kernel drbd... I would strongly suggest that the
> > in-kernel be brought closer to what people are mostly running, as it
> > stands it's basically unmaintained.
> 
> The changes we worked on over many Years in the more recent drbd-9.x
> branches are just too fundamental to do them in small chunks, we could
> upstream bit by bit.  We need to get that reviewed in a big series.  If I
> started to dump them on linux-block right away, nobody would look at it
> seriously, since it would be too much.  I intend to get people from red
> hat/suse assigned to do such a review. Then we will do that on linux-block,
> so that everyone who cares sees what happens.

Why do you think Red Hat, SUSE or any other distro vendor's engineers
should be made to review what amounts to be a massive dump of changes
you developed over years?

Presummably you have heard of "upstream first"!?  Why do you think it
doesn't apply to drbd?

It'd be one thing if drbd never went upstream but _it did_.  As is
your development model is completely wrong.

Mike

Re: state of drbd in mainline

[Jens Axboe]

On 3/30/22 9:23 AM, Philipp Reisner wrote:
>>> Jens, my intention is to keep it in-tree, and at some point update it.
>>> Regarding your questions:
>>
>> That'd be great, but it's been years since there was any significant
>> updates to the in-kernel drbd... I would strongly suggest that the
>> in-kernel be brought closer to what people are mostly running, as it
>> stands it's basically unmaintained.
> 
> The changes we worked on over many Years in the more recent drbd-9.x
> branches are just too fundamental to do them in small chunks, we could
> upstream bit by bit.  We need to get that reviewed in a big series.  If I

Your development model is fundamentally broken. You've allowed your 9.x
branch to totally drift from mainline, which just helps underline my
earlier point on that in-kernel drbd is effectively abandoned and
unmaintained.

> started to dump them on linux-block right away, nobody would look at it
> seriously, since it would be too much.  I intend to get people from red
> hat/suse assigned to do such a review. Then we will do that on linux-block,
> so that everyone who cares sees what happens.

You're just doing it totally wrong. Upstream kernel should match your
9.x branch, and it should have been developed in sync. What you appear
to have done is to ignore mainline, while it would've been correct and
much easier in the long run to ensure that development is regularly
synced to the mainline kernel. You know, like EVERY other driver that is
maintained does.

Now you've got a giant pile of patches, which probably don't adhere to
how we would've done the mainline commits in the first place, and it'll
cause a huge pain for not just you but upstream reviewers. I don't care
about the former, but I do care a lot about the latter. That's a giant
waste of the time of the folks involved on the block side, and
definitely not what a responsible kernel maintainer would do.

From your reply here and earlier ones, seems to me that you don't grasp
the gravity of the situation, which is also worrying.

>> The main discrepancy here is that there are apparently huge numbers of
>> in-tree users, yet no fixes or patches at all making it to mainline.
>> Either drbd is bug free and doesn't require any fixes at all, which I
>> very much would doubt, or fixes aren't being sent upstream.
> 
> It is the broad consent among the users of the drbd-8.4 branch (that is what
> is in-tree), is that it works for its purpose. It is for sure not bug-free,
> but people are not running into bugs anymore. So, call it free of relevant
> bugs, if you want.  No new features go into that branch, on purpose. To keep
> it that way.
> 
> Have a look at that one real bug-fix that was identified in the last Year.
> https://patchwork.kernel.org/project/linux-block/patch/20210426163032.3454129-1-christoph.boehmwalder@linbit.com/
> 
> When do you want to have that reposted to you?
> right now? Just before the next merge window opens?

That can go in anytime, so please do submit it.

-- 
Jens Axboe

Re: [Drbd-dev] state of drbd in mainline

[Kiselev, Oleg]

AWS RDS is a major user of drbd.  We are probably responsible for the substantial percentage of drbd users in the world.  We are currently on drbd-8.4.  We will eventually migrate to drbd-9, but at our scale this isn't going to happen very soon.  

The last time we needed to patch drbd was back in 2018.  We've not diagnosed any reportable issues with drbd-8.4 in the past ~4 years.   It's a stable, mature driver.   We would prefer for drbd-8 to continue being available in the upstream tree.  We'd also welcome drbd-9 being available in-tree as soon as possible: it will make life much easier for other AWS teams that are currently working on DRBD deployment.

On 3/30/22, 8:24 AM, "drbd-dev-bounces@lists.linbit.com on behalf of Philipp Reisner" <drbd-dev-bounces@lists.linbit.com on behalf of philipp.reisner@linbit.com> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



    > > Jens, my intention is to keep it in-tree, and at some point update it.
    > > Regarding your questions:
    >
    > That'd be great, but it's been years since there was any significant
    > updates to the in-kernel drbd... I would strongly suggest that the
    > in-kernel be brought closer to what people are mostly running, as it
    > stands it's basically unmaintained.

    The changes we worked on over many Years in the more recent drbd-9.x
    branches are just too fundamental to do them in small chunks, we could
    upstream bit by bit.  We need to get that reviewed in a big series.  If I
    started to dump them on linux-block right away, nobody would look at it
    seriously, since it would be too much.  I intend to get people from red
    hat/suse assigned to do such a review. Then we will do that on linux-block,
    so that everyone who cares sees what happens.

    [...]
    > The main discrepancy here is that there are apparently huge numbers of
    > in-tree users, yet no fixes or patches at all making it to mainline.
    > Either drbd is bug free and doesn't require any fixes at all, which I
    > very much would doubt, or fixes aren't being sent upstream.

    It is the broad consent among the users of the drbd-8.4 branch (that is what
    is in-tree), is that it works for its purpose. It is for sure not bug-free,
    but people are not running into bugs anymore. So, call it free of relevant
    bugs, if you want.  No new features go into that branch, on purpose. To keep
    it that way.

    Have a look at that one real bug-fix that was identified in the last Year.
    20210426163032.3454129-1-christoph.boehmwalder@linbit.com <https://patchwork.kernel.org/project/linux-block/patch/<a href=>/">https://patchwork.kernel.org/project/linux-block/patch/20210426163032.3454129-1-christoph.boehmwalder@linbit.com/

    When do you want to have that reposted to you?
    right now? Just before the next merge window opens?
    _______________________________________________
    drbd-dev mailing list
    drbd-dev@lists.linbit.com
    https://lists.linbit.com/mailman/listinfo/drbd-dev

Re: [Drbd-dev] state of drbd in mainline

[Jens Axboe]

On 3/30/22 1:31 PM, Kiselev, Oleg wrote:
> AWS RDS is a major user of drbd.  We are probably responsible for the
> substantial percentage of drbd users in the world.  We are currently
> on drbd-8.4.  We will eventually migrate to drbd-9, but at our scale
> this isn't going to happen very soon.  
> 
> The last time we needed to patch drbd was back in 2018.  We've not
> diagnosed any reportable issues with drbd-8.4 in the past ~4 years.
> It's a stable, mature driver.   We would prefer for drbd-8 to continue
> being available in the upstream tree.  We'd also welcome drbd-9 being
> available in-tree as soon as possible: it will make life much easier
> for other AWS teams that are currently working on DRBD deployment.

Thanks for the info, that's useful and good to know. My objection was as
much about ignoring patches that are sent for drbd, I don't recall
seeing any replies to those kinds of emails in the last 3-4 years. Part
of maintaining an in-tree driver is also dealing with those in a
reasonable amount of time, and ensuring that they make it upstream. And
the one bug fix that does look important was just sent upstream only
because of this discussion, and that's been lingering for a year it
seems.

Roughly ordered by date:

https://lore.kernel.org/all/20220327070918.8465-1-xiam0nd.tong@gmail.com/
https://lore.kernel.org/all/1648436049-4335-1-git-send-email-baihaowen@meizu.com/
https://lore.kernel.org/all/YbpERiPKO4ufe1hf@pc638.lan/
https://lore.kernel.org/all/20211213223331.135412-11-keescook@chromium.org/
[PATCH 0/2] drbd: Make use of PFN_UP helper macro (only on drbd list)
https://lore.kernel.org/all/20210525065925.3978-1-lyl2019@mail.ustc.edu.cn/
https://lore.kernel.org/all/1619774456-116567-1-git-send-email-jiapeng.chong@linux.alibaba.com/
https://lore.kernel.org/all/20210426163032.3454129-1-christoph.boehmwalder@linbit.com/ (the referenced patch)
https://lore.kernel.org/all/20210402070713.4069-1-lyl2019@mail.ustc.edu.cn/ (use after free, security issue?)
https://lore.kernel.org/all/20201026215043.3893318-1-arnd@kernel.org/

This is just going back a bit, by no means an exhaustive list. Looking back
to 2016-2017 and I see zero replies in emails like the above, not one.
Everybody misses patches here and there, that's common. What isn't common
is blatantly ignoring everything for, what, 5 years?

I'm not here to babysit unmaintained drivers. linbit need to get their
act together and maintain the driver, or it will get removed.

-- 
Jens Axboe

Re: [Drbd-dev] state of drbd in mainline

[Wolfgang Walter]

Am 2022-03-29 09:32, schrieb Christoph Hellwig:
> On Tue, Mar 29, 2022 at 09:22:26AM +0200, Philipp Reisner wrote:
>> Hi Christoph,
>> 
>> what do you expect for the DRBD changes? Usually, they fall into the
>> category: yes, obviously. When you are changing the block layer in 
>> this
>> way, then that is necessary for the in-tree DRBD code.
>> 
>> Regarding users: Yes, there are users of the in-tree DRBD code. Very 
>> big
>> corporations. And, yes, we see pressure building up that we get our 
>> act
>> together and re-sync the in-tree DRBD with the out-of tree DRBD.
> 
> The complete lack of bug reports and maintainer interaction usually
> suggests low to no use and heavy bitrot.  If that is not the case
> here that's fine, just asking..
> 
> _______________________________________________
> drbd-dev mailing list
> drbd-dev@lists.linbit.com
> https://lists.linbit.com/mailman/listinfo/drbd-dev

We use the in-tree DRBD with LTS (actually 5.15.32 and 5.4.188) and 
depend on it. It just works. We never had any issues in the last 5 
years.

Regards
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts