Keep bpf-next always open

Keep bpf-next always open

[Alexei Starovoitov]

BPF developers,

The merge window is 1.5 weeks away or 2.5 weeks if rc8 happens. In the past we
observed a rush of patches to get in before bpf-next closes for the duration of
the merge window. Then there is a flood of patches right after bpf-next
reopens. Both periods create unnecessary tension for developers and maintainers.
In order to mitigate these issues we're planning to keep bpf-next open
during upcoming merge window and if this experiment works out we will keep
doing it in the future. The problem that bpf-next cannot be fully open, since
during the merge window lots of trees get pulled by Linus with inevitable bugs
and conflicts. The merge window is the time to fix bugs that got exposed
because of merges and because more people test torvalds/linux.git than
bpf/bpf-next.git.

Hence starting roughly one week before the merge window few risky patches will
be applied to the 'next' branch in the bpf-next tree instead of
bpf-next/master. Then during the two weeks of the merge window the patches will
be reviewed as normal and will be applied to the 'next' branch as well. After
Linus cuts -rc1 and net-next reopens, we will fast forward bpf-next tree to
net-next tree and will try to merge the 'next' branch that accumulated the
patches over these three weeks. After fast-forward the bpf-next tree might look
very different vs its state before the merge window and there is a chance that
some of the patches in the 'next' branch will not apply. We will try to resolve
the conflicts as much as we can and apply them all. Essentially bpf-next/next
is a strong promise that the patches will land into bpf-next. This scheme will
allow developers to work on new features and post them for review and landing
regardless of the merge window or not. Having said that the bug fixing is
always a priority.

We've considered creating a bpf-next-next.git tree for this purpose, but decided
that bpf-next/next branch will be easier for everyone.

Thoughts and comments?

Re: Keep bpf-next always open

[Andrii Nakryiko]

On Wed, Sep 23, 2020 at 2:20 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> BPF developers,
>
> The merge window is 1.5 weeks away or 2.5 weeks if rc8 happens. In the past we
> observed a rush of patches to get in before bpf-next closes for the duration of
> the merge window. Then there is a flood of patches right after bpf-next
> reopens. Both periods create unnecessary tension for developers and maintainers.
> In order to mitigate these issues we're planning to keep bpf-next open
> during upcoming merge window and if this experiment works out we will keep
> doing it in the future. The problem that bpf-next cannot be fully open, since
> during the merge window lots of trees get pulled by Linus with inevitable bugs
> and conflicts. The merge window is the time to fix bugs that got exposed
> because of merges and because more people test torvalds/linux.git than
> bpf/bpf-next.git.
>
> Hence starting roughly one week before the merge window few risky patches will
> be applied to the 'next' branch in the bpf-next tree instead of

Riskiness would be up to maintainers to determine or should we mark
patches with a different tag (bpf-next-next?) explicitly?

> bpf-next/master. Then during the two weeks of the merge window the patches will
> be reviewed as normal and will be applied to the 'next' branch as well. After
> Linus cuts -rc1 and net-next reopens, we will fast forward bpf-next tree to
> net-next tree and will try to merge the 'next' branch that accumulated the
> patches over these three weeks. After fast-forward the bpf-next tree might look
> very different vs its state before the merge window and there is a chance that
> some of the patches in the 'next' branch will not apply. We will try to resolve
> the conflicts as much as we can and apply them all. Essentially bpf-next/next
> is a strong promise that the patches will land into bpf-next. This scheme will
> allow developers to work on new features and post them for review and landing
> regardless of the merge window or not. Having said that the bug fixing is
> always a priority.
>
> We've considered creating a bpf-next-next.git tree for this purpose, but decided
> that bpf-next/next branch will be easier for everyone.
>
> Thoughts and comments?

I like more continuous mode, thanks! bpf-next/next branch still means
that libbpf on Github is effectively frozen for the duration of the
merge window (merging an extra branch automatically is too much pain,
we have enough fun with bpf and bpf-next trees), but let's see how it
goes.

Re: Keep bpf-next always open

[Daniel Borkmann]

On 9/23/20 11:48 PM, Andrii Nakryiko wrote:
> On Wed, Sep 23, 2020 at 2:20 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
>>
>> BPF developers,
>>
>> The merge window is 1.5 weeks away or 2.5 weeks if rc8 happens. In the past we
>> observed a rush of patches to get in before bpf-next closes for the duration of
>> the merge window. Then there is a flood of patches right after bpf-next
>> reopens. Both periods create unnecessary tension for developers and maintainers.
>> In order to mitigate these issues we're planning to keep bpf-next open
>> during upcoming merge window and if this experiment works out we will keep
>> doing it in the future. The problem that bpf-next cannot be fully open, since
>> during the merge window lots of trees get pulled by Linus with inevitable bugs
>> and conflicts. The merge window is the time to fix bugs that got exposed
>> because of merges and because more people test torvalds/linux.git than
>> bpf/bpf-next.git.
>>
>> Hence starting roughly one week before the merge window few risky patches will
>> be applied to the 'next' branch in the bpf-next tree instead of
> 
> Riskiness would be up to maintainers to determine or should we mark
> patches with a different tag (bpf-next-next?) explicitly?

Imho, 'bpf-next-next' tag in subject line would be too confusing, so just sticking
to 'bpf-next' as-is and then up to maintainers whether to apply to master vs next
branch when we get to merge window time. We can see how that works out and adjust
later if necessary.

>> bpf-next/master. Then during the two weeks of the merge window the patches will
>> be reviewed as normal and will be applied to the 'next' branch as well. After
>> Linus cuts -rc1 and net-next reopens, we will fast forward bpf-next tree to
>> net-next tree and will try to merge the 'next' branch that accumulated the
>> patches over these three weeks. After fast-forward the bpf-next tree might look
>> very different vs its state before the merge window and there is a chance that
>> some of the patches in the 'next' branch will not apply. We will try to resolve
>> the conflicts as much as we can and apply them all. Essentially bpf-next/next
>> is a strong promise that the patches will land into bpf-next. This scheme will
>> allow developers to work on new features and post them for review and landing
>> regardless of the merge window or not. Having said that the bug fixing is
>> always a priority.
>>
>> We've considered creating a bpf-next-next.git tree for this purpose, but decided
>> that bpf-next/next branch will be easier for everyone.
>>
>> Thoughts and comments?
> 
> I like more continuous mode, thanks! bpf-next/next branch still means
> that libbpf on Github is effectively frozen for the duration of the
> merge window (merging an extra branch automatically is too much pain,
> we have enough fun with bpf and bpf-next trees), but let's see how it
> goes.

Re: Keep bpf-next always open

[Alexei Starovoitov]

On Wed, Sep 23, 2020 at 02:48:24PM -0700, Andrii Nakryiko wrote:
> On Wed, Sep 23, 2020 at 2:20 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > BPF developers,
> >
> > The merge window is 1.5 weeks away or 2.5 weeks if rc8 happens. In the past we
> > observed a rush of patches to get in before bpf-next closes for the duration of
> > the merge window. Then there is a flood of patches right after bpf-next
> > reopens. Both periods create unnecessary tension for developers and maintainers.
> > In order to mitigate these issues we're planning to keep bpf-next open
> > during upcoming merge window and if this experiment works out we will keep
> > doing it in the future. The problem that bpf-next cannot be fully open, since
> > during the merge window lots of trees get pulled by Linus with inevitable bugs
> > and conflicts. The merge window is the time to fix bugs that got exposed
> > because of merges and because more people test torvalds/linux.git than
> > bpf/bpf-next.git.
> >
> > Hence starting roughly one week before the merge window few risky patches will
> > be applied to the 'next' branch in the bpf-next tree instead of
> 
> Riskiness would be up to maintainers to determine or should we mark
> patches with a different tag (bpf-next-next?) explicitly?

"Risky" in a sense of needing a revert. The bpf tree and two plus -rc1 to -rc7
weeks should be enough to address any issues except the most fundamental ones.
Something like the recent bpf_tail_call support in subprograms I would consider
for the "next" branch if it was posted a day before the merge window.
In practice, I suspect, such cases will be rare.

I think bpf-next-next tag should not be used. All features are for [bpf-next].
Fixes are for [bpf]. The bpf-next/next is a temporary parking place for patches
while the merge window is ongoing.

Re: Keep bpf-next always open

[Song Liu]

> On Sep 23, 2020, at 3:14 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> 
> On Wed, Sep 23, 2020 at 02:48:24PM -0700, Andrii Nakryiko wrote:
>> On Wed, Sep 23, 2020 at 2:20 PM Alexei Starovoitov
>> <alexei.starovoitov@gmail.com> wrote:
>>> 
>>> BPF developers,
>>> 
>>> The merge window is 1.5 weeks away or 2.5 weeks if rc8 happens. In the past we
>>> observed a rush of patches to get in before bpf-next closes for the duration of
>>> the merge window. Then there is a flood of patches right after bpf-next
>>> reopens. Both periods create unnecessary tension for developers and maintainers.
>>> In order to mitigate these issues we're planning to keep bpf-next open
>>> during upcoming merge window and if this experiment works out we will keep
>>> doing it in the future. The problem that bpf-next cannot be fully open, since
>>> during the merge window lots of trees get pulled by Linus with inevitable bugs
>>> and conflicts. The merge window is the time to fix bugs that got exposed
>>> because of merges and because more people test torvalds/linux.git than
>>> bpf/bpf-next.git.
>>> 
>>> Hence starting roughly one week before the merge window few risky patches will
>>> be applied to the 'next' branch in the bpf-next tree instead of
>> 
>> Riskiness would be up to maintainers to determine or should we mark
>> patches with a different tag (bpf-next-next?) explicitly?
> 
> "Risky" in a sense of needing a revert. The bpf tree and two plus -rc1 to -rc7
> weeks should be enough to address any issues except the most fundamental ones.
> Something like the recent bpf_tail_call support in subprograms I would consider
> for the "next" branch if it was posted a day before the merge window.
> In practice, I suspect, such cases will be rare.
> 
> I think bpf-next-next tag should not be used. All features are for [bpf-next].
> Fixes are for [bpf]. The bpf-next/next is a temporary parking place for patches
> while the merge window is ongoing.

I wonder whether we can move/rename the branch around so that the developers can 
always base their work on bpf-next/master. Something like:

Long before merge window for 5.15:	
We only have bpf-next/master

1 week before merge window for 5.15:	
Clone bpf-next/master as bpf-next/for-5.15

From -1 week to the end of merge window
Risky features only goes to bpf-next/master, bug fix goes in both master and for-5.15

After merge window:
Fast forward bpf-next/master based on net-next. Deprecate for-5.15.

Would this work? 

Thanks,
Song

					 

Re: Keep bpf-next always open

[Alexei Starovoitov]

On Wed, Sep 23, 2020 at 10:23:51PM +0000, Song Liu wrote:
> 
> 
> > On Sep 23, 2020, at 3:14 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> > 
> > On Wed, Sep 23, 2020 at 02:48:24PM -0700, Andrii Nakryiko wrote:
> >> On Wed, Sep 23, 2020 at 2:20 PM Alexei Starovoitov
> >> <alexei.starovoitov@gmail.com> wrote:
> >>> 
> >>> BPF developers,
> >>> 
> >>> The merge window is 1.5 weeks away or 2.5 weeks if rc8 happens. In the past we
> >>> observed a rush of patches to get in before bpf-next closes for the duration of
> >>> the merge window. Then there is a flood of patches right after bpf-next
> >>> reopens. Both periods create unnecessary tension for developers and maintainers.
> >>> In order to mitigate these issues we're planning to keep bpf-next open
> >>> during upcoming merge window and if this experiment works out we will keep
> >>> doing it in the future. The problem that bpf-next cannot be fully open, since
> >>> during the merge window lots of trees get pulled by Linus with inevitable bugs
> >>> and conflicts. The merge window is the time to fix bugs that got exposed
> >>> because of merges and because more people test torvalds/linux.git than
> >>> bpf/bpf-next.git.
> >>> 
> >>> Hence starting roughly one week before the merge window few risky patches will
> >>> be applied to the 'next' branch in the bpf-next tree instead of
> >> 
> >> Riskiness would be up to maintainers to determine or should we mark
> >> patches with a different tag (bpf-next-next?) explicitly?
> > 
> > "Risky" in a sense of needing a revert. The bpf tree and two plus -rc1 to -rc7
> > weeks should be enough to address any issues except the most fundamental ones.
> > Something like the recent bpf_tail_call support in subprograms I would consider
> > for the "next" branch if it was posted a day before the merge window.
> > In practice, I suspect, such cases will be rare.
> > 
> > I think bpf-next-next tag should not be used. All features are for [bpf-next].
> > Fixes are for [bpf]. The bpf-next/next is a temporary parking place for patches
> > while the merge window is ongoing.
> 
> I wonder whether we can move/rename the branch around so that the developers can 
> always base their work on bpf-next/master. Something like:
> 
> Long before merge window for 5.15:	
> We only have bpf-next/master
> 
> 1 week before merge window for 5.15:	
> Clone bpf-next/master as bpf-next/for-5.15
> 
> From -1 week to the end of merge window
> Risky features only goes to bpf-next/master, bug fix goes in both master and for-5.15
> 
> After merge window:
> Fast forward bpf-next/master based on net-next. Deprecate for-5.15.
> 
> Would this work? 

It will create headaches for linux-next that merges bpf-next/master.
All linux-next trees should not add patches to those trees that are not going
into the merge window.

Re: Keep bpf-next always open

[Song Liu]

> On Sep 23, 2020, at 3:28 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> 
> On Wed, Sep 23, 2020 at 10:23:51PM +0000, Song Liu wrote:
>> 
>> 
>>> On Sep 23, 2020, at 3:14 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>>> 
>>> On Wed, Sep 23, 2020 at 02:48:24PM -0700, Andrii Nakryiko wrote:
>>>> On Wed, Sep 23, 2020 at 2:20 PM Alexei Starovoitov
>>>> <alexei.starovoitov@gmail.com> wrote:
>>>>> 
>>>>> BPF developers,
>>>>> 
>>>>> The merge window is 1.5 weeks away or 2.5 weeks if rc8 happens. In the past we
>>>>> observed a rush of patches to get in before bpf-next closes for the duration of
>>>>> the merge window. Then there is a flood of patches right after bpf-next
>>>>> reopens. Both periods create unnecessary tension for developers and maintainers.
>>>>> In order to mitigate these issues we're planning to keep bpf-next open
>>>>> during upcoming merge window and if this experiment works out we will keep
>>>>> doing it in the future. The problem that bpf-next cannot be fully open, since
>>>>> during the merge window lots of trees get pulled by Linus with inevitable bugs
>>>>> and conflicts. The merge window is the time to fix bugs that got exposed
>>>>> because of merges and because more people test torvalds/linux.git than
>>>>> bpf/bpf-next.git.
>>>>> 
>>>>> Hence starting roughly one week before the merge window few risky patches will
>>>>> be applied to the 'next' branch in the bpf-next tree instead of
>>>> 
>>>> Riskiness would be up to maintainers to determine or should we mark
>>>> patches with a different tag (bpf-next-next?) explicitly?
>>> 
>>> "Risky" in a sense of needing a revert. The bpf tree and two plus -rc1 to -rc7
>>> weeks should be enough to address any issues except the most fundamental ones.
>>> Something like the recent bpf_tail_call support in subprograms I would consider
>>> for the "next" branch if it was posted a day before the merge window.
>>> In practice, I suspect, such cases will be rare.
>>> 
>>> I think bpf-next-next tag should not be used. All features are for [bpf-next].
>>> Fixes are for [bpf]. The bpf-next/next is a temporary parking place for patches
>>> while the merge window is ongoing.
>> 
>> I wonder whether we can move/rename the branch around so that the developers can 
>> always base their work on bpf-next/master. Something like:
>> 
>> Long before merge window for 5.15:	
>> We only have bpf-next/master
>> 
>> 1 week before merge window for 5.15:	
>> Clone bpf-next/master as bpf-next/for-5.15
>> 
>> From -1 week to the end of merge window
>> Risky features only goes to bpf-next/master, bug fix goes in both master and for-5.15
>> 
>> After merge window:
>> Fast forward bpf-next/master based on net-next. Deprecate for-5.15.
>> 
>> Would this work? 
> 
> It will create headaches for linux-next that merges bpf-next/master.
> All linux-next trees should not add patches to those trees that are not going
> into the merge window.

I see. Keeping bpf-next/master for linux-next/master does make sense. 

How about we keep bpf-next/next always open, or maybe rename it as bpf-next/dev?
Developers could always base their work on bpf-next/dev. When the maintainer 
applies the patch, he can decide whether to apply it to both master and dev, or
just dev. 

Thanks,
Song