Draft talk proposal for MPTCP @ Linux Plumbers '22

Draft talk proposal for MPTCP @ Linux Plumbers '22

[Mat Martineau]

At our recent weekly MPTCP meetings, I've talked about proposing a Linux 
Plumbers talk this year. Matthieu and I have discussed collaborating on 
it.

My high-level idea was to focus on our new customization features: 
userspace path management and BPF packet schedulers.

Since MPTCP is a community effort, I'd like to share this draft so I 
can be sure to focus in the right areas and represent the community well. 
Please review and comment!

Side note: I've included the https://mptcp.dev URL even though it's not 
working yet. Matthieu, do you think we could get something set up there 
before September? The old https://is.gd/mptcp_upstream URL we used for the 
2019 talk doesn't work any more.

----

Multipath TCP (MPTCP) was initially supported in v5.6 of the Linux kernel. 
In later releases, the MPTCP development community has steadily expanded 
from the initial baseline feature set to now support a broad range of 
MPTCP features on the wire and through the socket and generic netlink 
APIs. The first phase of development focused on solid support of server 
use cases. Two current projects are intended to make MPTCP more 
customizable for client devices and unique use cases without requiring 
custom kernel modules.

The first of these customization features is the userspace path manager 
added in v5.19. On the kernel side, MPTCP's generic netlink API was 
updated with new commands that allow a userspace process to control 
multipath address advertisements and the additional subflows that can add 
paths to a MPTCP connection. The Multipath TCP Daemon (mptcpd) is the 
first userspace daemon to support these commands. With these components, 
systems can now customize MPTCP behavior on a per-connection basis. The 
in-kernel path manager, which remains available, has configuration options 
that affect all connections in a network namespace. We will show what the 
userspace and in-kernel path managers are best suited for and explain the 
tradeoffs between them.

A BPF packet scheduler framework is also in active development. When there 
are multiple active TCP subflows in a MPTCP connection, the MPTCP stack 
must decide which of those subflows to use to transmit each data packet. 
The default in-kernel scheduler tries to utilize all the available 
subflows to favor throughput, but is not tunable. Customizable schedulers 
could optimize for latency, redundancy, cost, carrier policy, or other 
factors. We will review the progress that MPTCP Linux developers have made 
on this feature to date, along with some of the challenges of integrating 
BPF into the MPTCP transmit path.

MPTCP development for the Linux kernel and mptcpd are public and open. You 
can find us at mptcp@lists.linux.dev, https://mptcp.dev, and 
https://github.com/intel/mptcpd

--
Mat Martineau
Intel

Re: Draft talk proposal for MPTCP @ Linux Plumbers '22 (2nd draft)

[Mat Martineau]

After some discussion at out last couple of weekly meetings, here's an 
updated draft.

Changes:

  * Moved BPF section up (before userspace PM)
  * Emphasized interactive discussion for BPF issues we've been trying to 
resolve, like the BPF/kernel API.
  * General edits

Suggestions, edits, feedback, etc, all welcome!


---

Multipath TCP (MPTCP) was initially supported in v5.6 of the Linux kernel. 
In later releases, the MPTCP development community has steadily expanded 
from the initial baseline feature set to now support a broad range of 
MPTCP features on the wire and through the socket and generic netlink 
APIs. Two current projects are intended to make MPTCP more customizable 
for client devices and unique use cases without requiring custom kernel 
modules.

One of these projects is the BPF MPTCP packet scheduler framework. When 
there are multiple active TCP subflows in a MPTCP connection, the MPTCP 
stack must decide which of those subflows to use to transmit each data 
packet. The default in-kernel scheduler tries to utilize all the available 
subflows to favor throughput, but it is not tunable. Customizable 
schedulers can optimize for latency, redundancy, cost, carrier policy, or 
other factors. We have patches implementing a proof-of-concept BPF packet 
scheduler, and hope to discuss with the netdev/BPF maintainers and 
audience how we might best structure the BPF/kernel API to balance 
long-term API stability, future evolution of MPTCP scheduler features, and 
usability for scheduler authors.

The next customization feature is the userspace path manager added in 
v5.19. On the kernel side, MPTCP's generic netlink API has new commands 
that allow a userspace process to control multipath address advertisements 
and the additional subflows that add paths to a MPTCP connection. The 
Multipath TCP Daemon (mptcpd) is the first userspace daemon to support 
these commands. With these components, systems can now customize MPTCP 
behavior on a per-connection basis. The in-kernel path manager, which is 
enabled by default, only has per-namespace configuration. We will show 
what the userspace and in-kernel path managers are best suited for and 
explain the tradeoffs between them.

MPTCP development for the Linux kernel and mptcpd are public and open. You 
can find us at mptcp@lists.linux.dev, https://mptcp.dev, and 
https://github.com/intel/mptcpd


--
Mat Martineau
Intel

Re: Draft talk proposal for MPTCP @ Linux Plumbers '22 (2nd draft)

[Matthieu Baerts]

Hi Mat,

On 15/07/2022 00:50, Mat Martineau wrote:
> 
> After some discussion at out last couple of weekly meetings, here's an
> updated draft.
> 
> Changes:
> 
>  * Moved BPF section up (before userspace PM)
>  * Emphasized interactive discussion for BPF issues we've been trying to
> resolve, like the BPF/kernel API.
>  * General edits

Thank you for this v2!

This looks really good. I have one general comment below:

> Suggestions, edits, feedback, etc, all welcome!
> 
> 
> ---
> 
> Multipath TCP (MPTCP) was initially supported in v5.6 of the Linux
> kernel. In later releases, the MPTCP development community has steadily
> expanded from the initial baseline feature set to now support a broad
> range of MPTCP features on the wire and through the socket and generic
> netlink APIs. Two current projects are intended to make MPTCP more
> customizable for client devices and unique use cases without requiring
> custom kernel modules.
> 
> One of these projects is the BPF MPTCP packet scheduler framework. When
> there are multiple active TCP subflows in a MPTCP connection, the MPTCP
> stack must decide which of those subflows to use to transmit each data
> packet. The default in-kernel scheduler tries to utilize all the
> available subflows to favor throughput, but it is not tunable.
> Customizable schedulers can optimize for latency, redundancy, cost,
> carrier policy, or other factors. We have patches implementing a
> proof-of-concept BPF packet scheduler, and hope to discuss with the
> netdev/BPF maintainers and audience how we might best structure the
> BPF/kernel API to balance long-term API stability, future evolution of
> MPTCP scheduler features, and usability for scheduler authors.

Should we insist on the fact that these discussions are not specific to
MPTCP but this is more general and MPTCP is used as one typical use-case?

I would suggest to explain that before this 2nd paragraph. Maybe saying
that we are trying to get answers to questions like: what's the
recommended way to extend a subsystem(?)? The recommended ways seems to
be with BPF and Netlink and they both have limitations. allowing
customisations with BPF bring more questions for us, e.g.: what about
long-term API stability and other questions you raised here at the end.

Then we can introduce the two use cases. The first one with the Packet
Scheduler where we need something on the kernel side as it is a decision
to take per packet. There are some uncertainties when using BPF but a
proof-of-concept implementation exists.

Then for the Path Manager, the decision can be done from the userspace
because the communications are per connections and not per packet. So
this is a another use-case where exposing a Netlink API seems to be a
good fit (even if there are some limitations if there is a very high
number of connections to deal with).

We can then compare BPF and Netlink and trying to get answers to our
questions.

What do you think?

Cheers,
Matt

> 
> The next customization feature is the userspace path manager added in
> v5.19. On the kernel side, MPTCP's generic netlink API has new commands
> that allow a userspace process to control multipath address
> advertisements and the additional subflows that add paths to a MPTCP
> connection. The Multipath TCP Daemon (mptcpd) is the first userspace
> daemon to support these commands. With these components, systems can now
> customize MPTCP behavior on a per-connection basis. The in-kernel path
> manager, which is enabled by default, only has per-namespace
> configuration. We will show what the userspace and in-kernel path
> managers are best suited for and explain the tradeoffs between them.
> 
> MPTCP development for the Linux kernel and mptcpd are public and open.
> You can find us at mptcp@lists.linux.dev, https://mptcp.dev, and
> https://github.com/intel/mptcpd
> 
> 
> -- 
> Mat Martineau
> Intel

-- 
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net

Re: Draft talk proposal for MPTCP @ Linux Plumbers '22 (2nd draft)

[Mat Martineau]

[-- Attachment #1: Type: text/plain, Size: 3391 bytes --]

On Fri, 15 Jul 2022, Matthieu Baerts wrote:

> Hi Mat,
>
> On 15/07/2022 00:50, Mat Martineau wrote:
>>
>> After some discussion at out last couple of weekly meetings, here's an
>> updated draft.
>>
>> Changes:
>>
>>  * Moved BPF section up (before userspace PM)
>>  * Emphasized interactive discussion for BPF issues we've been trying to
>> resolve, like the BPF/kernel API.
>>  * General edits
>
> Thank you for this v2!
>
> This looks really good. I have one general comment below:
>
>> Suggestions, edits, feedback, etc, all welcome!
>>
>>
>> ---
>>
>> Multipath TCP (MPTCP) was initially supported in v5.6 of the Linux
>> kernel. In later releases, the MPTCP development community has steadily
>> expanded from the initial baseline feature set to now support a broad
>> range of MPTCP features on the wire and through the socket and generic
>> netlink APIs. Two current projects are intended to make MPTCP more
>> customizable for client devices and unique use cases without requiring
>> custom kernel modules.
>>
>> One of these projects is the BPF MPTCP packet scheduler framework. When
>> there are multiple active TCP subflows in a MPTCP connection, the MPTCP
>> stack must decide which of those subflows to use to transmit each data
>> packet. The default in-kernel scheduler tries to utilize all the
>> available subflows to favor throughput, but it is not tunable.
>> Customizable schedulers can optimize for latency, redundancy, cost,
>> carrier policy, or other factors. We have patches implementing a
>> proof-of-concept BPF packet scheduler, and hope to discuss with the
>> netdev/BPF maintainers and audience how we might best structure the
>> BPF/kernel API to balance long-term API stability, future evolution of
>> MPTCP scheduler features, and usability for scheduler authors.
>
> Should we insist on the fact that these discussions are not specific to
> MPTCP but this is more general and MPTCP is used as one typical use-case?
>
> I would suggest to explain that before this 2nd paragraph. Maybe saying
> that we are trying to get answers to questions like: what's the
> recommended way to extend a subsystem(?)? The recommended ways seems to
> be with BPF and Netlink and they both have limitations. allowing
> customisations with BPF bring more questions for us, e.g.: what about
> long-term API stability and other questions you raised here at the end.
>
> Then we can introduce the two use cases. The first one with the Packet
> Scheduler where we need something on the kernel side as it is a decision
> to take per packet. There are some uncertainties when using BPF but a
> proof-of-concept implementation exists.
>
> Then for the Path Manager, the decision can be done from the userspace
> because the communications are per connections and not per packet. So
> this is a another use-case where exposing a Netlink API seems to be a
> good fit (even if there are some limitations if there is a very high
> number of connections to deal with).
>
> We can then compare BPF and Netlink and trying to get answers to our
> questions.
>
> What do you think?
>

I think that's a good way to frame the talk: emphasizes that the content 
can be useful for understanding general ideas about extending/customizing 
kernel functionality, and also cover what we've done in MPTCP for these 
two cases. I'll update the draft and post a v3 on Monday.


--
Mat Martineau
Intel

Re: Draft talk proposal for MPTCP @ Linux Plumbers '22 (3rd draft)

[Mat Martineau]

Here's a new draft based on Matthieu's feedback for v2 
(https://lore.kernel.org/mptcp/7cd436ad-6c05-d788-7af3-2ff08d647dfd@tessares.net/).

The main goal was to frame the proposal as an illustration of tradeoffs 
between different kernel extensibility options (BPF & genl).


---

Multipath TCP (MPTCP) was initially supported in v5.6 of the Linux kernel. 
In subsequent releases, the MPTCP development community has steadily 
expanded from the initial baseline feature set to now support a broad 
range of MPTCP features on the wire and through the socket and generic 
netlink APIs.

With core MPTCP functionality established, our next goal is to make MPTCP 
more extensible and customizable at runtime. The two most common tools in 
the kernel's networking subsystem for these purposes are generic netlink 
and BPF. Each has tradeoffs that make them better suited for different 
scenarios. Our choices for extending MPTCP show some of those tradeoffs, 
and also leave our community with some open questions about how to best 
use these interfaces and frameworks.

As one example, we are extending the MPTCP packet scheduler using BPF. 
When there are multiple active TCP subflows in a MPTCP connection, the 
MPTCP stack must decide which of those subflows to use to transmit each 
data packet. This requires low latency and low overhead, and direct access 
to low-level TCP connection information. Customizable schedulers can 
optimize for latency, redundancy, cost, carrier policy, or other factors. 
We have patches implementing a proof-of-concept BPF packet scheduler, and 
hope to discuss with the netdev/BPF maintainers and audience how we might 
best structure the BPF/kernel API to balance long-term API stability, 
future evolution of MPTCP scheduler features, and usability for scheduler 
authors.

The next customization feature is the userspace path manager added in 
v5.19. MPTCP path managers advertise addresses available for multipath 
connections, and establish or close additional TCP subflows using the 
available interfaces. There are a limited number of interactions with a 
path manager during the life of a MPTCP connection. Operations are not 
very sensitive to latency, and may need access to data from userspace. 
This led us to expand the MPTCP generic netlink API and update the 
Multipath TCP Daemon (mptcpd) to support the new commands. Generic netlink 
has been a good fit for path manager commands and events, the concepts are 
familiar and the message format makes it possible to maintain forward and 
backward compatibility between different kernel versions and userspace 
binaries.

MPTCP development for the Linux kernel and mptcpd are public and open. You 
can find us at mptcp@lists.linux.dev, https://mptcp.dev, and 
https://github.com/intel/mptcpd



--
Mat Martineau
Intel