Re: [MPTCP] MPTCP-API

Re: [MPTCP] MPTCP-API

[Matthieu Baerts]

[-- Attachment #1: Type: text/plain, Size: 2762 bytes --]

Hi Mat,

On 12/06/2019 18:21, Mat Martineau wrote:
> 
> On Tue, 11 Jun 2019, Christoph Paasch wrote:
> 
>> Hello,
>>
>>> On Jun 11, 2019, at 3:24 PM, Mat Martineau
>>> <mathew.j.martineau(a)linux.intel.com> wrote:
>>>
>>>
>>> Hi Christoph and Leif,
>>>
>>> On Tue, 11 Jun 2019, Christoph Paasch wrote:
>>>
>>>> I am bringing up something that we already once discussed quite a while
>>>> back. Apache Traffic Server has an option to enable MPTCP (currently,
>>>> through the socket-option MPTCP_ENABLED that is exposed on the
>>>> multipath-tcp.org kernel). Leif (in CC) is the main contributor &
>>>> maintainer
>>>> of ATS.
>>>>
>>>> Now, ATS will be doing a 9.0.0-release soon that will be the first
>>>> release
>>>> with this MPTCP-option in the ATS-config.
>>>>
>>>> We want to make sure that it would also work with the upcoming
>>>> upstream MPTCP
>>>> implementation. So, we can easily make this use IPPROTO_MPTCP. However,
>>>> there currently isn't a 100% guarantee that the API is stable (as
>>>> upstream
>>>> might request changes).
>>>>
>>>> So, we are wondering how we can do binary compatibility without
>>>> having to
>>>> wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a
>>>> sysctl
>>>> that indicates the "API-version" ? That way, upon startup ATS can
>>>> simply
>>>> check the API-version and if it doesn't match, just disable MPTCP.
>>>>
>>>>
>>>> What are your thoughts?
>>>
>>> There are a couple of approaches we could use to help ATS and other
>>> projects as upstream MPTCP evolves.
>>>
>>> One is that we should pick a likely-to-get-merged value for
>>> IPPROTO_MPTCP. The current prototype uses 99, but the IANA defines
>>> that as "any private encryption scheme). We had earlier prototypes
>>> with 262 (IPPROTO_TCP | 0x100) since the socket API uses 'int proto'
>>> instead of a single byte. ATS could consider a runtime-configurable
>>> proto value for MPTCP to help with this.
>>
>> Yes, let's pick a "likely-to-get-merged" value ASAP. Do you think 262
>> could work out?
>>
> 
> I'll test it out and send the patches, so they're ready to merge if we
> have a consensus.

Should we not request an official protocol number from the IANA? It
seems that all the other ones in the upstream Linux kernel are official.
Why not having one for MPTCP then. Especially if this will need to be
added later in the different libC.

I can check what to do and ask around how our TCP option official number
for MPTCP (30) was requested and assigned.

Best regards,
Matt
-- 
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

Re: [MPTCP] MPTCP-API

[Matthieu Baerts]

[-- Attachment #1: Type: text/plain, Size: 3038 bytes --]

Hi Mat,

On 13/06/2019 17:02, Mat Martineau wrote:
> 
> On Thu, 13 Jun 2019, Matthieu Baerts wrote:
> 
>> Hi Mat,
>>
>> On 12/06/2019 18:21, Mat Martineau wrote:
>>>
>>> On Tue, 11 Jun 2019, Christoph Paasch wrote:
>>>
>>>> Hello,
>>>>
>>>>> On Jun 11, 2019, at 3:24 PM, Mat Martineau
>>>>> <mathew.j.martineau(a)linux.intel.com> wrote:
>>>>>
>>>>>
>>>>> Hi Christoph and Leif,
>>>>>
>>>>> On Tue, 11 Jun 2019, Christoph Paasch wrote:
>>>>>
>>>>>> So, we are wondering how we can do binary compatibility without
>>>>>> having to wait for MPTCP to land upstream. Could the MPTCP-code
>>>>>> expose maybe a sysctl that indicates the "API-version" ? That way,
>>>>>> upon startup ATS can simply check the API-version and if it
>>>>>> doesn't match, just disable MPTCP.
>>>>>>
>>>>>>
>>>>>> What are your thoughts?
>>>>>
>>>>> There are a couple of approaches we could use to help ATS and other
>>>>> projects as upstream MPTCP evolves.
>>>>>
>>>>> One is that we should pick a likely-to-get-merged value for
>>>>> IPPROTO_MPTCP. The current prototype uses 99, but the IANA defines
>>>>> that as "any private encryption scheme). We had earlier prototypes
>>>>> with 262 (IPPROTO_TCP | 0x100) since the socket API uses 'int proto'
>>>>> instead of a single byte. ATS could consider a runtime-configurable
>>>>> proto value for MPTCP to help with this.
>>>>
>>>> Yes, let's pick a "likely-to-get-merged" value ASAP. Do you think 262
>>>> could work out?
>>>>
>>>
>>> I'll test it out and send the patches, so they're ready to merge if we
>>> have a consensus.
>>
>> Should we not request an official protocol number from the IANA? It
>> seems that all the other ones in the upstream Linux kernel are official.
>> Why not having one for MPTCP then. Especially if this will need to be
>> added later in the different libC.
>>
>> I can check what to do and ask around how our TCP option official number
>> for MPTCP (30) was requested and assigned.
> 
> I think the IANA would expect a new protocol number to be used
> over-the-wire, which would dramatically affect MPTCP's ability to
> pretend to be TCP as far as middleboxes and firewalls are concerned!
> 
> The Linux definitions already deviate somewhat from the IANA list (for
> example, IPPROTO_IP), which leads me to believe there is some
> flexibility in future definitions.

I agree with you, I didn't want to set a new protocol number visible on
the wire. I guess we cannot do that regarding the RFC6824 anyway.

My point is more to get an official number for this IPPROTO_MPTCP only
for libc and keep the same between the different OS.
For example, we have IPPROTO_RAW (255) which is marked as "Reserved" by
IANA.

We can discuss about that at our weekly meeting if you want to.

Cheers,
Matt

> -- 
> Mat Martineau
> Intel

-- 
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

Re: [MPTCP] MPTCP-API

[Mat Martineau]

[-- Attachment #1: Type: text/plain, Size: 2298 bytes --]


On Thu, 13 Jun 2019, Matthieu Baerts wrote:

> Hi Mat,
>
> On 12/06/2019 18:21, Mat Martineau wrote:
>>
>> On Tue, 11 Jun 2019, Christoph Paasch wrote:
>>
>>> Hello,
>>>
>>>> On Jun 11, 2019, at 3:24 PM, Mat Martineau
>>>> <mathew.j.martineau(a)linux.intel.com> wrote:
>>>>
>>>>
>>>> Hi Christoph and Leif,
>>>>
>>>> On Tue, 11 Jun 2019, Christoph Paasch wrote:
>>>>
>>>>> So, we are wondering how we can do binary compatibility without 
>>>>> having to wait for MPTCP to land upstream. Could the MPTCP-code 
>>>>> expose maybe a sysctl that indicates the "API-version" ? That way, 
>>>>> upon startup ATS can simply check the API-version and if it doesn't 
>>>>> match, just disable MPTCP.
>>>>>
>>>>>
>>>>> What are your thoughts?
>>>>
>>>> There are a couple of approaches we could use to help ATS and other
>>>> projects as upstream MPTCP evolves.
>>>>
>>>> One is that we should pick a likely-to-get-merged value for
>>>> IPPROTO_MPTCP. The current prototype uses 99, but the IANA defines
>>>> that as "any private encryption scheme). We had earlier prototypes
>>>> with 262 (IPPROTO_TCP | 0x100) since the socket API uses 'int proto'
>>>> instead of a single byte. ATS could consider a runtime-configurable
>>>> proto value for MPTCP to help with this.
>>>
>>> Yes, let's pick a "likely-to-get-merged" value ASAP. Do you think 262
>>> could work out?
>>>
>>
>> I'll test it out and send the patches, so they're ready to merge if we
>> have a consensus.
>
> Should we not request an official protocol number from the IANA? It
> seems that all the other ones in the upstream Linux kernel are official.
> Why not having one for MPTCP then. Especially if this will need to be
> added later in the different libC.
>
> I can check what to do and ask around how our TCP option official number
> for MPTCP (30) was requested and assigned.

I think the IANA would expect a new protocol number to be used 
over-the-wire, which would dramatically affect MPTCP's ability to pretend 
to be TCP as far as middleboxes and firewalls are concerned!

The Linux definitions already deviate somewhat from the IANA list (for 
example, IPPROTO_IP), which leads me to believe there is some flexibility 
in future definitions.

--
Mat Martineau
Intel

Re: [MPTCP] MPTCP-API

[Mat Martineau]

[-- Attachment #1: Type: text/plain, Size: 3435 bytes --]


On Tue, 11 Jun 2019, Christoph Paasch wrote:

> Hello,
>
>> On Jun 11, 2019, at 3:24 PM, Mat Martineau <mathew.j.martineau(a)linux.intel.com> wrote:
>>
>>
>> Hi Christoph and Leif,
>>
>> On Tue, 11 Jun 2019, Christoph Paasch wrote:
>>
>>> I am bringing up something that we already once discussed quite a while
>>> back. Apache Traffic Server has an option to enable MPTCP (currently,
>>> through the socket-option MPTCP_ENABLED that is exposed on the
>>> multipath-tcp.org kernel). Leif (in CC) is the main contributor & maintainer
>>> of ATS.
>>>
>>> Now, ATS will be doing a 9.0.0-release soon that will be the first release
>>> with this MPTCP-option in the ATS-config.
>>>
>>> We want to make sure that it would also work with the upcoming upstream MPTCP
>>> implementation. So, we can easily make this use IPPROTO_MPTCP. However,
>>> there currently isn't a 100% guarantee that the API is stable (as upstream
>>> might request changes).
>>>
>>> So, we are wondering how we can do binary compatibility without having to
>>> wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl
>>> that indicates the "API-version" ? That way, upon startup ATS can simply
>>> check the API-version and if it doesn't match, just disable MPTCP.
>>>
>>>
>>> What are your thoughts?
>>
>> There are a couple of approaches we could use to help ATS and other projects as upstream MPTCP evolves.
>>
>> One is that we should pick a likely-to-get-merged value for IPPROTO_MPTCP. The current prototype uses 99, but the IANA defines that as "any private encryption scheme). We had earlier prototypes with 262 (IPPROTO_TCP | 0x100) since the socket API uses 'int proto' instead of a single byte. ATS could consider a runtime-configurable proto value for MPTCP to help with this.
>
> Yes, let's pick a "likely-to-get-merged" value ASAP. Do you think 262 could work out?
>

I'll test it out and send the patches, so they're ready to merge if we 
have a consensus.

>> My impression is that the maintainers would not want a "MPTCP API version" sysctl where you'd read a number from a file and have v0, v1, v2 etc API since it would mainly exist to accomodate non-upstream kernels. The upstream API (once MPTCP is merged) is not supposed to change in ways that break userspace, after all.
>>
>> Given that the MPTCP features will get added to the upstream kernel over time, we may have an easier time justifying a MPTCP feature flag sysctl, kind of like the CPU flags in /proc/cpuinfo. That could indicate support for things like TFO, netlink path management, break-before-make, customizable schedulers, etc. The combination of features may be enough for userspace programs to figure out if they can make use of that MPTCP implementation.
>
> That sounds like a fine approach to me.
>
> In general, I am wondering, how do userspace applications handle runtime feature-checking as they may be running on different kernels? Is it always a try-and-error approach?

A project I am involved in makes use of recent additions to the keyctl API 
and various crypto algorithms on AF_ALG sockets. The checks are basically 
trial and error, if the feature is not there we expect EOPNOTSUPP from 
keyctl or bind() to fail:

https://git.kernel.org/pub/scm/libs/ell/ell.git/tree/ell/key.c#n788

https://git.kernel.org/pub/scm/libs/ell/ell.git/tree/ell/cipher.c#n621

--
Mat Martineau
Intel

Re: [MPTCP] MPTCP-API

[Paolo Abeni]

[-- Attachment #1: Type: text/plain, Size: 806 bytes --]

Hi again,

On Tue, 2019-06-11 at 14:13 -0700, Christoph Paasch wrote:
> So, we are wondering how we can do binary compatibility without having to
> wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl
> that indicates the "API-version" ? That way, upon startup ATS can simply
> check the API-version and if it doesn't match, just disable MPTCP.
> 
> 
> What are your thoughts?

Reporting some additional thoughts from Florian. He can't reply
directly ATM, but told me his opinion:

As far as upstream is concerned api should not change, so he is not
sure why any of the proposals would fly.  

AF_PACKET is really a special case, we should not take inspiration from
it, and there is no no abi version in nftables as far as he knows.

Paolo (voicing Florian ;)

Re: [MPTCP] MPTCP-API

[Paolo Abeni]

[-- Attachment #1: Type: text/plain, Size: 1196 bytes --]

On Tue, 2019-06-11 at 14:13 -0700, Christoph Paasch wrote:
> Now, ATS will be doing a 9.0.0-release soon that will be the first
> release with this MPTCP-option in the ATS-config.

This is probably a question more for Leif, how far/near is such
release?

> So, we are wondering how we can do binary compatibility without having to
> wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl
> that indicates the "API-version" ? That way, upon startup ATS can simply
> check the API-version and if it doesn't match, just disable MPTCP.
> 
> What are your thoughts?

I think we may look at other pieces of the kernel/networking stack that
alreday implement "versioning"

- AF_PACKET. A getsockopt() is used to retrieve the API version
(specifying e.g. the metadata layout). There is an additional possible
difference: packet sockets allow user space to set the API version via
setsockopt().

- nftables exposes an API version number via netlink (@Florian can you
give more details?)

- schedstat exposes the stats layout version number via a field in the
same procfs entry

- Others?!?

First 2 approaches look viable to me.

Cheers,

Paolo

Re: [MPTCP] MPTCP-API

[Christoph Paasch]

[-- Attachment #1: Type: text/plain, Size: 2846 bytes --]

Hello,

> On Jun 11, 2019, at 3:24 PM, Mat Martineau <mathew.j.martineau(a)linux.intel.com> wrote:
> 
> 
> Hi Christoph and Leif,
> 
> On Tue, 11 Jun 2019, Christoph Paasch wrote:
> 
>> I am bringing up something that we already once discussed quite a while
>> back. Apache Traffic Server has an option to enable MPTCP (currently,
>> through the socket-option MPTCP_ENABLED that is exposed on the
>> multipath-tcp.org kernel). Leif (in CC) is the main contributor & maintainer
>> of ATS.
>> 
>> Now, ATS will be doing a 9.0.0-release soon that will be the first release
>> with this MPTCP-option in the ATS-config.
>> 
>> We want to make sure that it would also work with the upcoming upstream MPTCP
>> implementation. So, we can easily make this use IPPROTO_MPTCP. However,
>> there currently isn't a 100% guarantee that the API is stable (as upstream
>> might request changes).
>> 
>> So, we are wondering how we can do binary compatibility without having to
>> wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl
>> that indicates the "API-version" ? That way, upon startup ATS can simply
>> check the API-version and if it doesn't match, just disable MPTCP.
>> 
>> 
>> What are your thoughts?
> 
> There are a couple of approaches we could use to help ATS and other projects as upstream MPTCP evolves.
> 
> One is that we should pick a likely-to-get-merged value for IPPROTO_MPTCP. The current prototype uses 99, but the IANA defines that as "any private encryption scheme). We had earlier prototypes with 262 (IPPROTO_TCP | 0x100) since the socket API uses 'int proto' instead of a single byte. ATS could consider a runtime-configurable proto value for MPTCP to help with this.

Yes, let's pick a "likely-to-get-merged" value ASAP. Do you think 262 could work out?

> My impression is that the maintainers would not want a "MPTCP API version" sysctl where you'd read a number from a file and have v0, v1, v2 etc API since it would mainly exist to accomodate non-upstream kernels. The upstream API (once MPTCP is merged) is not supposed to change in ways that break userspace, after all.
> 
> Given that the MPTCP features will get added to the upstream kernel over time, we may have an easier time justifying a MPTCP feature flag sysctl, kind of like the CPU flags in /proc/cpuinfo. That could indicate support for things like TFO, netlink path management, break-before-make, customizable schedulers, etc. The combination of features may be enough for userspace programs to figure out if they can make use of that MPTCP implementation.

That sounds like a fine approach to me.

In general, I am wondering, how do userspace applications handle runtime feature-checking as they may be running on different kernels? Is it always a try-and-error approach?


Christoph

Re: [MPTCP] MPTCP-API

[Mat Martineau]

[-- Attachment #1: Type: text/plain, Size: 2424 bytes --]


Hi Christoph and Leif,

On Tue, 11 Jun 2019, Christoph Paasch wrote:

> I am bringing up something that we already once discussed quite a while
> back. Apache Traffic Server has an option to enable MPTCP (currently,
> through the socket-option MPTCP_ENABLED that is exposed on the
> multipath-tcp.org kernel). Leif (in CC) is the main contributor & maintainer
> of ATS.
>
> Now, ATS will be doing a 9.0.0-release soon that will be the first release
> with this MPTCP-option in the ATS-config.
>
> We want to make sure that it would also work with the upcoming upstream MPTCP
> implementation. So, we can easily make this use IPPROTO_MPTCP. However,
> there currently isn't a 100% guarantee that the API is stable (as upstream
> might request changes).
>
> So, we are wondering how we can do binary compatibility without having to
> wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl
> that indicates the "API-version" ? That way, upon startup ATS can simply
> check the API-version and if it doesn't match, just disable MPTCP.
>
>
> What are your thoughts?

There are a couple of approaches we could use to help ATS and other 
projects as upstream MPTCP evolves.

One is that we should pick a likely-to-get-merged value for IPPROTO_MPTCP. 
The current prototype uses 99, but the IANA defines that as "any private 
encryption scheme). We had earlier prototypes with 262 (IPPROTO_TCP | 
0x100) since the socket API uses 'int proto' instead of a single byte. ATS 
could consider a runtime-configurable proto value for MPTCP to help with 
this.


My impression is that the maintainers would not want a "MPTCP API version" 
sysctl where you'd read a number from a file and have v0, v1, v2 etc API 
since it would mainly exist to accomodate non-upstream kernels. The 
upstream API (once MPTCP is merged) is not supposed to change in ways that 
break userspace, after all.

Given that the MPTCP features will get added to the upstream kernel over 
time, we may have an easier time justifying a MPTCP feature flag sysctl, 
kind of like the CPU flags in /proc/cpuinfo. That could indicate support 
for things like TFO, netlink path management, break-before-make, 
customizable schedulers, etc. The combination of features may be enough 
for userspace programs to figure out if they can make use of that MPTCP 
implementation.


--
Mat Martineau
Intel

[MPTCP] MPTCP-API

[Christoph Paasch]

[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]

Hello everyone,


I am bringing up something that we already once discussed quite a while
back. Apache Traffic Server has an option to enable MPTCP (currently,
through the socket-option MPTCP_ENABLED that is exposed on the
multipath-tcp.org kernel). Leif (in CC) is the main contributor & maintainer
of ATS.

Now, ATS will be doing a 9.0.0-release soon that will be the first release
with this MPTCP-option in the ATS-config.

We want to make sure that it would also work with the upcoming upstream MPTCP
implementation. So, we can easily make this use IPPROTO_MPTCP. However,
there currently isn't a 100% guarantee that the API is stable (as upstream
might request changes).

So, we are wondering how we can do binary compatibility without having to
wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl
that indicates the "API-version" ? That way, upon startup ATS can simply
check the API-version and if it doesn't match, just disable MPTCP.


What are your thoughts?


Thanks,
Christoph