Web10g TCP statistics patch - mainlining into kernel?

Web10g TCP statistics patch - mainlining into kernel?

[Valdis Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 862 bytes --]

I had a user who's working on tuning high-performance network file systems what
the chances of upstreaming the Web10G patch to provide the RFC4898 TCP Extended
Statistics MIB via netlink.

Yes, it's a tad on the intrusive side, and there's performance costs attached -
but so are a lot of *other* things that people use all the time for kernel
debugging, and it's a zero-hit thing for people who don't choose to configure
it into their kernel.  The added detailed status available from this will be
useful for people who are doing tuning and development (consider how useful
this would have been for the people who wrote the codel line discipline as part
of the bufferbloat project).

I'm willing to do the not-so-heavy lifting of getting the existing patch
cleaned up to upstream standards and sheparded through the process, if there's
any interest at all....


[-- Attachment #2: Type: application/pgp-signature, Size: 865 bytes --]

Re: [Web10g-user] Web10g TCP statistics patch - mainlining into kernel?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 884 bytes --]

On Fri, 25 Jan 2013 10:38:59 -0800, Dominic Hamon said:
> Hello Valdis
>
> I actually just finished patching a fork from the kernel github repo here:
> https://github.com/dominic-mlab/linux with a view to pushing it up. I
> haven't pushed a patch upstream before, so any guidance is welcome.

A quick browse over that git repo looks like it's unfortunately a tad
busticated - you've apparently got development on the 'master' branch, instead
of starting a separate branch.  This effectively means that it can't be easily
pulled upstream.  Fortunately, you only have one commit against it so
far, so you can probably recover by rejecting that commit so you're back
to a Linus tree, and then create a branch and do subsequent development there.

Question for the netdev people - you prefer pull requests based off
a Linus tree, or linux-next, davem/(net,net-next), or some other tree?


[-- Attachment #2: Type: application/pgp-signature, Size: 865 bytes --]

Re: [Web10g-user] Web10g TCP statistics patch - mainlining into kernel?

[rapier]

Hi,

My name is Chris Rapier and I'm on the Web10G dev team. We are 
interested in moving this into consideration for the mainline Linux 
kernel, in fact it's the primary goal of this project. We haven't 
brought this to the linux kernel community as of yet as we've not 
completed the quantification of performance/memory impact versus a 
vanilla baseline as of yet. Including the KIS we also have to test the 
impact the DKLM imposes (taking into account measurement granularity, 
number of concurrent connections, maximum throughput for a single 
stream, etc). Then we need to compare that against Web100 and write a 
paper. We're looking at around a 2 month time frame for that.

That being said, we welcome any and all insight and help on this that we 
can get from the community. Our desire to get the tests and paper worked 
out before bringing it to the kernel community was to simply make sure 
we would be able to answer any questions that might come up. 
Additionally, if you have any metrics of the impact of the KIS, DKLM, or 
any combination thereof we'd love to see them. The more data the better 
(for some definitions of better).

I and the rest of the development team are more than happy to answer any 
questions, address concerns, explain our thinking, discuss possible 
applications for this data, etc with anyone interested.

The web100-user list is probably the best way (for us at least) to do 
this. You do need to subscribe to post at 
https://lists.psc.edu/mailman/listinfo/web10g-user but I'll do my best 
to add any relevant posts that get caught up in the filter.

Thanks for your interest and I look forward to talking to anyone 
interested in our work.

Chris Rapier
Pittsburgh Supercomputing Center



On 1/25/13 1:31 PM, Valdis Kletnieks wrote:
> I had a user who's working on tuning high-performance network file systems what
> the chances of upstreaming the Web10G patch to provide the RFC4898 TCP Extended
> Statistics MIB via netlink.
>
> Yes, it's a tad on the intrusive side, and there's performance costs attached -
> but so are a lot of *other* things that people use all the time for kernel
> debugging, and it's a zero-hit thing for people who don't choose to configure
> it into their kernel.  The added detailed status available from this will be
> useful for people who are doing tuning and development (consider how useful
> this would have been for the people who wrote the codel line discipline as part
> of the bufferbloat project).
>
> I'm willing to do the not-so-heavy lifting of getting the existing patch
> cleaned up to upstream standards and sheparded through the process, if there's
> any interest at all....
>
>
>
> _______________________________________________
> Web10g-user mailing list
> Web10g-user@web10g.org
> https://lists.psc.edu/mailman/listinfo/web10g-user
>
> To UNSUBSCRIBE visit https://lists.psc.edu/mailman/unsubscribe/web10g-user
>

Re: [Web10g-user] Web10g TCP statistics patch - mainlining into kernel?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1013 bytes --]

On Fri, 25 Jan 2013 14:34:57 -0500, rapier said:

> My name is Chris Rapier and I'm on the Web10G dev team. We are
> interested in moving this into consideration for the mainline Linux
> kernel, in fact it's the primary goal of this project. We haven't
> brought this to the linux kernel community as of yet as we've not
> completed the quantification of performance/memory impact versus a
> vanilla baseline as of yet.

I already looked over the patch and it looks *fairly* sane.  Having said
that, the best way to proceed for the performance side is probably to
get some rough ballpark numbers and then get the patch into a state
that's upstream-able before doing the final measurements (because there's
actually a high likelihood that the final numbers will end up being
dependent on the exact details of the patch, plus having more eyeballs on
it from the netdev side may shake out better approaches).

Sorry if you've already done all that - I didn't see much evidence of
it from the web10g-user archives...

[-- Attachment #2: Type: application/pgp-signature, Size: 865 bytes --]

Re: [Web10g-user] Web10g TCP statistics patch - mainlining into kernel?

[rapier]

On 1/25/13 2:56 PM, Valdis.Kletnieks@vt.edu wrote:

> I already looked over the patch and it looks *fairly* sane.  Having said
> that, the best way to proceed for the performance side is probably to
> get some rough ballpark numbers and then get the patch into a state
> that's upstream-able before doing the final measurements (because there's
> actually a high likelihood that the final numbers will end up being
> dependent on the exact details of the patch, plus having more eyeballs on
> it from the netdev side may shake out better approaches).
 >
> Sorry if you've already done all that - I didn't see much evidence of
> it from the web10g-user archives...

The user side of the list has been pretty quiet to be sure. Most of the 
discussion happens internal to the team. The primary KIS developer is 
out the office until next week but I'll try to make sure he addresses 
this as soon as he can. I agree entirely with the more eyes approach and 
none of us have a lot of ego sunk into the details of the code in and of 
itself. As researchers, more than anything else, we want the concept 
moved forward. We believe, strongly, that having these metrics in the 
mainline can really do a lot to help the entire user community and spur 
some really neat development.

Chris

Re: [Web10g-user] Web10g TCP statistics patch - mainlining into kernel?

[Tom Herbert]

> The user side of the list has been pretty quiet to be sure. Most of the
> discussion happens internal to the team. The primary KIS developer is out
> the office until next week but I'll try to make sure he addresses this as
> soon as he can. I agree entirely with the more eyes approach and none of us
> have a lot of ego sunk into the details of the code in and of itself. As
> researchers, more than anything else, we want the concept moved forward. We
> believe, strongly, that having these metrics in the mainline can really do a
> lot to help the entire user community and spur some really neat development.
>
> Chris
>
Hi Chris, Valdis,

This is certainly interesting, and something that we'd really want to
try to get in upstream Linux since it's is so intrusive in the stack
(we have actually done quite a bit of work in this area already that
might be applicable).  Posting patches against davem's net-next branch
seems appropriate and you could certainly post them as RFC to start
some conversation.

Tom

> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Web10g TCP statistics patch - mainlining into kernel?

[Ben Hutchings]

On Fri, 2013-01-25 at 13:31 -0500, Valdis Kletnieks wrote:
[...]
> it's a zero-hit thing for people who don't choose to configure
> it into their kernel.
[...]

People with high-overhead changes always say this, but before long
distributions will be expected to enable it and then everyone pays the
price.  So I think you'll have to work on limiting the run-time overhead
when it's enabled at compile time but the user doesn't care about it.
Possibly it can be a run-time option?  (I haven't looked at the code and
don't expect to have time to do so.)

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: Web10g TCP statistics patch - mainlining into kernel?

[David Miller]

From: Ben Hutchings <bhutchings@solarflare.com>
Date: Tue, 29 Jan 2013 17:49:15 +0000

> On Fri, 2013-01-25 at 13:31 -0500, Valdis Kletnieks wrote:
> [...]
>> it's a zero-hit thing for people who don't choose to configure
>> it into their kernel.
> [...]
> 
> People with high-overhead changes always say this, but before long
> distributions will be expected to enable it and then everyone pays the
> price.  So I think you'll have to work on limiting the run-time overhead
> when it's enabled at compile time but the user doesn't care about it.
> Possibly it can be a run-time option?  (I haven't looked at the code and
> don't expect to have time to do so.)

+1

The run time overhead must be zero when the feature is configured into
the kernel, and this requirement is non-negotiable.

Re: [Web10g-user] Web10g TCP statistics patch - mainlining into kernel?

[rapier]

On 1/29/13 12:49 PM, Ben Hutchings wrote:
> On Fri, 2013-01-25 at 13:31 -0500, Valdis Kletnieks wrote:
> [...]
>> it's a zero-hit thing for people who don't choose to configure
>> it into their kernel.
> [...]
>
> People with high-overhead changes always say this, but before long
> distributions will be expected to enable it and then everyone pays the
> price.  So I think you'll have to work on limiting the run-time overhead
> when it's enabled at compile time but the user doesn't care about it.
> Possibly it can be a run-time option?  (I haven't looked at the code and
> don't expect to have time to do so.)

I should point out that Web10G is two stage solution. The first is the 
incorporation of the kernel instrument set (KIS, RFC 4898) into the 
stack. This is a pretty straightforward process where (basically) 
counters are incremented based on the existing events within the stack. 
At this point the data is still in kernel space. To move it up to the 
user we rely on a DKLM ABI based on netlink/genetlink. Since the ABI 
isn't loaded by default the question that the Web10G team is working on 
is how to quantify what sort of overhead the KIS actually imposes. We 
hope to have this information available to the community in the next month.

It's our hope that the KIS has negligible impact making it suitable for 
inclusion. The last thing we want to do is impact the performance of the 
kernel in production environments. The admin can choose to load the DKLM 
at their discretion. We'll be quantifying the impact a loaded DKLM has 
so people can make informed choices about this. As a note, since the KIS 
and the ABI are discrete components anyone could build a more efficient 
ABI once the KIS exists. At this time we are basically just looking at 
how to implement RFC4898 in a way that makes sense to the community as a 
whole.

Chris Rapier