XDP (eXpress Data Path) documentation

XDP (eXpress Data Path) documentation

[Jesper Dangaard Brouer via iovisor-dev]

Hi all,

As promised, I've started documenting the XDP eXpress Data Path):

 [1] https://prototype-kernel.readthedocs.io/en/latest/networking/XDP/index.html

IMHO the documentation have reached a stage where it is useful for the
XDP project, BUT I request collaboration on improving the documentation
from all. (Native English speakers are encouraged to send grammar fixes ;-))

You wouldn't believe it: But this pretty looking documentation actually
follows the new Kernel documentation format.  It is actually just
".rst" text files stored in my github repository under kernel/Documentation [2]

 [2] https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/Documentation

Thus, just git clone my repository and started editing and send me
patches (or github pull requests). Like:

 $ git clone https://github.com/netoptimizer/prototype-kernel
 $ cd prototype-kernel/kernel/Documentation/
 $ make html
 $ firefox _build/html/index.html &

This new documentation format combines the best of two worlds, pretty
online browser documentation with almost plain text files, and changes
being tracked via git commits [3] (and auto git hooks to generate the
readthedocs.org page). You got to love it! :-)

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

[3] https://github.com/netoptimizer/prototype-kernel/commits/master

Re: XDP (eXpress Data Path) documentation

[Tom Herbert]

On Tue, Sep 20, 2016 at 2:08 AM, Jesper Dangaard Brouer
<brouer@redhat.com> wrote:
> Hi all,
>
> As promised, I've started documenting the XDP eXpress Data Path):
>
>  [1] https://prototype-kernel.readthedocs.io/en/latest/networking/XDP/index.html
>
> IMHO the documentation have reached a stage where it is useful for the
> XDP project, BUT I request collaboration on improving the documentation
> from all. (Native English speakers are encouraged to send grammar fixes ;-))
>
Hi Jesper,

Thanks for taking the initiative on the this, The document reads more
like a design doc than description right now, that's probably okay
since we could use a design doc.

Under "Important to understand" there are some disclaimers that XDP
does not implement qdiscs or BQL and fairness otherwise. This is true
for it's own traffic, but it does not (or at least should not) affect
these mechanisms or normal stack traffic running simultaneously. I
think we've made assumptions about fairness between XDP and non-XDP
queues, we probably want to clarify fairness (and also validate
whatever assumptions we've made with testing).

Thanks,
Tom

> You wouldn't believe it: But this pretty looking documentation actually
> follows the new Kernel documentation format.  It is actually just
> ".rst" text files stored in my github repository under kernel/Documentation [2]
>
>  [2] https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/Documentation
>
> Thus, just git clone my repository and started editing and send me
> patches (or github pull requests). Like:
>
>  $ git clone https://github.com/netoptimizer/prototype-kernel
>  $ cd prototype-kernel/kernel/Documentation/
>  $ make html
>  $ firefox _build/html/index.html &
>
> This new documentation format combines the best of two worlds, pretty
> online browser documentation with almost plain text files, and changes
> being tracked via git commits [3] (and auto git hooks to generate the
> readthedocs.org page). You got to love it! :-)
>
> --
> Best regards,
>   Jesper Dangaard Brouer
>   MSc.CS, Principal Kernel Engineer at Red Hat
>   Author of http://www.iptv-analyzer.org
>   LinkedIn: http://www.linkedin.com/in/brouer
>
> [3] https://github.com/netoptimizer/prototype-kernel/commits/master

Re: XDP (eXpress Data Path) documentation

[Jesper Dangaard Brouer via iovisor-dev]

On Wed, 21 Sep 2016 17:03:24 -0700
Tom Herbert <tom-BjP2VixgY4xUbtYUoyoikg@public.gmane.org> wrote:

> On Tue, Sep 20, 2016 at 2:08 AM, Jesper Dangaard Brouer
> <brouer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> > Hi all,
> >
> > As promised, I've started documenting the XDP eXpress Data Path):
> >
> >  [1] https://prototype-kernel.readthedocs.io/en/latest/networking/XDP/index.html
> >
> > IMHO the documentation have reached a stage where it is useful for the
> > XDP project, BUT I request collaboration on improving the documentation
> > from all. (Native English speakers are encouraged to send grammar fixes ;-))
> >  
> Hi Jesper,
> 
> Thanks for taking the initiative on the this, The document reads more
> like a design doc than description right now, that's probably okay
> since we could use a design doc.

Yes, I fully agree.

I want to state very clearly, this document is not an attempt to hijack
the XDP project and control the "spec".  This is an attempt to collaborate.
We discuss things on the mailing list, each with our own vision of the
project, and most times we reach an agreement. But nobody document this
agreement. 

Month later, we make implementation choices that goes against these
agreements, because we simply forgot.  If someone remembers, we have to
reiterate the same arguments again (like it just happened with the
XDP_ABORTED action, my mistake).  And can anybody remember the
consensus around VLANs, it never got implemented that way...

I had to start the doc project somewhere, so I dumped my own vision
into the docs, and what I could remember from upstream discussions.
I need collaboration from others to adjust and "fix" my vision of the
project, into something that becomes a common ground we all can agree
on.

If some part of the docs provoke you, good, then you have a reason to
correct and fix it.  I'll do my best to keep an very open-mind about
any changes.  This should be a very "live" document.  


> Under "Important to understand" there are some disclaimers that XDP
> does not implement qdiscs or BQL and fairness otherwise. This is true
> for it's own traffic, but it does not (or at least should not) affect
> these mechanisms or normal stack traffic running simultaneously. I
> think we've made assumptions about fairness between XDP and non-XDP
> queues, we probably want to clarify fairness (and also validate
> whatever assumptions we've made with testing).

I love people pointing out mistakes in the documentation.  I want
update this ASAP when people point it out.  I'll even do the work of
integrating and committing these changes, for people too lazy to git
clone the repo themselves.

For you section Tom, I fully agree, but I don't know how to formulate
and adjust this in the text.

For people that want to edit the docs, notice the link "Edit on GitHub"
which takes you directly to the file you need to edit...



> > You wouldn't believe it: But this pretty looking documentation actually
> > follows the new Kernel documentation format.  It is actually just
> > ".rst" text files stored in my github repository under kernel/Documentation [2]
> >
> >  [2] https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/Documentation
> >
> > Thus, just git clone my repository and started editing and send me
> > patches (or github pull requests). Like:
> >
> >  $ git clone https://github.com/netoptimizer/prototype-kernel
> >  $ cd prototype-kernel/kernel/Documentation/
> >  $ make html
> >  $ firefox _build/html/index.html &
> >
> > This new documentation format combines the best of two worlds, pretty
> > online browser documentation with almost plain text files, and changes
> > being tracked via git commits [3] (and auto git hooks to generate the
> > readthedocs.org page). You got to love it! :-)
> >
> > --
> > Best regards,
> >   Jesper Dangaard Brouer
> >   MSc.CS, Principal Kernel Engineer at Red Hat
> >   Author of http://www.iptv-analyzer.org
> >   LinkedIn: http://www.linkedin.com/in/brouer
> >
> > [3] https://github.com/netoptimizer/prototype-kernel/commits/master  

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

Re: XDP (eXpress Data Path) documentation

[Alexei Starovoitov via iovisor-dev]

On Tue, Sep 20, 2016 at 11:08:44AM +0200, Jesper Dangaard Brouer via iovisor-dev wrote:
> Hi all,
> 
> As promised, I've started documenting the XDP eXpress Data Path):
> 
>  [1] https://prototype-kernel.readthedocs.io/en/latest/networking/XDP/index.html
> 
> IMHO the documentation have reached a stage where it is useful for the
> XDP project, BUT I request collaboration on improving the documentation
> from all. (Native English speakers are encouraged to send grammar fixes ;-))
> 
> You wouldn't believe it: But this pretty looking documentation actually
> follows the new Kernel documentation format.  It is actually just
> ".rst" text files stored in my github repository under kernel/Documentation [2]
> 
>  [2] https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/Documentation

Thanks so much for doing it. This is great start!
Some minor editing is needed here and there.
To make it into official doc do you mind preparing a patch for Jon's doc tree ?
If you think the doc is too volatile and not suitable for kernel.org,
another alternative is to host it on https://github.com/iovisor
since it's LF collaborative project it won't disappear suddenly.
You can be a maintainer of that repo if you like.

Re: XDP (eXpress Data Path) documentation

[Jesper Dangaard Brouer via iovisor-dev]

On Tue, 20 Sep 2016 19:47:07 -0700
Alexei Starovoitov <alexei.starovoitov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> On Tue, Sep 20, 2016 at 11:08:44AM +0200, Jesper Dangaard Brouer via iovisor-dev wrote:
> > Hi all,
> > 
> > As promised, I've started documenting the XDP eXpress Data Path):
> > 
> >  [1] https://prototype-kernel.readthedocs.io/en/latest/networking/XDP/index.html
> > 
> > IMHO the documentation have reached a stage where it is useful for the
> > XDP project, BUT I request collaboration on improving the documentation
> > from all. (Native English speakers are encouraged to send grammar fixes ;-))
> > 
> > You wouldn't believe it: But this pretty looking documentation actually
> > follows the new Kernel documentation format.  It is actually just
> > ".rst" text files stored in my github repository under kernel/Documentation [2]
> > 
> >  [2] https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/Documentation  
> 
> Thanks so much for doing it. This is great start!
> Some minor editing is needed here and there.
> To make it into official doc do you mind preparing a patch for Jon's doc tree ?
> If you think the doc is too volatile and not suitable for kernel.org,
> another alternative is to host it on https://github.com/iovisor
> since it's LF collaborative project it won't disappear suddenly.
> You can be a maintainer of that repo if you like.

I do see this as kernel documentation that eventually should end-up in
Jon's doc tree.  Right now it is too volatile.  Once XDP have
"stabilized" some more, I plan to push/submit the *relevant* pieces
into the kernel.  E.g. I plan to have a "proposals" section, which is
not meant upstream doc as it is an intermediate specification step
before implementing, after which it should move to another doc section.
Likewise some of the use-case documents might be "rejected" before
reaching upstream doc.

The reason I've not created a separate repository for XDP doc only, is
because I also plan to document other parts of the kernel in this
repo[3], not just XDP.  Like my page_pool work.  The documentation is
not really official documentation before it reach a kernel git tree,
together with the code it documents.

I hope this approach can help us document while developing, and
turn email discussions into specifications.  Like I forgot the
XDP_ABORTED not warning argument, which you had to re-iterate, but now
it is documented[4][5].

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

[3] https://github.com/netoptimizer/prototype-kernel/
[4] https://github.com/netoptimizer/prototype-kernel/commit/a4e60e2d7a894
[5] https://prototype-kernel.readthedocs.io/en/latest/networking/XDP/implementation/userspace_api.html#troubleshooting-and-monitoring

Re: XDP (eXpress Data Path) documentation

[Jesper Dangaard Brouer via iovisor-dev]

On Tue, 20 Sep 2016 11:08:44 +0200 Jesper Dangaard Brouer <brouer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> As promised, I've started documenting the XDP eXpress Data Path):
> 
>  [1] https://prototype-kernel.readthedocs.io/en/latest/networking/XDP/index.html
> 
> IMHO the documentation have reached a stage where it is useful for the
> XDP project, BUT I request collaboration on improving the documentation
> from all. (Native English speakers are encouraged to send grammar fixes ;-))

I want to publicly thanks Edward Cree for being the first contributor
to the XDP documentation with formulation and grammar fixes.

Pulled and pushed:
 https://github.com/netoptimizer/prototype-kernel/commit/fb6a3de95

Thanks!
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer