* netem: the reorder discussion
@ 2013-08-23 14:16 Teco Boot
2013-08-23 15:12 ` Eric Dumazet
0 siblings, 1 reply; 6+ messages in thread
From: Teco Boot @ 2013-08-23 14:16 UTC (permalink / raw)
To: Eric Dumazet, Ferry Huberts; +Cc: Netem, netdev, core-users
I had reorder problems with netem qdisc. I make use of the NRL CORE network emulator, which sets up virtual routers and links, build with network namespaces and netem. Typical CORE config for a n1<-->n2 link with rate (1024Kbps), delay (50ms) and no jitter:
qdisc tbf 1: dev n1.eth0.222 root refcnt 2 rate 1024Kbit burst 2999b lat 488.6ms
qdisc netem 10: dev n1.eth0.222 parent 1:1 limit 1000 delay 50.0ms
qdisc tbf 1: dev n2.eth0.222 root refcnt 2 rate 1024Kbit burst 2999b lat 488.6ms
qdisc netem 10: dev n2.eth0.222 parent 1:1 limit 1000 delay 50.0ms
I added jitter to this bi-directional link, e.g. 20ms. Now the delay for each packet is 50ms +/- 20ms is 30ms to 70ms.
However, this has some unexpected results: packets may be reordered. That is because the actual delay is calculated for each packet. Some packets have a larger delay, some have a smaller. If the inter-packet spacing is smaller than the time differences set by netem (up to 2x configured jitter), packets are reordered. In this example with +/20ms, this is the case with packet rate larger than 25pps. Reordering has bad effects on transport protocol throughputs. Reordering is less common on the Internet, so I don't want to emulate such. I don't say there is no reordering, I just say I don't want this netem behavior.
The netem guide mentions this unexpected results. It was caused by a change in version 1.1 (2.6.15). For people like me that do not want this, there is a work-around mentioned. However, this doesn't work anymore since somewhere before 2.6.31.
http://www.linuxfoundation.org/collaborate/workgroups/networking/netem
https://lists.linux-foundation.org/pipermail/netem/2011-April/001507.html
So what to do? The recent patch from Eric Dumazet eliminates jitter_reordering when netem rate is configured. Maybe not the clean approach, but acceptable. Ferry his patch eliminates jitter_reordering. Some people may use reorder by jitter, so this "feature" should not get removed. So Ferry his patch is not accepted. On the other hand, people like me are very confused by current behavior.
What we could do is keep existing features and describe what netem currently does. That is:
- netem with delay and jitter may reorder packets, if inter packet spacing is smaller than jitter
- reordering caused by delay and jitter can be turned off by using netem rate. rate can be set to very high value is no shaping is wanted.
Are we OK with this approach?
Adjust the linux-foundation & man page info? Who can do this?
Teco
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: netem: the reorder discussion
2013-08-23 14:16 netem: the reorder discussion Teco Boot
@ 2013-08-23 15:12 ` Eric Dumazet
2013-08-23 15:44 ` Stephen Hemminger
0 siblings, 1 reply; 6+ messages in thread
From: Eric Dumazet @ 2013-08-23 15:12 UTC (permalink / raw)
To: Teco Boot; +Cc: Ferry Huberts, Netem, netdev, core-users
On Fri, 2013-08-23 at 16:16 +0200, Teco Boot wrote:
> I had reorder problems with netem qdisc. I make use of the NRL CORE
> network emulator, which sets up virtual routers and links, build with
> network namespaces and netem. Typical CORE config for a n1<-->n2 link
> with rate (1024Kbps), delay (50ms) and no jitter:
> qdisc tbf 1: dev n1.eth0.222 root refcnt 2 rate 1024Kbit burst 2999b
> lat 488.6ms
> qdisc netem 10: dev n1.eth0.222 parent 1:1 limit 1000 delay 50.0ms
> qdisc tbf 1: dev n2.eth0.222 root refcnt 2 rate 1024Kbit burst 2999b
> lat 488.6ms
> qdisc netem 10: dev n2.eth0.222 parent 1:1 limit 1000 delay 50.0ms
>
> I added jitter to this bi-directional link, e.g. 20ms. Now the delay
> for each packet is 50ms +/- 20ms is 30ms to 70ms.
> However, this has some unexpected results: packets may be reordered.
> That is because the actual delay is calculated for each packet. Some
> packets have a larger delay, some have a smaller. If the inter-packet
> spacing is smaller than the time differences set by netem (up to 2x
> configured jitter), packets are reordered. In this example with
> +/20ms, this is the case with packet rate larger than 25pps.
> Reordering has bad effects on transport protocol throughputs.
> Reordering is less common on the Internet, so I don't want to emulate
> such. I don't say there is no reordering, I just say I don't want this
> netem behavior.
>
> The netem guide mentions this unexpected results. It was caused by a
> change in version 1.1 (2.6.15). For people like me that do not want
> this, there is a work-around mentioned. However, this doesn't work
> anymore since somewhere before 2.6.31.
> http://www.linuxfoundation.org/collaborate/workgroups/networking/netem
> https://lists.linux-foundation.org/pipermail/netem/2011-April/001507.html
>
> So what to do? The recent patch from Eric Dumazet eliminates
> jitter_reordering when netem rate is configured. Maybe not the clean
> approach, but acceptable. Ferry his patch eliminates
> jitter_reordering. Some people may use reorder by jitter, so this
> "feature" should not get removed. So Ferry his patch is not accepted.
> On the other hand, people like me are very confused by current
> behavior.
>
> What we could do is keep existing features and describe what netem
> currently does. That is:
> - netem with delay and jitter may reorder packets, if inter packet
> spacing is smaller than jitter
> - reordering caused by delay and jitter can be turned off by using
> netem rate. rate can be set to very high value is no shaping is
> wanted.
As long as one can define the expected behavior, you can add whatever
new netem parameter.
One could envision adding flow separation (skb->sk or rxhashing), so
that each flow can have his own local queue, to guarantee no reorders
per flow _if_ this is needed, even if per flow delays/jitter is/are
configured.
We also use netem to test on large scale, where the reordering stuff
needs fixes in transport stacks (And yes, we are working on TCP stack
to permit higher levels of reorders)
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: netem: the reorder discussion
2013-08-23 15:12 ` Eric Dumazet
@ 2013-08-23 15:44 ` Stephen Hemminger
2013-08-23 15:53 ` Teco Boot
0 siblings, 1 reply; 6+ messages in thread
From: Stephen Hemminger @ 2013-08-23 15:44 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Teco Boot, Ferry Huberts, Netem, netdev, core-users
On Fri, 23 Aug 2013 08:12:36 -0700
Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Fri, 2013-08-23 at 16:16 +0200, Teco Boot wrote:
> > I had reorder problems with netem qdisc. I make use of the NRL CORE
> > network emulator, which sets up virtual routers and links, build with
> > network namespaces and netem. Typical CORE config for a n1<-->n2 link
> > with rate (1024Kbps), delay (50ms) and no jitter:
> > qdisc tbf 1: dev n1.eth0.222 root refcnt 2 rate 1024Kbit burst 2999b
> > lat 488.6ms
> > qdisc netem 10: dev n1.eth0.222 parent 1:1 limit 1000 delay 50.0ms
> > qdisc tbf 1: dev n2.eth0.222 root refcnt 2 rate 1024Kbit burst 2999b
> > lat 488.6ms
> > qdisc netem 10: dev n2.eth0.222 parent 1:1 limit 1000 delay 50.0ms
> >
> > I added jitter to this bi-directional link, e.g. 20ms. Now the delay
> > for each packet is 50ms +/- 20ms is 30ms to 70ms.
> > However, this has some unexpected results: packets may be reordered.
> > That is because the actual delay is calculated for each packet. Some
> > packets have a larger delay, some have a smaller. If the inter-packet
> > spacing is smaller than the time differences set by netem (up to 2x
> > configured jitter), packets are reordered. In this example with
> > +/20ms, this is the case with packet rate larger than 25pps.
> > Reordering has bad effects on transport protocol throughputs.
> > Reordering is less common on the Internet, so I don't want to emulate
> > such. I don't say there is no reordering, I just say I don't want this
> > netem behavior.
> >
> > The netem guide mentions this unexpected results. It was caused by a
> > change in version 1.1 (2.6.15). For people like me that do not want
> > this, there is a work-around mentioned. However, this doesn't work
> > anymore since somewhere before 2.6.31.
> > http://www.linuxfoundation.org/collaborate/workgroups/networking/netem
> > https://lists.linux-foundation.org/pipermail/netem/2011-April/001507.html
> >
> > So what to do? The recent patch from Eric Dumazet eliminates
> > jitter_reordering when netem rate is configured. Maybe not the clean
> > approach, but acceptable. Ferry his patch eliminates
> > jitter_reordering. Some people may use reorder by jitter, so this
> > "feature" should not get removed. So Ferry his patch is not accepted.
> > On the other hand, people like me are very confused by current
> > behavior.
> >
> > What we could do is keep existing features and describe what netem
> > currently does. That is:
> > - netem with delay and jitter may reorder packets, if inter packet
> > spacing is smaller than jitter
> > - reordering caused by delay and jitter can be turned off by using
> > netem rate. rate can be set to very high value is no shaping is
> > wanted.
>
> As long as one can define the expected behavior, you can add whatever
> new netem parameter.
>
> One could envision adding flow separation (skb->sk or rxhashing), so
> that each flow can have his own local queue, to guarantee no reorders
> per flow _if_ this is needed, even if per flow delays/jitter is/are
> configured.
>
> We also use netem to test on large scale, where the reordering stuff
> needs fixes in transport stacks (And yes, we are working on TCP stack
> to permit higher levels of reorders)
I am happy with any solution that is:
* allows both always ordering and reordering based on random jitter of delay
* documented
I do get worried that people's tests get different results because of
netem behavior changes. Researchers like to have repeatable results.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: netem: the reorder discussion
2013-08-23 15:44 ` Stephen Hemminger
@ 2013-08-23 15:53 ` Teco Boot
2013-08-23 16:03 ` Eric Dumazet
0 siblings, 1 reply; 6+ messages in thread
From: Teco Boot @ 2013-08-23 15:53 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: Eric Dumazet, Ferry Huberts, Netem, netdev, core-users
Op 23 aug. 2013, om 17:44 heeft Stephen Hemminger <stephen@networkplumber.org> het volgende geschreven:
> On Fri, 23 Aug 2013 08:12:36 -0700
> Eric Dumazet <eric.dumazet@gmail.com> wrote:
>
>> On Fri, 2013-08-23 at 16:16 +0200, Teco Boot wrote:
>>> I had reorder problems with netem qdisc. I make use of the NRL CORE
>>> network emulator, which sets up virtual routers and links, build with
>>> network namespaces and netem. Typical CORE config for a n1<-->n2 link
>>> with rate (1024Kbps), delay (50ms) and no jitter:
>>> qdisc tbf 1: dev n1.eth0.222 root refcnt 2 rate 1024Kbit burst 2999b
>>> lat 488.6ms
>>> qdisc netem 10: dev n1.eth0.222 parent 1:1 limit 1000 delay 50.0ms
>>> qdisc tbf 1: dev n2.eth0.222 root refcnt 2 rate 1024Kbit burst 2999b
>>> lat 488.6ms
>>> qdisc netem 10: dev n2.eth0.222 parent 1:1 limit 1000 delay 50.0ms
>>>
>>> I added jitter to this bi-directional link, e.g. 20ms. Now the delay
>>> for each packet is 50ms +/- 20ms is 30ms to 70ms.
>>> However, this has some unexpected results: packets may be reordered.
>>> That is because the actual delay is calculated for each packet. Some
>>> packets have a larger delay, some have a smaller. If the inter-packet
>>> spacing is smaller than the time differences set by netem (up to 2x
>>> configured jitter), packets are reordered. In this example with
>>> +/20ms, this is the case with packet rate larger than 25pps.
>>> Reordering has bad effects on transport protocol throughputs.
>>> Reordering is less common on the Internet, so I don't want to emulate
>>> such. I don't say there is no reordering, I just say I don't want this
>>> netem behavior.
>>>
>>> The netem guide mentions this unexpected results. It was caused by a
>>> change in version 1.1 (2.6.15). For people like me that do not want
>>> this, there is a work-around mentioned. However, this doesn't work
>>> anymore since somewhere before 2.6.31.
>>> http://www.linuxfoundation.org/collaborate/workgroups/networking/netem
>>> https://lists.linux-foundation.org/pipermail/netem/2011-April/001507.html
>>>
>>> So what to do? The recent patch from Eric Dumazet eliminates
>>> jitter_reordering when netem rate is configured. Maybe not the clean
>>> approach, but acceptable. Ferry his patch eliminates
>>> jitter_reordering. Some people may use reorder by jitter, so this
>>> "feature" should not get removed. So Ferry his patch is not accepted.
>>> On the other hand, people like me are very confused by current
>>> behavior.
>>>
>>> What we could do is keep existing features and describe what netem
>>> currently does. That is:
>>> - netem with delay and jitter may reorder packets, if inter packet
>>> spacing is smaller than jitter
>>> - reordering caused by delay and jitter can be turned off by using
>>> netem rate. rate can be set to very high value is no shaping is
>>> wanted.
>>
>> As long as one can define the expected behavior, you can add whatever
>> new netem parameter.
>>
>> One could envision adding flow separation (skb->sk or rxhashing), so
>> that each flow can have his own local queue, to guarantee no reorders
>> per flow _if_ this is needed, even if per flow delays/jitter is/are
>> configured.
>>
>> We also use netem to test on large scale, where the reordering stuff
>> needs fixes in transport stacks (And yes, we are working on TCP stack
>> to permit higher levels of reorders)
>
> I am happy with any solution that is:
> * allows both always ordering and reordering based on random jitter of delay
> * documented
>
> I do get worried that people's tests get different results because of
> netem behavior changes. Researchers like to have repeatable results.
Questions that pops out of of my head: What is more important, understandable configuration or keep existing behavior for unmodified configuration? Do we want to reorder, even with configured rate?
My preference: make in understandable, document well and make all sensible options available.
Teco
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: netem: the reorder discussion
2013-08-23 15:53 ` Teco Boot
@ 2013-08-23 16:03 ` Eric Dumazet
2013-08-23 16:18 ` Teco Boot
0 siblings, 1 reply; 6+ messages in thread
From: Eric Dumazet @ 2013-08-23 16:03 UTC (permalink / raw)
To: Teco Boot; +Cc: Stephen Hemminger, Ferry Huberts, Netem, netdev, core-users
On Fri, 2013-08-23 at 17:53 +0200, Teco Boot wrote:
> Questions that pops out of of my head: What is more important,
> understandable configuration or keep existing behavior for unmodified
> configuration? Do we want to reorder, even with configured rate?
>
> My preference: make in understandable, document well and make all
> sensible options available.
The day 'rate' support was added, documentation became obsolete.
Its actually hard to have both rate and delay + jitters + reorders or
not.
If you can fix this properly, patches are welcome, but I suspect you
wanted to reintroduce the thing that made netem unusable for us.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: netem: the reorder discussion
2013-08-23 16:03 ` Eric Dumazet
@ 2013-08-23 16:18 ` Teco Boot
0 siblings, 0 replies; 6+ messages in thread
From: Teco Boot @ 2013-08-23 16:18 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Stephen Hemminger, Ferry Huberts, Netem, netdev, core-users
Op 23 aug. 2013 om 18:03 heeft Eric Dumazet <eric.dumazet@gmail.com> het volgende geschreven:
> On Fri, 2013-08-23 at 17:53 +0200, Teco Boot wrote:
>
>> Questions that pops out of of my head: What is more important,
>> understandable configuration or keep existing behavior for unmodified
>> configuration? Do we want to reorder, even with configured rate?
>>
>> My preference: make in understandable, document well and make all
>> sensible options available.
>
> The day 'rate' support was added, documentation became obsolete.
>
> Its actually hard to have both rate and delay + jitters + reorders or
> not.
>
> If you can fix this properly, patches are welcome, but I suspect you
> wanted to reintroduce the thing that made netem unusable for us.
No, for sure I do not want to make it unusable for you. Or anybody. The opposite is true.
I don't like the "look in souce" documentation. Or parameters with hard to understand side effects.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-08-23 16:19 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-08-23 14:16 netem: the reorder discussion Teco Boot
2013-08-23 15:12 ` Eric Dumazet
2013-08-23 15:44 ` Stephen Hemminger
2013-08-23 15:53 ` Teco Boot
2013-08-23 16:03 ` Eric Dumazet
2013-08-23 16:18 ` Teco Boot
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.