Using netem on one interface expands to all interfaces

Using netem on one interface expands to all interfaces

[Paul Hoffman]

Greetings, this is my first message to the list. In a testbed I am 
creating, I need to set different delays for packets going to different 
addresses on a single box. Because netem is only for classless qdiscs, I 
am not able to combine it with filtering, so I am forced to have 
multiple interfaces, one for each desired delay. However, I'm seeing 
something very troubling. FWIW, this is under Debian on a VM.

I use the following two commands:

# tc qdisc add dev enp0s18 root netem delay 200ms
# tc qdisc add dev enp0s8 root netem delay 1200ms

Then:

# tc qdisc list
qdisc noqueue 0: dev lo root refcnt 2
qdisc pfifo_fast 0: dev enp0s3 root refcnt 2 bands 3 priomap  1 2 2 2 1 
2 0 0 1 1 1 1 1 1 1 1
qdisc netem 8002: dev enp0s8 root refcnt 2 limit 1000 delay 1.2s
qdisc pfifo_fast 0: dev enp0s9 root refcnt 2 bands 3 priomap  1 2 2 2 1 
2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev enp0s10 root refcnt 2 bands 3 priomap  1 2 2 2 1 
2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev enp0s16 root refcnt 2 bands 3 priomap  1 2 2 2 1 
2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev enp0s17 root refcnt 2 bands 3 priomap  1 2 2 2 1 
2 0 0 1 1 1 1 1 1 1 1
qdisc netem 8001: dev enp0s18 root refcnt 2 limit 1000 delay 200.0ms
qdisc pfifo_fast 0: dev enp0s19 root refcnt 2 bands 3 priomap  1 2 2 2 1 
2 0 0 1 1 1 1 1 1 1 1

However, now pinging addresses on *any* interface on the box gets a 
1200ms delay. That is, pinging addresses on enp0s18 gets the same delay 
as was given to enp0s8, but even weirder, pinging addresses on enp0s9 
get the 1200ms delay as well.

Is this expected? Is it preventable? All clues are appreciated.

--Paul Hoffman

Re: Using netem on one interface expands to all interfaces

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 1010 bytes --]

On 02/13/2019 11:16 AM, Paul Hoffman wrote:
> However, now pinging addresses on *any* interface on the box gets a 
> 1200ms delay. That is, pinging addresses on enp0s18 gets the same delay 
> as was given to enp0s8, but even weirder, pinging addresses on enp0s9 
> get the 1200ms delay as well.
> 
> Is this expected? Is it preventable? All clues are appreciated.

I don't know if it's expected or not.

Are you sure that you're pinging the interface that you expect to be 
pinging?

Check the MAC address in the ARP / ND cache to see which physical 
interface packets are actually addressed to.

You don't show any IPs, so I don't know if you're putting multiple 
interfaces in the same subnet or using different subnets per interface.

You may need to fiddle with some /proc / sysctl settings to make sure 
Linux is actually using the interface that you expect.

Research "strong host model" for more details on what I think may be 
happening.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]

Re: Using netem on one interface expands to all interfaces

[Paul Hoffman]

Thanks for your quick response!

On 13 Feb 2019, at 11:47, Grant Taylor wrote:

> Are you sure that you're pinging the interface that you expect to be 
> pinging?

Yes.

> Check the MAC address in the ARP / ND cache to see which physical 
> interface packets are actually addressed to.

Yep.

> You don't show any IPs, so I don't know if you're putting multiple 
> interfaces in the same subnet or using different subnets per 
> interface.

Sorry, I didn't think they were needed. However...

> You may need to fiddle with some /proc / sysctl settings to make sure 
> Linux is actually using the interface that you expect.
>
> Research "strong host model" for more details on what I think may be 
> happening.

Unfortunately, that's exactly the right track. After some digging, it 
seems like the "weak host model" used by Linux/Debian is the fault here. 
I hope I can find a way to make the model strong with some sysctl 
settings, but I'm not hopeful.

--Paul Hoffman

Re: Using netem on one interface expands to all interfaces

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 731 bytes --]

On 02/13/2019 03:35 PM, Paul Hoffman wrote:
> Thanks for your quick response!

You're welcome.

> Sorry, I didn't think they were needed. However...

It's okay.

> Unfortunately, that's exactly the right track. After some digging, it 
> seems like the "weak host model" used by Linux/Debian is the fault here. 
> I hope I can find a way to make the model strong with some sysctl 
> settings, but I'm not hopeful.

I'm fairly certain that there are /proc / sysctl settings that you can 
tune to alter this.

Check the /proc/sys/net/ipv4/conf/*/arp_* files.

Every time I've looked at the kernel documentation for them, it has 
clearly explained what the value should be.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]

Re: Using netem on one interface expands to all interfaces

[Erik Auerswald]

Hi,

On Wed, Feb 13, 2019 at 02:35:18PM -0800, Paul Hoffman wrote:
> On 13 Feb 2019, at 11:47, Grant Taylor wrote:
> [...]
> 
> >You may need to fiddle with some /proc / sysctl settings to make
> >sure Linux is actually using the interface that you expect.
> >
> >Research "strong host model" for more details on what I think may
> >be happening.
> 
> Unfortunately, that's exactly the right track. After some digging,
> it seems like the "weak host model" used by Linux/Debian is the
> fault here. I hope I can find a way to make the model strong with
> some sysctl settings, but I'm not hopeful.

Some of the ARP sysctl settings should be able to help:

arp_ignore=1
  0 - (default): reply for any local target IP address, configured
                 on any interface
  1 - reply only if the target IP address is local address
                 configured on the incoming interface

That seems to me to be the most logical setting to ensure sensible
ARP replies.

arp_filter=1 seems to have a similar purpose.

arp_announce=2 seems to be more of a best-effort approach, i.e. any
interface MAC may be returned.

Please let us know if you found a solution.

Thanks,
Erik

P.S. I am no longer sure if "strong host model" vs. "weak host model"
     is the correct way to refer to this problem, as it concerns ARP,
     not IP routing / interface selection. Nowadays it seems to be
     called "ARP flux."
-- 
It's really easy to write a lousy random number generator, and it's not
at all obvious that it is lousy.
                        -- Bruce Schneier

Re: Using netem on one interface expands to all interfaces

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

On 02/14/2019 01:31 AM, Erik Auerswald wrote:
> P.S. I am no longer sure if "strong host model" vs. "weak host model" 
> is the correct way to refer to this problem, as it concerns ARP, not IP 
> routing / interface selection. Nowadays it seems to be called "ARP flux."

I seem to run into "{strong,weak} {host,end system} model" references 
monthly in the various circles that I travel in.

I can't remember the last time I heard "ARP flux".

"{strong,weak} {host,end system} model" might refer to routing and not 
directly ARP, but it is decidedly adjacent and applicable.  All be it 
possibly not technically precise.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]