traffic shaping with tc on Linux 5.4.x

traffic shaping with tc on Linux 5.4.x

[Lars Noodén]

If this is the right list to ask about tc, then I have a beginner
question about traffic shaping.  If not please point me to the correct
venue.

My questions is, given the rules below, how would I further subdivide
the SSH queue so that interactive sessions are prioritized over bulk
transfers?

The goal of the rules below are to give top priority to SSH, next
priority to HTTP/HTTPS, third priority to everything else, and, then,
with what's left over give something to IPFS.  General tips and
corrections also welcome, especially about nft instead of iptables.

Regards,
Lars

---

#!/bin/sh

PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin

if=wlp1s0

# remove existing qdiscs, classes, and filters from interface
tc qdisc del dev $if ingress
tc qdisc del dev $if root

# default class for unclassified traffic
tc qdisc replace dev $if root handle 1: htb default 30

# top level class with handle 1:1
tc class add dev $if parent 1: classid 1:1 htb rate 800kbit

# Class 1:10 is highest priority, SSH/SFTP
# Class 1:20 is next highest priority, HTTP/HTTPS
# Class 1:30 is next lowest priority, default traffic
# Class 1:40 is lowest priority but highest bandwidth, IPFS

tc class add dev $if parent 1:1 classid 1:10 htb rate 1mbit \
	ceil 200kbit prio 1
tc class add dev $if parent 1:1 classid 1:20 htb rate 1mbit \
	ceil 100kbit prio 2
tc class add dev $if parent 1:1 classid 1:30 htb rate 1mbit \
	ceil 100kbit prio 3
tc class add dev $if parent 1:1 classid 1:40 htb rate 1mbit \
	ceil 400kbit prio 4

# leaf qdisc to each child class
tc qdisc add dev $if parent 1:10 fq_codel
tc qdisc add dev $if parent 1:20 fq_codel
tc qdisc add dev $if parent 1:30 fq_codel
tc qdisc add dev $if parent 1:40 fq_codel

# add filters to prioritize traffic
tc filter add dev $if parent 1: handle 100 fw classid 1:10
tc filter add dev $if parent 1: handle 200 fw classid 1:20
tc filter add dev $if parent 1: handle 400 fw classid 1:40

# label outgoing traffic
iptables -Z; # zero counters
iptables -F; # flush (delete) rules
iptables -X; # delete all extra chains

iptables -t mangle -A OUTPUT -p tcp --match multiport \
	--sports 22 -j MARK --set-mark 100
iptables -t mangle -A OUTPUT -p tcp --match multiport \
	--sports 80,443 -j MARK --set-mark 200
iptables -t mangle -A OUTPUT -p tcp --match multiport \
	--sports 4001 -j MARK --set-mark 400

Re: traffic shaping with tc on Linux 5.4.x

[Lars Noodén]

On 2/16/21 6:42 PM, Dave Taht wrote:
> Me being me, I'd just turn on fq_codel as the qdisc, and anything
> interactive, not just ssh, would be automatically prioritized, no need
> for any further classification usually.

How would that be done?  I've been picking away at the problem over
time.  Below is the current state of my attempt at it.

> Or, if you want something that respects dscp markings, try cake, which
> is basically a one liner on the up and 4 on the down. I don't think
> you have a working tc mired line for shaping inbound above.
>
> To achieve basically everything you did above, if you have sch_cake (openwrt and
> linux 4.19 and after)
>
> tc qdisc add dev eth0 root cake bandwidth 800kbit ack-filter
> {docsis/dsl/ethernet)
>
> (cake also supports the connmark facility)
>
>
> I too am very interested in seeing ipfs perform well, and would be
> very interested in your results. However at 800kbit, oy... and you
> really need to shape inbound too.

The 800kbit cap is necessary because the shaping has to occur on a node
on the LAN not the LAN's router itself.  Above that rate, it gets in the
way of the other devices.

What I have below mostly seems to keep IPFS under control, but
establishing new SSH connections involves a very, very long delay while
IPFS is running full blast.  So I am welcoming any and all corrections.

/Lars

(Apologies for using iptables instead of nftables in this iteration.)

#!/bin/sh
PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin

if=wlan0

# remove all existing qdiscs, classes and filters from interface

tc qdisc del dev $if ingress   2>/dev/null
tc qdisc del dev $if root      2>/dev/null

# respect dscp markings using cake
tc qdisc add dev $if root cake bandwidth 800kbit ack-filter

# set default class for all unclassified traffic
tc qdisc replace dev $if root handle 1: htb default 30

# top level class with handle 1:1
# tc class add dev $if parent 1: classid 1:1 htb rate 800kbit

# Class 1:10 is highest priority path, outgoing SSH/SFTP
# Class 1:20 is outgoing SSH/SFTP bulk transfers
# Class 1:30 is next highest priority path, HTTP/HTTPS traffic
# Class 1:40 is default with next lowest priority
# Class 1:50 is lowest priority but highest total bandwidth, IPFS

tc class add dev $if parent 1:1 classid 1:10 htb \
 rate 600kbit ceil 1mbit prio 1
tc class add dev $if parent 1:1 classid 1:20 htb \
 rate 600kbit ceil 1mbit prio 2
tc class add dev $if parent 1:1 classid 1:30 htb \
 rate 100kbit ceil 1mbit prio 3
tc class add dev $if parent 1:1 classid 1:40 htb \
 rate 600kbit ceil 1mbit prio 4
tc class add dev $if parent 1:1 classid 1:50 htb \
 rate 790kbit prio 5

# leaf qdisc to each child class
tc qdisc add dev $if parent 1:10 fq_codel
tc qdisc add dev $if parent 1:20 fq_codel
tc qdisc add dev $if parent 1:30 fq_codel
tc qdisc add dev $if parent 1:40 fq_codel
tc qdisc add dev $if parent 1:50 fq_codel

# add filters to prioritize traffic
tc filter add dev $if parent 1: handle 100 fw classid 1:10
tc filter add dev $if parent 1: handle 200 fw classid 1:20
tc filter add dev $if parent 1: handle 400 fw classid 1:40
tc filter add dev $if parent 1: handle 500 fw classid 1:50

# drop whatever comes in too fast
tc qdisc add dev $if handle ffff: ingress
tc filter add dev $if parent ffff: protocol ip prio 50 u32 \
 match ip src 0.0.0.0/0 police rate 9mbit burst 9mbit drop flowid :1

# clear chains if possible
iptables -Z; # zero counters
iptables -t mangle -F; # flush (delete) rules
iptables -t mangle -X; # delete all extra chains


# label outgoing traffic
iptables -t mangle -A OUTPUT -p tcp -m tcp -m multiport --sports 22 \
	-m dscp --dscp 0x04 -m comment --comment sshinteractive \
	-j MARK --set-mark 100

iptables -t mangle -A OUTPUT -p tcp -m tcp -m multiport --sports 22 \
	-m dscp --dscp 0x02 -m comment --comment sshbulk \
	-j MARK --set-mark 200

iptables -t mangle -A OUTPUT -p tcp --match multiport \
        --sports 80,443,1965 \
        -j MARK --set-mark 400

iptables -t mangle -A OUTPUT -p tcp --match multiport --sports 4001 \
        -j MARK --set-mark 500

Re: traffic shaping with tc on Linux 5.4.x

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 696 bytes --]

On 2/16/21 9:04 AM, Erik Auerswald wrote:
> Hi,

Hi,

> You could look into the "dscp" module for iptables, and match on both 
> TCP port and DSCP markings.  OpenSSH uses different IPQoS configuration 
> settings for interactive sessions and bulk transfers.

Today I Learned that OpenSSH has IPQoS options to differentiate 
interactive vs non-interactive connections.

> You can check your settings with "ssh -G":
> 
>      $ ssh -G ssh.example.com | grep -i ipqos
>      ipqos lowdelay throughput

Search the ssh man page for IPQoS for more details.

TL;DR:  The first option is interactive and the second option is 
non-interactive.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4013 bytes --]

Re: traffic shaping with tc on Linux 5.4.x

[Dave Taht]

Me being me, I'd just turn on fq_codel as the qdisc, and anything
interactive, not just ssh, would be automatically prioritized, no need
for any further classification usually.

Or, if you want something that respects dscp markings, try cake, which
is basically a one liner on the up and 4 on the down. I don't think
you have a working tc mired line for shaping inbound above.

To achieve basically everything you did above, if you have sch_cake (openwrt and
linux 4.19 and after)

tc qdisc add dev eth0 root cake bandwidth 800kbit ack-filter
{docsis/dsl/ethernet)

(cake also supports the connmark facility)


I too am very interested in seeing ipfs perform well, and would be
very interested in your results. However at 800kbit, oy... and you
really need to shape inbound too.

Re: traffic shaping with tc on Linux 5.4.x

[Erik Auerswald]

Hi,

On Tue, Feb 16, 2021 at 02:34:20PM +0200, Lars Noodén wrote:
> [...]
> My question is, given the rules below, and if they are otherwise
> correct, how would I further subdivide the SSH queue so that interactive
> sessions are prioritized over bulk transfers?

You could look into the "dscp" module for iptables, and match on both
TCP port and DSCP markings.  OpenSSH uses different IPQoS configuration
settings for interactive sessions and bulk transfers.

You can check your settings with "ssh -G":

    $ ssh -G ssh.example.com | grep -i ipqos
    ipqos lowdelay throughput

Those old settings are pre-DSCP TOS byte values, newer OpenSSH defaults to
"IPQoS af21 cs1".  You could configure SSH to use DSCP values you like
and then match on those.

HTH,
Erik

traffic shaping with tc on Linux 5.4.x

[Lars Noodén]

Greetings.  I asked elsewhere but have been informed that this list
(Linux Advanced Routing and Traffic Control (LARTC)) is the place to ask
about traffic shaping on Linux with tc.

The goal of the rules below are to give top priority to SSH, next
priority to HTTP/HTTPS, third priority to everything else, and, then,
with what's left over give something to IPFS.  General tips and
corrections also welcome, especially about nft instead of iptables.

My question is, given the rules below, and if they are otherwise
correct, how would I further subdivide the SSH queue so that interactive
sessions are prioritized over bulk transfers?

Regards,
Lars

---

#!/bin/sh

PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin

if=wlp1s0

# remove existing qdiscs, classes, and filters from interface
tc qdisc del dev $if ingress
tc qdisc del dev $if root

# default class for unclassified traffic
tc qdisc replace dev $if root handle 1: htb default 30

# top level class with handle 1:1
tc class add dev $if parent 1: classid 1:1 htb rate 800kbit

# Class 1:10 is highest priority, SSH/SFTP
# Class 1:20 is next highest priority, HTTP/HTTPS
# Class 1:30 is next lowest priority, default traffic
# Class 1:40 is lowest priority but highest bandwidth, IPFS

tc class add dev $if parent 1:1 classid 1:10 htb rate 1mbit \
	ceil 200kbit prio 1
tc class add dev $if parent 1:1 classid 1:20 htb rate 1mbit \
	ceil 100kbit prio 2
tc class add dev $if parent 1:1 classid 1:30 htb rate 1mbit \
	ceil 100kbit prio 3
tc class add dev $if parent 1:1 classid 1:40 htb rate 1mbit \
	ceil 400kbit prio 4

# leaf qdisc to each child class
tc qdisc add dev $if parent 1:10 fq_codel
tc qdisc add dev $if parent 1:20 fq_codel
tc qdisc add dev $if parent 1:30 fq_codel
tc qdisc add dev $if parent 1:40 fq_codel

# add filters to prioritize traffic
tc filter add dev $if parent 1: handle 100 fw classid 1:10
tc filter add dev $if parent 1: handle 200 fw classid 1:20
tc filter add dev $if parent 1: handle 400 fw classid 1:40

# label outgoing traffic
iptables -Z; # zero counters
iptables -F; # flush (delete) rules
iptables -X; # delete all extra chains

iptables -t mangle -A OUTPUT -p tcp --match multiport \
	--sports 22 -j MARK --set-mark 100
iptables -t mangle -A OUTPUT -p tcp --match multiport \
	--sports 80,443 -j MARK --set-mark 200
iptables -t mangle -A OUTPUT -p tcp --match multiport \
	--sports 4001 -j MARK --set-mark 400

Re: traffic shaping with tc on Linux 5.4.x

[Frank Myhr]

On 2021/02/15 14:22, Lars Noodén wrote:
> If this is the right list to ask about tc, then I have a beginner
> question about traffic shaping.  If not please point me to the correct
> venue.

Hi Lars,

Probably you should ask your question on the LARTC mailing list:
https://lartc.org/#mailinglist

You could also try
https://serverfault.com/


> My questions is, given the rules below, how would I further subdivide
> the SSH queue so that interactive sessions are prioritized over bulk
> transfers?

Just some general comments:

Your question boils down to how to distinguish between interactive and 
bulk transfer SSH sessions. Some different cases:

* If interactive and bulk sessions typically originate from different 
hosts, then you could mark packets based on ip saddr and assign the 
marks to different routing classes.

* If the same hosts initiate both interactive and bulk transfers, then 
you could run sshd on two different ports, one for interactive, one for 
bulk transfers, and mark packets based on tcp dport. That would require 
you/your users to remember to use the correct ssh port depending on 
their intention.

* If you don't want to depend on clients connecting to a particular sshd 
port, you could try using SFQ to at least give interactive sessions a 
chance during bulk transfers. This *might* be enough, but is probably 
not the optimal way to achieve your goal.
https://lartc.org/howto/lartc.qdisc.classless.html#LARTC.SFQ

* Your general problem of achieving reasonable latency for interactive 
sessions while simultaneously providing high bandwidth for bulk 
transfers is exactly what HFSC is intended to solve:
https://en.wikipedia.org/wiki/Hierarchical_fair-service_curve

Worth a look!

* If none of the above ends up suitable for your purpose, you could use 
a meter with key ip saddr . tcp sport to mark ssh connections consuming 
more than a critical bandwidth for assignment to a bulk transfer class 
that gets lower priority. I'd use a set rather than a map because maps 
don't support timeouts, and you probably want the elements to expire 
automatically. I think a simple PRIO queue (within your ssh class) would 
work well with this config.

https://wiki.nftables.org/wiki-nftables/index.php/Meters
https://wiki.nftables.org/wiki-nftables/index.php/Rate_limiting_matchings
https://lartc.org/howto/lartc.qdisc.classful.html#AEN902

~

Would be interested in what you end up using... it would be great if you 
could follow up with a summary later. Good luck!

Best regards,
Frank

traffic shaping with tc on Linux 5.4.x

[Lars Noodén]

If this is the right list to ask about tc, then I have a beginner
question about traffic shaping.  If not please point me to the correct
venue.

My questions is, given the rules below, how would I further subdivide
the SSH queue so that interactive sessions are prioritized over bulk
transfers?

The goal of the rules below are to give top priority to SSH, next
priority to HTTP/HTTPS, third priority to everything else, and, then,
with what's left over give something to IPFS.  General tips and
corrections also welcome, especially about nft instead of iptables.

Regards,
Lars

---

#!/bin/sh

PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin

if=wlp1s0

# remove existing qdiscs, classes, and filters from interface
tc qdisc del dev $if ingress
tc qdisc del dev $if root

# default class for unclassified traffic
tc qdisc replace dev $if root handle 1: htb default 30

# top level class with handle 1:1
tc class add dev $if parent 1: classid 1:1 htb rate 800kbit

# Class 1:10 is highest priority, SSH/SFTP
# Class 1:20 is next highest priority, HTTP/HTTPS
# Class 1:30 is next lowest priority, default traffic
# Class 1:40 is lowest priority but highest bandwidth, IPFS

tc class add dev $if parent 1:1 classid 1:10 htb rate 1mbit \
	ceil 200kbit prio 1
tc class add dev $if parent 1:1 classid 1:20 htb rate 1mbit \
	ceil 100kbit prio 2
tc class add dev $if parent 1:1 classid 1:30 htb rate 1mbit \
	ceil 100kbit prio 3
tc class add dev $if parent 1:1 classid 1:40 htb rate 1mbit \
	ceil 400kbit prio 4

# leaf qdisc to each child class
tc qdisc add dev $if parent 1:10 fq_codel
tc qdisc add dev $if parent 1:20 fq_codel
tc qdisc add dev $if parent 1:30 fq_codel
tc qdisc add dev $if parent 1:40 fq_codel

# add filters to prioritize traffic
tc filter add dev $if parent 1: handle 100 fw classid 1:10
tc filter add dev $if parent 1: handle 200 fw classid 1:20
tc filter add dev $if parent 1: handle 400 fw classid 1:40

# label outgoing traffic
iptables -Z; # zero counters
iptables -F; # flush (delete) rules
iptables -X; # delete all extra chains

iptables -t mangle -A OUTPUT -p tcp --match multiport \
	--sports 22 -j MARK --set-mark 100
iptables -t mangle -A OUTPUT -p tcp --match multiport \
	--sports 80,443 -j MARK --set-mark 200
iptables -t mangle -A OUTPUT -p tcp --match multiport \
	--sports 4001 -j MARK --set-mark 400