From mboxrd@z Thu Jan 1 00:00:00 1970 From: Grant Taylor Date: Tue, 24 Mar 2020 17:57:46 +0000 Subject: Re: tc question about ingress bandwidth splitting Message-Id: <1e8ec56e-4456-0e07-77bd-13c17e36a968@tnetconsulting.net> MIME-Version: 1 Content-Type: multipart/mixed; boundary="------------ms010105050500010201000105" List-Id: References: <74CFEE65-9CE8-4CF7-9706-2E2E67B24E08@redfish-solutions.com> In-Reply-To: <74CFEE65-9CE8-4CF7-9706-2E2E67B24E08@redfish-solutions.com> To: lartc@vger.kernel.org This is a cryptographically signed message in MIME format. --------------ms010105050500010201000105 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 3/24/20 12:51 AM, Philip Prindeville wrote: > Hi Grant, Hi, > Well, it=E2=80=99s exactly because it *isn=E2=80=99t* 1Gbps each direct= ion that I=20 > need good shaping. I could get more, but I=E2=80=99d also pay more. Fair enough. > No. The idea being that =E2=80=9Cguest=E2=80=9D relies on the kindness= of strangers=E2=80=A6=20 > whereas =E2=80=9Cproduction=E2=80=9D has a guaranteed SLA of at least 4= 0/8 mbps. QoS has the ability to guarantee an SLA of 40 & 8 to production. Think about it this way: 1) Production gets up to it's SLA. 2) Guest gets up to it's SLA. 3) Production and / or guest get any unused bandwidth. Each class is guaranteed their SLA, and can optionally use any remaining = bandwidth (unused bandwidth of other classes). > Right. In this case I=E2=80=99m limiting (or pacing) the ACKs so that = the=20 > sender paces his data. That's not what I was referring to. QoS can rate limit what is sent out the internal interfaces at the 40 /=20 10 Mbps values. The thing that it can not do is rate limit want comes in the outside=20 interface. There may be ~12 Mbps of incoming traffic for guests. But=20 the router will only send 10 Mbps of that out it's inside interface.=20 Thus the router is rate limiting what guest receives to 10 Mbps. It's=20 just that there is an additional 2 Mbps that the router is dropping on=20 the floor. Does that make sense? > For UDP not at all. For TCP you can apply back pressure, as above.=20 > If the sender has filled his window, and I hold back any ACKs, he=20 > can=E2=80=99t send anything more until I do send an ACK. See above. It's possible for a router to use QoS to rate limit any type of traffic. = The router quite literally receives the traffic on one interface and=20 sends it out another interface. The rate that the traffic is sent is=20 what is rate limited. TCP, UDP, ICMP, it doesn't matter what type of traffic. > Correct. Eventually the sender will back off in an attempt to reach=20 > a congestion-free steady state. I would bet that a "congestion-free steady state" is /never/ achieved.=20 The very design of most protocols is to send as fast as possible. When=20 they detect errors, they /may/ slow down /for/ /a/ /little/ /while/.=20 But they will speed back up. Even if a given flow could achieve something resembling a=20 congestion-free steady state, the nature of Internet traffic is so=20 inconsistent that you have flows starting & stopping all the time. Thus = you have wildly shifting demands on traffic. > My scenario, as I said, is a SoHo router. I don=E2=80=99t have a lot o= f=20 > servers behind it that receive bursts of incoming traffic=20 > asynchronously from outside (other than email, which I host=20 > locally). IMHO servers are actually less of a problem than the average person=20 surfing the web. Every single web page you go to is at least one new and short lived=20 flow. Many web pages are 100s of new and short lived flows. Most of=20 them start at about the same time. The more web surfers you have, the more of these types of traffic=20 patterns that you have. It's also very random when they will happen.=20 You could have anywhere between 0 and the number of people on your=20 network at the same time. Also, multiple windows / tabs mean that more and more of these can=20 happen at the same time. > If my daughter decides to watch an HD movie on an iPad during the day=20 > while I=E2=80=99m working, I don=E2=80=99t want that traffic overrunnin= g my network=20 > and causing me to not be able to work. In that scenario, the=20 > connection is originating internally and going outbound, and it=E2=80=99= s=20 > long-lived (where "long-lived" is any duration of 20 or more RTT=E2=80=99= s). That's one of the things that QoS is quite good at dealing with. Though I question how long lived your daughter's streams actually are.=20 I know for a fact that YouTube is a series of small downloads. So each=20 download is relatively short lived. It's not one long lived connection=20 that lasts for the duration of the video. There also the fact that YouTube prefers QUIC, which is UDP based, to=20 TCP if it can use it. > Only slightly less for me: I did a traffic-shaper plugin for Arno=E2=80= =99s=20 > Internet Firewall (AIF) about 12 years ago. I=E2=80=99ve since forgott= en=20 > everything. Tempus fugit. > Yup. And I=E2=80=99m hoping to be able to not need ifb to do it. I forgot about ifb. I think it would do similar to what I was=20 suggesting with network namespaces. Though I do wonder how complicated=20 having multiple things in the same namespace will make tc rules. > Sure, for the total. I meant =E2=80=9Cguest=E2=80=9D bursting over his= allotted 10/2=20 > mbps for a short duration, say 600ms (I came up with that as being 5=20 > RTT=E2=80=99s of 120ms). I figure that=E2=80=99s enough for slow-start= to ramp up=20 > into steady state=E2=80=A6 See above comments about steady state. > Well, know you=E2=80=99ve got me confused. Because if each can borrow = from=20 > the other, where=E2=80=99s the SLA? Where=E2=80=99s the cap? Who gets= prioritized? I think I explained it above. Each is guaranteed the availability of it's SLA. The unused traffic=20 over the SLA is (can be) fair game. Meaning that if production is using 15 & 3, there is 25 & 5 that guest=20 could use if allowed to. Similarly, if guests are sleeping, there is an additional 10 & 2 that=20 production could take advantage of. > I could be completely unshaped, and have both borrowing from each=20 > other=E2=80=A6 which is the degenerate case. That's why each is guaranteed their SLA *FIRST* and then can use=20 whatever is unused *SECOND*. This allows optimal use of the bandwidth=20 while still guaranteeing SLAs. > Yeah, and indeed that=E2=80=99s what HTB excels at. Yep. If memory serves, HTB is one of many that can do it. But HTB was one of = the earlier options. > Agreed. >=20 > Although=E2=80=A6 in the case of the =E2=80=9Cguest=E2=80=9D network, I= don=E2=80=99t ever want it=20 > performing better than the hard SLA of 10/2 mbps, or people will=20 > complain when they don=E2=80=99t get extra bandwidth. If they=E2=80=99= re conditioned=20 > to think that =E2=80=9CI=E2=80=99m on the guest network, and 10/2 mbps = is all I=E2=80=99m=20 > going to get=E2=80=9D then they=E2=80=99ll be happy with it and won=E2=80= =99t complain. Okay. That is a hard policy decision that you are making. I have no objection = to that. It also means that guest doesn't get to borrow unused=20 bandwidth from production. > I don=E2=80=99t want to hear, =E2=80=9Cwell, this was so much better tw= o days ago!=E2=80=9D >=20 > My answer is, =E2=80=9CIt=E2=80=99s free. You=E2=80=99re getting it by= someone else=E2=80=99s good=20 > graces=E2=80=A6 be grateful you=E2=80=99re getting anything at all.=E2=80= =9D ACK > Some ISPs were actually squashing the bits, and got spanked severely=20 > by the FCC. Okay. I don't recall that. I wonder why they wanted to stomp on ECN.=20 Especially seeing as how ECN is to alert about congestion. Lack of=20 congestion notification encourages additional ramp up. I'm assuming that ISPs were clearing ECN. Maybe I have this backwards.=20 Maybe they were artificially setting it to induce slowdowns. > Also, some older router=E2=80=99s IP stacks were not ECN aware, and had= the=20 > older bit definitions (remember that RFC 3168 and ECN borrowed the=20 > ECT1 bit from TOS/LOWCOST from RFC 791 and 1349). My experience has been that most routers ignore QoS / ECN. > I=E2=80=99m assuming a 3.18 kernel or later and iproute2 + iptables. N= othing=20 > else. And sch_htb is present. Unfortunately there are a LOT of possible combination in that mix. I also know that the Ubiquity EdgeRouter Lite uses a 2.6 kernel. I=20 don't know about other EdgeOS (Ubiquity Linux distro). But I wouldn't=20 be surprised to learn that EdgeOS is 2.6 period. > This is the same problem that ifb solves, right? Probably. (See above.) > I=E2=80=99m not sure I want to assume that Namespaces are available in = all=20 > scenarios. Fair enough. I run old PCs with standard Linux distros as routers. So I can easily=20 add what I want to them. > Yeah, for now I=E2=80=99m not concerned about internal traffic. Yet. That's something to keep in mind when creating the QoS configuration.=20 As in you might need to take it into account and make sure that you=20 don't artificially slow it down. > Agreed. >=20 > As I said, I don=E2=80=99t want to have to explain to anyone later that= =20 > =E2=80=9C35mbps might have been available Sunday, but today I=E2=80=99m= running=20 > Carbonite and it=E2=80=99s hogging all the bandwidth while I download t= hese=20 > 10 new VM=E2=80=99s I created this morning, so suck it.=E2=80=9D ACK > No, but it can cause other traffic destined to the production network=20 > to get dropped, which is the scenario I=E2=80=99m trying to avoid. I understand. > As I remember, some of the newer (model-based) congestion avoidance=20 > algorithms (like BBR) were really much better at fairness and=20 > avoiding dropped packets=E2=80=A6 My understanding is that BBR is rather aggressive in that it tries to=20 identify what the bandwidth is and then use all of that bandwidth that=20 it can. It's particularly aggressive at finding the bandwidth too. See above comments about the transient nature of flows. > Thanks. You're welcome. --=20 Grant. . . . unix || die --------------ms010105050500010201000105 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC CzkwggUhMIIECaADAgECAhA53zcXtFD9dENby64EqrKqMA0GCSqGSIb3DQEBCwUAMIGWMQsw CQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxm b3JkMRgwFgYDVQQKEw9TZWN0aWdvIExpbWl0ZWQxPjA8BgNVBAMTNVNlY3RpZ28gUlNBIENs aWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBMB4XDTE5MTExOTAwMDAw MFoXDTIwMTExODIzNTk1OVowKzEpMCcGCSqGSIb3DQEJARYaZ3RheWxvckB0bmV0Y29uc3Vs dGluZy5uZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCwIZcEJcuE7mUfxJnD I8oOSX/TvAhoP11agD++8L7Ok8fFJhJK0lOVRsq1M6lF2E2Vzuyffg2ppbecWvHcIRadsaiG imnrJQasdkhj/JUtqPUXnC0SVA0AzYLrLReQB+9j/jTgB5JnFLyC2lEn9KTA6JmDGjvVkv2T k+I2+v24nI4/2lGjD+jIKQiFXkE1uqablXJAw1c9Mh9d4/wjnIM9zLGv1i3xxOLdQ1PXSUZL 12wOy1r7CsGAnNSNhGaceB2tdhdleFEyIHgSgDWtWResHdu/ubZqFiHxaLRJlafOHMj3yC6x NOA1IdcNJsaRkQHxSkayKzeE5JK3TxlV83dbAgMBAAGjggHTMIIBzzAfBgNVHSMEGDAWgBQJ wPL8C9qU21/+K9+omULPyeCtADAdBgNVHQ4EFgQUU6bXebmKM+efFHN0MBjYuJO9Za8wDgYD VR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUF BwMCMEAGA1UdIAQ5MDcwNQYMKwYBBAGyMQECAQEBMCUwIwYIKwYBBQUHAgEWF2h0dHBzOi8v c2VjdGlnby5jb20vQ1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuc2VjdGlnby5j b20vU2VjdGlnb1JTQUNsaWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmww gYoGCCsGAQUFBwEBBH4wfDBVBggrBgEFBQcwAoZJaHR0cDovL2NydC5zZWN0aWdvLmNvbS9T ZWN0aWdvUlNBQ2xpZW50QXV0aGVudGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAjBggr BgEFBQcwAYYXaHR0cDovL29jc3Auc2VjdGlnby5jb20wJQYDVR0RBB4wHIEaZ3RheWxvckB0 bmV0Y29uc3VsdGluZy5uZXQwDQYJKoZIhvcNAQELBQADggEBADOWdJFXVQvdVPUy4ChriEyS 3wiEdWmLb3CGko4ps7uXgHoCk0V9oU38LjKTrcm/KOhLhBh2Wz3LxirbtgTP+YxpgkPxDEWO ee/o/TiLhVrTLiqZJIwjlZmY1lTmHuoXWQK3M0MJZYVrGgMJgQg0/+mZkRlEa67N4WETh7MH rKglv3HHy3LeU835KA8cpMxRbDvPiA8wdKHWgrl4LXOJKtI8rgmMJxUOCQdgI6DSEo/yYve0 /TxLLBlWAhve7e+/aYjKn3V5CpNOmqkRi7V2d6ZJ+RMQrJDtqitQAkzq8cH+CSTGagHzAxQp e00hH+aVwNioyaoNBezCCLirOjVdlFIwggYQMIID+KADAgECAhBNlCwQ1DvglAnFgS06KwZP MA0GCSqGSIb3DQEBDAUAMIGIMQswCQYDVQQGEwJVUzETMBEGA1UECBMKTmV3IEplcnNleTEU MBIGA1UEBxMLSmVyc2V5IENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEu MCwGA1UEAxMlVVNFUlRydXN0IFJTQSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0xODEx MDIwMDAwMDBaFw0zMDEyMzEyMzU5NTlaMIGWMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3Jl YXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRgwFgYDVQQKEw9TZWN0aWdvIExp bWl0ZWQxPjA8BgNVBAMTNVNlY3RpZ28gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQg U2VjdXJlIEVtYWlsIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyjztlApB /975Rrno1jvm2pK/KxBOqhq8gr2+JhwpKirSzZxQgT9tlC7zl6hn1fXjSo5MqXUfItMltrMa XqcESJuK8dtK56NCSrq4iDKaKq9NxOXFmqXX2zN8HHGjQ2b2Xv0v1L5Nk1MQPKA19xeWQcpG EGFUUd0kN+oHox+L9aV1rjfNiCj3bJk6kJaOPabPi2503nn/ITX5e8WfPnGw4VuZ79Khj1YB rf24k5Ee1sLTHsLtpiK9OjG4iQRBdq6Z/TlVx/hGAez5h36bBJMxqdHLpdwIUkTqT8se3ed0 PewDch/8kHPo5fZl5u1B0ecpq/sDN/5sCG52Ds+QU5O5EwIDAQABo4IBZDCCAWAwHwYDVR0j BBgwFoAUU3m/WqorSs9UgOHYm8Cd8rIDZsswHQYDVR0OBBYEFAnA8vwL2pTbX/4r36iZQs/J 4K0AMA4GA1UdDwEB/wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdJQQWMBQGCCsG AQUFBwMCBggrBgEFBQcDBDARBgNVHSAECjAIMAYGBFUdIAAwUAYDVR0fBEkwRzBFoEOgQYY/ aHR0cDovL2NybC51c2VydHJ1c3QuY29tL1VTRVJUcnVzdFJTQUNlcnRpZmljYXRpb25BdXRo b3JpdHkuY3JsMHYGCCsGAQUFBwEBBGowaDA/BggrBgEFBQcwAoYzaHR0cDovL2NydC51c2Vy dHJ1c3QuY29tL1VTRVJUcnVzdFJTQUFkZFRydXN0Q0EuY3J0MCUGCCsGAQUFBzABhhlodHRw Oi8vb2NzcC51c2VydHJ1c3QuY29tMA0GCSqGSIb3DQEBDAUAA4ICAQBBRHUAqznCFfXejpVt MnFojADdF9d6HBA4kMjjsb0XMZHztuOCtKF+xswhh2GqkW5JQrM8zVlU+A2VP72Ky2nlRA1G wmIPgou74TZ/XTarHG8zdMSgaDrkVYzz1g3nIVO9IHk96VwsacIvBF8JfqIs+8aWH2PfSUrN xP6Ys7U0sZYx4rXD6+cqFq/ZW5BUfClN/rhk2ddQXyn7kkmka2RQb9d90nmNHdgKrwfQ49mQ 2hWQNDkJJIXwKjYA6VUR/fZUFeCUisdDe/0ABLTI+jheXUV1eoYV7lNwNBKpeHdNuO6Aacb5 33JlfeUHxvBz9OfYWUiXu09sMAviM11Q0DuMZ5760CdO2VnpsXP4KxaYIhvqPqUMWqRdWyn7 crItNkZeroXaecG03i3mM7dkiPaCkgocBg0EBYsbZDZ8bsG3a08LwEsL1Ygz3SBsyECa0waq 4hOf/Z85F2w2ZpXfP+w8q4ifwO90SGZZV+HR/Jh6rEaVPDRF/CEGVqR1hiuQOZ1YL5ezMTX0 ZSLwrymUE0pwi/KDaiYB15uswgeIAcA6JzPFf9pLkAFFWs1QNyN++niFhsM47qodx/PL+5jR 87myx5uYdBEQkkDc+lKB1Wct6ucXqm2EmsaQ0M95QjTmy+rDWjkDYdw3Ms6mSWE3Bn7i5Zgt wCLXgAIe5W8mybM2JzGCBDIwggQuAgEBMIGrMIGWMQswCQYDVQQGEwJHQjEbMBkGA1UECBMS R3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRgwFgYDVQQKEw9TZWN0aWdv IExpbWl0ZWQxPjA8BgNVBAMTNVNlY3RpZ28gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBh bmQgU2VjdXJlIEVtYWlsIENBAhA53zcXtFD9dENby64EqrKqMA0GCWCGSAFlAwQCAQUAoIIC VzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0yMDAzMjQxNzU3 NDZaMC8GCSqGSIb3DQEJBDEiBCDh6k0pPjtRGqvpLBTURrl6dGUQnHw8doiPzUKHevTkijBs BgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcw DgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEo MIG8BgkrBgEEAYI3EAQxga4wgaswgZYxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVy IE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGltaXRl ZDE+MDwGA1UEAxM1U2VjdGlnbyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1 cmUgRW1haWwgQ0ECEDnfNxe0UP10Q1vLrgSqsqowgb4GCyqGSIb3DQEJEAILMYGuoIGrMIGW MQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdT YWxmb3JkMRgwFgYDVQQKEw9TZWN0aWdvIExpbWl0ZWQxPjA8BgNVBAMTNVNlY3RpZ28gUlNB IENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhA53zcXtFD9dENb y64EqrKqMA0GCSqGSIb3DQEBAQUABIIBAD6jZXzgb1au91uPwgJBer1NXSpCBz8D4K3lvBeM Igqke1QS5dal7lUSvs+Goy/zCQRWTFz9FF0zff4u3PbcfvXbqKPQauITWM+Adq9SYa3fRKu2 JpjwuSzTXrYhJymyO1VbzU/8E2YNiM6ZRgCtFh7DEX66Jl50afQJZw9t5fDvL+Yae2LzGLwy 0fszDE/x8XHnVdQSsmGJLZFdLtKxMiMXsMWrlEdeIG1MEpWAw+Hfo9KkP5kmZBD8P016LU5B dxaWpgHvlzFWdlJ79ZLq52b7z/lI4Kpl/Ufj3+yUIajPOhitkOh5eUqrALE6AbLXuOQylFh2 d1eam5Vfu5NSRf4AAAAAAAA= --------------ms010105050500010201000105--