From mboxrd@z Thu Jan 1 00:00:00 1970 From: Xin Long Subject: Re: [PATCH net] sctp: not allow to set rto_min with a value below 200 msecs Date: Mon, 4 Jun 2018 20:15:17 +0800 Message-ID: References: <20180525191319.GB392@hmswarspite.think-freely.org> <71FD541D-25FE-4100-980B-C3A0CEAF6CD4@lurchi.franken.de> <20180527010059.GA15798@neilslaptop.think-freely.org> <20180528194315.GB3788@localhost.localdomain> <20180529114111.GA24144@hmswarspite.think-freely.org> <559FFFDD-E508-4936-9544-CACE606AF40F@lurchi.franken.de> <20180529154514.GC3788@localhost.localdomain> <20180529170604.GD3788@localhost.localdomain> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Cc: Marcelo Ricardo Leitner , Neal Cardwell , Michael Tuexen , Neil Horman , Netdev , linux-sctp@vger.kernel.org, David Miller , David Ahern , Eric Dumazet , syzkaller To: Dmitry Vyukov Return-path: Received: from mail-io0-f195.google.com ([209.85.223.195]:38181 "EHLO mail-io0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751970AbeFDMPT (ORCPT ); Mon, 4 Jun 2018 08:15:19 -0400 In-Reply-To: Sender: netdev-owner@vger.kernel.org List-ID: On Mon, Jun 4, 2018 at 4:34 PM, Dmitry Vyukov wrote: > On Tue, May 29, 2018 at 7:45 PM, Xin Long wrote: >> On Wed, May 30, 2018 at 1:06 AM, Marcelo Ricardo Leitner >> wrote: >>> On Tue, May 29, 2018 at 12:03:46PM -0400, Neal Cardwell wrote: >>>> On Tue, May 29, 2018 at 11:45 AM Marcelo Ricardo Leitner < >>>> marcelo.leitner@gmail.com> wrote: >>>> > - patch2 - fix rtx attack vector >>>> > - Add the floor value to rto_min to HZ/20 (which fits the values >>>> > that Michael shared on the other email) >>>> >>>> I would encourage allowing minimum RTO values down to 5ms, if the ACK >>>> policy in the receiver makes this feasible. Our experience is that in >>>> datacenter environments it can be advantageous to allow timer-based loss >>>> recoveries using timeout values as low as 5ms, e.g.: >>> >>> Thanks Neal. On Xin's tests, the hearbeat timer becomes an issue at >>> ~25ms already. Xin, can you share more details on the hw, which CPU >>> was used? > > Hi, > > Did we reach any decision on this? This continues to produce bug > reports on syzbot. I will post a patch later today for the suggestion: - patch1 - fix issue at hand - Use the max_t above to fix this. As for patch2 and patch3: - patch2 - fix rtx attack vector - Add the floor value to rto_min to HZ/20 (which fits the values that Michael shared on the other email) - patch3 - speed up initial HB again - change sctp_cmd_hb_timers_start() so hb timers are kickstarted when the association is established. AFAICT RFC doesn't specify when these initial ones should be sent, and I see no issues with speeding them up. They are more like improvements, we will do it in the future after getting more information. > > I am not sure whom you are asking, because Xin is you unless I am > missing something :) > But if you mean syzbot hardware, then it's GCE VMs with modern Intel > CPUs but an important aspect is a heavy-debug config (which you can > take from here https://syzkaller.appspot.com/bug?extid=3dcd59a1f907245f891f) > and systematic bug reporting. So if it's any flaky in your testing, it > will produce dozens of bug emails on syzbot. > > >> It was on a KVM guest, "-smp 2,cores=1,threads=1,sockets=2" >> # lscpu >> Architecture: x86_64 >> CPU op-mode(s): 32-bit, 64-bit >> Byte Order: Little Endian >> CPU(s): 2 >> On-line CPU(s) list: 0,1 >> Thread(s) per core: 1 >> Core(s) per socket: 1 >> Socket(s): 2 >> NUMA node(s): 1 >> Vendor ID: GenuineIntel >> CPU family: 6 >> Model: 13 >> Model name: QEMU Virtual CPU version 1.5.3 >> Stepping: 3 >> CPU MHz: 2397.222 >> BogoMIPS: 4794.44 >> Hypervisor vendor: KVM >> Virtualization type: full >> L1d cache: 32K >> L1i cache: 32K >> L2 cache: 4096K >> NUMA node0 CPU(s): 0,1 >> Flags: fpu de pse tsc msr pae mce cx8 apic sep mtrr >> pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good >> nopl cpuid pni cx16 hypervisor lahf_lm abm pti >> >> If we're counting on max_t to fix this CPU stuck. It should not that >> matter if min rto < the value causing that stuck. >> >>> >>> Anyway, what about we add a floor to rto_max too, so that RTO can >>> actually grow into something bigger that don't hog the CPU? Like: >>> rto_min floor = 5ms >>> rto_max floor = 50ms >>> >>> Marcelo From mboxrd@z Thu Jan 1 00:00:00 1970 From: Xin Long Date: Mon, 04 Jun 2018 12:15:17 +0000 Subject: Re: [PATCH net] sctp: not allow to set rto_min with a value below 200 msecs Message-Id: List-Id: References: <20180525191319.GB392@hmswarspite.think-freely.org> <71FD541D-25FE-4100-980B-C3A0CEAF6CD4@lurchi.franken.de> <20180527010059.GA15798@neilslaptop.think-freely.org> <20180528194315.GB3788@localhost.localdomain> <20180529114111.GA24144@hmswarspite.think-freely.org> <559FFFDD-E508-4936-9544-CACE606AF40F@lurchi.franken.de> <20180529154514.GC3788@localhost.localdomain> <20180529170604.GD3788@localhost.localdomain> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Dmitry Vyukov Cc: Marcelo Ricardo Leitner , Neal Cardwell , Michael Tuexen , Neil Horman , Netdev , linux-sctp@vger.kernel.org, David Miller , David Ahern , Eric Dumazet , syzkaller On Mon, Jun 4, 2018 at 4:34 PM, Dmitry Vyukov wrote: > On Tue, May 29, 2018 at 7:45 PM, Xin Long wrote: >> On Wed, May 30, 2018 at 1:06 AM, Marcelo Ricardo Leitner >> wrote: >>> On Tue, May 29, 2018 at 12:03:46PM -0400, Neal Cardwell wrote: >>>> On Tue, May 29, 2018 at 11:45 AM Marcelo Ricardo Leitner < >>>> marcelo.leitner@gmail.com> wrote: >>>> > - patch2 - fix rtx attack vector >>>> > - Add the floor value to rto_min to HZ/20 (which fits the values >>>> > that Michael shared on the other email) >>>> >>>> I would encourage allowing minimum RTO values down to 5ms, if the ACK >>>> policy in the receiver makes this feasible. Our experience is that in >>>> datacenter environments it can be advantageous to allow timer-based loss >>>> recoveries using timeout values as low as 5ms, e.g.: >>> >>> Thanks Neal. On Xin's tests, the hearbeat timer becomes an issue at >>> ~25ms already. Xin, can you share more details on the hw, which CPU >>> was used? > > Hi, > > Did we reach any decision on this? This continues to produce bug > reports on syzbot. I will post a patch later today for the suggestion: - patch1 - fix issue at hand - Use the max_t above to fix this. As for patch2 and patch3: - patch2 - fix rtx attack vector - Add the floor value to rto_min to HZ/20 (which fits the values that Michael shared on the other email) - patch3 - speed up initial HB again - change sctp_cmd_hb_timers_start() so hb timers are kickstarted when the association is established. AFAICT RFC doesn't specify when these initial ones should be sent, and I see no issues with speeding them up. They are more like improvements, we will do it in the future after getting more information. > > I am not sure whom you are asking, because Xin is you unless I am > missing something :) > But if you mean syzbot hardware, then it's GCE VMs with modern Intel > CPUs but an important aspect is a heavy-debug config (which you can > take from here https://syzkaller.appspot.com/bug?extid=cd59a1f907245f891f) > and systematic bug reporting. So if it's any flaky in your testing, it > will produce dozens of bug emails on syzbot. > > >> It was on a KVM guest, "-smp 2,cores=1,threads=1,sockets=2" >> # lscpu >> Architecture: x86_64 >> CPU op-mode(s): 32-bit, 64-bit >> Byte Order: Little Endian >> CPU(s): 2 >> On-line CPU(s) list: 0,1 >> Thread(s) per core: 1 >> Core(s) per socket: 1 >> Socket(s): 2 >> NUMA node(s): 1 >> Vendor ID: GenuineIntel >> CPU family: 6 >> Model: 13 >> Model name: QEMU Virtual CPU version 1.5.3 >> Stepping: 3 >> CPU MHz: 2397.222 >> BogoMIPS: 4794.44 >> Hypervisor vendor: KVM >> Virtualization type: full >> L1d cache: 32K >> L1i cache: 32K >> L2 cache: 4096K >> NUMA node0 CPU(s): 0,1 >> Flags: fpu de pse tsc msr pae mce cx8 apic sep mtrr >> pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good >> nopl cpuid pni cx16 hypervisor lahf_lm abm pti >> >> If we're counting on max_t to fix this CPU stuck. It should not that >> matter if min rto < the value causing that stuck. >> >>> >>> Anyway, what about we add a floor to rto_max too, so that RTO can >>> actually grow into something bigger that don't hog the CPU? Like: >>> rto_min floor = 5ms >>> rto_max floor = 50ms >>> >>> Marcelo