Re: [RFC] [PATCH net-next v6 3/3] r8169: Implement dynamic ASPM mechanism - Kai-Heng Feng

From: Kai-Heng Feng <kai.heng.feng@canonical.com>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: Heiner Kallweit <hkallweit1@gmail.com>,
	nic_swsd <nic_swsd@realtek.com>,
	Bjorn Helgaas <bhelgaas@google.com>,
	David Miller <davem@davemloft.net>,
	Jakub Kicinski <kuba@kernel.org>,
	Anthony Wong <anthony.wong@canonical.com>,
	Linux Netdev List <netdev@vger.kernel.org>,
	Linux PCI <linux-pci@vger.kernel.org>,
	LKML <linux-kernel@vger.kernel.org>
Subject: Re: [RFC] [PATCH net-next v6 3/3] r8169: Implement dynamic ASPM mechanism
Date: Fri, 15 Oct 2021 12:11:47 +0800	[thread overview]
Message-ID: <CAAd53p5w_tE8URs0R7eog6X-kMSUQAeLiGS-CvDvnfQq+=i3TA@mail.gmail.com> (raw)
In-Reply-To: <20211008135821.GA1326355@bhelgaas>

On Fri, Oct 8, 2021 at 9:58 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Fri, Oct 08, 2021 at 02:18:55PM +0800, Kai-Heng Feng wrote:
> > On Fri, Oct 8, 2021 at 3:11 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > On Fri, Oct 08, 2021 at 12:15:52AM +0800, Kai-Heng Feng wrote:
> > > > r8169 NICs on some platforms have abysmal speed when ASPM is enabled.
> > > > Same issue can be observed with older vendor drivers.
> > > >
> > > > The issue is however solved by the latest vendor driver. There's a new
> > > > mechanism, which disables r8169's internal ASPM when the NIC traffic has
> > > > more than 10 packets per second, and vice versa. The possible reason for
> > > > this is likely because the buffer on the chip is too small for its ASPM
> > > > exit latency.
> > > > ...
>
> > > I suppose that on the Intel system, if we enable ASPM, the link goes
> > > to L1.2, and the NIC immediately receives 1000 packets in that second
> > > before we can disable ASPM again, we probably drop a few packets?
> > >
> > > Whereas on the AMD system, we probably *never* drop any packets even
> > > with L1.2 enabled all the time?
> >
> > Yes and yes.
>
> The fact that we drop some packets with dynamic ASPM on the Intel
> system means we must be giving up some performance.
>
> And I guess that on the AMD system, we should get full performance but
> we must be using a little more power (probably unmeasurable) because
> ASPM *could* be always enabled but dynamic ASPM disables it some of
> the time.

Yes that's the case here.

>
> > > And if we actually knew the root cause and could set the correct LTR
> > > values or whatever is wrong on the Intel system, we probably wouldn't
> > > need this dynamic scheme?
> >
> > Because Realtek already implemented the dynamic ASPM workaround in
> > their Windows and Linux driver, they never bother to find the root
> > cause.
> > So we'll never know what really happens here.
>
> Looks like it.  Somebody with a PCIe analyzer could probably make
> progress, but I agree, that doesn't seem likely.
>
> Realtek no doubt has the equipment to do this, but apparently they
> don't think it's worthwhile.  In their defense, the Linux ASPM code is
> pretty impenetrable and there could be a problem there that causes or
> contributes to this.

I do hope they can put more effort on their ethernet driver like what
they do on their wireless drivers.

Kai-Heng

>
> Bjorn