RE: Re: [PATCH] net: ena: Add PCI shutdown handler to allow safe kexec

RE: Re: [PATCH] net: ena: Add PCI shutdown handler to allow safe kexec

[Jubran, Samih]

> -----Original Message-----
> From: Jubran, Samih
> Sent: Tuesday, March 24, 2020 7:03 PM
> To: 'David Miller' <davem@davemloft.net>; 'gpiccoli@canonical.com'
> <gpiccoli@canonical.com>
> Cc: Belgazal, Netanel <netanel@amazon.com>; Kiyanovski, Arthur
> <akiyano@amazon.com>; 'netdev@vger.kernel.org'
> <netdev@vger.kernel.org>; Tzalik, Guy <gtzalik@amazon.com>; Bshara,
> Saeed <saeedb@amazon.com>; Machulsky, Zorik <zorik@amazon.com>;
> 'kernel@gpiccoli.net' <kernel@gpiccoli.net>; 'gshan@redhat.com'
> <gshan@redhat.com>; 'gavin.guo@canonical.com'
> <gavin.guo@canonical.com>; 'jay.vosburgh@canonical.com'
> <jay.vosburgh@canonical.com>; 'pedro.principeza@canonical.com'
> <pedro.principeza@canonical.com>
> Subject: RE: Re: [PATCH] net: ena: Add PCI shutdown handler to allow safe
> kexec
> 
> 
> 
> > -----Original Message-----
> > From: netdev-owner@vger.kernel.org <netdev-owner@vger.kernel.org>
> On
> > Behalf Of David Miller <davem@davemloft.net>
> > Sent: Tuesday, March 24, 2020 6:05 AM
> > To: gpiccoli@canonical.com
> > Cc: netanel@amazon.com; akiyano@amazon.com;
> netdev@vger.kernel.org;
> > gtzalik@amazon.com; saeedb@amazon.com; zorik@amazon.com;
> > kernel@gpiccoli.net; gshan@redhat.com; gavin.guo@canonical.com;
> > jay.vosburgh@canonical.com; pedro.principeza@canonical.com
> > Subject: Re: [PATCH] net: ena: Add PCI shutdown handler to allow safe
> > kexec
> >
> > From: "Guilherme G. Piccoli" <gpiccoli@canonical.com>
> > Date: Fri, 20 Mar 2020 09:55:34 -0300
> >
> > > Currently ENA only provides the PCI remove() handler, used during
> > > rmmod for example. This is not called on shutdown/kexec path; we are
> > > potentially creating a failure scenario on kexec:
> > >
> > > (a) Kexec is triggered, no shutdown() / remove() handler is called
> > > for ENA; instead pci_device_shutdown() clears the master bit of the
> > > PCI device, stopping all DMA transactions;
> > >
> > > (b) Kexec reboot happens and the device gets enabled again, likely
> > > having its FW with that DMA transaction buffered; then it may
> > > trigger the (now
> > > invalid) memory operation in the new kernel, corrupting kernel
> > > memory
> > area.
> > >
> > > This patch aims to prevent this, by implementing a shutdown()
> > > handler quite similar to the remove() one - the difference being the
> > > handling of the netdev, which is unregistered on remove(), but
> > > following the convention observed in other drivers, it's only detached on
> shutdown().
> > >
> > > This prevents an odd issue in AWS Nitro instances, in which after
> > > the 2nd kexec the next one will fail with an initrd corruption,
> > > caused by a wild DMA write to invalid kernel memory. The lspci
> > > output for the adapter present in my instance is:
> > >
> > > 00:05.0 Ethernet controller [0200]: Amazon.com, Inc. Elastic Network
> > > Adapter (ENA) [1d0f:ec20]
> > >
> > > Suggested-by: Gavin Shan <gshan@redhat.com>
> > > Signed-off-by: Guilherme G. Piccoli <gpiccoli@canonical.com>
> >
> > Amazon folks, please review.
> 
> The patch is still under review we will reply as soon as we have finished
> testing it, Thanks

Acked-by: Sameeh Jubran <sameehj@amazon.com>

RE: Re: [PATCH] net: ena: Add PCI shutdown handler to allow safe kexec

[Jubran, Samih]

> -----Original Message-----
> From: netdev-owner@vger.kernel.org <netdev-owner@vger.kernel.org>
> On Behalf Of David Miller <davem@davemloft.net>
> Sent: Tuesday, March 24, 2020 6:05 AM
> To: gpiccoli@canonical.com
> Cc: netanel@amazon.com; akiyano@amazon.com; netdev@vger.kernel.org;
> gtzalik@amazon.com; saeedb@amazon.com; zorik@amazon.com;
> kernel@gpiccoli.net; gshan@redhat.com; gavin.guo@canonical.com;
> jay.vosburgh@canonical.com; pedro.principeza@canonical.com
> Subject: Re: [PATCH] net: ena: Add PCI shutdown handler to allow safe kexec
> 
> From: "Guilherme G. Piccoli" <gpiccoli@canonical.com>
> Date: Fri, 20 Mar 2020 09:55:34 -0300
> 
> > Currently ENA only provides the PCI remove() handler, used during
> > rmmod for example. This is not called on shutdown/kexec path; we are
> > potentially creating a failure scenario on kexec:
> >
> > (a) Kexec is triggered, no shutdown() / remove() handler is called for
> > ENA; instead pci_device_shutdown() clears the master bit of the PCI
> > device, stopping all DMA transactions;
> >
> > (b) Kexec reboot happens and the device gets enabled again, likely
> > having its FW with that DMA transaction buffered; then it may trigger
> > the (now
> > invalid) memory operation in the new kernel, corrupting kernel memory
> area.
> >
> > This patch aims to prevent this, by implementing a shutdown() handler
> > quite similar to the remove() one - the difference being the handling
> > of the netdev, which is unregistered on remove(), but following the
> > convention observed in other drivers, it's only detached on shutdown().
> >
> > This prevents an odd issue in AWS Nitro instances, in which after the
> > 2nd kexec the next one will fail with an initrd corruption, caused by
> > a wild DMA write to invalid kernel memory. The lspci output for the
> > adapter present in my instance is:
> >
> > 00:05.0 Ethernet controller [0200]: Amazon.com, Inc. Elastic Network
> > Adapter (ENA) [1d0f:ec20]
> >
> > Suggested-by: Gavin Shan <gshan@redhat.com>
> > Signed-off-by: Guilherme G. Piccoli <gpiccoli@canonical.com>
> 
> Amazon folks, please review.

The patch is still under review we will reply as soon as we have finished testing it,
Thanks