[PATCH V1 net-next 0/5] Add devlink support to ena

[PATCH V1 net-next 0/5] Add devlink support to ena

[David Arinzon]

This patchset adds devlink support to the ena driver.

David Arinzon (5):
  net: ena: Register ena device to devlink
  net: ena: Add devlink reload functionality
  net: ena: Configure large LLQ using devlink params
  net: ena: Several changes to support large LLQ configuration
  net: ena: Add devlink documentation

 .../device_drivers/ethernet/amazon/ena.rst    |  30 +++
 drivers/net/ethernet/amazon/Kconfig           |   1 +
 drivers/net/ethernet/amazon/ena/Makefile      |   2 +-
 drivers/net/ethernet/amazon/ena/ena_devlink.c | 215 ++++++++++++++++++
 drivers/net/ethernet/amazon/ena/ena_devlink.h |  22 ++
 drivers/net/ethernet/amazon/ena/ena_netdev.c  | 123 +++++++---
 drivers/net/ethernet/amazon/ena/ena_netdev.h  |  13 ++
 7 files changed, 379 insertions(+), 27 deletions(-)
 create mode 100644 drivers/net/ethernet/amazon/ena/ena_devlink.c
 create mode 100644 drivers/net/ethernet/amazon/ena/ena_devlink.h

-- 
2.38.1

[PATCH V1 net-next 1/5] net: ena: Register ena device to devlink

[David Arinzon]

This patch registers ena as a device that supports devlink.
This makes it listed when running
        $ devlink dev show

The patch lands the base upon which the driver's devlink callbacks will
be added.

Signed-off-by: Shay Agroskin <shayagr@amazon.com>
Signed-off-by: David Arinzon <darinzon@amazon.com>
---
 drivers/net/ethernet/amazon/Kconfig           |  1 +
 drivers/net/ethernet/amazon/ena/Makefile      |  2 +-
 drivers/net/ethernet/amazon/ena/ena_devlink.c | 42 +++++++++++++++++++
 drivers/net/ethernet/amazon/ena/ena_devlink.h | 20 +++++++++
 drivers/net/ethernet/amazon/ena/ena_netdev.c  | 24 ++++++++++-
 drivers/net/ethernet/amazon/ena/ena_netdev.h  |  2 +
 6 files changed, 88 insertions(+), 3 deletions(-)
 create mode 100644 drivers/net/ethernet/amazon/ena/ena_devlink.c
 create mode 100644 drivers/net/ethernet/amazon/ena/ena_devlink.h

diff --git a/drivers/net/ethernet/amazon/Kconfig b/drivers/net/ethernet/amazon/Kconfig
index c37fa393b99e..8f1b3302a89e 100644
--- a/drivers/net/ethernet/amazon/Kconfig
+++ b/drivers/net/ethernet/amazon/Kconfig
@@ -18,6 +18,7 @@ if NET_VENDOR_AMAZON
 
 config ENA_ETHERNET
 	tristate "Elastic Network Adapter (ENA) support"
+	select NET_DEVLINK
 	depends on PCI_MSI && !CPU_BIG_ENDIAN
 	select DIMLIB
 	help
diff --git a/drivers/net/ethernet/amazon/ena/Makefile b/drivers/net/ethernet/amazon/ena/Makefile
index f1f752a8f7bb..34abcedd9132 100644
--- a/drivers/net/ethernet/amazon/ena/Makefile
+++ b/drivers/net/ethernet/amazon/ena/Makefile
@@ -5,4 +5,4 @@
 
 obj-$(CONFIG_ENA_ETHERNET) += ena.o
 
-ena-y := ena_netdev.o ena_com.o ena_eth_com.o ena_ethtool.o
+ena-y := ena_netdev.o ena_com.o ena_eth_com.o ena_ethtool.o ena_devlink.o
diff --git a/drivers/net/ethernet/amazon/ena/ena_devlink.c b/drivers/net/ethernet/amazon/ena/ena_devlink.c
new file mode 100644
index 000000000000..6897d60d8376
--- /dev/null
+++ b/drivers/net/ethernet/amazon/ena/ena_devlink.c
@@ -0,0 +1,42 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/*
+ * Copyright 2015-2021 Amazon.com, Inc. or its affiliates. All rights reserved.
+ */
+
+#include "linux/pci.h"
+
+#include "ena_devlink.h"
+
+static const struct devlink_ops ena_devlink_ops = {};
+
+struct devlink *ena_devlink_alloc(struct ena_adapter *adapter)
+{
+	struct device *dev = &adapter->pdev->dev;
+	struct devlink *devlink;
+
+	devlink = devlink_alloc(&ena_devlink_ops, sizeof(struct ena_adapter *), dev);
+	if (!devlink) {
+		netdev_err(adapter->netdev, "Failed to allocate devlink struct\n");
+		return NULL;
+	}
+
+	ENA_DEVLINK_PRIV(devlink) = adapter;
+	adapter->devlink = devlink;
+
+	return devlink;
+}
+
+void ena_devlink_free(struct devlink *devlink)
+{
+	devlink_free(devlink);
+}
+
+void ena_devlink_register(struct devlink *devlink, struct device *dev)
+{
+	devlink_register(devlink);
+}
+
+void ena_devlink_unregister(struct devlink *devlink)
+{
+	devlink_unregister(devlink);
+}
diff --git a/drivers/net/ethernet/amazon/ena/ena_devlink.h b/drivers/net/ethernet/amazon/ena/ena_devlink.h
new file mode 100644
index 000000000000..6f737884b850
--- /dev/null
+++ b/drivers/net/ethernet/amazon/ena/ena_devlink.h
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/*
+ * Copyright 2015-2021 Amazon.com, Inc. or its affiliates. All rights reserved.
+ */
+
+#ifndef DEVLINK_H
+#define DEVLINK_H
+
+#include "ena_netdev.h"
+#include <net/devlink.h>
+
+#define ENA_DEVLINK_PRIV(devlink) \
+	(*(struct ena_adapter **)devlink_priv(devlink))
+
+struct devlink *ena_devlink_alloc(struct ena_adapter *adapter);
+void ena_devlink_free(struct devlink *devlink);
+void ena_devlink_register(struct devlink *devlink, struct device *dev);
+void ena_devlink_unregister(struct devlink *devlink);
+
+#endif /* DEVLINK_H */
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index e8ad5ea31aff..ce79a0c42e6a 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -22,6 +22,8 @@
 #include <linux/bpf_trace.h>
 #include "ena_pci_id_tbl.h"
 
+#include "ena_devlink.h"
+
 MODULE_AUTHOR("Amazon.com, Inc. or its affiliates");
 MODULE_DESCRIPTION(DEVICE_NAME);
 MODULE_LICENSE("GPL");
@@ -4243,6 +4245,7 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	struct ena_adapter *adapter;
 	struct net_device *netdev;
 	static int adapters_found;
+	struct devlink *devlink;
 	u32 max_num_io_queues;
 	bool wd_state;
 	int bars, rc;
@@ -4308,12 +4311,18 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	pci_set_drvdata(pdev, adapter);
 
-	rc = ena_device_init(ena_dev, pdev, &get_feat_ctx, &wd_state);
+	devlink = ena_devlink_alloc(adapter);
+	if (!devlink) {
+		netdev_err(netdev, "ena_devlink_alloc failed\n");
+		goto err_netdev_destroy;
+	}
+
+	rc = ena_device_init(adapter, pdev, &get_feat_ctx, &wd_state);
 	if (rc) {
 		dev_err(&pdev->dev, "ENA device init failed\n");
 		if (rc == -ETIME)
 			rc = -EPROBE_DEFER;
-		goto err_netdev_destroy;
+		goto err_devlink_destroy;
 	}
 
 	rc = ena_map_llq_mem_bar(pdev, ena_dev, bars);
@@ -4362,6 +4371,7 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 			"Failed to query interrupt moderation feature\n");
 		goto err_device_destroy;
 	}
+
 	ena_init_io_rings(adapter,
 			  0,
 			  adapter->xdp_num_queues +
@@ -4420,6 +4430,8 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	adapters_found++;
 
+	ena_devlink_register(devlink, &pdev->dev);
+
 	return 0;
 
 err_rss:
@@ -4436,6 +4448,8 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 err_device_destroy:
 	ena_com_delete_host_info(ena_dev);
 	ena_com_admin_destroy(ena_dev);
+err_devlink_destroy:
+	ena_devlink_free(devlink);
 err_netdev_destroy:
 	free_netdev(netdev);
 err_free_region:
@@ -4462,10 +4476,15 @@ static void __ena_shutoff(struct pci_dev *pdev, bool shutdown)
 	struct ena_adapter *adapter = pci_get_drvdata(pdev);
 	struct ena_com_dev *ena_dev;
 	struct net_device *netdev;
+	struct devlink *devlink;
 
 	ena_dev = adapter->ena_dev;
 	netdev = adapter->netdev;
 
+	devlink = adapter->devlink;
+	ena_devlink_unregister(devlink);
+	ena_devlink_free(devlink);
+
 #ifdef CONFIG_RFS_ACCEL
 	if ((adapter->msix_vecs >= 1) && (netdev->rx_cpu_rmap)) {
 		free_irq_cpu_rmap(netdev->rx_cpu_rmap);
@@ -4482,6 +4501,7 @@ static void __ena_shutoff(struct pci_dev *pdev, bool shutdown)
 	rtnl_lock(); /* lock released inside the below if-else block */
 	adapter->reset_reason = ENA_REGS_RESET_SHUTDOWN;
 	ena_destroy_device(adapter, true);
+
 	if (shutdown) {
 		netif_device_detach(netdev);
 		dev_close(netdev);
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.h b/drivers/net/ethernet/amazon/ena/ena_netdev.h
index 2cb141079474..c6132aa229df 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.h
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.h
@@ -313,6 +313,8 @@ struct ena_adapter {
 	struct net_device *netdev;
 	struct pci_dev *pdev;
 
+	struct devlink *devlink;
+
 	/* rx packets that shorter that this len will be copied to the skb
 	 * header
 	 */
-- 
2.38.1

Re: [PATCH V1 net-next 1/5] net: ena: Register ena device to devlink

[Kuniyuki Iwashima]

From: David Arinzon <darinzon@amazon.com>
Date:   Sun, 8 Jan 2023 10:35:29 +0000
> This patch registers ena as a device that supports devlink.
> This makes it listed when running
>         $ devlink dev show
> 
> The patch lands the base upon which the driver's devlink callbacks will
> be added.
> 
> Signed-off-by: Shay Agroskin <shayagr@amazon.com>
> Signed-off-by: David Arinzon <darinzon@amazon.com>
> ---
>  drivers/net/ethernet/amazon/Kconfig           |  1 +
>  drivers/net/ethernet/amazon/ena/Makefile      |  2 +-
>  drivers/net/ethernet/amazon/ena/ena_devlink.c | 42 +++++++++++++++++++
>  drivers/net/ethernet/amazon/ena/ena_devlink.h | 20 +++++++++
>  drivers/net/ethernet/amazon/ena/ena_netdev.c  | 24 ++++++++++-
>  drivers/net/ethernet/amazon/ena/ena_netdev.h  |  2 +
>  6 files changed, 88 insertions(+), 3 deletions(-)
>  create mode 100644 drivers/net/ethernet/amazon/ena/ena_devlink.c
>  create mode 100644 drivers/net/ethernet/amazon/ena/ena_devlink.h
> 
> diff --git a/drivers/net/ethernet/amazon/Kconfig b/drivers/net/ethernet/amazon/Kconfig
> index c37fa393b99e..8f1b3302a89e 100644
> --- a/drivers/net/ethernet/amazon/Kconfig
> +++ b/drivers/net/ethernet/amazon/Kconfig
> @@ -18,6 +18,7 @@ if NET_VENDOR_AMAZON
>  
>  config ENA_ETHERNET
>  	tristate "Elastic Network Adapter (ENA) support"
> +	select NET_DEVLINK
>  	depends on PCI_MSI && !CPU_BIG_ENDIAN
>  	select DIMLIB
>  	help
> diff --git a/drivers/net/ethernet/amazon/ena/Makefile b/drivers/net/ethernet/amazon/ena/Makefile
> index f1f752a8f7bb..34abcedd9132 100644
> --- a/drivers/net/ethernet/amazon/ena/Makefile
> +++ b/drivers/net/ethernet/amazon/ena/Makefile
> @@ -5,4 +5,4 @@
>  
>  obj-$(CONFIG_ENA_ETHERNET) += ena.o
>  
> -ena-y := ena_netdev.o ena_com.o ena_eth_com.o ena_ethtool.o
> +ena-y := ena_netdev.o ena_com.o ena_eth_com.o ena_ethtool.o ena_devlink.o
> diff --git a/drivers/net/ethernet/amazon/ena/ena_devlink.c b/drivers/net/ethernet/amazon/ena/ena_devlink.c
> new file mode 100644
> index 000000000000..6897d60d8376
> --- /dev/null
> +++ b/drivers/net/ethernet/amazon/ena/ena_devlink.c
> @@ -0,0 +1,42 @@
> +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> +/*
> + * Copyright 2015-2021 Amazon.com, Inc. or its affiliates. All rights reserved.

nit: s/2015-2021 //

---8<---
Copyright dates are not needed in open source code that Amazon creates. I.e.,
“Copyright [YEAR]-[YEAR] Amazon.com Inc. or its affiliates” will become
“Copyright Amazon.com Inc. or its affiliates.”
---8<---


> + */
> +
> +#include "linux/pci.h"
> +
> +#include "ena_devlink.h"
> +
> +static const struct devlink_ops ena_devlink_ops = {};
> +
> +struct devlink *ena_devlink_alloc(struct ena_adapter *adapter)
> +{
> +	struct device *dev = &adapter->pdev->dev;
> +	struct devlink *devlink;
> +
> +	devlink = devlink_alloc(&ena_devlink_ops, sizeof(struct ena_adapter *), dev);
> +	if (!devlink) {
> +		netdev_err(adapter->netdev, "Failed to allocate devlink struct\n");
> +		return NULL;
> +	}
> +
> +	ENA_DEVLINK_PRIV(devlink) = adapter;
> +	adapter->devlink = devlink;
> +
> +	return devlink;
> +}
> +
> +void ena_devlink_free(struct devlink *devlink)
> +{
> +	devlink_free(devlink);
> +}
> +
> +void ena_devlink_register(struct devlink *devlink, struct device *dev)
> +{
> +	devlink_register(devlink);
> +}
> +
> +void ena_devlink_unregister(struct devlink *devlink)
> +{
> +	devlink_unregister(devlink);
> +}
> diff --git a/drivers/net/ethernet/amazon/ena/ena_devlink.h b/drivers/net/ethernet/amazon/ena/ena_devlink.h
> new file mode 100644
> index 000000000000..6f737884b850
> --- /dev/null
> +++ b/drivers/net/ethernet/amazon/ena/ena_devlink.h
> @@ -0,0 +1,20 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
> +/*
> + * Copyright 2015-2021 Amazon.com, Inc. or its affiliates. All rights reserved.

Same here.


> + */
> +
> +#ifndef DEVLINK_H
> +#define DEVLINK_H
> +
> +#include "ena_netdev.h"
> +#include <net/devlink.h>
> +
> +#define ENA_DEVLINK_PRIV(devlink) \
> +	(*(struct ena_adapter **)devlink_priv(devlink))
> +
> +struct devlink *ena_devlink_alloc(struct ena_adapter *adapter);
> +void ena_devlink_free(struct devlink *devlink);
> +void ena_devlink_register(struct devlink *devlink, struct device *dev);
> +void ena_devlink_unregister(struct devlink *devlink);
> +
> +#endif /* DEVLINK_H */
> diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
> index e8ad5ea31aff..ce79a0c42e6a 100644
> --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
> +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
> @@ -22,6 +22,8 @@
>  #include <linux/bpf_trace.h>
>  #include "ena_pci_id_tbl.h"
>  
> +#include "ena_devlink.h"
> +
>  MODULE_AUTHOR("Amazon.com, Inc. or its affiliates");
>  MODULE_DESCRIPTION(DEVICE_NAME);
>  MODULE_LICENSE("GPL");
> @@ -4243,6 +4245,7 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
>  	struct ena_adapter *adapter;
>  	struct net_device *netdev;
>  	static int adapters_found;
> +	struct devlink *devlink;
>  	u32 max_num_io_queues;
>  	bool wd_state;
>  	int bars, rc;
> @@ -4308,12 +4311,18 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
>  
>  	pci_set_drvdata(pdev, adapter);
>  
> -	rc = ena_device_init(ena_dev, pdev, &get_feat_ctx, &wd_state);
> +	devlink = ena_devlink_alloc(adapter);
> +	if (!devlink) {
> +		netdev_err(netdev, "ena_devlink_alloc failed\n");
> +		goto err_netdev_destroy;
> +	}
> +
> +	rc = ena_device_init(adapter, pdev, &get_feat_ctx, &wd_state);
>  	if (rc) {
>  		dev_err(&pdev->dev, "ENA device init failed\n");
>  		if (rc == -ETIME)
>  			rc = -EPROBE_DEFER;
> -		goto err_netdev_destroy;
> +		goto err_devlink_destroy;
>  	}
>  
>  	rc = ena_map_llq_mem_bar(pdev, ena_dev, bars);
> @@ -4362,6 +4371,7 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
>  			"Failed to query interrupt moderation feature\n");
>  		goto err_device_destroy;
>  	}
> +
>  	ena_init_io_rings(adapter,
>  			  0,
>  			  adapter->xdp_num_queues +
> @@ -4420,6 +4430,8 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
>  
>  	adapters_found++;
>  
> +	ena_devlink_register(devlink, &pdev->dev);
> +
>  	return 0;
>  
>  err_rss:
> @@ -4436,6 +4448,8 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
>  err_device_destroy:
>  	ena_com_delete_host_info(ena_dev);
>  	ena_com_admin_destroy(ena_dev);
> +err_devlink_destroy:
> +	ena_devlink_free(devlink);
>  err_netdev_destroy:
>  	free_netdev(netdev);
>  err_free_region:
> @@ -4462,10 +4476,15 @@ static void __ena_shutoff(struct pci_dev *pdev, bool shutdown)
>  	struct ena_adapter *adapter = pci_get_drvdata(pdev);
>  	struct ena_com_dev *ena_dev;
>  	struct net_device *netdev;
> +	struct devlink *devlink;
>  
>  	ena_dev = adapter->ena_dev;
>  	netdev = adapter->netdev;
>  
> +	devlink = adapter->devlink;
> +	ena_devlink_unregister(devlink);
> +	ena_devlink_free(devlink);
> +
>  #ifdef CONFIG_RFS_ACCEL
>  	if ((adapter->msix_vecs >= 1) && (netdev->rx_cpu_rmap)) {
>  		free_irq_cpu_rmap(netdev->rx_cpu_rmap);
> @@ -4482,6 +4501,7 @@ static void __ena_shutoff(struct pci_dev *pdev, bool shutdown)
>  	rtnl_lock(); /* lock released inside the below if-else block */
>  	adapter->reset_reason = ENA_REGS_RESET_SHUTDOWN;
>  	ena_destroy_device(adapter, true);
> +
>  	if (shutdown) {
>  		netif_device_detach(netdev);
>  		dev_close(netdev);
> diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.h b/drivers/net/ethernet/amazon/ena/ena_netdev.h
> index 2cb141079474..c6132aa229df 100644
> --- a/drivers/net/ethernet/amazon/ena/ena_netdev.h
> +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.h
> @@ -313,6 +313,8 @@ struct ena_adapter {
>  	struct net_device *netdev;
>  	struct pci_dev *pdev;
>  
> +	struct devlink *devlink;
> +
>  	/* rx packets that shorter that this len will be copied to the skb
>  	 * header
>  	 */
> -- 
> 2.38.1

RE: [PATCH V1 net-next 1/5] net: ena: Register ena device to devlink

[Arinzon, David]

Thank you! This feedback will be addressed in the next version of the patchset.

> -----Original Message-----
> From: Iwashima, Kuniyuki <kuniyu@amazon.co.jp>
> Sent: Monday, January 9, 2023 7:59 AM
> To: Arinzon, David <darinzon@amazon.com>
> Cc: Kiyanovski, Arthur <akiyano@amazon.com>; Saidi, Ali
> <alisaidi@amazon.com>; davem@davemloft.net; Itzko, Shahar
> <itzko@amazon.com>; kuba@kernel.org; Matushevsky, Alexander
> <matua@amazon.com>; Bshara, Nafea <nafea@amazon.com>; Dagan,
> Noam <ndagan@amazon.com>; netdev@vger.kernel.org; Abboud, Osama
> <osamaabb@amazon.com>; Bshara, Saeed <saeedb@amazon.com>;
> Agroskin, Shay <shayagr@amazon.com>; Machulsky, Zorik
> <zorik@amazon.com>; Iwashima, Kuniyuki <kuniyu@amazon.co.jp>
> Subject: Re: [PATCH V1 net-next 1/5] net: ena: Register ena device to
> devlink
> 
> From: David Arinzon <darinzon@amazon.com>
> Date:   Sun, 8 Jan 2023 10:35:29 +0000
> > This patch registers ena as a device that supports devlink.
> > This makes it listed when running
> >         $ devlink dev show
> >
> > The patch lands the base upon which the driver's devlink callbacks
> > will be added.
> >
> > Signed-off-by: Shay Agroskin <shayagr@amazon.com>
> > Signed-off-by: David Arinzon <darinzon@amazon.com>
> > ---
> >  drivers/net/ethernet/amazon/Kconfig           |  1 +
> >  drivers/net/ethernet/amazon/ena/Makefile      |  2 +-
> >  drivers/net/ethernet/amazon/ena/ena_devlink.c | 42
> > +++++++++++++++++++
> drivers/net/ethernet/amazon/ena/ena_devlink.h |
> > 20 +++++++++  drivers/net/ethernet/amazon/ena/ena_netdev.c  | 24
> > ++++++++++-  drivers/net/ethernet/amazon/ena/ena_netdev.h  |  2 +
> >  6 files changed, 88 insertions(+), 3 deletions(-)  create mode 100644
> > drivers/net/ethernet/amazon/ena/ena_devlink.c
> >  create mode 100644 drivers/net/ethernet/amazon/ena/ena_devlink.h
> >
> > diff --git a/drivers/net/ethernet/amazon/Kconfig
> > b/drivers/net/ethernet/amazon/Kconfig
> > index c37fa393b99e..8f1b3302a89e 100644
> > --- a/drivers/net/ethernet/amazon/Kconfig
> > +++ b/drivers/net/ethernet/amazon/Kconfig
> > @@ -18,6 +18,7 @@ if NET_VENDOR_AMAZON
> >
> >  config ENA_ETHERNET
> >  	tristate "Elastic Network Adapter (ENA) support"
> > +	select NET_DEVLINK
> >  	depends on PCI_MSI && !CPU_BIG_ENDIAN
> >  	select DIMLIB
> >  	help
> > diff --git a/drivers/net/ethernet/amazon/ena/Makefile
> > b/drivers/net/ethernet/amazon/ena/Makefile
> > index f1f752a8f7bb..34abcedd9132 100644
> > --- a/drivers/net/ethernet/amazon/ena/Makefile
> > +++ b/drivers/net/ethernet/amazon/ena/Makefile
> > @@ -5,4 +5,4 @@
> >
> >  obj-$(CONFIG_ENA_ETHERNET) += ena.o
> >
> > -ena-y := ena_netdev.o ena_com.o ena_eth_com.o ena_ethtool.o
> > +ena-y := ena_netdev.o ena_com.o ena_eth_com.o ena_ethtool.o
> > +ena_devlink.o
> > diff --git a/drivers/net/ethernet/amazon/ena/ena_devlink.c
> > b/drivers/net/ethernet/amazon/ena/ena_devlink.c
> > new file mode 100644
> > index 000000000000..6897d60d8376
> > --- /dev/null
> > +++ b/drivers/net/ethernet/amazon/ena/ena_devlink.c
> > @@ -0,0 +1,42 @@
> > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > +/*
> > + * Copyright 2015-2021 Amazon.com, Inc. or its affiliates. All rights
> reserved.
> 
> nit: s/2015-2021 //
> 
> ---8<---
> Copyright dates are not needed in open source code that Amazon creates.
> I.e., “Copyright [YEAR]-[YEAR] Amazon.com Inc. or its affiliates” will
> become “Copyright Amazon.com Inc. or its affiliates.”
> ---8<---
> 
> 
> > + */
> > +
> > +#include "linux/pci.h"
> > +
> > +#include "ena_devlink.h"
> > +
> > +static const struct devlink_ops ena_devlink_ops = {};
> > +
> > +struct devlink *ena_devlink_alloc(struct ena_adapter *adapter) {
> > +	struct device *dev = &adapter->pdev->dev;
> > +	struct devlink *devlink;
> > +
> > +	devlink = devlink_alloc(&ena_devlink_ops, sizeof(struct
> ena_adapter *), dev);
> > +	if (!devlink) {
> > +		netdev_err(adapter->netdev, "Failed to allocate devlink
> struct\n");
> > +		return NULL;
> > +	}
> > +
> > +	ENA_DEVLINK_PRIV(devlink) = adapter;
> > +	adapter->devlink = devlink;
> > +
> > +	return devlink;
> > +}
> > +
> > +void ena_devlink_free(struct devlink *devlink) {
> > +	devlink_free(devlink);
> > +}
> > +
> > +void ena_devlink_register(struct devlink *devlink, struct device
> > +*dev) {
> > +	devlink_register(devlink);
> > +}
> > +
> > +void ena_devlink_unregister(struct devlink *devlink) {
> > +	devlink_unregister(devlink);
> > +}
> > diff --git a/drivers/net/ethernet/amazon/ena/ena_devlink.h
> > b/drivers/net/ethernet/amazon/ena/ena_devlink.h
> > new file mode 100644
> > index 000000000000..6f737884b850
> > --- /dev/null
> > +++ b/drivers/net/ethernet/amazon/ena/ena_devlink.h
> > @@ -0,0 +1,20 @@
> > +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
> > +/*
> > + * Copyright 2015-2021 Amazon.com, Inc. or its affiliates. All rights
> reserved.
> 
> Same here.
> 
> 
> > + */
> > +
> > +#ifndef DEVLINK_H
> > +#define DEVLINK_H
> > +
> > +#include "ena_netdev.h"
> > +#include <net/devlink.h>
> > +
> > +#define ENA_DEVLINK_PRIV(devlink) \
> > +	(*(struct ena_adapter **)devlink_priv(devlink))
> > +
> > +struct devlink *ena_devlink_alloc(struct ena_adapter *adapter); void
> > +ena_devlink_free(struct devlink *devlink); void
> > +ena_devlink_register(struct devlink *devlink, struct device *dev);
> > +void ena_devlink_unregister(struct devlink *devlink);
> > +
> > +#endif /* DEVLINK_H */
> > diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c
> > b/drivers/net/ethernet/amazon/ena/ena_netdev.c
> > index e8ad5ea31aff..ce79a0c42e6a 100644
> > --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
> > +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
> > @@ -22,6 +22,8 @@
> >  #include <linux/bpf_trace.h>
> >  #include "ena_pci_id_tbl.h"
> >
> > +#include "ena_devlink.h"
> > +
> >  MODULE_AUTHOR("Amazon.com, Inc. or its affiliates");
> > MODULE_DESCRIPTION(DEVICE_NAME);  MODULE_LICENSE("GPL"); @@ -
> 4243,6
> > +4245,7 @@ static int ena_probe(struct pci_dev *pdev, const struct
> pci_device_id *ent)
> >  	struct ena_adapter *adapter;
> >  	struct net_device *netdev;
> >  	static int adapters_found;
> > +	struct devlink *devlink;
> >  	u32 max_num_io_queues;
> >  	bool wd_state;
> >  	int bars, rc;
> > @@ -4308,12 +4311,18 @@ static int ena_probe(struct pci_dev *pdev,
> > const struct pci_device_id *ent)
> >
> >  	pci_set_drvdata(pdev, adapter);
> >
> > -	rc = ena_device_init(ena_dev, pdev, &get_feat_ctx, &wd_state);
> > +	devlink = ena_devlink_alloc(adapter);
> > +	if (!devlink) {
> > +		netdev_err(netdev, "ena_devlink_alloc failed\n");
> > +		goto err_netdev_destroy;
> > +	}
> > +
> > +	rc = ena_device_init(adapter, pdev, &get_feat_ctx, &wd_state);
> >  	if (rc) {
> >  		dev_err(&pdev->dev, "ENA device init failed\n");
> >  		if (rc == -ETIME)
> >  			rc = -EPROBE_DEFER;
> > -		goto err_netdev_destroy;
> > +		goto err_devlink_destroy;
> >  	}
> >
> >  	rc = ena_map_llq_mem_bar(pdev, ena_dev, bars); @@ -4362,6
> +4371,7 @@
> > static int ena_probe(struct pci_dev *pdev, const struct pci_device_id
> *ent)
> >  			"Failed to query interrupt moderation feature\n");
> >  		goto err_device_destroy;
> >  	}
> > +
> >  	ena_init_io_rings(adapter,
> >  			  0,
> >  			  adapter->xdp_num_queues +
> > @@ -4420,6 +4430,8 @@ static int ena_probe(struct pci_dev *pdev,
> const
> > struct pci_device_id *ent)
> >
> >  	adapters_found++;
> >
> > +	ena_devlink_register(devlink, &pdev->dev);
> > +
> >  	return 0;
> >
> >  err_rss:
> > @@ -4436,6 +4448,8 @@ static int ena_probe(struct pci_dev *pdev,
> const
> > struct pci_device_id *ent)
> >  err_device_destroy:
> >  	ena_com_delete_host_info(ena_dev);
> >  	ena_com_admin_destroy(ena_dev);
> > +err_devlink_destroy:
> > +	ena_devlink_free(devlink);
> >  err_netdev_destroy:
> >  	free_netdev(netdev);
> >  err_free_region:
> > @@ -4462,10 +4476,15 @@ static void __ena_shutoff(struct pci_dev
> *pdev, bool shutdown)
> >  	struct ena_adapter *adapter = pci_get_drvdata(pdev);
> >  	struct ena_com_dev *ena_dev;
> >  	struct net_device *netdev;
> > +	struct devlink *devlink;
> >
> >  	ena_dev = adapter->ena_dev;
> >  	netdev = adapter->netdev;
> >
> > +	devlink = adapter->devlink;
> > +	ena_devlink_unregister(devlink);
> > +	ena_devlink_free(devlink);
> > +
> >  #ifdef CONFIG_RFS_ACCEL
> >  	if ((adapter->msix_vecs >= 1) && (netdev->rx_cpu_rmap)) {
> >  		free_irq_cpu_rmap(netdev->rx_cpu_rmap);
> > @@ -4482,6 +4501,7 @@ static void __ena_shutoff(struct pci_dev
> *pdev, bool shutdown)
> >  	rtnl_lock(); /* lock released inside the below if-else block */
> >  	adapter->reset_reason = ENA_REGS_RESET_SHUTDOWN;
> >  	ena_destroy_device(adapter, true);
> > +
> >  	if (shutdown) {
> >  		netif_device_detach(netdev);
> >  		dev_close(netdev);
> > diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.h
> > b/drivers/net/ethernet/amazon/ena/ena_netdev.h
> > index 2cb141079474..c6132aa229df 100644
> > --- a/drivers/net/ethernet/amazon/ena/ena_netdev.h
> > +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.h
> > @@ -313,6 +313,8 @@ struct ena_adapter {
> >  	struct net_device *netdev;
> >  	struct pci_dev *pdev;
> >
> > +	struct devlink *devlink;
> > +
> >  	/* rx packets that shorter that this len will be copied to the skb
> >  	 * header
> >  	 */
> > --
> > 2.38.1

[PATCH V1 net-next 2/5] net: ena: Add devlink reload functionality

[David Arinzon]

This system allows to reload most of the driver's resources.
This functionality is added to our driver to apply configurations which
require a reset (such as changing LLQ entry size).

For the implementation of reload functionality, the driver performs
the same sequence that device reset performs with a few exceptions:
  - The reset occurs immediately rather than setting a reset flag which
    would cause the timer routine to trigger the reset.
    This is done to provide a smoother user experience,
    which makes sure the reset operation is done by the time
    the 'devlink reload' command returns.

  - Destruction of driver resources (using ena_destroy_device()) and
    their re-initialization (using ena_restore_device()) is done without
    holding the rtnl_lock() throughout the 'devlink reload' execution,
    but rather with holding it for *each* of the operations separately.

Signed-off-by: Shay Agroskin <shayagr@amazon.com>
Signed-off-by: David Arinzon <darinzon@amazon.com>
---
 drivers/net/ethernet/amazon/ena/ena_devlink.c | 73 ++++++++++++++++++-
 drivers/net/ethernet/amazon/ena/ena_netdev.c  |  6 +-
 drivers/net/ethernet/amazon/ena/ena_netdev.h  |  3 +
 3 files changed, 77 insertions(+), 5 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_devlink.c b/drivers/net/ethernet/amazon/ena/ena_devlink.c
index 6897d60d8376..2568ade34c2a 100644
--- a/drivers/net/ethernet/amazon/ena/ena_devlink.c
+++ b/drivers/net/ethernet/amazon/ena/ena_devlink.c
@@ -7,7 +7,76 @@
 
 #include "ena_devlink.h"
 
-static const struct devlink_ops ena_devlink_ops = {};
+static int ena_devlink_reload_down(struct devlink *devlink,
+				   bool netns_change,
+				   enum devlink_reload_action action,
+				   enum devlink_reload_limit limit,
+				   struct netlink_ext_ack *extack)
+{
+	struct ena_adapter *adapter = ENA_DEVLINK_PRIV(devlink);
+
+	if (netns_change) {
+		NL_SET_ERR_MSG_MOD(extack, "Namespace change is not supported");
+		return -EOPNOTSUPP;
+	}
+
+	if (action != DEVLINK_RELOAD_ACTION_DRIVER_REINIT) {
+		NL_SET_ERR_MSG_MOD(extack, "Action is not supported");
+		return -EOPNOTSUPP;
+	}
+
+	if (limit != DEVLINK_RELOAD_LIMIT_UNSPEC) {
+		NL_SET_ERR_MSG_MOD(extack, "Driver reload doesn't support limitations");
+		return -EOPNOTSUPP;
+	}
+
+	rtnl_lock();
+	ena_destroy_device(adapter, false);
+	rtnl_unlock();
+
+	return 0;
+}
+
+static int ena_devlink_reload_up(struct devlink *devlink,
+				 enum devlink_reload_action action,
+				 enum devlink_reload_limit limit,
+				 u32 *actions_performed,
+				 struct netlink_ext_ack *extack)
+{
+	struct ena_adapter *adapter = ENA_DEVLINK_PRIV(devlink);
+	int err = 0;
+
+	if (action != DEVLINK_RELOAD_ACTION_DRIVER_REINIT) {
+		NL_SET_ERR_MSG_MOD(extack, "Action is not supported");
+		return -EOPNOTSUPP;
+	}
+
+	if (limit != DEVLINK_RELOAD_LIMIT_UNSPEC) {
+		NL_SET_ERR_MSG_MOD(extack, "Driver reload doesn't support limitations");
+		return -EOPNOTSUPP;
+	}
+
+	rtnl_lock();
+	/* Check that no other routine initialized the device (e.g.
+	 * ena_fw_reset_device()). Also we're under devlink_mutex here,
+	 * so devlink isn't freed under our feet.
+	 */
+	if (!test_bit(ENA_FLAG_DEVICE_RUNNING, &adapter->flags))
+		err = ena_restore_device(adapter);
+
+	rtnl_unlock();
+
+	if (!err)
+		*actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
+
+	return err;
+}
+
+static const struct devlink_ops ena_devlink_ops = {
+	.reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
+	.reload_down	= ena_devlink_reload_down,
+	.reload_up	= ena_devlink_reload_up,
+};
 
 struct devlink *ena_devlink_alloc(struct ena_adapter *adapter)
 {
@@ -20,6 +89,8 @@ struct devlink *ena_devlink_alloc(struct ena_adapter *adapter)
 		return NULL;
 	}
 
+	devlink_set_features(devlink, DEVLINK_F_RELOAD);
+
 	ENA_DEVLINK_PRIV(devlink) = adapter;
 	adapter->devlink = devlink;
 
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index ce79a0c42e6a..a42db781472c 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -44,8 +44,6 @@ MODULE_DEVICE_TABLE(pci, ena_pci_tbl);
 
 static int ena_rss_init_default(struct ena_adapter *adapter);
 static void check_for_admin_com_state(struct ena_adapter *adapter);
-static void ena_destroy_device(struct ena_adapter *adapter, bool graceful);
-static int ena_restore_device(struct ena_adapter *adapter);
 
 static void ena_init_io_rings(struct ena_adapter *adapter,
 			      int first_index, int count);
@@ -3587,7 +3585,7 @@ static int ena_enable_msix_and_set_admin_interrupts(struct ena_adapter *adapter)
 	return rc;
 }
 
-static void ena_destroy_device(struct ena_adapter *adapter, bool graceful)
+void ena_destroy_device(struct ena_adapter *adapter, bool graceful)
 {
 	struct net_device *netdev = adapter->netdev;
 	struct ena_com_dev *ena_dev = adapter->ena_dev;
@@ -3633,7 +3631,7 @@ static void ena_destroy_device(struct ena_adapter *adapter, bool graceful)
 	clear_bit(ENA_FLAG_DEVICE_RUNNING, &adapter->flags);
 }
 
-static int ena_restore_device(struct ena_adapter *adapter)
+int ena_restore_device(struct ena_adapter *adapter)
 {
 	struct ena_com_dev_get_features_ctx get_feat_ctx;
 	struct ena_com_dev *ena_dev = adapter->ena_dev;
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.h b/drivers/net/ethernet/amazon/ena/ena_netdev.h
index c6132aa229df..244c80af6974 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.h
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.h
@@ -452,4 +452,7 @@ static inline enum ena_xdp_errors_t ena_xdp_allowed(struct ena_adapter *adapter)
 	return rc;
 }
 
+void ena_destroy_device(struct ena_adapter *adapter, bool graceful);
+int ena_restore_device(struct ena_adapter *adapter);
+
 #endif /* !(ENA_H) */
-- 
2.38.1

[PATCH V1 net-next 3/5] net: ena: Configure large LLQ using devlink params

[David Arinzon]

This change introduces devlink params infrastructure
to the ena driver as well as the ability to enable large LLQ
configuration through the infrastructure.

Default LLQ entry size is 128 bytes. 128 bytes entry size
allows for a maximum of 96 bytes of packet header size which
sometimes is not enough (e.g. when using tunneling).
Increasing LLQ entry size to 256 bytes, by enabling large
LLQ through devlink, allows a maximum header size of 224 bytes.
This comes with the penalty of reducing the number of LLQ
entries in the TX queue by 2 (i.e. from 1024 to 512).

Signed-off-by: Shay Agroskin <shayagr@amazon.com>
Signed-off-by: David Arinzon <darinzon@amazon.com>
---
 drivers/net/ethernet/amazon/ena/ena_devlink.c | 106 +++++++++++++++++-
 drivers/net/ethernet/amazon/ena/ena_devlink.h |   2 +
 drivers/net/ethernet/amazon/ena/ena_netdev.c  |  47 +++++++-
 drivers/net/ethernet/amazon/ena/ena_netdev.h  |   8 ++
 4 files changed, 157 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_devlink.c b/drivers/net/ethernet/amazon/ena/ena_devlink.c
index 2568ade34c2a..25194c365299 100644
--- a/drivers/net/ethernet/amazon/ena/ena_devlink.c
+++ b/drivers/net/ethernet/amazon/ena/ena_devlink.c
@@ -7,6 +7,72 @@
 
 #include "ena_devlink.h"
 
+static int ena_devlink_llq_header_validate(struct devlink *devlink, u32 id,
+					   union devlink_param_value val,
+					   struct netlink_ext_ack *extack);
+
+enum ena_devlink_param_id {
+	ENA_DEVLINK_PARAM_ID_BASE = DEVLINK_PARAM_GENERIC_ID_MAX,
+	ENA_DEVLINK_PARAM_ID_LLQ_HEADER_SIZE,
+};
+
+static const struct devlink_param ena_devlink_params[] = {
+	DEVLINK_PARAM_DRIVER(ENA_DEVLINK_PARAM_ID_LLQ_HEADER_SIZE,
+			     "large_llq_header", DEVLINK_PARAM_TYPE_BOOL,
+			     BIT(DEVLINK_PARAM_CMODE_DRIVERINIT),
+			     NULL, NULL, ena_devlink_llq_header_validate),
+};
+
+static int ena_devlink_llq_header_validate(struct devlink *devlink, u32 id,
+					   union devlink_param_value val,
+					   struct netlink_ext_ack *extack)
+{
+	struct ena_adapter *adapter = ENA_DEVLINK_PRIV(devlink);
+	bool value = val.vbool;
+
+	if (!value)
+		return 0;
+
+	if (adapter->ena_dev->tx_mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_HOST) {
+		NL_SET_ERR_MSG_MOD(extack, "Instance doesn't support LLQ");
+		return -EOPNOTSUPP;
+	}
+
+	if (!adapter->large_llq_header_supported) {
+		NL_SET_ERR_MSG_MOD(extack, "Instance doesn't support large LLQ");
+		return -EOPNOTSUPP;
+	}
+
+	return 0;
+}
+
+void ena_devlink_params_get(struct devlink *devlink)
+{
+	struct ena_adapter *adapter = ENA_DEVLINK_PRIV(devlink);
+	union devlink_param_value val;
+	int err;
+
+	err = devlink_param_driverinit_value_get(devlink,
+						 ENA_DEVLINK_PARAM_ID_LLQ_HEADER_SIZE,
+						 &val);
+	if (err) {
+		netdev_err(adapter->netdev, "Failed to query LLQ header size param\n");
+		return;
+	}
+
+	adapter->large_llq_header_enabled = val.vbool;
+}
+
+void ena_devlink_disable_large_llq_header_param(struct devlink *devlink)
+{
+	union devlink_param_value value;
+
+	value.vbool = false;
+	devlink_param_driverinit_value_set(devlink,
+					   ENA_DEVLINK_PARAM_ID_LLQ_HEADER_SIZE,
+					   value);
+}
+
 static int ena_devlink_reload_down(struct devlink *devlink,
 				   bool netns_change,
 				   enum devlink_reload_action action,
@@ -78,6 +144,29 @@ static const struct devlink_ops ena_devlink_ops = {
 	.reload_up	= ena_devlink_reload_up,
 };
 
+static int ena_devlink_configure_params(struct devlink *devlink)
+{
+	struct ena_adapter *adapter = ENA_DEVLINK_PRIV(devlink);
+	union devlink_param_value value;
+	int rc;
+
+	rc = devlink_params_register(devlink, ena_devlink_params,
+				     ARRAY_SIZE(ena_devlink_params));
+	if (rc) {
+		netdev_err(adapter->netdev, "Failed to register devlink params\n");
+		return rc;
+	}
+
+	value.vbool = adapter->large_llq_header_enabled;
+	devlink_param_driverinit_value_set(devlink,
+					   ENA_DEVLINK_PARAM_ID_LLQ_HEADER_SIZE,
+					   value);
+
+	devlink_set_features(devlink, DEVLINK_F_RELOAD);
+
+	return 0;
+}
+
 struct devlink *ena_devlink_alloc(struct ena_adapter *adapter)
 {
 	struct device *dev = &adapter->pdev->dev;
@@ -89,16 +178,29 @@ struct devlink *ena_devlink_alloc(struct ena_adapter *adapter)
 		return NULL;
 	}
 
-	devlink_set_features(devlink, DEVLINK_F_RELOAD);
-
 	ENA_DEVLINK_PRIV(devlink) = adapter;
 	adapter->devlink = devlink;
 
+	if (ena_devlink_configure_params(devlink))
+		goto free_devlink;
+
 	return devlink;
+free_devlink:
+	devlink_free(devlink);
+
+	return NULL;
+}
+
+static void ena_devlink_configure_params_clean(struct devlink *devlink)
+{
+	devlink_params_unregister(devlink, ena_devlink_params,
+				  ARRAY_SIZE(ena_devlink_params));
 }
 
 void ena_devlink_free(struct devlink *devlink)
 {
+	ena_devlink_configure_params_clean(devlink);
+
 	devlink_free(devlink);
 }
 
diff --git a/drivers/net/ethernet/amazon/ena/ena_devlink.h b/drivers/net/ethernet/amazon/ena/ena_devlink.h
index 6f737884b850..9db6038ecd62 100644
--- a/drivers/net/ethernet/amazon/ena/ena_devlink.h
+++ b/drivers/net/ethernet/amazon/ena/ena_devlink.h
@@ -16,5 +16,7 @@ struct devlink *ena_devlink_alloc(struct ena_adapter *adapter);
 void ena_devlink_free(struct devlink *devlink);
 void ena_devlink_register(struct devlink *devlink, struct device *dev);
 void ena_devlink_unregister(struct devlink *devlink);
+void ena_devlink_params_get(struct devlink *devlink);
+void ena_devlink_disable_large_llq_header_param(struct devlink *devlink);
 
 #endif /* DEVLINK_H */
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index a42db781472c..24b95765bb04 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -3385,13 +3385,30 @@ static int ena_device_validate_params(struct ena_adapter *adapter,
 	return 0;
 }
 
-static void set_default_llq_configurations(struct ena_llq_configurations *llq_config)
+static void set_default_llq_configurations(struct ena_adapter *adapter,
+					   struct ena_llq_configurations *llq_config,
+					   struct ena_admin_feature_llq_desc *llq)
 {
+	struct ena_com_dev *ena_dev = adapter->ena_dev;
+
 	llq_config->llq_header_location = ENA_ADMIN_INLINE_HEADER;
 	llq_config->llq_stride_ctrl = ENA_ADMIN_MULTIPLE_DESCS_PER_ENTRY;
 	llq_config->llq_num_decs_before_header = ENA_ADMIN_LLQ_NUM_DESCS_BEFORE_HEADER_2;
-	llq_config->llq_ring_entry_size = ENA_ADMIN_LIST_ENTRY_SIZE_128B;
-	llq_config->llq_ring_entry_size_value = 128;
+
+	adapter->large_llq_header_supported =
+		!!(ena_dev->supported_features & (1 << ENA_ADMIN_LLQ));
+	adapter->large_llq_header_supported &=
+		!!(llq->entry_size_ctrl_supported &
+			ENA_ADMIN_LIST_ENTRY_SIZE_256B);
+
+	if ((llq->entry_size_ctrl_supported & ENA_ADMIN_LIST_ENTRY_SIZE_256B) &&
+	    adapter->large_llq_header_enabled) {
+		llq_config->llq_ring_entry_size = ENA_ADMIN_LIST_ENTRY_SIZE_256B;
+		llq_config->llq_ring_entry_size_value = 256;
+	} else {
+		llq_config->llq_ring_entry_size = ENA_ADMIN_LIST_ENTRY_SIZE_128B;
+		llq_config->llq_ring_entry_size_value = 128;
+	}
 }
 
 static int ena_set_queues_placement_policy(struct pci_dev *pdev,
@@ -3493,6 +3510,8 @@ static int ena_device_init(struct ena_com_dev *ena_dev, struct pci_dev *pdev,
 		goto err_mmio_read_less;
 	}
 
+	ena_devlink_params_get(adapter->devlink);
+
 	/* ENA admin level init */
 	rc = ena_com_admin_init(ena_dev, &aenq_handlers);
 	if (rc) {
@@ -3533,7 +3552,7 @@ static int ena_device_init(struct ena_com_dev *ena_dev, struct pci_dev *pdev,
 
 	*wd_state = !!(aenq_groups & BIT(ENA_ADMIN_KEEP_ALIVE));
 
-	set_default_llq_configurations(&llq_config);
+	set_default_llq_configurations(adapter, &llq_config, &get_feat_ctx->llq);
 
 	rc = ena_set_queues_placement_policy(pdev, ena_dev, &get_feat_ctx->llq,
 					     &llq_config);
@@ -4212,6 +4231,26 @@ static void ena_calc_io_queue_size(struct ena_adapter *adapter,
 	max_tx_queue_size = rounddown_pow_of_two(max_tx_queue_size);
 	max_rx_queue_size = rounddown_pow_of_two(max_rx_queue_size);
 
+	/* When forcing large headers, we multiply the entry size by 2,
+	 * and therefore divide the queue size by 2, leaving the amount
+	 * of memory used by the queues unchanged.
+	 */
+	if (adapter->large_llq_header_enabled) {
+		if ((llq->entry_size_ctrl_supported & ENA_ADMIN_LIST_ENTRY_SIZE_256B) &&
+		    (ena_dev->tx_mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_DEV)) {
+			max_tx_queue_size /= 2;
+			dev_info(&adapter->pdev->dev,
+				 "Forcing large headers and decreasing maximum TX queue size to %d\n",
+				 max_tx_queue_size);
+		} else {
+			dev_err(&adapter->pdev->dev,
+				"Forcing large headers failed: LLQ is disabled or device does not support large headers\n");
+
+			adapter->large_llq_header_enabled = false;
+			ena_devlink_disable_large_llq_header_param(adapter->devlink);
+		}
+	}
+
 	tx_queue_size = clamp_val(tx_queue_size, ENA_MIN_RING_SIZE,
 				  max_tx_queue_size);
 	rx_queue_size = clamp_val(rx_queue_size, ENA_MIN_RING_SIZE,
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.h b/drivers/net/ethernet/amazon/ena/ena_netdev.h
index 244c80af6974..b6faf48373d2 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.h
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.h
@@ -336,6 +336,14 @@ struct ena_adapter {
 
 	u32 msg_enable;
 
+	/* The flag is used for two purposes:
+	 * 1. Indicates that large LLQ has been requested.
+	 * 2. Indicates whether large LLQ is set or not after device
+	 *    initialization / configuration.
+	 */
+	bool large_llq_header_enabled;
+	bool large_llq_header_supported;
+
 	u16 max_tx_sgl_size;
 	u16 max_rx_sgl_size;
 
-- 
2.38.1

[PATCH V1 net-next 4/5] net: ena: Several changes to support large LLQ configuration

[David Arinzon]

1. Allow queue related parameters to be recalculated upon every
   VF reset.
2. Handling of queues configured before ena_device_init() function
   is called.
3. Prevention of large LLQ configuration if LLQ is not
   supported.

Signed-off-by: Shay Agroskin <shayagr@amazon.com>
Signed-off-by: David Arinzon <darinzon@amazon.com>
---
 drivers/net/ethernet/amazon/ena/ena_netdev.c | 48 +++++++++++++-------
 1 file changed, 31 insertions(+), 17 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index 24b95765bb04..fac8c913de6b 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -44,6 +44,8 @@ MODULE_DEVICE_TABLE(pci, ena_pci_tbl);
 
 static int ena_rss_init_default(struct ena_adapter *adapter);
 static void check_for_admin_com_state(struct ena_adapter *adapter);
+static void ena_calc_io_queue_size(struct ena_adapter *adapter,
+				   struct ena_com_dev_get_features_ctx *get_feat_ctx);
 
 static void ena_init_io_rings(struct ena_adapter *adapter,
 			      int first_index, int count);
@@ -3427,6 +3429,13 @@ static int ena_set_queues_placement_policy(struct pci_dev *pdev,
 		return 0;
 	}
 
+	if (!ena_dev->mem_bar) {
+		netdev_err(ena_dev->net_device,
+			   "LLQ is advertised as supported but device doesn't expose mem bar\n");
+		ena_dev->tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_HOST;
+		return 0;
+	}
+
 	rc = ena_com_config_dev_mode(ena_dev, llq, llq_default_configurations);
 	if (unlikely(rc)) {
 		dev_err(&pdev->dev,
@@ -3442,15 +3451,8 @@ static int ena_map_llq_mem_bar(struct pci_dev *pdev, struct ena_com_dev *ena_dev
 {
 	bool has_mem_bar = !!(bars & BIT(ENA_MEM_BAR));
 
-	if (!has_mem_bar) {
-		if (ena_dev->tx_mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_DEV) {
-			dev_err(&pdev->dev,
-				"ENA device does not expose LLQ bar. Fallback to host mode policy.\n");
-			ena_dev->tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_HOST;
-		}
-
+	if (!has_mem_bar)
 		return 0;
-	}
 
 	ena_dev->mem_bar = devm_ioremap_wc(&pdev->dev,
 					   pci_resource_start(pdev, ENA_MEM_BAR),
@@ -3462,10 +3464,11 @@ static int ena_map_llq_mem_bar(struct pci_dev *pdev, struct ena_com_dev *ena_dev
 	return 0;
 }
 
-static int ena_device_init(struct ena_com_dev *ena_dev, struct pci_dev *pdev,
+static int ena_device_init(struct ena_adapter *adapter, struct pci_dev *pdev,
 			   struct ena_com_dev_get_features_ctx *get_feat_ctx,
 			   bool *wd_state)
 {
+	struct ena_com_dev *ena_dev = adapter->ena_dev;
 	struct ena_llq_configurations llq_config;
 	struct device *dev = &pdev->dev;
 	bool readless_supported;
@@ -3561,6 +3564,8 @@ static int ena_device_init(struct ena_com_dev *ena_dev, struct pci_dev *pdev,
 		goto err_admin_init;
 	}
 
+	ena_calc_io_queue_size(adapter, get_feat_ctx);
+
 	return 0;
 
 err_admin_init:
@@ -3659,7 +3664,7 @@ int ena_restore_device(struct ena_adapter *adapter)
 	int rc;
 
 	set_bit(ENA_FLAG_ONGOING_RESET, &adapter->flags);
-	rc = ena_device_init(ena_dev, adapter->pdev, &get_feat_ctx, &wd_state);
+	rc = ena_device_init(adapter, adapter->pdev, &get_feat_ctx, &wd_state);
 	if (rc) {
 		dev_err(&pdev->dev, "Can not initialize device\n");
 		goto err;
@@ -4187,9 +4192,19 @@ static void ena_calc_io_queue_size(struct ena_adapter *adapter,
 	struct ena_com_dev *ena_dev = adapter->ena_dev;
 	u32 tx_queue_size = ENA_DEFAULT_RING_SIZE;
 	u32 rx_queue_size = ENA_DEFAULT_RING_SIZE;
+	bool tx_configured, rx_configured;
 	u32 max_tx_queue_size;
 	u32 max_rx_queue_size;
 
+	/* If this function is called after driver load, the ring sizes have
+	 * already been configured. Take it into account when recalculating ring
+	 * size.
+	 */
+	tx_configured = !!adapter->tx_ring[0].ring_size;
+	rx_configured = !!adapter->rx_ring[0].ring_size;
+	tx_queue_size = tx_configured ? adapter->tx_ring[0].ring_size : tx_queue_size;
+	rx_queue_size = rx_configured ? adapter->rx_ring[0].ring_size : rx_queue_size;
+
 	if (ena_dev->supported_features & BIT(ENA_ADMIN_MAX_QUEUES_EXT)) {
 		struct ena_admin_queue_ext_feature_fields *max_queue_ext =
 			&get_feat_ctx->max_queue_ext.max_queue_ext;
@@ -4354,6 +4369,12 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 		goto err_netdev_destroy;
 	}
 
+	rc = ena_map_llq_mem_bar(pdev, ena_dev, bars);
+	if (rc) {
+		dev_err(&pdev->dev, "ENA LLQ bar mapping failed\n");
+		goto err_devlink_destroy;
+	}
+
 	rc = ena_device_init(adapter, pdev, &get_feat_ctx, &wd_state);
 	if (rc) {
 		dev_err(&pdev->dev, "ENA device init failed\n");
@@ -4362,12 +4383,6 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 		goto err_devlink_destroy;
 	}
 
-	rc = ena_map_llq_mem_bar(pdev, ena_dev, bars);
-	if (rc) {
-		dev_err(&pdev->dev, "ENA llq bar mapping failed\n");
-		goto err_device_destroy;
-	}
-
 	/* Initial TX and RX interrupt delay. Assumes 1 usec granularity.
 	 * Updated during device initialization with the real granularity
 	 */
@@ -4375,7 +4390,6 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	ena_dev->intr_moder_rx_interval = ENA_INTR_INITIAL_RX_INTERVAL_USECS;
 	ena_dev->intr_delay_resolution = ENA_DEFAULT_INTR_DELAY_RESOLUTION;
 	max_num_io_queues = ena_calc_max_io_queue_num(pdev, ena_dev, &get_feat_ctx);
-	ena_calc_io_queue_size(adapter, &get_feat_ctx);
 	if (unlikely(!max_num_io_queues)) {
 		rc = -EFAULT;
 		goto err_device_destroy;
-- 
2.38.1

[PATCH V1 net-next 5/5] net: ena: Add devlink documentation

[David Arinzon]

Update the documentation with a devlink section, the
added files, as well as large LLQ enablement.

Signed-off-by: Shay Agroskin <shayagr@amazon.com>
Signed-off-by: David Arinzon <darinzon@amazon.com>
---
 .../device_drivers/ethernet/amazon/ena.rst    | 30 +++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst
index 8bcb173e0353..1229732a8c91 100644
--- a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst
+++ b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst
@@ -53,6 +53,7 @@ ena_common_defs.h   Common definitions for ena_com layer.
 ena_regs_defs.h     Definition of ENA PCI memory-mapped (MMIO) registers.
 ena_netdev.[ch]     Main Linux kernel driver.
 ena_ethtool.c       ethtool callbacks.
+ena_devlink.[ch]    devlink files (see `devlink support`_ for more info)
 ena_pci_id_tbl.h    Supported device IDs.
 =================   ======================================================
 
@@ -253,6 +254,35 @@ RSS
 - The user can provide a hash key, hash function, and configure the
   indirection table through `ethtool(8)`.
 
+.. _`devlink support`:
+DEVLINK SUPPORT
+===============
+.. _`devlink`: https://www.kernel.org/doc/html/latest/networking/devlink/index.html
+
+`devlink`_ supports toggling LLQ entry size between the default 128 bytes and 256
+bytes.
+A 128 bytes entry size allows for a maximum of 96 bytes of packet header size
+which sometimes is not enough (e.g. when using tunneling).
+Increasing LLQ entry size to 256 bytes, allows a maximum header size of 224
+bytes. This comes with the penalty of reducing the number of LLQ entries in the
+TX queue by 2 (i.e. from 1024 to 512).
+
+The entry size can be toggled by enabling/disabling the large_llq_header devlink
+param and reloading the driver to make it take effect, e.g.
+
+.. code-block:: shell
+
+  sudo devlink dev param set pci/0000:00:06.0 name large_llq_header value true cmode driverinit
+  sudo devlink dev reload pci/0000:00:06.0
+
+One way to verify that the TX queue entry size has indeed increased is to check
+that the maximum TX queue depth is 512. This can be checked, for example, by
+using:
+
+.. code-block:: shell
+
+  ethtool -g [interface]
+
 DATA PATH
 =========
 
-- 
2.38.1

Re: [PATCH V1 net-next 5/5] net: ena: Add devlink documentation

[kernel test robot]

[-- Attachment #1: Type: text/plain, Size: 2267 bytes --]

Hi David,

I love your patch! Perhaps something to improve:

[auto build test WARNING on net-next/master]

url:    https://github.com/intel-lab-lkp/linux/commits/David-Arinzon/net-ena-Register-ena-device-to-devlink/20230108-183920
patch link:    https://lore.kernel.org/r/20230108103533.10104-6-darinzon%40amazon.com
patch subject: [PATCH V1 net-next 5/5] net: ena: Add devlink documentation
reproduce:
        # https://github.com/intel-lab-lkp/linux/commit/677aa62d9ff1bc108e291e910ad4e0d090530b7c
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review David-Arinzon/net-ena-Register-ena-device-to-devlink/20230108-183920
        git checkout 677aa62d9ff1bc108e291e910ad4e0d090530b7c
        make menuconfig
        # enable CONFIG_COMPILE_TEST, CONFIG_WARN_MISSING_DOCUMENTS, CONFIG_WARN_ABI_ERRORS
        make htmldocs

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

>> Documentation/networking/device_drivers/ethernet/amazon/ena.rst:258: WARNING: Explicit markup ends without a blank line; unexpected unindent.

vim +258 Documentation/networking/device_drivers/ethernet/amazon/ena.rst

   242	
   243	- The ENA device supports RSS that allows flexible Rx traffic
   244	  steering.
   245	- Toeplitz and CRC32 hash functions are supported.
   246	- Different combinations of L2/L3/L4 fields can be configured as
   247	  inputs for hash functions.
   248	- The driver configures RSS settings using the AQ SetFeature command
   249	  (ENA_ADMIN_RSS_HASH_FUNCTION, ENA_ADMIN_RSS_HASH_INPUT and
   250	  ENA_ADMIN_RSS_INDIRECTION_TABLE_CONFIG properties).
   251	- If the NETIF_F_RXHASH flag is set, the 32-bit result of the hash
   252	  function delivered in the Rx CQ descriptor is set in the received
   253	  SKB.
   254	- The user can provide a hash key, hash function, and configure the
   255	  indirection table through `ethtool(8)`.
   256	
   257	.. _`devlink support`:
 > 258	DEVLINK SUPPORT
   259	===============
   260	.. _`devlink`: https://www.kernel.org/doc/html/latest/networking/devlink/index.html
   261	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests

[-- Attachment #2: config --]
[-- Type: text/plain, Size: 39372 bytes --]

#
# Automatically generated file; DO NOT EDIT.
# Linux/x86_64 6.2.0-rc2 Kernel Configuration
#
CONFIG_CC_VERSION_TEXT="gcc-11 (Debian 11.3.0-8) 11.3.0"
CONFIG_CC_IS_GCC=y
CONFIG_GCC_VERSION=110300
CONFIG_CLANG_VERSION=0
CONFIG_AS_IS_GNU=y
CONFIG_AS_VERSION=23900
CONFIG_LD_IS_BFD=y
CONFIG_LD_VERSION=23900
CONFIG_LLD_VERSION=0
CONFIG_CC_CAN_LINK=y
CONFIG_CC_CAN_LINK_STATIC=y
CONFIG_CC_HAS_ASM_GOTO_OUTPUT=y
CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y
CONFIG_CC_HAS_ASM_INLINE=y
CONFIG_CC_HAS_NO_PROFILE_FN_ATTR=y
CONFIG_PAHOLE_VERSION=123
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_TABLE_SORT=y
CONFIG_THREAD_INFO_IN_TASK=y

#
# General setup
#
CONFIG_BROKEN_ON_SMP=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_COMPILE_TEST=y
# CONFIG_WERROR is not set
CONFIG_LOCALVERSION=""
CONFIG_BUILD_SALT=""
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y
CONFIG_HAVE_KERNEL_ZSTD=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
# CONFIG_KERNEL_LZ4 is not set
# CONFIG_KERNEL_ZSTD is not set
CONFIG_DEFAULT_INIT=""
CONFIG_DEFAULT_HOSTNAME="(none)"
# CONFIG_SYSVIPC is not set
# CONFIG_WATCH_QUEUE is not set
# CONFIG_CROSS_MEMORY_ATTACH is not set
# CONFIG_USELIB is not set
CONFIG_HAVE_ARCH_AUDITSYSCALL=y

#
# IRQ subsystem
#
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_HARDIRQS_SW_RESEND=y
CONFIG_IRQ_DOMAIN=y
CONFIG_IRQ_DOMAIN_HIERARCHY=y
CONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR=y
CONFIG_GENERIC_IRQ_RESERVATION_MODE=y
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
# end of IRQ subsystem

CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ARCH_CLOCKSOURCE_INIT=y
CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_HAVE_POSIX_CPU_TIMERS_TASK_WORK=y
CONFIG_POSIX_CPU_TIMERS_TASK_WORK=y

#
# Timers subsystem
#
CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ is not set
# CONFIG_HIGH_RES_TIMERS is not set
CONFIG_CLOCKSOURCE_WATCHDOG_MAX_SKEW_US=100
# end of Timers subsystem

CONFIG_HAVE_EBPF_JIT=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y

#
# BPF subsystem
#
# CONFIG_BPF_SYSCALL is not set
# end of BPF subsystem

CONFIG_PREEMPT_NONE_BUILD=y
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
# CONFIG_PREEMPT_DYNAMIC is not set

#
# CPU/Task time and stats accounting
#
CONFIG_TICK_CPU_ACCOUNTING=y
# CONFIG_VIRT_CPU_ACCOUNTING_GEN is not set
# CONFIG_IRQ_TIME_ACCOUNTING is not set
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_PSI is not set
# end of CPU/Task time and stats accounting

CONFIG_CPU_ISOLATION=y

#
# RCU Subsystem
#
CONFIG_TINY_RCU=y
# CONFIG_RCU_EXPERT is not set
CONFIG_SRCU=y
CONFIG_TINY_SRCU=y
# end of RCU Subsystem

# CONFIG_IKCONFIG is not set
# CONFIG_IKHEADERS is not set
CONFIG_LOG_BUF_SHIFT=17
CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=13
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y

#
# Scheduler features
#
# end of Scheduler features

CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH=y
CONFIG_CC_HAS_INT128=y
CONFIG_CC_IMPLICIT_FALLTHROUGH="-Wimplicit-fallthrough=5"
CONFIG_GCC12_NO_ARRAY_BOUNDS=y
CONFIG_ARCH_SUPPORTS_INT128=y
# CONFIG_CGROUPS is not set
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_TIME_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_CHECKPOINT_RESTORE is not set
# CONFIG_SCHED_AUTOGROUP is not set
# CONFIG_SYSFS_DEPRECATED is not set
# CONFIG_RELAY is not set
# CONFIG_BLK_DEV_INITRD is not set
# CONFIG_BOOT_CONFIG is not set
# CONFIG_INITRAMFS_PRESERVE_MTIME is not set
CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_LD_ORPHAN_WARN=y
CONFIG_LD_ORPHAN_WARN_LEVEL="warn"
CONFIG_SYSCTL=y
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
# CONFIG_EXPERT is not set
CONFIG_MULTIUSER=y
CONFIG_SGETMASK_SYSCALL=y
CONFIG_SYSFS_SYSCALL=y
CONFIG_FHANDLE=y
CONFIG_POSIX_TIMERS=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_FUTEX_PI=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_IO_URING=y
CONFIG_ADVISE_SYSCALLS=y
CONFIG_MEMBARRIER=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_SELFTEST is not set
CONFIG_KALLSYMS_BASE_RELATIVE=y
CONFIG_ARCH_HAS_MEMBARRIER_SYNC_CORE=y
CONFIG_RSEQ=y
# CONFIG_EMBEDDED is not set
CONFIG_HAVE_PERF_EVENTS=y

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
# end of Kernel Performance Events And Counters

# CONFIG_PROFILING is not set
# end of General setup

CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_MMU=y
CONFIG_ARCH_MMAP_RND_BITS_MIN=28
CONFIG_ARCH_MMAP_RND_BITS_MAX=32
CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=8
CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MAX=16
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_PGTABLE_LEVELS=4
CONFIG_CC_HAS_SANE_STACKPROTECTOR=y

#
# Processor type and features
#
# CONFIG_SMP is not set
CONFIG_X86_FEATURE_NAMES=y
CONFIG_X86_MPPARSE=y
# CONFIG_GOLDFISH is not set
# CONFIG_X86_CPU_RESCTRL is not set
# CONFIG_X86_EXTENDED_PLATFORM is not set
# CONFIG_SCHED_OMIT_FRAME_POINTER is not set
# CONFIG_HYPERVISOR_GUEST is not set
# CONFIG_MK8 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_GENERIC_CPU=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_IA32_FEAT_CTL=y
CONFIG_X86_VMX_FEATURE_NAMES=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_HYGON=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_CPU_SUP_ZHAOXIN=y
CONFIG_HPET_TIMER=y
CONFIG_DMI=y
CONFIG_NR_CPUS_RANGE_BEGIN=1
CONFIG_NR_CPUS_RANGE_END=1
CONFIG_NR_CPUS_DEFAULT=1
CONFIG_NR_CPUS=1
CONFIG_UP_LATE_INIT=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
# CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS is not set
# CONFIG_X86_MCE is not set

#
# Performance monitoring
#
# CONFIG_PERF_EVENTS_AMD_POWER is not set
# CONFIG_PERF_EVENTS_AMD_UNCORE is not set
# CONFIG_PERF_EVENTS_AMD_BRS is not set
# end of Performance monitoring

CONFIG_X86_16BIT=y
CONFIG_X86_ESPFIX64=y
CONFIG_X86_VSYSCALL_EMULATION=y
# CONFIG_X86_IOPL_IOPERM is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set
# CONFIG_X86_5LEVEL is not set
CONFIG_X86_DIRECT_GBPAGES=y
# CONFIG_AMD_MEM_ENCRYPT is not set
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
# CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
CONFIG_MTRR=y
# CONFIG_MTRR_SANITIZER is not set
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_X86_UMIP=y
CONFIG_CC_HAS_IBT=y
# CONFIG_X86_KERNEL_IBT is not set
# CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS is not set
CONFIG_X86_INTEL_TSX_MODE_OFF=y
# CONFIG_X86_INTEL_TSX_MODE_ON is not set
# CONFIG_X86_INTEL_TSX_MODE_AUTO is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x1000000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x200000
CONFIG_LEGACY_VSYSCALL_XONLY=y
# CONFIG_LEGACY_VSYSCALL_NONE is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_MODIFY_LDT_SYSCALL=y
# CONFIG_STRICT_SIGALTSTACK_SIZE is not set
CONFIG_HAVE_LIVEPATCH=y
# end of Processor type and features

CONFIG_CC_HAS_SLS=y
CONFIG_CC_HAS_RETURN_THUNK=y
CONFIG_CC_HAS_ENTRY_PADDING=y
CONFIG_FUNCTION_PADDING_CFI=11
CONFIG_FUNCTION_PADDING_BYTES=16
# CONFIG_SPECULATION_MITIGATIONS is not set
CONFIG_ARCH_HAS_ADD_PAGES=y
CONFIG_ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE=y

#
# Power management and ACPI options
#
# CONFIG_SUSPEND is not set
# CONFIG_PM is not set
CONFIG_ARCH_SUPPORTS_ACPI=y
# CONFIG_ACPI is not set

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set
# end of CPU Frequency scaling

#
# CPU Idle
#
# CONFIG_CPU_IDLE is not set
# end of CPU Idle
# end of Power management and ACPI options

#
# Bus options (PCI etc.)
#
CONFIG_ISA_DMA_API=y
# end of Bus options (PCI etc.)

#
# Binary Emulations
#
# CONFIG_IA32_EMULATION is not set
# CONFIG_X86_X32_ABI is not set
# end of Binary Emulations

CONFIG_HAVE_KVM=y
# CONFIG_VIRTUALIZATION is not set
CONFIG_AS_AVX512=y
CONFIG_AS_SHA1_NI=y
CONFIG_AS_SHA256_NI=y
CONFIG_AS_TPAUSE=y

#
# General architecture-dependent options
#
CONFIG_GENERIC_ENTRY=y
# CONFIG_JUMP_LABEL is not set
# CONFIG_STATIC_CALL_SELFTEST is not set
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_ARCH_CORRECT_STACKTRACE_ON_KRETPROBE=y
CONFIG_HAVE_FUNCTION_ERROR_INJECTION=y
CONFIG_HAVE_NMI=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_TRACE_IRQFLAGS_NMI_SUPPORT=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_CONTIGUOUS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_ARCH_HAS_FORTIFY_SOURCE=y
CONFIG_ARCH_HAS_SET_MEMORY=y
CONFIG_ARCH_HAS_SET_DIRECT_MAP=y
CONFIG_HAVE_ARCH_THREAD_STRUCT_WHITELIST=y
CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT=y
CONFIG_ARCH_WANTS_NO_INSTR=y
CONFIG_HAVE_ASM_MODVERSIONS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_RSEQ=y
CONFIG_HAVE_RUST=y
CONFIG_HAVE_FUNCTION_ARG_ACCESS_API=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_HARDLOCKUP_DETECTOR_PERF=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE=y
CONFIG_MMU_GATHER_MERGE_VMAS=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_ARCH_HAS_NMI_SAFE_THIS_CPU_OPS=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_HAVE_ARCH_SECCOMP=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
# CONFIG_SECCOMP is not set
CONFIG_HAVE_ARCH_STACKLEAK=y
CONFIG_HAVE_STACKPROTECTOR=y
# CONFIG_STACKPROTECTOR is not set
CONFIG_ARCH_SUPPORTS_LTO_CLANG=y
CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y
CONFIG_LTO_NONE=y
CONFIG_ARCH_SUPPORTS_CFI_CLANG=y
CONFIG_HAVE_ARCH_WITHIN_STACK_FRAMES=y
CONFIG_HAVE_CONTEXT_TRACKING_USER=y
CONFIG_HAVE_CONTEXT_TRACKING_USER_OFFSTACK=y
CONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=y
CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
CONFIG_HAVE_MOVE_PUD=y
CONFIG_HAVE_MOVE_PMD=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD=y
CONFIG_HAVE_ARCH_HUGE_VMAP=y
CONFIG_HAVE_ARCH_HUGE_VMALLOC=y
CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
CONFIG_HAVE_ARCH_SOFT_DIRTY=y
CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK=y
CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK=y
CONFIG_SOFTIRQ_ON_OWN_STACK=y
CONFIG_ARCH_HAS_ELF_RANDOMIZE=y
CONFIG_HAVE_ARCH_MMAP_RND_BITS=y
CONFIG_HAVE_EXIT_THREAD=y
CONFIG_ARCH_MMAP_RND_BITS=28
CONFIG_PAGE_SIZE_LESS_THAN_64KB=y
CONFIG_PAGE_SIZE_LESS_THAN_256KB=y
CONFIG_HAVE_OBJTOOL=y
CONFIG_HAVE_JUMP_LABEL_HACK=y
CONFIG_HAVE_NOINSTR_HACK=y
CONFIG_HAVE_NOINSTR_VALIDATION=y
CONFIG_HAVE_UACCESS_VALIDATION=y
CONFIG_HAVE_STACK_VALIDATION=y
CONFIG_HAVE_RELIABLE_STACKTRACE=y
# CONFIG_COMPAT_32BIT_TIME is not set
CONFIG_HAVE_ARCH_VMAP_STACK=y
# CONFIG_VMAP_STACK is not set
CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET=y
CONFIG_RANDOMIZE_KSTACK_OFFSET=y
# CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT is not set
CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=y
CONFIG_STRICT_KERNEL_RWX=y
CONFIG_ARCH_HAS_STRICT_MODULE_RWX=y
CONFIG_HAVE_ARCH_PREL32_RELOCATIONS=y
CONFIG_ARCH_HAS_MEM_ENCRYPT=y
CONFIG_HAVE_STATIC_CALL=y
CONFIG_HAVE_STATIC_CALL_INLINE=y
CONFIG_HAVE_PREEMPT_DYNAMIC=y
CONFIG_HAVE_PREEMPT_DYNAMIC_CALL=y
CONFIG_ARCH_WANT_LD_ORPHAN_WARN=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_ARCH_SUPPORTS_PAGE_TABLE_CHECK=y
CONFIG_ARCH_HAS_ELFCORE_COMPAT=y
CONFIG_ARCH_HAS_PARANOID_L1D_FLUSH=y
CONFIG_DYNAMIC_SIGFRAME=y
CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG=y

#
# GCOV-based kernel profiling
#
CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
# end of GCOV-based kernel profiling

CONFIG_HAVE_GCC_PLUGINS=y
# CONFIG_GCC_PLUGINS is not set
CONFIG_FUNCTION_ALIGNMENT_4B=y
CONFIG_FUNCTION_ALIGNMENT_16B=y
CONFIG_FUNCTION_ALIGNMENT=16
# end of General architecture-dependent options

CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
# CONFIG_MODULES is not set
CONFIG_BLOCK=y
# CONFIG_BLOCK_LEGACY_AUTOLOAD is not set
# CONFIG_BLK_DEV_BSGLIB is not set
# CONFIG_BLK_DEV_INTEGRITY is not set
# CONFIG_BLK_DEV_ZONED is not set
# CONFIG_BLK_WBT is not set
# CONFIG_BLK_SED_OPAL is not set
# CONFIG_BLK_INLINE_ENCRYPTION is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_EFI_PARTITION=y
# end of Partition Types

#
# IO Schedulers
#
# CONFIG_MQ_IOSCHED_DEADLINE is not set
# CONFIG_MQ_IOSCHED_KYBER is not set
# CONFIG_IOSCHED_BFQ is not set
# end of IO Schedulers

CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_INLINE_READ_UNLOCK=y
CONFIG_INLINE_READ_UNLOCK_IRQ=y
CONFIG_INLINE_WRITE_UNLOCK=y
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
CONFIG_ARCH_USE_QUEUED_SPINLOCKS=y
CONFIG_ARCH_USE_QUEUED_RWLOCKS=y
CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE=y
CONFIG_ARCH_HAS_SYNC_CORE_BEFORE_USERMODE=y
CONFIG_ARCH_HAS_SYSCALL_WRAPPER=y

#
# Executable file formats
#
# CONFIG_BINFMT_ELF is not set
# CONFIG_BINFMT_SCRIPT is not set
# CONFIG_BINFMT_MISC is not set
CONFIG_COREDUMP=y
# end of Executable file formats

#
# Memory Management options
#
# CONFIG_SWAP is not set

#
# SLAB allocator options
#
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLAB_MERGE_DEFAULT is not set
# CONFIG_SLAB_FREELIST_RANDOM is not set
# CONFIG_SLAB_FREELIST_HARDENED is not set
# CONFIG_SLUB_STATS is not set
# end of SLAB allocator options

# CONFIG_SHUFFLE_PAGE_ALLOCATOR is not set
# CONFIG_COMPAT_BRK is not set
CONFIG_SPARSEMEM=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
# CONFIG_SPARSEMEM_VMEMMAP is not set
CONFIG_HAVE_FAST_GUP=y
CONFIG_EXCLUSIVE_SYSTEM_RAM=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
# CONFIG_MEMORY_HOTPLUG is not set
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK=y
# CONFIG_COMPACTION is not set
# CONFIG_PAGE_REPORTING is not set
CONFIG_PHYS_ADDR_T_64BIT=y
# CONFIG_KSM is not set
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
CONFIG_ARCH_WANTS_THP_SWAP=y
# CONFIG_TRANSPARENT_HUGEPAGE is not set
CONFIG_NEED_PER_CPU_KM=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
# CONFIG_CMA is not set
CONFIG_GENERIC_EARLY_IOREMAP=y
# CONFIG_IDLE_PAGE_TRACKING is not set
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_ARCH_HAS_CURRENT_STACK_POINTER=y
CONFIG_ARCH_HAS_PTE_DEVMAP=y
CONFIG_ZONE_DMA=y
CONFIG_ZONE_DMA32=y
CONFIG_VM_EVENT_COUNTERS=y
# CONFIG_PERCPU_STATS is not set

#
# GUP_TEST needs to have DEBUG_FS enabled
#
CONFIG_ARCH_HAS_PTE_SPECIAL=y
CONFIG_SECRETMEM=y
# CONFIG_ANON_VMA_NAME is not set
# CONFIG_USERFAULTFD is not set
# CONFIG_LRU_GEN is not set

#
# Data Access Monitoring
#
# CONFIG_DAMON is not set
# end of Data Access Monitoring
# end of Memory Management options

# CONFIG_NET is not set

#
# Device Drivers
#
CONFIG_HAVE_EISA=y
# CONFIG_EISA is not set
CONFIG_HAVE_PCI=y
# CONFIG_PCI is not set
# CONFIG_PCCARD is not set

#
# Generic Driver Options
#
# CONFIG_UEVENT_HELPER is not set
# CONFIG_DEVTMPFS is not set
# CONFIG_STANDALONE is not set
# CONFIG_PREVENT_FIRMWARE_BUILD is not set

#
# Firmware loader
#
CONFIG_FW_LOADER=y
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_FW_LOADER_USER_HELPER is not set
# CONFIG_FW_LOADER_COMPRESS is not set
# CONFIG_FW_UPLOAD is not set
# end of Firmware loader

CONFIG_ALLOW_DEV_COREDUMP=y
CONFIG_GENERIC_CPU_AUTOPROBE=y
CONFIG_GENERIC_CPU_VULNERABILITIES=y
# end of Generic Driver Options

#
# Bus devices
#
# CONFIG_ARM_INTEGRATOR_LM is not set
# CONFIG_BT1_APB is not set
# CONFIG_BT1_AXI is not set
# CONFIG_HISILICON_LPC is not set
# CONFIG_INTEL_IXP4XX_EB is not set
# CONFIG_QCOM_EBI2 is not set
# CONFIG_MHI_BUS is not set
# CONFIG_MHI_BUS_EP is not set
# end of Bus devices

#
# Firmware Drivers
#

#
# ARM System Control and Management Interface Protocol
#
# CONFIG_ARM_SCMI_PROTOCOL is not set
# end of ARM System Control and Management Interface Protocol

# CONFIG_EDD is not set
CONFIG_FIRMWARE_MEMMAP=y
# CONFIG_DMIID is not set
# CONFIG_DMI_SYSFS is not set
CONFIG_DMI_SCAN_MACHINE_NON_EFI_FALLBACK=y
# CONFIG_FW_CFG_SYSFS is not set
# CONFIG_SYSFB_SIMPLEFB is not set
# CONFIG_BCM47XX_NVRAM is not set
# CONFIG_GOOGLE_FIRMWARE is not set

#
# Tegra firmware driver
#
# end of Tegra firmware driver
# end of Firmware Drivers

# CONFIG_GNSS is not set
# CONFIG_MTD is not set
# CONFIG_OF is not set
CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
# CONFIG_PARPORT is not set
# CONFIG_BLK_DEV is not set

#
# NVME Support
#
# CONFIG_NVME_FC is not set
# end of NVME Support

#
# Misc devices
#
# CONFIG_DUMMY_IRQ is not set
# CONFIG_ATMEL_SSC is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_SMPRO_ERRMON is not set
# CONFIG_SMPRO_MISC is not set
# CONFIG_QCOM_COINCELL is not set
# CONFIG_SRAM is not set
# CONFIG_XILINX_SDFEC is not set
# CONFIG_C2PORT is not set

#
# EEPROM support
#
# CONFIG_EEPROM_93CX6 is not set
# end of EEPROM support

#
# Texas Instruments shared transport line discipline
#
# end of Texas Instruments shared transport line discipline

#
# Altera FPGA firmware download module (requires I2C)
#
# CONFIG_ECHO is not set
# CONFIG_PVPANIC is not set
# end of Misc devices

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
# CONFIG_RAID_ATTRS is not set
# CONFIG_SCSI is not set
# end of SCSI device support

# CONFIG_ATA is not set
# CONFIG_MD is not set
# CONFIG_TARGET_CORE is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_FIREWIRE is not set
# end of IEEE 1394 (FireWire) support

# CONFIG_MACINTOSH_DRIVERS is not set

#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_SPARSEKMAP is not set
# CONFIG_INPUT_MATRIXKMAP is not set

#
# Userland interfaces
#
# CONFIG_INPUT_MOUSEDEV is not set
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
# CONFIG_INPUT_KEYBOARD is not set
# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set
# CONFIG_RMI4_CORE is not set

#
# Hardware I/O ports
#
# CONFIG_SERIO is not set
CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y
# CONFIG_GAMEPORT is not set
# end of Hardware I/O ports
# end of Input device support

#
# Character devices
#
CONFIG_TTY=y
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
# CONFIG_LEGACY_TIOCSTI is not set
# CONFIG_LDISC_AUTOLOAD is not set

#
# Serial drivers
#
# CONFIG_SERIAL_8250 is not set

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_AMBA_PL010 is not set
# CONFIG_SERIAL_MESON is not set
# CONFIG_SERIAL_CLPS711X is not set
# CONFIG_SERIAL_SAMSUNG is not set
# CONFIG_SERIAL_TEGRA is not set
# CONFIG_SERIAL_IMX is not set
# CONFIG_SERIAL_UARTLITE is not set
# CONFIG_SERIAL_SH_SCI is not set
# CONFIG_SERIAL_MSM is not set
# CONFIG_SERIAL_VT8500 is not set
# CONFIG_SERIAL_OMAP is not set
# CONFIG_SERIAL_LANTIQ is not set
# CONFIG_SERIAL_SCCNXP is not set
# CONFIG_SERIAL_TIMBERDALE is not set
# CONFIG_SERIAL_BCM63XX is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_MXS_AUART is not set
# CONFIG_SERIAL_MPS2_UART is not set
# CONFIG_SERIAL_ARC is not set
# CONFIG_SERIAL_FSL_LPUART is not set
# CONFIG_SERIAL_FSL_LINFLEXUART is not set
# CONFIG_SERIAL_ST_ASC is not set
# CONFIG_SERIAL_STM32 is not set
# CONFIG_SERIAL_OWL is not set
# CONFIG_SERIAL_RDA is not set
# CONFIG_SERIAL_LITEUART is not set
# CONFIG_SERIAL_SUNPLUS is not set
# end of Serial drivers

# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NULL_TTY is not set
# CONFIG_SERIAL_DEV_BUS is not set
# CONFIG_VIRTIO_CONSOLE is not set
# CONFIG_IPMI_HANDLER is not set
# CONFIG_ASPEED_KCS_IPMI_BMC is not set
# CONFIG_NPCM7XX_KCS_IPMI_BMC is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_MWAVE is not set
# CONFIG_DEVMEM is not set
# CONFIG_NVRAM is not set
# CONFIG_HANGCHECK_TIMER is not set
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
# end of Character devices

#
# I2C support
#
# CONFIG_I2C is not set
# end of I2C support

# CONFIG_I3C is not set
# CONFIG_SPI is not set
# CONFIG_SPMI is not set
# CONFIG_HSI is not set
# CONFIG_PPS is not set

#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK_OPTIONAL=y

#
# Enable PHYLIB and NETWORK_PHY_TIMESTAMPING to see the additional clocks.
#
# end of PTP clock support

# CONFIG_PINCTRL is not set
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
# CONFIG_POWER_RESET is not set
# CONFIG_POWER_SUPPLY is not set
# CONFIG_HWMON is not set
# CONFIG_THERMAL is not set
# CONFIG_WATCHDOG is not set
CONFIG_SSB_POSSIBLE=y
# CONFIG_SSB is not set
CONFIG_BCMA_POSSIBLE=y
# CONFIG_BCMA is not set

#
# Multifunction device drivers
#
# CONFIG_MFD_SUN4I_GPADC is not set
# CONFIG_MFD_AT91_USART is not set
# CONFIG_MFD_MADERA is not set
# CONFIG_MFD_EXYNOS_LPASS is not set
# CONFIG_MFD_MXS_LRADC is not set
# CONFIG_MFD_MX25_TSADC is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_MFD_KEMPLD is not set
# CONFIG_MFD_MT6397 is not set
# CONFIG_MFD_PM8XXX is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_ABX500_CORE is not set
# CONFIG_MFD_SUN6I_PRCM is not set
# CONFIG_MFD_SYSCON is not set
# CONFIG_MFD_TI_AM335X_TSCADC is not set
# CONFIG_MFD_TQMX86 is not set
# CONFIG_MFD_STM32_LPTIMER is not set
# CONFIG_MFD_STM32_TIMERS is not set
# end of Multifunction device drivers

# CONFIG_REGULATOR is not set
# CONFIG_RC_CORE is not set

#
# CEC support
#
# CONFIG_MEDIA_CEC_SUPPORT is not set
# end of CEC support

# CONFIG_MEDIA_SUPPORT is not set

#
# Graphics support
#
# CONFIG_IMX_IPUV3_CORE is not set
# CONFIG_DRM is not set

#
# ARM devices
#
# end of ARM devices

#
# Frame buffer Devices
#
# CONFIG_FB is not set
# CONFIG_MMP_DISP is not set
# end of Frame buffer Devices

#
# Backlight & LCD device support
#
# CONFIG_LCD_CLASS_DEVICE is not set
# CONFIG_BACKLIGHT_CLASS_DEVICE is not set
# end of Backlight & LCD device support

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_DUMMY_CONSOLE_COLUMNS=80
CONFIG_DUMMY_CONSOLE_ROWS=25
# end of Console display driver support
# end of Graphics support

# CONFIG_SOUND is not set

#
# HID support
#
# CONFIG_HID is not set
# end of HID support

CONFIG_USB_OHCI_LITTLE_ENDIAN=y
# CONFIG_USB_SUPPORT is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
CONFIG_EDAC_ATOMIC_SCRUB=y
CONFIG_EDAC_SUPPORT=y
CONFIG_RTC_LIB=y
CONFIG_RTC_MC146818_LIB=y
# CONFIG_RTC_CLASS is not set
# CONFIG_DMADEVICES is not set

#
# DMABUF options
#
# CONFIG_SYNC_FILE is not set
# CONFIG_DMABUF_HEAPS is not set
# end of DMABUF options

# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
# CONFIG_VFIO is not set
# CONFIG_VIRT_DRIVERS is not set
# CONFIG_VIRTIO_MENU is not set
# CONFIG_VHOST_MENU is not set

#
# Microsoft Hyper-V guest support
#
# end of Microsoft Hyper-V guest support

# CONFIG_GREYBUS is not set
# CONFIG_COMEDI is not set
# CONFIG_STAGING is not set
# CONFIG_CHROME_PLATFORMS is not set
# CONFIG_MELLANOX_PLATFORM is not set
# CONFIG_OLPC_XO175 is not set
# CONFIG_SURFACE_PLATFORMS is not set
# CONFIG_X86_PLATFORM_DEVICES is not set
# CONFIG_COMMON_CLK is not set
# CONFIG_HWSPINLOCK is not set

#
# Clock Source drivers
#
CONFIG_CLKEVT_I8253=y
CONFIG_I8253_LOCK=y
CONFIG_CLKBLD_I8253=y
# CONFIG_BCM2835_TIMER is not set
# CONFIG_BCM_KONA_TIMER is not set
# CONFIG_DAVINCI_TIMER is not set
# CONFIG_DIGICOLOR_TIMER is not set
# CONFIG_OMAP_DM_TIMER is not set
# CONFIG_DW_APB_TIMER is not set
# CONFIG_FTTMR010_TIMER is not set
# CONFIG_IXP4XX_TIMER is not set
# CONFIG_MESON6_TIMER is not set
# CONFIG_OWL_TIMER is not set
# CONFIG_RDA_TIMER is not set
# CONFIG_SUN4I_TIMER is not set
# CONFIG_TEGRA_TIMER is not set
# CONFIG_VT8500_TIMER is not set
# CONFIG_NPCM7XX_TIMER is not set
# CONFIG_ASM9260_TIMER is not set
# CONFIG_CLKSRC_DBX500_PRCMU is not set
# CONFIG_CLPS711X_TIMER is not set
# CONFIG_MXS_TIMER is not set
# CONFIG_NSPIRE_TIMER is not set
# CONFIG_INTEGRATOR_AP_TIMER is not set
# CONFIG_CLKSRC_PISTACHIO is not set
# CONFIG_CLKSRC_STM32_LP is not set
# CONFIG_ARMV7M_SYSTICK is not set
# CONFIG_ATMEL_PIT is not set
# CONFIG_ATMEL_ST is not set
# CONFIG_CLKSRC_SAMSUNG_PWM is not set
# CONFIG_FSL_FTM_TIMER is not set
# CONFIG_OXNAS_RPS_TIMER is not set
# CONFIG_MTK_TIMER is not set
# CONFIG_SH_TIMER_CMT is not set
# CONFIG_SH_TIMER_MTU2 is not set
# CONFIG_RENESAS_OSTM is not set
# CONFIG_SH_TIMER_TMU is not set
# CONFIG_EM_TIMER_STI is not set
# CONFIG_CLKSRC_PXA is not set
# CONFIG_TIMER_IMX_SYS_CTR is not set
# CONFIG_CLKSRC_ST_LPC is not set
# CONFIG_GXP_TIMER is not set
# CONFIG_MSC313E_TIMER is not set
# CONFIG_MICROCHIP_PIT64B is not set
# end of Clock Source drivers

# CONFIG_MAILBOX is not set
# CONFIG_IOMMU_SUPPORT is not set

#
# Remoteproc drivers
#
# CONFIG_REMOTEPROC is not set
# end of Remoteproc drivers

#
# Rpmsg drivers
#
# CONFIG_RPMSG_VIRTIO is not set
# end of Rpmsg drivers

#
# SOC (System On Chip) specific Drivers
#

#
# Amlogic SoC drivers
#
# CONFIG_MESON_CANVAS is not set
# CONFIG_MESON_CLK_MEASURE is not set
# CONFIG_MESON_GX_SOCINFO is not set
# CONFIG_MESON_MX_SOCINFO is not set
# end of Amlogic SoC drivers

#
# Apple SoC drivers
#
# CONFIG_APPLE_SART is not set
# end of Apple SoC drivers

#
# ASPEED SoC drivers
#
# CONFIG_ASPEED_LPC_CTRL is not set
# CONFIG_ASPEED_LPC_SNOOP is not set
# CONFIG_ASPEED_UART_ROUTING is not set
# CONFIG_ASPEED_P2A_CTRL is not set
# CONFIG_ASPEED_SOCINFO is not set
# end of ASPEED SoC drivers

# CONFIG_AT91_SOC_ID is not set
# CONFIG_AT91_SOC_SFR is not set

#
# Broadcom SoC drivers
#
# CONFIG_SOC_BCM63XX is not set
# CONFIG_SOC_BRCMSTB is not set
# end of Broadcom SoC drivers

#
# NXP/Freescale QorIQ SoC drivers
#
# end of NXP/Freescale QorIQ SoC drivers

#
# fujitsu SoC drivers
#
# end of fujitsu SoC drivers

#
# i.MX SoC drivers
#
# CONFIG_SOC_IMX8M is not set
# CONFIG_SOC_IMX9 is not set
# end of i.MX SoC drivers

#
# IXP4xx SoC drivers
#
# CONFIG_IXP4XX_QMGR is not set
# CONFIG_IXP4XX_NPE is not set
# end of IXP4xx SoC drivers

#
# Enable LiteX SoC Builder specific drivers
#
# CONFIG_LITEX_SOC_CONTROLLER is not set
# end of Enable LiteX SoC Builder specific drivers

# CONFIG_LOONGSON2_GUTS is not set

#
# MediaTek SoC drivers
#
# CONFIG_MTK_CMDQ is not set
# CONFIG_MTK_DEVAPC is not set
# CONFIG_MTK_INFRACFG is not set
# CONFIG_MTK_MMSYS is not set
# end of MediaTek SoC drivers

#
# Qualcomm SoC drivers
#
# CONFIG_QCOM_GENI_SE is not set
# CONFIG_QCOM_GSBI is not set
# CONFIG_QCOM_LLCC is not set
# CONFIG_QCOM_RPMH is not set
# CONFIG_QCOM_SPM is not set
# CONFIG_QCOM_ICC_BWMON is not set
# end of Qualcomm SoC drivers

# CONFIG_SOC_RENESAS is not set
# CONFIG_ROCKCHIP_GRF is not set
# CONFIG_SOC_SAMSUNG is not set
# CONFIG_SOC_TI is not set
# CONFIG_UX500_SOC_ID is not set

#
# Xilinx SoC drivers
#
# end of Xilinx SoC drivers
# end of SOC (System On Chip) specific Drivers

# CONFIG_PM_DEVFREQ is not set
# CONFIG_EXTCON is not set
# CONFIG_MEMORY is not set
# CONFIG_IIO is not set
# CONFIG_PWM is not set

#
# IRQ chip support
#
# CONFIG_RENESAS_INTC_IRQPIN is not set
# CONFIG_RENESAS_IRQC is not set
# CONFIG_RENESAS_RZA1_IRQC is not set
# CONFIG_RENESAS_RZG2L_IRQC is not set
# CONFIG_SL28CPLD_INTC is not set
# CONFIG_TS4800_IRQ is not set
# CONFIG_INGENIC_TCU_IRQ is not set
# CONFIG_IRQ_UNIPHIER_AIDET is not set
# CONFIG_MESON_IRQ_GPIO is not set
# CONFIG_IMX_IRQSTEER is not set
# CONFIG_IMX_INTMUX is not set
# CONFIG_EXYNOS_IRQ_COMBINER is not set
# CONFIG_MST_IRQ is not set
# CONFIG_MCHP_EIC is not set
# CONFIG_SUNPLUS_SP7021_INTC is not set
# end of IRQ chip support

# CONFIG_IPACK_BUS is not set
# CONFIG_RESET_CONTROLLER is not set

#
# PHY Subsystem
#
# CONFIG_GENERIC_PHY is not set
# CONFIG_PHY_PISTACHIO_USB is not set
# CONFIG_PHY_CAN_TRANSCEIVER is not set

#
# PHY drivers for Broadcom platforms
#
# CONFIG_PHY_BCM63XX_USBH is not set
# CONFIG_BCM_KONA_USB2_PHY is not set
# end of PHY drivers for Broadcom platforms

# CONFIG_PHY_HI6220_USB is not set
# CONFIG_PHY_HI3660_USB is not set
# CONFIG_PHY_HI3670_USB is not set
# CONFIG_PHY_HI3670_PCIE is not set
# CONFIG_PHY_HISTB_COMBPHY is not set
# CONFIG_PHY_HISI_INNO_USB2 is not set
# CONFIG_PHY_PXA_28NM_HSIC is not set
# CONFIG_PHY_PXA_28NM_USB2 is not set
# CONFIG_PHY_PXA_USB is not set
# CONFIG_PHY_MMP3_USB is not set
# CONFIG_PHY_MMP3_HSIC is not set
# CONFIG_PHY_MT7621_PCI is not set
# CONFIG_PHY_RALINK_USB is not set
# CONFIG_PHY_R8A779F0_ETHERNET_SERDES is not set
# CONFIG_PHY_RCAR_GEN3_USB3 is not set
# CONFIG_PHY_ROCKCHIP_DPHY_RX0 is not set
# CONFIG_PHY_ROCKCHIP_PCIE is not set
# CONFIG_PHY_ROCKCHIP_SNPS_PCIE3 is not set
# CONFIG_PHY_EXYNOS_MIPI_VIDEO is not set
# CONFIG_PHY_SAMSUNG_USB2 is not set
# CONFIG_PHY_ST_SPEAR1310_MIPHY is not set
# CONFIG_PHY_ST_SPEAR1340_MIPHY is not set
# CONFIG_PHY_TEGRA194_P2U is not set
# CONFIG_PHY_DA8XX_USB is not set
# CONFIG_OMAP_CONTROL_PHY is not set
# CONFIG_TI_PIPE3 is not set
# CONFIG_PHY_INTEL_KEEMBAY_EMMC is not set
# CONFIG_PHY_INTEL_KEEMBAY_USB is not set
# CONFIG_PHY_INTEL_LGM_EMMC is not set
# CONFIG_PHY_XILINX_ZYNQMP is not set
# end of PHY Subsystem

# CONFIG_POWERCAP is not set
# CONFIG_MCB is not set

#
# Performance monitor support
#
# CONFIG_ARM_CCN is not set
# CONFIG_ARM_CMN is not set
# CONFIG_FSL_IMX8_DDR_PMU is not set
# CONFIG_XGENE_PMU is not set
# CONFIG_ARM_DMC620_PMU is not set
# CONFIG_MARVELL_CN10K_TAD_PMU is not set
# CONFIG_ALIBABA_UNCORE_DRW_PMU is not set
# CONFIG_MARVELL_CN10K_DDR_PMU is not set
# CONFIG_MESON_DDR_PMU is not set
# end of Performance monitor support

# CONFIG_RAS is not set

#
# Android
#
# CONFIG_ANDROID_BINDER_IPC is not set
# end of Android

# CONFIG_DAX is not set
# CONFIG_NVMEM is not set

#
# HW tracing support
#
# CONFIG_STM is not set
# CONFIG_INTEL_TH is not set
# end of HW tracing support

# CONFIG_FPGA is not set
# CONFIG_TEE is not set
# CONFIG_SIOX is not set
# CONFIG_SLIMBUS is not set
# CONFIG_INTERCONNECT is not set
# CONFIG_COUNTER is not set
# CONFIG_PECI is not set
# CONFIG_HTE is not set
# end of Device Drivers

#
# File systems
#
CONFIG_DCACHE_WORD_ACCESS=y
# CONFIG_VALIDATE_FS_PARSER is not set
# CONFIG_EXT2_FS is not set
# CONFIG_EXT3_FS is not set
# CONFIG_EXT4_FS is not set
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_XFS_FS is not set
# CONFIG_GFS2_FS is not set
# CONFIG_BTRFS_FS is not set
# CONFIG_NILFS2_FS is not set
# CONFIG_F2FS_FS is not set
CONFIG_EXPORTFS=y
# CONFIG_EXPORTFS_BLOCK_OPS is not set
CONFIG_FILE_LOCKING=y
# CONFIG_FS_ENCRYPTION is not set
# CONFIG_FS_VERITY is not set
# CONFIG_DNOTIFY is not set
# CONFIG_INOTIFY_USER is not set
# CONFIG_FANOTIFY is not set
# CONFIG_QUOTA is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_AUTOFS_FS is not set
# CONFIG_FUSE_FS is not set
# CONFIG_OVERLAY_FS is not set

#
# Caches
#
# CONFIG_FSCACHE is not set
# end of Caches

#
# CD-ROM/DVD Filesystems
#
# CONFIG_ISO9660_FS is not set
# CONFIG_UDF_FS is not set
# end of CD-ROM/DVD Filesystems

#
# DOS/FAT/EXFAT/NT Filesystems
#
# CONFIG_MSDOS_FS is not set
# CONFIG_VFAT_FS is not set
# CONFIG_EXFAT_FS is not set
# CONFIG_NTFS_FS is not set
# CONFIG_NTFS3_FS is not set
# end of DOS/FAT/EXFAT/NT Filesystems

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
# CONFIG_PROC_KCORE is not set
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
# CONFIG_PROC_CHILDREN is not set
CONFIG_PROC_PID_ARCH_STATUS=y
CONFIG_KERNFS=y
CONFIG_SYSFS=y
# CONFIG_TMPFS is not set
# CONFIG_HUGETLBFS is not set
CONFIG_ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP=y
CONFIG_ARCH_HAS_GIGANTIC_PAGE=y
# CONFIG_CONFIGFS_FS is not set
# end of Pseudo filesystems

# CONFIG_MISC_FILESYSTEMS is not set
# CONFIG_NLS is not set
# CONFIG_UNICODE is not set
CONFIG_IO_WQ=y
# end of File systems

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITYFS is not set
CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR=y
# CONFIG_HARDENED_USERCOPY is not set
# CONFIG_FORTIFY_SOURCE is not set
# CONFIG_STATIC_USERMODEHELPER is not set
CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_LSM="landlock,lockdown,yama,loadpin,safesetid,integrity,bpf"

#
# Kernel hardening options
#

#
# Memory initialization
#
CONFIG_INIT_STACK_NONE=y
# CONFIG_INIT_ON_ALLOC_DEFAULT_ON is not set
# CONFIG_INIT_ON_FREE_DEFAULT_ON is not set
CONFIG_CC_HAS_ZERO_CALL_USED_REGS=y
# CONFIG_ZERO_CALL_USED_REGS is not set
# end of Memory initialization

CONFIG_RANDSTRUCT_NONE=y
# end of Kernel hardening options
# end of Security options

# CONFIG_CRYPTO is not set

#
# Library routines
#
# CONFIG_PACKING is not set
CONFIG_BITREVERSE=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
# CONFIG_CORDIC is not set
# CONFIG_PRIME_NUMBERS is not set
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_ARCH_USE_CMPXCHG_LOCKREF=y
CONFIG_ARCH_HAS_FAST_MULTIPLIER=y
CONFIG_ARCH_USE_SYM_ANNOTATIONS=y

#
# Crypto library routines
#
CONFIG_CRYPTO_LIB_BLAKE2S_GENERIC=y
# CONFIG_CRYPTO_LIB_CHACHA is not set
# CONFIG_CRYPTO_LIB_CURVE25519 is not set
CONFIG_CRYPTO_LIB_POLY1305_RSIZE=11
# CONFIG_CRYPTO_LIB_POLY1305 is not set
# end of Crypto library routines

# CONFIG_CRC_CCITT is not set
# CONFIG_CRC16 is not set
# CONFIG_CRC_T10DIF is not set
# CONFIG_CRC64_ROCKSOFT is not set
# CONFIG_CRC_ITU_T is not set
CONFIG_CRC32=y
# CONFIG_CRC32_SELFTEST is not set
CONFIG_CRC32_SLICEBY8=y
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
# CONFIG_CRC32_BIT is not set
# CONFIG_CRC64 is not set
# CONFIG_CRC4 is not set
# CONFIG_CRC7 is not set
# CONFIG_LIBCRC32C is not set
# CONFIG_CRC8 is not set
# CONFIG_RANDOM32_SELFTEST is not set
# CONFIG_XZ_DEC is not set
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT_MAP=y
CONFIG_HAS_DMA=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_SWIOTLB=y
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_IRQ_POLL is not set
CONFIG_HAVE_GENERIC_VDSO=y
CONFIG_GENERIC_GETTIMEOFDAY=y
CONFIG_GENERIC_VDSO_TIME_NS=y
CONFIG_ARCH_HAS_PMEM_API=y
CONFIG_ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION=y
CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE=y
CONFIG_ARCH_HAS_COPY_MC=y
CONFIG_ARCH_STACKWALK=y
CONFIG_STACKDEPOT=y
CONFIG_SBITMAP=y
# CONFIG_PARMAN is not set
# CONFIG_OBJAGG is not set
# end of Library routines

#
# Kernel hacking
#

#
# printk and dmesg options
#
# CONFIG_PRINTK_TIME is not set
# CONFIG_PRINTK_CALLER is not set
# CONFIG_STACKTRACE_BUILD_ID is not set
CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7
CONFIG_CONSOLE_LOGLEVEL_QUIET=4
CONFIG_MESSAGE_LOGLEVEL_DEFAULT=4
# CONFIG_DYNAMIC_DEBUG is not set
# CONFIG_DYNAMIC_DEBUG_CORE is not set
# CONFIG_SYMBOLIC_ERRNAME is not set
CONFIG_DEBUG_BUGVERBOSE=y
# end of printk and dmesg options

# CONFIG_DEBUG_KERNEL is not set

#
# Compile-time checks and compiler options
#
CONFIG_AS_HAS_NON_CONST_LEB128=y
CONFIG_FRAME_WARN=2048
# CONFIG_STRIP_ASM_SYMS is not set
# CONFIG_HEADERS_INSTALL is not set
CONFIG_DEBUG_SECTION_MISMATCH=y
CONFIG_SECTION_MISMATCH_WARN_ONLY=y
CONFIG_OBJTOOL=y
# end of Compile-time checks and compiler options

#
# Generic Kernel Debugging Instruments
#
# CONFIG_MAGIC_SYSRQ is not set
# CONFIG_DEBUG_FS is not set
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_ARCH_HAS_UBSAN_SANITIZE_ALL=y
# CONFIG_UBSAN is not set
CONFIG_HAVE_ARCH_KCSAN=y
CONFIG_HAVE_KCSAN_COMPILER=y
# end of Generic Kernel Debugging Instruments

#
# Networking Debugging
#
# end of Networking Debugging

#
# Memory Debugging
#
# CONFIG_PAGE_EXTENSION is not set
CONFIG_SLUB_DEBUG=y
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_PAGE_TABLE_CHECK is not set
# CONFIG_PAGE_POISONING is not set
# CONFIG_DEBUG_RODATA_TEST is not set
CONFIG_ARCH_HAS_DEBUG_WX=y
# CONFIG_DEBUG_WX is not set
CONFIG_GENERIC_PTDUMP=y
CONFIG_HAVE_DEBUG_KMEMLEAK=y
CONFIG_ARCH_HAS_DEBUG_VM_PGTABLE=y
# CONFIG_DEBUG_VM_PGTABLE is not set
CONFIG_ARCH_HAS_DEBUG_VIRTUAL=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP=y
CONFIG_HAVE_ARCH_KASAN=y
CONFIG_HAVE_ARCH_KASAN_VMALLOC=y
CONFIG_CC_HAS_KASAN_GENERIC=y
CONFIG_CC_HAS_WORKING_NOSANITIZE_ADDRESS=y
# CONFIG_KASAN is not set
CONFIG_HAVE_ARCH_KFENCE=y
# CONFIG_KFENCE is not set
CONFIG_HAVE_ARCH_KMSAN=y
# end of Memory Debugging

#
# Debug Oops, Lockups and Hangs
#
# CONFIG_PANIC_ON_OOPS is not set
CONFIG_PANIC_ON_OOPS_VALUE=0
CONFIG_PANIC_TIMEOUT=0
CONFIG_HARDLOCKUP_CHECK_TIMESTAMP=y
# end of Debug Oops, Lockups and Hangs

#
# Scheduler Debugging
#
# end of Scheduler Debugging

# CONFIG_DEBUG_TIMEKEEPING is not set

#
# Lock Debugging (spinlocks, mutexes, etc...)
#
CONFIG_LOCK_DEBUGGING_SUPPORT=y
# CONFIG_WW_MUTEX_SELFTEST is not set
# end of Lock Debugging (spinlocks, mutexes, etc...)

# CONFIG_DEBUG_IRQFLAGS is not set
CONFIG_STACKTRACE=y
# CONFIG_WARN_ALL_UNSEEDED_RANDOM is not set

#
# Debug kernel data structures
#
# CONFIG_BUG_ON_DATA_CORRUPTION is not set
# end of Debug kernel data structures

#
# RCU Debugging
#
# end of RCU Debugging

CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_HAVE_RETHOOK=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS=y
CONFIG_HAVE_DYNAMIC_FTRACE_NO_PATCHABLE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_FENTRY=y
CONFIG_HAVE_OBJTOOL_MCOUNT=y
CONFIG_HAVE_OBJTOOL_NOP_MCOUNT=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_HAVE_BUILDTIME_MCOUNT_SORT=y
CONFIG_TRACING_SUPPORT=y
# CONFIG_FTRACE is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_SAMPLE_FTRACE_DIRECT=y
CONFIG_HAVE_SAMPLE_FTRACE_DIRECT_MULTI=y
CONFIG_ARCH_HAS_DEVMEM_IS_ALLOWED=y

#
# x86 Debugging
#
# CONFIG_X86_VERBOSE_BOOTUP is not set
CONFIG_EARLY_PRINTK=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_UNWINDER_ORC=y
# CONFIG_UNWINDER_FRAME_POINTER is not set
# end of x86 Debugging

#
# Kernel Testing and Coverage
#
# CONFIG_KUNIT is not set
CONFIG_ARCH_HAS_KCOV=y
CONFIG_CC_HAS_SANCOV_TRACE_PC=y
# CONFIG_KCOV is not set
# CONFIG_RUNTIME_TESTING_MENU is not set
CONFIG_ARCH_USE_MEMTEST=y
# CONFIG_MEMTEST is not set
# end of Kernel Testing and Coverage

#
# Rust hacking
#
# end of Rust hacking

CONFIG_WARN_MISSING_DOCUMENTS=y
CONFIG_WARN_ABI_ERRORS=y
# end of Kernel hacking

Re: [PATCH V1 net-next 5/5] net: ena: Add devlink documentation

[Kuniyuki Iwashima]

From:   David Arinzon <darinzon@amazon.com>
Date:   Sun, 8 Jan 2023 10:35:33 +0000
> Update the documentation with a devlink section, the
> added files, as well as large LLQ enablement.
> 
> Signed-off-by: Shay Agroskin <shayagr@amazon.com>
> Signed-off-by: David Arinzon <darinzon@amazon.com>
> ---
>  .../device_drivers/ethernet/amazon/ena.rst    | 30 +++++++++++++++++++

Each driver's devlink doc exists under Documentation/networking/devlink/ and
linked from index.html there.

We should duplicate this doc under Documentation/networking/devlink/ or link from
the index.html ?


>  1 file changed, 30 insertions(+)
> 
> diff --git a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst
> index 8bcb173e0353..1229732a8c91 100644
> --- a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst
> +++ b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst
> @@ -53,6 +53,7 @@ ena_common_defs.h   Common definitions for ena_com layer.
>  ena_regs_defs.h     Definition of ENA PCI memory-mapped (MMIO) registers.
>  ena_netdev.[ch]     Main Linux kernel driver.
>  ena_ethtool.c       ethtool callbacks.
> +ena_devlink.[ch]    devlink files (see `devlink support`_ for more info)
>  ena_pci_id_tbl.h    Supported device IDs.
>  =================   ======================================================
>  
> @@ -253,6 +254,35 @@ RSS
>  - The user can provide a hash key, hash function, and configure the
>    indirection table through `ethtool(8)`.
>  
> +.. _`devlink support`:
> +DEVLINK SUPPORT
> +===============
> +.. _`devlink`: https://www.kernel.org/doc/html/latest/networking/devlink/index.html
> +
> +`devlink`_ supports toggling LLQ entry size between the default 128 bytes and 256
> +bytes.
> +A 128 bytes entry size allows for a maximum of 96 bytes of packet header size
> +which sometimes is not enough (e.g. when using tunneling).
> +Increasing LLQ entry size to 256 bytes, allows a maximum header size of 224
> +bytes. This comes with the penalty of reducing the number of LLQ entries in the
> +TX queue by 2 (i.e. from 1024 to 512).
> +
> +The entry size can be toggled by enabling/disabling the large_llq_header devlink
> +param and reloading the driver to make it take effect, e.g.
> +
> +.. code-block:: shell
> +
> +  sudo devlink dev param set pci/0000:00:06.0 name large_llq_header value true cmode driverinit
> +  sudo devlink dev reload pci/0000:00:06.0
> +
> +One way to verify that the TX queue entry size has indeed increased is to check
> +that the maximum TX queue depth is 512. This can be checked, for example, by
> +using:
> +
> +.. code-block:: shell
> +
> +  ethtool -g [interface]
> +
>  DATA PATH
>  =========
>  
> -- 
> 2.38.1

RE: [PATCH V1 net-next 5/5] net: ena: Add devlink documentation

[Arinzon, David]

Good catch! Thanks.
This will be addressed in the next version of the patchset.

> -----Original Message-----
> From: Iwashima, Kuniyuki <kuniyu@amazon.co.jp>
> Sent: Monday, January 9, 2023 8:00 AM
> To: Arinzon, David <darinzon@amazon.com>
> Cc: Kiyanovski, Arthur <akiyano@amazon.com>; Saidi, Ali
> <alisaidi@amazon.com>; davem@davemloft.net; Itzko, Shahar
> <itzko@amazon.com>; kuba@kernel.org; Matushevsky, Alexander
> <matua@amazon.com>; Bshara, Nafea <nafea@amazon.com>; Dagan,
> Noam <ndagan@amazon.com>; netdev@vger.kernel.org; Abboud, Osama
> <osamaabb@amazon.com>; Bshara, Saeed <saeedb@amazon.com>;
> Agroskin, Shay <shayagr@amazon.com>; Machulsky, Zorik
> <zorik@amazon.com>; Iwashima, Kuniyuki <kuniyu@amazon.co.jp>
> Subject: Re: [PATCH V1 net-next 5/5] net: ena: Add devlink documentation
> 
> From:   David Arinzon <darinzon@amazon.com>
> Date:   Sun, 8 Jan 2023 10:35:33 +0000
> > Update the documentation with a devlink section, the added files, as
> > well as large LLQ enablement.
> >
> > Signed-off-by: Shay Agroskin <shayagr@amazon.com>
> > Signed-off-by: David Arinzon <darinzon@amazon.com>
> > ---
> >  .../device_drivers/ethernet/amazon/ena.rst    | 30
> +++++++++++++++++++
> 
> Each driver's devlink doc exists under Documentation/networking/devlink/
> and linked from index.html there.
> 
> We should duplicate this doc under Documentation/networking/devlink/
> or link from the index.html ?
> 
> 
> >  1 file changed, 30 insertions(+)
> >
> > diff --git
> > a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst
> > b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst
> > index 8bcb173e0353..1229732a8c91 100644
> > ---
> a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst
> > +++
> b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst
> > @@ -53,6 +53,7 @@ ena_common_defs.h   Common definitions for
> ena_com layer.
> >  ena_regs_defs.h     Definition of ENA PCI memory-mapped (MMIO)
> registers.
> >  ena_netdev.[ch]     Main Linux kernel driver.
> >  ena_ethtool.c       ethtool callbacks.
> > +ena_devlink.[ch]    devlink files (see `devlink support`_ for more info)
> >  ena_pci_id_tbl.h    Supported device IDs.
> >  =================
> ======================================================
> >
> > @@ -253,6 +254,35 @@ RSS
> >  - The user can provide a hash key, hash function, and configure the
> >    indirection table through `ethtool(8)`.
> >
> > +.. _`devlink support`:
> > +DEVLINK SUPPORT
> > +===============
> > +.. _`devlink`:
> >
> +https://www.kernel.org/doc/html/latest/networking/devlink/index.html
> > +
> > +`devlink`_ supports toggling LLQ entry size between the default 128
> > +bytes and 256 bytes.
> > +A 128 bytes entry size allows for a maximum of 96 bytes of packet
> > +header size which sometimes is not enough (e.g. when using tunneling).
> > +Increasing LLQ entry size to 256 bytes, allows a maximum header size
> > +of 224 bytes. This comes with the penalty of reducing the number of
> > +LLQ entries in the TX queue by 2 (i.e. from 1024 to 512).
> > +
> > +The entry size can be toggled by enabling/disabling the
> > +large_llq_header devlink param and reloading the driver to make it take
> effect, e.g.
> > +
> > +.. code-block:: shell
> > +
> > +  sudo devlink dev param set pci/0000:00:06.0 name large_llq_header
> > + value true cmode driverinit  sudo devlink dev reload
> > + pci/0000:00:06.0
> > +
> > +One way to verify that the TX queue entry size has indeed increased
> > +is to check that the maximum TX queue depth is 512. This can be
> > +checked, for example, by
> > +using:
> > +
> > +.. code-block:: shell
> > +
> > +  ethtool -g [interface]
> > +
> >  DATA PATH
> >  =========
> >
> > --
> > 2.38.1

Re: [PATCH V1 net-next 0/5] Add devlink support to ena

[Jakub Kicinski]

On Sun, 8 Jan 2023 10:35:28 +0000 David Arinzon wrote:
> This patchset adds devlink support to the ena driver.

Wrong place, please take a look at 

	struct kernel_ethtool_ringparam::tx_push

and ETHTOOL_A_RINGS_TX_PUSH. I think you just want to configure 
the max size of the TX push, right?

The reload is also an overkill, reload should re-register all driver
objects but the devlink instance, IIRC. You're not even unregistering
the netdev. You should handle this change the same way you handle any
ring size changes.


For future reference - if you ever _actually_ need devlink please use
the devl_* APIs and take the instance locks explicitly. There has not
been a single devlink reload implementation which would get locking
right using the devlink_* APIs 😔️

RE: [PATCH V1 net-next 0/5] Add devlink support to ena

[Arinzon, David]

> On Sun, 8 Jan 2023 10:35:28 +0000 David Arinzon wrote:
> > This patchset adds devlink support to the ena driver.
> 
> Wrong place, please take a look at
> 
>         struct kernel_ethtool_ringparam::tx_push
> 
> and ETHTOOL_A_RINGS_TX_PUSH. I think you just want to configure the
> max size of the TX push, right?
> 

Hi Jakub,
Thank you for the feedbacks.

We're not configuring the max size of the TX push, but effectively the
maximal packet header size to be pushed to the device.
This is noted in the documentation on patch 5/5 in this patchset.
AFAIK, there's no relevant ethtool parameter for this configuration.

> The reload is also an overkill, reload should re-register all driver objects
> but the devlink instance, IIRC. You're not even unregistering the netdev.
> You should handle this change the same way you handle any ring size
> changes.
> 
> 

The LLQ configuration is different from other configurations set via ethtool
(like queue size and number of queues). LLQ requires re-negotiation
with the device and requires a reset, which is not performed in the ethtool
configurations case. It may be possible to unregister/register the netdev,
but it is unnecessary in this case, as most of the changes are reflected in the
interface and structures between the driver and the device.

> For future reference - if you ever _actually_ need devlink please use the
> devl_* APIs and take the instance locks explicitly. There has not been a
> single devlink reload implementation which would get locking right using
> the devlink_* APIs 😔️

This operation can happen in parallel to a reset of the device from a different
context which is unrelated to devlink. Our intention is to avoid such cases,
therefore, holding the devlink lock using devl_lock APIs will not be sufficient.
The driver holds the RTNL_LOCK in key places, either explicitly or implicitly,
as in ethtool configuration changes for example.

Re: [PATCH V1 net-next 0/5] Add devlink support to ena

[Jakub Kicinski]

On Tue, 10 Jan 2023 20:11:23 +0000 Arinzon, David wrote:
> > On Sun, 8 Jan 2023 10:35:28 +0000 David Arinzon wrote:  
> > > This patchset adds devlink support to the ena driver.  
> > 
> > Wrong place, please take a look at
> > 
> >         struct kernel_ethtool_ringparam::tx_push
> > 
> > and ETHTOOL_A_RINGS_TX_PUSH. I think you just want to configure the
> > max size of the TX push, right?
> 
> We're not configuring the max size of the TX push, but effectively the
> maximal packet header size to be pushed to the device.
> This is noted in the documentation on patch 5/5 in this patchset.
> AFAIK, there's no relevant ethtool parameter for this configuration.

Perhaps I should have complained about the low quality of that
documentation to make it clear that I have in fact read it :/

I read it again - and I still don't know what you're doing.
I sounds like inline header length configuration yet you also use LLQ
all over the place. And LLQ for ENA is documented as basically tx_push:

  - **Low Latency Queue (LLQ) mode or "push-mode":**

Please explain this in a way which assumes zero Amazon-specific
knowledge :(

> > The reload is also an overkill, reload should re-register all driver objects
> > but the devlink instance, IIRC. You're not even unregistering the netdev.
> > You should handle this change the same way you handle any ring size
> > changes.
> 
> The LLQ configuration is different from other configurations set via ethtool
> (like queue size and number of queues). LLQ requires re-negotiation
> with the device and requires a reset, which is not performed in the ethtool
> configurations case.

What do you mean when you say that reset is not required in the ethool
configuration case?

AFAIK ethtool config should not (although sadly very often it does)
cause any loss of unrelated configuration. But you can certainly reset
HW blocks or reneg features with FW or whatever else...

> It may be possible to unregister/register the netdev,
> but it is unnecessary in this case, as most of the changes are reflected in the
> interface and structures between the driver and the device.
> 
> > For future reference - if you ever _actually_ need devlink please use the
> > devl_* APIs and take the instance locks explicitly. There has not been a
> > single devlink reload implementation which would get locking right using
> > the devlink_* APIs 😔️  
> 
> This operation can happen in parallel to a reset of the device from a different
> context which is unrelated to devlink. Our intention is to avoid such cases,
> therefore, holding the devlink lock using devl_lock APIs will not be sufficient.
> The driver holds the RTNL_LOCK in key places, either explicitly or implicitly,
> as in ethtool configuration changes for example.

Yeah, which is why you should not be using devlink for this.

RE: [PATCH V1 net-next 0/5] Add devlink support to ena

[Arinzon, David]

> On Tue, 10 Jan 2023 20:11:23 +0000 Arinzon, David wrote:
> > > On Sun, 8 Jan 2023 10:35:28 +0000 David Arinzon wrote:
> > > > This patchset adds devlink support to the ena driver.
> > >
> > > Wrong place, please take a look at
> > >
> > >         struct kernel_ethtool_ringparam::tx_push
> > >
> > > and ETHTOOL_A_RINGS_TX_PUSH. I think you just want to configure
> the
> > > max size of the TX push, right?
> >
> > We're not configuring the max size of the TX push, but effectively the
> > maximal packet header size to be pushed to the device.
> > This is noted in the documentation on patch 5/5 in this patchset.
> > AFAIK, there's no relevant ethtool parameter for this configuration.
> 
> Perhaps I should have complained about the low quality of that
> documentation to make it clear that I have in fact read it :/
> 

Noted, we will see how to improve documentation going forward.

> I read it again - and I still don't know what you're doing.
> I sounds like inline header length configuration yet you also use LLQ all
> over the place. And LLQ for ENA is documented as basically tx_push:
> 
>   - **Low Latency Queue (LLQ) mode or "push-mode":**
> 
> Please explain this in a way which assumes zero Amazon-specific
> knowledge :(
> 

Low Latency Queues (LLQ) is a mode of operation where the packet headers
(up to a defined length) are being written directly to the device memory.
Therefore, you are right, the description is similar to tx_push. However,
This is not a configurable option while ETHTOOL_A_RINGS_TX_PUSH
configures whether to work in a mode or not.
If I'm understanding the intent behind ETHTOOL_A_RINGS_TX_PUSH
and the implementation in the driver that introduced the feature, it
refers to a push of the packet and not just the headers, which is not what
the ena driver does.

In this patchset, we allow the configuration of an extended size of the
Low Latency Queue, meaning, allow enabled another, larger, pre-defined
size to be used as a max size of the packet header to be pushed directly to
device memory. It is not configurable in value, therefore, it was defined as
large LLQ.

I hope this provides more clarification, if not, I'll be happy to elaborate further.

> > > The reload is also an overkill, reload should re-register all driver
> > > objects but the devlink instance, IIRC. You're not even unregistering
> the netdev.
> > > You should handle this change the same way you handle any ring size
> > > changes.
> >
> > The LLQ configuration is different from other configurations set via
> > ethtool (like queue size and number of queues). LLQ requires
> > re-negotiation with the device and requires a reset, which is not
> > performed in the ethtool configurations case.
> 
> What do you mean when you say that reset is not required in the ethool
> configuration case?
> 
> AFAIK ethtool config should not (although sadly very often it does) cause
> any loss of unrelated configuration. But you can certainly reset HW blocks
> or reneg features with FW or whatever else...
> 

The ena driver currently supports various configurations via ethtool,
for example, changing the number of queues/channels and the
queue/channel size. These options, for example, require changes in the
netdev and the interfaces with the kernel, therefore, we perform a
reconfiguration on this level. But, these configurations do not require
the interface between the driver and device to be reset.
Low Latency Queue mode change from standard to large (what's being
configured in this pathset) is more significant functionality change and
requires this reset.

> > It may be possible to unregister/register the netdev, but it is
> > unnecessary in this case, as most of the changes are reflected in the
> > interface and structures between the driver and the device.
> >
> > > For future reference - if you ever _actually_ need devlink please
> > > use the
> > > devl_* APIs and take the instance locks explicitly. There has not
> > > been a single devlink reload implementation which would get locking
> > > right using the devlink_* APIs 😔️
> >
> > This operation can happen in parallel to a reset of the device from a
> > different context which is unrelated to devlink. Our intention is to
> > avoid such cases, therefore, holding the devlink lock using devl_lock APIs
> will not be sufficient.
> > The driver holds the RTNL_LOCK in key places, either explicitly or
> > implicitly, as in ethtool configuration changes for example.
> 
> Yeah, which is why you should not be using devlink for this.

Though using ethtool would've been more convenient as RTNL_LOCK is being
held while the command takes place, this functionality is specific to Amazon and
adding a private flag, for example, is not desirable. Creating a new ethtool or
updating an existing one would be specific only for Amazon as well.
Per our understanding, ethtool is used for interface operations, while devlink
is used for device operations. Performing the discussed configuration change
is a device configuration, as it affects the way the device configures its own memory.

Re: [PATCH V1 net-next 0/5] Add devlink support to ena

[Jakub Kicinski]

On Wed, 11 Jan 2023 08:58:46 +0000 Arinzon, David wrote:
> > I read it again - and I still don't know what you're doing.
> > I sounds like inline header length configuration yet you also use LLQ all
> > over the place. And LLQ for ENA is documented as basically tx_push:
> > 
> >   - **Low Latency Queue (LLQ) mode or "push-mode":**
> > 
> > Please explain this in a way which assumes zero Amazon-specific
> > knowledge :(
> 
> Low Latency Queues (LLQ) is a mode of operation where the packet headers
> (up to a defined length) are being written directly to the device memory.
> Therefore, you are right, the description is similar to tx_push. However,
> This is not a configurable option while ETHTOOL_A_RINGS_TX_PUSH
> configures whether to work in a mode or not.
> If I'm understanding the intent behind ETHTOOL_A_RINGS_TX_PUSH
> and the implementation in the driver that introduced the feature, it
> refers to a push of the packet and not just the headers, which is not what
> the ena driver does.
> 
> In this patchset, we allow the configuration of an extended size of the
> Low Latency Queue, meaning, allow enabled another, larger, pre-defined
> size to be used as a max size of the packet header to be pushed directly to
> device memory. It is not configurable in value, therefore, it was defined as
> large LLQ.
> 
> I hope this provides more clarification, if not, I'll be happy to elaborate further.

Thanks, the large missing piece in my understanding is still what 
the user visible impact of this change is. Without increasing 
the LLQ entry size, a user who sends packet with long headers will:
 a) see higher latency thru the NIC, but everything else is the same
 b) see higher latency and lower overall throughput in terms of PPS
 c) will have limited access to offloads, because the device requires
    full access to headers via LLQ for some offloads

which one of the three is the closest?

RE: [PATCH V1 net-next 0/5] Add devlink support to ena

[Arinzon, David]

> -----Original Message-----
> From: Jakub Kicinski <kuba@kernel.org>
> Sent: Wednesday, January 11, 2023 9:01 PM
> To: Arinzon, David <darinzon@amazon.com>
> Cc: David Miller <davem@davemloft.net>; netdev@vger.kernel.org;
> Machulsky, Zorik <zorik@amazon.com>; Matushevsky, Alexander
> <matua@amazon.com>; Bshara, Saeed <saeedb@amazon.com>; Bshara,
> Nafea <nafea@amazon.com>; Saidi, Ali <alisaidi@amazon.com>;
> Kiyanovski, Arthur <akiyano@amazon.com>; Dagan, Noam
> <ndagan@amazon.com>; Agroskin, Shay <shayagr@amazon.com>; Itzko,
> Shahar <itzko@amazon.com>; Abboud, Osama <osamaabb@amazon.com>
> Subject: RE: [EXTERNAL][PATCH V1 net-next 0/5] Add devlink support to
> ena
> 
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and
> know the content is safe.
> 
> 
> 
> On Wed, 11 Jan 2023 08:58:46 +0000 Arinzon, David wrote:
> > > I read it again - and I still don't know what you're doing.
> > > I sounds like inline header length configuration yet you also use
> > > LLQ all over the place. And LLQ for ENA is documented as basically
> tx_push:
> > >
> > >   - **Low Latency Queue (LLQ) mode or "push-mode":**
> > >
> > > Please explain this in a way which assumes zero Amazon-specific
> > > knowledge :(
> >
> > Low Latency Queues (LLQ) is a mode of operation where the packet
> > headers (up to a defined length) are being written directly to the device
> memory.
> > Therefore, you are right, the description is similar to tx_push.
> > However, This is not a configurable option while
> > ETHTOOL_A_RINGS_TX_PUSH configures whether to work in a mode or
> not.
> > If I'm understanding the intent behind ETHTOOL_A_RINGS_TX_PUSH
> and the
> > implementation in the driver that introduced the feature, it refers to
> > a push of the packet and not just the headers, which is not what the
> > ena driver does.
> >
> > In this patchset, we allow the configuration of an extended size of
> > the Low Latency Queue, meaning, allow enabled another, larger,
> > pre-defined size to be used as a max size of the packet header to be
> > pushed directly to device memory. It is not configurable in value,
> > therefore, it was defined as large LLQ.
> >
> > I hope this provides more clarification, if not, I'll be happy to elaborate
> further.
> 
> Thanks, the large missing piece in my understanding is still what the user
> visible impact of this change is. Without increasing the LLQ entry size, a
> user who sends packet with long headers will:
>  a) see higher latency thru the NIC, but everything else is the same
>  b) see higher latency and lower overall throughput in terms of PPS
>  c) will have limited access to offloads, because the device requires
>     full access to headers via LLQ for some offloads
> 
> which one of the three is the closest?

You're right, I went through the documentation again, and I see that the implications
of a case where packet headers are longer than the LLQ entry size are not mentioned
properly. We'll rework it to explain the motivation to turn on this mode in relevant use cases.
If the packet network headers are not within the size of the LLQ entry, then the packet will
be dropped. So I'll say that c) describes the impact the best given that certain types of
traffic will not succeed or have disruptions due to dropped TX packets.

Re: [PATCH V1 net-next 0/5] Add devlink support to ena

[Jakub Kicinski]

On Wed, 11 Jan 2023 19:31:39 +0000 Arinzon, David wrote:
> If the packet network headers are not within the size of the LLQ entry, then the packet will
> be dropped. So I'll say that c) describes the impact the best given that certain types of
> traffic will not succeed or have disruptions due to dropped TX packets.

I see. Sounds like it could be a fit for
DEVLINK_ATTR_ESWITCH_INLINE_MODE ? But that one configures the depth of
the headers copied inline, rather than bytes. We could add a value for
"tunneled" and have that imply 256B LLQ in case of ena.

The other option is to introduce the concept of "max length of inline
data" to ethtool, and add a new netlink attribute to ethtool -g.

Either way - the length of "inline|push" data is not a ena-specific
concept, so using private knobs (devlink params or ethtool flags) is
not appropriate upstream. We should add a bona fide uAPI for it.

RE: [PATCH V1 net-next 0/5] Add devlink support to ena

[Arinzon, David]

> -----Original Message-----
> From: Jakub Kicinski <kuba@kernel.org>
> Sent: Wednesday, January 11, 2023 10:00 PM
> To: Arinzon, David <darinzon@amazon.com>
> Cc: David Miller <davem@davemloft.net>; netdev@vger.kernel.org;
> Machulsky, Zorik <zorik@amazon.com>; Matushevsky, Alexander
> <matua@amazon.com>; Bshara, Saeed <saeedb@amazon.com>; Bshara,
> Nafea <nafea@amazon.com>; Saidi, Ali <alisaidi@amazon.com>;
> Kiyanovski, Arthur <akiyano@amazon.com>; Dagan, Noam
> <ndagan@amazon.com>; Agroskin, Shay <shayagr@amazon.com>; Itzko,
> Shahar <itzko@amazon.com>; Abboud, Osama <osamaabb@amazon.com>
> Subject: RE: [EXTERNAL][PATCH V1 net-next 0/5] Add devlink support to
> ena
> 
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and
> know the content is safe.
> 
> 
> 
> On Wed, 11 Jan 2023 19:31:39 +0000 Arinzon, David wrote:
> > If the packet network headers are not within the size of the LLQ
> > entry, then the packet will be dropped. So I'll say that c) describes
> > the impact the best given that certain types of traffic will not succeed or
> have disruptions due to dropped TX packets.
> 
> I see. Sounds like it could be a fit for
> DEVLINK_ATTR_ESWITCH_INLINE_MODE ? But that one configures the
> depth of the headers copied inline, rather than bytes. We could add a value
> for "tunneled" and have that imply 256B LLQ in case of ena.
> 
> The other option is to introduce the concept of "max length of inline data"
> to ethtool, and add a new netlink attribute to ethtool -g.
> 
> Either way - the length of "inline|push" data is not a ena-specific concept,
> so using private knobs (devlink params or ethtool flags) is not appropriate
> upstream. We should add a bona fide uAPI for it.

I've looked into the original commit of DEVLINK_ATTR_ESWITCH_INLINE_MODE
and it seems to be more related to encapsulation and adding a tunneling concept
here would be misleading, as it doesn't cover all the use-cases affected here.
It is true that in case of tunneling and if there's a need to access the encapsulated
headers, the length of the headers might be larger than the default value in LLQ,
but this may be also the case for non-tunneled traffic, for example,
IPv6 with TCP SACK option.

I'll note again that this is not a configurable value, meaning, the only option is
to have 128B (standard) or 256B (large) LLQ, there's no option to set other values,
but only choose between the modes, therefore, I don't see how having such an
option through ethtool, as you suggested (setting the max length) can be
beneficial in our use-case (might be good overall, as you noted, it's a more
generic concept). Will put more thought into it.

Re: [PATCH V1 net-next 0/5] Add devlink support to ena

[Jakub Kicinski]

On Wed, 11 Jan 2023 21:21:16 +0000 Arinzon, David wrote:
> I'll note again that this is not a configurable value, meaning, the only option is
> to have 128B (standard) or 256B (large) LLQ, there's no option to set other values,
> but only choose between the modes, therefore, I don't see how having such an
> option through ethtool, as you suggested (setting the max length) can be
> beneficial in our use-case (might be good overall, as you noted, it's a more
> generic concept). Will put more thought into it.

FWIW you can just return an error via extack if user tries to set an
unsupported value, like:

	NL_SET_ERR_MSG(extack, "only X and Y are supported");

and it will pop up as part of the error in ethtool CLI. Or do some
rounding up - a lot of drivers does that for ring params already.

Re: [PATCH V1 net-next 0/5] Add devlink support to ena

[Gal Pressman]

On 11/01/2023 22:00, Jakub Kicinski wrote:
> On Wed, 11 Jan 2023 19:31:39 +0000 Arinzon, David wrote:
>> If the packet network headers are not within the size of the LLQ entry, then the packet will
>> be dropped. So I'll say that c) describes the impact the best given that certain types of
>> traffic will not succeed or have disruptions due to dropped TX packets.
> 
> I see. Sounds like it could be a fit for
> DEVLINK_ATTR_ESWITCH_INLINE_MODE ? But that one configures the depth of
> the headers copied inline, rather than bytes. We could add a value for
> "tunneled" and have that imply 256B LLQ in case of ena.
> 
> The other option is to introduce the concept of "max length of inline
> data" to ethtool, and add a new netlink attribute to ethtool -g.

TX copybreak? When the user sets it to > 96 bytes, use the large LLQ.

BTW, shouldn't ethtool's tx_push reflect the fact that LLQs are being
used? I don't see it used in ena.

Re: [PATCH V1 net-next 0/5] Add devlink support to ena

[Shay Agroskin]

Gal Pressman <gal@nvidia.com> writes:

> On 11/01/2023 22:00, Jakub Kicinski wrote:
>> On Wed, 11 Jan 2023 19:31:39 +0000 Arinzon, David wrote:
>>> If the packet network headers are not within the size of the 
>>> LLQ entry, then the packet will
>>> be dropped. So I'll say that c) describes the impact the best 
>>> given that certain types of
>>> traffic will not succeed or have disruptions due to dropped TX 
>>> packets.
>> 
>> I see. Sounds like it could be a fit for
>> DEVLINK_ATTR_ESWITCH_INLINE_MODE ? But that one configures the 
>> depth of
>> the headers copied inline, rather than bytes. We could add a 
>> value for
>> "tunneled" and have that imply 256B LLQ in case of ena.
>> 
>> The other option is to introduce the concept of "max length of 
>> inline
>> data" to ethtool, and add a new netlink attribute to ethtool 
>> -g.
>
> TX copybreak? When the user sets it to > 96 bytes, use the large 
> LLQ.
>
> BTW, shouldn't ethtool's tx_push reflect the fact that LLQs are 
> being
> used? I don't see it used in ena.

Using tx copybreak does sound like it can work for our use 
case. Thanks for the tip Gal (:

Jakub, do you see an issue with utilizing tx_copybreak ethtool 
parameter instead of the devlink param in this patchset ?

Thanks,
Shay

Re: [PATCH V1 net-next 0/5] Add devlink support to ena

[Jakub Kicinski]

On Thu, 12 Jan 2023 15:47:13 +0200 Shay Agroskin wrote:
> Gal Pressman <gal@nvidia.com> writes:
> > TX copybreak? When the user sets it to > 96 bytes, use the large 
> > LLQ.
> >
> > BTW, shouldn't ethtool's tx_push reflect the fact that LLQs are 
> > being
> > used? I don't see it used in ena.  
> 
> Using tx copybreak does sound like it can work for our use 
> case. Thanks for the tip Gal (:
> 
> Jakub, do you see an issue with utilizing tx_copybreak ethtool 
> parameter instead of the devlink param in this patchset ?

IDK, the semantics don't feel close enough.

As a user I'd set tx_copybreak only on systems which have IOMMU enabled 
(or otherwise have high cost of DMA mapping), to save CPU cycles.

The ena feature does not seem to be about CPU cycle saving (likely 
the opposite, in fact), and does not operate on full segments AFAIU.

Hence my preference to expose it as a new tx_push_buf_len, combining
the semantics of tx_push and rx_buf_len.

Re: [PATCH V1 net-next 0/5] Add devlink support to ena

[Gal Pressman]

On 12/01/2023 21:56, Jakub Kicinski wrote:
> On Thu, 12 Jan 2023 15:47:13 +0200 Shay Agroskin wrote:
>> Gal Pressman <gal@nvidia.com> writes:
>>> TX copybreak? When the user sets it to > 96 bytes, use the large 
>>> LLQ.
>>>
>>> BTW, shouldn't ethtool's tx_push reflect the fact that LLQs are 
>>> being
>>> used? I don't see it used in ena.  
>>
>> Using tx copybreak does sound like it can work for our use 
>> case. Thanks for the tip Gal (:
>>
>> Jakub, do you see an issue with utilizing tx_copybreak ethtool 
>> parameter instead of the devlink param in this patchset ?
> 
> IDK, the semantics don't feel close enough.
> 
> As a user I'd set tx_copybreak only on systems which have IOMMU enabled 
> (or otherwise have high cost of DMA mapping), to save CPU cycles.
> 
> The ena feature does not seem to be about CPU cycle saving (likely 
> the opposite, in fact), and does not operate on full segments AFAIU.

Segments?

> Hence my preference to expose it as a new tx_push_buf_len, combining
> the semantics of tx_push and rx_buf_len.

Sounds like a good idea.
To clarify, buf_len here refers to the size of the inline'd part, not
the WQE itself, correct? The driver will use whatever WQE size it needs
in order to accommodate the requested inline size?

Re: [PATCH V1 net-next 0/5] Add devlink support to ena

[Jakub Kicinski]

On Sun, 15 Jan 2023 12:05:33 +0200 Gal Pressman wrote:
> > IDK, the semantics don't feel close enough.
> > 
> > As a user I'd set tx_copybreak only on systems which have IOMMU enabled 
> > (or otherwise have high cost of DMA mapping), to save CPU cycles.
> > 
> > The ena feature does not seem to be about CPU cycle saving (likely 
> > the opposite, in fact), and does not operate on full segments AFAIU.  
> 
> Segments?

Complete DMA buffers. Basically whether the optimization
only kicks in if skb->len < configured_len or 
skb_headlen() < configured_len.

> > Hence my preference to expose it as a new tx_push_buf_len, combining
> > the semantics of tx_push and rx_buf_len.  
> 
> Sounds like a good idea.
> To clarify, buf_len here refers to the size of the inline'd part, not
> the WQE itself, correct? The driver will use whatever WQE size it needs
> in order to accommodate the requested inline size?

We can decide either way, but I _think_ rx_buf_len refers to the size
as allocated, not necessarily usable size (in case the first buffer has
padding / headroom). But as long as we clearly document - either way is
fine.

Re: [PATCH V1 net-next 0/5] Add devlink support to ena

[Shay Agroskin]

Jakub Kicinski <kuba@kernel.org> writes:

> On Thu, 12 Jan 2023 15:47:13 +0200 Shay Agroskin wrote:
>> Gal Pressman <gal@nvidia.com> writes:
>> > TX copybreak? When the user sets it to > 96 bytes, use the 
>> > large 
>> > LLQ.
>> >
>> > BTW, shouldn't ethtool's tx_push reflect the fact that LLQs 
>> > are 
>> > being
>> > used? I don't see it used in ena.  
>> 
>> Using tx copybreak does sound like it can work for our use 
>> case. Thanks for the tip Gal (:
>> 
>> Jakub, do you see an issue with utilizing tx_copybreak ethtool 
>> parameter instead of the devlink param in this patchset ?
>
> IDK, the semantics don't feel close enough.
>
> As a user I'd set tx_copybreak only on systems which have IOMMU 
> enabled 
> (or otherwise have high cost of DMA mapping), to save CPU 
> cycles.
>
> The ena feature does not seem to be about CPU cycle saving 
> (likely 
> the opposite, in fact), and does not operate on full segments 
> AFAIU.
>
> Hence my preference to expose it as a new tx_push_buf_len, 
> combining
> the semantics of tx_push and rx_buf_len.

We'll proceed with working on and RFC that adds the new param to 
ethtool.

Going forward, can I ask what's the community's stand on adding 
sysfs or procfse entries to the driver as means to tweak custom 
device attributes ?

Thanks,
Shay

Re: [PATCH V1 net-next 0/5] Add devlink support to ena

[Jakub Kicinski]

On Mon, 16 Jan 2023 16:23:56 +0200 Shay Agroskin wrote:
> Going forward, can I ask what's the community's stand on adding 
> sysfs or procfse entries to the driver as means to tweak custom 
> device attributes ?

We prefer to avoid both of those.

If you're worried about user having hard time changing the parameter
because the user space ethtool command is out of date - our solution 
to that will likely be the "Netlink protocol descriptions in YAML".

With that thing you don't really need a custom CLI tool to issue
Netlink commands. Until it gets merged you may need to hand-craft 
the right netlink message in python or another scripting language 
of choice :(