[PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Mika Westerberg]

Hi all,

This patch series adds support for Thunderbolt security levels, which were
first introduced in Intel Falcon Ridge Thunderbolt controller, to prevent
DMA attacks when PCIe is tunneled over Thunderbolt fabric. This is needed
if there is no IOMMU available for various reasons.

Most PCs out there having Falcon Ridge or newer have security level set to
"user" which means that user authorization is needed before PCIe tunnel is
creaded (the PCIe device appears). This effectively means that without
driver support the user needs to configure security level from BIOS to
"none" to get Thunderbolt devices connected. With these patches the user
can authorize devices using sysfs attributes like:

  # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized

In addition these patches add support for upgrading NVM firmware running on
a host or device by running something like:

  # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem
  # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate

This is documented with more details in patch [23/24].

This series is based on Amir's networking patches [1] but instead of
splitting the functionality between kernel driver and userspace daemon, we
take advantage of Linux driver core by converting the existing driver to
expose a Linux bus (domain) and devices (switches). Notifications to the
userspace about plugged/unplugged devices is handled by standard uevents
when a device is added to/removed from the Thunderbolt bus.

Since thunderbolt device identification and authorization can be done
directly through sysfs attributes there is no need for userspace daemon.
However, there still should be an application that promps user for unknown
devices and allows selecting between "single connect" and "connect always"
keeping this information in a database or similar persistent storage. This
patch series only provides mechanism for userspace applications to achieve
that.

Where Internal Connection Manager (ICM) firmware is available and usable,
we use it in the driver. This also includes newer Apple Macbooks with
Alpine Ridge. For older Macbooks the driver works as before but in addition
the Thunderbolt bus is available there as well (including possibility to
upgrade NVM firmware of connected devices).

We are also in works of porting Amir's networking driver to work on top of
the new Thunderbolt bus pretty much the same way firewire networking is
currently done. In addition this makes is possible to introduce other
protocols like a char device that allows userspace directly to communicate
accross Thunderbolt domains.

Note for Macs the Linux native PCIe hotplug support does not work well with
the Thunderbolt PCIe topologies where there is need to put all available
resources to the PCIe downstream port where the PCIe chain is extended.
This is something we need to fix. In the mean time is a way to work it
around by passing "pci=hpbussize=10,hpmemsize=2M" or so to the kernel
command line.

These patches use uuid_be from uuid.h but I've learned that there is a work
to remove the type completely in favor of new uuid_t [2]. I'm not sure what
to do regarding that because those patches are not yet in the mainline.

[1] https://lkml.org/lkml/2016/11/9/341
[2] http://git.infradead.org/users/hch/vfs.git/shortlog/refs/heads/uuid-types

Mika Westerberg (24):
  thunderbolt: Use const buffer pointer in write operations
  thunderbolt: Do not try to read UID if DROM offset is read as 0
  thunderbolt: Do not warn about newer DROM versions
  thunderbolt: Add MSI-X support
  thunderbolt: Rework capability handling
  thunderbolt: Introduce thunderbolt bus and connection manager
  thunderbolt: Convert switch to a device
  thunderbolt: Fail switch adding operation if reading DROM fails
  thunderbolt: Do not fail if DROM data CRC32 is invalid
  thunderbolt: Read vendor and device name from DROM
  thunderbolt: Move control channel messages to tb_msgs.h
  thunderbolt: Expose get_route() to other files
  thunderbolt: Expose make_header() to other files
  thunderbolt: Let the connection manager handle all notifications
  thunderbolt: Rework control channel to be more reliable
  thunderbolt: Add Thunderbolt 3 PCI IDs
  thunderbolt: Add support for NHI mailbox
  thunderbolt: Store Thunderbolt generation in the switch structure
  thunderbolt: Add support for DMA configuration based mailbox
  thunderbolt: Do not touch the hardware if the NHI is gone on resume
  thunderbolt: Add support for Internal Connection Manager (ICM)
  thunderbolt: Add support for host and device NVM firmware upgrade
  thunderbolt: Add documentation how Thunderbolt bus can be used
  MAINTAINERS: Add maintainers for Thunderbolt driver

 Documentation/ABI/testing/sysfs-bus-thunderbolt |  108 +++
 Documentation/admin-guide/index.rst             |    1 +
 Documentation/admin-guide/thunderbolt.rst       |  197 ++++
 MAINTAINERS                                     |    3 +
 drivers/thunderbolt/Kconfig                     |   13 +-
 drivers/thunderbolt/Makefile                    |    2 +-
 drivers/thunderbolt/cap.c                       |  169 ++--
 drivers/thunderbolt/ctl.c                       |  655 +++++++++----
 drivers/thunderbolt/ctl.h                       |  105 ++-
 drivers/thunderbolt/dma_port.c                  |  524 +++++++++++
 drivers/thunderbolt/dma_port.h                  |   34 +
 drivers/thunderbolt/domain.c                    |  455 ++++++++++
 drivers/thunderbolt/eeprom.c                    |   84 +-
 drivers/thunderbolt/icm.c                       | 1098 ++++++++++++++++++++++
 drivers/thunderbolt/nhi.c                       |  302 +++++-
 drivers/thunderbolt/nhi.h                       |   91 +-
 drivers/thunderbolt/nhi_regs.h                  |   27 +
 drivers/thunderbolt/switch.c                    | 1109 +++++++++++++++++++++--
 drivers/thunderbolt/tb.c                        |  237 ++---
 drivers/thunderbolt/tb.h                        |  242 ++++-
 drivers/thunderbolt/tb_msgs.h                   |  260 ++++++
 drivers/thunderbolt/tb_regs.h                   |   31 +-
 drivers/thunderbolt/tunnel_pci.c                |   17 +-
 23 files changed, 5213 insertions(+), 551 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-bus-thunderbolt
 create mode 100644 Documentation/admin-guide/thunderbolt.rst
 create mode 100644 drivers/thunderbolt/dma_port.c
 create mode 100644 drivers/thunderbolt/dma_port.h
 create mode 100644 drivers/thunderbolt/domain.c
 create mode 100644 drivers/thunderbolt/icm.c
 create mode 100644 drivers/thunderbolt/tb_msgs.h

-- 
2.11.0

[PATCH 01/24] thunderbolt: Use const buffer pointer in write operations

[Mika Westerberg]

These functions should not (and do not) modify the argument in any way
so make it const.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 drivers/thunderbolt/ctl.c | 8 ++++----
 drivers/thunderbolt/ctl.h | 4 ++--
 drivers/thunderbolt/tb.h  | 2 +-
 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/thunderbolt/ctl.c b/drivers/thunderbolt/ctl.c
index 1146ff4210a9..1031d97407a8 100644
--- a/drivers/thunderbolt/ctl.c
+++ b/drivers/thunderbolt/ctl.c
@@ -273,7 +273,7 @@ static void tb_cfg_print_error(struct tb_ctl *ctl,
 	}
 }
 
-static void cpu_to_be32_array(__be32 *dst, u32 *src, size_t len)
+static void cpu_to_be32_array(__be32 *dst, const u32 *src, size_t len)
 {
 	int i;
 	for (i = 0; i < len; i++)
@@ -333,7 +333,7 @@ static void tb_ctl_tx_callback(struct tb_ring *ring, struct ring_frame *frame,
  *
  * Return: Returns 0 on success or an error code on failure.
  */
-static int tb_ctl_tx(struct tb_ctl *ctl, void *data, size_t len,
+static int tb_ctl_tx(struct tb_ctl *ctl, const void *data, size_t len,
 		     enum tb_cfg_pkg_type type)
 {
 	int res;
@@ -650,7 +650,7 @@ struct tb_cfg_result tb_cfg_read_raw(struct tb_ctl *ctl, void *buffer,
  *
  * Offset and length are in dwords.
  */
-struct tb_cfg_result tb_cfg_write_raw(struct tb_ctl *ctl, void *buffer,
+struct tb_cfg_result tb_cfg_write_raw(struct tb_ctl *ctl, const void *buffer,
 		u64 route, u32 port, enum tb_cfg_space space,
 		u32 offset, u32 length, int timeout_msec)
 {
@@ -695,7 +695,7 @@ int tb_cfg_read(struct tb_ctl *ctl, void *buffer, u64 route, u32 port,
 	return res.err;
 }
 
-int tb_cfg_write(struct tb_ctl *ctl, void *buffer, u64 route, u32 port,
+int tb_cfg_write(struct tb_ctl *ctl, const void *buffer, u64 route, u32 port,
 		 enum tb_cfg_space space, u32 offset, u32 length)
 {
 	struct tb_cfg_result res = tb_cfg_write_raw(ctl, buffer, route, port,
diff --git a/drivers/thunderbolt/ctl.h b/drivers/thunderbolt/ctl.h
index ba87d6e731dd..83ae54947082 100644
--- a/drivers/thunderbolt/ctl.h
+++ b/drivers/thunderbolt/ctl.h
@@ -61,13 +61,13 @@ struct tb_cfg_result tb_cfg_read_raw(struct tb_ctl *ctl, void *buffer,
 				     u64 route, u32 port,
 				     enum tb_cfg_space space, u32 offset,
 				     u32 length, int timeout_msec);
-struct tb_cfg_result tb_cfg_write_raw(struct tb_ctl *ctl, void *buffer,
+struct tb_cfg_result tb_cfg_write_raw(struct tb_ctl *ctl, const void *buffer,
 				      u64 route, u32 port,
 				      enum tb_cfg_space space, u32 offset,
 				      u32 length, int timeout_msec);
 int tb_cfg_read(struct tb_ctl *ctl, void *buffer, u64 route, u32 port,
 		enum tb_cfg_space space, u32 offset, u32 length);
-int tb_cfg_write(struct tb_ctl *ctl, void *buffer, u64 route, u32 port,
+int tb_cfg_write(struct tb_ctl *ctl, const void *buffer, u64 route, u32 port,
 		 enum tb_cfg_space space, u32 offset, u32 length);
 int tb_cfg_get_upstream_port(struct tb_ctl *ctl, u64 route);
 
diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
index 61d57ba64035..ba2b85750335 100644
--- a/drivers/thunderbolt/tb.h
+++ b/drivers/thunderbolt/tb.h
@@ -173,7 +173,7 @@ static inline int tb_port_read(struct tb_port *port, void *buffer,
 			   length);
 }
 
-static inline int tb_port_write(struct tb_port *port, void *buffer,
+static inline int tb_port_write(struct tb_port *port, const void *buffer,
 				enum tb_cfg_space space, u32 offset, u32 length)
 {
 	return tb_cfg_write(port->sw->tb->ctl,
-- 
2.11.0

[PATCH 02/24] thunderbolt: Do not try to read UID if DROM offset is read as 0

[Mika Westerberg]

At least Falcon Ridge when in host mode does not have any kind of DROM
available and reading DROM offset returns 0 for these. Do not try to
read DROM any further in that case.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 drivers/thunderbolt/eeprom.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/thunderbolt/eeprom.c b/drivers/thunderbolt/eeprom.c
index 6392990c984d..e4e64b130514 100644
--- a/drivers/thunderbolt/eeprom.c
+++ b/drivers/thunderbolt/eeprom.c
@@ -276,6 +276,9 @@ int tb_drom_read_uid_only(struct tb_switch *sw, u64 *uid)
 	if (res)
 		return res;
 
+	if (drom_offset == 0)
+		return -ENODEV;
+
 	/* read uid */
 	res = tb_eeprom_read_n(sw, drom_offset, data, 9);
 	if (res)
-- 
2.11.0

[PATCH 03/24] thunderbolt: Do not warn about newer DROM versions

[Mika Westerberg]

DROM version 2 is compatible with the previous generation so no need to
warn about that.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 drivers/thunderbolt/eeprom.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/thunderbolt/eeprom.c b/drivers/thunderbolt/eeprom.c
index e4e64b130514..eb2179c98b09 100644
--- a/drivers/thunderbolt/eeprom.c
+++ b/drivers/thunderbolt/eeprom.c
@@ -488,7 +488,7 @@ int tb_drom_read(struct tb_switch *sw)
 		goto err;
 	}
 
-	if (header->device_rom_revision > 1)
+	if (header->device_rom_revision > 2)
 		tb_sw_warn(sw, "drom device_rom_revision %#x unknown\n",
 			header->device_rom_revision);
 
-- 
2.11.0

[PATCH 04/24] thunderbolt: Add MSI-X support

[Mika Westerberg]

Intel Thunderbolt controllers support up to 16 MSI-X vectors. Using
MSI-X is preferred over MSI or legacy interrupt and may bring additional
performance because there is no need to check the status registers which
interrupt was triggered.

While there we convert comments in structs tb_ring and tb_nhi to follow
kernel-doc format more closely.

This code is based on the work done by Amir Levy and Michael Jamet.

Signed-off-by: Michael Jamet <michael.jamet@intel.com>
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
---
 drivers/thunderbolt/ctl.c      |   4 +-
 drivers/thunderbolt/nhi.c      | 159 ++++++++++++++++++++++++++++++++++++-----
 drivers/thunderbolt/nhi.h      |  56 +++++++++++----
 drivers/thunderbolt/nhi_regs.h |   9 +++
 4 files changed, 196 insertions(+), 32 deletions(-)

diff --git a/drivers/thunderbolt/ctl.c b/drivers/thunderbolt/ctl.c
index 1031d97407a8..889a32dd21e7 100644
--- a/drivers/thunderbolt/ctl.c
+++ b/drivers/thunderbolt/ctl.c
@@ -488,11 +488,11 @@ struct tb_ctl *tb_ctl_alloc(struct tb_nhi *nhi, hotplug_cb cb, void *cb_data)
 	if (!ctl->frame_pool)
 		goto err;
 
-	ctl->tx = ring_alloc_tx(nhi, 0, 10);
+	ctl->tx = ring_alloc_tx(nhi, 0, 10, RING_FLAG_NO_SUSPEND);
 	if (!ctl->tx)
 		goto err;
 
-	ctl->rx = ring_alloc_rx(nhi, 0, 10);
+	ctl->rx = ring_alloc_rx(nhi, 0, 10, RING_FLAG_NO_SUSPEND);
 	if (!ctl->rx)
 		goto err;
 
diff --git a/drivers/thunderbolt/nhi.c b/drivers/thunderbolt/nhi.c
index a8c20413dbda..17f3b1bdb7da 100644
--- a/drivers/thunderbolt/nhi.c
+++ b/drivers/thunderbolt/nhi.c
@@ -21,6 +21,12 @@
 
 #define RING_TYPE(ring) ((ring)->is_tx ? "TX ring" : "RX ring")
 
+/*
+ * Minimal number of vectors when we use MSI-X. Two for control channel
+ * Rx/Tx and the rest four are for cross domain DMA paths.
+ */
+#define MSIX_MIN_VECS		6
+#define MSIX_MAX_VECS		16
 
 static int ring_interrupt_index(struct tb_ring *ring)
 {
@@ -239,8 +245,82 @@ int __ring_enqueue(struct tb_ring *ring, struct ring_frame *frame)
 	return ret;
 }
 
+static irqreturn_t ring_msix(int irq, void *data)
+{
+	struct tb_ring *ring = data;
+
+	schedule_work(&ring->work);
+	return IRQ_HANDLED;
+}
+
+static void ring_map_unmap_msix(struct tb_ring *ring, bool map)
+{
+	u32 step, shift, ivr, misc;
+	int index;
+
+	if (ring->irq <= 0)
+		return;
+
+	if (ring->is_tx)
+		index = ring->hop;
+	else
+		index = ring->hop + ring->nhi->hop_count;
+
+	/*
+	 * Ask the hardware to clear interrupt status bits automatically
+	 * since we already know which interrupt was triggered.
+	 */
+	misc = ioread32(ring->nhi->iobase + REG_DMA_MISC);
+	if (!(misc & REG_DMA_MISC_INT_AUTO_CLEAR)) {
+		misc |= REG_DMA_MISC_INT_AUTO_CLEAR;
+		iowrite32(misc, ring->nhi->iobase + REG_DMA_MISC);
+	}
+
+	step = index / REG_INT_VEC_ALLOC_REGS * REG_INT_VEC_ALLOC_BITS;
+	shift = index % REG_INT_VEC_ALLOC_REGS * REG_INT_VEC_ALLOC_BITS;
+	ivr = ioread32(ring->nhi->iobase + REG_INT_VEC_ALLOC_BASE + step);
+	ivr &= ~(REG_INT_VEC_ALLOC_MASK << shift);
+	if (map)
+		ivr |= ring->vector << shift;
+	iowrite32(ivr, ring->nhi->iobase + REG_INT_VEC_ALLOC_BASE + step);
+}
+
+static int ring_request_msix(struct tb_ring *ring, bool no_suspend)
+{
+	struct tb_nhi *nhi = ring->nhi;
+	unsigned long irqflags;
+	int ret;
+
+	if (!nhi->pdev->msix_enabled)
+		return 0;
+
+	ret = ida_simple_get(&nhi->msix_ida, 0, MSIX_MAX_VECS, GFP_KERNEL);
+	if (ret < 0)
+		return ret;
+
+	ring->vector = ret;
+
+	ring->irq = pci_irq_vector(ring->nhi->pdev, ring->vector);
+	if (ring->irq < 0)
+		return ring->irq;
+
+	irqflags = no_suspend ? IRQF_NO_SUSPEND : 0;
+	return request_irq(ring->irq, ring_msix, irqflags, "thunderbolt", ring);
+}
+
+static void ring_release_msix(struct tb_ring *ring)
+{
+	if (ring->irq <= 0)
+		return;
+
+	free_irq(ring->irq, ring);
+	ida_simple_remove(&ring->nhi->msix_ida, ring->vector);
+	ring->vector = 0;
+	ring->irq = 0;
+}
+
 static struct tb_ring *ring_alloc(struct tb_nhi *nhi, u32 hop, int size,
-				  bool transmit)
+				  bool transmit, unsigned int flags)
 {
 	struct tb_ring *ring = NULL;
 	dev_info(&nhi->pdev->dev, "allocating %s ring %d of size %d\n",
@@ -271,9 +351,14 @@ static struct tb_ring *ring_alloc(struct tb_nhi *nhi, u32 hop, int size,
 	ring->hop = hop;
 	ring->is_tx = transmit;
 	ring->size = size;
+	ring->flags = flags;
 	ring->head = 0;
 	ring->tail = 0;
 	ring->running = false;
+
+	if (ring_request_msix(ring, flags & RING_FLAG_NO_SUSPEND))
+		goto err;
+
 	ring->descriptors = dma_alloc_coherent(&ring->nhi->pdev->dev,
 			size * sizeof(*ring->descriptors),
 			&ring->descriptors_dma, GFP_KERNEL | __GFP_ZERO);
@@ -295,14 +380,16 @@ static struct tb_ring *ring_alloc(struct tb_nhi *nhi, u32 hop, int size,
 	return NULL;
 }
 
-struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size)
+struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size,
+			      unsigned int flags)
 {
-	return ring_alloc(nhi, hop, size, true);
+	return ring_alloc(nhi, hop, size, true, flags);
 }
 
-struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size)
+struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size,
+			      unsigned int flags)
 {
-	return ring_alloc(nhi, hop, size, false);
+	return ring_alloc(nhi, hop, size, false, flags);
 }
 
 /**
@@ -334,6 +421,7 @@ void ring_start(struct tb_ring *ring)
 		ring_iowrite32options(ring,
 				      RING_FLAG_ENABLE | RING_FLAG_RAW, 0);
 	}
+	ring_map_unmap_msix(ring, true);
 	ring_interrupt_active(ring, true);
 	ring->running = true;
 err:
@@ -366,6 +454,7 @@ void ring_stop(struct tb_ring *ring)
 		goto err;
 	}
 	ring_interrupt_active(ring, false);
+	ring_map_unmap_msix(ring, false);
 
 	ring_iowrite32options(ring, 0, 0);
 	ring_iowrite64desc(ring, 0, 0);
@@ -413,6 +502,8 @@ void ring_free(struct tb_ring *ring)
 			 RING_TYPE(ring), ring->hop);
 	}
 
+	ring_release_msix(ring);
+
 	dma_free_coherent(&ring->nhi->pdev->dev,
 			  ring->size * sizeof(*ring->descriptors),
 			  ring->descriptors, ring->descriptors_dma);
@@ -528,9 +619,51 @@ static void nhi_shutdown(struct tb_nhi *nhi)
 	 * We have to release the irq before calling flush_work. Otherwise an
 	 * already executing IRQ handler could call schedule_work again.
 	 */
-	devm_free_irq(&nhi->pdev->dev, nhi->pdev->irq, nhi);
+	if (!nhi->pdev->msix_enabled)
+		devm_free_irq(&nhi->pdev->dev, nhi->pdev->irq, nhi);
 	flush_work(&nhi->interrupt_work);
 	mutex_destroy(&nhi->lock);
+	ida_destroy(&nhi->msix_ida);
+}
+
+static int nhi_init_msi(struct tb_nhi *nhi)
+{
+	struct pci_dev *pdev = nhi->pdev;
+	int res, irq, nvec;
+
+	/* In case someone left them on. */
+	nhi_disable_interrupts(nhi);
+
+	ida_init(&nhi->msix_ida);
+
+	/*
+	 * The NHI has 16 MSI-X vectors or a single MSI. We first try to
+	 * get all MSI-X vectors and if we succeed, each ring will have
+	 * one MSI-X. If for some reason that does not work out, we
+	 * fallback to a single MSI.
+	 */
+	nvec = pci_alloc_irq_vectors(pdev, MSIX_MIN_VECS, MSIX_MAX_VECS,
+				     PCI_IRQ_MSIX);
+	if (nvec < 0) {
+		nvec = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
+		if (nvec < 0)
+			return nvec;
+
+		INIT_WORK(&nhi->interrupt_work, nhi_interrupt_work);
+
+		irq = pci_irq_vector(nhi->pdev, 0);
+		if (irq < 0)
+			return irq;
+
+		res = devm_request_irq(&pdev->dev, irq, nhi_msi,
+				       IRQF_NO_SUSPEND, "thunderbolt", nhi);
+		if (res) {
+			dev_err(&pdev->dev, "request_irq failed, aborting\n");
+			return res;
+		}
+	}
+
+	return 0;
 }
 
 static int nhi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
@@ -545,12 +678,6 @@ static int nhi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		return res;
 	}
 
-	res = pci_enable_msi(pdev);
-	if (res) {
-		dev_err(&pdev->dev, "cannot enable MSI, aborting\n");
-		return res;
-	}
-
 	res = pcim_iomap_regions(pdev, 1 << 0, "thunderbolt");
 	if (res) {
 		dev_err(&pdev->dev, "cannot obtain PCI resources, aborting\n");
@@ -568,7 +695,6 @@ static int nhi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (nhi->hop_count != 12 && nhi->hop_count != 32)
 		dev_warn(&pdev->dev, "unexpected hop count: %d\n",
 			 nhi->hop_count);
-	INIT_WORK(&nhi->interrupt_work, nhi_interrupt_work);
 
 	nhi->tx_rings = devm_kcalloc(&pdev->dev, nhi->hop_count,
 				     sizeof(*nhi->tx_rings), GFP_KERNEL);
@@ -577,12 +703,9 @@ static int nhi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (!nhi->tx_rings || !nhi->rx_rings)
 		return -ENOMEM;
 
-	nhi_disable_interrupts(nhi); /* In case someone left them on. */
-	res = devm_request_irq(&pdev->dev, pdev->irq, nhi_msi,
-			       IRQF_NO_SUSPEND, /* must work during _noirq */
-			       "thunderbolt", nhi);
+	res = nhi_init_msi(nhi);
 	if (res) {
-		dev_err(&pdev->dev, "request_irq failed, aborting\n");
+		dev_err(&pdev->dev, "cannot enable MSI, aborting\n");
 		return res;
 	}
 
diff --git a/drivers/thunderbolt/nhi.h b/drivers/thunderbolt/nhi.h
index 317242939b31..630f44140530 100644
--- a/drivers/thunderbolt/nhi.h
+++ b/drivers/thunderbolt/nhi.h
@@ -7,45 +7,75 @@
 #ifndef DSL3510_H_
 #define DSL3510_H_
 
+#include <linux/idr.h>
 #include <linux/mutex.h>
 #include <linux/workqueue.h>
 
 /**
  * struct tb_nhi - thunderbolt native host interface
+ * @lock: Must be held during ring creation/destruction. Is acquired by
+ *	  interrupt_work when dispatching interrupts to individual rings.
+ * @pdev: Pointer to the PCI device
+ * @iobase: MMIO space of the NHI
+ * @tx_rings: All Tx rings available on this host controller
+ * @rx_rings: All Rx rings available on this host controller
+ * @msix_ida: Used to allocate MSI-X vectors for rings
+ * @interrupt_work: Work scheduled to handle ring interrupt when no
+ *		    MSI-X is used.
+ * @hop_count: Number of rings (end point hops) supported by NHI.
  */
 struct tb_nhi {
-	struct mutex lock; /*
-			    * Must be held during ring creation/destruction.
-			    * Is acquired by interrupt_work when dispatching
-			    * interrupts to individual rings.
-			    **/
+	struct mutex lock;
 	struct pci_dev *pdev;
 	void __iomem *iobase;
 	struct tb_ring **tx_rings;
 	struct tb_ring **rx_rings;
+	struct ida msix_ida;
 	struct work_struct interrupt_work;
-	u32 hop_count; /* Number of rings (end point hops) supported by NHI. */
+	u32 hop_count;
 };
 
 /**
  * struct tb_ring - thunderbolt TX or RX ring associated with a NHI
+ * @lock: Lock serializing actions to this ring. Must be acquired after
+ *	  nhi->lock.
+ * @nhi: Pointer to the native host controller interface
+ * @size: Size of the ring
+ * @hop: Hop (DMA channel) associated with this ring
+ * @head: Head of the ring (write next descriptor here)
+ * @tail: Tail of the ring (complete next descriptor here)
+ * @descriptors: Allocated descriptors for this ring
+ * @queue: Queue holding frames to be transferred over this ring
+ * @in_flight: Queue holding frames that are currently in flight
+ * @work: Interrupt work structure
+ * @is_tx: Is the ring Tx or Rx
+ * @running: Is the ring running
+ * @irq: MSI-X irq number if the ring uses MSI-X. %0 otherwise.
+ * @vector: MSI-X vector number the ring uses (only set if @irq is > 0)
+ * @flags: Ring specific flags
  */
 struct tb_ring {
-	struct mutex lock; /* must be acquired after nhi->lock */
+	struct mutex lock;
 	struct tb_nhi *nhi;
 	int size;
 	int hop;
-	int head; /* write next descriptor here */
-	int tail; /* complete next descriptor here */
+	int head;
+	int tail;
 	struct ring_desc *descriptors;
 	dma_addr_t descriptors_dma;
 	struct list_head queue;
 	struct list_head in_flight;
 	struct work_struct work;
-	bool is_tx:1; /* rx otherwise */
+	bool is_tx:1;
 	bool running:1;
+	int irq;
+	u8 vector;
+	unsigned int flags;
 };
 
+/* Leave ring interrupt enabled on suspend */
+#define RING_FLAG_NO_SUSPEND	BIT(0)
+
 struct ring_frame;
 typedef void (*ring_cb)(struct tb_ring*, struct ring_frame*, bool canceled);
 
@@ -64,8 +94,10 @@ struct ring_frame {
 
 #define TB_FRAME_SIZE 0x100    /* minimum size for ring_rx */
 
-struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size);
-struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size);
+struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size,
+			      unsigned int flags);
+struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size,
+			      unsigned int flags);
 void ring_start(struct tb_ring *ring);
 void ring_stop(struct tb_ring *ring);
 void ring_free(struct tb_ring *ring);
diff --git a/drivers/thunderbolt/nhi_regs.h b/drivers/thunderbolt/nhi_regs.h
index 75cf0691e6c5..48b98d3c7e6a 100644
--- a/drivers/thunderbolt/nhi_regs.h
+++ b/drivers/thunderbolt/nhi_regs.h
@@ -95,7 +95,16 @@ struct ring_desc {
 #define REG_RING_INTERRUPT_BASE	0x38200
 #define RING_INTERRUPT_REG_COUNT(nhi) ((31 + 2 * nhi->hop_count) / 32)
 
+/* Interrupt Vector Allocation */
+#define REG_INT_VEC_ALLOC_BASE	0x38c40
+#define REG_INT_VEC_ALLOC_BITS	4
+#define REG_INT_VEC_ALLOC_MASK	GENMASK(3, 0)
+#define REG_INT_VEC_ALLOC_REGS	(32 / REG_INT_VEC_ALLOC_BITS)
+
 /* The last 11 bits contain the number of hops supported by the NHI port. */
 #define REG_HOP_COUNT		0x39640
 
+#define REG_DMA_MISC			0x39864
+#define REG_DMA_MISC_INT_AUTO_CLEAR     BIT(2)
+
 #endif
-- 
2.11.0

[PATCH 05/24] thunderbolt: Rework capability handling

[Mika Westerberg]

Organization of the capabilities in switches and ports is not so random
after all. Rework the capability handling functionality so that it
follows how capabilities are organized and provide two new functions
(tb_switch_find_vsec_cap() and tb_port_find_cap()) which can be used to
extract capabilities for ports and switches. Then convert the current
users over these.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 drivers/thunderbolt/cap.c        | 169 +++++++++++++++++++++------------------
 drivers/thunderbolt/switch.c     |   6 +-
 drivers/thunderbolt/tb.c         |   8 +-
 drivers/thunderbolt/tb.h         |   3 +-
 drivers/thunderbolt/tb_regs.h    |  31 ++++---
 drivers/thunderbolt/tunnel_pci.c |   8 +-
 6 files changed, 124 insertions(+), 101 deletions(-)

diff --git a/drivers/thunderbolt/cap.c b/drivers/thunderbolt/cap.c
index a7b47e7cddbd..0dd852c3df8d 100644
--- a/drivers/thunderbolt/cap.c
+++ b/drivers/thunderbolt/cap.c
@@ -9,6 +9,8 @@
 
 #include "tb.h"
 
+#define CAP_OFFSET_MAX		0xff
+#define VSEC_CAP_OFFSET_MAX	0xffff
 
 struct tb_cap_any {
 	union {
@@ -18,99 +20,110 @@ struct tb_cap_any {
 	};
 } __packed;
 
-static bool tb_cap_is_basic(struct tb_cap_any *cap)
-{
-	/* basic.cap is u8. This checks only the lower 8 bit of cap. */
-	return cap->basic.cap != 5;
-}
-
-static bool tb_cap_is_long(struct tb_cap_any *cap)
+/**
+ * tb_port_find_cap() - Find port capability
+ * @port: Port to find the capability for
+ * @cap: Capability to look
+ *
+ * Returns offset to start of capability or %-ENOENT if no such
+ * capability was found. Negative errno is returned if there was an
+ * error.
+ */
+int tb_port_find_cap(struct tb_port *port, enum tb_port_cap cap)
 {
-	return !tb_cap_is_basic(cap)
-	       && cap->extended_short.next == 0
-	       && cap->extended_short.length == 0;
-}
+	u32 offset;
 
-static enum tb_cap tb_cap(struct tb_cap_any *cap)
-{
-	if (tb_cap_is_basic(cap))
-		return cap->basic.cap;
+	/*
+	 * DP out adapters claim to implement TMU capability but in
+	 * reality they do not so we hard code the adapter specific
+	 * capability offset here.
+	 */
+	if (port->config.type == TB_TYPE_DP_HDMI_OUT)
+		offset = 0x39;
 	else
-		/* extended_short/long have cap at the same offset. */
-		return cap->extended_short.cap;
+		offset = 0x1;
+
+	do {
+		struct tb_cap_any header;
+		int ret;
+
+		ret = tb_port_read(port, &header, TB_CFG_PORT, offset, 1);
+		if (ret)
+			return ret;
+
+		if (header.basic.cap == cap)
+			return offset;
+
+		offset = header.basic.next;
+	} while (offset);
+
+	return -ENOENT;
 }
 
-static u32 tb_cap_next(struct tb_cap_any *cap, u32 offset)
+static int tb_switch_find_cap(struct tb_switch *sw, enum tb_switch_cap cap)
 {
-	int next;
-	if (offset == 1) {
-		/*
-		 * The first pointer is part of the switch header and always
-		 * a simple pointer.
-		 */
-		next = cap->basic.next;
-	} else {
-		/*
-		 * Somehow Intel decided to use 3 different types of capability
-		 * headers. It is not like anyone could have predicted that
-		 * single byte offsets are not enough...
-		 */
-		if (tb_cap_is_basic(cap))
-			next = cap->basic.next;
-		else if (!tb_cap_is_long(cap))
-			next = cap->extended_short.next;
-		else
-			next = cap->extended_long.next;
+	int offset = sw->config.first_cap_offset;
+
+	while (offset > 0 && offset < CAP_OFFSET_MAX) {
+		struct tb_cap_any header;
+		int ret;
+
+		ret = tb_sw_read(sw, &header, TB_CFG_SWITCH, offset, 1);
+		if (ret)
+			return ret;
+
+		if (header.basic.cap == cap)
+			return offset;
+
+		offset = header.basic.next;
 	}
-	/*
-	 * "Hey, we could terminate some capability lists with a null offset
-	 *  and others with a pointer to the last element." - "Great idea!"
-	 */
-	if (next == offset)
-		return 0;
-	return next;
+
+	return -ENOENT;
 }
 
 /**
- * tb_find_cap() - find a capability
+ * tb_switch_find_vsec_cap() - Find switch vendor specific capability
+ * @sw: Switch to find the capability for
+ * @vsec: Vendor specific capability to look
  *
- * Return: Returns a positive offset if the capability was found and 0 if not.
- * Returns an error code on failure.
+ * Functions enumerates vendor specific (VSEC) capabilities of a switch
+ * and returns offset when capability matching @vsed is found. If no
+ * such capability is found returns %-ENOENT. In case of error returns
+ * negative errno.
  */
-int tb_find_cap(struct tb_port *port, enum tb_cfg_space space, enum tb_cap cap)
+int tb_switch_find_vsec_cap(struct tb_switch *sw, enum tb_switch_vsec_cap vsec)
 {
-	u32 offset = 1;
 	struct tb_cap_any header;
-	int res;
-	int retries = 10;
-	while (retries--) {
-		res = tb_port_read(port, &header, space, offset, 1);
-		if (res) {
-			/* Intel needs some help with linked lists. */
-			if (space == TB_CFG_PORT && offset == 0xa
-			    && port->config.type == TB_TYPE_DP_HDMI_OUT) {
-				offset = 0x39;
-				continue;
-			}
-			return res;
-		}
-		if (offset != 1) {
-			if (tb_cap(&header) == cap)
+	int offset;
+
+	offset = tb_switch_find_cap(sw, TB_SWITCH_CAP_VSEC);
+	if (offset < 0)
+		return offset;
+
+	while (offset > 0 && offset < VSEC_CAP_OFFSET_MAX) {
+		int ret;
+
+		ret = tb_sw_read(sw, &header, TB_CFG_SWITCH, offset, 2);
+		if (ret)
+			return ret;
+
+		/*
+		 * Extended vendor specific capabilities come in two
+		 * flavors: short and long. The latter is used when
+		 * offset is over 0xff.
+		 */
+		if (offset >= CAP_OFFSET_MAX) {
+			if (header.extended_long.vsec_id == vsec)
 				return offset;
-			if (tb_cap_is_long(&header)) {
-				/* tb_cap_extended_long is 2 dwords */
-				res = tb_port_read(port, &header, space,
-						   offset, 2);
-				if (res)
-					return res;
-			}
+			offset = header.extended_long.next;
+		} else {
+			if (header.extended_short.vsec_id == vsec)
+				return offset;
+			if (!header.extended_short.length)
+				return -ENOENT;
+			offset = header.extended_short.next;
 		}
-		offset = tb_cap_next(&header, offset);
-		if (!offset)
-			return 0;
 	}
-	tb_port_WARN(port,
-		     "run out of retries while looking for cap %#x in config space %d, last offset: %#x\n",
-		     cap, space, offset);
-	return -EIO;
+
+	return -ENOENT;
 }
diff --git a/drivers/thunderbolt/switch.c b/drivers/thunderbolt/switch.c
index c6f30b1695a9..5bae49d72d2c 100644
--- a/drivers/thunderbolt/switch.c
+++ b/drivers/thunderbolt/switch.c
@@ -192,7 +192,7 @@ static int tb_init_port(struct tb_port *port)
 
 	/* Port 0 is the switch itself and has no PHY. */
 	if (port->config.type == TB_TYPE_PORT && port->port != 0) {
-		cap = tb_find_cap(port, TB_CFG_PORT, TB_CAP_PHY);
+		cap = tb_port_find_cap(port, TB_PORT_CAP_PHY);
 
 		if (cap > 0)
 			port->cap_phy = cap;
@@ -394,9 +394,9 @@ struct tb_switch *tb_switch_alloc(struct tb *tb, u64 route)
 		sw->ports[i].port = i;
 	}
 
-	cap = tb_find_cap(&sw->ports[0], TB_CFG_SWITCH, TB_CAP_PLUG_EVENTS);
+	cap = tb_switch_find_vsec_cap(sw, TB_VSEC_CAP_PLUG_EVENTS);
 	if (cap < 0) {
-		tb_sw_warn(sw, "cannot find TB_CAP_PLUG_EVENTS aborting\n");
+		tb_sw_warn(sw, "cannot find TB_VSEC_CAP_PLUG_EVENTS aborting\n");
 		goto err;
 	}
 	sw->cap_plug_events = cap;
diff --git a/drivers/thunderbolt/tb.c b/drivers/thunderbolt/tb.c
index 24b6d30c3c86..6b44076e1380 100644
--- a/drivers/thunderbolt/tb.c
+++ b/drivers/thunderbolt/tb.c
@@ -121,8 +121,8 @@ static struct tb_port *tb_find_unused_down_port(struct tb_switch *sw)
 			continue;
 		if (sw->ports[i].config.type != TB_TYPE_PCIE_DOWN)
 			continue;
-		cap = tb_find_cap(&sw->ports[i], TB_CFG_PORT, TB_CAP_PCIE);
-		if (cap <= 0)
+		cap = tb_port_find_cap(&sw->ports[i], TB_PORT_CAP_ADAP);
+		if (cap < 0)
 			continue;
 		res = tb_port_read(&sw->ports[i], &data, TB_CFG_PORT, cap, 1);
 		if (res < 0)
@@ -165,8 +165,8 @@ static void tb_activate_pcie_devices(struct tb *tb)
 		}
 
 		/* check whether port is already activated */
-		cap = tb_find_cap(up_port, TB_CFG_PORT, TB_CAP_PCIE);
-		if (cap <= 0)
+		cap = tb_port_find_cap(up_port, TB_PORT_CAP_ADAP);
+		if (cap < 0)
 			continue;
 		if (tb_port_read(up_port, &data, TB_CFG_PORT, cap, 1))
 			continue;
diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
index ba2b85750335..9b60e71562dc 100644
--- a/drivers/thunderbolt/tb.h
+++ b/drivers/thunderbolt/tb.h
@@ -233,7 +233,8 @@ int tb_wait_for_port(struct tb_port *port, bool wait_if_unplugged);
 int tb_port_add_nfc_credits(struct tb_port *port, int credits);
 int tb_port_clear_counter(struct tb_port *port, int counter);
 
-int tb_find_cap(struct tb_port *port, enum tb_cfg_space space, enum tb_cap cap);
+int tb_switch_find_vsec_cap(struct tb_switch *sw, enum tb_switch_vsec_cap vsec);
+int tb_port_find_cap(struct tb_port *port, enum tb_port_cap cap);
 
 struct tb_path *tb_path_alloc(struct tb *tb, int num_hops);
 void tb_path_free(struct tb_path *path);
diff --git a/drivers/thunderbolt/tb_regs.h b/drivers/thunderbolt/tb_regs.h
index 1e2a4a8046be..57fdfa61e136 100644
--- a/drivers/thunderbolt/tb_regs.h
+++ b/drivers/thunderbolt/tb_regs.h
@@ -23,15 +23,22 @@
  */
 #define TB_MAX_CONFIG_RW_LENGTH 60
 
-enum tb_cap {
-	TB_CAP_PHY		= 0x0001,
-	TB_CAP_TIME1		= 0x0003,
-	TB_CAP_PCIE		= 0x0004,
-	TB_CAP_I2C		= 0x0005,
-	TB_CAP_PLUG_EVENTS	= 0x0105, /* also EEPROM */
-	TB_CAP_TIME2		= 0x0305,
-	TB_CAP_IECS		= 0x0405,
-	TB_CAP_LINK_CONTROLLER	= 0x0605, /* also IECS */
+enum tb_switch_cap {
+	TB_SWITCH_CAP_VSEC		= 0x05,
+};
+
+enum tb_switch_vsec_cap {
+	TB_VSEC_CAP_PLUG_EVENTS		= 0x01, /* also EEPROM */
+	TB_VSEC_CAP_TIME2		= 0x03,
+	TB_VSEC_CAP_IECS		= 0x04,
+	TB_VSEC_CAP_LINK_CONTROLLER	= 0x06, /* also IECS */
+};
+
+enum tb_port_cap {
+	TB_PORT_CAP_PHY			= 0x01,
+	TB_PORT_CAP_TIME1		= 0x03,
+	TB_PORT_CAP_ADAP		= 0x04,
+	TB_PORT_CAP_VSEC		= 0x05,
 };
 
 enum tb_port_state {
@@ -51,13 +58,15 @@ struct tb_cap_basic {
 
 struct tb_cap_extended_short {
 	u8 next; /* if next and length are zero then we have a long cap */
-	enum tb_cap cap:16;
+	u8 cap;
+	u8 vsec_id;
 	u8 length;
 } __packed;
 
 struct tb_cap_extended_long {
 	u8 zero1;
-	enum tb_cap cap:16;
+	u8 cap;
+	u8 vsec_id;
 	u8 zero2;
 	u16 next;
 	u16 length;
diff --git a/drivers/thunderbolt/tunnel_pci.c b/drivers/thunderbolt/tunnel_pci.c
index baf1cd370446..f4ce9845e42a 100644
--- a/drivers/thunderbolt/tunnel_pci.c
+++ b/drivers/thunderbolt/tunnel_pci.c
@@ -147,10 +147,10 @@ bool tb_pci_is_invalid(struct tb_pci_tunnel *tunnel)
 static int tb_pci_port_active(struct tb_port *port, bool active)
 {
 	u32 word = active ? 0x80000000 : 0x0;
-	int cap = tb_find_cap(port, TB_CFG_PORT, TB_CAP_PCIE);
-	if (cap <= 0) {
-		tb_port_warn(port, "TB_CAP_PCIE not found: %d\n", cap);
-		return cap ? cap : -ENXIO;
+	int cap = tb_port_find_cap(port, TB_PORT_CAP_ADAP);
+	if (cap < 0) {
+		tb_port_warn(port, "TB_PORT_CAP_ADAP not found: %d\n", cap);
+		return cap;
 	}
 	return tb_port_write(port, &word, TB_CFG_PORT, cap, 1);
 }
-- 
2.11.0

[PATCH 06/24] thunderbolt: Introduce thunderbolt bus and connection manager

[Mika Westerberg]

Thunderbolt fabric consists of one or more switches. This fabric is
called domain and it is controlled by an entity called connection
manager. The connection manager can be either internal (driven by a
firmware running on the host controller) or external (software driver).
This driver currently implements support for the latter.

In order to manage switches and their properties more easily we model
this domain structure as a Linux bus. Each host controller adds a domain
device to this bus, and these devices are named as domainN where N
stands for index or id of the current domain.

We then abstract connection manager specific operations into a new
structure tb_cm_ops and convert the existing tb.c to fill those
accordingly. This makes it easier to add support for the internal
connection manager in subsequent patches.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 drivers/thunderbolt/Makefile     |   2 +-
 drivers/thunderbolt/domain.c     | 230 +++++++++++++++++++++++++++++++++++++++
 drivers/thunderbolt/nhi.c        |  31 ++++--
 drivers/thunderbolt/tb.c         | 156 ++++++++++++--------------
 drivers/thunderbolt/tb.h         |  70 +++++++++---
 drivers/thunderbolt/tunnel_pci.c |   9 +-
 6 files changed, 377 insertions(+), 121 deletions(-)
 create mode 100644 drivers/thunderbolt/domain.c

diff --git a/drivers/thunderbolt/Makefile b/drivers/thunderbolt/Makefile
index 5d1053cdfa54..e276a9a62261 100644
--- a/drivers/thunderbolt/Makefile
+++ b/drivers/thunderbolt/Makefile
@@ -1,3 +1,3 @@
 obj-${CONFIG_THUNDERBOLT} := thunderbolt.o
 thunderbolt-objs := nhi.o ctl.o tb.o switch.o cap.o path.o tunnel_pci.o eeprom.o
-
+thunderbolt-objs += domain.o
diff --git a/drivers/thunderbolt/domain.c b/drivers/thunderbolt/domain.c
new file mode 100644
index 000000000000..b5bfca95415d
--- /dev/null
+++ b/drivers/thunderbolt/domain.c
@@ -0,0 +1,230 @@
+/*
+ * Thunderbolt bus support
+ *
+ * Copyright (C) 2017, Intel Corporation
+ * Author:  Mika Westerberg <mika.westerberg@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/device.h>
+#include <linux/idr.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+
+#include "tb.h"
+
+static DEFINE_IDA(tb_domain_ida);
+
+struct bus_type tb_bus_type = {
+	.name = "thunderbolt",
+};
+
+static void tb_domain_release(struct device *dev)
+{
+	struct tb *tb = container_of(dev, struct tb, dev);
+
+	if (tb->ctl)
+		tb_ctl_free(tb->ctl);
+
+	destroy_workqueue(tb->wq);
+	ida_simple_remove(&tb_domain_ida, tb->index);
+	kfree(tb);
+}
+
+struct device_type tb_domain_type = {
+	.name = "thunderbolt_domain",
+	.release = tb_domain_release,
+};
+
+/**
+ * tb_domain_alloc() - Allocate a domain
+ * @nhi: Pointer to the host controller
+ * @privsize: Size of the connection manager private data
+ *
+ * Allocates and initializes a new Thunderbolt domain. Connection
+ * managers are expected to call this and then fill in @cm_ops
+ * accordingly.
+ *
+ * Call tb_domain_put() to release the domain before it has been added
+ * to the system.
+ *
+ * Return: allocated domain structure on %NULL in case of error
+ */
+struct tb *tb_domain_alloc(struct tb_nhi *nhi, size_t privsize)
+{
+	struct tb *tb;
+
+	/*
+	 * Make sure the structure sizes map with that the hardware
+	 * expects because bit-fields are being used.
+	 */
+	BUILD_BUG_ON(sizeof(struct tb_regs_switch_header) != 5 * 4);
+	BUILD_BUG_ON(sizeof(struct tb_regs_port_header) != 8 * 4);
+	BUILD_BUG_ON(sizeof(struct tb_regs_hop) != 2 * 4);
+
+	tb = kzalloc(sizeof(*tb) + privsize, GFP_KERNEL);
+	if (!tb)
+		return NULL;
+
+	tb->nhi = nhi;
+	mutex_init(&tb->lock);
+
+	tb->index = ida_simple_get(&tb_domain_ida, 0, 0, GFP_KERNEL);
+	if (tb->index < 0)
+		goto err_free;
+
+	tb->wq = alloc_ordered_workqueue("thunderbolt%d", 0, tb->index);
+	if (!tb->wq)
+		goto err_remove_ida;
+
+	tb->dev.parent = &nhi->pdev->dev;
+	tb->dev.bus = &tb_bus_type;
+	tb->dev.type = &tb_domain_type;
+	dev_set_name(&tb->dev, "domain%d", tb->index);
+	device_initialize(&tb->dev);
+
+	return tb;
+
+err_remove_ida:
+	ida_simple_remove(&tb_domain_ida, tb->index);
+err_free:
+	kfree(tb);
+
+	return NULL;
+}
+
+/**
+ * tb_domain_add() - Add domain to the system
+ * @tb: Domain to add
+ *
+ * Starts the domain and adds it to the system. Hotplugging devices will
+ * work after this has been returned successfully. In order to remove
+ * and release the domain after this function has been called, call
+ * tb_domain_remove().
+ *
+ * Return: %0 in case of success and negative errno in case of error
+ */
+int tb_domain_add(struct tb *tb)
+{
+	int ret;
+
+	if (WARN_ON(!tb->cm_ops))
+		return -EINVAL;
+
+	mutex_lock(&tb->lock);
+
+	tb->ctl = tb_ctl_alloc(tb->nhi, tb->cm_ops->hotplug, tb);
+	if (!tb->ctl) {
+		ret = -ENOMEM;
+		goto err_unlock;
+	}
+
+	/*
+	 * tb_schedule_hotplug_handler may be called as soon as the config
+	 * channel is started. Thats why we have to hold the lock here.
+	 */
+	tb_ctl_start(tb->ctl);
+
+	ret = device_add(&tb->dev);
+	if (ret)
+		goto err_ctl_stop;
+
+	/* Start the domain */
+	if (tb->cm_ops->start) {
+		ret = tb->cm_ops->start(tb);
+		if (ret)
+			goto err_domain_del;
+	}
+
+	/* This starts event processing */
+	mutex_unlock(&tb->lock);
+
+	return 0;
+
+err_domain_del:
+	device_del(&tb->dev);
+err_ctl_stop:
+	tb_ctl_stop(tb->ctl);
+err_unlock:
+	mutex_unlock(&tb->lock);
+
+	return ret;
+}
+
+/**
+ * tb_domain_remove() - Removes and releases a domain
+ * @tb: Domain to remove
+ *
+ * Stops the domain, removes it from the system and releases all
+ * resources once the last reference has been released.
+ */
+void tb_domain_remove(struct tb *tb)
+{
+	mutex_lock(&tb->lock);
+	if (tb->cm_ops->stop)
+		tb->cm_ops->stop(tb);
+	/* Stop the domain control traffic */
+	tb_ctl_stop(tb->ctl);
+	mutex_unlock(&tb->lock);
+
+	flush_workqueue(tb->wq);
+	device_unregister(&tb->dev);
+}
+
+/**
+ * tb_domain_suspend_noirq() - Suspend a domain
+ * @tb: Domain to suspend
+ *
+ * Suspends all devices in the domain and stops the control channel.
+ */
+int tb_domain_suspend_noirq(struct tb *tb)
+{
+	int ret = 0;
+
+	/*
+	 * The control channel interrupt is left enabled during suspend
+	 * and taking the lock here prevents any events happening before
+	 * we actually have stopped the domain and the control channel.
+	 */
+	mutex_lock(&tb->lock);
+	if (tb->cm_ops->suspend_noirq)
+		ret = tb->cm_ops->suspend_noirq(tb);
+	if (!ret)
+		tb_ctl_stop(tb->ctl);
+	mutex_unlock(&tb->lock);
+
+	return ret;
+}
+
+/**
+ * tb_domain_resume_noirq() - Resume a domain
+ * @tb: Domain to resume
+ *
+ * Re-starts the control channel, and resumes all devices connected to
+ * the domain.
+ */
+int tb_domain_resume_noirq(struct tb *tb)
+{
+	int ret = 0;
+
+	mutex_lock(&tb->lock);
+	tb_ctl_start(tb->ctl);
+	if (tb->cm_ops->resume_noirq)
+		ret = tb->cm_ops->resume_noirq(tb);
+	mutex_unlock(&tb->lock);
+
+	return ret;
+}
+
+int tb_domain_init(void)
+{
+	return bus_register(&tb_bus_type);
+}
+
+void tb_domain_exit(void)
+{
+	bus_unregister(&tb_bus_type);
+}
diff --git a/drivers/thunderbolt/nhi.c b/drivers/thunderbolt/nhi.c
index 17f3b1bdb7da..bf69ee42da68 100644
--- a/drivers/thunderbolt/nhi.c
+++ b/drivers/thunderbolt/nhi.c
@@ -589,16 +589,16 @@ static int nhi_suspend_noirq(struct device *dev)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
 	struct tb *tb = pci_get_drvdata(pdev);
-	thunderbolt_suspend(tb);
-	return 0;
+
+	return tb_domain_suspend_noirq(tb);
 }
 
 static int nhi_resume_noirq(struct device *dev)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
 	struct tb *tb = pci_get_drvdata(pdev);
-	thunderbolt_resume(tb);
-	return 0;
+
+	return tb_domain_resume_noirq(tb);
 }
 
 static void nhi_shutdown(struct tb_nhi *nhi)
@@ -717,12 +717,17 @@ static int nhi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	iowrite32(3906250 / 10000, nhi->iobase + 0x38c00);
 
 	dev_info(&nhi->pdev->dev, "NHI initialized, starting thunderbolt\n");
-	tb = thunderbolt_alloc_and_start(nhi);
-	if (!tb) {
+	tb = tb_probe(nhi);
+	if (!tb)
+		return -ENODEV;
+
+	res = tb_domain_add(tb);
+	if (res) {
 		/*
 		 * At this point the RX/TX rings might already have been
 		 * activated. Do a proper shutdown.
 		 */
+		tb_domain_put(tb);
 		nhi_shutdown(nhi);
 		return -EIO;
 	}
@@ -735,7 +740,8 @@ static void nhi_remove(struct pci_dev *pdev)
 {
 	struct tb *tb = pci_get_drvdata(pdev);
 	struct tb_nhi *nhi = tb->nhi;
-	thunderbolt_shutdown_and_free(tb);
+
+	tb_domain_remove(tb);
 	nhi_shutdown(nhi);
 }
 
@@ -799,14 +805,23 @@ static struct pci_driver nhi_driver = {
 
 static int __init nhi_init(void)
 {
+	int ret;
+
 	if (!dmi_match(DMI_BOARD_VENDOR, "Apple Inc."))
 		return -ENOSYS;
-	return pci_register_driver(&nhi_driver);
+	ret = tb_domain_init();
+	if (ret)
+		return ret;
+	ret = pci_register_driver(&nhi_driver);
+	if (ret)
+		tb_domain_exit();
+	return ret;
 }
 
 static void __exit nhi_unload(void)
 {
 	pci_unregister_driver(&nhi_driver);
+	tb_domain_exit();
 }
 
 module_init(nhi_init);
diff --git a/drivers/thunderbolt/tb.c b/drivers/thunderbolt/tb.c
index 6b44076e1380..9f00a0f28d53 100644
--- a/drivers/thunderbolt/tb.c
+++ b/drivers/thunderbolt/tb.c
@@ -12,6 +12,18 @@
 #include "tb_regs.h"
 #include "tunnel_pci.h"
 
+/**
+ * struct tb_cm - Simple Thunderbolt connection manager
+ * @tunnel_list: List of active tunnels
+ * @hotplug_active: tb_handle_hotplug will stop progressing plug
+ *		    events and exit if this is not set (it needs to
+ *		    acquire the lock one more time). Used to drain wq
+ *		    after cfg has been paused.
+ */
+struct tb_cm {
+	struct list_head tunnel_list;
+	bool hotplug_active;
+};
 
 /* enumeration & hot plug handling */
 
@@ -62,12 +74,14 @@ static void tb_scan_port(struct tb_port *port)
  */
 static void tb_free_invalid_tunnels(struct tb *tb)
 {
+	struct tb_cm *tcm = tb_priv(tb);
 	struct tb_pci_tunnel *tunnel;
 	struct tb_pci_tunnel *n;
-	list_for_each_entry_safe(tunnel, n, &tb->tunnel_list, list)
-	{
+
+	list_for_each_entry_safe(tunnel, n, &tcm->tunnel_list, list) {
 		if (tb_pci_is_invalid(tunnel)) {
 			tb_pci_deactivate(tunnel);
+			list_del(&tunnel->list);
 			tb_pci_free(tunnel);
 		}
 	}
@@ -149,6 +163,8 @@ static void tb_activate_pcie_devices(struct tb *tb)
 	struct tb_port *up_port;
 	struct tb_port *down_port;
 	struct tb_pci_tunnel *tunnel;
+	struct tb_cm *tcm = tb_priv(tb);
+
 	/* scan for pcie devices at depth 1*/
 	for (i = 1; i <= tb->root_switch->config.max_port_number; i++) {
 		if (tb_is_upstream_port(&tb->root_switch->ports[i]))
@@ -195,6 +211,7 @@ static void tb_activate_pcie_devices(struct tb *tb)
 			tb_pci_free(tunnel);
 		}
 
+		list_add(&tunnel->list, &tcm->tunnel_list);
 	}
 }
 
@@ -217,10 +234,11 @@ static void tb_handle_hotplug(struct work_struct *work)
 {
 	struct tb_hotplug_event *ev = container_of(work, typeof(*ev), work);
 	struct tb *tb = ev->tb;
+	struct tb_cm *tcm = tb_priv(tb);
 	struct tb_switch *sw;
 	struct tb_port *port;
 	mutex_lock(&tb->lock);
-	if (!tb->hotplug_active)
+	if (!tcm->hotplug_active)
 		goto out; /* during init, suspend or shutdown */
 
 	sw = get_switch_at_route(tb->root_switch, ev->route);
@@ -296,22 +314,14 @@ static void tb_schedule_hotplug_handler(void *data, u64 route, u8 port,
 	queue_work(tb->wq, &ev->work);
 }
 
-/**
- * thunderbolt_shutdown_and_free() - shutdown everything
- *
- * Free all switches and the config channel.
- *
- * Used in the error path of thunderbolt_alloc_and_start.
- */
-void thunderbolt_shutdown_and_free(struct tb *tb)
+static void tb_stop(struct tb *tb)
 {
+	struct tb_cm *tcm = tb_priv(tb);
 	struct tb_pci_tunnel *tunnel;
 	struct tb_pci_tunnel *n;
 
-	mutex_lock(&tb->lock);
-
 	/* tunnels are only present after everything has been initialized */
-	list_for_each_entry_safe(tunnel, n, &tb->tunnel_list, list) {
+	list_for_each_entry_safe(tunnel, n, &tcm->tunnel_list, list) {
 		tb_pci_deactivate(tunnel);
 		tb_pci_free(tunnel);
 	}
@@ -320,98 +330,44 @@ void thunderbolt_shutdown_and_free(struct tb *tb)
 		tb_switch_free(tb->root_switch);
 	tb->root_switch = NULL;
 
-	if (tb->ctl) {
-		tb_ctl_stop(tb->ctl);
-		tb_ctl_free(tb->ctl);
-	}
-	tb->ctl = NULL;
-	tb->hotplug_active = false; /* signal tb_handle_hotplug to quit */
-
-	/* allow tb_handle_hotplug to acquire the lock */
-	mutex_unlock(&tb->lock);
-	if (tb->wq) {
-		flush_workqueue(tb->wq);
-		destroy_workqueue(tb->wq);
-		tb->wq = NULL;
-	}
-	mutex_destroy(&tb->lock);
-	kfree(tb);
+	tcm->hotplug_active = false; /* signal tb_handle_hotplug to quit */
 }
 
-/**
- * thunderbolt_alloc_and_start() - setup the thunderbolt bus
- *
- * Allocates a tb_cfg control channel, initializes the root switch, enables
- * plug events and activates pci devices.
- *
- * Return: Returns NULL on error.
- */
-struct tb *thunderbolt_alloc_and_start(struct tb_nhi *nhi)
+static int tb_start(struct tb *tb)
 {
-	struct tb *tb;
-
-	BUILD_BUG_ON(sizeof(struct tb_regs_switch_header) != 5 * 4);
-	BUILD_BUG_ON(sizeof(struct tb_regs_port_header) != 8 * 4);
-	BUILD_BUG_ON(sizeof(struct tb_regs_hop) != 2 * 4);
-
-	tb = kzalloc(sizeof(*tb), GFP_KERNEL);
-	if (!tb)
-		return NULL;
-
-	tb->nhi = nhi;
-	mutex_init(&tb->lock);
-	mutex_lock(&tb->lock);
-	INIT_LIST_HEAD(&tb->tunnel_list);
-
-	tb->wq = alloc_ordered_workqueue("thunderbolt", 0);
-	if (!tb->wq)
-		goto err_locked;
-
-	tb->ctl = tb_ctl_alloc(tb->nhi, tb_schedule_hotplug_handler, tb);
-	if (!tb->ctl)
-		goto err_locked;
-	/*
-	 * tb_schedule_hotplug_handler may be called as soon as the config
-	 * channel is started. Thats why we have to hold the lock here.
-	 */
-	tb_ctl_start(tb->ctl);
+	struct tb_cm *tcm = tb_priv(tb);
 
 	tb->root_switch = tb_switch_alloc(tb, 0);
 	if (!tb->root_switch)
-		goto err_locked;
+		return -ENOMEM;
 
 	/* Full scan to discover devices added before the driver was loaded. */
 	tb_scan_switch(tb->root_switch);
 	tb_activate_pcie_devices(tb);
 
 	/* Allow tb_handle_hotplug to progress events */
-	tb->hotplug_active = true;
-	mutex_unlock(&tb->lock);
-	return tb;
-
-err_locked:
-	mutex_unlock(&tb->lock);
-	thunderbolt_shutdown_and_free(tb);
-	return NULL;
+	tcm->hotplug_active = true;
+	return 0;
 }
 
-void thunderbolt_suspend(struct tb *tb)
+static int tb_suspend_noirq(struct tb *tb)
 {
+	struct tb_cm *tcm = tb_priv(tb);
+
 	tb_info(tb, "suspending...\n");
-	mutex_lock(&tb->lock);
 	tb_switch_suspend(tb->root_switch);
-	tb_ctl_stop(tb->ctl);
-	tb->hotplug_active = false; /* signal tb_handle_hotplug to quit */
-	mutex_unlock(&tb->lock);
+	tcm->hotplug_active = false; /* signal tb_handle_hotplug to quit */
 	tb_info(tb, "suspend finished\n");
+
+	return 0;
 }
 
-void thunderbolt_resume(struct tb *tb)
+static int tb_resume_noirq(struct tb *tb)
 {
+	struct tb_cm *tcm = tb_priv(tb);
 	struct tb_pci_tunnel *tunnel, *n;
+
 	tb_info(tb, "resuming...\n");
-	mutex_lock(&tb->lock);
-	tb_ctl_start(tb->ctl);
 
 	/* remove any pci devices the firmware might have setup */
 	tb_switch_reset(tb, 0);
@@ -419,9 +375,9 @@ void thunderbolt_resume(struct tb *tb)
 	tb_switch_resume(tb->root_switch);
 	tb_free_invalid_tunnels(tb);
 	tb_free_unplugged_children(tb->root_switch);
-	list_for_each_entry_safe(tunnel, n, &tb->tunnel_list, list)
+	list_for_each_entry_safe(tunnel, n, &tcm->tunnel_list, list)
 		tb_pci_restart(tunnel);
-	if (!list_empty(&tb->tunnel_list)) {
+	if (!list_empty(&tcm->tunnel_list)) {
 		/*
 		 * the pcie links need some time to get going.
 		 * 100ms works for me...
@@ -430,7 +386,33 @@ void thunderbolt_resume(struct tb *tb)
 		msleep(100);
 	}
 	 /* Allow tb_handle_hotplug to progress events */
-	tb->hotplug_active = true;
-	mutex_unlock(&tb->lock);
+	tcm->hotplug_active = true;
 	tb_info(tb, "resume finished\n");
+
+	return 0;
+}
+
+static const struct tb_cm_ops tb_cm_ops = {
+	.start = tb_start,
+	.stop = tb_stop,
+	.suspend_noirq = tb_suspend_noirq,
+	.resume_noirq = tb_resume_noirq,
+	.hotplug = tb_schedule_hotplug_handler,
+};
+
+struct tb *tb_probe(struct tb_nhi *nhi)
+{
+	struct tb_cm *tcm;
+	struct tb *tb;
+
+	tb = tb_domain_alloc(nhi, sizeof(*tcm));
+	if (!tb)
+		return NULL;
+
+	tb->cm_ops = &tb_cm_ops;
+
+	tcm = tb_priv(tb);
+	INIT_LIST_HEAD(&tcm->tunnel_list);
+
+	return tb;
 }
diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
index 9b60e71562dc..ce9db64e3333 100644
--- a/drivers/thunderbolt/tb.h
+++ b/drivers/thunderbolt/tb.h
@@ -92,29 +92,52 @@ struct tb_path {
 	int path_length; /* number of hops */
 };
 
+/**
+ * struct tb_cm_ops - Connection manager specific operations vector
+ * @start: Starts the domain
+ * @stop: Stops the domain
+ * @suspend_noirq: Connection manager specific suspend_noirq
+ * @resume_noirq: Connection manager specific resume_noirq
+ * @hotplug: Handle hotplug event
+ */
+struct tb_cm_ops {
+	int (*start)(struct tb *tb);
+	void (*stop)(struct tb *tb);
+	int (*suspend_noirq)(struct tb *tb);
+	int (*resume_noirq)(struct tb *tb);
+	hotplug_cb hotplug;
+};
 
 /**
  * struct tb - main thunderbolt bus structure
+ * @dev: Domain device
+ * @lock: Big lock. Must be held when accessing cfg or any struct
+ *	  tb_switch / struct tb_port.
+ * @nhi: Pointer to the NHI structure
+ * @ctl: Control channel for this domain
+ * @wq: Ordered workqueue for all domain specific work
+ * @root_switch: Root switch of this domain
+ * @cm_ops: Connection manager specific operations vector
+ * @index: Linux assigned domain number
+ * @privdata: Private connection manager specific data
  */
 struct tb {
-	struct mutex lock;	/*
-				 * Big lock. Must be held when accessing cfg or
-				 * any struct tb_switch / struct tb_port.
-				 */
+	struct device dev;
+	struct mutex lock;
 	struct tb_nhi *nhi;
 	struct tb_ctl *ctl;
-	struct workqueue_struct *wq; /* ordered workqueue for plug events */
+	struct workqueue_struct *wq;
 	struct tb_switch *root_switch;
-	struct list_head tunnel_list; /* list of active PCIe tunnels */
-	bool hotplug_active; /*
-			      * tb_handle_hotplug will stop progressing plug
-			      * events and exit if this is not set (it needs to
-			      * acquire the lock one more time). Used to drain
-			      * wq after cfg has been paused.
-			      */
-
+	const struct tb_cm_ops *cm_ops;
+	int index;
+	unsigned long privdata[0];
 };
 
+static inline void *tb_priv(struct tb *tb)
+{
+	return (void *)tb->privdata;
+}
+
 /* helper functions & macros */
 
 /**
@@ -215,11 +238,24 @@ static inline int tb_port_write(struct tb_port *port, const void *buffer,
 #define tb_port_info(port, fmt, arg...) \
 	__TB_PORT_PRINT(tb_info, port, fmt, ##arg)
 
+struct tb *tb_probe(struct tb_nhi *nhi);
+
+extern struct bus_type tb_bus_type;
+extern struct device_type tb_domain_type;
+
+int tb_domain_init(void);
+void tb_domain_exit(void);
 
-struct tb *thunderbolt_alloc_and_start(struct tb_nhi *nhi);
-void thunderbolt_shutdown_and_free(struct tb *tb);
-void thunderbolt_suspend(struct tb *tb);
-void thunderbolt_resume(struct tb *tb);
+struct tb *tb_domain_alloc(struct tb_nhi *nhi, size_t privsize);
+int tb_domain_add(struct tb *tb);
+void tb_domain_remove(struct tb *tb);
+int tb_domain_suspend_noirq(struct tb *tb);
+int tb_domain_resume_noirq(struct tb *tb);
+
+static inline void tb_domain_put(struct tb *tb)
+{
+	put_device(&tb->dev);
+}
 
 struct tb_switch *tb_switch_alloc(struct tb *tb, u64 route);
 void tb_switch_free(struct tb_switch *sw);
diff --git a/drivers/thunderbolt/tunnel_pci.c b/drivers/thunderbolt/tunnel_pci.c
index f4ce9845e42a..ca4475907d7a 100644
--- a/drivers/thunderbolt/tunnel_pci.c
+++ b/drivers/thunderbolt/tunnel_pci.c
@@ -194,19 +194,13 @@ int tb_pci_restart(struct tb_pci_tunnel *tunnel)
  */
 int tb_pci_activate(struct tb_pci_tunnel *tunnel)
 {
-	int res;
 	if (tunnel->path_to_up->activated || tunnel->path_to_down->activated) {
 		tb_tunnel_WARN(tunnel,
 			       "trying to activate an already activated tunnel\n");
 		return -EINVAL;
 	}
 
-	res = tb_pci_restart(tunnel);
-	if (res)
-		return res;
-
-	list_add(&tunnel->list, &tunnel->tb->tunnel_list);
-	return 0;
+	return tb_pci_restart(tunnel);
 }
 
 
@@ -227,6 +221,5 @@ void tb_pci_deactivate(struct tb_pci_tunnel *tunnel)
 		tb_path_deactivate(tunnel->path_to_down);
 	if (tunnel->path_to_up->activated)
 		tb_path_deactivate(tunnel->path_to_up);
-	list_del_init(&tunnel->list);
 }
 
-- 
2.11.0

[PATCH 07/24] thunderbolt: Convert switch to a device

[Mika Westerberg]

Thunderbolt domain consists of switches that are connected to each
other, forming a bus. This will convert each switch into a real Linux
device structure and adds them to the domain. The advantage here is
that we get all the goodies from the driver core, like reference
counting and sysfs hierarchy for free.

Also expose device identification information to the userspace via new
sysfs attributes.

In order to support internal connection manager (ICM) we separate switch
configuration into its own function (tb_switch_configure()) which is
only called by the existing native connection manager implementation
used on Macs.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 Documentation/ABI/testing/sysfs-bus-thunderbolt |  20 ++
 drivers/thunderbolt/eeprom.c                    |   2 +
 drivers/thunderbolt/switch.c                    | 242 +++++++++++++++++++-----
 drivers/thunderbolt/tb.c                        |  40 +++-
 drivers/thunderbolt/tb.h                        |  45 ++++-
 5 files changed, 285 insertions(+), 64 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-bus-thunderbolt

diff --git a/Documentation/ABI/testing/sysfs-bus-thunderbolt b/Documentation/ABI/testing/sysfs-bus-thunderbolt
new file mode 100644
index 000000000000..a3dac3becd1e
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-bus-thunderbolt
@@ -0,0 +1,20 @@
+What:		/sys/bus/thunderbolt/devices/.../device
+Date:		Sep 2017
+KernelVersion:	4.13
+Contact:	thunderbolt-software@lists.01.org
+Description:	This attribute contains id of this device extracted from
+		the device DROM.
+
+What:		/sys/bus/thunderbolt/devices/.../vendor
+Date:		Sep 2017
+KernelVersion:	4.13
+Contact:	thunderbolt-software@lists.01.org
+Description:	This attribute contains vendor id of this device extracted
+		from the device DROM.
+
+What:		/sys/bus/thunderbolt/devices/.../unique_id
+Date:		Sep 2017
+KernelVersion:	4.13
+Contact:	thunderbolt-software@lists.01.org
+Description:	This attribute contains unique id (UUID) of this device
+		extracted from the device DROM or hardware registers.
diff --git a/drivers/thunderbolt/eeprom.c b/drivers/thunderbolt/eeprom.c
index eb2179c98b09..7e485e3ef27e 100644
--- a/drivers/thunderbolt/eeprom.c
+++ b/drivers/thunderbolt/eeprom.c
@@ -479,6 +479,8 @@ int tb_drom_read(struct tb_switch *sw)
 		goto err;
 	}
 	sw->uid = header->uid;
+	sw->vendor = header->vendor_id;
+	sw->device = header->model_id;
 
 	crc = tb_crc32(sw->drom + TB_DROM_DATA_START, header->data_len);
 	if (crc != header->data_crc32) {
diff --git a/drivers/thunderbolt/switch.c b/drivers/thunderbolt/switch.c
index 5bae49d72d2c..a03548763180 100644
--- a/drivers/thunderbolt/switch.c
+++ b/drivers/thunderbolt/switch.c
@@ -281,6 +281,9 @@ static int tb_plug_events_active(struct tb_switch *sw, bool active)
 	u32 data;
 	int res;
 
+	if (!sw->config.enabled)
+		return 0;
+
 	sw->config.plug_events_delay = 0xff;
 	res = tb_sw_write(sw, ((u32 *) &sw->config) + 4, TB_CFG_SWITCH, 4, 1);
 	if (res)
@@ -307,36 +310,71 @@ static int tb_plug_events_active(struct tb_switch *sw, bool active)
 			   sw->cap_plug_events + 1, 1);
 }
 
+static ssize_t device_show(struct device *dev, struct device_attribute *attr,
+			   char *buf)
+{
+	struct tb_switch *sw = tb_to_switch(dev);
 
-/**
- * tb_switch_free() - free a tb_switch and all downstream switches
- */
-void tb_switch_free(struct tb_switch *sw)
+	return sprintf(buf, "%#x\n", sw->device);
+}
+static DEVICE_ATTR_RO(device);
+
+static ssize_t vendor_show(struct device *dev, struct device_attribute *attr,
+			   char *buf)
 {
-	int i;
-	/* port 0 is the switch itself and never has a remote */
-	for (i = 1; i <= sw->config.max_port_number; i++) {
-		if (tb_is_upstream_port(&sw->ports[i]))
-			continue;
-		if (sw->ports[i].remote)
-			tb_switch_free(sw->ports[i].remote->sw);
-		sw->ports[i].remote = NULL;
-	}
+	struct tb_switch *sw = tb_to_switch(dev);
 
-	if (!sw->is_unplugged)
-		tb_plug_events_active(sw, false);
+	return sprintf(buf, "%#x\n", sw->vendor);
+}
+static DEVICE_ATTR_RO(vendor);
+
+static ssize_t unique_id_show(struct device *dev, struct device_attribute *attr,
+			      char *buf)
+{
+	struct tb_switch *sw = tb_to_switch(dev);
 
+	return sprintf(buf, "%pUb\n", sw->uuid);
+}
+static DEVICE_ATTR_RO(unique_id);
+
+static struct attribute *switch_attrs[] = {
+	&dev_attr_device.attr,
+	&dev_attr_vendor.attr,
+	&dev_attr_unique_id.attr,
+	NULL,
+};
+ATTRIBUTE_GROUPS(switch);
+
+static void tb_switch_release(struct device *dev)
+{
+	struct tb_switch *sw = tb_to_switch(dev);
+
+	kfree(sw->uuid);
 	kfree(sw->ports);
 	kfree(sw->drom);
 	kfree(sw);
 }
 
+struct device_type tb_switch_type = {
+	.name = "thunderbolt_device",
+	.release = tb_switch_release,
+};
+
 /**
- * tb_switch_alloc() - allocate and initialize a switch
+ * tb_switch_alloc() - allocate a switch
+ * @tb: Pointer to the owning domain
+ * @parent: Parent device for this switch
+ * @route: Route string for this switch
  *
- * Return: Returns a NULL on failure.
+ * Allocates and initializes a switch. Will not upload configuration to
+ * the switch. For that you need to call tb_switch_configure()
+ * separately. The returned switch should be released by calling
+ * tb_switch_put().
+ *
+ * Return: Pointer to the allocated switch or %NULL in case of failure
  */
-struct tb_switch *tb_switch_alloc(struct tb *tb, u64 route)
+struct tb_switch *tb_switch_alloc(struct tb *tb, struct device *parent,
+				  u64 route)
 {
 	int i;
 	int cap;
@@ -352,10 +390,8 @@ struct tb_switch *tb_switch_alloc(struct tb *tb, u64 route)
 	sw->tb = tb;
 	if (tb_cfg_read(tb->ctl, &sw->config, route, 0, TB_CFG_SWITCH, 0, 5))
 		goto err;
-	tb_info(tb,
-		"initializing Switch at %#llx (depth: %d, up port: %d)\n",
-		route, tb_route_length(route), upstream_port);
-	tb_info(tb, "old switch config:\n");
+
+	tb_info(tb, "current switch config:\n");
 	tb_dump_switch(tb, &sw->config);
 
 	/* configure switch */
@@ -363,24 +399,7 @@ struct tb_switch *tb_switch_alloc(struct tb *tb, u64 route)
 	sw->config.depth = tb_route_length(route);
 	sw->config.route_lo = route;
 	sw->config.route_hi = route >> 32;
-	sw->config.enabled = 1;
-	/* from here on we may use the tb_sw_* functions & macros */
-
-	if (sw->config.vendor_id != 0x8086)
-		tb_sw_warn(sw, "unknown switch vendor id %#x\n",
-			   sw->config.vendor_id);
-
-	if (sw->config.device_id != PCI_DEVICE_ID_INTEL_LIGHT_RIDGE &&
-	    sw->config.device_id != PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C &&
-	    sw->config.device_id != PCI_DEVICE_ID_INTEL_PORT_RIDGE &&
-	    sw->config.device_id != PCI_DEVICE_ID_INTEL_FALCON_RIDGE_2C_BRIDGE &&
-	    sw->config.device_id != PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_BRIDGE)
-		tb_sw_warn(sw, "unsupported switch device id %#x\n",
-			   sw->config.device_id);
-
-	/* upload configuration */
-	if (tb_sw_write(sw, 1 + (u32 *) &sw->config, TB_CFG_SWITCH, 1, 3))
-		goto err;
+	sw->config.enabled = 0;
 
 	/* initialize ports */
 	sw->ports = kcalloc(sw->config.max_port_number + 1, sizeof(*sw->ports),
@@ -401,31 +420,152 @@ struct tb_switch *tb_switch_alloc(struct tb *tb, u64 route)
 	}
 	sw->cap_plug_events = cap;
 
+	device_initialize(&sw->dev);
+	sw->dev.parent = parent;
+	sw->dev.bus = &tb_bus_type;
+	sw->dev.type = &tb_switch_type;
+	sw->dev.groups = switch_groups;
+	dev_set_name(&sw->dev, "%u-%llx", tb->index, tb_route(sw));
+
+	return sw;
+err:
+	kfree(sw->ports);
+	kfree(sw);
+	return NULL;
+}
+
+/**
+ * tb_switch_configure() - Uploads configuration to the switch
+ * @sw: Switch to configure
+ *
+ * Call this function before the switch is added to the system. It will
+ * upload configuration to the switch and makes it available for the
+ * connection manager to use.
+ *
+ * Return: %0 in case of success and negative errno in case of failure
+ */
+int tb_switch_configure(struct tb_switch *sw)
+{
+	struct tb *tb = sw->tb;
+	u64 route;
+	int ret;
+
+	route = tb_route(sw);
+	tb_info(tb,
+		"initializing Switch at %#llx (depth: %d, up port: %d)\n",
+		route, tb_route_length(route), sw->config.upstream_port_number);
+
+	if (sw->config.vendor_id != PCI_VENDOR_ID_INTEL)
+		tb_sw_warn(sw, "unknown switch vendor id %#x\n",
+			   sw->config.vendor_id);
+
+	if (sw->config.device_id != PCI_DEVICE_ID_INTEL_LIGHT_RIDGE &&
+	    sw->config.device_id != PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C &&
+	    sw->config.device_id != PCI_DEVICE_ID_INTEL_PORT_RIDGE &&
+	    sw->config.device_id != PCI_DEVICE_ID_INTEL_FALCON_RIDGE_2C_BRIDGE &&
+	    sw->config.device_id != PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_BRIDGE)
+		tb_sw_warn(sw, "unsupported switch device id %#x\n",
+			   sw->config.device_id);
+
+	sw->config.enabled = 1;
+
+	/* upload configuration */
+	ret = tb_sw_write(sw, 1 + (u32 *) &sw->config, TB_CFG_SWITCH, 1, 3);
+	if (ret)
+		return ret;
+
+	return tb_plug_events_active(sw, true);
+}
+
+static void tb_switch_set_uuid(struct tb_switch *sw)
+{
+	u32 uuid[4];
+	int cap;
+
+	if (sw->uuid)
+		return;
+
+	/*
+	 * By default the UUID will be based on UID where upper two
+	 * dwords are filled with ones.
+	 */
+	uuid[0] = sw->uid & 0xffffffff;
+	uuid[1] = (sw->uid >> 32) & 0xffffffff;
+	uuid[2] = 0xffffffff;
+	uuid[3] = 0xffffffff;
+
+	/*
+	 * The newer controllers include fused UUID as part of link
+	 * controller specific registers
+	 */
+	cap = tb_switch_find_vsec_cap(sw, TB_VSEC_CAP_LINK_CONTROLLER);
+	if (cap > 0)
+		tb_sw_read(sw, uuid, TB_CFG_SWITCH, cap + 3, 4);
+
+	sw->uuid = kmemdup(uuid, sizeof(uuid), GFP_KERNEL);
+}
+
+/**
+ * tb_switch_add() - Add a switch to the domain
+ * @sw: Switch to add
+ *
+ * This is the last step in adding switch to the domain. It will read
+ * identification information from DROM and initializes ports so that
+ * they can be used to connect other switches. The switch will be
+ * exposed to the userspace when this function successfully returns. To
+ * remove and release the switch, call tb_switch_remove().
+ *
+ * Return: %0 in case of success and negative errno in case of failure
+ */
+int tb_switch_add(struct tb_switch *sw)
+{
+	int i, ret;
+
 	/* read drom */
 	if (tb_drom_read(sw))
 		tb_sw_warn(sw, "tb_eeprom_read_rom failed, continuing\n");
 	tb_sw_info(sw, "uid: %#llx\n", sw->uid);
 
+	tb_switch_set_uuid(sw);
+
 	for (i = 0; i <= sw->config.max_port_number; i++) {
 		if (sw->ports[i].disabled) {
 			tb_port_info(&sw->ports[i], "disabled by eeprom\n");
 			continue;
 		}
-		if (tb_init_port(&sw->ports[i]))
-			goto err;
+		ret = tb_init_port(&sw->ports[i]);
+		if (ret)
+			return ret;
 	}
 
-	/* TODO: I2C, IECS, link controller */
+	return device_add(&sw->dev);
+}
 
-	if (tb_plug_events_active(sw, true))
-		goto err;
+/**
+ * tb_switch_remove() - Remove and release a switch
+ * @sw: Switch to remove
+ *
+ * This will remove the switch from the domain and release it after last
+ * reference count drops to zero. If there are switches connected below
+ * this switch, they will be removed as well.
+ */
+void tb_switch_remove(struct tb_switch *sw)
+{
+	int i;
 
-	return sw;
-err:
-	kfree(sw->ports);
-	kfree(sw->drom);
-	kfree(sw);
-	return NULL;
+	/* port 0 is the switch itself and never has a remote */
+	for (i = 1; i <= sw->config.max_port_number; i++) {
+		if (tb_is_upstream_port(&sw->ports[i]))
+			continue;
+		if (sw->ports[i].remote)
+			tb_switch_remove(sw->ports[i].remote->sw);
+		sw->ports[i].remote = NULL;
+	}
+
+	if (!sw->is_unplugged)
+		tb_plug_events_active(sw, false);
+
+	device_unregister(&sw->dev);
 }
 
 /**
diff --git a/drivers/thunderbolt/tb.c b/drivers/thunderbolt/tb.c
index 9f00a0f28d53..94ecac012428 100644
--- a/drivers/thunderbolt/tb.c
+++ b/drivers/thunderbolt/tb.c
@@ -61,9 +61,21 @@ static void tb_scan_port(struct tb_port *port)
 		tb_port_WARN(port, "port already has a remote!\n");
 		return;
 	}
-	sw = tb_switch_alloc(port->sw->tb, tb_downstream_route(port));
+	sw = tb_switch_alloc(port->sw->tb, &port->sw->dev,
+			     tb_downstream_route(port));
 	if (!sw)
 		return;
+
+	if (tb_switch_configure(sw)) {
+		tb_switch_put(sw);
+		return;
+	}
+
+	if (tb_switch_add(sw)) {
+		tb_switch_put(sw);
+		return;
+	}
+
 	port->remote = tb_upstream_port(sw);
 	tb_upstream_port(sw)->remote = port;
 	tb_scan_switch(sw);
@@ -100,7 +112,7 @@ static void tb_free_unplugged_children(struct tb_switch *sw)
 		if (!port->remote)
 			continue;
 		if (port->remote->sw->is_unplugged) {
-			tb_switch_free(port->remote->sw);
+			tb_switch_remove(port->remote->sw);
 			port->remote = NULL;
 		} else {
 			tb_free_unplugged_children(port->remote->sw);
@@ -266,7 +278,7 @@ static void tb_handle_hotplug(struct work_struct *work)
 			tb_port_info(port, "unplugged\n");
 			tb_sw_set_unplugged(port->remote->sw);
 			tb_free_invalid_tunnels(tb);
-			tb_switch_free(port->remote->sw);
+			tb_switch_remove(port->remote->sw);
 			port->remote = NULL;
 		} else {
 			tb_port_info(port,
@@ -325,22 +337,32 @@ static void tb_stop(struct tb *tb)
 		tb_pci_deactivate(tunnel);
 		tb_pci_free(tunnel);
 	}
-
-	if (tb->root_switch)
-		tb_switch_free(tb->root_switch);
-	tb->root_switch = NULL;
-
+	tb_switch_remove(tb->root_switch);
 	tcm->hotplug_active = false; /* signal tb_handle_hotplug to quit */
 }
 
 static int tb_start(struct tb *tb)
 {
 	struct tb_cm *tcm = tb_priv(tb);
+	int ret;
 
-	tb->root_switch = tb_switch_alloc(tb, 0);
+	tb->root_switch = tb_switch_alloc(tb, &tb->dev, 0);
 	if (!tb->root_switch)
 		return -ENOMEM;
 
+	ret = tb_switch_configure(tb->root_switch);
+	if (ret) {
+		tb_switch_put(tb->root_switch);
+		return ret;
+	}
+
+	/* Announce the switch to the world */
+	ret = tb_switch_add(tb->root_switch);
+	if (ret) {
+		tb_switch_put(tb->root_switch);
+		return ret;
+	}
+
 	/* Full scan to discover devices added before the driver was loaded. */
 	tb_scan_switch(tb->root_switch);
 	tb_activate_pcie_devices(tb);
diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
index ce9db64e3333..350c3f21924e 100644
--- a/drivers/thunderbolt/tb.h
+++ b/drivers/thunderbolt/tb.h
@@ -8,20 +8,36 @@
 #define TB_H_
 
 #include <linux/pci.h>
+#include <linux/uuid.h>
 
 #include "tb_regs.h"
 #include "ctl.h"
 
 /**
  * struct tb_switch - a thunderbolt switch
+ * @dev: Device for the switch
+ * @config: Switch configuration
+ * @ports: Ports in this switch
+ * @tb: Pointer to the domain the switch belongs to
+ * @uid: Unique ID of the switch
+ * @uuid: UUID of the switch (or %NULL if not supported)
+ * @vendor: Vendor ID of the switch
+ * @device: Device ID of the switch
+ * @cap_plug_events: Offset to the plug events capability (%0 if not found)
+ * @is_unplugged: The switch is going away
+ * @drom: DROM of the switch (%NULL if not found)
  */
 struct tb_switch {
+	struct device dev;
 	struct tb_regs_switch_header config;
 	struct tb_port *ports;
 	struct tb *tb;
 	u64 uid;
-	int cap_plug_events; /* offset, zero if not found */
-	bool is_unplugged; /* unplugged, will go away */
+	uuid_be *uuid;
+	u16 vendor;
+	u16 device;
+	int cap_plug_events;
+	bool is_unplugged;
 	u8 *drom;
 };
 
@@ -242,6 +258,7 @@ struct tb *tb_probe(struct tb_nhi *nhi);
 
 extern struct bus_type tb_bus_type;
 extern struct device_type tb_domain_type;
+extern struct device_type tb_switch_type;
 
 int tb_domain_init(void);
 void tb_domain_exit(void);
@@ -257,14 +274,34 @@ static inline void tb_domain_put(struct tb *tb)
 	put_device(&tb->dev);
 }
 
-struct tb_switch *tb_switch_alloc(struct tb *tb, u64 route);
-void tb_switch_free(struct tb_switch *sw);
+struct tb_switch *tb_switch_alloc(struct tb *tb, struct device *parent,
+				  u64 route);
+int tb_switch_configure(struct tb_switch *sw);
+int tb_switch_add(struct tb_switch *sw);
+void tb_switch_remove(struct tb_switch *sw);
 void tb_switch_suspend(struct tb_switch *sw);
 int tb_switch_resume(struct tb_switch *sw);
 int tb_switch_reset(struct tb *tb, u64 route);
 void tb_sw_set_unplugged(struct tb_switch *sw);
 struct tb_switch *get_switch_at_route(struct tb_switch *sw, u64 route);
 
+static inline void tb_switch_put(struct tb_switch *sw)
+{
+	put_device(&sw->dev);
+}
+
+static inline bool tb_is_switch(const struct device *dev)
+{
+	return dev->type == &tb_switch_type;
+}
+
+static inline struct tb_switch *tb_to_switch(struct device *dev)
+{
+	if (tb_is_switch(dev))
+		return container_of(dev, struct tb_switch, dev);
+	return NULL;
+}
+
 int tb_wait_for_port(struct tb_port *port, bool wait_if_unplugged);
 int tb_port_add_nfc_credits(struct tb_port *port, int credits);
 int tb_port_clear_counter(struct tb_port *port, int counter);
-- 
2.11.0

[PATCH 08/24] thunderbolt: Fail switch adding operation if reading DROM fails

[Mika Westerberg]

All non-root switches are expected to have DROM so if the operation
fails, it might be due the user unlugging the device. There is no point
continuing adding the switch further in that case. Just bail out.

For root switches (hosts) the DROM is either retrieved from a EFI
variable, NVM or hard-coded.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 drivers/thunderbolt/switch.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/thunderbolt/switch.c b/drivers/thunderbolt/switch.c
index a03548763180..4a961d174cad 100644
--- a/drivers/thunderbolt/switch.c
+++ b/drivers/thunderbolt/switch.c
@@ -522,8 +522,11 @@ int tb_switch_add(struct tb_switch *sw)
 	int i, ret;
 
 	/* read drom */
-	if (tb_drom_read(sw))
-		tb_sw_warn(sw, "tb_eeprom_read_rom failed, continuing\n");
+	ret = tb_drom_read(sw);
+	if (ret) {
+		tb_sw_warn(sw, "tb_eeprom_read_rom failed\n");
+		return ret;
+	}
 	tb_sw_info(sw, "uid: %#llx\n", sw->uid);
 
 	tb_switch_set_uuid(sw);
-- 
2.11.0

[PATCH 09/24] thunderbolt: Do not fail if DROM data CRC32 is invalid

[Mika Westerberg]

There are devices out there where CRC32 of the DROM is not correct. One
reason for this is that the ICM firmware does not validate it and it
seems that neither does the Apple driver. To be able to support such
devices we continue parsing the DROM contents regardless of whether
CRC32 failed or not. We still keep the warning there.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 drivers/thunderbolt/eeprom.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/thunderbolt/eeprom.c b/drivers/thunderbolt/eeprom.c
index 7e485e3ef27e..e2c1f8a45522 100644
--- a/drivers/thunderbolt/eeprom.c
+++ b/drivers/thunderbolt/eeprom.c
@@ -485,9 +485,8 @@ int tb_drom_read(struct tb_switch *sw)
 	crc = tb_crc32(sw->drom + TB_DROM_DATA_START, header->data_len);
 	if (crc != header->data_crc32) {
 		tb_sw_warn(sw,
-			"drom data crc32 mismatch (expected: %#x, got: %#x), aborting\n",
+			"drom data crc32 mismatch (expected: %#x, got: %#x), continuing\n",
 			header->data_crc32, crc);
-		goto err;
 	}
 
 	if (header->device_rom_revision > 2)
-- 
2.11.0

[PATCH 10/24] thunderbolt: Read vendor and device name from DROM

[Mika Westerberg]

The device DROM contains name of the vendor and device among other
things. Extract this information and expose it to the userspace via two
new attributes.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 Documentation/ABI/testing/sysfs-bus-thunderbolt | 14 ++++++++++++++
 drivers/thunderbolt/eeprom.c                    | 23 ++++++++++++++++++++++-
 drivers/thunderbolt/switch.c                    | 22 ++++++++++++++++++++++
 drivers/thunderbolt/tb.h                        |  4 ++++
 4 files changed, 62 insertions(+), 1 deletion(-)

diff --git a/Documentation/ABI/testing/sysfs-bus-thunderbolt b/Documentation/ABI/testing/sysfs-bus-thunderbolt
index a3dac3becd1e..2f352c787431 100644
--- a/Documentation/ABI/testing/sysfs-bus-thunderbolt
+++ b/Documentation/ABI/testing/sysfs-bus-thunderbolt
@@ -5,6 +5,13 @@ Contact:	thunderbolt-software@lists.01.org
 Description:	This attribute contains id of this device extracted from
 		the device DROM.
 
+What:		/sys/bus/thunderbolt/devices/.../device_name
+Date:		Sep 2017
+KernelVersion:	4.13
+Contact:	thunderbolt-software@lists.01.org
+Description:	This attribute contains name of this device extracted from
+		the device DROM.
+
 What:		/sys/bus/thunderbolt/devices/.../vendor
 Date:		Sep 2017
 KernelVersion:	4.13
@@ -12,6 +19,13 @@ Contact:	thunderbolt-software@lists.01.org
 Description:	This attribute contains vendor id of this device extracted
 		from the device DROM.
 
+What:		/sys/bus/thunderbolt/devices/.../vendor_name
+Date:		Sep 2017
+KernelVersion:	4.13
+Contact:	thunderbolt-software@lists.01.org
+Description:	This attribute contains vendor name of this device extracted
+		from the device DROM.
+
 What:		/sys/bus/thunderbolt/devices/.../unique_id
 Date:		Sep 2017
 KernelVersion:	4.13
diff --git a/drivers/thunderbolt/eeprom.c b/drivers/thunderbolt/eeprom.c
index e2c1f8a45522..f688fb255042 100644
--- a/drivers/thunderbolt/eeprom.c
+++ b/drivers/thunderbolt/eeprom.c
@@ -204,6 +204,11 @@ struct tb_drom_entry_header {
 	enum tb_drom_entry_type type:1;
 } __packed;
 
+struct tb_drom_entry_generic {
+	struct tb_drom_entry_header header;
+	u8 data[0];
+} __packed;
+
 struct tb_drom_entry_port {
 	/* BYTES 0-1 */
 	struct tb_drom_entry_header header;
@@ -304,6 +309,15 @@ static void tb_drom_parse_port_entry(struct tb_port *port,
 				&port->sw->ports[entry->dual_link_port_nr];
 }
 
+static void tb_drom_parse_generic_entry(struct tb_switch *sw,
+		struct tb_drom_entry_generic *entry)
+{
+	if (entry->header.index == 1)
+		sw->vendor_name = kstrdup((char *)entry->data, GFP_KERNEL);
+	else if (entry->header.index == 2)
+		sw->device_name = kstrdup((char *)entry->data, GFP_KERNEL);
+}
+
 static int tb_drom_parse_entry(struct tb_switch *sw,
 		struct tb_drom_entry_header *header)
 {
@@ -311,8 +325,15 @@ static int tb_drom_parse_entry(struct tb_switch *sw,
 	int res;
 	enum tb_port_type type;
 
-	if (header->type != TB_DROM_ENTRY_PORT)
+	switch (header->type) {
+	case TB_DROM_ENTRY_PORT:
+		break;
+	case TB_DROM_ENTRY_GENERIC:
+		tb_drom_parse_generic_entry(sw,
+			(struct tb_drom_entry_generic *)header);
+	default:
 		return 0;
+	}
 
 	port = &sw->ports[header->index];
 	port->disabled = header->port_disabled;
diff --git a/drivers/thunderbolt/switch.c b/drivers/thunderbolt/switch.c
index 4a961d174cad..b06de0efbdfc 100644
--- a/drivers/thunderbolt/switch.c
+++ b/drivers/thunderbolt/switch.c
@@ -319,6 +319,15 @@ static ssize_t device_show(struct device *dev, struct device_attribute *attr,
 }
 static DEVICE_ATTR_RO(device);
 
+static ssize_t
+device_name_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	struct tb_switch *sw = tb_to_switch(dev);
+
+	return sprintf(buf, "%s\n", sw->device_name ? sw->device_name : "");
+}
+static DEVICE_ATTR_RO(device_name);
+
 static ssize_t vendor_show(struct device *dev, struct device_attribute *attr,
 			   char *buf)
 {
@@ -328,6 +337,15 @@ static ssize_t vendor_show(struct device *dev, struct device_attribute *attr,
 }
 static DEVICE_ATTR_RO(vendor);
 
+static ssize_t
+vendor_name_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	struct tb_switch *sw = tb_to_switch(dev);
+
+	return sprintf(buf, "%s\n", sw->vendor_name ? sw->vendor_name : "");
+}
+static DEVICE_ATTR_RO(vendor_name);
+
 static ssize_t unique_id_show(struct device *dev, struct device_attribute *attr,
 			      char *buf)
 {
@@ -339,7 +357,9 @@ static DEVICE_ATTR_RO(unique_id);
 
 static struct attribute *switch_attrs[] = {
 	&dev_attr_device.attr,
+	&dev_attr_device_name.attr,
 	&dev_attr_vendor.attr,
+	&dev_attr_vendor_name.attr,
 	&dev_attr_unique_id.attr,
 	NULL,
 };
@@ -350,6 +370,8 @@ static void tb_switch_release(struct device *dev)
 	struct tb_switch *sw = tb_to_switch(dev);
 
 	kfree(sw->uuid);
+	kfree(sw->device_name);
+	kfree(sw->vendor_name);
 	kfree(sw->ports);
 	kfree(sw->drom);
 	kfree(sw);
diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
index 350c3f21924e..5e66dce53c65 100644
--- a/drivers/thunderbolt/tb.h
+++ b/drivers/thunderbolt/tb.h
@@ -23,6 +23,8 @@
  * @uuid: UUID of the switch (or %NULL if not supported)
  * @vendor: Vendor ID of the switch
  * @device: Device ID of the switch
+ * @vendor_name: Name of the vendor (or %NULL if not known)
+ * @device_name: Name of the device (or %NULL if not known)
  * @cap_plug_events: Offset to the plug events capability (%0 if not found)
  * @is_unplugged: The switch is going away
  * @drom: DROM of the switch (%NULL if not found)
@@ -36,6 +38,8 @@ struct tb_switch {
 	uuid_be *uuid;
 	u16 vendor;
 	u16 device;
+	const char *vendor_name;
+	const char *device_name;
 	int cap_plug_events;
 	bool is_unplugged;
 	u8 *drom;
-- 
2.11.0

[PATCH 11/24] thunderbolt: Move control channel messages to tb_msgs.h

[Mika Westerberg]

We will be forwarding notifications received from the control channel to
the connection manager implementations. This way they can decide what to
do if anything when a notification is received.

To be able to use control channel messages from other files, move them
to tb_msgs.h.

No functional changes intended.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 drivers/thunderbolt/ctl.c     |  76 -----------------------------
 drivers/thunderbolt/ctl.h     |  16 +------
 drivers/thunderbolt/tb_msgs.h | 108 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 109 insertions(+), 91 deletions(-)
 create mode 100644 drivers/thunderbolt/tb_msgs.h

diff --git a/drivers/thunderbolt/ctl.c b/drivers/thunderbolt/ctl.c
index 889a32dd21e7..08bdb7145d89 100644
--- a/drivers/thunderbolt/ctl.c
+++ b/drivers/thunderbolt/ctl.c
@@ -52,82 +52,6 @@ struct tb_ctl {
 #define tb_ctl_info(ctl, format, arg...) \
 	dev_info(&(ctl)->nhi->pdev->dev, format, ## arg)
 
-
-/* configuration packets definitions */
-
-enum tb_cfg_pkg_type {
-	TB_CFG_PKG_READ = 1,
-	TB_CFG_PKG_WRITE = 2,
-	TB_CFG_PKG_ERROR = 3,
-	TB_CFG_PKG_NOTIFY_ACK = 4,
-	TB_CFG_PKG_EVENT = 5,
-	TB_CFG_PKG_XDOMAIN_REQ = 6,
-	TB_CFG_PKG_XDOMAIN_RESP = 7,
-	TB_CFG_PKG_OVERRIDE = 8,
-	TB_CFG_PKG_RESET = 9,
-	TB_CFG_PKG_PREPARE_TO_SLEEP = 0xd,
-};
-
-/* common header */
-struct tb_cfg_header {
-	u32 route_hi:22;
-	u32 unknown:10; /* highest order bit is set on replies */
-	u32 route_lo;
-} __packed;
-
-/* additional header for read/write packets */
-struct tb_cfg_address {
-	u32 offset:13; /* in dwords */
-	u32 length:6; /* in dwords */
-	u32 port:6;
-	enum tb_cfg_space space:2;
-	u32 seq:2; /* sequence number  */
-	u32 zero:3;
-} __packed;
-
-/* TB_CFG_PKG_READ, response for TB_CFG_PKG_WRITE */
-struct cfg_read_pkg {
-	struct tb_cfg_header header;
-	struct tb_cfg_address addr;
-} __packed;
-
-/* TB_CFG_PKG_WRITE, response for TB_CFG_PKG_READ */
-struct cfg_write_pkg {
-	struct tb_cfg_header header;
-	struct tb_cfg_address addr;
-	u32 data[64]; /* maximum size, tb_cfg_address.length has 6 bits */
-} __packed;
-
-/* TB_CFG_PKG_ERROR */
-struct cfg_error_pkg {
-	struct tb_cfg_header header;
-	enum tb_cfg_error error:4;
-	u32 zero1:4;
-	u32 port:6;
-	u32 zero2:2; /* Both should be zero, still they are different fields. */
-	u32 zero3:16;
-} __packed;
-
-/* TB_CFG_PKG_EVENT */
-struct cfg_event_pkg {
-	struct tb_cfg_header header;
-	u32 port:6;
-	u32 zero:25;
-	bool unplug:1;
-} __packed;
-
-/* TB_CFG_PKG_RESET */
-struct cfg_reset_pkg {
-	struct tb_cfg_header header;
-} __packed;
-
-/* TB_CFG_PKG_PREPARE_TO_SLEEP */
-struct cfg_pts_pkg {
-	struct tb_cfg_header header;
-	u32 data;
-} __packed;
-
-
 /* utility functions */
 
 static u64 get_route(struct tb_cfg_header header)
diff --git a/drivers/thunderbolt/ctl.h b/drivers/thunderbolt/ctl.h
index 83ae54947082..610980e3232f 100644
--- a/drivers/thunderbolt/ctl.h
+++ b/drivers/thunderbolt/ctl.h
@@ -8,6 +8,7 @@
 #define _TB_CFG
 
 #include "nhi.h"
+#include "tb_msgs.h"
 
 /* control channel */
 struct tb_ctl;
@@ -23,21 +24,6 @@ void tb_ctl_free(struct tb_ctl *ctl);
 
 #define TB_CFG_DEFAULT_TIMEOUT 5000 /* msec */
 
-enum tb_cfg_space {
-	TB_CFG_HOPS = 0,
-	TB_CFG_PORT = 1,
-	TB_CFG_SWITCH = 2,
-	TB_CFG_COUNTERS = 3,
-};
-
-enum tb_cfg_error {
-	TB_CFG_ERROR_PORT_NOT_CONNECTED = 0,
-	TB_CFG_ERROR_INVALID_CONFIG_SPACE = 2,
-	TB_CFG_ERROR_NO_SUCH_PORT = 4,
-	TB_CFG_ERROR_ACK_PLUG_EVENT = 7, /* send as reply to TB_CFG_PKG_EVENT */
-	TB_CFG_ERROR_LOOP = 8,
-};
-
 struct tb_cfg_result {
 	u64 response_route;
 	u32 response_port; /*
diff --git a/drivers/thunderbolt/tb_msgs.h b/drivers/thunderbolt/tb_msgs.h
new file mode 100644
index 000000000000..761d56287149
--- /dev/null
+++ b/drivers/thunderbolt/tb_msgs.h
@@ -0,0 +1,108 @@
+/*
+ * Thunderbolt control channel messages
+ *
+ * Copyright (C) 2014 Andreas Noever <andreas.noever@gmail.com>
+ * Copyright (C) 2017, Intel Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef _TB_MSGS
+#define _TB_MSGS
+
+#include <linux/types.h>
+
+enum tb_cfg_pkg_type {
+	TB_CFG_PKG_READ = 1,
+	TB_CFG_PKG_WRITE = 2,
+	TB_CFG_PKG_ERROR = 3,
+	TB_CFG_PKG_NOTIFY_ACK = 4,
+	TB_CFG_PKG_EVENT = 5,
+	TB_CFG_PKG_XDOMAIN_REQ = 6,
+	TB_CFG_PKG_XDOMAIN_RESP = 7,
+	TB_CFG_PKG_OVERRIDE = 8,
+	TB_CFG_PKG_RESET = 9,
+	TB_CFG_PKG_PREPARE_TO_SLEEP = 0xd,
+
+};
+
+enum tb_cfg_space {
+	TB_CFG_HOPS = 0,
+	TB_CFG_PORT = 1,
+	TB_CFG_SWITCH = 2,
+	TB_CFG_COUNTERS = 3,
+};
+
+enum tb_cfg_error {
+	TB_CFG_ERROR_PORT_NOT_CONNECTED = 0,
+	TB_CFG_ERROR_LINK_ERROR = 1,
+	TB_CFG_ERROR_INVALID_CONFIG_SPACE = 2,
+	TB_CFG_ERROR_NO_SUCH_PORT = 4,
+	TB_CFG_ERROR_ACK_PLUG_EVENT = 7, /* send as reply to TB_CFG_PKG_EVENT */
+	TB_CFG_ERROR_LOOP = 8,
+	TB_CFG_ERROR_HEC_ERROR_DETECTED = 12,
+	TB_CFG_ERROR_FLOW_CONTROL_ERROR = 13,
+};
+
+/* common header */
+struct tb_cfg_header {
+	u32 route_hi:22;
+	u32 unknown:10; /* highest order bit is set on replies */
+	u32 route_lo;
+} __packed;
+
+/* additional header for read/write packets */
+struct tb_cfg_address {
+	u32 offset:13; /* in dwords */
+	u32 length:6; /* in dwords */
+	u32 port:6;
+	enum tb_cfg_space space:2;
+	u32 seq:2; /* sequence number  */
+	u32 zero:3;
+} __packed;
+
+/* TB_CFG_PKG_READ, response for TB_CFG_PKG_WRITE */
+struct cfg_read_pkg {
+	struct tb_cfg_header header;
+	struct tb_cfg_address addr;
+} __packed;
+
+/* TB_CFG_PKG_WRITE, response for TB_CFG_PKG_READ */
+struct cfg_write_pkg {
+	struct tb_cfg_header header;
+	struct tb_cfg_address addr;
+	u32 data[64]; /* maximum size, tb_cfg_address.length has 6 bits */
+} __packed;
+
+/* TB_CFG_PKG_ERROR */
+struct cfg_error_pkg {
+	struct tb_cfg_header header;
+	enum tb_cfg_error error:4;
+	u32 zero1:4;
+	u32 port:6;
+	u32 zero2:2; /* Both should be zero, still they are different fields. */
+	u32 zero3:16;
+} __packed;
+
+/* TB_CFG_PKG_EVENT */
+struct cfg_event_pkg {
+	struct tb_cfg_header header;
+	u32 port:6;
+	u32 zero:25;
+	bool unplug:1;
+} __packed;
+
+/* TB_CFG_PKG_RESET */
+struct cfg_reset_pkg {
+	struct tb_cfg_header header;
+} __packed;
+
+/* TB_CFG_PKG_PREPARE_TO_SLEEP */
+struct cfg_pts_pkg {
+	struct tb_cfg_header header;
+	u32 data;
+} __packed;
+
+#endif
-- 
2.11.0

[PATCH 12/24] thunderbolt: Expose get_route() to other files

[Mika Westerberg]

We are going to use it when we change the connection manager to handle
events itself. Also rename it to follow naming convention used in
functions exposed in ctl.h.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 drivers/thunderbolt/ctl.c | 19 +++++++------------
 drivers/thunderbolt/ctl.h |  4 ++++
 2 files changed, 11 insertions(+), 12 deletions(-)

diff --git a/drivers/thunderbolt/ctl.c b/drivers/thunderbolt/ctl.c
index 08bdb7145d89..86902a4d681e 100644
--- a/drivers/thunderbolt/ctl.c
+++ b/drivers/thunderbolt/ctl.c
@@ -54,11 +54,6 @@ struct tb_ctl {
 
 /* utility functions */
 
-static u64 get_route(struct tb_cfg_header header)
-{
-	return (u64) header.route_hi << 32 | header.route_lo;
-}
-
 static struct tb_cfg_header make_header(u64 route)
 {
 	struct tb_cfg_header header = {
@@ -66,7 +61,7 @@ static struct tb_cfg_header make_header(u64 route)
 		.route_lo = route,
 	};
 	/* check for overflow, route_hi is not 32 bits! */
-	WARN_ON(get_route(header) != route);
+	WARN_ON(tb_cfg_get_route(&header) != route);
 	return header;
 }
 
@@ -91,9 +86,9 @@ static int check_header(struct ctl_pkg *pkg, u32 len, enum tb_cfg_pkg_type type,
 	if (WARN(header->unknown != 1 << 9,
 			"header->unknown is %#x\n", header->unknown))
 		return -EIO;
-	if (WARN(route != get_route(*header),
+	if (WARN(route != tb_cfg_get_route(header),
 			"wrong route (expected %llx, got %llx)",
-			route, get_route(*header)))
+			route, tb_cfg_get_route(header)))
 		return -EIO;
 	return 0;
 }
@@ -126,10 +121,10 @@ static struct tb_cfg_result decode_error(struct ctl_pkg *response)
 {
 	struct cfg_error_pkg *pkg = response->buffer;
 	struct tb_cfg_result res = { 0 };
-	res.response_route = get_route(pkg->header);
+	res.response_route = tb_cfg_get_route(&pkg->header);
 	res.response_port = 0;
 	res.err = check_header(response, sizeof(*pkg), TB_CFG_PKG_ERROR,
-			       get_route(pkg->header));
+			       tb_cfg_get_route(&pkg->header));
 	if (res.err)
 		return res;
 
@@ -153,7 +148,7 @@ static struct tb_cfg_result parse_header(struct ctl_pkg *pkg, u32 len,
 		return decode_error(pkg);
 
 	res.response_port = 0; /* will be updated later for cfg_read/write */
-	res.response_route = get_route(*header);
+	res.response_route = tb_cfg_get_route(header);
 	res.err = check_header(pkg, len, type, route);
 	return res;
 }
@@ -294,7 +289,7 @@ static void tb_ctl_handle_plug_event(struct tb_ctl *ctl,
 				     struct ctl_pkg *response)
 {
 	struct cfg_event_pkg *pkg = response->buffer;
-	u64 route = get_route(pkg->header);
+	u64 route = tb_cfg_get_route(&pkg->header);
 
 	if (check_header(response, sizeof(*pkg), TB_CFG_PKG_EVENT, route)) {
 		tb_ctl_warn(ctl, "malformed TB_CFG_PKG_EVENT\n");
diff --git a/drivers/thunderbolt/ctl.h b/drivers/thunderbolt/ctl.h
index 610980e3232f..9812b1c86d4f 100644
--- a/drivers/thunderbolt/ctl.h
+++ b/drivers/thunderbolt/ctl.h
@@ -38,6 +38,10 @@ struct tb_cfg_result {
 	enum tb_cfg_error tb_error; /* valid if err == 1 */
 };
 
+static inline u64 tb_cfg_get_route(const struct tb_cfg_header *header)
+{
+	return (u64) header->route_hi << 32 | header->route_lo;
+}
 
 int tb_cfg_error(struct tb_ctl *ctl, u64 route, u32 port,
 		 enum tb_cfg_error error);
-- 
2.11.0

[PATCH 13/24] thunderbolt: Expose make_header() to other files

[Mika Westerberg]

We will be using this function in files introduced in subsequent
patches. While there the function is renamed to tb_cfg_make_header()
following tb_cfg_get_route().

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 drivers/thunderbolt/ctl.c | 19 ++++---------------
 drivers/thunderbolt/ctl.h | 11 +++++++++++
 2 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/drivers/thunderbolt/ctl.c b/drivers/thunderbolt/ctl.c
index 86902a4d681e..87880dc53199 100644
--- a/drivers/thunderbolt/ctl.c
+++ b/drivers/thunderbolt/ctl.c
@@ -54,17 +54,6 @@ struct tb_ctl {
 
 /* utility functions */
 
-static struct tb_cfg_header make_header(u64 route)
-{
-	struct tb_cfg_header header = {
-		.route_hi = route >> 32,
-		.route_lo = route,
-	};
-	/* check for overflow, route_hi is not 32 bits! */
-	WARN_ON(tb_cfg_get_route(&header) != route);
-	return header;
-}
-
 static int check_header(struct ctl_pkg *pkg, u32 len, enum tb_cfg_pkg_type type,
 			u64 route)
 {
@@ -497,7 +486,7 @@ int tb_cfg_error(struct tb_ctl *ctl, u64 route, u32 port,
 		 enum tb_cfg_error error)
 {
 	struct cfg_error_pkg pkg = {
-		.header = make_header(route),
+		.header = tb_cfg_make_header(route),
 		.port = port,
 		.error = error,
 	};
@@ -516,7 +505,7 @@ struct tb_cfg_result tb_cfg_reset(struct tb_ctl *ctl, u64 route,
 				  int timeout_msec)
 {
 	int err;
-	struct cfg_reset_pkg request = { .header = make_header(route) };
+	struct cfg_reset_pkg request = { .header = tb_cfg_make_header(route) };
 	struct tb_cfg_header reply;
 
 	err = tb_ctl_tx(ctl, &request, sizeof(request), TB_CFG_PKG_RESET);
@@ -538,7 +527,7 @@ struct tb_cfg_result tb_cfg_read_raw(struct tb_ctl *ctl, void *buffer,
 {
 	struct tb_cfg_result res = { 0 };
 	struct cfg_read_pkg request = {
-		.header = make_header(route),
+		.header = tb_cfg_make_header(route),
 		.addr = {
 			.port = port,
 			.space = space,
@@ -575,7 +564,7 @@ struct tb_cfg_result tb_cfg_write_raw(struct tb_ctl *ctl, const void *buffer,
 {
 	struct tb_cfg_result res = { 0 };
 	struct cfg_write_pkg request = {
-		.header = make_header(route),
+		.header = tb_cfg_make_header(route),
 		.addr = {
 			.port = port,
 			.space = space,
diff --git a/drivers/thunderbolt/ctl.h b/drivers/thunderbolt/ctl.h
index 9812b1c86d4f..914da86ec77d 100644
--- a/drivers/thunderbolt/ctl.h
+++ b/drivers/thunderbolt/ctl.h
@@ -43,6 +43,17 @@ static inline u64 tb_cfg_get_route(const struct tb_cfg_header *header)
 	return (u64) header->route_hi << 32 | header->route_lo;
 }
 
+static inline struct tb_cfg_header tb_cfg_make_header(u64 route)
+{
+	struct tb_cfg_header header = {
+		.route_hi = route >> 32,
+		.route_lo = route,
+	};
+	/* check for overflow, route_hi is not 32 bits! */
+	WARN_ON(tb_cfg_get_route(&header) != route);
+	return header;
+}
+
 int tb_cfg_error(struct tb_ctl *ctl, u64 route, u32 port,
 		 enum tb_cfg_error error);
 struct tb_cfg_result tb_cfg_reset(struct tb_ctl *ctl, u64 route,
-- 
2.11.0

[PATCH 14/24] thunderbolt: Let the connection manager handle all notifications

[Mika Westerberg]

Currently the control channel (ctl.c) handles the one supported
notification (PLUG_EVENT) and sends back ACK accordingly. However, we
are going to add support for the internal connection manager (ICM) that
needs to handle a different notifications. So instead of dealing
everything in the control channel, we change the callback to take an
arbitrary thunderbolt packet and convert the native connection manager
to handle the event itself.

In addition we only push replies we know of to the response FIFO.
Everything else is treated as notification (or request) and is expected
to be dealt by the connection manager implementation.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 drivers/thunderbolt/ctl.c    | 86 ++++++++++++++++++++++++++++++--------------
 drivers/thunderbolt/ctl.h    |  5 +--
 drivers/thunderbolt/domain.c | 15 +++++++-
 drivers/thunderbolt/tb.c     | 30 ++++++++++++----
 drivers/thunderbolt/tb.h     |  5 +--
 5 files changed, 103 insertions(+), 38 deletions(-)

diff --git a/drivers/thunderbolt/ctl.c b/drivers/thunderbolt/ctl.c
index 87880dc53199..b3ee755fdb37 100644
--- a/drivers/thunderbolt/ctl.c
+++ b/drivers/thunderbolt/ctl.c
@@ -35,7 +35,7 @@ struct tb_ctl {
 	DECLARE_KFIFO(response_fifo, struct ctl_pkg*, 16);
 	struct completion response_ready;
 
-	hotplug_cb callback;
+	event_cb callback;
 	void *callback_data;
 };
 
@@ -52,6 +52,9 @@ struct tb_ctl {
 #define tb_ctl_info(ctl, format, arg...) \
 	dev_info(&(ctl)->nhi->pdev->dev, format, ## arg)
 
+#define tb_ctl_dbg(ctl, format, arg...) \
+	dev_dbg(&(ctl)->nhi->pdev->dev, format, ## arg)
+
 /* utility functions */
 
 static int check_header(struct ctl_pkg *pkg, u32 len, enum tb_cfg_pkg_type type,
@@ -272,24 +275,12 @@ static int tb_ctl_tx(struct tb_ctl *ctl, const void *data, size_t len,
 }
 
 /**
- * tb_ctl_handle_plug_event() - acknowledge a plug event, invoke ctl->callback
+ * tb_ctl_handle_event() - acknowledge a plug event, invoke ctl->callback
  */
-static void tb_ctl_handle_plug_event(struct tb_ctl *ctl,
-				     struct ctl_pkg *response)
+static void tb_ctl_handle_event(struct tb_ctl *ctl, enum tb_cfg_pkg_type type,
+				struct ctl_pkg *pkg, size_t size)
 {
-	struct cfg_event_pkg *pkg = response->buffer;
-	u64 route = tb_cfg_get_route(&pkg->header);
-
-	if (check_header(response, sizeof(*pkg), TB_CFG_PKG_EVENT, route)) {
-		tb_ctl_warn(ctl, "malformed TB_CFG_PKG_EVENT\n");
-		return;
-	}
-
-	if (tb_cfg_error(ctl, route, pkg->port, TB_CFG_ERROR_ACK_PLUG_EVENT))
-		tb_ctl_warn(ctl, "could not ack plug event on %llx:%x\n",
-			    route, pkg->port);
-	WARN(pkg->zero, "pkg->zero is %#x\n", pkg->zero);
-	ctl->callback(ctl->callback_data, route, pkg->port, pkg->unplug);
+	ctl->callback(ctl->callback_data, type, pkg->buffer, size);
 }
 
 static void tb_ctl_rx_submit(struct ctl_pkg *pkg)
@@ -302,10 +293,29 @@ static void tb_ctl_rx_submit(struct ctl_pkg *pkg)
 					     */
 }
 
+static int tb_async_error(const struct ctl_pkg *pkg)
+{
+	const struct cfg_error_pkg *error = (const struct cfg_error_pkg *)pkg;
+
+	if (pkg->frame.eof != TB_CFG_PKG_ERROR)
+		return false;
+
+	switch (error->error) {
+	case TB_CFG_ERROR_LINK_ERROR:
+	case TB_CFG_ERROR_HEC_ERROR_DETECTED:
+	case TB_CFG_ERROR_FLOW_CONTROL_ERROR:
+		return true;
+
+	default:
+		return false;
+	}
+}
+
 static void tb_ctl_rx_callback(struct tb_ring *ring, struct ring_frame *frame,
 			       bool canceled)
 {
 	struct ctl_pkg *pkg = container_of(frame, typeof(*pkg), frame);
+	__be32 crc32;
 
 	if (canceled)
 		return; /*
@@ -320,18 +330,42 @@ static void tb_ctl_rx_callback(struct tb_ring *ring, struct ring_frame *frame,
 	}
 
 	frame->size -= 4; /* remove checksum */
-	if (*(__be32 *) (pkg->buffer + frame->size)
-			!= tb_crc(pkg->buffer, frame->size)) {
-		tb_ctl_err(pkg->ctl,
-			   "RX: checksum mismatch, dropping packet\n");
-		goto rx;
-	}
+	crc32 = tb_crc(pkg->buffer, frame->size);
 	be32_to_cpu_array(pkg->buffer, pkg->buffer, frame->size / 4);
 
-	if (frame->eof == TB_CFG_PKG_EVENT) {
-		tb_ctl_handle_plug_event(pkg->ctl, pkg);
+	switch (frame->eof) {
+	case TB_CFG_PKG_READ:
+	case TB_CFG_PKG_WRITE:
+	case TB_CFG_PKG_ERROR:
+	case TB_CFG_PKG_OVERRIDE:
+	case TB_CFG_PKG_RESET:
+		if (*(__be32 *)(pkg->buffer + frame->size) != crc32) {
+			tb_ctl_err(pkg->ctl,
+				   "RX: checksum mismatch, dropping packet\n");
+			goto rx;
+		}
+		if (tb_async_error(pkg)) {
+			tb_ctl_handle_event(pkg->ctl, frame->eof,
+					    pkg, frame->size);
+			goto rx;
+		}
+		break;
+
+	case TB_CFG_PKG_EVENT:
+		if (*(__be32 *)(pkg->buffer + frame->size) != crc32) {
+			tb_ctl_err(pkg->ctl,
+				   "RX: checksum mismatch, dropping packet\n");
+			goto rx;
+		}
+		tb_ctl_handle_event(pkg->ctl, frame->eof, pkg, frame->size);
+		goto rx;
+
+	default:
+		tb_ctl_dbg(pkg->ctl, "RX: unknown package %#x, dropping\n",
+			   frame->eof);
 		goto rx;
 	}
+
 	if (!kfifo_put(&pkg->ctl->response_fifo, pkg)) {
 		tb_ctl_err(pkg->ctl, "RX: fifo is full\n");
 		goto rx;
@@ -379,7 +413,7 @@ static struct tb_cfg_result tb_ctl_rx(struct tb_ctl *ctl, void *buffer,
  *
  * Return: Returns a pointer on success or NULL on failure.
  */
-struct tb_ctl *tb_ctl_alloc(struct tb_nhi *nhi, hotplug_cb cb, void *cb_data)
+struct tb_ctl *tb_ctl_alloc(struct tb_nhi *nhi, event_cb cb, void *cb_data)
 {
 	int i;
 	struct tb_ctl *ctl = kzalloc(sizeof(*ctl), GFP_KERNEL);
diff --git a/drivers/thunderbolt/ctl.h b/drivers/thunderbolt/ctl.h
index 914da86ec77d..2b23e030a85b 100644
--- a/drivers/thunderbolt/ctl.h
+++ b/drivers/thunderbolt/ctl.h
@@ -13,9 +13,10 @@
 /* control channel */
 struct tb_ctl;
 
-typedef void (*hotplug_cb)(void *data, u64 route, u8 port, bool unplug);
+typedef void (*event_cb)(void *data, enum tb_cfg_pkg_type type,
+			 const void *buf, size_t size);
 
-struct tb_ctl *tb_ctl_alloc(struct tb_nhi *nhi, hotplug_cb cb, void *cb_data);
+struct tb_ctl *tb_ctl_alloc(struct tb_nhi *nhi, event_cb cb, void *cb_data);
 void tb_ctl_start(struct tb_ctl *ctl);
 void tb_ctl_stop(struct tb_ctl *ctl);
 void tb_ctl_free(struct tb_ctl *ctl);
diff --git a/drivers/thunderbolt/domain.c b/drivers/thunderbolt/domain.c
index b5bfca95415d..99fe34d16b18 100644
--- a/drivers/thunderbolt/domain.c
+++ b/drivers/thunderbolt/domain.c
@@ -96,6 +96,19 @@ struct tb *tb_domain_alloc(struct tb_nhi *nhi, size_t privsize)
 	return NULL;
 }
 
+static void tb_domain_event_cb(void *data, enum tb_cfg_pkg_type type,
+			       const void *buf, size_t size)
+{
+	struct tb *tb = data;
+
+	if (!tb->cm_ops->handle_event) {
+		tb_warn(tb, "domain does not have event handler\n");
+		return;
+	}
+
+	tb->cm_ops->handle_event(tb, type, buf, size);
+}
+
 /**
  * tb_domain_add() - Add domain to the system
  * @tb: Domain to add
@@ -116,7 +129,7 @@ int tb_domain_add(struct tb *tb)
 
 	mutex_lock(&tb->lock);
 
-	tb->ctl = tb_ctl_alloc(tb->nhi, tb->cm_ops->hotplug, tb);
+	tb->ctl = tb_ctl_alloc(tb->nhi, tb_domain_event_cb, tb);
 	if (!tb->ctl) {
 		ret = -ENOMEM;
 		goto err_unlock;
diff --git a/drivers/thunderbolt/tb.c b/drivers/thunderbolt/tb.c
index 94ecac012428..ea9de49b5e10 100644
--- a/drivers/thunderbolt/tb.c
+++ b/drivers/thunderbolt/tb.c
@@ -311,18 +311,34 @@ static void tb_handle_hotplug(struct work_struct *work)
  *
  * Delegates to tb_handle_hotplug.
  */
-static void tb_schedule_hotplug_handler(void *data, u64 route, u8 port,
-					bool unplug)
+static void tb_handle_event(struct tb *tb, enum tb_cfg_pkg_type type,
+			    const void *buf, size_t size)
 {
-	struct tb *tb = data;
-	struct tb_hotplug_event *ev = kmalloc(sizeof(*ev), GFP_KERNEL);
+	const struct cfg_event_pkg *pkg = buf;
+	struct tb_hotplug_event *ev;
+	u64 route;
+
+	if (type != TB_CFG_PKG_EVENT) {
+		tb_warn(tb, "unexpected event %#x, ignoring\n", type);
+		return;
+	}
+
+	route = tb_cfg_get_route(&pkg->header);
+
+	if (tb_cfg_error(tb->ctl, route, pkg->port,
+			 TB_CFG_ERROR_ACK_PLUG_EVENT)) {
+		tb_warn(tb, "could not ack plug event on %llx:%x\n", route,
+			pkg->port);
+	}
+
+	ev = kmalloc(sizeof(*ev), GFP_KERNEL);
 	if (!ev)
 		return;
 	INIT_WORK(&ev->work, tb_handle_hotplug);
 	ev->tb = tb;
 	ev->route = route;
-	ev->port = port;
-	ev->unplug = unplug;
+	ev->port = pkg->port;
+	ev->unplug = pkg->unplug;
 	queue_work(tb->wq, &ev->work);
 }
 
@@ -419,7 +435,7 @@ static const struct tb_cm_ops tb_cm_ops = {
 	.stop = tb_stop,
 	.suspend_noirq = tb_suspend_noirq,
 	.resume_noirq = tb_resume_noirq,
-	.hotplug = tb_schedule_hotplug_handler,
+	.handle_event = tb_handle_event,
 };
 
 struct tb *tb_probe(struct tb_nhi *nhi)
diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
index 5e66dce53c65..73db614fa913 100644
--- a/drivers/thunderbolt/tb.h
+++ b/drivers/thunderbolt/tb.h
@@ -118,14 +118,15 @@ struct tb_path {
  * @stop: Stops the domain
  * @suspend_noirq: Connection manager specific suspend_noirq
  * @resume_noirq: Connection manager specific resume_noirq
- * @hotplug: Handle hotplug event
+ * @handle_event: Handle thunderbolt event
  */
 struct tb_cm_ops {
 	int (*start)(struct tb *tb);
 	void (*stop)(struct tb *tb);
 	int (*suspend_noirq)(struct tb *tb);
 	int (*resume_noirq)(struct tb *tb);
-	hotplug_cb hotplug;
+	void (*handle_event)(struct tb *tb, enum tb_cfg_pkg_type,
+			     const void *buf, size_t size);
 };
 
 /**
-- 
2.11.0

[PATCH 15/24] thunderbolt: Rework control channel to be more reliable

[Mika Westerberg]

If a request times out the response might arrive right after the request
is failed. This response is pushed to the kfifo and next request will
read it instead. Since it most likely will not pass our validation
checks in parse_header() the next request will fail as well, and
response to that request will be pushed to the kfifo, ad infinitum.

We end up in a situation where all requests fail and no devices can be
added anymore until the driver is unloaded and reloaded again.

To overcome this, rework the control channel so that we will have a
queue of outstanding requests. Each request will be handled in turn and
the response is validated against what is expected. Unexpected packets
(for example responses for requests that have been timed out) are
dropped. This model is copied from Greybus implementation with small
changes here and there to get it cope with Thunderbolt control packets.

In addition the configuration packets support sequence number which the
switch is supposed to copy from the request to response. We use this to
drop responses that are already timed out. Taking advantage of the
sequence number, we automatically retry configuration read/write 4 times
before giving up.

Also timeout is not a programming error so there is no need to trigger a
scary backtrace (WARN), instead we just log a warning.  After all
Thunderbolt devices are hot-pluggable by definition which means user can
unplug a device any time and that is totally acceptable.

With this change there is no need to take the global domain lock when
sending configuration packets anymore. This is useful when we add
support for cross-domain (XDomain) communication later on.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 drivers/thunderbolt/ctl.c | 471 +++++++++++++++++++++++++++++++++++++++-------
 drivers/thunderbolt/ctl.h |  65 +++++++
 drivers/thunderbolt/tb.h  |   2 +-
 3 files changed, 467 insertions(+), 71 deletions(-)

diff --git a/drivers/thunderbolt/ctl.c b/drivers/thunderbolt/ctl.c
index b3ee755fdb37..2b1255cbf3c9 100644
--- a/drivers/thunderbolt/ctl.c
+++ b/drivers/thunderbolt/ctl.c
@@ -5,22 +5,17 @@
  */
 
 #include <linux/crc32.h>
+#include <linux/delay.h>
 #include <linux/slab.h>
 #include <linux/pci.h>
 #include <linux/dmapool.h>
 #include <linux/workqueue.h>
-#include <linux/kfifo.h>
 
 #include "ctl.h"
 
 
-struct ctl_pkg {
-	struct tb_ctl *ctl;
-	void *buffer;
-	struct ring_frame frame;
-};
-
-#define TB_CTL_RX_PKG_COUNT 10
+#define TB_CTL_RX_PKG_COUNT	10
+#define TB_CTL_RETRIES		4
 
 /**
  * struct tb_cfg - thunderbolt control channel
@@ -32,8 +27,9 @@ struct tb_ctl {
 
 	struct dma_pool *frame_pool;
 	struct ctl_pkg *rx_packets[TB_CTL_RX_PKG_COUNT];
-	DECLARE_KFIFO(response_fifo, struct ctl_pkg*, 16);
-	struct completion response_ready;
+	struct mutex request_lock;
+	struct list_head request_queue;
+	bool running;
 
 	event_cb callback;
 	void *callback_data;
@@ -55,10 +51,115 @@ struct tb_ctl {
 #define tb_ctl_dbg(ctl, format, arg...) \
 	dev_dbg(&(ctl)->nhi->pdev->dev, format, ## arg)
 
+static DECLARE_WAIT_QUEUE_HEAD(tb_cfg_request_cancel_queue);
+
+/**
+ * tb_cfg_request_alloc() - Allocates a new config request
+ *
+ * This is refcounted object so when you are done with this, call
+ * tb_cfg_request_put() to it.
+ */
+struct tb_cfg_request *tb_cfg_request_alloc(void)
+{
+	struct tb_cfg_request *req;
+
+	req = kzalloc(sizeof(*req), GFP_KERNEL);
+	if (!req)
+		return NULL;
+
+	kref_init(&req->kref);
+
+	return req;
+}
+
+/**
+ * tb_cfg_request_get() - Increase refcount of a request
+ * @req: Request whose refcount is increased
+ */
+void tb_cfg_request_get(struct tb_cfg_request *req)
+{
+	kref_get(&req->kref);
+}
+
+static void tb_cfg_request_destroy(struct kref *kref)
+{
+	struct tb_cfg_request *req = container_of(kref, typeof(*req), kref);
+
+	kfree(req);
+}
+
+/**
+ * tb_cfg_request_put() - Decrease refcount and possibly release the request
+ * @req: Request whose refcount is decreased
+ *
+ * Call this function when you are done with the request. When refcount
+ * goes to %0 the object is released.
+ */
+void tb_cfg_request_put(struct tb_cfg_request *req)
+{
+	kref_put(&req->kref, tb_cfg_request_destroy);
+}
+
+static int tb_cfg_request_enqueue(struct tb_ctl *ctl,
+				  struct tb_cfg_request *req)
+{
+	WARN_ON(test_bit(TB_CFG_REQUEST_ACTIVE, &req->flags));
+	WARN_ON(req->ctl);
+
+	mutex_lock(&ctl->request_lock);
+	if (!ctl->running) {
+		mutex_unlock(&ctl->request_lock);
+		return -ENOTCONN;
+	}
+	req->ctl = ctl;
+	list_add_tail(&req->list, &ctl->request_queue);
+	set_bit(TB_CFG_REQUEST_ACTIVE, &req->flags);
+	mutex_unlock(&ctl->request_lock);
+	return 0;
+}
+
+static void tb_cfg_request_dequeue(struct tb_cfg_request *req)
+{
+	struct tb_ctl *ctl = req->ctl;
+
+	mutex_lock(&ctl->request_lock);
+	list_del(&req->list);
+	clear_bit(TB_CFG_REQUEST_ACTIVE, &req->flags);
+	if (test_bit(TB_CFG_REQUEST_CANCELED, &req->flags))
+		wake_up(&tb_cfg_request_cancel_queue);
+	mutex_unlock(&ctl->request_lock);
+}
+
+static bool tb_cfg_request_is_active(struct tb_cfg_request *req)
+{
+	return test_bit(TB_CFG_REQUEST_ACTIVE, &req->flags);
+}
+
+static struct tb_cfg_request *
+tb_cfg_request_find(struct tb_ctl *ctl, struct ctl_pkg *pkg)
+{
+	struct tb_cfg_request *req;
+	bool found = false;
+
+	mutex_lock(&pkg->ctl->request_lock);
+	list_for_each_entry(req, &pkg->ctl->request_queue, list) {
+		tb_cfg_request_get(req);
+		if (req->match(req, pkg)) {
+			found = true;
+			break;
+		}
+		tb_cfg_request_put(req);
+	}
+	mutex_unlock(&pkg->ctl->request_lock);
+
+	return found ? req : NULL;
+}
+
 /* utility functions */
 
-static int check_header(struct ctl_pkg *pkg, u32 len, enum tb_cfg_pkg_type type,
-			u64 route)
+
+static int check_header(const struct ctl_pkg *pkg, u32 len,
+			enum tb_cfg_pkg_type type, u64 route)
 {
 	struct tb_cfg_header *header = pkg->buffer;
 
@@ -100,8 +201,6 @@ static int check_config_address(struct tb_cfg_address addr,
 	if (WARN(length != addr.length, "wrong space (expected %x, got %x\n)",
 			length, addr.length))
 		return -EIO;
-	if (WARN(addr.seq, "addr.seq is %#x\n", addr.seq))
-		return -EIO;
 	/*
 	 * We cannot check addr->port as it is set to the upstream port of the
 	 * sender.
@@ -109,7 +208,7 @@ static int check_config_address(struct tb_cfg_address addr,
 	return 0;
 }
 
-static struct tb_cfg_result decode_error(struct ctl_pkg *response)
+static struct tb_cfg_result decode_error(const struct ctl_pkg *response)
 {
 	struct cfg_error_pkg *pkg = response->buffer;
 	struct tb_cfg_result res = { 0 };
@@ -130,7 +229,7 @@ static struct tb_cfg_result decode_error(struct ctl_pkg *response)
 
 }
 
-static struct tb_cfg_result parse_header(struct ctl_pkg *pkg, u32 len,
+static struct tb_cfg_result parse_header(const struct ctl_pkg *pkg, u32 len,
 					 enum tb_cfg_pkg_type type, u64 route)
 {
 	struct tb_cfg_header *header = pkg->buffer;
@@ -198,7 +297,7 @@ static void be32_to_cpu_array(u32 *dst, __be32 *src, size_t len)
 		dst[i] = be32_to_cpu(src[i]);
 }
 
-static __be32 tb_crc(void *data, size_t len)
+static __be32 tb_crc(const void *data, size_t len)
 {
 	return cpu_to_be32(~__crc32c_le(~0, data, len));
 }
@@ -315,6 +414,7 @@ static void tb_ctl_rx_callback(struct tb_ring *ring, struct ring_frame *frame,
 			       bool canceled)
 {
 	struct ctl_pkg *pkg = container_of(frame, typeof(*pkg), frame);
+	struct tb_cfg_request *req;
 	__be32 crc32;
 
 	if (canceled)
@@ -361,48 +461,135 @@ static void tb_ctl_rx_callback(struct tb_ring *ring, struct ring_frame *frame,
 		goto rx;
 
 	default:
-		tb_ctl_dbg(pkg->ctl, "RX: unknown package %#x, dropping\n",
-			   frame->eof);
-		goto rx;
+		break;
 	}
 
-	if (!kfifo_put(&pkg->ctl->response_fifo, pkg)) {
-		tb_ctl_err(pkg->ctl, "RX: fifo is full\n");
-		goto rx;
+	/*
+	 * The received packet will be processed only if there is an
+	 * active request and that the packet is what is expected. This
+	 * prevents packets such as replies coming after timeout has
+	 * triggered from messing with the active requests.
+	 */
+	req = tb_cfg_request_find(pkg->ctl, pkg);
+	if (req) {
+		if (req->copy(req, pkg))
+			schedule_work(&req->work);
+		tb_cfg_request_put(req);
 	}
-	complete(&pkg->ctl->response_ready);
-	return;
+
 rx:
 	tb_ctl_rx_submit(pkg);
 }
 
+static void tb_cfg_request_work(struct work_struct *work)
+{
+	struct tb_cfg_request *req = container_of(work, typeof(*req), work);
+
+	if (!test_bit(TB_CFG_REQUEST_CANCELED, &req->flags))
+		req->callback(req->callback_data);
+
+	tb_cfg_request_dequeue(req);
+	tb_cfg_request_put(req);
+}
+
 /**
- * tb_ctl_rx() - receive a packet from the control channel
+ * tb_cfg_request() - Start control request not waiting for it to complete
+ * @ctl: Control channel to use
+ * @req: Request to start
+ * @callback: Callback called when the request is completed
+ * @callback_data: Data to be passed to @callback
+ *
+ * This queues @req on the given control channel without waiting for it
+ * to complete. When the request completes @callback is called.
  */
-static struct tb_cfg_result tb_ctl_rx(struct tb_ctl *ctl, void *buffer,
-				      size_t length, int timeout_msec,
-				      u64 route, enum tb_cfg_pkg_type type)
+int tb_cfg_request(struct tb_ctl *ctl, struct tb_cfg_request *req,
+		   void (*callback)(void *), void *callback_data)
 {
-	struct tb_cfg_result res;
-	struct ctl_pkg *pkg;
+	int ret;
 
-	if (!wait_for_completion_timeout(&ctl->response_ready,
-					 msecs_to_jiffies(timeout_msec))) {
-		tb_ctl_WARN(ctl, "RX: timeout\n");
-		return (struct tb_cfg_result) { .err = -ETIMEDOUT };
-	}
-	if (!kfifo_get(&ctl->response_fifo, &pkg)) {
-		tb_ctl_WARN(ctl, "empty kfifo\n");
-		return (struct tb_cfg_result) { .err = -EIO };
-	}
+	req->flags = 0;
+	req->callback = callback;
+	req->callback_data = callback_data;
+	INIT_WORK(&req->work, tb_cfg_request_work);
+	INIT_LIST_HEAD(&req->list);
 
-	res = parse_header(pkg, length, type, route);
-	if (!res.err)
-		memcpy(buffer, pkg->buffer, length);
-	tb_ctl_rx_submit(pkg);
-	return res;
+	tb_cfg_request_get(req);
+	ret = tb_cfg_request_enqueue(ctl, req);
+	if (ret)
+		goto err_put;
+
+	ret = tb_ctl_tx(ctl, req->request, req->request_size,
+			req->request_type);
+	if (ret)
+		goto err_dequeue;
+
+	if (!req->response)
+		schedule_work(&req->work);
+
+	return 0;
+
+err_dequeue:
+	tb_cfg_request_dequeue(req);
+err_put:
+	tb_cfg_request_put(req);
+
+	return ret;
+}
+
+/**
+ * tb_cfg_request_cancel() - Cancel a control request
+ * @req: Request to cancel
+ * @err: Error to assign to the request
+ *
+ * This function can be used to cancel ongoing request. It will wait
+ * until the request is not active anymore.
+ */
+void tb_cfg_request_cancel(struct tb_cfg_request *req, int err)
+{
+	set_bit(TB_CFG_REQUEST_CANCELED, &req->flags);
+	schedule_work(&req->work);
+	wait_event(tb_cfg_request_cancel_queue, !tb_cfg_request_is_active(req));
+	req->result.err = err;
 }
 
+static void tb_cfg_request_complete(void *data)
+{
+	complete(data);
+}
+
+/**
+ * tb_cfg_request_sync() - Start control request and wait until it completes
+ * @ctl: Control channel to use
+ * @req: Request to start
+ * @timeout_msec: Timeout how long to wait @req to complete
+ *
+ * Starts a control request and waits until it completes. If timeout
+ * triggers the request is canceled before function returns. Note the
+ * caller needs to make sure only one message for given switch is active
+ * at a time.
+ */
+struct tb_cfg_result tb_cfg_request_sync(struct tb_ctl *ctl,
+					 struct tb_cfg_request *req,
+					 int timeout_msec)
+{
+	unsigned long timeout = msecs_to_jiffies(timeout_msec);
+	struct tb_cfg_result res = { 0 };
+	DECLARE_COMPLETION_ONSTACK(done);
+	int ret;
+
+	ret = tb_cfg_request(ctl, req, tb_cfg_request_complete, &done);
+	if (ret) {
+		res.err = ret;
+		return res;
+	}
+
+	if (!wait_for_completion_timeout(&done, timeout))
+		tb_cfg_request_cancel(req, -ETIMEDOUT);
+
+	flush_work(&req->work);
+
+	return req->result;
+}
 
 /* public interface, alloc/start/stop/free */
 
@@ -423,8 +610,8 @@ struct tb_ctl *tb_ctl_alloc(struct tb_nhi *nhi, event_cb cb, void *cb_data)
 	ctl->callback = cb;
 	ctl->callback_data = cb_data;
 
-	init_completion(&ctl->response_ready);
-	INIT_KFIFO(ctl->response_fifo);
+	mutex_init(&ctl->request_lock);
+	INIT_LIST_HEAD(&ctl->request_queue);
 	ctl->frame_pool = dma_pool_create("thunderbolt_ctl", &nhi->pdev->dev,
 					 TB_FRAME_SIZE, 4, 0);
 	if (!ctl->frame_pool)
@@ -488,6 +675,8 @@ void tb_ctl_start(struct tb_ctl *ctl)
 	ring_start(ctl->rx);
 	for (i = 0; i < TB_CTL_RX_PKG_COUNT; i++)
 		tb_ctl_rx_submit(ctl->rx_packets[i]);
+
+	ctl->running = true;
 }
 
 /**
@@ -500,12 +689,16 @@ void tb_ctl_start(struct tb_ctl *ctl)
  */
 void tb_ctl_stop(struct tb_ctl *ctl)
 {
+	mutex_lock(&ctl->request_lock);
+	ctl->running = false;
+	mutex_unlock(&ctl->request_lock);
+
 	ring_stop(ctl->rx);
 	ring_stop(ctl->tx);
 
-	if (!kfifo_is_empty(&ctl->response_fifo))
-		tb_ctl_WARN(ctl, "dangling response in response_fifo\n");
-	kfifo_reset(&ctl->response_fifo);
+	if (!list_empty(&ctl->request_queue))
+		tb_ctl_WARN(ctl, "dangling request in request_queue\n");
+	INIT_LIST_HEAD(&ctl->request_queue);
 	tb_ctl_info(ctl, "control channel stopped\n");
 }
 
@@ -528,6 +721,49 @@ int tb_cfg_error(struct tb_ctl *ctl, u64 route, u32 port,
 	return tb_ctl_tx(ctl, &pkg, sizeof(pkg), TB_CFG_PKG_ERROR);
 }
 
+static bool tb_cfg_match(const struct tb_cfg_request *req,
+			 const struct ctl_pkg *pkg)
+{
+	u64 route = tb_cfg_get_route(pkg->buffer) & ~BIT_ULL(63);
+
+	if (pkg->frame.eof == TB_CFG_PKG_ERROR)
+		return true;
+
+	if (pkg->frame.eof != req->response_type)
+		return false;
+	if (route != tb_cfg_get_route(req->request))
+		return false;
+	if (pkg->frame.size != req->response_size)
+		return false;
+
+	if (pkg->frame.eof == TB_CFG_PKG_READ ||
+	    pkg->frame.eof == TB_CFG_PKG_WRITE) {
+		const struct cfg_read_pkg *req_hdr = req->request;
+		const struct cfg_read_pkg *res_hdr = pkg->buffer;
+
+		if (req_hdr->addr.seq != res_hdr->addr.seq)
+			return false;
+	}
+
+	return true;
+}
+
+static bool tb_cfg_copy(struct tb_cfg_request *req, const struct ctl_pkg *pkg)
+{
+	struct tb_cfg_result res;
+
+	/* Now make sure it is in expected format */
+	res = parse_header(pkg, req->response_size, req->response_type,
+			   tb_cfg_get_route(req->request));
+	if (!res.err)
+		memcpy(req->response, pkg->buffer, req->response_size);
+
+	req->result = res;
+
+	/* Always complete when first response is received */
+	return true;
+}
+
 /**
  * tb_cfg_reset() - send a reset packet and wait for a response
  *
@@ -538,16 +774,31 @@ int tb_cfg_error(struct tb_ctl *ctl, u64 route, u32 port,
 struct tb_cfg_result tb_cfg_reset(struct tb_ctl *ctl, u64 route,
 				  int timeout_msec)
 {
-	int err;
 	struct cfg_reset_pkg request = { .header = tb_cfg_make_header(route) };
+	struct tb_cfg_result res = { 0 };
 	struct tb_cfg_header reply;
+	struct tb_cfg_request *req;
+
+	req = tb_cfg_request_alloc();
+	if (!req) {
+		res.err = -ENOMEM;
+		return res;
+	}
+
+	req->match = tb_cfg_match;
+	req->copy = tb_cfg_copy;
+	req->request = &request;
+	req->request_size = sizeof(request);
+	req->request_type = TB_CFG_PKG_RESET;
+	req->response = &reply;
+	req->response_size = sizeof(reply);
+	req->response_type = sizeof(TB_CFG_PKG_RESET);
+
+	res = tb_cfg_request_sync(ctl, req, timeout_msec);
 
-	err = tb_ctl_tx(ctl, &request, sizeof(request), TB_CFG_PKG_RESET);
-	if (err)
-		return (struct tb_cfg_result) { .err = err };
+	tb_cfg_request_put(req);
 
-	return tb_ctl_rx(ctl, &reply, sizeof(reply), timeout_msec, route,
-			 TB_CFG_PKG_RESET);
+	return res;
 }
 
 /**
@@ -570,13 +821,39 @@ struct tb_cfg_result tb_cfg_read_raw(struct tb_ctl *ctl, void *buffer,
 		},
 	};
 	struct cfg_write_pkg reply;
+	int retries = 0;
 
-	res.err = tb_ctl_tx(ctl, &request, sizeof(request), TB_CFG_PKG_READ);
-	if (res.err)
-		return res;
+	while (retries < TB_CTL_RETRIES) {
+		struct tb_cfg_request *req;
+
+		req = tb_cfg_request_alloc();
+		if (!req) {
+			res.err = -ENOMEM;
+			return res;
+		}
+
+		request.addr.seq = retries++;
+
+		req->match = tb_cfg_match;
+		req->copy = tb_cfg_copy;
+		req->request = &request;
+		req->request_size = sizeof(request);
+		req->request_type = TB_CFG_PKG_READ;
+		req->response = &reply;
+		req->response_size = 12 + 4 * length;
+		req->response_type = TB_CFG_PKG_READ;
+
+		res = tb_cfg_request_sync(ctl, req, timeout_msec);
+
+		tb_cfg_request_put(req);
+
+		if (res.err != -ETIMEDOUT)
+			break;
+
+		/* Wait a bit (arbitrary time) until we send a retry */
+		usleep_range(10, 100);
+	}
 
-	res = tb_ctl_rx(ctl, &reply, 12 + 4 * length, timeout_msec, route,
-			TB_CFG_PKG_READ);
 	if (res.err)
 		return res;
 
@@ -607,15 +884,41 @@ struct tb_cfg_result tb_cfg_write_raw(struct tb_ctl *ctl, const void *buffer,
 		},
 	};
 	struct cfg_read_pkg reply;
+	int retries = 0;
 
 	memcpy(&request.data, buffer, length * 4);
 
-	res.err = tb_ctl_tx(ctl, &request, 12 + 4 * length, TB_CFG_PKG_WRITE);
-	if (res.err)
-		return res;
+	while (retries < TB_CTL_RETRIES) {
+		struct tb_cfg_request *req;
+
+		req = tb_cfg_request_alloc();
+		if (!req) {
+			res.err = -ENOMEM;
+			return res;
+		}
+
+		request.addr.seq = retries++;
+
+		req->match = tb_cfg_match;
+		req->copy = tb_cfg_copy;
+		req->request = &request;
+		req->request_size = 12 + 4 * length;
+		req->request_type = TB_CFG_PKG_WRITE;
+		req->response = &reply;
+		req->response_size = sizeof(reply);
+		req->response_type = TB_CFG_PKG_WRITE;
+
+		res = tb_cfg_request_sync(ctl, req, timeout_msec);
+
+		tb_cfg_request_put(req);
+
+		if (res.err != -ETIMEDOUT)
+			break;
+
+		/* Wait a bit (arbitrary time) until we send a retry */
+		usleep_range(10, 100);
+	}
 
-	res = tb_ctl_rx(ctl, &reply, sizeof(reply), timeout_msec, route,
-			TB_CFG_PKG_WRITE);
 	if (res.err)
 		return res;
 
@@ -629,11 +932,25 @@ int tb_cfg_read(struct tb_ctl *ctl, void *buffer, u64 route, u32 port,
 {
 	struct tb_cfg_result res = tb_cfg_read_raw(ctl, buffer, route, port,
 			space, offset, length, TB_CFG_DEFAULT_TIMEOUT);
-	if (res.err == 1) {
+	switch (res.err) {
+	case 0:
+		/* Success */
+		break;
+
+	case 1:
+		/* Thunderbolt error, tb_error holds the actual number */
 		tb_cfg_print_error(ctl, &res);
 		return -EIO;
+
+	case -ETIMEDOUT:
+		tb_ctl_warn(ctl, "timeout reading config space %u from %#x\n",
+			    space, offset);
+		break;
+
+	default:
+		WARN(1, "tb_cfg_read: %d\n", res.err);
+		break;
 	}
-	WARN(res.err, "tb_cfg_read: %d\n", res.err);
 	return res.err;
 }
 
@@ -642,11 +959,25 @@ int tb_cfg_write(struct tb_ctl *ctl, const void *buffer, u64 route, u32 port,
 {
 	struct tb_cfg_result res = tb_cfg_write_raw(ctl, buffer, route, port,
 			space, offset, length, TB_CFG_DEFAULT_TIMEOUT);
-	if (res.err == 1) {
+	switch (res.err) {
+	case 0:
+		/* Success */
+		break;
+
+	case 1:
+		/* Thunderbolt error, tb_error holds the actual number */
 		tb_cfg_print_error(ctl, &res);
 		return -EIO;
+
+	case -ETIMEDOUT:
+		tb_ctl_warn(ctl, "timeout writing config space %u to %#x\n",
+			    space, offset);
+		break;
+
+	default:
+		WARN(1, "tb_cfg_write: %d\n", res.err);
+		break;
 	}
-	WARN(res.err, "tb_cfg_write: %d\n", res.err);
 	return res.err;
 }
 
diff --git a/drivers/thunderbolt/ctl.h b/drivers/thunderbolt/ctl.h
index 2b23e030a85b..36fd28b1c1c5 100644
--- a/drivers/thunderbolt/ctl.h
+++ b/drivers/thunderbolt/ctl.h
@@ -7,6 +7,8 @@
 #ifndef _TB_CFG
 #define _TB_CFG
 
+#include <linux/kref.h>
+
 #include "nhi.h"
 #include "tb_msgs.h"
 
@@ -39,6 +41,69 @@ struct tb_cfg_result {
 	enum tb_cfg_error tb_error; /* valid if err == 1 */
 };
 
+struct ctl_pkg {
+	struct tb_ctl *ctl;
+	void *buffer;
+	struct ring_frame frame;
+};
+
+/**
+ * struct tb_cfg_request - Control channel request
+ * @kref: Reference count
+ * @ctl: Pointer to the control channel structure. Only set when the
+ *	 request is queued.
+ * @request_size: Size of the request packet (in bytes)
+ * @request_type: Type of the request packet
+ * @response: Response is stored here
+ * @response_size: Maximum size of one response packet
+ * @response_type: Expected type of the response packet
+ * @npackets: Number of packets expected to be returned with this request
+ * @match: Function used to match the incoming packet
+ * @copy: Function used to copy the incoming packet to @response
+ * @callback: Callback called when the request is finished successfully
+ * @callback_data: Data to be passed to @callback
+ * @flags: Flags for the request
+ * @work: Work item used to complete the request
+ * @result: Result after the request has been completed
+ * @list: Requests are queued using this field
+ *
+ * An arbitrary request over Thunderbolt control channel. For standard
+ * control channel message, one should use tb_cfg_read/write() and
+ * friends if possible.
+ */
+struct tb_cfg_request {
+	struct kref kref;
+	struct tb_ctl *ctl;
+	const void *request;
+	size_t request_size;
+	enum tb_cfg_pkg_type request_type;
+	void *response;
+	size_t response_size;
+	enum tb_cfg_pkg_type response_type;
+	size_t npackets;
+	bool (*match)(const struct tb_cfg_request *req,
+		      const struct ctl_pkg *pkg);
+	bool (*copy)(struct tb_cfg_request *req, const struct ctl_pkg *pkg);
+	void (*callback)(void *callback_data);
+	void *callback_data;
+	unsigned long flags;
+	struct work_struct work;
+	struct tb_cfg_result result;
+	struct list_head list;
+};
+
+#define TB_CFG_REQUEST_ACTIVE		0
+#define TB_CFG_REQUEST_CANCELED		1
+
+struct tb_cfg_request *tb_cfg_request_alloc(void);
+void tb_cfg_request_get(struct tb_cfg_request *req);
+void tb_cfg_request_put(struct tb_cfg_request *req);
+int tb_cfg_request(struct tb_ctl *ctl, struct tb_cfg_request *req,
+		   void (*callback)(void *), void *callback_data);
+void tb_cfg_request_cancel(struct tb_cfg_request *req, int err);
+struct tb_cfg_result tb_cfg_request_sync(struct tb_ctl *ctl,
+			struct tb_cfg_request *req, int timeout_msec);
+
 static inline u64 tb_cfg_get_route(const struct tb_cfg_header *header)
 {
 	return (u64) header->route_hi << 32 | header->route_lo;
diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
index 73db614fa913..0be989069941 100644
--- a/drivers/thunderbolt/tb.h
+++ b/drivers/thunderbolt/tb.h
@@ -132,7 +132,7 @@ struct tb_cm_ops {
 /**
  * struct tb - main thunderbolt bus structure
  * @dev: Domain device
- * @lock: Big lock. Must be held when accessing cfg or any struct
+ * @lock: Big lock. Must be held when accessing any struct
  *	  tb_switch / struct tb_port.
  * @nhi: Pointer to the NHI structure
  * @ctl: Control channel for this domain
-- 
2.11.0

[PATCH 16/24] thunderbolt: Add Thunderbolt 3 PCI IDs

[Mika Westerberg]

Add Intel Alpine Ridge Thunderbolt 3 controller PCI IDs to the list of
supported devices.

This code is based on the work done by Amir Levy and Michael Jamet.

Signed-off-by: Michael Jamet <michael.jamet@intel.com>
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
---
 drivers/thunderbolt/nhi.c    | 11 +++++++++++
 drivers/thunderbolt/nhi.h    | 15 +++++++++++++++
 drivers/thunderbolt/switch.c | 19 ++++++++++++++-----
 3 files changed, 40 insertions(+), 5 deletions(-)

diff --git a/drivers/thunderbolt/nhi.c b/drivers/thunderbolt/nhi.c
index bf69ee42da68..1ebc845163c9 100644
--- a/drivers/thunderbolt/nhi.c
+++ b/drivers/thunderbolt/nhi.c
@@ -789,6 +789,17 @@ static struct pci_device_id nhi_ids[] = {
 		.device = PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_NHI,
 		.subvendor = PCI_ANY_ID, .subdevice = PCI_ANY_ID,
 	},
+
+	/* Thunderbolt 3 */
+	{ PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_2C_NHI) },
+	{ PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_4C_NHI) },
+	{ PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_USBONLY_NHI) },
+	{ PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_NHI) },
+	{ PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_USBONLY_NHI) },
+	{ PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_2C_NHI) },
+	{ PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_4C_NHI) },
+	{ PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_USBONLY_NHI) },
+
 	{ 0,}
 };
 
diff --git a/drivers/thunderbolt/nhi.h b/drivers/thunderbolt/nhi.h
index 630f44140530..8a443ad67133 100644
--- a/drivers/thunderbolt/nhi.h
+++ b/drivers/thunderbolt/nhi.h
@@ -143,4 +143,19 @@ static inline int ring_tx(struct tb_ring *ring, struct ring_frame *frame)
 	return __ring_enqueue(ring, frame);
 }
 
+/*
+ * PCI IDs used in this driver from Alpine Ridge forward. There is no
+ * need for the PCI quirk anymore as we will use ICM also on Apple
+ * hardware.
+ */
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_NHI		0x15bf
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_BRIDGE	0x15c0
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_4C_NHI	0x15d2
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_4C_BRIDGE	0x15d3
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_2C_NHI	0x15d9
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_2C_BRIDGE	0x15da
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_USBONLY_NHI	0x15dc
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_USBONLY_NHI	0x15dd
+#define PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_USBONLY_NHI	0x15de
+
 #endif
diff --git a/drivers/thunderbolt/switch.c b/drivers/thunderbolt/switch.c
index b06de0efbdfc..396e00ab7723 100644
--- a/drivers/thunderbolt/switch.c
+++ b/drivers/thunderbolt/switch.c
@@ -481,13 +481,22 @@ int tb_switch_configure(struct tb_switch *sw)
 		tb_sw_warn(sw, "unknown switch vendor id %#x\n",
 			   sw->config.vendor_id);
 
-	if (sw->config.device_id != PCI_DEVICE_ID_INTEL_LIGHT_RIDGE &&
-	    sw->config.device_id != PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C &&
-	    sw->config.device_id != PCI_DEVICE_ID_INTEL_PORT_RIDGE &&
-	    sw->config.device_id != PCI_DEVICE_ID_INTEL_FALCON_RIDGE_2C_BRIDGE &&
-	    sw->config.device_id != PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_BRIDGE)
+	switch (sw->config.device_id) {
+	case PCI_DEVICE_ID_INTEL_LIGHT_RIDGE:
+	case PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C:
+	case PCI_DEVICE_ID_INTEL_PORT_RIDGE:
+	case PCI_DEVICE_ID_INTEL_FALCON_RIDGE_2C_BRIDGE:
+	case PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_BRIDGE:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_2C_BRIDGE:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_4C_BRIDGE:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_2C_BRIDGE:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_4C_BRIDGE:
+		break;
+
+	default:
 		tb_sw_warn(sw, "unsupported switch device id %#x\n",
 			   sw->config.device_id);
+	}
 
 	sw->config.enabled = 1;
 
-- 
2.11.0

[PATCH 17/24] thunderbolt: Add support for NHI mailbox

[Mika Westerberg]

The host controller includes two sets of registers that are used to
communicate with the firmware. Add functions that can be used to access
these registers.

This code is based on the work done by Amir Levy and Michael Jamet.

Signed-off-by: Michael Jamet <michael.jamet@intel.com>
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
---
 drivers/thunderbolt/nhi.c      | 58 ++++++++++++++++++++++++++++++++++++++++++
 drivers/thunderbolt/nhi.h      | 16 ++++++++++++
 drivers/thunderbolt/nhi_regs.h | 11 ++++++++
 3 files changed, 85 insertions(+)

diff --git a/drivers/thunderbolt/nhi.c b/drivers/thunderbolt/nhi.c
index 1ebc845163c9..db9dca11e5db 100644
--- a/drivers/thunderbolt/nhi.c
+++ b/drivers/thunderbolt/nhi.c
@@ -14,6 +14,7 @@
 #include <linux/interrupt.h>
 #include <linux/module.h>
 #include <linux/dmi.h>
+#include <linux/delay.h>
 
 #include "nhi.h"
 #include "nhi_regs.h"
@@ -28,6 +29,8 @@
 #define MSIX_MIN_VECS		6
 #define MSIX_MAX_VECS		16
 
+#define NHI_MAILBOX_TIMEOUT	500 /* ms */
+
 static int ring_interrupt_index(struct tb_ring *ring)
 {
 	int bit = ring->hop;
@@ -528,6 +531,61 @@ void ring_free(struct tb_ring *ring)
 	kfree(ring);
 }
 
+/**
+ * nhi_mailbox_cmd() - Send a command through NHI mailbox
+ * @nhi: Pointer to the NHI structure
+ * @cmd: Command to send
+ * @data: Data to be send with the command
+ *
+ * Sends mailbox command to the firmware running on NHI. Returns %0 in
+ * case of success and negative errno in case of failure.
+ */
+int nhi_mailbox_cmd(struct tb_nhi *nhi, enum nhi_mailbox_cmd cmd, u32 data)
+{
+	ktime_t timeout;
+	u32 val;
+
+	iowrite32(data, nhi->iobase + REG_INMAIL_DATA);
+
+	val = ioread32(nhi->iobase + REG_INMAIL_CMD);
+	val &= ~(REG_INMAIL_CMD_MASK | REG_INMAIL_ERROR);
+	val |= REG_INMAIL_OP_REQUEST | cmd;
+	iowrite32(val, nhi->iobase + REG_INMAIL_CMD);
+
+	timeout = ktime_add_ms(ktime_get(), NHI_MAILBOX_TIMEOUT);
+	do {
+		val = ioread32(nhi->iobase + REG_INMAIL_CMD);
+		if (!(val & REG_INMAIL_OP_REQUEST))
+			break;
+		usleep_range(10, 20);
+	} while (ktime_before(ktime_get(), timeout));
+
+	if (val & REG_INMAIL_OP_REQUEST)
+		return -ETIMEDOUT;
+	if (val & REG_INMAIL_ERROR)
+		return -EIO;
+
+	return 0;
+}
+
+/**
+ * nhi_mailbox_mode() - Return current firmware operation mode
+ * @nhi: Pointer to the NHI structure
+ *
+ * The function reads current firmware operation mode using NHI mailbox
+ * registers and returns it to the caller.
+ */
+enum nhi_fw_mode nhi_mailbox_mode(struct tb_nhi *nhi)
+{
+	u32 val;
+
+	val = ioread32(nhi->iobase + REG_OUTMAIL_CMD);
+	val &= REG_OUTMAIL_CMD_OPMODE_MASK;
+	val >>= REG_OUTMAIL_CMD_OPMODE_SHIFT;
+
+	return (enum nhi_fw_mode)val;
+}
+
 static void nhi_interrupt_work(struct work_struct *work)
 {
 	struct tb_nhi *nhi = container_of(work, typeof(*nhi), interrupt_work);
diff --git a/drivers/thunderbolt/nhi.h b/drivers/thunderbolt/nhi.h
index 8a443ad67133..730ece3a8a5a 100644
--- a/drivers/thunderbolt/nhi.h
+++ b/drivers/thunderbolt/nhi.h
@@ -143,6 +143,22 @@ static inline int ring_tx(struct tb_ring *ring, struct ring_frame *frame)
 	return __ring_enqueue(ring, frame);
 }
 
+enum nhi_fw_mode {
+	NHI_FW_SAFE_MODE,
+	NHI_FW_AUTH_MODE,
+	NHI_FW_EP_MODE,
+	NHI_FW_CM_MODE,
+};
+
+enum nhi_mailbox_cmd {
+	NHI_MAILBOX_SAVE_DEVS = 0x05,
+	NHI_MAILBOX_DRV_UNLOADS = 0x07,
+	NHI_MAILBOX_ALLOW_ALL_DEVS = 0x23,
+};
+
+int nhi_mailbox_cmd(struct tb_nhi *nhi, enum nhi_mailbox_cmd cmd, u32 data);
+enum nhi_fw_mode nhi_mailbox_mode(struct tb_nhi *nhi);
+
 /*
  * PCI IDs used in this driver from Alpine Ridge forward. There is no
  * need for the PCI quirk anymore as we will use ICM also on Apple
diff --git a/drivers/thunderbolt/nhi_regs.h b/drivers/thunderbolt/nhi_regs.h
index 48b98d3c7e6a..322fe1fa3a3c 100644
--- a/drivers/thunderbolt/nhi_regs.h
+++ b/drivers/thunderbolt/nhi_regs.h
@@ -107,4 +107,15 @@ struct ring_desc {
 #define REG_DMA_MISC			0x39864
 #define REG_DMA_MISC_INT_AUTO_CLEAR     BIT(2)
 
+#define REG_INMAIL_DATA			0x39900
+
+#define REG_INMAIL_CMD			0x39904
+#define REG_INMAIL_CMD_MASK		GENMASK(7, 0)
+#define REG_INMAIL_ERROR		BIT(30)
+#define REG_INMAIL_OP_REQUEST		BIT(31)
+
+#define REG_OUTMAIL_CMD			0x3990c
+#define REG_OUTMAIL_CMD_OPMODE_SHIFT	8
+#define REG_OUTMAIL_CMD_OPMODE_MASK	GENMASK(11, 8)
+
 #endif
-- 
2.11.0

[PATCH 18/24] thunderbolt: Store Thunderbolt generation in the switch structure

[Mika Westerberg]

In some cases it is useful to know what is the Thunderbolt generation
the switch supports. This introduces a new field to struct switch that
stores the generation of the switch based on the device ID.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 drivers/thunderbolt/switch.c | 24 ++++++++++++++++++++++++
 drivers/thunderbolt/tb.h     |  2 ++
 2 files changed, 26 insertions(+)

diff --git a/drivers/thunderbolt/switch.c b/drivers/thunderbolt/switch.c
index 396e00ab7723..9c91d397d3b3 100644
--- a/drivers/thunderbolt/switch.c
+++ b/drivers/thunderbolt/switch.c
@@ -382,6 +382,28 @@ struct device_type tb_switch_type = {
 	.release = tb_switch_release,
 };
 
+static void tb_switch_set_generation(struct tb_switch *sw)
+{
+	switch (sw->config.device_id) {
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_BRIDGE:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_2C_BRIDGE:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_4C_BRIDGE:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_2C_BRIDGE:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_4C_BRIDGE:
+		sw->generation = 3;
+		break;
+
+	case PCI_DEVICE_ID_INTEL_FALCON_RIDGE_2C_BRIDGE:
+	case PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_BRIDGE:
+		sw->generation = 2;
+		break;
+
+	default:
+		sw->generation = 1;
+		break;
+	}
+}
+
 /**
  * tb_switch_alloc() - allocate a switch
  * @tb: Pointer to the owning domain
@@ -442,6 +464,8 @@ struct tb_switch *tb_switch_alloc(struct tb *tb, struct device *parent,
 	}
 	sw->cap_plug_events = cap;
 
+	tb_switch_set_generation(sw);
+
 	device_initialize(&sw->dev);
 	sw->dev.parent = parent;
 	sw->dev.bus = &tb_bus_type;
diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
index 0be989069941..b3cda7605619 100644
--- a/drivers/thunderbolt/tb.h
+++ b/drivers/thunderbolt/tb.h
@@ -25,6 +25,7 @@
  * @device: Device ID of the switch
  * @vendor_name: Name of the vendor (or %NULL if not known)
  * @device_name: Name of the device (or %NULL if not known)
+ * @generation: Switch Thunderbolt generation
  * @cap_plug_events: Offset to the plug events capability (%0 if not found)
  * @is_unplugged: The switch is going away
  * @drom: DROM of the switch (%NULL if not found)
@@ -40,6 +41,7 @@ struct tb_switch {
 	u16 device;
 	const char *vendor_name;
 	const char *device_name;
+	unsigned int generation;
 	int cap_plug_events;
 	bool is_unplugged;
 	u8 *drom;
-- 
2.11.0

[PATCH 19/24] thunderbolt: Add support for DMA configuration based mailbox

[Mika Westerberg]

The DMA (NHI) port of a switch provides access to the NVM of the host
controller (and devices starting from Intel Alpine Ridge). The NVM
contains also more complete DROM for the root switch including vendor
and device identification strings.

This will look for the DMA port capability for each switch and if found
populates sw->dma_port. We then teach tb_drom_read() to read the DROM
information from NVM if available for the root switch.

The DMA port capability also supports upgrading the NVM for both host
controller and devices which will be added in subsequent patches.

This code is based on the work done by Amir Levy and Michael Jamet.

Signed-off-by: Michael Jamet <michael.jamet@intel.com>
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
---
 drivers/thunderbolt/Makefile   |   2 +-
 drivers/thunderbolt/dma_port.c | 524 +++++++++++++++++++++++++++++++++++++++++
 drivers/thunderbolt/dma_port.h |  34 +++
 drivers/thunderbolt/eeprom.c   |  51 +++-
 drivers/thunderbolt/switch.c   |  30 +++
 drivers/thunderbolt/tb.h       |   5 +
 6 files changed, 644 insertions(+), 2 deletions(-)
 create mode 100644 drivers/thunderbolt/dma_port.c
 create mode 100644 drivers/thunderbolt/dma_port.h

diff --git a/drivers/thunderbolt/Makefile b/drivers/thunderbolt/Makefile
index e276a9a62261..9828e862dd35 100644
--- a/drivers/thunderbolt/Makefile
+++ b/drivers/thunderbolt/Makefile
@@ -1,3 +1,3 @@
 obj-${CONFIG_THUNDERBOLT} := thunderbolt.o
 thunderbolt-objs := nhi.o ctl.o tb.o switch.o cap.o path.o tunnel_pci.o eeprom.o
-thunderbolt-objs += domain.o
+thunderbolt-objs += domain.o dma_port.o
diff --git a/drivers/thunderbolt/dma_port.c b/drivers/thunderbolt/dma_port.c
new file mode 100644
index 000000000000..af6dde347bee
--- /dev/null
+++ b/drivers/thunderbolt/dma_port.c
@@ -0,0 +1,524 @@
+/*
+ * Thunderbolt DMA configuration based mailbox support
+ *
+ * Copyright (C) 2017, Intel Corporation
+ * Authors: Michael Jamet <michael.jamet@intel.com>
+ *          Mika Westerberg <mika.westerberg@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/delay.h>
+#include <linux/slab.h>
+
+#include "dma_port.h"
+#include "tb_regs.h"
+
+#define DMA_PORT_CAP			0x3e
+
+#define MAIL_DATA			1
+#define MAIL_DATA_DWORDS		16
+
+#define MAIL_IN				17
+#define MAIL_IN_CMD_SHIFT		28
+#define MAIL_IN_CMD_MASK		GENMASK(31, 28)
+#define MAIL_IN_CMD_FLASH_WRITE		0x0
+#define MAIL_IN_CMD_FLASH_UPDATE_AUTH	0x1
+#define MAIL_IN_CMD_FLASH_READ		0x2
+#define MAIL_IN_CMD_POWER_CYCLE		0x4
+#define MAIL_IN_DWORDS_SHIFT		24
+#define MAIL_IN_DWORDS_MASK		GENMASK(27, 24)
+#define MAIL_IN_ADDRESS_SHIFT		2
+#define MAIL_IN_ADDRESS_MASK		GENMASK(23, 2)
+#define MAIL_IN_CSS			BIT(1)
+#define MAIL_IN_OP_REQUEST		BIT(0)
+
+#define MAIL_OUT			18
+#define MAIL_OUT_STATUS_RESPONSE	BIT(29)
+#define MAIL_OUT_STATUS_CMD_SHIFT	4
+#define MAIL_OUT_STATUS_CMD_MASK	GENMASK(7, 4)
+#define MAIL_OUT_STATUS_MASK		GENMASK(3, 0)
+#define MAIL_OUT_STATUS_COMPLETED	0
+#define MAIL_OUT_STATUS_ERR_AUTH	1
+#define MAIL_OUT_STATUS_ERR_ACCESS	2
+
+#define DMA_PORT_TIMEOUT		5000 /* ms */
+#define DMA_PORT_RETRIES		3
+
+/**
+ * struct tb_dma_port - DMA control port
+ * @sw: Switch the DMA port belongs to
+ * @port: Switch port number where DMA capability is found
+ * @base: Start offset of the mailbox registers
+ * @buf: Temporary buffer to store a single block
+ */
+struct tb_dma_port {
+	struct tb_switch *sw;
+	u8 port;
+	u32 base;
+	u8 *buf;
+};
+
+/*
+ * When the switch is in safe mode it supports very little functionality
+ * so we don't validate that much here.
+ */
+static bool dma_port_match(const struct tb_cfg_request *req,
+			   const struct ctl_pkg *pkg)
+{
+	u64 route = tb_cfg_get_route(pkg->buffer) & ~BIT_ULL(63);
+
+	if (pkg->frame.eof == TB_CFG_PKG_ERROR)
+		return true;
+	if (pkg->frame.eof != req->response_type)
+		return false;
+	if (route != tb_cfg_get_route(req->request))
+		return false;
+	if (pkg->frame.size != req->response_size)
+		return false;
+
+	return true;
+}
+
+static bool dma_port_copy(struct tb_cfg_request *req, const struct ctl_pkg *pkg)
+{
+	memcpy(req->response, pkg->buffer, req->response_size);
+	return true;
+}
+
+static int dma_port_read(struct tb_ctl *ctl, void *buffer, u64 route,
+			 u32 port, u32 offset, u32 length, int timeout_msec)
+{
+	struct cfg_read_pkg request = {
+		.header = tb_cfg_make_header(route),
+		.addr = {
+			.seq = 1,
+			.port = port,
+			.space = TB_CFG_PORT,
+			.offset = offset,
+			.length = length,
+		},
+	};
+	struct tb_cfg_request *req;
+	struct cfg_write_pkg reply;
+	struct tb_cfg_result res;
+
+	req = tb_cfg_request_alloc();
+	if (!req)
+		return -ENOMEM;
+
+	req->match = dma_port_match;
+	req->copy = dma_port_copy;
+	req->request = &request;
+	req->request_size = sizeof(request);
+	req->request_type = TB_CFG_PKG_READ;
+	req->response = &reply;
+	req->response_size = 12 + 4 * length;
+	req->response_type = TB_CFG_PKG_READ;
+
+	res = tb_cfg_request_sync(ctl, req, timeout_msec);
+
+	tb_cfg_request_put(req);
+
+	if (res.err)
+		return res.err;
+
+	memcpy(buffer, &reply.data, 4 * length);
+	return 0;
+}
+
+static int dma_port_write(struct tb_ctl *ctl, const void *buffer, u64 route,
+			  u32 port, u32 offset, u32 length, int timeout_msec)
+{
+	struct cfg_write_pkg request = {
+		.header = tb_cfg_make_header(route),
+		.addr = {
+			.seq = 1,
+			.port = port,
+			.space = TB_CFG_PORT,
+			.offset = offset,
+			.length = length,
+		},
+	};
+	struct tb_cfg_request *req;
+	struct cfg_read_pkg reply;
+	struct tb_cfg_result res;
+
+	memcpy(&request.data, buffer, length * 4);
+
+	req = tb_cfg_request_alloc();
+	if (!req)
+		return -ENOMEM;
+
+	req->match = dma_port_match;
+	req->copy = dma_port_copy;
+	req->request = &request;
+	req->request_size = 12 + 4 * length;
+	req->request_type = TB_CFG_PKG_WRITE;
+	req->response = &reply;
+	req->response_size = sizeof(reply);
+	req->response_type = TB_CFG_PKG_WRITE;
+
+	res = tb_cfg_request_sync(ctl, req, timeout_msec);
+
+	tb_cfg_request_put(req);
+
+	return res.err;
+}
+
+static int dma_find_port(struct tb_switch *sw)
+{
+	int port, ret;
+	u32 type;
+
+	/*
+	 * The DMA (NHI) port is either 3 or 5 depending on the
+	 * controller. Try both starting from 5 which is more common.
+	 */
+	port = 5;
+	ret = dma_port_read(sw->tb->ctl, &type, tb_route(sw), port, 2, 1,
+			    DMA_PORT_TIMEOUT);
+	if (!ret && (type & 0xffffff) == TB_TYPE_NHI)
+		return port;
+
+	port = 3;
+	ret = dma_port_read(sw->tb->ctl, &type, tb_route(sw), port, 2, 1,
+			    DMA_PORT_TIMEOUT);
+	if (!ret && (type & 0xffffff) == TB_TYPE_NHI)
+		return port;
+
+	return -ENODEV;
+}
+
+/**
+ * dma_port_alloc() - Finds DMA control port from a switch pointed by route
+ * @sw: Switch from where find the DMA port
+ *
+ * Function checks if the switch NHI port supports DMA configuration
+ * based mailbox capability and if it does, allocates and initializes
+ * DMA port structure. Returns %NULL if the capabity was not found.
+ *
+ * The DMA control port is functional also when the switch is in safe
+ * mode.
+ */
+struct tb_dma_port *dma_port_alloc(struct tb_switch *sw)
+{
+	struct tb_dma_port *dma;
+	int port;
+
+	port = dma_find_port(sw);
+	if (port < 0)
+		return NULL;
+
+	dma = kzalloc(sizeof(*dma), GFP_KERNEL);
+	if (!dma)
+		return NULL;
+
+	dma->buf = kmalloc_array(MAIL_DATA_DWORDS, sizeof(u32), GFP_KERNEL);
+	if (!dma->buf) {
+		kfree(dma);
+		return NULL;
+	}
+
+	dma->sw = sw;
+	dma->port = port;
+	dma->base = DMA_PORT_CAP;
+
+	return dma;
+}
+
+/**
+ * dma_port_free() - Release DMA control port structure
+ * @dma: DMA control port
+ */
+void dma_port_free(struct tb_dma_port *dma)
+{
+	if (dma) {
+		kfree(dma->buf);
+		kfree(dma);
+	}
+}
+
+static int dma_port_wait_for_completion(struct tb_dma_port *dma,
+					unsigned int timeout)
+{
+	unsigned long end = jiffies + msecs_to_jiffies(timeout);
+	struct tb_switch *sw = dma->sw;
+
+	do {
+		int ret;
+		u32 in;
+
+		ret = dma_port_read(sw->tb->ctl, &in, tb_route(sw), dma->port,
+				    dma->base + MAIL_IN, 1, 50);
+		if (ret) {
+			if (ret != -ETIMEDOUT)
+				return ret;
+		} else if (!(in & MAIL_IN_OP_REQUEST)) {
+			return 0;
+		}
+
+		usleep_range(50, 100);
+	} while (time_before(jiffies, end));
+
+	return -ETIMEDOUT;
+}
+
+static int status_to_errno(u32 status)
+{
+	switch (status & MAIL_OUT_STATUS_MASK) {
+	case MAIL_OUT_STATUS_COMPLETED:
+		return 0;
+	case MAIL_OUT_STATUS_ERR_AUTH:
+		return -EINVAL;
+	case MAIL_OUT_STATUS_ERR_ACCESS:
+		return -EACCES;
+	}
+
+	return -EIO;
+}
+
+static int dma_port_request(struct tb_dma_port *dma, u32 in,
+			    unsigned int timeout)
+{
+	struct tb_switch *sw = dma->sw;
+	u32 out;
+	int ret;
+
+	ret = dma_port_write(sw->tb->ctl, &in, tb_route(sw), dma->port,
+			     dma->base + MAIL_IN, 1, DMA_PORT_TIMEOUT);
+	if (ret)
+		return ret;
+
+	ret = dma_port_wait_for_completion(dma, timeout);
+	if (ret)
+		return ret;
+
+	ret = dma_port_read(sw->tb->ctl, &out, tb_route(sw), dma->port,
+			    dma->base + MAIL_OUT, 1, DMA_PORT_TIMEOUT);
+	if (ret)
+		return ret;
+
+	return status_to_errno(out);
+}
+
+static int dma_port_flash_read_block(struct tb_dma_port *dma, u32 address,
+				     void *buf, u32 size)
+{
+	struct tb_switch *sw = dma->sw;
+	u32 in, dwaddress, dwords;
+	int ret;
+
+	dwaddress = address / 4;
+	dwords = size / 4;
+
+	in = MAIL_IN_CMD_FLASH_READ << MAIL_IN_CMD_SHIFT;
+	if (dwords < MAIL_DATA_DWORDS)
+		in |= (dwords << MAIL_IN_DWORDS_SHIFT) & MAIL_IN_DWORDS_MASK;
+	in |= (dwaddress << MAIL_IN_ADDRESS_SHIFT) & MAIL_IN_ADDRESS_MASK;
+	in |= MAIL_IN_OP_REQUEST;
+
+	ret = dma_port_request(dma, in, DMA_PORT_TIMEOUT);
+	if (ret)
+		return ret;
+
+	return dma_port_read(sw->tb->ctl, buf, tb_route(sw), dma->port,
+			     dma->base + MAIL_DATA, dwords, DMA_PORT_TIMEOUT);
+}
+
+static int dma_port_flash_write_block(struct tb_dma_port *dma, u32 address,
+				      const void *buf, u32 size)
+{
+	struct tb_switch *sw = dma->sw;
+	u32 in, dwaddress, dwords;
+	int ret;
+
+	dwords = size / 4;
+
+	/* Write the block to MAIL_DATA registers */
+	ret = dma_port_write(sw->tb->ctl, buf, tb_route(sw), dma->port,
+			    dma->base + MAIL_DATA, dwords, DMA_PORT_TIMEOUT);
+
+	in = MAIL_IN_CMD_FLASH_WRITE << MAIL_IN_CMD_SHIFT;
+
+	/* CSS header write is always done to the same magic address */
+	if (address >= DMA_PORT_CSS_ADDRESS) {
+		dwaddress = DMA_PORT_CSS_ADDRESS;
+		in |= MAIL_IN_CSS;
+	} else {
+		dwaddress = address / 4;
+	}
+
+	in |= ((dwords - 1) << MAIL_IN_DWORDS_SHIFT) & MAIL_IN_DWORDS_MASK;
+	in |= (dwaddress << MAIL_IN_ADDRESS_SHIFT) & MAIL_IN_ADDRESS_MASK;
+	in |= MAIL_IN_OP_REQUEST;
+
+	return dma_port_request(dma, in, DMA_PORT_TIMEOUT);
+}
+
+/**
+ * dma_port_flash_read() - Read from active flash region
+ * @dma: DMA control port
+ * @address: Address relative to the start of active region
+ * @buf: Buffer where the data is read
+ * @size: Size of the buffer
+ */
+int dma_port_flash_read(struct tb_dma_port *dma, unsigned int address,
+			void *buf, size_t size)
+{
+	unsigned int retries = DMA_PORT_RETRIES;
+	unsigned int offset;
+
+	offset = address & 3;
+	address = address & ~3;
+
+	do {
+		u32 nbytes = min_t(u32, size, MAIL_DATA_DWORDS * 4);
+		int ret;
+
+		ret = dma_port_flash_read_block(dma, address, dma->buf,
+						ALIGN(nbytes, 4));
+		if (ret) {
+			if (ret == -ETIMEDOUT) {
+				if (retries--)
+					continue;
+				ret = -EIO;
+			}
+			return ret;
+		}
+
+		memcpy(buf, dma->buf + offset, nbytes);
+
+		size -= nbytes;
+		address += nbytes;
+		buf += nbytes;
+	} while (size > 0);
+
+	return 0;
+}
+
+/**
+ * dma_port_flash_write() - Write to non-active flash region
+ * @dma: DMA control port
+ * @address: Address relative to the start of non-active region
+ * @buf: Data to write
+ * @size: Size of the buffer
+ *
+ * Writes block of data to the non-active flash region of the switch. If
+ * the address is given as %DMA_PORT_CSS_ADDRESS the block is written
+ * using CSS command.
+ */
+int dma_port_flash_write(struct tb_dma_port *dma, unsigned int address,
+			 const void *buf, size_t size)
+{
+	unsigned int retries = DMA_PORT_RETRIES;
+	unsigned int offset;
+
+	if (address >= DMA_PORT_CSS_ADDRESS) {
+		offset = 0;
+		if (size > DMA_PORT_CSS_MAX_SIZE)
+			return -E2BIG;
+	} else {
+		offset = address & 3;
+		address = address & ~3;
+	}
+
+	do {
+		u32 nbytes = min_t(u32, size, MAIL_DATA_DWORDS * 4);
+		int ret;
+
+		memcpy(dma->buf + offset, buf, nbytes);
+
+		ret = dma_port_flash_write_block(dma, address, buf, nbytes);
+		if (ret) {
+			if (ret == -ETIMEDOUT) {
+				if (retries--)
+					continue;
+				ret = -EIO;
+			}
+			return ret;
+		}
+
+		size -= nbytes;
+		address += nbytes;
+		buf += nbytes;
+	} while (size > 0);
+
+	return 0;
+}
+
+/**
+ * dma_port_flash_update_auth() - Starts flash authenticate cycle
+ * @dma: DMA control port
+ *
+ * Starts the flash update authentication cycle. If the image in the
+ * non-active area was valid, the switch starts upgrade process where
+ * active and non-active area get swapped in the end. Caller should call
+ * dma_port_flash_update_auth_status() to get status of this command.
+ * This is because if the switch in question is root switch the
+ * thunderbolt host controller gets reset as well.
+ */
+int dma_port_flash_update_auth(struct tb_dma_port *dma)
+{
+	u32 in;
+
+	in = MAIL_IN_CMD_FLASH_UPDATE_AUTH << MAIL_IN_CMD_SHIFT;
+	in |= MAIL_IN_OP_REQUEST;
+
+	return dma_port_request(dma, in, 150);
+}
+
+/**
+ * dma_port_flash_update_auth_status() - Reads status of update auth command
+ * @dma: DMA control port
+ * @status: Status code of the operation
+ *
+ * The function checks if there is status available from the last update
+ * auth command. Returns %0 if there is no status and no further
+ * action is required. If there is status, %1 is returned instead and
+ * @status holds the failure code.
+ *
+ * Negative return means there was an error reading status from the
+ * switch.
+ */
+int dma_port_flash_update_auth_status(struct tb_dma_port *dma, u32 *status)
+{
+	struct tb_switch *sw = dma->sw;
+	u32 out, cmd;
+	int ret;
+
+	ret = dma_port_read(sw->tb->ctl, &out, tb_route(sw), dma->port,
+			    dma->base + MAIL_OUT, 1, DMA_PORT_TIMEOUT);
+	if (ret)
+		return ret;
+
+	/* Check if the status relates to flash update auth */
+	cmd = (out & MAIL_OUT_STATUS_CMD_MASK) >> MAIL_OUT_STATUS_CMD_SHIFT;
+	if (cmd == MAIL_IN_CMD_FLASH_UPDATE_AUTH) {
+		if (status)
+			*status = out & MAIL_OUT_STATUS_MASK;
+
+		/* Reset is needed in any case */
+		return 1;
+	}
+
+	return 0;
+}
+
+/**
+ * dma_port_power_cycle() - Power cycles the switch
+ * @dma: DMA control port
+ *
+ * Triggers power cycle to the switch.
+ */
+int dma_port_power_cycle(struct tb_dma_port *dma)
+{
+	u32 in;
+
+	in = MAIL_IN_CMD_POWER_CYCLE << MAIL_IN_CMD_SHIFT;
+	in |= MAIL_IN_OP_REQUEST;
+
+	return dma_port_request(dma, in, 150);
+}
diff --git a/drivers/thunderbolt/dma_port.h b/drivers/thunderbolt/dma_port.h
new file mode 100644
index 000000000000..c4a69e0fbff7
--- /dev/null
+++ b/drivers/thunderbolt/dma_port.h
@@ -0,0 +1,34 @@
+/*
+ * Thunderbolt DMA configuration based mailbox support
+ *
+ * Copyright (C) 2017, Intel Corporation
+ * Authors: Michael Jamet <michael.jamet@intel.com>
+ *          Mika Westerberg <mika.westerberg@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef DMA_PORT_H_
+#define DMA_PORT_H_
+
+#include "tb.h"
+
+struct tb_switch;
+struct tb_dma_port;
+
+#define DMA_PORT_CSS_ADDRESS		0x3fffff
+#define DMA_PORT_CSS_MAX_SIZE		SZ_128
+
+struct tb_dma_port *dma_port_alloc(struct tb_switch *sw);
+void dma_port_free(struct tb_dma_port *dma);
+int dma_port_flash_read(struct tb_dma_port *dma, unsigned int address,
+			void *buf, size_t size);
+int dma_port_flash_update_auth(struct tb_dma_port *dma);
+int dma_port_flash_update_auth_status(struct tb_dma_port *dma, u32 *status);
+int dma_port_flash_write(struct tb_dma_port *dma, unsigned int address,
+			 const void *buf, size_t size);
+int dma_port_power_cycle(struct tb_dma_port *dma);
+
+#endif
diff --git a/drivers/thunderbolt/eeprom.c b/drivers/thunderbolt/eeprom.c
index f688fb255042..d37befc33674 100644
--- a/drivers/thunderbolt/eeprom.c
+++ b/drivers/thunderbolt/eeprom.c
@@ -418,6 +418,50 @@ static int tb_drom_copy_efi(struct tb_switch *sw, u16 *size)
 	return -EINVAL;
 }
 
+static int tb_drom_copy_nvm(struct tb_switch *sw, u16 *size)
+{
+	u32 drom_offset;
+	int ret;
+
+	if (!sw->dma_port)
+		return -ENODEV;
+
+	ret = tb_sw_read(sw, &drom_offset, TB_CFG_SWITCH,
+			 sw->cap_plug_events + 12, 1);
+	if (ret)
+		return ret;
+
+	if (!drom_offset)
+		return -ENODEV;
+
+	ret = dma_port_flash_read(sw->dma_port, drom_offset + 14, size,
+				  sizeof(*size));
+	if (ret)
+		return ret;
+
+	/* Size includes CRC8 + UID + CRC32 */
+	*size += 1 + 8 + 4;
+	sw->drom = kzalloc(*size, GFP_KERNEL);
+	if (!sw->drom)
+		return -ENOMEM;
+
+	ret = dma_port_flash_read(sw->dma_port, drom_offset, sw->drom, *size);
+	if (ret)
+		goto err_free;
+
+	/*
+	 * Read UID from the minimal DROM because the one in NVM is just
+	 * a placeholder.
+	 */
+	tb_drom_read_uid_only(sw, &sw->uid);
+	return 0;
+
+err_free:
+	kfree(sw->drom);
+	sw->drom = NULL;
+	return ret;
+}
+
 /**
  * tb_drom_read - copy drom to sw->drom and parse it
  */
@@ -439,6 +483,10 @@ int tb_drom_read(struct tb_switch *sw)
 		if (tb_drom_copy_efi(sw, &size) == 0)
 			goto parse;
 
+		/* Non-Apple hardware has the DROM as part of NVM */
+		if (tb_drom_copy_nvm(sw, &size) == 0)
+			goto parse;
+
 		/*
 		 * The root switch contains only a dummy drom (header only,
 		 * no entries). Hardcode the configuration here.
@@ -499,7 +547,8 @@ int tb_drom_read(struct tb_switch *sw)
 			header->uid_crc8, crc);
 		goto err;
 	}
-	sw->uid = header->uid;
+	if (!sw->uid)
+		sw->uid = header->uid;
 	sw->vendor = header->vendor_id;
 	sw->device = header->model_id;
 
diff --git a/drivers/thunderbolt/switch.c b/drivers/thunderbolt/switch.c
index 9c91d397d3b3..a75db3fa5e7a 100644
--- a/drivers/thunderbolt/switch.c
+++ b/drivers/thunderbolt/switch.c
@@ -369,6 +369,8 @@ static void tb_switch_release(struct device *dev)
 {
 	struct tb_switch *sw = tb_to_switch(dev);
 
+	dma_port_free(sw->dma_port);
+
 	kfree(sw->uuid);
 	kfree(sw->device_name);
 	kfree(sw->vendor_name);
@@ -560,6 +562,25 @@ static void tb_switch_set_uuid(struct tb_switch *sw)
 	sw->uuid = kmemdup(uuid, sizeof(uuid), GFP_KERNEL);
 }
 
+static void tb_switch_add_dma_port(struct tb_switch *sw)
+{
+	switch (sw->generation) {
+	case 3:
+		break;
+
+	case 2:
+		/* Only root switch can be upgraded */
+		if (tb_route(sw))
+			return;
+		break;
+
+	default:
+		return;
+	}
+
+	sw->dma_port = dma_port_alloc(sw);
+}
+
 /**
  * tb_switch_add() - Add a switch to the domain
  * @sw: Switch to add
@@ -576,6 +597,15 @@ int tb_switch_add(struct tb_switch *sw)
 {
 	int i, ret;
 
+	/*
+	 * Initialize DMA control port now before we read DROM. Recent
+	 * host controllers have more complete DROM on NVM that includes
+	 * vendor and model identification strings which we then expose
+	 * to the userspace. NVM can be accessed through DMA
+	 * configuration based mailbox.
+	 */
+	tb_switch_add_dma_port(sw);
+
 	/* read drom */
 	ret = tb_drom_read(sw);
 	if (ret) {
diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
index b3cda7605619..3a7d14b9b7d0 100644
--- a/drivers/thunderbolt/tb.h
+++ b/drivers/thunderbolt/tb.h
@@ -12,12 +12,16 @@
 
 #include "tb_regs.h"
 #include "ctl.h"
+#include "dma_port.h"
 
 /**
  * struct tb_switch - a thunderbolt switch
  * @dev: Device for the switch
  * @config: Switch configuration
  * @ports: Ports in this switch
+ * @dma_port: If the switch has port supporting DMA configuration based
+ *	      mailbox this will hold the pointer to that. Otherwise
+ *	      will be set to %NULL.
  * @tb: Pointer to the domain the switch belongs to
  * @uid: Unique ID of the switch
  * @uuid: UUID of the switch (or %NULL if not supported)
@@ -34,6 +38,7 @@ struct tb_switch {
 	struct device dev;
 	struct tb_regs_switch_header config;
 	struct tb_port *ports;
+	struct tb_dma_port *dma_port;
 	struct tb *tb;
 	u64 uid;
 	uuid_be *uuid;
-- 
2.11.0

[PATCH 20/24] thunderbolt: Do not touch the hardware if the NHI is gone on resume

[Mika Westerberg]

On PCs the NHI host controller is only present when there is a device
connected. When the last device is disconnected the host controller will
dissappear shortly (within 10s). Now if that happens when we are
suspended we should not try to touch the hardware anymore, so add a flag
for this and check it before we re-enable rings.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 drivers/thunderbolt/nhi.c | 14 ++++++++++++++
 drivers/thunderbolt/nhi.h |  3 +++
 2 files changed, 17 insertions(+)

diff --git a/drivers/thunderbolt/nhi.c b/drivers/thunderbolt/nhi.c
index db9dca11e5db..c11e4009325b 100644
--- a/drivers/thunderbolt/nhi.c
+++ b/drivers/thunderbolt/nhi.c
@@ -404,6 +404,8 @@ void ring_start(struct tb_ring *ring)
 {
 	mutex_lock(&ring->nhi->lock);
 	mutex_lock(&ring->lock);
+	if (ring->nhi->going_away)
+		goto err;
 	if (ring->running) {
 		dev_WARN(&ring->nhi->pdev->dev, "ring already started\n");
 		goto err;
@@ -451,6 +453,8 @@ void ring_stop(struct tb_ring *ring)
 	mutex_lock(&ring->lock);
 	dev_info(&ring->nhi->pdev->dev, "stopping %s %d\n",
 		 RING_TYPE(ring), ring->hop);
+	if (ring->nhi->going_away)
+		goto err;
 	if (!ring->running) {
 		dev_WARN(&ring->nhi->pdev->dev, "%s %d already stopped\n",
 			 RING_TYPE(ring), ring->hop);
@@ -655,6 +659,16 @@ static int nhi_resume_noirq(struct device *dev)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
 	struct tb *tb = pci_get_drvdata(pdev);
+	u32 vid;
+
+	/*
+	 * Check that the device is still there. It may be that the user
+	 * unplugged last device which causes the host controller to go
+	 * away on PCs.
+	 */
+	pci_read_config_dword(pdev, PCI_VENDOR_ID, &vid);
+	if (vid == ~0)
+		tb->nhi->going_away = true;
 
 	return tb_domain_resume_noirq(tb);
 }
diff --git a/drivers/thunderbolt/nhi.h b/drivers/thunderbolt/nhi.h
index 730ece3a8a5a..243f5fcfa367 100644
--- a/drivers/thunderbolt/nhi.h
+++ b/drivers/thunderbolt/nhi.h
@@ -20,6 +20,8 @@
  * @tx_rings: All Tx rings available on this host controller
  * @rx_rings: All Rx rings available on this host controller
  * @msix_ida: Used to allocate MSI-X vectors for rings
+ * @going_away: The host controller device is about to disappear so when
+ *		this flag is set, avoid touching the hardware anymore.
  * @interrupt_work: Work scheduled to handle ring interrupt when no
  *		    MSI-X is used.
  * @hop_count: Number of rings (end point hops) supported by NHI.
@@ -31,6 +33,7 @@ struct tb_nhi {
 	struct tb_ring **tx_rings;
 	struct tb_ring **rx_rings;
 	struct ida msix_ida;
+	bool going_away;
 	struct work_struct interrupt_work;
 	u32 hop_count;
 };
-- 
2.11.0

[PATCH 21/24] thunderbolt: Add support for Internal Connection Manager (ICM)

[Mika Westerberg]

Starting from Intel Falcon Ridge the internal connection manager running
on the Thunderbolt host controller has been supporting 4 security
levels. One reason for this is to prevent DMA attacks and only allow
connecting devices the user trusts.

The internal connection manager (ICM) is the preferred way of connecting
Thunderbolt devices over software only implementation typically used on
Macs. The driver communicates with ICM using special Thunderbolt ring 0
(control channel) messages. In order to handle these messages we add
support for the ICM messages to the control channel.

The security levels are as follows:

  none - No security, all tunnels are created automatically
  user - User needs to approve the device before tunnels are created
  secure - User need to approve the device before tunnels are created.
	   The device is sent a challenge on future connects to be able
	   to verify it is actually the approved device.
  dponly - Only Display Port and USB tunnels can be created and those
           are created automatically.

The security levels are typically configurable from the system BIOS and
by default it is set to "user" on many systems.

In this patch each Thunderbolt device will have either one or two new
sysfs attributes: authorized and key. The latter appears for devices
that support secure connect.

In order to identify the device the user can read identication
information, including UUID and name of the device from sysfs and based
on that make a decision to authorize the device. The device is
authorized by simply writing 1 to the "authorized" sysfs attribute. This
is following the USB bus device authorization mechanism. The secure
connect requires an additional challenge step (writing 2 to the
"authorized" attribute) in future connects when the key has already been
stored to the NVM of the device.

Non-ICM systems (before Alpine Ridge) continue to use the existing
functionality and the security level is set to none. For systems with
Alpine Ridge, even on Apple hardware, we will use ICM.

This code is based on the work done by Amir Levy and Michael Jamet.

Signed-off-by: Michael Jamet <michael.jamet@intel.com>
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
---
 Documentation/ABI/testing/sysfs-bus-thunderbolt |   48 +
 drivers/thunderbolt/Kconfig                     |   12 +-
 drivers/thunderbolt/Makefile                    |    2 +-
 drivers/thunderbolt/ctl.c                       |    2 +
 drivers/thunderbolt/domain.c                    |  195 +++++
 drivers/thunderbolt/icm.c                       | 1067 +++++++++++++++++++++++
 drivers/thunderbolt/nhi.c                       |   33 +-
 drivers/thunderbolt/nhi_regs.h                  |    7 +
 drivers/thunderbolt/switch.c                    |  218 ++++-
 drivers/thunderbolt/tb.c                        |    7 +
 drivers/thunderbolt/tb.h                        |   72 ++
 drivers/thunderbolt/tb_msgs.h                   |  152 ++++
 12 files changed, 1802 insertions(+), 13 deletions(-)
 create mode 100644 drivers/thunderbolt/icm.c

diff --git a/Documentation/ABI/testing/sysfs-bus-thunderbolt b/Documentation/ABI/testing/sysfs-bus-thunderbolt
index 2f352c787431..c100bdeacfaf 100644
--- a/Documentation/ABI/testing/sysfs-bus-thunderbolt
+++ b/Documentation/ABI/testing/sysfs-bus-thunderbolt
@@ -1,3 +1,51 @@
+What: /sys/bus/thunderbolt/devices/.../domainX/security
+Date:		Sep 2017
+KernelVersion:	4.13
+Contact:	thunderbolt-software@lists.01.org
+Description:	This attribute holds current Thunderbolt security level
+		set by the system BIOS. Possible values are:
+
+		none: All devices are automatically authorized
+		user: Devices are only authorized based on writing
+		      appropriate value to the authorized attribute
+		secure: Require devices that support secure connect at
+			minimum. User needs to authorize each device.
+		dponly: Automatically tunnel Display port (and USB). No
+			PCIe tunnels are created.
+
+What: /sys/bus/thunderbolt/devices/.../authorized
+Date:		Sep 2017
+KernelVersion:	4.13
+Contact:	thunderbolt-software@lists.01.org
+Description:	This attribute is used to authorize Thunderbolt devices
+		after they have been connected. If the device is not
+		authorized, no devices such as PCIe and Display port are
+		available to the system.
+
+		Contents of this attribute will be 0 when the device is not
+		yet authorized.
+
+		Possible values are supported:
+		1: The device will be authorized and connected
+
+		When key attribute contains 32 byte hex string the possible
+		values are:
+		1: The 32 byte hex string is added to the device NVM and
+		   the device is authorized.
+		2: Send a challenge based on the 32 byte hex string. If the
+		   challenge response from device is valid, the device is
+		   authorized. In case of failure errno will be ENOKEY if
+		   the device did not contain a key at all, and
+		   EKEYREJECTED if the challenge response did not match.
+
+What: /sys/bus/thunderbolt/devices/.../key
+Date:		Sep 2017
+KernelVersion:	4.13
+Contact:	thunderbolt-software@lists.01.org
+Description:	When a devices supports Thunderbolt secure connect it will
+		have this attribute. Writing 32 byte hex string changes
+		authorization to use the secure connection method instead.
+
 What:		/sys/bus/thunderbolt/devices/.../device
 Date:		Sep 2017
 KernelVersion:	4.13
diff --git a/drivers/thunderbolt/Kconfig b/drivers/thunderbolt/Kconfig
index d35db16aa43f..a9cc724985ad 100644
--- a/drivers/thunderbolt/Kconfig
+++ b/drivers/thunderbolt/Kconfig
@@ -1,15 +1,15 @@
 menuconfig THUNDERBOLT
-	tristate "Thunderbolt support for Apple devices"
+	tristate "Thunderbolt support"
 	depends on PCI
 	depends on X86 || COMPILE_TEST
 	select APPLE_PROPERTIES if EFI_STUB && X86
 	select CRC32
+	select CRYPTO
+	select CRYPTO_HASH
 	help
-	  Cactus Ridge Thunderbolt Controller driver
-	  This driver is required if you want to hotplug Thunderbolt devices on
-	  Apple hardware.
-
-	  Device chaining is currently not supported.
+	  Thunderbolt Controller driver. This driver is required if you
+	  want to hotplug Thunderbolt devices on Apple hardware or on PCs
+	  with Intel Falcon Ridge or newer.
 
 	  To compile this driver a module, choose M here. The module will be
 	  called thunderbolt.
diff --git a/drivers/thunderbolt/Makefile b/drivers/thunderbolt/Makefile
index 9828e862dd35..4900febc6c8a 100644
--- a/drivers/thunderbolt/Makefile
+++ b/drivers/thunderbolt/Makefile
@@ -1,3 +1,3 @@
 obj-${CONFIG_THUNDERBOLT} := thunderbolt.o
 thunderbolt-objs := nhi.o ctl.o tb.o switch.o cap.o path.o tunnel_pci.o eeprom.o
-thunderbolt-objs += domain.o dma_port.o
+thunderbolt-objs += domain.o dma_port.o icm.o
diff --git a/drivers/thunderbolt/ctl.c b/drivers/thunderbolt/ctl.c
index 2b1255cbf3c9..32e5bce56ac9 100644
--- a/drivers/thunderbolt/ctl.c
+++ b/drivers/thunderbolt/ctl.c
@@ -457,6 +457,8 @@ static void tb_ctl_rx_callback(struct tb_ring *ring, struct ring_frame *frame,
 				   "RX: checksum mismatch, dropping packet\n");
 			goto rx;
 		}
+		/* Fall through */
+	case TB_CFG_PKG_ICM_EVENT:
 		tb_ctl_handle_event(pkg->ctl, frame->eof, pkg, frame->size);
 		goto rx;
 
diff --git a/drivers/thunderbolt/domain.c b/drivers/thunderbolt/domain.c
index 99fe34d16b18..b8754c222ab4 100644
--- a/drivers/thunderbolt/domain.c
+++ b/drivers/thunderbolt/domain.c
@@ -13,11 +13,43 @@
 #include <linux/idr.h>
 #include <linux/module.h>
 #include <linux/slab.h>
+#include <linux/random.h>
+#include <crypto/hash.h>
 
 #include "tb.h"
 
 static DEFINE_IDA(tb_domain_ida);
 
+static const char * const tb_security_names[] = {
+	[TB_SECURITY_NONE] = "none",
+	[TB_SECURITY_USER] = "user",
+	[TB_SECURITY_SECURE] = "secure",
+	[TB_SECURITY_DPONLY] = "dponly",
+};
+
+static ssize_t security_show(struct device *dev, struct device_attribute *attr,
+			     char *buf)
+{
+	struct tb *tb = container_of(dev, struct tb, dev);
+
+	return sprintf(buf, "%s\n", tb_security_names[tb->security_level]);
+}
+static DEVICE_ATTR_RO(security);
+
+static struct attribute *domain_attrs[] = {
+	&dev_attr_security.attr,
+	NULL,
+};
+
+static struct attribute_group domain_attr_group = {
+	.attrs = domain_attrs,
+};
+
+static const struct attribute_group *domain_attr_groups[] = {
+	&domain_attr_group,
+	NULL,
+};
+
 struct bus_type tb_bus_type = {
 	.name = "thunderbolt",
 };
@@ -83,6 +115,7 @@ struct tb *tb_domain_alloc(struct tb_nhi *nhi, size_t privsize)
 	tb->dev.parent = &nhi->pdev->dev;
 	tb->dev.bus = &tb_bus_type;
 	tb->dev.type = &tb_domain_type;
+	tb->dev.groups = domain_attr_groups;
 	dev_set_name(&tb->dev, "domain%d", tb->index);
 	device_initialize(&tb->dev);
 
@@ -141,6 +174,12 @@ int tb_domain_add(struct tb *tb)
 	 */
 	tb_ctl_start(tb->ctl);
 
+	if (tb->cm_ops->driver_ready) {
+		ret = tb->cm_ops->driver_ready(tb);
+		if (ret)
+			goto err_ctl_stop;
+	}
+
 	ret = device_add(&tb->dev);
 	if (ret)
 		goto err_ctl_stop;
@@ -232,6 +271,162 @@ int tb_domain_resume_noirq(struct tb *tb)
 	return ret;
 }
 
+int tb_domain_suspend(struct tb *tb)
+{
+	int ret;
+
+	mutex_lock(&tb->lock);
+	if (tb->cm_ops->suspend) {
+		ret = tb->cm_ops->suspend(tb);
+		if (ret) {
+			mutex_unlock(&tb->lock);
+			return ret;
+		}
+	}
+	mutex_unlock(&tb->lock);
+	return 0;
+}
+
+void tb_domain_complete(struct tb *tb)
+{
+	mutex_lock(&tb->lock);
+	if (tb->cm_ops->complete)
+		tb->cm_ops->complete(tb);
+	mutex_unlock(&tb->lock);
+}
+
+/**
+ * tb_domain_approve_switch() - Approve switch
+ * @tb: Domain the switch belongs to
+ * @sw: Switch to approve
+ *
+ * This will approve switch by connection manager specific means. In
+ * case of success the connection manager will create tunnels for all
+ * supported protocols.
+ */
+int tb_domain_approve_switch(struct tb *tb, struct tb_switch *sw)
+{
+	struct tb_switch *parent_sw;
+
+	if (!tb->cm_ops->approve_switch)
+		return -EPERM;
+
+	/* The parent switch must be authorized before this one */
+	parent_sw = tb_to_switch(sw->dev.parent);
+	if (!parent_sw || !parent_sw->authorized)
+		return -EINVAL;
+
+	return tb->cm_ops->approve_switch(tb, sw);
+}
+
+/**
+ * tb_domain_approve_switch_key() - Approve switch and add key
+ * @tb: Domain the switch belongs to
+ * @sw: Switch to approve
+ *
+ * For switches that support secure connect, this function first adds
+ * key to the switch NVM using connection manager specific means. If
+ * adding the key is successful, the switch is approved and connected.
+ *
+ * Return: %0 on success and negative errno in case of failure.
+ */
+int tb_domain_approve_switch_key(struct tb *tb, struct tb_switch *sw)
+{
+	struct tb_switch *parent_sw;
+	int ret;
+
+	if (!tb->cm_ops->approve_switch || !tb->cm_ops->add_switch_key)
+		return -EPERM;
+
+	/* The parent switch must be authorized before this one */
+	parent_sw = tb_to_switch(sw->dev.parent);
+	if (!parent_sw || !parent_sw->authorized)
+		return -EINVAL;
+
+	ret = tb->cm_ops->add_switch_key(tb, sw);
+	if (ret)
+		return ret;
+
+	return tb->cm_ops->approve_switch(tb, sw);
+}
+
+/**
+ * tb_domain_challenge_switch_key() - Challenge and approve switch
+ * @tb: Domain the switch belongs to
+ * @sw: Switch to approve
+ *
+ * For switches that support secure connect, this function generates
+ * random challenge and sends it to the switch. The switch responds to
+ * this and if the response matches our random challenge, the switch is
+ * approved and connected.
+ *
+ * Return: %0 on success and negative errno in case of failure.
+ */
+int tb_domain_challenge_switch_key(struct tb *tb, struct tb_switch *sw)
+{
+	u8 challenge[TB_SWITCH_KEY_SIZE];
+	u8 response[TB_SWITCH_KEY_SIZE];
+	u8 hmac[TB_SWITCH_KEY_SIZE];
+	struct tb_switch *parent_sw;
+	struct crypto_shash *tfm;
+	struct shash_desc *shash;
+	int ret;
+
+	if (!tb->cm_ops->approve_switch || !tb->cm_ops->challenge_switch_key)
+		return -EPERM;
+
+	/* The parent switch must be authorized before this one */
+	parent_sw = tb_to_switch(sw->dev.parent);
+	if (!parent_sw || !parent_sw->authorized)
+		return -EINVAL;
+
+	get_random_bytes(challenge, sizeof(challenge));
+	ret = tb->cm_ops->challenge_switch_key(tb, sw, challenge, response);
+	if (ret)
+		return ret;
+
+	tfm = crypto_alloc_shash("hmac(sha256)", 0, 0);
+	if (IS_ERR(tfm))
+		return PTR_ERR(tfm);
+
+	ret = crypto_shash_setkey(tfm, sw->key, TB_SWITCH_KEY_SIZE);
+	if (ret)
+		goto err_free_tfm;
+
+	shash = kzalloc(sizeof(*shash) + crypto_shash_descsize(tfm),
+			GFP_KERNEL);
+	if (!shash) {
+		ret = -ENOMEM;
+		goto err_free_tfm;
+	}
+
+	shash->tfm = tfm;
+	shash->flags = CRYPTO_TFM_REQ_MAY_SLEEP;
+
+	memset(hmac, 0, sizeof(hmac));
+	ret = crypto_shash_digest(shash, challenge, sizeof(hmac), hmac);
+	if (ret)
+		goto err_free_shash;
+
+	/* The returned HMAC must match the one we calculated */
+	if (memcmp(response, hmac, sizeof(hmac))) {
+		ret = -EKEYREJECTED;
+		goto err_free_shash;
+	}
+
+	crypto_free_shash(tfm);
+	kfree(shash);
+
+	return tb->cm_ops->approve_switch(tb, sw);
+
+err_free_shash:
+	kfree(shash);
+err_free_tfm:
+	crypto_free_shash(tfm);
+
+	return ret;
+}
+
 int tb_domain_init(void)
 {
 	return bus_register(&tb_bus_type);
diff --git a/drivers/thunderbolt/icm.c b/drivers/thunderbolt/icm.c
new file mode 100644
index 000000000000..c99b2ae0e7ab
--- /dev/null
+++ b/drivers/thunderbolt/icm.c
@@ -0,0 +1,1067 @@
+/*
+ * Internal Thunderbolt Connection Manager. This is a firmware running on
+ * the Thunderbolt host controller performing most of the low-level
+ * handling.
+ *
+ * Copyright (C) 2017, Intel Corporation
+ * Authors: Michael Jamet <michael.jamet@intel.com>
+ *          Mika Westerberg <mika.westerberg@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/delay.h>
+#include <linux/dmi.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/sizes.h>
+#include <linux/slab.h>
+#include <linux/workqueue.h>
+
+#include "ctl.h"
+#include "nhi_regs.h"
+#include "tb.h"
+
+#define PCIE2CIO_CMD			0x30
+#define PCIE2CIO_CMD_TIMEOUT		BIT(31)
+#define PCIE2CIO_CMD_START		BIT(30)
+#define PCIE2CIO_CMD_WRITE		BIT(21)
+#define PCIE2CIO_CMD_CS_MASK		GENMASK(20, 19)
+#define PCIE2CIO_CMD_CS_SHIFT		19
+#define PCIE2CIO_CMD_PORT_MASK		GENMASK(18, 13)
+#define PCIE2CIO_CMD_PORT_SHIFT		13
+
+#define PCIE2CIO_WRDATA			0x34
+#define PCIE2CIO_RDDATA			0x38
+
+#define PHY_PORT_CS1			0x37
+#define PHY_PORT_CS1_LINK_DISABLE	BIT(14)
+#define PHY_PORT_CS1_LINK_STATE_MASK	GENMASK(29, 26)
+#define PHY_PORT_CS1_LINK_STATE_SHIFT	26
+
+#define ICM_TIMEOUT			5000 /* ms */
+#define ICM_MAX_LINK			4
+#define ICM_MAX_DEPTH			6
+
+/* Each physical port contains 2 links on modern controllers */
+#define TB_LINKS_PER_PHYS_PORT		2
+
+/*
+ * Calculate physical port number (Zero-based numbering) from link which
+ * starts from 1.
+ */
+#define TB_PHYS_PORT_FROM_LINK(link)	(((link) - 1) / TB_LINKS_PER_PHYS_PORT)
+
+/**
+ * struct icm - Internal connection manager private data
+ * @request_lock: Makes sure only one message is send to ICM at time
+ * @rescan_work: Work used to rescan the surviving switches after resume
+ * @upstream_port: Pointer to the PCIe upstream port this host
+ *		   controller is connected. This is only set for systems
+ *		   where ICM needs to be started manually
+ * @vnd_cap: Vendor defined capability where PCIe2CIO mailbox resides
+ *	     (only set when @upstream_port is not %NULL)
+ * @is_supported: Checks if we can support ICM on this controller
+ * @get_mode: Read and return the ICM firmware mode (optional)
+ * @get_route: Find a route string for given switch
+ * @device_connected: Handle device connected ICM message
+ * @device_disconnected: Handle device disconnected ICM message
+ */
+struct icm {
+	struct mutex request_lock;
+	struct delayed_work rescan_work;
+	struct pci_dev *upstream_port;
+	int vnd_cap;
+	bool (*is_supported)(struct tb *tb);
+	int (*get_mode)(struct tb *tb);
+	int (*get_route)(struct tb *tb, u8 link, u8 depth, u64 *route);
+	void (*device_connected)(struct tb *tb,
+				 const struct icm_pkg_header *hdr);
+	void (*device_disconnected)(struct tb *tb,
+				    const struct icm_pkg_header *hdr);
+};
+
+struct icm_notification {
+	struct work_struct work;
+	struct icm_pkg_header *pkg;
+	struct tb *tb;
+};
+
+static inline struct tb *icm_to_tb(struct icm *icm)
+{
+	return ((void *)icm - sizeof(struct tb));
+}
+
+static inline u8 phys_port_from_route(u64 route, u8 depth)
+{
+	return TB_PHYS_PORT_FROM_LINK(route >> ((depth - 1) * 8));
+}
+
+static inline u8 dual_link_from_link(u8 link)
+{
+	return link ? ((link - 1) ^ 0x01) + 1 : 0;
+}
+
+static inline u64 get_route(u32 route_hi, u32 route_lo)
+{
+	return (u64)route_hi << 32 | route_lo;
+}
+
+static inline bool is_apple(void)
+{
+	return dmi_match(DMI_BOARD_VENDOR, "Apple Inc.");
+}
+
+static bool icm_match(const struct tb_cfg_request *req,
+		      const struct ctl_pkg *pkg)
+{
+	const struct icm_pkg_header *res_hdr = pkg->buffer;
+	const struct icm_pkg_header *req_hdr = req->request;
+
+	if (pkg->frame.eof != req->response_type)
+		return false;
+	if (res_hdr->code != req_hdr->code)
+		return false;
+
+	return true;
+}
+
+static bool icm_copy(struct tb_cfg_request *req, const struct ctl_pkg *pkg)
+{
+	const struct icm_pkg_header *hdr = pkg->buffer;
+
+	if (hdr->packet_id < req->npackets) {
+		size_t offset = hdr->packet_id * req->response_size;
+
+		memcpy(req->response + offset, pkg->buffer, req->response_size);
+	}
+
+	return hdr->packet_id == hdr->total_packets - 1;
+}
+
+static int icm_request(struct tb *tb, const void *request, size_t request_size,
+		       void *response, size_t response_size, size_t npackets,
+		       unsigned int timeout_msec)
+{
+	struct icm *icm = tb_priv(tb);
+	int retries = 3;
+
+	do {
+		struct tb_cfg_request *req;
+		struct tb_cfg_result res;
+
+		req = tb_cfg_request_alloc();
+		if (!req)
+			return -ENOMEM;
+
+		req->match = icm_match;
+		req->copy = icm_copy;
+		req->request = request;
+		req->request_size = request_size;
+		req->request_type = TB_CFG_PKG_ICM_CMD;
+		req->response = response;
+		req->npackets = npackets;
+		req->response_size = response_size;
+		req->response_type = TB_CFG_PKG_ICM_RESP;
+
+		mutex_lock(&icm->request_lock);
+		res = tb_cfg_request_sync(tb->ctl, req, timeout_msec);
+		mutex_unlock(&icm->request_lock);
+
+		tb_cfg_request_put(req);
+
+		if (res.err != -ETIMEDOUT)
+			return res.err == 1 ? -EIO : res.err;
+
+		usleep_range(20, 50);
+	} while (retries--);
+
+	return -ETIMEDOUT;
+}
+
+static bool icm_fr_is_supported(struct tb *tb)
+{
+	return !is_apple();
+}
+
+static inline int icm_fr_get_switch_index(u32 port)
+{
+	int index;
+
+	if ((port & ICM_PORT_TYPE_MASK) != TB_TYPE_PORT)
+		return 0;
+
+	index = port >> ICM_PORT_INDEX_SHIFT;
+	return index != 0xff ? index : 0;
+}
+
+static int icm_fr_get_route(struct tb *tb, u8 link, u8 depth, u64 *route)
+{
+	struct icm_fr_pkg_get_topology_response *switches, *sw;
+	struct icm_fr_pkg_get_topology request = {
+		.hdr = { .code = ICM_GET_TOPOLOGY },
+	};
+	size_t npackets = ICM_GET_TOPOLOGY_PACKETS;
+	int ret, index;
+	u8 i;
+
+	switches = kcalloc(npackets, sizeof(*switches), GFP_KERNEL);
+	if (!switches)
+		return -ENOMEM;
+
+	ret = icm_request(tb, &request, sizeof(request), switches,
+			  sizeof(*switches), npackets, ICM_TIMEOUT);
+	if (ret)
+		goto err_free;
+
+	sw = &switches[0];
+	index = icm_fr_get_switch_index(sw->ports[link]);
+	if (!index) {
+		ret = -ENODEV;
+		goto err_free;
+	}
+
+	sw = &switches[index];
+	for (i = 1; i < depth; i++) {
+		unsigned int j;
+
+		if (!(sw->first_data & ICM_SWITCH_USED)) {
+			ret = -ENODEV;
+			goto err_free;
+		}
+
+		for (j = 0; j < ARRAY_SIZE(sw->ports); j++) {
+			index = icm_fr_get_switch_index(sw->ports[j]);
+			if (index > sw->switch_index) {
+				sw = &switches[index];
+				break;
+			}
+		}
+	}
+
+	*route = get_route(sw->route_hi, sw->route_lo);
+
+err_free:
+	kfree(switches);
+	return ret;
+}
+
+static int icm_fr_approve_switch(struct tb *tb, struct tb_switch *sw)
+{
+	struct icm_fr_pkg_approve_device request;
+	struct icm_fr_pkg_approve_device reply;
+	int ret;
+
+	memset(&request, 0, sizeof(request));
+	memcpy(&request.ep_uuid, sw->uuid, sizeof(request.ep_uuid));
+	request.hdr.code = ICM_APPROVE_DEVICE;
+	request.connection_id = sw->connection_id;
+	request.connection_key = sw->connection_key;
+
+	memset(&reply, 0, sizeof(reply));
+	/* Use larger timeout as establishing tunnels can take some time */
+	ret = icm_request(tb, &request, sizeof(request), &reply, sizeof(reply),
+			  1, 10000);
+	if (ret)
+		return ret;
+
+	if (reply.hdr.flags & ICM_FLAGS_ERROR) {
+		tb_warn(tb, "PCIe tunnel creation failed\n");
+		return -EIO;
+	}
+
+	return 0;
+}
+
+static int icm_fr_add_switch_key(struct tb *tb, struct tb_switch *sw)
+{
+	struct icm_fr_pkg_add_device_key request;
+	struct icm_fr_pkg_add_device_key_response reply;
+	int ret;
+
+	memset(&request, 0, sizeof(request));
+	memcpy(&request.ep_uuid, sw->uuid, sizeof(request.ep_uuid));
+	request.hdr.code = ICM_ADD_DEVICE_KEY;
+	request.connection_id = sw->connection_id;
+	request.connection_key = sw->connection_key;
+	memcpy(request.key, sw->key, TB_SWITCH_KEY_SIZE);
+
+	memset(&reply, 0, sizeof(reply));
+	ret = icm_request(tb, &request, sizeof(request), &reply, sizeof(reply),
+			  1, ICM_TIMEOUT);
+	if (ret)
+		return ret;
+
+	if (reply.hdr.flags & ICM_FLAGS_ERROR) {
+		tb_warn(tb, "Adding key to switch failed\n");
+		return -EIO;
+	}
+
+	return 0;
+}
+
+static int icm_fr_challenge_switch_key(struct tb *tb, struct tb_switch *sw,
+				       const u8 *challenge, u8 *response)
+{
+	struct icm_fr_pkg_challenge_device request;
+	struct icm_fr_pkg_challenge_device_response reply;
+	int ret;
+
+	memset(&request, 0, sizeof(request));
+	memcpy(&request.ep_uuid, sw->uuid, sizeof(request.ep_uuid));
+	request.hdr.code = ICM_CHALLENGE_DEVICE;
+	request.connection_id = sw->connection_id;
+	request.connection_key = sw->connection_key;
+	memcpy(request.challenge, challenge, TB_SWITCH_KEY_SIZE);
+
+	memset(&reply, 0, sizeof(reply));
+	ret = icm_request(tb, &request, sizeof(request), &reply, sizeof(reply),
+			  1, ICM_TIMEOUT);
+	if (ret)
+		return ret;
+
+	if (reply.hdr.flags & ICM_FLAGS_ERROR)
+		return -EKEYREJECTED;
+	if (reply.hdr.flags & ICM_FLAGS_NO_KEY)
+		return -ENOKEY;
+
+	memcpy(response, reply.response, TB_SWITCH_KEY_SIZE);
+
+	return 0;
+}
+
+static void remove_switch(struct tb_switch *sw)
+{
+	struct tb_switch *parent_sw;
+
+	parent_sw = tb_to_switch(sw->dev.parent);
+	tb_port_at(tb_route(sw), parent_sw)->remote = NULL;
+	tb_switch_remove(sw);
+}
+
+static void
+icm_fr_device_connected(struct tb *tb, const struct icm_pkg_header *hdr)
+{
+	const struct icm_fr_event_device_connected *pkg =
+		(const struct icm_fr_event_device_connected *)hdr;
+	struct tb_switch *sw, *parent_sw;
+	struct icm *icm = tb_priv(tb);
+	bool authorized = false;
+	u8 link, depth;
+	u64 route;
+	int ret;
+
+	link = pkg->link_info & ICM_LINK_INFO_LINK_MASK;
+	depth = (pkg->link_info & ICM_LINK_INFO_DEPTH_MASK) >>
+		ICM_LINK_INFO_DEPTH_SHIFT;
+	authorized = pkg->link_info & ICM_LINK_INFO_APPROVED;
+
+	ret = icm->get_route(tb, link, depth, &route);
+	if (ret) {
+		tb_err(tb, "failed to find route string for switch at %u.%u\n",
+		       link, depth);
+		return;
+	}
+
+	sw = tb_switch_find_by_uuid(tb, &pkg->ep_uuid);
+	if (sw) {
+		u8 phys_port, sw_phys_port;
+
+		parent_sw = tb_to_switch(sw->dev.parent);
+		sw_phys_port = phys_port_from_route(tb_route(sw), sw->depth);
+		phys_port = phys_port_from_route(route, depth);
+
+		/*
+		 * On resume ICM will send us connected events for the
+		 * devices that still are present. However, that
+		 * information might have changed for example by the
+		 * fact that a switch on a dual-link connection might
+		 * have been enumerated using the other link now. Make
+		 * sure our book keeping matches that.
+		 */
+		if (sw->depth == depth && sw_phys_port == phys_port &&
+		    !!sw->authorized == authorized) {
+			tb_port_at(tb_route(sw), parent_sw)->remote = NULL;
+			tb_port_at(route, parent_sw)->remote =
+				   tb_upstream_port(sw);
+			sw->config.route_hi = upper_32_bits(route);
+			sw->config.route_lo = lower_32_bits(route);
+			sw->connection_id = pkg->connection_id;
+			sw->connection_key = pkg->connection_key;
+			sw->link = link;
+			sw->depth = depth;
+			sw->is_unplugged = false;
+			tb_switch_put(sw);
+			return;
+		}
+
+		/*
+		 * User connected the same switch to another physical
+		 * port or to another part of the topology. Remove the
+		 * existing switch now before adding the new one.
+		 */
+		remove_switch(sw);
+		tb_switch_put(sw);
+	}
+
+	/*
+	 * If the switch was not found by UUID, look for a switch on
+	 * same physical port (taking possible link aggregation into
+	 * account) and depth. If we found one it is definitely a stale
+	 * one so remove it first.
+	 */
+	sw = tb_switch_find_by_link_depth(tb, link, depth);
+	if (!sw) {
+		u8 dual_link;
+
+		dual_link = dual_link_from_link(link);
+		if (dual_link)
+			sw = tb_switch_find_by_link_depth(tb, dual_link, depth);
+	}
+	if (sw) {
+		remove_switch(sw);
+		tb_switch_put(sw);
+	}
+
+	parent_sw = tb_switch_find_by_link_depth(tb, link, depth - 1);
+	if (!parent_sw) {
+		tb_err(tb, "failed to find parent switch for %u.%u\n",
+		       link, depth);
+		return;
+	}
+
+	sw = tb_switch_alloc(tb, &parent_sw->dev, route);
+	if (!sw) {
+		tb_switch_put(parent_sw);
+		return;
+	}
+
+	sw->uuid = kmemdup(&pkg->ep_uuid, sizeof(pkg->ep_uuid), GFP_KERNEL);
+	sw->connection_id = pkg->connection_id;
+	sw->connection_key = pkg->connection_key;
+	sw->link = link;
+	sw->depth = depth;
+	sw->authorized = authorized;
+	sw->security_level = (pkg->hdr.flags & ICM_FLAGS_SLEVEL_MASK) >>
+				ICM_FLAGS_SLEVEL_SHIFT;
+
+	/* Link the two switches now */
+	tb_port_at(route, parent_sw)->remote = tb_upstream_port(sw);
+	tb_upstream_port(sw)->remote = tb_port_at(route, parent_sw);
+
+	ret = tb_switch_add(sw);
+	if (ret) {
+		tb_port_at(tb_route(sw), parent_sw)->remote = NULL;
+		tb_switch_put(sw);
+	}
+	tb_switch_put(parent_sw);
+}
+
+static void
+icm_fr_device_disconnected(struct tb *tb, const struct icm_pkg_header *hdr)
+{
+	const struct icm_fr_event_device_disconnected *pkg =
+		(const struct icm_fr_event_device_disconnected *)hdr;
+	struct tb_switch *sw;
+	u8 link, depth;
+
+	link = pkg->link_info & ICM_LINK_INFO_LINK_MASK;
+	depth = (pkg->link_info & ICM_LINK_INFO_DEPTH_MASK) >>
+		ICM_LINK_INFO_DEPTH_SHIFT;
+
+	if (link > ICM_MAX_LINK || depth > ICM_MAX_DEPTH) {
+		tb_warn(tb, "invalid topology %u.%u, ignoring\n", link, depth);
+		return;
+	}
+
+	sw = tb_switch_find_by_link_depth(tb, link, depth);
+	if (!sw) {
+		tb_warn(tb, "no switch exists at %u.%u, ignoring\n", link,
+			depth);
+		return;
+	}
+
+	remove_switch(sw);
+	tb_switch_put(sw);
+}
+
+static struct pci_dev *get_upstream_port(struct pci_dev *pdev)
+{
+	struct pci_dev *parent;
+
+	parent = pci_upstream_bridge(pdev);
+	while (parent) {
+		if (!pci_is_pcie(parent))
+			return NULL;
+		if (pci_pcie_type(parent) == PCI_EXP_TYPE_UPSTREAM)
+			break;
+		parent = pci_upstream_bridge(parent);
+	}
+
+	if (!parent)
+		return NULL;
+
+	switch (parent->device) {
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_2C_BRIDGE:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_4C_BRIDGE:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_BRIDGE:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_4C_BRIDGE:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_2C_BRIDGE:
+		return parent;
+	}
+
+	return NULL;
+}
+
+static bool icm_ar_is_supported(struct tb *tb)
+{
+	struct pci_dev *upstream_port;
+	struct icm *icm = tb_priv(tb);
+
+	/*
+	 * Starting from Alpine Ridge we can use ICM on Apple machines
+	 * as well. We just need to reset and re-enable it first.
+	 */
+	if (!is_apple())
+		return true;
+
+	/*
+	 * Find the upstream PCIe port in case we need to do reset
+	 * through its vendor specific registers.
+	 */
+	upstream_port = get_upstream_port(tb->nhi->pdev);
+	if (upstream_port) {
+		int cap;
+
+		cap = pci_find_ext_capability(upstream_port,
+					      PCI_EXT_CAP_ID_VNDR);
+		if (cap > 0) {
+			icm->upstream_port = upstream_port;
+			icm->vnd_cap = cap;
+
+			return true;
+		}
+	}
+
+	return false;
+}
+
+static int icm_ar_get_mode(struct tb *tb)
+{
+	struct tb_nhi *nhi = tb->nhi;
+	int retries = 5;
+	u32 val;
+
+	do {
+		val = ioread32(nhi->iobase + REG_FW_STS);
+		if (val & REG_FW_STS_NVM_AUTH_DONE)
+			break;
+		msleep(30);
+	} while (--retries);
+
+	if (!retries) {
+		dev_err(&nhi->pdev->dev, "ICM firmware not authenticated\n");
+		return -ENODEV;
+	}
+
+	return nhi_mailbox_mode(nhi);
+}
+
+static int icm_ar_get_route(struct tb *tb, u8 link, u8 depth, u64 *route)
+{
+	struct icm_ar_pkg_get_route_response reply;
+	struct icm_ar_pkg_get_route request = {
+		.hdr = { .code = ICM_GET_ROUTE },
+		.link_info = depth << ICM_LINK_INFO_DEPTH_SHIFT | link,
+	};
+	int ret;
+
+	memset(&reply, 0, sizeof(reply));
+	ret = icm_request(tb, &request, sizeof(request), &reply, sizeof(reply),
+			  1, ICM_TIMEOUT);
+	if (ret)
+		return ret;
+
+	if (reply.hdr.flags & ICM_FLAGS_ERROR)
+		return -EIO;
+
+	*route = get_route(reply.route_hi, reply.route_lo);
+	return 0;
+}
+
+static void icm_handle_notification(struct work_struct *work)
+{
+	struct icm_notification *n = container_of(work, typeof(*n), work);
+	struct tb *tb = n->tb;
+	struct icm *icm = tb_priv(tb);
+
+	mutex_lock(&tb->lock);
+
+	switch (n->pkg->code) {
+	case ICM_EVENT_DEVICE_CONNECTED:
+		icm->device_connected(tb, n->pkg);
+		break;
+	case ICM_EVENT_DEVICE_DISCONNECTED:
+		icm->device_disconnected(tb, n->pkg);
+		break;
+	}
+
+	mutex_unlock(&tb->lock);
+
+	kfree(n->pkg);
+	kfree(n);
+}
+
+static void icm_handle_event(struct tb *tb, enum tb_cfg_pkg_type type,
+			     const void *buf, size_t size)
+{
+	struct icm_notification *n;
+
+	n = kmalloc(sizeof(*n), GFP_KERNEL);
+	if (!n)
+		return;
+
+	INIT_WORK(&n->work, icm_handle_notification);
+	n->pkg = kmemdup(buf, size, GFP_KERNEL);
+	n->tb = tb;
+
+	queue_work(tb->wq, &n->work);
+}
+
+static int
+__icm_driver_ready(struct tb *tb, enum tb_security_level *security_level)
+{
+	struct icm_pkg_driver_ready_response reply;
+	struct icm_pkg_driver_ready request = {
+		.hdr.code = ICM_DRIVER_READY,
+	};
+	unsigned int retries = 10;
+	int ret;
+
+	memset(&reply, 0, sizeof(reply));
+	ret = icm_request(tb, &request, sizeof(request), &reply, sizeof(reply),
+			  1, ICM_TIMEOUT);
+	if (ret)
+		return ret;
+
+	if (security_level)
+		*security_level = reply.security_level & 0xf;
+
+	/*
+	 * Hold on here until the switch config space is accessible so
+	 * that we can read root switch config successfully.
+	 */
+	do {
+		struct tb_cfg_result res;
+		u32 tmp;
+
+		res = tb_cfg_read_raw(tb->ctl, &tmp, 0, 0, TB_CFG_SWITCH,
+				      0, 1, 100);
+		if (!res.err)
+			return 0;
+
+		msleep(50);
+	} while (--retries);
+
+	return -ETIMEDOUT;
+}
+
+static int pci2cio_wait_completion(struct icm *icm, unsigned long timeout_msec)
+{
+	unsigned long end = jiffies + msecs_to_jiffies(timeout_msec);
+	u32 cmd;
+
+	do {
+		pci_read_config_dword(icm->upstream_port,
+				      icm->vnd_cap + PCIE2CIO_CMD, &cmd);
+		if (!(cmd & PCIE2CIO_CMD_START)) {
+			if (cmd & PCIE2CIO_CMD_TIMEOUT)
+				break;
+			return 0;
+		}
+
+		msleep(50);
+	} while (time_before(jiffies, end));
+
+	return -ETIMEDOUT;
+}
+
+static int pcie2cio_read(struct icm *icm, enum tb_cfg_space cs,
+			 unsigned int port, unsigned int index, u32 *data)
+{
+	struct pci_dev *pdev = icm->upstream_port;
+	int ret, vnd_cap = icm->vnd_cap;
+	u32 cmd;
+
+	cmd = index;
+	cmd |= (port << PCIE2CIO_CMD_PORT_SHIFT) & PCIE2CIO_CMD_PORT_MASK;
+	cmd |= (cs << PCIE2CIO_CMD_CS_SHIFT) & PCIE2CIO_CMD_CS_MASK;
+	cmd |= PCIE2CIO_CMD_START;
+	pci_write_config_dword(pdev, vnd_cap + PCIE2CIO_CMD, cmd);
+
+	ret = pci2cio_wait_completion(icm, 5000);
+	if (ret)
+		return ret;
+
+	pci_read_config_dword(pdev, vnd_cap + PCIE2CIO_RDDATA, data);
+	return 0;
+}
+
+static int pcie2cio_write(struct icm *icm, enum tb_cfg_space cs,
+			  unsigned int port, unsigned int index, u32 data)
+{
+	struct pci_dev *pdev = icm->upstream_port;
+	int vnd_cap = icm->vnd_cap;
+	u32 cmd;
+
+	pci_write_config_dword(pdev, vnd_cap + PCIE2CIO_WRDATA, data);
+
+	cmd = index;
+	cmd |= (port << PCIE2CIO_CMD_PORT_SHIFT) & PCIE2CIO_CMD_PORT_MASK;
+	cmd |= (cs << PCIE2CIO_CMD_CS_SHIFT) & PCIE2CIO_CMD_CS_MASK;
+	cmd |= PCIE2CIO_CMD_WRITE | PCIE2CIO_CMD_START;
+	pci_write_config_dword(pdev, vnd_cap + PCIE2CIO_CMD, cmd);
+
+	return pci2cio_wait_completion(icm, 5000);
+}
+
+static int icm_firmware_reset(struct tb *tb, struct tb_nhi *nhi)
+{
+	struct icm *icm = tb_priv(tb);
+	u32 val;
+
+	/* Put ARC to wait for CIO reset event to happen */
+	val = ioread32(nhi->iobase + REG_FW_STS);
+	val |= REG_FW_STS_CIO_RESET_REQ;
+	iowrite32(val, nhi->iobase + REG_FW_STS);
+
+	/* Re-start ARC */
+	val = ioread32(nhi->iobase + REG_FW_STS);
+	val |= REG_FW_STS_ICM_EN_INVERT;
+	val |= REG_FW_STS_ICM_EN_CPU;
+	iowrite32(val, nhi->iobase + REG_FW_STS);
+
+	/* Trigger CIO reset now */
+	return pcie2cio_write(icm, TB_CFG_SWITCH, 0, 0x50, BIT(9));
+}
+
+static int icm_firmware_start(struct tb *tb, struct tb_nhi *nhi)
+{
+	unsigned int retries = 10;
+	int ret;
+	u32 val;
+
+	/* Check if the ICM firmware is already running */
+	val = ioread32(nhi->iobase + REG_FW_STS);
+	if (val & REG_FW_STS_ICM_EN)
+		return 0;
+
+	dev_info(&nhi->pdev->dev, "starting ICM firmware\n");
+
+	ret = icm_firmware_reset(tb, nhi);
+	if (ret)
+		return ret;
+
+	/* Wait until the ICM firmware tells us it is up and running */
+	do {
+		/* Check that the ICM firmware is running */
+		val = ioread32(nhi->iobase + REG_FW_STS);
+		if (val & REG_FW_STS_NVM_AUTH_DONE)
+			return 0;
+
+		msleep(300);
+	} while (--retries);
+
+	return -ETIMEDOUT;
+}
+
+static int icm_reset_phys_port(struct tb *tb, int phys_port)
+{
+	struct icm *icm = tb_priv(tb);
+	u32 state0, state1;
+	int port0, port1;
+	u32 val0, val1;
+	int ret;
+
+	if (!icm->upstream_port)
+		return 0;
+
+	if (phys_port) {
+		port0 = 3;
+		port1 = 4;
+	} else {
+		port0 = 1;
+		port1 = 2;
+	}
+
+	/*
+	 * Read link status of both null ports belonging to a single
+	 * physical port.
+	 */
+	ret = pcie2cio_read(icm, TB_CFG_PORT, port0, PHY_PORT_CS1, &val0);
+	if (ret)
+		return ret;
+	ret = pcie2cio_read(icm, TB_CFG_PORT, port1, PHY_PORT_CS1, &val1);
+	if (ret)
+		return ret;
+
+	state0 = val0 & PHY_PORT_CS1_LINK_STATE_MASK;
+	state0 >>= PHY_PORT_CS1_LINK_STATE_SHIFT;
+	state1 = val1 & PHY_PORT_CS1_LINK_STATE_MASK;
+	state1 >>= PHY_PORT_CS1_LINK_STATE_SHIFT;
+
+	/* If they are both up we need to reset them now */
+	if (state0 != TB_PORT_UP || state1 != TB_PORT_UP)
+		return 0;
+
+	val0 |= PHY_PORT_CS1_LINK_DISABLE;
+	ret = pcie2cio_write(icm, TB_CFG_PORT, port0, PHY_PORT_CS1, val0);
+	if (ret)
+		return ret;
+
+	val1 |= PHY_PORT_CS1_LINK_DISABLE;
+	ret = pcie2cio_write(icm, TB_CFG_PORT, port1, PHY_PORT_CS1, val1);
+	if (ret)
+		return ret;
+
+	/* Wait a bit and then re-enable both ports */
+	usleep_range(10, 100);
+
+	ret = pcie2cio_read(icm, TB_CFG_PORT, port0, PHY_PORT_CS1, &val0);
+	if (ret)
+		return ret;
+	ret = pcie2cio_read(icm, TB_CFG_PORT, port1, PHY_PORT_CS1, &val1);
+	if (ret)
+		return ret;
+
+	val0 &= ~PHY_PORT_CS1_LINK_DISABLE;
+	ret = pcie2cio_write(icm, TB_CFG_PORT, port0, PHY_PORT_CS1, val0);
+	if (ret)
+		return ret;
+
+	val1 &= ~PHY_PORT_CS1_LINK_DISABLE;
+	return pcie2cio_write(icm, TB_CFG_PORT, port1, PHY_PORT_CS1, val1);
+}
+
+static int icm_firmware_init(struct tb *tb)
+{
+	struct icm *icm = tb_priv(tb);
+	struct tb_nhi *nhi = tb->nhi;
+	int ret;
+
+	ret = icm_firmware_start(tb, nhi);
+	if (ret) {
+		dev_err(&nhi->pdev->dev, "could not start ICM firmware\n");
+		return ret;
+	}
+
+	if (icm->get_mode) {
+		ret = icm->get_mode(tb);
+
+		switch (ret) {
+		case NHI_FW_CM_MODE:
+			/* Ask ICM to accept all Thunderbolt devices */
+			nhi_mailbox_cmd(nhi, NHI_MAILBOX_ALLOW_ALL_DEVS, 0);
+			break;
+
+		default:
+			tb_err(tb, "ICM firmware is in wrong mode: %u\n", ret);
+			return -ENODEV;
+		}
+	}
+
+	/*
+	 * Reset both physical ports if there is anything connected to
+	 * them already.
+	 */
+	ret = icm_reset_phys_port(tb, 0);
+	if (ret)
+		dev_warn(&nhi->pdev->dev, "failed to reset links on port0\n");
+	ret = icm_reset_phys_port(tb, 1);
+	if (ret)
+		dev_warn(&nhi->pdev->dev, "failed to reset links on port1\n");
+
+	return 0;
+}
+
+static int icm_driver_ready(struct tb *tb)
+{
+	int ret;
+
+	ret = icm_firmware_init(tb);
+	if (ret)
+		return ret;
+
+	return __icm_driver_ready(tb, &tb->security_level);
+}
+
+static int icm_suspend(struct tb *tb)
+{
+	return nhi_mailbox_cmd(tb->nhi, NHI_MAILBOX_SAVE_DEVS, 0);
+}
+
+/*
+ * Mark all switches (except root switch) below this one unplugged. ICM
+ * firmware will send us an updated list of switches after we have send
+ * it driver ready command. If a switch is not in that list it will be
+ * removed when we perform rescan.
+ */
+static void icm_unplug_children(struct tb_switch *sw)
+{
+	unsigned int i;
+
+	if (tb_route(sw))
+		sw->is_unplugged = true;
+
+	for (i = 1; i <= sw->config.max_port_number; i++) {
+		struct tb_port *port = &sw->ports[i];
+
+		if (tb_is_upstream_port(port))
+			continue;
+		if (!port->remote)
+			continue;
+
+		icm_unplug_children(port->remote->sw);
+	}
+}
+
+static void icm_free_unplugged_children(struct tb_switch *sw)
+{
+	unsigned int i;
+
+	for (i = 1; i <= sw->config.max_port_number; i++) {
+		struct tb_port *port = &sw->ports[i];
+
+		if (tb_is_upstream_port(port))
+			continue;
+		if (!port->remote)
+			continue;
+
+		if (port->remote->sw->is_unplugged) {
+			tb_switch_remove(port->remote->sw);
+			port->remote = NULL;
+		} else {
+			icm_free_unplugged_children(port->remote->sw);
+		}
+	}
+}
+
+static void icm_rescan_work(struct work_struct *work)
+{
+	struct icm *icm = container_of(work, struct icm, rescan_work.work);
+	struct tb *tb = icm_to_tb(icm);
+
+	mutex_lock(&tb->lock);
+	if (tb->root_switch)
+		icm_free_unplugged_children(tb->root_switch);
+	mutex_unlock(&tb->lock);
+}
+
+static void icm_complete(struct tb *tb)
+{
+	struct icm *icm = tb_priv(tb);
+
+	if (tb->nhi->going_away)
+		return;
+
+	icm_unplug_children(tb->root_switch);
+
+	/*
+	 * Now all existing children should be resumed, start events
+	 * from ICM to get updated status.
+	 */
+	__icm_driver_ready(tb, NULL);
+
+	/*
+	 * We do not get notifications of devices that have been
+	 * unplugged during suspend so schedule rescan to clean them up
+	 * if any.
+	 */
+	queue_delayed_work(tb->wq, &icm->rescan_work, msecs_to_jiffies(500));
+}
+
+static int icm_start(struct tb *tb)
+{
+	int ret;
+
+	tb->root_switch = tb_switch_alloc(tb, &tb->dev, 0);
+	if (!tb->root_switch)
+		return -ENODEV;
+
+	ret = tb_switch_add(tb->root_switch);
+	if (ret)
+		tb_switch_put(tb->root_switch);
+
+	return ret;
+}
+
+static void icm_stop(struct tb *tb)
+{
+	struct icm *icm = tb_priv(tb);
+
+	cancel_delayed_work(&icm->rescan_work);
+	tb_switch_remove(tb->root_switch);
+	tb->root_switch = NULL;
+	nhi_mailbox_cmd(tb->nhi, NHI_MAILBOX_DRV_UNLOADS, 0);
+}
+
+/* Falcon Ridge and Alpine Ridge */
+static const struct tb_cm_ops icm_fr_ops = {
+	.driver_ready = icm_driver_ready,
+	.start = icm_start,
+	.stop = icm_stop,
+	.suspend = icm_suspend,
+	.complete = icm_complete,
+	.handle_event = icm_handle_event,
+	.approve_switch = icm_fr_approve_switch,
+	.add_switch_key = icm_fr_add_switch_key,
+	.challenge_switch_key = icm_fr_challenge_switch_key,
+};
+
+struct tb *icm_probe(struct tb_nhi *nhi)
+{
+	struct icm *icm;
+	struct tb *tb;
+
+	tb = tb_domain_alloc(nhi, sizeof(struct icm));
+	if (!tb)
+		return NULL;
+
+	icm = tb_priv(tb);
+	INIT_DELAYED_WORK(&icm->rescan_work, icm_rescan_work);
+	mutex_init(&icm->request_lock);
+
+	switch (nhi->pdev->device) {
+	case PCI_DEVICE_ID_INTEL_FALCON_RIDGE_2C_NHI:
+	case PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_NHI:
+		icm->is_supported = icm_fr_is_supported;
+		icm->get_route = icm_fr_get_route;
+		icm->device_connected = icm_fr_device_connected;
+		icm->device_disconnected = icm_fr_device_disconnected;
+		tb->cm_ops = &icm_fr_ops;
+		break;
+
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_2C_NHI:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_4C_NHI:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_NHI:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_4C_NHI:
+	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_2C_NHI:
+		icm->is_supported = icm_ar_is_supported;
+		icm->get_mode = icm_ar_get_mode;
+		icm->get_route = icm_ar_get_route;
+		icm->device_connected = icm_fr_device_connected;
+		icm->device_disconnected = icm_fr_device_disconnected;
+		tb->cm_ops = &icm_fr_ops;
+		break;
+	}
+
+	if (!icm->is_supported || !icm->is_supported(tb)) {
+		dev_dbg(&nhi->pdev->dev, "ICM not supported on this controller\n");
+		tb_domain_put(tb);
+		return NULL;
+	}
+
+	return tb;
+}
diff --git a/drivers/thunderbolt/nhi.c b/drivers/thunderbolt/nhi.c
index c11e4009325b..4b328bf577d9 100644
--- a/drivers/thunderbolt/nhi.c
+++ b/drivers/thunderbolt/nhi.c
@@ -13,7 +13,6 @@
 #include <linux/pci.h>
 #include <linux/interrupt.h>
 #include <linux/module.h>
-#include <linux/dmi.h>
 #include <linux/delay.h>
 
 #include "nhi.h"
@@ -673,6 +672,22 @@ static int nhi_resume_noirq(struct device *dev)
 	return tb_domain_resume_noirq(tb);
 }
 
+static int nhi_suspend(struct device *dev)
+{
+	struct pci_dev *pdev = to_pci_dev(dev);
+	struct tb *tb = pci_get_drvdata(pdev);
+
+	return tb_domain_suspend(tb);
+}
+
+static void nhi_complete(struct device *dev)
+{
+	struct pci_dev *pdev = to_pci_dev(dev);
+	struct tb *tb = pci_get_drvdata(pdev);
+
+	tb_domain_complete(tb);
+}
+
 static void nhi_shutdown(struct tb_nhi *nhi)
 {
 	int i;
@@ -788,10 +803,16 @@ static int nhi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	/* magic value - clock related? */
 	iowrite32(3906250 / 10000, nhi->iobase + 0x38c00);
 
-	dev_info(&nhi->pdev->dev, "NHI initialized, starting thunderbolt\n");
-	tb = tb_probe(nhi);
+	tb = icm_probe(nhi);
 	if (!tb)
+		tb = tb_probe(nhi);
+	if (!tb) {
+		dev_err(&nhi->pdev->dev,
+			"failed to determine connection manager, aborting\n");
 		return -ENODEV;
+	}
+
+	dev_info(&nhi->pdev->dev, "NHI initialized, starting thunderbolt\n");
 
 	res = tb_domain_add(tb);
 	if (res) {
@@ -830,6 +851,10 @@ static const struct dev_pm_ops nhi_pm_ops = {
 					    * pci-tunnels stay alive.
 					    */
 	.restore_noirq = nhi_resume_noirq,
+	.suspend = nhi_suspend,
+	.freeze = nhi_suspend,
+	.poweroff = nhi_suspend,
+	.complete = nhi_complete,
 };
 
 static struct pci_device_id nhi_ids[] = {
@@ -890,8 +915,6 @@ static int __init nhi_init(void)
 {
 	int ret;
 
-	if (!dmi_match(DMI_BOARD_VENDOR, "Apple Inc."))
-		return -ENOSYS;
 	ret = tb_domain_init();
 	if (ret)
 		return ret;
diff --git a/drivers/thunderbolt/nhi_regs.h b/drivers/thunderbolt/nhi_regs.h
index 322fe1fa3a3c..09ed574e92ff 100644
--- a/drivers/thunderbolt/nhi_regs.h
+++ b/drivers/thunderbolt/nhi_regs.h
@@ -118,4 +118,11 @@ struct ring_desc {
 #define REG_OUTMAIL_CMD_OPMODE_SHIFT	8
 #define REG_OUTMAIL_CMD_OPMODE_MASK	GENMASK(11, 8)
 
+#define REG_FW_STS			0x39944
+#define REG_FW_STS_NVM_AUTH_DONE	BIT(31)
+#define REG_FW_STS_CIO_RESET_REQ	BIT(30)
+#define REG_FW_STS_ICM_EN_CPU		BIT(2)
+#define REG_FW_STS_ICM_EN_INVERT	BIT(1)
+#define REG_FW_STS_ICM_EN		BIT(0)
+
 #endif
diff --git a/drivers/thunderbolt/switch.c b/drivers/thunderbolt/switch.c
index a75db3fa5e7a..5dc96ad1fc68 100644
--- a/drivers/thunderbolt/switch.c
+++ b/drivers/thunderbolt/switch.c
@@ -9,6 +9,9 @@
 
 #include "tb.h"
 
+/* Switch authorization from userspace is serialized by this lock */
+static DEFINE_MUTEX(switch_lock);
+
 /* port utility functions */
 
 static const char *tb_port_type(struct tb_regs_port_header *port)
@@ -310,6 +313,72 @@ static int tb_plug_events_active(struct tb_switch *sw, bool active)
 			   sw->cap_plug_events + 1, 1);
 }
 
+static ssize_t authorized_show(struct device *dev,
+			       struct device_attribute *attr,
+			       char *buf)
+{
+	struct tb_switch *sw = tb_to_switch(dev);
+
+	return sprintf(buf, "%u\n", sw->authorized);
+}
+
+static int tb_switch_set_authorized(struct tb_switch *sw, unsigned int val)
+{
+	int ret = -EINVAL;
+
+	if (mutex_lock_interruptible(&switch_lock))
+		return -ERESTARTSYS;
+
+	if (sw->authorized)
+		goto unlock;
+
+	switch (val) {
+	/* Approve switch */
+	case 1:
+		if (sw->key)
+			ret = tb_domain_approve_switch_key(sw->tb, sw);
+		else
+			ret = tb_domain_approve_switch(sw->tb, sw);
+		break;
+
+	/* Challenge switch */
+	case 2:
+		if (sw->key)
+			ret = tb_domain_challenge_switch_key(sw->tb, sw);
+		break;
+
+	default:
+		break;
+	}
+
+	if (!ret)
+		sw->authorized = val;
+
+unlock:
+	mutex_unlock(&switch_lock);
+	return ret;
+}
+
+static ssize_t authorized_store(struct device *dev,
+				struct device_attribute *attr,
+				const char *buf, size_t count)
+{
+	struct tb_switch *sw = tb_to_switch(dev);
+	unsigned int val;
+	ssize_t ret;
+
+	ret = kstrtouint(buf, 0, &val);
+	if (ret)
+		return ret;
+	if (val > 2)
+		return -EINVAL;
+
+	ret = tb_switch_set_authorized(sw, val);
+
+	return ret ? ret : count;
+}
+static DEVICE_ATTR_RW(authorized);
+
 static ssize_t device_show(struct device *dev, struct device_attribute *attr,
 			   char *buf)
 {
@@ -356,6 +425,7 @@ static ssize_t unique_id_show(struct device *dev, struct device_attribute *attr,
 static DEVICE_ATTR_RO(unique_id);
 
 static struct attribute *switch_attrs[] = {
+	&dev_attr_authorized.attr,
 	&dev_attr_device.attr,
 	&dev_attr_device_name.attr,
 	&dev_attr_vendor.attr,
@@ -365,6 +435,66 @@ static struct attribute *switch_attrs[] = {
 };
 ATTRIBUTE_GROUPS(switch);
 
+static ssize_t key_show(struct device *dev, struct device_attribute *attr,
+			char *buf)
+{
+	struct tb_switch *sw = tb_to_switch(dev);
+	ssize_t ret;
+
+	if (mutex_lock_interruptible(&switch_lock))
+		return -ERESTARTSYS;
+
+	if (sw->key)
+		ret = sprintf(buf, "%*phN\n", TB_SWITCH_KEY_SIZE, sw->key);
+	else
+		ret = sprintf(buf, "\n");
+
+	mutex_unlock(&switch_lock);
+	return ret;
+}
+
+static ssize_t key_store(struct device *dev, struct device_attribute *attr,
+			 const char *buf, size_t count)
+{
+	struct tb_switch *sw = tb_to_switch(dev);
+	u8 key[TB_SWITCH_KEY_SIZE];
+	ssize_t ret = count;
+
+	if (count < 64)
+		return -EINVAL;
+
+	if (hex2bin(key, buf, sizeof(key)))
+		return -EINVAL;
+
+	if (mutex_lock_interruptible(&switch_lock))
+		return -ERESTARTSYS;
+
+	if (sw->authorized) {
+		ret = -EBUSY;
+	} else {
+		kfree(sw->key);
+		sw->key = kmemdup(key, sizeof(key), GFP_KERNEL);
+		if (!sw->key)
+			ret = -ENOMEM;
+	}
+
+	mutex_unlock(&switch_lock);
+	return ret;
+}
+static DEVICE_ATTR_RW(key);
+
+static struct attribute *secure_switch_attrs[] = {
+	&dev_attr_authorized.attr,
+	&dev_attr_device.attr,
+	&dev_attr_device_name.attr,
+	&dev_attr_key.attr,
+	&dev_attr_vendor.attr,
+	&dev_attr_vendor_name.attr,
+	&dev_attr_unique_id.attr,
+	NULL,
+};
+ATTRIBUTE_GROUPS(secure_switch);
+
 static void tb_switch_release(struct device *dev)
 {
 	struct tb_switch *sw = tb_to_switch(dev);
@@ -376,6 +506,7 @@ static void tb_switch_release(struct device *dev)
 	kfree(sw->vendor_name);
 	kfree(sw->ports);
 	kfree(sw->drom);
+	kfree(sw->key);
 	kfree(sw);
 }
 
@@ -468,11 +599,14 @@ struct tb_switch *tb_switch_alloc(struct tb *tb, struct device *parent,
 
 	tb_switch_set_generation(sw);
 
+	/* Root switch is always authorized */
+	if (!route)
+		sw->authorized = true;
+
 	device_initialize(&sw->dev);
 	sw->dev.parent = parent;
 	sw->dev.bus = &tb_bus_type;
 	sw->dev.type = &tb_switch_type;
-	sw->dev.groups = switch_groups;
 	dev_set_name(&sw->dev, "%u-%llx", tb->index, tb_route(sw));
 
 	return sw;
@@ -626,6 +760,11 @@ int tb_switch_add(struct tb_switch *sw)
 			return ret;
 	}
 
+	if (sw->security_level == TB_SECURITY_SECURE)
+		sw->dev.groups = secure_switch_groups;
+	else
+		sw->dev.groups = switch_groups;
+
 	return device_add(&sw->dev);
 }
 
@@ -737,3 +876,80 @@ void tb_switch_suspend(struct tb_switch *sw)
 	 * effect?
 	 */
 }
+
+struct tb_sw_lookup {
+	struct tb *tb;
+	u8 link;
+	u8 depth;
+	const uuid_be *uuid;
+};
+
+static int tb_switch_match(struct device *dev, void *data)
+{
+	struct tb_switch *sw = tb_to_switch(dev);
+	struct tb_sw_lookup *lookup = data;
+
+	if (!sw)
+		return 0;
+	if (sw->tb != lookup->tb)
+		return 0;
+
+	if (lookup->uuid)
+		return !memcmp(sw->uuid, lookup->uuid, sizeof(*lookup->uuid));
+
+	/* Root switch is matched only by depth */
+	if (!lookup->depth)
+		return !sw->depth;
+
+	return sw->link == lookup->link && sw->depth == lookup->depth;
+}
+
+/**
+ * tb_switch_find_by_link_depth() - Find switch by link and depth
+ * @tb: Domain the switch belongs
+ * @link: Link number the switch is connected
+ * @depth: Depth of the switch in link
+ *
+ * Returned switch has reference count increased so the caller needs to
+ * call tb_switch_put() when done with the switch.
+ */
+struct tb_switch *tb_switch_find_by_link_depth(struct tb *tb, u8 link, u8 depth)
+{
+	struct tb_sw_lookup lookup;
+	struct device *dev;
+
+	memset(&lookup, 0, sizeof(lookup));
+	lookup.tb = tb;
+	lookup.link = link;
+	lookup.depth = depth;
+
+	dev = bus_find_device(&tb_bus_type, NULL, &lookup, tb_switch_match);
+	if (dev)
+		return tb_to_switch(dev);
+
+	return NULL;
+}
+
+/**
+ * tb_switch_find_by_link_depth() - Find switch by UUID
+ * @tb: Domain the switch belongs
+ * @uuid: UUID to look for
+ *
+ * Returned switch has reference count increased so the caller needs to
+ * call tb_switch_put() when done with the switch.
+ */
+struct tb_switch *tb_switch_find_by_uuid(struct tb *tb, const uuid_be *uuid)
+{
+	struct tb_sw_lookup lookup;
+	struct device *dev;
+
+	memset(&lookup, 0, sizeof(lookup));
+	lookup.tb = tb;
+	lookup.uuid = uuid;
+
+	dev = bus_find_device(&tb_bus_type, NULL, &lookup, tb_switch_match);
+	if (dev)
+		return tb_to_switch(dev);
+
+	return NULL;
+}
diff --git a/drivers/thunderbolt/tb.c b/drivers/thunderbolt/tb.c
index ea9de49b5e10..ad2304bad592 100644
--- a/drivers/thunderbolt/tb.c
+++ b/drivers/thunderbolt/tb.c
@@ -7,6 +7,7 @@
 #include <linux/slab.h>
 #include <linux/errno.h>
 #include <linux/delay.h>
+#include <linux/dmi.h>
 
 #include "tb.h"
 #include "tb_regs.h"
@@ -71,6 +72,8 @@ static void tb_scan_port(struct tb_port *port)
 		return;
 	}
 
+	sw->authorized = true;
+
 	if (tb_switch_add(sw)) {
 		tb_switch_put(sw);
 		return;
@@ -443,10 +446,14 @@ struct tb *tb_probe(struct tb_nhi *nhi)
 	struct tb_cm *tcm;
 	struct tb *tb;
 
+	if (!dmi_match(DMI_BOARD_VENDOR, "Apple Inc."))
+		return NULL;
+
 	tb = tb_domain_alloc(nhi, sizeof(*tcm));
 	if (!tb)
 		return NULL;
 
+	tb->security_level = TB_SECURITY_NONE;
 	tb->cm_ops = &tb_cm_ops;
 
 	tcm = tb_priv(tb);
diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
index 3a7d14b9b7d0..9449497cea4f 100644
--- a/drivers/thunderbolt/tb.h
+++ b/drivers/thunderbolt/tb.h
@@ -15,6 +15,22 @@
 #include "dma_port.h"
 
 /**
+ * enum tb_security_level - Thunderbolt security level
+ * @TB_SECURITY_NONE: No security, legacy mode
+ * @TB_SECURITY_USER: User approval required at minimum
+ * @TB_SECURITY_SECURE: One time saved key required at minimum
+ * @TB_SECURITY_DPONLY: Only tunnel Display port (and USB)
+ */
+enum tb_security_level {
+	TB_SECURITY_NONE,
+	TB_SECURITY_USER,
+	TB_SECURITY_SECURE,
+	TB_SECURITY_DPONLY,
+};
+
+#define TB_SWITCH_KEY_SIZE	32
+
+/**
  * struct tb_switch - a thunderbolt switch
  * @dev: Device for the switch
  * @config: Switch configuration
@@ -33,6 +49,19 @@
  * @cap_plug_events: Offset to the plug events capability (%0 if not found)
  * @is_unplugged: The switch is going away
  * @drom: DROM of the switch (%NULL if not found)
+ * @authorized: Whether the switch is authorized by user or policy
+ * @work: Work used to automatically authorize a switch
+ * @security_level: Switch supported security level
+ * @key: Contains the key used to challenge the device or %NULL if not
+ *	 supported. Size of the key is %TB_SWITCH_KEY_SIZE.
+ * @connection_id: Connection ID used with ICM messaging
+ * @connection_key: Connection key used with ICM messaging
+ * @link: Root switch link this switch is connected (ICM only)
+ * @depth: Depth in the chain this switch is connected (ICM only)
+ *
+ * When the switch is being added or removed to the domain (other
+ * switches) you need to have domain lock held. For switch authorization
+ * internal switch_lock is enough.
  */
 struct tb_switch {
 	struct device dev;
@@ -50,6 +79,14 @@ struct tb_switch {
 	int cap_plug_events;
 	bool is_unplugged;
 	u8 *drom;
+	unsigned int authorized;
+	struct work_struct work;
+	enum tb_security_level security_level;
+	u8 *key;
+	u8 connection_id;
+	u8 connection_key;
+	u8 link;
+	u8 depth;
 };
 
 /**
@@ -121,19 +158,33 @@ struct tb_path {
 
 /**
  * struct tb_cm_ops - Connection manager specific operations vector
+ * @driver_ready: Called right after control channel is started. Used by
+ *		  ICM to send driver ready message to the firmware.
  * @start: Starts the domain
  * @stop: Stops the domain
  * @suspend_noirq: Connection manager specific suspend_noirq
  * @resume_noirq: Connection manager specific resume_noirq
+ * @suspend: Connection manager specific suspend
+ * @complete: Connection manager specific complete
  * @handle_event: Handle thunderbolt event
+ * @approve_switch: Approve switch
+ * @add_switch_key: Add key to switch
+ * @challenge_switch_key: Challenge switch using key
  */
 struct tb_cm_ops {
+	int (*driver_ready)(struct tb *tb);
 	int (*start)(struct tb *tb);
 	void (*stop)(struct tb *tb);
 	int (*suspend_noirq)(struct tb *tb);
 	int (*resume_noirq)(struct tb *tb);
+	int (*suspend)(struct tb *tb);
+	void (*complete)(struct tb *tb);
 	void (*handle_event)(struct tb *tb, enum tb_cfg_pkg_type,
 			     const void *buf, size_t size);
+	int (*approve_switch)(struct tb *tb, struct tb_switch *sw);
+	int (*add_switch_key)(struct tb *tb, struct tb_switch *sw);
+	int (*challenge_switch_key)(struct tb *tb, struct tb_switch *sw,
+				    const u8 *challenge, u8 *response);
 };
 
 /**
@@ -147,6 +198,7 @@ struct tb_cm_ops {
  * @root_switch: Root switch of this domain
  * @cm_ops: Connection manager specific operations vector
  * @index: Linux assigned domain number
+ * @security_level: Current security level
  * @privdata: Private connection manager specific data
  */
 struct tb {
@@ -158,6 +210,7 @@ struct tb {
 	struct tb_switch *root_switch;
 	const struct tb_cm_ops *cm_ops;
 	int index;
+	enum tb_security_level security_level;
 	unsigned long privdata[0];
 };
 
@@ -188,6 +241,16 @@ static inline u64 tb_route(struct tb_switch *sw)
 	return ((u64) sw->config.route_hi) << 32 | sw->config.route_lo;
 }
 
+static inline struct tb_port *tb_port_at(u64 route, struct tb_switch *sw)
+{
+	u8 port;
+
+	port = route >> (sw->config.depth * 8);
+	if (WARN_ON(port > sw->config.max_port_number))
+		return NULL;
+	return &sw->ports[port];
+}
+
 static inline int tb_sw_read(struct tb_switch *sw, void *buffer,
 			     enum tb_cfg_space space, u32 offset, u32 length)
 {
@@ -266,6 +329,7 @@ static inline int tb_port_write(struct tb_port *port, const void *buffer,
 #define tb_port_info(port, fmt, arg...) \
 	__TB_PORT_PRINT(tb_info, port, fmt, ##arg)
 
+struct tb *icm_probe(struct tb_nhi *nhi);
 struct tb *tb_probe(struct tb_nhi *nhi);
 
 extern struct bus_type tb_bus_type;
@@ -280,6 +344,11 @@ int tb_domain_add(struct tb *tb);
 void tb_domain_remove(struct tb *tb);
 int tb_domain_suspend_noirq(struct tb *tb);
 int tb_domain_resume_noirq(struct tb *tb);
+int tb_domain_suspend(struct tb *tb);
+void tb_domain_complete(struct tb *tb);
+int tb_domain_approve_switch(struct tb *tb, struct tb_switch *sw);
+int tb_domain_approve_switch_key(struct tb *tb, struct tb_switch *sw);
+int tb_domain_challenge_switch_key(struct tb *tb, struct tb_switch *sw);
 
 static inline void tb_domain_put(struct tb *tb)
 {
@@ -296,6 +365,9 @@ int tb_switch_resume(struct tb_switch *sw);
 int tb_switch_reset(struct tb *tb, u64 route);
 void tb_sw_set_unplugged(struct tb_switch *sw);
 struct tb_switch *get_switch_at_route(struct tb_switch *sw, u64 route);
+struct tb_switch *tb_switch_find_by_link_depth(struct tb *tb, u8 link,
+					       u8 depth);
+struct tb_switch *tb_switch_find_by_uuid(struct tb *tb, const uuid_be *uuid);
 
 static inline void tb_switch_put(struct tb_switch *sw)
 {
diff --git a/drivers/thunderbolt/tb_msgs.h b/drivers/thunderbolt/tb_msgs.h
index 761d56287149..85b6d33c0919 100644
--- a/drivers/thunderbolt/tb_msgs.h
+++ b/drivers/thunderbolt/tb_msgs.h
@@ -13,6 +13,7 @@
 #define _TB_MSGS
 
 #include <linux/types.h>
+#include <linux/uuid.h>
 
 enum tb_cfg_pkg_type {
 	TB_CFG_PKG_READ = 1,
@@ -24,6 +25,9 @@ enum tb_cfg_pkg_type {
 	TB_CFG_PKG_XDOMAIN_RESP = 7,
 	TB_CFG_PKG_OVERRIDE = 8,
 	TB_CFG_PKG_RESET = 9,
+	TB_CFG_PKG_ICM_EVENT = 10,
+	TB_CFG_PKG_ICM_CMD = 11,
+	TB_CFG_PKG_ICM_RESP = 12,
 	TB_CFG_PKG_PREPARE_TO_SLEEP = 0xd,
 
 };
@@ -105,4 +109,152 @@ struct cfg_pts_pkg {
 	u32 data;
 } __packed;
 
+/* ICM messages */
+
+enum icm_pkg_code {
+	ICM_GET_TOPOLOGY = 0x1,
+	ICM_DRIVER_READY = 0x3,
+	ICM_APPROVE_DEVICE = 0x4,
+	ICM_CHALLENGE_DEVICE = 0x5,
+	ICM_ADD_DEVICE_KEY = 0x6,
+	ICM_GET_ROUTE = 0xa,
+};
+
+enum icm_event_code {
+	ICM_EVENT_DEVICE_CONNECTED = 3,
+	ICM_EVENT_DEVICE_DISCONNECTED = 4,
+};
+
+struct icm_pkg_header {
+	u8 code;
+	u8 flags;
+	u8 packet_id;
+	u8 total_packets;
+} __packed;
+
+#define ICM_FLAGS_ERROR			BIT(0)
+#define ICM_FLAGS_NO_KEY		BIT(1)
+#define ICM_FLAGS_SLEVEL_SHIFT		3
+#define ICM_FLAGS_SLEVEL_MASK		GENMASK(4, 3)
+
+struct icm_pkg_driver_ready {
+	struct icm_pkg_header hdr;
+} __packed;
+
+struct icm_pkg_driver_ready_response {
+	struct icm_pkg_header hdr;
+	u8 romver;
+	u8 ramver;
+	u16 security_level;
+} __packed;
+
+/* Falcon Ridge & Alpine Ridge common messages */
+
+struct icm_fr_pkg_get_topology {
+	struct icm_pkg_header hdr;
+} __packed;
+
+#define ICM_GET_TOPOLOGY_PACKETS	14
+
+struct icm_fr_pkg_get_topology_response {
+	struct icm_pkg_header hdr;
+	u32 route_lo;
+	u32 route_hi;
+	u8 first_data;
+	u8 second_data;
+	u8 drom_i2c_address_index;
+	u8 switch_index;
+	u32 reserved[2];
+	u32 ports[16];
+	u32 port_hop_info[16];
+} __packed;
+
+#define ICM_SWITCH_USED			BIT(0)
+#define ICM_SWITCH_UPSTREAM_PORT_MASK	GENMASK(7, 1)
+#define ICM_SWITCH_UPSTREAM_PORT_SHIFT	1
+
+#define ICM_PORT_TYPE_MASK		GENMASK(23, 0)
+#define ICM_PORT_INDEX_SHIFT		24
+#define ICM_PORT_INDEX_MASK		GENMASK(31, 24)
+
+struct icm_fr_event_device_connected {
+	struct icm_pkg_header hdr;
+	uuid_be ep_uuid;
+	u8 connection_key;
+	u8 connection_id;
+	u16 link_info;
+	u32 ep_name[55];
+} __packed;
+
+#define ICM_LINK_INFO_LINK_MASK		0x7
+#define ICM_LINK_INFO_DEPTH_SHIFT	4
+#define ICM_LINK_INFO_DEPTH_MASK	GENMASK(7, 4)
+#define ICM_LINK_INFO_APPROVED		BIT(8)
+
+struct icm_fr_pkg_approve_device {
+	struct icm_pkg_header hdr;
+	uuid_be ep_uuid;
+	u8 connection_key;
+	u8 connection_id;
+	u16 reserved;
+} __packed;
+
+struct icm_fr_event_device_disconnected {
+	struct icm_pkg_header hdr;
+	u16 reserved;
+	u16 link_info;
+} __packed;
+
+struct icm_fr_pkg_add_device_key {
+	struct icm_pkg_header hdr;
+	uuid_be ep_uuid;
+	u8 connection_key;
+	u8 connection_id;
+	u16 reserved;
+	u32 key[8];
+} __packed;
+
+struct icm_fr_pkg_add_device_key_response {
+	struct icm_pkg_header hdr;
+	uuid_be ep_uuid;
+	u8 connection_key;
+	u8 connection_id;
+	u16 reserved;
+} __packed;
+
+struct icm_fr_pkg_challenge_device {
+	struct icm_pkg_header hdr;
+	uuid_be ep_uuid;
+	u8 connection_key;
+	u8 connection_id;
+	u16 reserved;
+	u32 challenge[8];
+} __packed;
+
+struct icm_fr_pkg_challenge_device_response {
+	struct icm_pkg_header hdr;
+	uuid_be ep_uuid;
+	u8 connection_key;
+	u8 connection_id;
+	u16 reserved;
+	u32 challenge[8];
+	u32 response[8];
+} __packed;
+
+/* Alpine Ridge only messages */
+
+struct icm_ar_pkg_get_route {
+	struct icm_pkg_header hdr;
+	u16 reserved;
+	u16 link_info;
+} __packed;
+
+struct icm_ar_pkg_get_route_response {
+	struct icm_pkg_header hdr;
+	u16 reserved;
+	u16 link_info;
+	u32 route_hi;
+	u32 route_lo;
+} __packed;
+
 #endif
-- 
2.11.0

[PATCH 22/24] thunderbolt: Add support for host and device NVM firmware upgrade

[Mika Westerberg]

Starting from Intel Falcon Ridge the NVM firmware can be upgraded by
using DMA configuration based mailbox commands. If we detect that the
host or device (device support starts from Intel Alpine Ridge) has the
DMA configuration based mailbox we expose NVM information to the
userspace as two separate Linux NVMem devices: nvm_active and
nvm_non_active. The former is read-only portion of the active NVM which
firmware upgrade tools can be use to find out suitable NVM image if the
device identification strings are not enough.

The latter is write-only portion where the new NVM image is to be
written by the userspace. It is up to the userspace to find out right
NVM image (the kernel does very minimal validation). The ICM firmware
itself authenticates the new NVM firmware and fails the operation if it
is not what is expected.

We also expose two new sysfs files per each switch: nvm_version and
nvm_authenticate which can be used to read the active NVM version and
start the upgrade process.

We also introduce safe mode which is the mode a switch goes when it does
not have properly authenticated firmware. In this mode the switch only
accepts a couple of commands including flashing a new NVM firmware image
and triggering power cycle.

This code is based on the work done by Amir Levy and Michael Jamet.

Signed-off-by: Michael Jamet <michael.jamet@intel.com>
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
---
 Documentation/ABI/testing/sysfs-bus-thunderbolt |  26 ++
 drivers/thunderbolt/Kconfig                     |   1 +
 drivers/thunderbolt/domain.c                    |  17 +
 drivers/thunderbolt/icm.c                       |  33 +-
 drivers/thunderbolt/nhi.h                       |   1 +
 drivers/thunderbolt/switch.c                    | 595 +++++++++++++++++++++++-
 drivers/thunderbolt/tb.h                        |  36 ++
 7 files changed, 684 insertions(+), 25 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-bus-thunderbolt b/Documentation/ABI/testing/sysfs-bus-thunderbolt
index c100bdeacfaf..ed88f0eb593b 100644
--- a/Documentation/ABI/testing/sysfs-bus-thunderbolt
+++ b/Documentation/ABI/testing/sysfs-bus-thunderbolt
@@ -80,3 +80,29 @@ KernelVersion:	4.13
 Contact:	thunderbolt-software@lists.01.org
 Description:	This attribute contains unique id (UUID) of this device
 		extracted from the device DROM or hardware registers.
+
+What:		/sys/bus/thunderbolt/devices/.../nvm_version
+Date:		Sep 2017
+KernelVersion:	4.13
+Contact:	thunderbolt-software@lists.01.org
+Description:	If the device has upgradeable firmware the version
+		number is available here. Format: %x.%x, major.minor.
+		If the device is in safe mode reading the file returns
+		-ENODATA instead as the NVM version is not available.
+
+What:		/sys/bus/thunderbolt/devices/.../nvm_authenticate
+Date:		Sep 2017
+KernelVersion:	4.13
+Contact:	thunderbolt-software@lists.01.org
+Description:	When new NVM image is written to the non-active NVM
+		area (through non_activeX NVMem device), the
+		authentication procedure is started by writing 1 to
+		this file. If everything goes well, the device is
+		restarted with the new NVM firmware. If the image
+		verification fails an error code is returned instead.
+
+		When read holds status of the last authentication
+		operation if an error occurred during the process. This
+		is directly the status value from the DMA configuration
+		based mailbox before the device is power cycled. Writing
+		0 here clears the status.
diff --git a/drivers/thunderbolt/Kconfig b/drivers/thunderbolt/Kconfig
index a9cc724985ad..f4869c38c7e4 100644
--- a/drivers/thunderbolt/Kconfig
+++ b/drivers/thunderbolt/Kconfig
@@ -6,6 +6,7 @@ menuconfig THUNDERBOLT
 	select CRC32
 	select CRYPTO
 	select CRYPTO_HASH
+	select NVMEM
 	help
 	  Thunderbolt Controller driver. This driver is required if you
 	  want to hotplug Thunderbolt devices on Apple hardware or on PCs
diff --git a/drivers/thunderbolt/domain.c b/drivers/thunderbolt/domain.c
index b8754c222ab4..d5075266b202 100644
--- a/drivers/thunderbolt/domain.c
+++ b/drivers/thunderbolt/domain.c
@@ -427,6 +427,23 @@ int tb_domain_challenge_switch_key(struct tb *tb, struct tb_switch *sw)
 	return ret;
 }
 
+/**
+ * tb_domain_disconnect_pcie_paths() - Disconnect all PCIe paths
+ * @tb: Domain whose PCIe paths to disconnect
+ *
+ * This needs to be called in preparation for NVM upgrade of the host
+ * controller. Makes sure all PCIe paths are disconnected.
+ *
+ * Return %0 on success and negative errno in case of error.
+ */
+int tb_domain_disconnect_pcie_paths(struct tb *tb)
+{
+	if (!tb->cm_ops->disconnect_pcie_paths)
+		return -EPERM;
+
+	return tb->cm_ops->disconnect_pcie_paths(tb);
+}
+
 int tb_domain_init(void)
 {
 	return bus_register(&tb_bus_type);
diff --git a/drivers/thunderbolt/icm.c b/drivers/thunderbolt/icm.c
index c99b2ae0e7ab..27fcba91aba2 100644
--- a/drivers/thunderbolt/icm.c
+++ b/drivers/thunderbolt/icm.c
@@ -63,6 +63,7 @@
  *		   where ICM needs to be started manually
  * @vnd_cap: Vendor defined capability where PCIe2CIO mailbox resides
  *	     (only set when @upstream_port is not %NULL)
+ * @safe_mode: ICM is in safe mode
  * @is_supported: Checks if we can support ICM on this controller
  * @get_mode: Read and return the ICM firmware mode (optional)
  * @get_route: Find a route string for given switch
@@ -74,6 +75,7 @@ struct icm {
 	struct delayed_work rescan_work;
 	struct pci_dev *upstream_port;
 	int vnd_cap;
+	bool safe_mode;
 	bool (*is_supported)(struct tb *tb);
 	int (*get_mode)(struct tb *tb);
 	int (*get_route)(struct tb *tb, u8 link, u8 depth, u64 *route);
@@ -861,6 +863,10 @@ static int icm_firmware_init(struct tb *tb)
 		ret = icm->get_mode(tb);
 
 		switch (ret) {
+		case NHI_FW_SAFE_MODE:
+			icm->safe_mode = true;
+			break;
+
 		case NHI_FW_CM_MODE:
 			/* Ask ICM to accept all Thunderbolt devices */
 			nhi_mailbox_cmd(nhi, NHI_MAILBOX_ALLOW_ALL_DEVS, 0);
@@ -888,12 +894,20 @@ static int icm_firmware_init(struct tb *tb)
 
 static int icm_driver_ready(struct tb *tb)
 {
+	struct icm *icm = tb_priv(tb);
 	int ret;
 
 	ret = icm_firmware_init(tb);
 	if (ret)
 		return ret;
 
+	if (icm->safe_mode) {
+		tb_info(tb, "Thunderbolt host controller is in safe mode.\n");
+		tb_info(tb, "You need to update NVM firmware of the controller before it can be used.\n");
+		tb_info(tb, "For latest updates check https://thunderbolttechnology.net/updates.\n");
+		return 0;
+	}
+
 	return __icm_driver_ready(tb, &tb->security_level);
 }
 
@@ -984,12 +998,23 @@ static void icm_complete(struct tb *tb)
 
 static int icm_start(struct tb *tb)
 {
+	struct icm *icm = tb_priv(tb);
 	int ret;
 
-	tb->root_switch = tb_switch_alloc(tb, &tb->dev, 0);
+	if (icm->safe_mode)
+		tb->root_switch = tb_switch_alloc_safe_mode(tb, &tb->dev, 0);
+	else
+		tb->root_switch = tb_switch_alloc(tb, &tb->dev, 0);
 	if (!tb->root_switch)
 		return -ENODEV;
 
+	/*
+	 * NVM upgrade has not been tested on Apple systems and they
+	 * don't provide images publicly either. To be on the safe side
+	 * prevent root switch NVM upgrade on Macs for now.
+	 */
+	tb->root_switch->no_nvm_upgrade = is_apple();
+
 	ret = tb_switch_add(tb->root_switch);
 	if (ret)
 		tb_switch_put(tb->root_switch);
@@ -1007,6 +1032,11 @@ static void icm_stop(struct tb *tb)
 	nhi_mailbox_cmd(tb->nhi, NHI_MAILBOX_DRV_UNLOADS, 0);
 }
 
+static int icm_disconnect_pcie_paths(struct tb *tb)
+{
+	return nhi_mailbox_cmd(tb->nhi, NHI_MAILBOX_DISCONNECT_PCIE_PATHS, 0);
+}
+
 /* Falcon Ridge and Alpine Ridge */
 static const struct tb_cm_ops icm_fr_ops = {
 	.driver_ready = icm_driver_ready,
@@ -1018,6 +1048,7 @@ static const struct tb_cm_ops icm_fr_ops = {
 	.approve_switch = icm_fr_approve_switch,
 	.add_switch_key = icm_fr_add_switch_key,
 	.challenge_switch_key = icm_fr_challenge_switch_key,
+	.disconnect_pcie_paths = icm_disconnect_pcie_paths,
 };
 
 struct tb *icm_probe(struct tb_nhi *nhi)
diff --git a/drivers/thunderbolt/nhi.h b/drivers/thunderbolt/nhi.h
index 243f5fcfa367..6acf07594758 100644
--- a/drivers/thunderbolt/nhi.h
+++ b/drivers/thunderbolt/nhi.h
@@ -155,6 +155,7 @@ enum nhi_fw_mode {
 
 enum nhi_mailbox_cmd {
 	NHI_MAILBOX_SAVE_DEVS = 0x05,
+	NHI_MAILBOX_DISCONNECT_PCIE_PATHS = 0x06,
 	NHI_MAILBOX_DRV_UNLOADS = 0x07,
 	NHI_MAILBOX_ALLOW_ALL_DEVS = 0x23,
 };
diff --git a/drivers/thunderbolt/switch.c b/drivers/thunderbolt/switch.c
index 5dc96ad1fc68..d6f6880ec6b9 100644
--- a/drivers/thunderbolt/switch.c
+++ b/drivers/thunderbolt/switch.c
@@ -5,13 +5,474 @@
  */
 
 #include <linux/delay.h>
+#include <linux/idr.h>
+#include <linux/nvmem-provider.h>
+#include <linux/sizes.h>
 #include <linux/slab.h>
+#include <linux/vmalloc.h>
 
 #include "tb.h"
 
 /* Switch authorization from userspace is serialized by this lock */
 static DEFINE_MUTEX(switch_lock);
 
+/* Switch NVM support */
+
+#define NVM_DEVID		0x05
+#define NVM_VERSION		0x08
+#define NVM_CSS			0x10
+#define NVM_FLASH_SIZE		0x45
+
+#define NVM_MIN_SIZE		SZ_32K
+#define NVM_MAX_SIZE		SZ_512K
+
+static DEFINE_IDA(nvm_ida);
+
+struct nvm_auth_status {
+	struct list_head list;
+	uuid_be uuid;
+	u32 status;
+};
+
+/*
+ * Hold NVM authentication failure status per switch This information
+ * needs to stay around even when the switch gets power cycled so we
+ * keep it separately.
+ */
+static LIST_HEAD(nvm_auth_status_cache);
+static DEFINE_MUTEX(nvm_auth_status_lock);
+
+static struct nvm_auth_status *__nvm_get_auth_status(const struct tb_switch *sw)
+{
+	struct nvm_auth_status *st;
+
+	list_for_each_entry(st, &nvm_auth_status_cache, list) {
+		if (!uuid_be_cmp(st->uuid, *sw->uuid))
+			return st;
+	}
+
+	return NULL;
+}
+
+static void nvm_get_auth_status(const struct tb_switch *sw, u32 *status)
+{
+	struct nvm_auth_status *st;
+
+	mutex_lock(&nvm_auth_status_lock);
+	st = __nvm_get_auth_status(sw);
+	mutex_unlock(&nvm_auth_status_lock);
+
+	*status = st ? st->status : 0;
+}
+
+static void nvm_set_auth_status(const struct tb_switch *sw, u32 status)
+{
+	struct nvm_auth_status *st;
+
+	if (WARN_ON(!sw->uuid))
+		return;
+
+	mutex_lock(&nvm_auth_status_lock);
+	st = __nvm_get_auth_status(sw);
+
+	if (!st) {
+		st = kzalloc(sizeof(*st), GFP_KERNEL);
+		if (!st)
+			goto unlock;
+
+		memcpy(&st->uuid, sw->uuid, sizeof(st->uuid));
+		INIT_LIST_HEAD(&st->list);
+		list_add_tail(&st->list, &nvm_auth_status_cache);
+	}
+
+	st->status = status;
+unlock:
+	mutex_unlock(&nvm_auth_status_lock);
+}
+
+static void nvm_clear_auth_status(const struct tb_switch *sw)
+{
+	struct nvm_auth_status *st;
+
+	mutex_lock(&nvm_auth_status_lock);
+	st = __nvm_get_auth_status(sw);
+	if (st) {
+		list_del(&st->list);
+		kfree(st);
+	}
+	mutex_unlock(&nvm_auth_status_lock);
+}
+
+static int nvm_validate_and_write(struct tb_switch *sw)
+{
+	unsigned int image_size, hdr_size;
+	const u8 *buf = sw->nvm->buf;
+	u16 ds_size;
+	int ret;
+
+	if (!buf)
+		return -EINVAL;
+
+	image_size = sw->nvm->buf_data_size;
+	if (image_size < NVM_MIN_SIZE || image_size > NVM_MAX_SIZE)
+		return -EINVAL;
+
+	/*
+	 * FARB pointer must point inside the image and must at least
+	 * contain parts of the digital section we will be reading here.
+	 */
+	hdr_size = (*(u32 *)buf) & 0xffffff;
+	if (hdr_size + NVM_DEVID + 2 >= image_size)
+		return -EINVAL;
+
+	/* Digital section start should be aligned to 4k page */
+	if (!IS_ALIGNED(hdr_size, SZ_4K))
+		return -EINVAL;
+
+	/*
+	 * Read digital section size and check that it also fits inside
+	 * the image.
+	 */
+	ds_size = *(u16 *)(buf + hdr_size);
+	if (ds_size >= image_size)
+		return -EINVAL;
+
+	if (!sw->safe_mode) {
+		u16 device_id;
+
+		/*
+		 * Make sure the device ID in the image matches the one
+		 * we read from the switch config space.
+		 */
+		device_id = *(u16 *)(buf + hdr_size + NVM_DEVID);
+		if (device_id != sw->config.device_id)
+			return -EINVAL;
+
+		if (sw->generation < 3) {
+			/* Write CSS headers first */
+			ret = dma_port_flash_write(sw->dma_port,
+				DMA_PORT_CSS_ADDRESS, buf + NVM_CSS,
+				DMA_PORT_CSS_MAX_SIZE);
+			if (ret)
+				return ret;
+		}
+
+		/* Skip headers in the image */
+		buf += hdr_size;
+		image_size -= hdr_size;
+	}
+
+	return dma_port_flash_write(sw->dma_port, 0, buf, image_size);
+}
+
+static int nvm_authenticate_host(struct tb_switch *sw)
+{
+	int ret;
+
+	/*
+	 * Root switch NVM upgrade requires that we disconnect the
+	 * existing PCIe paths first (in case it is not in safe mode
+	 * already).
+	 */
+	if (!sw->safe_mode) {
+		ret = tb_domain_disconnect_pcie_paths(sw->tb);
+		if (ret)
+			return ret;
+		/*
+		 * The host controller goes away pretty soon after this if
+		 * everything goes well so getting timeout is expected.
+		 */
+		ret = dma_port_flash_update_auth(sw->dma_port);
+		return ret == -ETIMEDOUT ? 0 : ret;
+	}
+
+	/*
+	 * From safe mode we can get out by just power cycling the
+	 * switch.
+	 */
+	dma_port_power_cycle(sw->dma_port);
+	return 0;
+}
+
+static int nvm_authenticate_device(struct tb_switch *sw)
+{
+	int ret, retries = 10;
+
+	ret = dma_port_flash_update_auth(sw->dma_port);
+	if (ret && ret != -ETIMEDOUT)
+		return ret;
+
+	/*
+	 * Poll here for the authentication status. It takes some time
+	 * for the device to respond (we get timeout for a while). Once
+	 * we get response the device needs to be power cycled in order
+	 * to the new NVM to be taken into use.
+	 */
+	do {
+		u32 status;
+
+		ret = dma_port_flash_update_auth_status(sw->dma_port, &status);
+		if (ret < 0 && ret != -ETIMEDOUT)
+			return ret;
+		if (ret > 0) {
+			if (status) {
+				tb_sw_warn(sw, "failed to authenticate NVM\n");
+				nvm_set_auth_status(sw, status);
+			}
+
+			tb_sw_info(sw, "power cycling the switch now\n");
+			dma_port_power_cycle(sw->dma_port);
+			return 0;
+		}
+
+		msleep(500);
+	} while (--retries);
+
+	return -ETIMEDOUT;
+}
+
+static ssize_t nvm_authenticate_show(struct device *dev,
+	struct device_attribute *attr, char *buf)
+{
+	struct tb_switch *sw = tb_to_switch(dev);
+	u32 status;
+
+	nvm_get_auth_status(sw, &status);
+	return sprintf(buf, "%#x\n", status);
+}
+
+static ssize_t nvm_authenticate_store(struct device *dev,
+	struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct tb_switch *sw = tb_to_switch(dev);
+	unsigned int val;
+	int ret;
+
+	if (mutex_lock_interruptible(&switch_lock))
+		return -ERESTARTSYS;
+
+	ret = kstrtouint(buf, 0, &val);
+	if (ret)
+		goto unlock;
+
+	switch (val) {
+	case 0:
+		/* Just clear the authentication status */
+		nvm_clear_auth_status(sw);
+		break;
+
+	case 1:
+		ret = nvm_validate_and_write(sw);
+		if (ret)
+			goto unlock;
+
+		sw->nvm->authenticating = true;
+
+		if (!tb_route(sw))
+			ret = nvm_authenticate_host(sw);
+		else
+			ret = nvm_authenticate_device(sw);
+		break;
+
+	default:
+		ret = -EINVAL;
+		break;
+	}
+
+unlock:
+	mutex_unlock(&switch_lock);
+
+	if (ret)
+		return ret;
+	return count;
+}
+static DEVICE_ATTR_RW(nvm_authenticate);
+
+static ssize_t nvm_version_show(struct device *dev,
+				struct device_attribute *attr, char *buf)
+{
+	struct tb_switch *sw = tb_to_switch(dev);
+
+	if (sw->safe_mode)
+		return -ENODATA;
+	return sprintf(buf, "%x.%x\n", sw->nvm->major, sw->nvm->minor);
+}
+static DEVICE_ATTR_RO(nvm_version);
+
+static struct attribute *nvm_attrs[] = {
+	&dev_attr_nvm_authenticate.attr,
+	&dev_attr_nvm_version.attr,
+	NULL,
+};
+
+static struct attribute_group nvm_group = {
+	.attrs = nvm_attrs,
+};
+
+static int tb_switch_nvm_read(void *priv, unsigned int offset, void *val,
+			      size_t bytes)
+{
+	struct tb_switch *sw = priv;
+
+	return dma_port_flash_read(sw->dma_port, offset, val, bytes);
+}
+
+static int tb_switch_nvm_write(void *priv, unsigned int offset, void *val,
+			       size_t bytes)
+{
+	struct tb_switch *sw = priv;
+	int ret = 0;
+
+	if (mutex_lock_interruptible(&switch_lock))
+		return -ERESTARTSYS;
+
+	/*
+	 * Since writing the NVM image might require some special steps,
+	 * for example when CSS headers are written, we cache the image
+	 * locally here and handle the special cases when the user asks
+	 * us to authenticate the image.
+	 */
+	if (!sw->nvm->buf) {
+		sw->nvm->buf = vmalloc(NVM_MAX_SIZE);
+		if (!sw->nvm->buf) {
+			ret = -ENOMEM;
+			goto unlock;
+		}
+	}
+
+	sw->nvm->buf_data_size = offset + bytes;
+	memcpy(sw->nvm->buf + offset, val, bytes);
+
+unlock:
+	mutex_unlock(&switch_lock);
+
+	return ret;
+}
+
+static struct nvmem_device *register_nvmem(struct tb_switch *sw, int id,
+					   size_t size, bool active)
+{
+	struct nvmem_config config;
+
+	memset(&config, 0, sizeof(config));
+
+	if (active) {
+		config.name = "nvm_active";
+		config.reg_read = tb_switch_nvm_read;
+	} else {
+		config.name = "nvm_non_active";
+		config.reg_write = tb_switch_nvm_write;
+	}
+
+	config.id = id;
+	config.stride = 4;
+	config.word_size = 4;
+	config.size = size;
+	config.dev = &sw->dev;
+	config.owner = THIS_MODULE;
+	config.root_only = true;
+	config.priv = sw;
+
+	return nvmem_register(&config);
+}
+
+static int tb_switch_nvm_add(struct tb_switch *sw)
+{
+	struct nvmem_device *nvm_dev;
+	struct tb_switch_nvm *nvm;
+	u32 val;
+	int ret;
+
+	if (!sw->dma_port)
+		return 0;
+
+	nvm = kzalloc(sizeof(*nvm), GFP_KERNEL);
+	if (!nvm)
+		return -ENOMEM;
+
+	nvm->id = ida_simple_get(&nvm_ida, 0, 0, GFP_KERNEL);
+
+	/*
+	 * If the switch is in safe-mode the only accessible portion of
+	 * the NVM is the non-active one where userspace is expected to
+	 * write new functional NVM.
+	 */
+	if (!sw->safe_mode) {
+		u32 nvm_size, hdr_size;
+
+		ret = dma_port_flash_read(sw->dma_port, NVM_FLASH_SIZE, &val,
+					  sizeof(val));
+		if (ret)
+			goto err_ida;
+
+		hdr_size = sw->generation < 3 ? SZ_8K : SZ_16K;
+		nvm_size = (SZ_1M << (val & 7)) / 8;
+		nvm_size = (nvm_size - hdr_size) / 2;
+
+		ret = dma_port_flash_read(sw->dma_port, NVM_VERSION, &val,
+					  sizeof(val));
+		if (ret)
+			goto err_ida;
+
+		nvm->major = val >> 16 & 0xff;
+		nvm->minor = val >> 8 & 0xff;
+
+		nvm_dev = register_nvmem(sw, nvm->id, nvm_size, true);
+		if (IS_ERR(nvm_dev)) {
+			ret = PTR_ERR(nvm_dev);
+			goto err_ida;
+		}
+		nvm->active = nvm_dev;
+	}
+
+	nvm_dev = register_nvmem(sw, nvm->id, NVM_MAX_SIZE, false);
+	if (IS_ERR(nvm_dev)) {
+		ret = PTR_ERR(nvm_dev);
+		goto err_nvm_active;
+	}
+	nvm->non_active = nvm_dev;
+
+	sw->nvm = nvm;
+
+	ret = sysfs_create_group(&sw->dev.kobj, &nvm_group);
+	if (ret) {
+		sw->nvm = NULL;
+		goto err_nvm_nonactive;
+	}
+
+	return 0;
+
+err_nvm_nonactive:
+	nvmem_unregister(nvm->non_active);
+err_nvm_active:
+	if (nvm->active)
+		nvmem_unregister(nvm->active);
+err_ida:
+	ida_simple_remove(&nvm_ida, nvm->id);
+	kfree(nvm);
+
+	return ret;
+}
+
+static void tb_switch_nvm_remove(struct tb_switch *sw)
+{
+	if (!sw->nvm)
+		return;
+
+	/* Remove authentication status in case the switch is unplugged */
+	if (!sw->nvm->authenticating)
+		nvm_clear_auth_status(sw);
+
+	sysfs_remove_group(&sw->dev.kobj, &nvm_group);
+	nvmem_unregister(sw->nvm->non_active);
+	if (sw->nvm->active)
+		nvmem_unregister(sw->nvm->active);
+	ida_simple_remove(&nvm_ida, sw->nvm->id);
+	vfree(sw->nvm->buf);
+	kfree(sw->nvm);
+	sw->nvm = NULL;
+}
+
 /* port utility functions */
 
 static const char *tb_port_type(struct tb_regs_port_header *port)
@@ -617,6 +1078,44 @@ struct tb_switch *tb_switch_alloc(struct tb *tb, struct device *parent,
 }
 
 /**
+ * tb_switch_alloc_safe_mode() - allocate a switch that is in safe mode
+ * @tb: Pointer to the owning domain
+ * @parent: Parent device for this switch
+ * @route: Route string for this switch
+ *
+ * This creates a switch in safe mode. This means the switch pretty much
+ * lacks all capabilities except DMA configuration port before it is
+ * flashed with a valid NVM firmware.
+ *
+ * The returned switch must be released by calling tb_switch_put().
+ *
+ * Return: Pointer to the allocated switch or %NULL in case of failure
+ */
+struct tb_switch *
+tb_switch_alloc_safe_mode(struct tb *tb, struct device *parent, u64 route)
+{
+	struct tb_switch *sw;
+
+	sw = kzalloc(sizeof(*sw), GFP_KERNEL);
+	if (!sw)
+		return NULL;
+
+	sw->tb = tb;
+	sw->config.depth = tb_route_length(route);
+	sw->config.route_hi = upper_32_bits(route);
+	sw->config.route_lo = lower_32_bits(route);
+	sw->safe_mode = true;
+
+	device_initialize(&sw->dev);
+	sw->dev.parent = parent;
+	sw->dev.bus = &tb_bus_type;
+	sw->dev.type = &tb_switch_type;
+	dev_set_name(&sw->dev, "%u-%llx", tb->index, tb_route(sw));
+
+	return sw;
+}
+
+/**
  * tb_switch_configure() - Uploads configuration to the switch
  * @sw: Switch to configure
  *
@@ -696,8 +1195,11 @@ static void tb_switch_set_uuid(struct tb_switch *sw)
 	sw->uuid = kmemdup(uuid, sizeof(uuid), GFP_KERNEL);
 }
 
-static void tb_switch_add_dma_port(struct tb_switch *sw)
+static int tb_switch_add_dma_port(struct tb_switch *sw)
 {
+	u32 status;
+	int ret;
+
 	switch (sw->generation) {
 	case 3:
 		break;
@@ -705,14 +1207,49 @@ static void tb_switch_add_dma_port(struct tb_switch *sw)
 	case 2:
 		/* Only root switch can be upgraded */
 		if (tb_route(sw))
-			return;
+			return 0;
 		break;
 
 	default:
-		return;
+		/*
+		 * DMA port is the only thing available when the switch
+		 * is in safe mode.
+		 */
+		if (!sw->safe_mode)
+			return 0;
+		break;
 	}
 
+	if (sw->no_nvm_upgrade)
+		return 0;
+
 	sw->dma_port = dma_port_alloc(sw);
+	if (!sw->dma_port)
+		return 0;
+
+	/*
+	 * Check status of the previous flash authentication. If there
+	 * is one we need to power cycle the switch in any case to make
+	 * it functional again.
+	 */
+	ret = dma_port_flash_update_auth_status(sw->dma_port, &status);
+	if (ret <= 0)
+		return ret;
+
+	if (status) {
+		tb_sw_info(sw, "switch flash authentication failed\n");
+		tb_switch_set_uuid(sw);
+		nvm_set_auth_status(sw, status);
+	}
+
+	tb_sw_info(sw, "power cycling the switch now\n");
+	dma_port_power_cycle(sw->dma_port);
+
+	/*
+	 * We return error here which causes the switch adding failure.
+	 * It should appear back after power cycle is complete.
+	 */
+	return -ESHUTDOWN;
 }
 
 /**
@@ -738,34 +1275,43 @@ int tb_switch_add(struct tb_switch *sw)
 	 * to the userspace. NVM can be accessed through DMA
 	 * configuration based mailbox.
 	 */
-	tb_switch_add_dma_port(sw);
-
-	/* read drom */
-	ret = tb_drom_read(sw);
-	if (ret) {
-		tb_sw_warn(sw, "tb_eeprom_read_rom failed\n");
+	ret = tb_switch_add_dma_port(sw);
+	if (ret)
 		return ret;
-	}
-	tb_sw_info(sw, "uid: %#llx\n", sw->uid);
 
-	tb_switch_set_uuid(sw);
+	if (!sw->safe_mode) {
+		/* read drom */
+		ret = tb_drom_read(sw);
+		if (ret) {
+			tb_sw_warn(sw, "tb_eeprom_read_rom failed\n");
+			return ret;
+		}
+		tb_sw_info(sw, "uid: %#llx\n", sw->uid);
 
-	for (i = 0; i <= sw->config.max_port_number; i++) {
-		if (sw->ports[i].disabled) {
-			tb_port_info(&sw->ports[i], "disabled by eeprom\n");
-			continue;
+		tb_switch_set_uuid(sw);
+
+		for (i = 0; i <= sw->config.max_port_number; i++) {
+			if (sw->ports[i].disabled) {
+				tb_port_info(&sw->ports[i], "disabled by eeprom\n");
+				continue;
+			}
+			ret = tb_init_port(&sw->ports[i]);
+			if (ret)
+				return ret;
 		}
-		ret = tb_init_port(&sw->ports[i]);
-		if (ret)
-			return ret;
+
+		if (sw->security_level == TB_SECURITY_SECURE)
+			sw->dev.groups = secure_switch_groups;
+		else
+			sw->dev.groups = switch_groups;
 	}
 
-	if (sw->security_level == TB_SECURITY_SECURE)
-		sw->dev.groups = secure_switch_groups;
-	else
-		sw->dev.groups = switch_groups;
+	ret = device_add(&sw->dev);
+	if (ret)
+		return ret;
 
-	return device_add(&sw->dev);
+	tb_switch_nvm_add(sw);
+	return 0;
 }
 
 /**
@@ -792,6 +1338,7 @@ void tb_switch_remove(struct tb_switch *sw)
 	if (!sw->is_unplugged)
 		tb_plug_events_active(sw, false);
 
+	tb_switch_nvm_remove(sw);
 	device_unregister(&sw->dev);
 }
 
diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
index 9449497cea4f..b3029cebe714 100644
--- a/drivers/thunderbolt/tb.h
+++ b/drivers/thunderbolt/tb.h
@@ -7,6 +7,7 @@
 #ifndef TB_H_
 #define TB_H_
 
+#include <linux/nvmem-provider.h>
 #include <linux/pci.h>
 #include <linux/uuid.h>
 
@@ -15,6 +16,30 @@
 #include "dma_port.h"
 
 /**
+ * struct tb_switch_nvm - Structure holding switch NVM information
+ * @major: Major version number of the active NVM portion
+ * @minor: Minor version number of the active NVM portion
+ * @id: Identifier used with both NVM portions
+ * @active: Active portion NVMem device
+ * @non_active: Non-active portion NVMem device
+ * @buf: Buffer where the NVM image is stored before it is written to
+ *	 the actual NVM flash device
+ * @buf_data_size: Number of bytes actually consumed by the new NVM
+ *		   image
+ * @authenticating: The switch is authenticating the new NVM
+ */
+struct tb_switch_nvm {
+	u8 major;
+	u8 minor;
+	int id;
+	struct nvmem_device *active;
+	struct nvmem_device *non_active;
+	void *buf;
+	size_t buf_data_size;
+	bool authenticating;
+};
+
+/**
  * enum tb_security_level - Thunderbolt security level
  * @TB_SECURITY_NONE: No security, legacy mode
  * @TB_SECURITY_USER: User approval required at minimum
@@ -49,6 +74,9 @@ enum tb_security_level {
  * @cap_plug_events: Offset to the plug events capability (%0 if not found)
  * @is_unplugged: The switch is going away
  * @drom: DROM of the switch (%NULL if not found)
+ * @nvm: Pointer to the NVM if the switch has one (%NULL otherwise)
+ * @no_nvm_upgrade: Prevent NVM upgrade of this switch
+ * @safe_mode: The switch is in safe-mode
  * @authorized: Whether the switch is authorized by user or policy
  * @work: Work used to automatically authorize a switch
  * @security_level: Switch supported security level
@@ -79,6 +107,9 @@ struct tb_switch {
 	int cap_plug_events;
 	bool is_unplugged;
 	u8 *drom;
+	struct tb_switch_nvm *nvm;
+	bool no_nvm_upgrade;
+	bool safe_mode;
 	unsigned int authorized;
 	struct work_struct work;
 	enum tb_security_level security_level;
@@ -170,6 +201,7 @@ struct tb_path {
  * @approve_switch: Approve switch
  * @add_switch_key: Add key to switch
  * @challenge_switch_key: Challenge switch using key
+ * @disconnect_pcie_paths: Disconnects PCIe paths before NVM update
  */
 struct tb_cm_ops {
 	int (*driver_ready)(struct tb *tb);
@@ -185,6 +217,7 @@ struct tb_cm_ops {
 	int (*add_switch_key)(struct tb *tb, struct tb_switch *sw);
 	int (*challenge_switch_key)(struct tb *tb, struct tb_switch *sw,
 				    const u8 *challenge, u8 *response);
+	int (*disconnect_pcie_paths)(struct tb *tb);
 };
 
 /**
@@ -349,6 +382,7 @@ void tb_domain_complete(struct tb *tb);
 int tb_domain_approve_switch(struct tb *tb, struct tb_switch *sw);
 int tb_domain_approve_switch_key(struct tb *tb, struct tb_switch *sw);
 int tb_domain_challenge_switch_key(struct tb *tb, struct tb_switch *sw);
+int tb_domain_disconnect_pcie_paths(struct tb *tb);
 
 static inline void tb_domain_put(struct tb *tb)
 {
@@ -357,6 +391,8 @@ static inline void tb_domain_put(struct tb *tb)
 
 struct tb_switch *tb_switch_alloc(struct tb *tb, struct device *parent,
 				  u64 route);
+struct tb_switch *tb_switch_alloc_safe_mode(struct tb *tb,
+			struct device *parent, u64 route);
 int tb_switch_configure(struct tb_switch *sw);
 int tb_switch_add(struct tb_switch *sw);
 void tb_switch_remove(struct tb_switch *sw);
-- 
2.11.0

[PATCH 23/24] thunderbolt: Add documentation how Thunderbolt bus can be used

[Mika Westerberg]

Since there are no such tool yet that handles all the low-level details
of connecting devices and upgrading their firmware, add a small document
that shows how the Thunderbolt bus can be used directly from command
line.

Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Reviewed-by: Michael Jamet <michael.jamet@intel.com>
---
 Documentation/admin-guide/index.rst       |   1 +
 Documentation/admin-guide/thunderbolt.rst | 197 ++++++++++++++++++++++++++++++
 2 files changed, 198 insertions(+)
 create mode 100644 Documentation/admin-guide/thunderbolt.rst

diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst
index 8c60a8a32a1a..6d99a7ce6e21 100644
--- a/Documentation/admin-guide/index.rst
+++ b/Documentation/admin-guide/index.rst
@@ -61,6 +61,7 @@ configure specific aspects of kernel behavior to your liking.
    java
    ras
    pm/index
+   thunderbolt
 
 .. only::  subproject and html
 
diff --git a/Documentation/admin-guide/thunderbolt.rst b/Documentation/admin-guide/thunderbolt.rst
new file mode 100644
index 000000000000..c985241c8521
--- /dev/null
+++ b/Documentation/admin-guide/thunderbolt.rst
@@ -0,0 +1,197 @@
+=============
+ Thunderbolt
+=============
+The interface presented here is not meant for end users. Instead there
+should be a userspace tool that handles all the low-level details, keeps
+database of the authorized devices and prompts user for new connections.
+
+More details about the sysfs interface for Thunderbolt devices can be
+found in ``Documentation/ABI/testing/sysfs-bus-thunderbolt``.
+
+Those users who just want to connect any device without any sort of
+manual work, can add following line to
+``/etc/udev/rules.d/99-local.rules``::
+
+  ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1"
+
+This will authorize all devices automatically when they appear. However,
+keep in mind that this bypasses the security levels and makes the system
+vulnerable to DMA attacks.
+
+Security levels and how to use them
+-----------------------------------
+Starting from Intel Falcon Ridge Thunderbolt controller there are 4
+security levels available. The reason for these is the fact that the
+connected devices can be DMA masters and thus read contents of the host
+memory without CPU and OS knowing about it. There are ways to prevent
+this by setting up an IOMMU but it is not always available for various
+reasons.
+
+The security levels are as follows:
+
+  none
+    All devices are automatically connected by the firmware. No user
+    approval is needed. In BIOS settings this is typically called
+    *Legacy mode*.
+
+  user
+    User is asked whether the device is allowed to be connected.
+    Based on the device identification information available through
+    ``/sys/bus/thunderbolt/devices``. user then can do the decision.
+    In BIOS settings this is typically called *Unique ID*.
+
+  secure
+    User is asked whether the device is allowed to be connected. In
+    addition to UUID the device (if it supports secure connect) is sent
+    a challenge that should match the expected one based on a random key
+    written to ``key`` sysfs attribute. In BIOS settings this is
+    typically called *One time saved key*.
+
+  dponly
+    The firmware automatically creates tunnels for Display Port and
+    USB. No PCIe tunneling is done. In BIOS settings this is
+    typically called *Display Port Only*.
+
+The current security level can be read from
+``/sys/bus/thunderbolt/devices/domainX/security`` where ``domainX`` is
+the Thunderbolt domain the host controller manages. There is typically
+one domain per Thunderbolt host controller.
+
+If the security level reads as ``user`` or ``secure`` the connected
+device must be authorized by the user before PCIe tunnels are created
+(e.g the PCIe device appears).
+
+Each Thunderbolt device plugged in will appear in sysfs under
+``/sys/bus/thunderbolt/devices``. The device directory carries
+information that can be used to identify the particular device,
+including its name and UUID.
+
+Authorizing devices when security level is ``user`` or ``secure``
+-----------------------------------------------------------------
+When a device is plugged in it will appear in sysfs as follows::
+
+  /sys/bus/thunderbolt/devices/0-1/authorized	- 0
+  /sys/bus/thunderbolt/devices/0-1/device	- 0x8004
+  /sys/bus/thunderbolt/devices/0-1/device_name	- Thunderbolt to FireWire Adapter
+  /sys/bus/thunderbolt/devices/0-1/vendor	- 0x1
+  /sys/bus/thunderbolt/devices/0-1/vendor_name	- Apple, Inc.
+  /sys/bus/thunderbolt/devices/0-1/unique_id	- e0376f00-0300-0100-ffff-ffffffffffff
+
+The ``authorized`` attribute reads 0 which means no PCIe tunnels are
+created yet. The user can authorize the device by simply::
+
+  # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized
+
+This will create the PCIe tunnels and the device is now connected.
+
+If the device supports secure connect it has an additional attribute
+``key`` which can hold a random value used for authorization and
+challenging the device in future connects::
+
+  /sys/bus/thunderbolt/devices/0-3/authorized	- 0
+  /sys/bus/thunderbolt/devices/0-3/device	- 0x305
+  /sys/bus/thunderbolt/devices/0-3/device_name	- AKiTiO Thunder3 PCIe Box
+  /sys/bus/thunderbolt/devices/0-3/key		-
+  /sys/bus/thunderbolt/devices/0-3/vendor	- 0x41
+  /sys/bus/thunderbolt/devices/0-3/vendor_name	- inXtron
+  /sys/bus/thunderbolt/devices/0-3/unique_id	- dc010000-0000-8508-a22d-32ca6421cb16
+
+Notice the key is empty by default.
+
+If the user does not want to use secure connect it can just echo 1 to
+the ``authorized`` attribute and the PCIe tunnels will be created.
+
+If the user wants to use secure connect, the first time the device is
+plugged a key needs to be created and send to the device::
+
+  # key=$(openssl rand -hex 32)
+  # echo $key > /sys/bus/thunderbolt/devices/0-3/key
+  # echo 1 > /sys/bus/thunderbolt/devices/0-3/authorized
+
+Now the device is connected (PCIe tunnels are created) and in addition
+the key is stored on the device NVM.
+
+Next time the device is plugged in the user can verify (challenge) the
+device using the same key::
+
+  # echo $key > /sys/bus/thunderbolt/devices/0-3/key
+  # echo 2 > /sys/bus/thunderbolt/devices/0-3/authorized
+
+If the challenge the device returns back matches the one we expect based
+on the key, the device is connected and the PCIe tunnels are created.
+However, if the challenge failed no tunnels are created and error is
+returned to the user.
+
+If the user still wants to connect the device it can either approve
+the device without a key or write new key and write 1 to the
+``authorized`` file to get the new key stored on the device NVM.
+
+Upgrading NVM on Thunderbolt device or host
+-------------------------------------------
+Since most of the functionality is handled in a firmware running on a
+host controller or a device, it is important that the firmware can be
+upgraded to the latest where possible bugs in it have been fixed.
+Typically OEMs provide this firmware from their support site.
+
+There is also a central site which has links where to download firmwares
+for some machines:
+
+  `Thunderbolt Updates <https://thunderbolttechnology.net/updates>`_
+
+Before you upgrade firmware on a device or host, please make sure it is
+the suitable. Failing to do that may render the device (or host) in a
+state where it cannot be used properly anymore without special tools!
+
+Host NVM upgrade on Apple Macs is not supported.
+
+Once the NVM image has been downloaded, you need to plug in a
+Thunderbolt device so that the host controller appears. It does not
+matter which device is connected (unless you are upgrading NVM on a
+device - then you need to connect that particular device).
+
+Note OEM-specific method to power the controller up ("force power") may
+be available for your system in which case there is no need to plug in a
+Thunderbolt device.
+
+After that we can write the firmware to the non-active parts of the NVM
+of the host or device. As an example here is how Intel NUC6i7KYK (Skull
+Canyon) Thunderbolt controller NVM is upgraded::
+
+  # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem
+
+Once the operation completes we can trigger NVM authentication and
+upgrade process as follows::
+
+  # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate
+
+If no errors are returned, the host controller shortly disappears. Once
+it comes back the driver notices it and initiates a full power cycle.
+After a while the host controller appears again and this time it should
+be fully functional.
+
+We can verify that the new NVM firmware is active by running following
+commands::
+
+  # cat /sys/bus/thunderbolt/devices/0-0/nvm_authenticate
+  0x0
+  # cat /sys/bus/thunderbolt/devices/0-0/nvm_version
+  18.0
+
+If ``nvm_authenticate`` contains anything else than 0x0 it is the error
+code from the last authentication cycle, which means the authentication
+of the NVM image failed.
+
+Note names of the NVMem devices ``nvm_activeN`` and ``nvm_non_activeN``
+depends on the order they are registered in the NVMem subsystem. N in
+the name is the identifier added by the NVMem subsystem.
+
+Upgrading NVM when host controller is in safe mode
+--------------------------------------------------
+If the existing NVM is not properly authenticated (or is missing) the
+host controller goes into safe mode which means that only available
+functionality is flashing new NVM image. When in this mode the reading
+``nvm_version`` fails with ``ENODATA`` and the device identification
+information is missing.
+
+To recover from this mode, one needs to flash a valid NVM image to the
+host host controller in the same way it is done in the previous chapter.
-- 
2.11.0

[PATCH 24/24] MAINTAINERS: Add maintainers for Thunderbolt driver

[Mika Westerberg]

We will be helping Andreas to maintain the Thunderbolt driver.

Signed-off-by: Michael Jamet <michael.jamet@intel.com>
Signed-off-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
---
 MAINTAINERS | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index f7d568b8f133..4aa5616a5a78 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11325,6 +11325,9 @@ F:	Documentation/tee.txt
 
 THUNDERBOLT DRIVER
 M:	Andreas Noever <andreas.noever@gmail.com>
+M:	Michael Jamet <michael.jamet@intel.com>
+M:	Mika Westerberg <mika.westerberg@linux.intel.com>
+M:	Yehezkel Bernat <yehezkel.bernat@intel.com>
 S:	Maintained
 F:	drivers/thunderbolt/
 
-- 
2.11.0

Re: [PATCH 05/24] thunderbolt: Rework capability handling

[Andy Shevchenko]

On Thu, May 18, 2017 at 5:38 PM, Mika Westerberg
<mika.westerberg@linux.intel.com> wrote:
> Organization of the capabilities in switches and ports is not so random
> after all. Rework the capability handling functionality so that it
> follows how capabilities are organized and provide two new functions
> (tb_switch_find_vsec_cap() and tb_port_find_cap()) which can be used to
> extract capabilities for ports and switches. Then convert the current
> users over these.

One nit here.

> +int tb_port_find_cap(struct tb_port *port, enum tb_port_cap cap)
>  {

> +       u32 offset;
>

> +       /*
> +        * DP out adapters claim to implement TMU capability but in
> +        * reality they do not so we hard code the adapter specific
> +        * capability offset here.
> +        */
> +       if (port->config.type == TB_TYPE_DP_HDMI_OUT)
> +               offset = 0x39;
>         else

> +               offset = 0x1;
> +
> +       do {
> +               struct tb_cap_any header;
> +               int ret;
> +
> +               ret = tb_port_read(port, &header, TB_CFG_PORT, offset, 1);
> +               if (ret)
> +                       return ret;
> +
> +               if (header.basic.cap == cap)
> +                       return offset;
> +
> +               offset = header.basic.next;
> +       } while (offset);
> +
> +       return -ENOENT;
>  }

> +static int tb_switch_find_cap(struct tb_switch *sw, enum tb_switch_cap cap)
>  {

> +       int offset = sw->config.first_cap_offset;
> +
> +       while (offset > 0 && offset < CAP_OFFSET_MAX) {
> +               struct tb_cap_any header;
> +               int ret;
> +
> +               ret = tb_sw_read(sw, &header, TB_CFG_SWITCH, offset, 1);
> +               if (ret)
> +                       return ret;
> +
> +               if (header.basic.cap == cap)
> +                       return offset;
> +
> +               offset = header.basic.next;
>         }

> +
> +       return -ENOENT;
>  }

Both has quite similar bodies.
Wouldn't be nice to split out a helper which takes initial offset and
type as parameters?


-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH 06/24] thunderbolt: Introduce thunderbolt bus and connection manager

[Andy Shevchenko]

On Thu, May 18, 2017 at 5:38 PM, Mika Westerberg
<mika.westerberg@linux.intel.com> wrote:
> Thunderbolt fabric consists of one or more switches. This fabric is
> called domain and it is controlled by an entity called connection
> manager. The connection manager can be either internal (driven by a
> firmware running on the host controller) or external (software driver).
> This driver currently implements support for the latter.
>
> In order to manage switches and their properties more easily we model
> this domain structure as a Linux bus. Each host controller adds a domain
> device to this bus, and these devices are named as domainN where N
> stands for index or id of the current domain.
>
> We then abstract connection manager specific operations into a new
> structure tb_cm_ops and convert the existing tb.c to fill those
> accordingly. This makes it easier to add support for the internal
> connection manager in subsequent patches.

One nit below.

> +static void tb_domain_release(struct device *dev)
> +{
> +       struct tb *tb = container_of(dev, struct tb, dev);
> +
> +       if (tb->ctl)
> +               tb_ctl_free(tb->ctl);

The usual pattern is to put such checks inside *_free() type of functions.

> +
> +       destroy_workqueue(tb->wq);
> +       ida_simple_remove(&tb_domain_ida, tb->index);
> +       kfree(tb);
> +}

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH 07/24] thunderbolt: Convert switch to a device

[Andy Shevchenko]

On Thu, May 18, 2017 at 5:38 PM, Mika Westerberg
<mika.westerberg@linux.intel.com> wrote:
> Thunderbolt domain consists of switches that are connected to each
> other, forming a bus. This will convert each switch into a real Linux
> device structure and adds them to the domain. The advantage here is
> that we get all the goodies from the driver core, like reference
> counting and sysfs hierarchy for free.
>
> Also expose device identification information to the userspace via new
> sysfs attributes.
>
> In order to support internal connection manager (ICM) we separate switch
> configuration into its own function (tb_switch_configure()) which is
> only called by the existing native connection manager implementation
> used on Macs.
>

Couple of nits below.

> +
> +       return sw;
> +err:

Perhaps
err_free_sw_ports:
?

> +       kfree(sw->ports);
> +       kfree(sw);
> +       return NULL;
> +}

> +       /* upload configuration */
> +       ret = tb_sw_write(sw, 1 + (u32 *) &sw->config, TB_CFG_SWITCH, 1, 3);

Extra space before &.

> +       if (ret)
> +               return ret;

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH 10/24] thunderbolt: Read vendor and device name from DROM

[Andy Shevchenko]

On Thu, May 18, 2017 at 5:39 PM, Mika Westerberg
<mika.westerberg@linux.intel.com> wrote:
> The device DROM contains name of the vendor and device among other
> things. Extract this information and expose it to the userspace via two
> new attributes.

One nit below.

> +       switch (header->type) {
> +       case TB_DROM_ENTRY_PORT:
> +               break;
> +       case TB_DROM_ENTRY_GENERIC:

> +               tb_drom_parse_generic_entry(sw,
> +                       (struct tb_drom_entry_generic *)header);

Can it be one line?
Is fall through intended?

> +       default:
>                 return 0;
> +       }

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH 22/24] thunderbolt: Add support for host and device NVM firmware upgrade

[Andy Shevchenko]

On Thu, May 18, 2017 at 5:39 PM, Mika Westerberg
<mika.westerberg@linux.intel.com> wrote:
> Starting from Intel Falcon Ridge the NVM firmware can be upgraded by
> using DMA configuration based mailbox commands. If we detect that the
> host or device (device support starts from Intel Alpine Ridge) has the
> DMA configuration based mailbox we expose NVM information to the
> userspace as two separate Linux NVMem devices: nvm_active and
> nvm_non_active. The former is read-only portion of the active NVM which
> firmware upgrade tools can be use to find out suitable NVM image if the
> device identification strings are not enough.
>
> The latter is write-only portion where the new NVM image is to be
> written by the userspace. It is up to the userspace to find out right
> NVM image (the kernel does very minimal validation). The ICM firmware
> itself authenticates the new NVM firmware and fails the operation if it
> is not what is expected.
>
> We also expose two new sysfs files per each switch: nvm_version and
> nvm_authenticate which can be used to read the active NVM version and
> start the upgrade process.
>
> We also introduce safe mode which is the mode a switch goes when it does
> not have properly authenticated firmware. In this mode the switch only
> accepts a couple of commands including flashing a new NVM firmware image
> and triggering power cycle.
>
> This code is based on the work done by Amir Levy and Michael Jamet.

Couple of nitpicks below.

> +static ssize_t nvm_authenticate_store(struct device *dev,
> +       struct device_attribute *attr, const char *buf, size_t count)
> +{
> +       struct tb_switch *sw = tb_to_switch(dev);
> +       unsigned int val;
> +       int ret;
> +
> +       if (mutex_lock_interruptible(&switch_lock))
> +               return -ERESTARTSYS;
> +
> +       ret = kstrtouint(buf, 0, &val);
> +       if (ret)
> +               goto unlock;

Looking below it would be
ret = kstrtobool(..., &x);
if (ret)
 goto ...;

if (x) {
} else {
}

exit_unlock:
...

> +
> +       switch (val) {
> +       case 0:
> +               /* Just clear the authentication status */
> +               nvm_clear_auth_status(sw);
> +               break;
> +
> +       case 1:
> +               ret = nvm_validate_and_write(sw);
> +               if (ret)
> +                       goto unlock;
> +
> +               sw->nvm->authenticating = true;
> +
> +               if (!tb_route(sw))
> +                       ret = nvm_authenticate_host(sw);
> +               else
> +                       ret = nvm_authenticate_device(sw);
> +               break;
> +
> +       default:
> +               ret = -EINVAL;
> +               break;
> +       }
> +
> +unlock:
> +       mutex_unlock(&switch_lock);
> +
> +       if (ret)
> +               return ret;
> +       return count;
> +}

> +               nvm->major = val >> 16 & 0xff;
> +               nvm->minor = val >> 8 & 0xff;

If lvalue is u8 or alike the conjunction becomes redundant.

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH 05/24] thunderbolt: Rework capability handling

[Mika Westerberg]

On Thu, May 18, 2017 at 07:38:29PM +0300, Andy Shevchenko wrote:
> On Thu, May 18, 2017 at 5:38 PM, Mika Westerberg
> <mika.westerberg@linux.intel.com> wrote:
> > Organization of the capabilities in switches and ports is not so random
> > after all. Rework the capability handling functionality so that it
> > follows how capabilities are organized and provide two new functions
> > (tb_switch_find_vsec_cap() and tb_port_find_cap()) which can be used to
> > extract capabilities for ports and switches. Then convert the current
> > users over these.
> 
> One nit here.
> 
> > +int tb_port_find_cap(struct tb_port *port, enum tb_port_cap cap)
> >  {
> 
> > +       u32 offset;
> >
> 
> > +       /*
> > +        * DP out adapters claim to implement TMU capability but in
> > +        * reality they do not so we hard code the adapter specific
> > +        * capability offset here.
> > +        */
> > +       if (port->config.type == TB_TYPE_DP_HDMI_OUT)
> > +               offset = 0x39;
> >         else
> 
> > +               offset = 0x1;
> > +
> > +       do {
> > +               struct tb_cap_any header;
> > +               int ret;
> > +
> > +               ret = tb_port_read(port, &header, TB_CFG_PORT, offset, 1);
> > +               if (ret)
> > +                       return ret;
> > +
> > +               if (header.basic.cap == cap)
> > +                       return offset;
> > +
> > +               offset = header.basic.next;
> > +       } while (offset);
> > +
> > +       return -ENOENT;
> >  }
> 
> > +static int tb_switch_find_cap(struct tb_switch *sw, enum tb_switch_cap cap)
> >  {
> 
> > +       int offset = sw->config.first_cap_offset;
> > +
> > +       while (offset > 0 && offset < CAP_OFFSET_MAX) {
> > +               struct tb_cap_any header;
> > +               int ret;
> > +
> > +               ret = tb_sw_read(sw, &header, TB_CFG_SWITCH, offset, 1);
> > +               if (ret)
> > +                       return ret;
> > +
> > +               if (header.basic.cap == cap)
> > +                       return offset;
> > +
> > +               offset = header.basic.next;
> >         }
> 
> > +
> > +       return -ENOENT;
> >  }
> 
> Both has quite similar bodies.
> Wouldn't be nice to split out a helper which takes initial offset and
> type as parameters?

The whole point of this rework was to separate port vs. switch
capability to follow how the hardware is organized instead of having one
more complex function handling everything :)

Sure, I can merge them back together but IMHO it will be not that
readable anymore.

Re: [PATCH 06/24] thunderbolt: Introduce thunderbolt bus and connection manager

[Mika Westerberg]

On Thu, May 18, 2017 at 07:43:13PM +0300, Andy Shevchenko wrote:
> On Thu, May 18, 2017 at 5:38 PM, Mika Westerberg
> <mika.westerberg@linux.intel.com> wrote:
> > Thunderbolt fabric consists of one or more switches. This fabric is
> > called domain and it is controlled by an entity called connection
> > manager. The connection manager can be either internal (driven by a
> > firmware running on the host controller) or external (software driver).
> > This driver currently implements support for the latter.
> >
> > In order to manage switches and their properties more easily we model
> > this domain structure as a Linux bus. Each host controller adds a domain
> > device to this bus, and these devices are named as domainN where N
> > stands for index or id of the current domain.
> >
> > We then abstract connection manager specific operations into a new
> > structure tb_cm_ops and convert the existing tb.c to fill those
> > accordingly. This makes it easier to add support for the internal
> > connection manager in subsequent patches.
> 
> One nit below.
> 
> > +static void tb_domain_release(struct device *dev)
> > +{
> > +       struct tb *tb = container_of(dev, struct tb, dev);
> > +
> > +       if (tb->ctl)
> > +               tb_ctl_free(tb->ctl);
> 
> The usual pattern is to put such checks inside *_free() type of functions.

tb_ctl_free() and these patterns are from the original code. I just
moved the calls from tb.c intto here. No problem doing that change but I
suppose that should be done first in a separate patch before this patch.

> > +
> > +       destroy_workqueue(tb->wq);
> > +       ida_simple_remove(&tb_domain_ida, tb->index);
> > +       kfree(tb);
> > +}
> 
> -- 
> With Best Regards,
> Andy Shevchenko

Re: [PATCH 07/24] thunderbolt: Convert switch to a device

[Mika Westerberg]

On Thu, May 18, 2017 at 07:49:21PM +0300, Andy Shevchenko wrote:
> On Thu, May 18, 2017 at 5:38 PM, Mika Westerberg
> <mika.westerberg@linux.intel.com> wrote:
> > Thunderbolt domain consists of switches that are connected to each
> > other, forming a bus. This will convert each switch into a real Linux
> > device structure and adds them to the domain. The advantage here is
> > that we get all the goodies from the driver core, like reference
> > counting and sysfs hierarchy for free.
> >
> > Also expose device identification information to the userspace via new
> > sysfs attributes.
> >
> > In order to support internal connection manager (ICM) we separate switch
> > configuration into its own function (tb_switch_configure()) which is
> > only called by the existing native connection manager implementation
> > used on Macs.
> >
> 
> Couple of nits below.
> 
> > +
> > +       return sw;
> > +err:
> 
> Perhaps
> err_free_sw_ports:
> ?

Works for me :)

> > +       kfree(sw->ports);
> > +       kfree(sw);
> > +       return NULL;
> > +}
> 
> > +       /* upload configuration */
> > +       ret = tb_sw_write(sw, 1 + (u32 *) &sw->config, TB_CFG_SWITCH, 1, 3);
> 
> Extra space before &.

This comes from the original code but I'll change it.

> > +       if (ret)
> > +               return ret;
> 
> -- 
> With Best Regards,
> Andy Shevchenko

Re: [PATCH 10/24] thunderbolt: Read vendor and device name from DROM

[Mika Westerberg]

On Thu, May 18, 2017 at 10:19:11PM +0300, Andy Shevchenko wrote:
> On Thu, May 18, 2017 at 5:39 PM, Mika Westerberg
> <mika.westerberg@linux.intel.com> wrote:
> > The device DROM contains name of the vendor and device among other
> > things. Extract this information and expose it to the userspace via two
> > new attributes.
> 
> One nit below.
> 
> > +       switch (header->type) {
> > +       case TB_DROM_ENTRY_PORT:
> > +               break;
> > +       case TB_DROM_ENTRY_GENERIC:
> 
> > +               tb_drom_parse_generic_entry(sw,
> > +                       (struct tb_drom_entry_generic *)header);
> 
> Can it be one line?

It does not fit into 80 char limit.

> Is fall through intended?

Yes.

> > +       default:
> >                 return 0;
> > +       }
> 
> -- 
> With Best Regards,
> Andy Shevchenko

Re: [PATCH 22/24] thunderbolt: Add support for host and device NVM firmware upgrade

[Mika Westerberg]

On Thu, May 18, 2017 at 10:35:19PM +0300, Andy Shevchenko wrote:
> On Thu, May 18, 2017 at 5:39 PM, Mika Westerberg
> <mika.westerberg@linux.intel.com> wrote:
> > Starting from Intel Falcon Ridge the NVM firmware can be upgraded by
> > using DMA configuration based mailbox commands. If we detect that the
> > host or device (device support starts from Intel Alpine Ridge) has the
> > DMA configuration based mailbox we expose NVM information to the
> > userspace as two separate Linux NVMem devices: nvm_active and
> > nvm_non_active. The former is read-only portion of the active NVM which
> > firmware upgrade tools can be use to find out suitable NVM image if the
> > device identification strings are not enough.
> >
> > The latter is write-only portion where the new NVM image is to be
> > written by the userspace. It is up to the userspace to find out right
> > NVM image (the kernel does very minimal validation). The ICM firmware
> > itself authenticates the new NVM firmware and fails the operation if it
> > is not what is expected.
> >
> > We also expose two new sysfs files per each switch: nvm_version and
> > nvm_authenticate which can be used to read the active NVM version and
> > start the upgrade process.
> >
> > We also introduce safe mode which is the mode a switch goes when it does
> > not have properly authenticated firmware. In this mode the switch only
> > accepts a couple of commands including flashing a new NVM firmware image
> > and triggering power cycle.
> >
> > This code is based on the work done by Amir Levy and Michael Jamet.
> 
> Couple of nitpicks below.
> 
> > +static ssize_t nvm_authenticate_store(struct device *dev,
> > +       struct device_attribute *attr, const char *buf, size_t count)
> > +{
> > +       struct tb_switch *sw = tb_to_switch(dev);
> > +       unsigned int val;
> > +       int ret;
> > +
> > +       if (mutex_lock_interruptible(&switch_lock))
> > +               return -ERESTARTSYS;
> > +
> > +       ret = kstrtouint(buf, 0, &val);
> > +       if (ret)
> > +               goto unlock;
> 
> Looking below it would be
> ret = kstrtobool(..., &x);
> if (ret)
>  goto ...;
> 
> if (x) {
> } else {
> }
> 
> exit_unlock:
> ...

OK.

> > +
> > +       switch (val) {
> > +       case 0:
> > +               /* Just clear the authentication status */
> > +               nvm_clear_auth_status(sw);
> > +               break;
> > +
> > +       case 1:
> > +               ret = nvm_validate_and_write(sw);
> > +               if (ret)
> > +                       goto unlock;
> > +
> > +               sw->nvm->authenticating = true;
> > +
> > +               if (!tb_route(sw))
> > +                       ret = nvm_authenticate_host(sw);
> > +               else
> > +                       ret = nvm_authenticate_device(sw);
> > +               break;
> > +
> > +       default:
> > +               ret = -EINVAL;
> > +               break;
> > +       }
> > +
> > +unlock:
> > +       mutex_unlock(&switch_lock);
> > +
> > +       if (ret)
> > +               return ret;
> > +       return count;
> > +}
> 
> > +               nvm->major = val >> 16 & 0xff;
> > +               nvm->minor = val >> 8 & 0xff;
> 
> If lvalue is u8 or alike the conjunction becomes redundant.

It is indeed u8 so I can drop the & 0xff. Thanks.

Re: [PATCH 10/24] thunderbolt: Read vendor and device name from DROM

[Lukas Wunner]

Hi Mika,

nice work, by now I've picked up my jaw from the floor and can
offer a few comments...


On Thu, May 18, 2017 at 05:39:00PM +0300, Mika Westerberg wrote:
> The device DROM contains name of the vendor and device among other
> things.

What exactly are these other things?  Apple uses 0x30 to store a
serial number.  Is this attribute number assigned by Intel to Apple
or is it reserved for vendor use or did they arbitrarily choose it?

If there can be many attributes, should they be stored in a list
rather than adding a char* pointer for each one to struct tb_switch?
The latter doesn't scale.


> +static void tb_drom_parse_generic_entry(struct tb_switch *sw,
> +		struct tb_drom_entry_generic *entry)
> +{
> +	if (entry->header.index == 1)
> +		sw->vendor_name = kstrdup((char *)entry->data, GFP_KERNEL);
> +	else if (entry->header.index == 2)
> +		sw->device_name = kstrdup((char *)entry->data, GFP_KERNEL);
> +}

This assumes that these are properly null-terminated strings, but the DROM
may contain complete garbage.  The existing drom parser is very careful
to validate and sanitize everything.


>  static int tb_drom_parse_entry(struct tb_switch *sw,
>  		struct tb_drom_entry_header *header)
>  {
> @@ -311,8 +325,15 @@ static int tb_drom_parse_entry(struct tb_switch *sw,
>  	int res;
>  	enum tb_port_type type;
>  
> -	if (header->type != TB_DROM_ENTRY_PORT)
> +	switch (header->type) {
> +	case TB_DROM_ENTRY_PORT:
> +		break;
> +	case TB_DROM_ENTRY_GENERIC:
> +		tb_drom_parse_generic_entry(sw,
> +			(struct tb_drom_entry_generic *)header);
> +	default:
>  		return 0;
> +	}
>  
>  	port = &sw->ports[header->index];
>  	port->disabled = header->port_disabled;

I'm afraid this control flow is not very pretty, the stuff below the
switch/case statement is essentially the parser for TB_DROM_ENTRY_PORT
whereas the parser for TB_DROM_ENTRY_GENERIC is in a separate function.
It would be easier to follow the control flow if the parser for
TB_DROM_ENTRY_PORT was in a separate function tb_drom_parse_port_entry().

In fact I wrote patches to do just that one and a half years ago but
haven't upstreamed them so far, mostly because I was unsure how many
attributes there can be, if they should be stored in a list, etc.
I didn't have access to the same resources as you do.

https://github.com/l1k/linux/commit/b6c9db73258b
https://github.com/l1k/linux/commit/d1b46362b528

Feel free to include them in full or in part in your series
or modify as you see fit.

The latter patch also includes a sanitizer for generic entries.

Thanks,

Lukas

Re: [PATCH 10/24] thunderbolt: Read vendor and device name from DROM

[Mika Westerberg]

On Fri, May 19, 2017 at 12:07:10PM +0200, Lukas Wunner wrote:
> Hi Mika,
> 
> nice work, by now I've picked up my jaw from the floor and can
> offer a few comments...

Thanks! :)

> On Thu, May 18, 2017 at 05:39:00PM +0300, Mika Westerberg wrote:
> > The device DROM contains name of the vendor and device among other
> > things.
> 
> What exactly are these other things?  Apple uses 0x30 to store a
> serial number.  Is this attribute number assigned by Intel to Apple
> or is it reserved for vendor use or did they arbitrarily choose it?

It is part of the DROM specification. The 0x30 - 0x3e are vendor
specific entries.

There are couple of other things but I don't think they are useful to
us to be honest.

> If there can be many attributes, should they be stored in a list
> rather than adding a char* pointer for each one to struct tb_switch?
> The latter doesn't scale.

I don't think we need other attributes (well, at least right now). The
device/vendor name is useful because that's what we expose to the
userspace for device identification along with the device/vendor ID.

> > +static void tb_drom_parse_generic_entry(struct tb_switch *sw,
> > +		struct tb_drom_entry_generic *entry)
> > +{
> > +	if (entry->header.index == 1)
> > +		sw->vendor_name = kstrdup((char *)entry->data, GFP_KERNEL);
> > +	else if (entry->header.index == 2)
> > +		sw->device_name = kstrdup((char *)entry->data, GFP_KERNEL);
> > +}
> 
> This assumes that these are properly null-terminated strings, but the DROM
> may contain complete garbage.  The existing drom parser is very careful
> to validate and sanitize everything.

The DROM specification says they must be null-terminated but I yes, it
is possible that some of the devices have it wrong. The generic entry
includes length field so I suppose we can use that + kmemdup() instead
here?

> >  static int tb_drom_parse_entry(struct tb_switch *sw,
> >  		struct tb_drom_entry_header *header)
> >  {
> > @@ -311,8 +325,15 @@ static int tb_drom_parse_entry(struct tb_switch *sw,
> >  	int res;
> >  	enum tb_port_type type;
> >  
> > -	if (header->type != TB_DROM_ENTRY_PORT)
> > +	switch (header->type) {
> > +	case TB_DROM_ENTRY_PORT:
> > +		break;
> > +	case TB_DROM_ENTRY_GENERIC:
> > +		tb_drom_parse_generic_entry(sw,
> > +			(struct tb_drom_entry_generic *)header);
> > +	default:
> >  		return 0;
> > +	}
> >  
> >  	port = &sw->ports[header->index];
> >  	port->disabled = header->port_disabled;
> 
> I'm afraid this control flow is not very pretty, the stuff below the
> switch/case statement is essentially the parser for TB_DROM_ENTRY_PORT
> whereas the parser for TB_DROM_ENTRY_GENERIC is in a separate function.
> It would be easier to follow the control flow if the parser for
> TB_DROM_ENTRY_PORT was in a separate function tb_drom_parse_port_entry().
> 
> In fact I wrote patches to do just that one and a half years ago but
> haven't upstreamed them so far, mostly because I was unsure how many
> attributes there can be, if they should be stored in a list, etc.
> I didn't have access to the same resources as you do.
> 
> https://github.com/l1k/linux/commit/b6c9db73258b
> https://github.com/l1k/linux/commit/d1b46362b528

Cool.

> Feel free to include them in full or in part in your series
> or modify as you see fit.
> 
> The latter patch also includes a sanitizer for generic entries.

OK, I'll take a look at them and see if we can use them here :)

Re: [PATCH 06/24] thunderbolt: Introduce thunderbolt bus and connection manager

[Andy Shevchenko]

On Fri, May 19, 2017 at 11:15 AM, Mika Westerberg
<mika.westerberg@linux.intel.com> wrote:
> On Thu, May 18, 2017 at 07:43:13PM +0300, Andy Shevchenko wrote:
>> On Thu, May 18, 2017 at 5:38 PM, Mika Westerberg
>> <mika.westerberg@linux.intel.com> wrote:

>> One nit below.
>>
>> > +static void tb_domain_release(struct device *dev)
>> > +{
>> > +       struct tb *tb = container_of(dev, struct tb, dev);
>> > +
>> > +       if (tb->ctl)
>> > +               tb_ctl_free(tb->ctl);
>>
>> The usual pattern is to put such checks inside *_free() type of functions.
>
> tb_ctl_free() and these patterns are from the original code. I just
> moved the calls from tb.c intto here. No problem doing that change but I
> suppose that should be done first in a separate patch before this patch.

Precisely!

>> > +
>> > +       destroy_workqueue(tb->wq);
>> > +       ida_simple_remove(&tb_domain_ida, tb->index);
>> > +       kfree(tb);
>> > +}

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH 05/24] thunderbolt: Rework capability handling

[Andy Shevchenko]

On Fri, May 19, 2017 at 11:12 AM, Mika Westerberg
<mika.westerberg@linux.intel.com> wrote:
> On Thu, May 18, 2017 at 07:38:29PM +0300, Andy Shevchenko wrote:
>> On Thu, May 18, 2017 at 5:38 PM, Mika Westerberg
>> <mika.westerberg@linux.intel.com> wrote:

>> One nit here.

>> Both has quite similar bodies.
>> Wouldn't be nice to split out a helper which takes initial offset and
>> type as parameters?
>
> The whole point of this rework was to separate port vs. switch
> capability to follow how the hardware is organized instead of having one
> more complex function handling everything :)
>
> Sure, I can merge them back together but IMHO it will be not that
> readable anymore.

Up to you. I'm fine with either.

-- 
With Best Regards,
Andy Shevchenko

RE: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Mario.Limonciello]

> -----Original Message-----
> From: Mika Westerberg [mailto:mika.westerberg@linux.intel.com]
> Sent: Thursday, May 18, 2017 9:39 AM
> To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Cc: Andreas Noever <andreas.noever@gmail.com>; Michael Jamet
> <michael.jamet@intel.com>; Yehezkel Bernat <yehezkel.bernat@intel.com>; Lukas
> Wunner <lukas@wunner.de>; Amir Levy <amir.jer.levy@intel.com>; Andy
> Lutomirski <luto@kernel.org>; Limonciello, Mario <Mario_Limonciello@Dell.com>;
> Dominguez, Jared <Jared_Dominguez@DELL.com>; Andy Shevchenko
> <andriy.shevchenko@linux.intel.com>; Mika Westerberg
> <mika.westerberg@linux.intel.com>; linux-kernel@vger.kernel.org
> Subject: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade
> 
> Hi all,
> 
> This patch series adds support for Thunderbolt security levels, which were
> first introduced in Intel Falcon Ridge Thunderbolt controller, to prevent
> DMA attacks when PCIe is tunneled over Thunderbolt fabric. This is needed
> if there is no IOMMU available for various reasons.
> 
> Most PCs out there having Falcon Ridge or newer have security level set to
> "user" which means that user authorization is needed before PCIe tunnel is
> creaded (the PCIe device appears). This effectively means that without
> driver support the user needs to configure security level from BIOS to
> "none" to get Thunderbolt devices connected. With these patches the user
> can authorize devices using sysfs attributes like:
> 
>   # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized
> 
> In addition these patches add support for upgrading NVM firmware running on
> a host or device by running something like:
> 
>   # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-
> 0/nvm_non_active0/nvmem
>   # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate
> 
> This is documented with more details in patch [23/24].
> 
> This series is based on Amir's networking patches [1] but instead of
> splitting the functionality between kernel driver and userspace daemon, we
> take advantage of Linux driver core by converting the existing driver to
> expose a Linux bus (domain) and devices (switches). Notifications to the
> userspace about plugged/unplugged devices is handled by standard uevents
> when a device is added to/removed from the Thunderbolt bus.
> 
> Since thunderbolt device identification and authorization can be done
> directly through sysfs attributes there is no need for userspace daemon.
> However, there still should be an application that promps user for unknown
> devices and allows selecting between "single connect" and "connect always"
> keeping this information in a database or similar persistent storage. This
> patch series only provides mechanism for userspace applications to achieve
> that.
> 
> Where Internal Connection Manager (ICM) firmware is available and usable,
> we use it in the driver. This also includes newer Apple Macbooks with
> Alpine Ridge. For older Macbooks the driver works as before but in addition
> the Thunderbolt bus is available there as well (including possibility to
> upgrade NVM firmware of connected devices).
> 
> We are also in works of porting Amir's networking driver to work on top of
> the new Thunderbolt bus pretty much the same way firewire networking is
> currently done. In addition this makes is possible to introduce other
> protocols like a char device that allows userspace directly to communicate
> accross Thunderbolt domains.
> 
> Note for Macs the Linux native PCIe hotplug support does not work well with
> the Thunderbolt PCIe topologies where there is need to put all available
> resources to the PCIe downstream port where the PCIe chain is extended.
> This is something we need to fix. In the mean time is a way to work it
> around by passing "pci=hpbussize=10,hpmemsize=2M" or so to the kernel
> command line.
> 
> These patches use uuid_be from uuid.h but I've learned that there is a work
> to remove the type completely in favor of new uuid_t [2]. I'm not sure what
> to do regarding that because those patches are not yet in the mainline.
> 
> [1] https://lkml.org/lkml/2016/11/9/341
> [2] http://git.infradead.org/users/hch/vfs.git/shortlog/refs/heads/uuid-types
> 
> Mika Westerberg (24):
>   thunderbolt: Use const buffer pointer in write operations
>   thunderbolt: Do not try to read UID if DROM offset is read as 0
>   thunderbolt: Do not warn about newer DROM versions
>   thunderbolt: Add MSI-X support
>   thunderbolt: Rework capability handling
>   thunderbolt: Introduce thunderbolt bus and connection manager
>   thunderbolt: Convert switch to a device
>   thunderbolt: Fail switch adding operation if reading DROM fails
>   thunderbolt: Do not fail if DROM data CRC32 is invalid
>   thunderbolt: Read vendor and device name from DROM
>   thunderbolt: Move control channel messages to tb_msgs.h
>   thunderbolt: Expose get_route() to other files
>   thunderbolt: Expose make_header() to other files
>   thunderbolt: Let the connection manager handle all notifications
>   thunderbolt: Rework control channel to be more reliable
>   thunderbolt: Add Thunderbolt 3 PCI IDs
>   thunderbolt: Add support for NHI mailbox
>   thunderbolt: Store Thunderbolt generation in the switch structure
>   thunderbolt: Add support for DMA configuration based mailbox
>   thunderbolt: Do not touch the hardware if the NHI is gone on resume
>   thunderbolt: Add support for Internal Connection Manager (ICM)
>   thunderbolt: Add support for host and device NVM firmware upgrade
>   thunderbolt: Add documentation how Thunderbolt bus can be used
>   MAINTAINERS: Add maintainers for Thunderbolt driver
> 
>  Documentation/ABI/testing/sysfs-bus-thunderbolt |  108 +++
>  Documentation/admin-guide/index.rst             |    1 +
>  Documentation/admin-guide/thunderbolt.rst       |  197 ++++
>  MAINTAINERS                                     |    3 +
>  drivers/thunderbolt/Kconfig                     |   13 +-
>  drivers/thunderbolt/Makefile                    |    2 +-
>  drivers/thunderbolt/cap.c                       |  169 ++--
>  drivers/thunderbolt/ctl.c                       |  655 +++++++++----
>  drivers/thunderbolt/ctl.h                       |  105 ++-
>  drivers/thunderbolt/dma_port.c                  |  524 +++++++++++
>  drivers/thunderbolt/dma_port.h                  |   34 +
>  drivers/thunderbolt/domain.c                    |  455 ++++++++++
>  drivers/thunderbolt/eeprom.c                    |   84 +-
>  drivers/thunderbolt/icm.c                       | 1098 ++++++++++++++++++++++
>  drivers/thunderbolt/nhi.c                       |  302 +++++-
>  drivers/thunderbolt/nhi.h                       |   91 +-
>  drivers/thunderbolt/nhi_regs.h                  |   27 +
>  drivers/thunderbolt/switch.c                    | 1109 +++++++++++++++++++++--
>  drivers/thunderbolt/tb.c                        |  237 ++---
>  drivers/thunderbolt/tb.h                        |  242 ++++-
>  drivers/thunderbolt/tb_msgs.h                   |  260 ++++++
>  drivers/thunderbolt/tb_regs.h                   |   31 +-
>  drivers/thunderbolt/tunnel_pci.c                |   17 +-
>  23 files changed, 5213 insertions(+), 551 deletions(-)
>  create mode 100644 Documentation/ABI/testing/sysfs-bus-thunderbolt
>  create mode 100644 Documentation/admin-guide/thunderbolt.rst
>  create mode 100644 drivers/thunderbolt/dma_port.c
>  create mode 100644 drivers/thunderbolt/dma_port.h
>  create mode 100644 drivers/thunderbolt/domain.c
>  create mode 100644 drivers/thunderbolt/icm.c
>  create mode 100644 drivers/thunderbolt/tb_msgs.h
> 
> --
> 2.11.0

Mika,

Thanks for submitting this series.  
I've tested security level stuff a little bit, 
but I'm running into what I think is some odd behavior.

Here's my setup:
System: I'm using is an XPS 9350 (Has Alpine Ridge).  It's got NVM 16.0.  BIOS 1.4.13
TBT Device: Dell TB16 (which has AR in the cable and in dock - both NVM 16.0).

I created a udev rule that will automatically authorize the dock and cable.
#dell cable
ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{vendor}=="0xd4", ATTR{device}=="0xb051", ATTR{authorized}="1"
#dell dock
ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{vendor}=="0xd4", ATTR{device}=="0xb054", ATTR{authorized}="1"

If I boot the system with the dock connected the cable shows up and authorizes but the dock doesn't.
---
[    6.916669] thunderbolt 0000:03:00.0: current switch config:
[    6.916671] thunderbolt 0000:03:00.0:  Switch: 8086:1576 (Revision: 4, TB Version: 2)
[    6.916673] thunderbolt 0000:03:00.0:   Max Port Number: 11
[    6.916673] thunderbolt 0000:03:00.0:   Config:
[    6.916675] thunderbolt 0000:03:00.0:    Upstream Port Number: 5 Depth: 0 Route String: 0x0 Enabled: 1, PlugEventsDelay: 254ms
[    6.916676] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
[    6.927293] thunderbolt 0000:03:00.0: 0: uid: 0x8086a3422131ea10
[    6.927632] thunderbolt 0000:03:00.0:  Port 0: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[    6.927633] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
[    6.927634] thunderbolt 0000:03:00.0:   Max counters: 8
[    6.927635] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[    6.927760] thunderbolt 0000:03:00.0:  Port 1: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[    6.927761] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[    6.927762] thunderbolt 0000:03:00.0:   Max counters: 16
[    6.927763] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
[    6.927890] thunderbolt 0000:03:00.0:  Port 2: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[    6.927895] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[    6.927896] thunderbolt 0000:03:00.0:   Max counters: 16
[    6.927897] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
[    6.928091] thunderbolt 0000:03:00.0:  Port 3: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[    6.928092] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[    6.928092] thunderbolt 0000:03:00.0:   Max counters: 16
[    6.928093] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
[    6.928217] thunderbolt 0000:03:00.0:  Port 4: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[    6.928219] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[    6.928219] thunderbolt 0000:03:00.0:   Max counters: 16
[    6.928220] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
[    6.928222] thunderbolt 0000:03:00.0: 0:5: disabled by eeprom
[    6.928261] thunderbolt 0000:03:00.0:  Port 6: 8086:1576 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
[    6.928262] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[    6.928262] thunderbolt 0000:03:00.0:   Max counters: 2
[    6.928263] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[    6.928304] thunderbolt 0000:03:00.0:  Port 7: 8086:1576 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
[    6.928305] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[    6.928306] thunderbolt 0000:03:00.0:   Max counters: 2
[    6.928306] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[    6.930860] thunderbolt 0000:03:00.0:  Port 8: 8086:1576 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0102))
[    6.930862] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
[    6.930863] thunderbolt 0000:03:00.0:   Max counters: 2
[    6.930864] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[    6.930865] thunderbolt 0000:03:00.0: 0:9: disabled by eeprom
[    6.930908] thunderbolt 0000:03:00.0:  Port 10: 8086:1576 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0101))
[    6.930909] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
[    6.930910] thunderbolt 0000:03:00.0:   Max counters: 2
[    6.930911] thunderbolt 0000:03:00.0:   NFC Credits: 0x1000000
[    6.930912] thunderbolt 0000:03:00.0: 0:b: disabled by eeprom
[    6.932249] thunderbolt 0000:03:00.0: current switch config:
[    6.932252] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB Version: 2)
[    6.932253] thunderbolt 0000:03:00.0:   Max Port Number: 11
[    6.932254] thunderbolt 0000:03:00.0:   Config:
[    6.932255] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 1 Route String: 0x1 Enabled: 1, PlugEventsDelay: 254ms
[    6.932256] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
[    6.940642] thunderbolt 0000:03:00.0: 1: reading drom (length: 0x6e)
[    7.040048] thunderbolt 0000:03:00.0: 1: drom data crc32 mismatch (expected: 0xaf438340, got: 0xaf4383c0), continuing
[    7.040247] thunderbolt 0000:03:00.0: 1: uid: 0xd40f7a7928c300
[    7.040296] thunderbolt 0000:03:00.0:  Port 0: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[    7.040297] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
[    7.040298] thunderbolt 0000:03:00.0:   Max counters: 8
[    7.040299] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[    7.040574] thunderbolt 0000:03:00.0:  Port 1: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[    7.040575] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[    7.040576] thunderbolt 0000:03:00.0:   Max counters: 16
[    7.040577] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
[    7.040711] thunderbolt 0000:03:00.0:  Port 2: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[    7.040712] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[    7.040713] thunderbolt 0000:03:00.0:   Max counters: 16
[    7.040714] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
[    7.040848] thunderbolt 0000:03:00.0:  Port 3: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[    7.040849] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[    7.040850] thunderbolt 0000:03:00.0:   Max counters: 16
[    7.040851] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
[    7.040987] thunderbolt 0000:03:00.0:  Port 4: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[    7.040988] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[    7.040990] thunderbolt 0000:03:00.0:   Max counters: 16
[    7.040991] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
[    7.040992] thunderbolt 0000:03:00.0: 1:5: disabled by eeprom
[    7.041033] thunderbolt 0000:03:00.0:  Port 6: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100102))
[    7.041034] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[    7.041035] thunderbolt 0000:03:00.0:   Max counters: 2
[    7.041036] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[    7.041078] thunderbolt 0000:03:00.0:  Port 7: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
[    7.041080] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[    7.041080] thunderbolt 0000:03:00.0:   Max counters: 2
[    7.041081] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[    7.041083] thunderbolt 0000:03:00.0: 1:8: disabled by eeprom
[    7.041084] thunderbolt 0000:03:00.0: 1:9: disabled by eeprom
[    7.041085] thunderbolt 0000:03:00.0: 1:a: disabled by eeprom
[    7.041085] thunderbolt 0000:03:00.0: 1:b: disabled by eeprom
[    7.041827] thunderbolt 0000:03:00.0: current switch config:
[    7.041829] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB Version: 2)
[    7.041829] thunderbolt 0000:03:00.0:   Max Port Number: 11
[    7.041830] thunderbolt 0000:03:00.0:   Config:
[    7.041831] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 2 Route String: 0x301 Enabled: 1, PlugEventsDelay: 254ms
[    7.041832] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
[    7.416678] pci 0000:04:00.0: [8086:1578] type 01 class 0x060400
[    7.416925] pci 0000:04:00.0: supports D1 D2
[    7.416926] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    7.428433] pci_bus 0000:05: [bus 05] partially hidden behind bridge 0000:04 [bus 04]
[    7.428500] pci_bus 0000:39: busn_res: can not insert [bus 39] under [bus 02-05] (conflicts with (null) [bus 02-05])
[    7.428503] pcieport 0000:02:02.0: PCI bridge to [bus 39]
[    7.428512] pcieport 0000:02:02.0:   bridge window [mem 0xd9f00000-0xd9ffffff]
[    7.428519] pci_bus 0000:39: [bus 39] partially hidden behind bridge 0000:02 [bus 02-05]
[    7.428542] pci_bus 0000:02: Allocating resources
[    7.428544] pcieport 0000:02:02.0: can't claim BAR 14 [mem 0xd9f00000-0xd9ffffff]: no compatible bridge window
[    7.451018] Bluetooth: RFCOMM TTY layer initialized
[    7.451022] Bluetooth: RFCOMM socket layer initialized
[    7.451026] Bluetooth: RFCOMM ver 1.11
[   11.958457] wlp58s0: authenticate with 50:6a:03:a7:1c:27
[   11.969167] wlp58s0: send auth to 50:6a:03:a7:1c:27 (try 1/3)
[   11.975768] wlp58s0: authenticated
[   11.976386] wlp58s0: associate with 50:6a:03:a7:1c:27 (try 1/3)
[   11.984817] wlp58s0: RX AssocResp from 50:6a:03:a7:1c:27 (capab=0x1411 status=0 aid=2)
[   12.001864] wlp58s0: associated
[   12.001913] IPv6: ADDRCONF(NETDEV_CHANGE): wlp58s0: link becomes ready
[   27.616672] thunderbolt 0000:03:00.0: timeout reading config space 2 from 0x5
[   27.616674] thunderbolt 0000:03:00.0: 301: cannot find TB_VSEC_CAP_PLUG_EVENTS aborting

If I then unplug the cable and plug it back in the dock does show up and authorize properly, but is a traceback
along the way.
---
[ 1653.226296] pcieport 0000:02:02.0: Refused to change power state, currently in D3
[ 1653.729116] thunderbolt 0000:03:00.0: stopping RX ring 0
[ 1653.729130] thunderbolt 0000:03:00.0: disabling interrupt at register 0x38200 bit 12 (0xffffffff -> 0xffffefff)
[ 1653.729159] thunderbolt 0000:03:00.0: stopping TX ring 0
[ 1653.729168] thunderbolt 0000:03:00.0: disabling interrupt at register 0x38200 bit 0 (0xffffffff -> 0xfffffffe)
[ 1653.729195] thunderbolt 0000:03:00.0: control channel stopped
[ 1653.729362] thunderbolt 0000:03:00.0: freeing RX ring 0
[ 1653.729381] thunderbolt 0000:03:00.0: freeing TX ring 0
[ 1653.729407] thunderbolt 0000:03:00.0: shutdown
[ 1653.753091] pcieport 0000:02:00.0: Refused to change power state, currently in D3
[ 1653.756383] pci_bus 0000:03: busn_res: [bus 03] is released
[ 1653.756676] pci_bus 0000:04: busn_res: [bus 04] is released
[ 1653.757479] pci_bus 0000:02: busn_res: [bus 02-05] is released
[ 1660.846964] ACPI Error: [SPRT] Namespace lookup failure, AE_ALREADY_EXISTS (20170303/dswload2-330)
[ 1660.846979] ACPI Exception: AE_ALREADY_EXISTS, During name lookup/catalog (20170303/psobject-241)
[ 1660.846985] ACPI Error: Method parse/execution failed [\_GPE._E42] (Node ffff8ba73016b488), AE_ALREADY_EXISTS (20170303/psparse-543)
[ 1660.846996] ACPI Error: Method parse/execution failed [\_GPE._E42] (Node ffff8ba73016b488), AE_ALREADY_EXISTS (20170303/psparse-543)
[ 1660.847009] ACPI Exception: AE_ALREADY_EXISTS, while evaluating GPE method [_E42] (20170303/evgpe-646)
[ 1660.893399] pci 0000:01:00.0: [8086:1576] type 01 class 0x060400
[ 1660.893529] pci 0000:01:00.0: supports D1 D2
[ 1660.893530] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1660.893625] pci 0000:01:00.0: System wakeup disabled by ACPI
[ 1660.893776] pci 0000:02:00.0: [8086:1576] type 01 class 0x060400
[ 1660.894047] pci 0000:02:00.0: supports D1 D2
[ 1660.894049] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1660.894178] pci 0000:02:01.0: [8086:1576] type 01 class 0x060400
[ 1660.894320] pci 0000:02:01.0: supports D1 D2
[ 1660.894321] pci 0000:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1660.894409] pci 0000:02:02.0: [8086:1576] type 01 class 0x060400
[ 1660.894542] pci 0000:02:02.0: supports D1 D2
[ 1660.894543] pci 0000:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1660.894637] pci 0000:01:00.0: PCI bridge to [bus 02-39]
[ 1660.894646] pci 0000:01:00.0:   bridge window [mem 0xc4000000-0xda0fffff]
[ 1660.894652] pci 0000:01:00.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[ 1660.894716] pci 0000:03:00.0: [8086:1575] type 00 class 0x088000
[ 1660.894744] pci 0000:03:00.0: reg 0x10: [mem 0xda000000-0xda03ffff]
[ 1660.894756] pci 0000:03:00.0: reg 0x14: [mem 0xda040000-0xda040fff]
[ 1660.894910] pci 0000:03:00.0: supports D1 D2
[ 1660.894911] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1660.895079] pci 0000:02:00.0: PCI bridge to [bus 03]
[ 1660.895089] pci 0000:02:00.0:   bridge window [mem 0xda000000-0xda0fffff]
[ 1660.895152] pci 0000:02:01.0: PCI bridge to [bus 04-38]
[ 1660.895161] pci 0000:02:01.0:   bridge window [mem 0xc4000000-0xd9efffff]
[ 1660.895168] pci 0000:02:01.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[ 1660.895222] pci 0000:02:02.0: PCI bridge to [bus 39]
[ 1660.895231] pci 0000:02:02.0:   bridge window [mem 0xd9f00000-0xd9ffffff]
[ 1660.895263] pci_bus 0000:02: Allocating resources
[ 1660.895296] pci 0000:02:01.0: bridge window [io  0x1000-0x0fff] to [bus 04-38] add_size 1000
[ 1660.895314] pci 0000:01:00.0: bridge window [io  0x1000-0x0fff] to [bus 02-39] add_size 1000
[ 1660.895318] pci 0000:01:00.0: BAR 13: assigned [io  0x2000-0x2fff]
[ 1660.895321] pci 0000:02:01.0: BAR 13: assigned [io  0x2000-0x2fff]
[ 1660.895322] pci 0000:02:00.0: PCI bridge to [bus 03]
[ 1660.895329] pci 0000:02:00.0:   bridge window [mem 0xda000000-0xda0fffff]
[ 1660.895338] pci 0000:02:01.0: PCI bridge to [bus 04-38]
[ 1660.895341] pci 0000:02:01.0:   bridge window [io  0x2000-0x2fff]
[ 1660.895346] pci 0000:02:01.0:   bridge window [mem 0xc4000000-0xd9efffff]
[ 1660.895351] pci 0000:02:01.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[ 1660.895357] pci 0000:02:02.0: PCI bridge to [bus 39]
[ 1660.895363] pci 0000:02:02.0:   bridge window [mem 0xd9f00000-0xd9ffffff]
[ 1660.895372] pci 0000:01:00.0: PCI bridge to [bus 02-39]
[ 1660.895374] pci 0000:01:00.0:   bridge window [io  0x2000-0x2fff]
[ 1660.895379] pci 0000:01:00.0:   bridge window [mem 0xc4000000-0xda0fffff]
[ 1660.895383] pci 0000:01:00.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[ 1661.283910] pcieport 0000:01:00.0: enabling device (0006 -> 0007)
[ 1661.284659] pcieport 0000:02:01.0: enabling device (0006 -> 0007)
[ 1661.285546] thunderbolt 0000:03:00.0: NHI initialized, starting thunderbolt
[ 1661.285550] thunderbolt 0000:03:00.0: allocating TX ring 0 of size 10
[ 1661.285565] thunderbolt 0000:03:00.0: allocating RX ring 0 of size 10
[ 1661.285582] thunderbolt 0000:03:00.0: control channel created
[ 1661.285583] thunderbolt 0000:03:00.0: control channel starting...
[ 1661.285584] thunderbolt 0000:03:00.0: starting TX ring 0
[ 1661.285591] thunderbolt 0000:03:00.0: enabling interrupt at register 0x38200 bit 0 (0x0 -> 0x1)
[ 1661.285592] thunderbolt 0000:03:00.0: starting RX ring 0
[ 1661.285599] thunderbolt 0000:03:00.0: enabling interrupt at register 0x38200 bit 12 (0x1 -> 0x1001)
[ 1661.731122] ACPI Error: Cannot release Mutex [PATM], not acquired (20170303/exmutex-393)
[ 1661.731143] ACPI Error: Method parse/execution failed [\_SB.PCI0.LPCB.ECDV._Q66] (Node ffff8ba73017bf00), AE_AML_MUTEX_NOT_ACQUIRED (20170303/psparse-543)
[ 1662.538307] thunderbolt 0000:03:00.0: current switch config:
[ 1662.538315] thunderbolt 0000:03:00.0:  Switch: 8086:1576 (Revision: 4, TB Version: 2)
[ 1662.538319] thunderbolt 0000:03:00.0:   Max Port Number: 11
[ 1662.538322] thunderbolt 0000:03:00.0:   Config:
[ 1662.538329] thunderbolt 0000:03:00.0:    Upstream Port Number: 5 Depth: 0 Route String: 0x0 Enabled: 1, PlugEventsDelay: 254ms
[ 1662.538333] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
[ 1662.561266] thunderbolt 0000:03:00.0: 0: uid: 0x8086a3422131ea10
[ 1662.561856] thunderbolt 0000:03:00.0:  Port 0: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1662.561860] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
[ 1662.561864] thunderbolt 0000:03:00.0:   Max counters: 8
[ 1662.561868] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[ 1662.562078] thunderbolt 0000:03:00.0:  Port 1: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1662.562082] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[ 1662.562086] thunderbolt 0000:03:00.0:   Max counters: 16
[ 1662.562090] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
[ 1662.562323] thunderbolt 0000:03:00.0:  Port 2: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1662.562328] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[ 1662.562331] thunderbolt 0000:03:00.0:   Max counters: 16
[ 1662.562335] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
[ 1662.562568] thunderbolt 0000:03:00.0:  Port 3: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1662.562575] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[ 1662.562579] thunderbolt 0000:03:00.0:   Max counters: 16
[ 1662.562583] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
[ 1662.562797] thunderbolt 0000:03:00.0:  Port 4: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1662.562801] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[ 1662.562805] thunderbolt 0000:03:00.0:   Max counters: 16
[ 1662.562808] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
[ 1662.562813] thunderbolt 0000:03:00.0: 0:5: disabled by eeprom
[ 1662.562873] thunderbolt 0000:03:00.0:  Port 6: 8086:1576 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
[ 1662.562877] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[ 1662.562880] thunderbolt 0000:03:00.0:   Max counters: 2
[ 1662.562884] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[ 1662.562938] thunderbolt 0000:03:00.0:  Port 7: 8086:1576 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
[ 1662.562942] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[ 1662.562945] thunderbolt 0000:03:00.0:   Max counters: 2
[ 1662.562949] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[ 1662.563005] thunderbolt 0000:03:00.0:  Port 8: 8086:1576 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0102))
[ 1662.563009] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
[ 1662.563012] thunderbolt 0000:03:00.0:   Max counters: 2
[ 1662.563016] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[ 1662.563019] thunderbolt 0000:03:00.0: 0:9: disabled by eeprom
[ 1662.563082] thunderbolt 0000:03:00.0:  Port 10: 8086:1576 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0101))
[ 1662.563085] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
[ 1662.563089] thunderbolt 0000:03:00.0:   Max counters: 2
[ 1662.563092] thunderbolt 0000:03:00.0:   NFC Credits: 0x1000000
[ 1662.563096] thunderbolt 0000:03:00.0: 0:b: disabled by eeprom
[ 1662.564973] thunderbolt 0000:03:00.0: current switch config:
[ 1662.564983] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB Version: 2)
[ 1662.564989] thunderbolt 0000:03:00.0:   Max Port Number: 11
[ 1662.564994] thunderbolt 0000:03:00.0:   Config:
[ 1662.565004] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 1 Route String: 0x1 Enabled: 1, PlugEventsDelay: 254ms
[ 1662.565011] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
[ 1662.586185] thunderbolt 0000:03:00.0: 1: reading drom (length: 0x6e)
[ 1662.685656] thunderbolt 0000:03:00.0: 1: drom data crc32 mismatch (expected: 0xaf438340, got: 0xaf4383c0), continuing
[ 1662.685923] thunderbolt 0000:03:00.0: 1: uid: 0xd40f7a7928c300
[ 1662.685970] thunderbolt 0000:03:00.0:  Port 0: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1662.685972] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
[ 1662.685974] thunderbolt 0000:03:00.0:   Max counters: 8
[ 1662.685976] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[ 1662.686163] thunderbolt 0000:03:00.0:  Port 1: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1662.686165] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[ 1662.686167] thunderbolt 0000:03:00.0:   Max counters: 16
[ 1662.686169] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
[ 1662.686346] thunderbolt 0000:03:00.0:  Port 2: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1662.686348] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[ 1662.686350] thunderbolt 0000:03:00.0:   Max counters: 16
[ 1662.686351] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
[ 1662.686520] thunderbolt 0000:03:00.0:  Port 3: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1662.686522] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[ 1662.686523] thunderbolt 0000:03:00.0:   Max counters: 16
[ 1662.686525] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
[ 1662.686697] thunderbolt 0000:03:00.0:  Port 4: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1662.686698] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[ 1662.686700] thunderbolt 0000:03:00.0:   Max counters: 16
[ 1662.686702] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
[ 1662.686704] thunderbolt 0000:03:00.0: 1:5: disabled by eeprom
[ 1662.686757] thunderbolt 0000:03:00.0:  Port 6: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100102))
[ 1662.686758] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[ 1662.686760] thunderbolt 0000:03:00.0:   Max counters: 2
[ 1662.686762] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[ 1662.686816] thunderbolt 0000:03:00.0:  Port 7: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
[ 1662.686818] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[ 1662.686820] thunderbolt 0000:03:00.0:   Max counters: 2
[ 1662.686822] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[ 1662.686825] thunderbolt 0000:03:00.0: 1:8: disabled by eeprom
[ 1662.686827] thunderbolt 0000:03:00.0: 1:9: disabled by eeprom
[ 1662.686830] thunderbolt 0000:03:00.0: 1:a: disabled by eeprom
[ 1662.686832] thunderbolt 0000:03:00.0: 1:b: disabled by eeprom
[ 1662.688262] thunderbolt 0000:03:00.0: current switch config:
[ 1662.688271] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB Version: 2)
[ 1662.688273] thunderbolt 0000:03:00.0:   Max Port Number: 11
[ 1662.688275] thunderbolt 0000:03:00.0:   Config:
[ 1662.688284] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 2 Route String: 0x301 Enabled: 1, PlugEventsDelay: 254ms
[ 1662.688287] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
[ 1666.261728] pci 0000:04:00.0: [8086:1578] type 01 class 0x060400
[ 1666.262033] pci 0000:04:00.0: supports D1 D2
[ 1666.262034] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1666.262442] sysfs: cannot create duplicate filename '/class/pci_bus/0000:05'
[ 1666.262459] ------------[ cut here ]------------
[ 1666.262470] WARNING: CPU: 2 PID: 2606 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x56/0x70
[ 1666.262470] Modules linked in: ctr ccm rfcomm uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core videodev media btusb btrtl nvram msr hid_multitouch snd_soc_skl snd_soc_skl_ipc snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_codec_hdmi snd_hda_ext_core snd_soc_sst_match snd_soc_core snd_hda_codec_realtek snd_hda_codec_generic snd_compress snd_pcm_dmaengine bnep ac97_bus snd_hda_intel snd_hda_codec i2c_designware_platform snd_hda_core i2c_designware_core snd_hwdep arc4 snd_pcm intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp dcdbas snd_seq_midi snd_seq_midi_event kvm_intel kvm binfmt_misc irqbypass snd_rawmidi crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc nls_iso8859_1 aesni_intel iwlmvm aes_x86_64 crypto_simd glue_helper cryptd mac80211 snd_seq snd_seq_device joydev
[ 1666.262536]  snd_timer input_leds snd serio_raw iwlwifi cfg80211 rtsx_pci_ms soundcore memstick thunderbolt idma64 nvmem_core virt_dma hci_uart intel_pch_thermal btbcm intel_lpss_pci mei_me processor_thermal_device shpchp mei btqca intel_soc_dts_iosf btintel bluetooth ecdh_generic intel_lpss_acpi intel_hid intel_lpss int3403_thermal sparse_keymap int340x_thermal_zone int3400_thermal tpm_crb acpi_thermal_rel mac_hid acpi_pad acpi_als kfifo_buf industrialio parport_pc ppdev lp parport autofs4 btrfs xor raid6_pq usbhid i915 rtsx_pci_sdmmc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect psmouse sysimgblt fb_sys_fops rtsx_pci nvme drm nvme_core ahci libahci i2c_hid wmi hid pinctrl_sunrisepoint video pinctrl_intel
[ 1666.262598] CPU: 2 PID: 2606 Comm: kworker/u8:3 Not tainted 4.12.0-rc1+ #6
[ 1666.262600] Hardware name: Dell Inc. XPS 13 9350/09JHRY, BIOS 1.4.13 12/28/2016
[ 1666.262607] Workqueue: kacpi_hotplug acpi_hotplug_work_fn
[ 1666.262609] task: ffff8ba7256d8ec0 task.stack: ffffa924c2158000
[ 1666.262613] RIP: 0010:sysfs_warn_dup+0x56/0x70
[ 1666.262614] RSP: 0018:ffffa924c215ba08 EFLAGS: 00010286
[ 1666.262615] RAX: 0000000000000040 RBX: ffff8ba72af7c000 RCX: 0000000000000006
[ 1666.262617] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff8ba740d0dd40
[ 1666.262618] RBP: ffffa924c215ba20 R08: 0000000000000001 R09: 0000000000000526
[ 1666.262618] R10: ffff8ba72e6a2000 R11: 0000000000000526 R12: ffff8ba7295135a0
[ 1666.262619] R13: ffff8ba72dcdd3c0 R14: 0000000000000001 R15: ffffffffffffffef
[ 1666.262620] FS:  0000000000000000(0000) GS:ffff8ba740d00000(0000) knlGS:0000000000000000
[ 1666.262621] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1666.262622] CR2: 00005650509e0738 CR3: 000000011ce09000 CR4: 00000000003406e0
[ 1666.262624] Call Trace:
[ 1666.262630]  sysfs_do_create_link_sd.isra.2+0x9e/0xb0
[ 1666.262634]  sysfs_create_link+0x25/0x40
[ 1666.262641]  device_add+0x2e8/0x670
[ 1666.262643]  device_register+0x1a/0x20
[ 1666.262649]  pci_add_new_bus+0x179/0x420
[ 1666.262651]  pci_scan_bridge+0x5fc/0x640
[ 1666.262654]  pci_scan_child_bus+0xa4/0x180
[ 1666.262658]  pci_scan_bridge+0x447/0x640
[ 1666.262660]  ? pci_scan_single_device+0x51/0xd0
[ 1666.262662]  pci_scan_child_bus+0xa4/0x180
[ 1666.262664]  pci_scan_bridge+0x447/0x640
[ 1666.262669]  enable_slot+0xc6/0x2f0
[ 1666.262670]  ? trim_stale_devices+0xab/0x150
[ 1666.262672]  acpiphp_check_bridge.part.7+0xff/0x140
[ 1666.262674]  acpiphp_hotplug_notify+0x173/0x200
[ 1666.262677]  ? free_bridge+0x130/0x130
[ 1666.262681]  acpi_device_hotplug+0x9d/0x480
[ 1666.262685]  acpi_hotplug_work_fn+0x1e/0x30
[ 1666.262688]  process_one_work+0x156/0x3f0
[ 1666.262691]  worker_thread+0x4b/0x410
[ 1666.262694]  kthread+0x109/0x140
[ 1666.262696]  ? process_one_work+0x3f0/0x3f0
[ 1666.262699]  ? kthread_create_on_node+0x70/0x70
[ 1666.262706]  ret_from_fork+0x2c/0x40
[ 1666.262709] Code: 85 c0 48 89 c3 74 12 b9 00 10 00 00 48 89 c2 31 f6 4c 89 ef e8 ac c8 ff ff 4c 89 e2 48 89 de 48 c7 c7 58 59 ca 93 e8 77 27 ee ff <0f> ff 48 89 df e8 a0 fb f4 ff 5b 41 5c 41 5d 5d c3 66 0f 1f 84 
[ 1666.262744] ---[ end trace a1df410cd1bc506d ]---
[ 1666.262829] ------------[ cut here ]------------
[ 1666.262837] WARNING: CPU: 2 PID: 2606 at drivers/pci/probe.c:894 pci_add_new_bus+0x394/0x420
[ 1666.262838] Modules linked in: ctr ccm rfcomm uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core videodev media btusb btrtl nvram msr hid_multitouch snd_soc_skl snd_soc_skl_ipc snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_codec_hdmi snd_hda_ext_core snd_soc_sst_match snd_soc_core snd_hda_codec_realtek snd_hda_codec_generic snd_compress snd_pcm_dmaengine bnep ac97_bus snd_hda_intel snd_hda_codec i2c_designware_platform snd_hda_core i2c_designware_core snd_hwdep arc4 snd_pcm intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp dcdbas snd_seq_midi snd_seq_midi_event kvm_intel kvm binfmt_misc irqbypass snd_rawmidi crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc nls_iso8859_1 aesni_intel iwlmvm aes_x86_64 crypto_simd glue_helper cryptd mac80211 snd_seq snd_seq_device joydev
[ 1666.262869]  snd_timer input_leds snd serio_raw iwlwifi cfg80211 rtsx_pci_ms soundcore memstick thunderbolt idma64 nvmem_core virt_dma hci_uart intel_pch_thermal btbcm intel_lpss_pci mei_me processor_thermal_device shpchp mei btqca intel_soc_dts_iosf btintel bluetooth ecdh_generic intel_lpss_acpi intel_hid intel_lpss int3403_thermal sparse_keymap int340x_thermal_zone int3400_thermal tpm_crb acpi_thermal_rel mac_hid acpi_pad acpi_als kfifo_buf industrialio parport_pc ppdev lp parport autofs4 btrfs xor raid6_pq usbhid i915 rtsx_pci_sdmmc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect psmouse sysimgblt fb_sys_fops rtsx_pci nvme drm nvme_core ahci libahci i2c_hid wmi hid pinctrl_sunrisepoint video pinctrl_intel
[ 1666.262922] CPU: 2 PID: 2606 Comm: kworker/u8:3 Tainted: G        W       4.12.0-rc1+ #6
[ 1666.262923] Hardware name: Dell Inc. XPS 13 9350/09JHRY, BIOS 1.4.13 12/28/2016
[ 1666.262928] Workqueue: kacpi_hotplug acpi_hotplug_work_fn
[ 1666.262930] task: ffff8ba7256d8ec0 task.stack: ffffa924c2158000
[ 1666.262933] RIP: 0010:pci_add_new_bus+0x394/0x420
[ 1666.262934] RSP: 0018:ffffa924c215bae8 EFLAGS: 00010282
[ 1666.262935] RAX: 00000000ffffffef RBX: ffff8ba72cbf6000 RCX: 0000000000000000
[ 1666.262936] RDX: 0000000000000001 RSI: ffff8ba72cbf6168 RDI: 0000000000000001
[ 1666.262936] RBP: ffffa924c215bb28 R08: 000000000001efa0 R09: ffffffff935a0dcb
[ 1666.262937] R10: ffffce3451a83600 R11: 0000000000000000 R12: ffff8ba72cbf6800
[ 1666.262938] R13: ffff8ba72af7a000 R14: ffff8ba72af7a000 R15: ffff8ba72cbf6120
[ 1666.262939] FS:  0000000000000000(0000) GS:ffff8ba740d00000(0000) knlGS:0000000000000000
[ 1666.262939] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1666.262940] CR2: 00005650509e0738 CR3: 000000011ce09000 CR4: 00000000003406e0
[ 1666.262941] Call Trace:
[ 1666.262945]  pci_scan_bridge+0x5fc/0x640
[ 1666.262947]  pci_scan_child_bus+0xa4/0x180
[ 1666.262949]  pci_scan_bridge+0x447/0x640
[ 1666.262951]  ? pci_scan_single_device+0x51/0xd0
[ 1666.262952]  pci_scan_child_bus+0xa4/0x180
[ 1666.262954]  pci_scan_bridge+0x447/0x640
[ 1666.262957]  enable_slot+0xc6/0x2f0
[ 1666.262959]  ? trim_stale_devices+0xab/0x150
[ 1666.262961]  acpiphp_check_bridge.part.7+0xff/0x140
[ 1666.262962]  acpiphp_hotplug_notify+0x173/0x200
[ 1666.262964]  ? free_bridge+0x130/0x130
[ 1666.262966]  acpi_device_hotplug+0x9d/0x480
[ 1666.262968]  acpi_hotplug_work_fn+0x1e/0x30
[ 1666.262971]  process_one_work+0x156/0x3f0
[ 1666.262973]  worker_thread+0x4b/0x410
[ 1666.262975]  kthread+0x109/0x140
[ 1666.262976]  ? process_one_work+0x3f0/0x3f0
[ 1666.262978]  ? kthread_create_on_node+0x70/0x70
[ 1666.262981]  ret_from_fork+0x2c/0x40
[ 1666.262982] Code: 0f 84 80 00 00 00 83 c2 02 48 63 d2 0f b6 82 79 be a6 93 eb 0b 83 f8 01 19 c0 83 e0 03 83 c0 0c 88 83 e3 00 00 00 e9 7d fe ff ff <0f> ff e9 e6 fd ff ff 0f ff 48 c7 c6 4a a4 cc 93 89 c2 4c 89 ff 
[ 1666.263011] ---[ end trace a1df410cd1bc506e ]---
[ 1666.263055] pci 0000:05:01.0: [8086:1578] type 01 class 0x060400
[ 1666.263393] pci 0000:05:01.0: supports D1 D2
[ 1666.263395] pci 0000:05:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1666.263590] pci 0000:05:04.0: [8086:1578] type 01 class 0x060400
[ 1666.263860] pci 0000:05:04.0: supports D1 D2
[ 1666.263861] pci 0000:05:04.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1666.264032] pci 0000:04:00.0: PCI bridge to [bus 05-38]
[ 1666.264049] pci 0000:04:00.0:   bridge window [mem 0xc4000000-0xd9efffff]
[ 1666.264061] pci 0000:04:00.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[ 1666.264150] pci 0000:05:01.0: PCI bridge to [bus 06]
[ 1666.264260] pci 0000:05:04.0: PCI bridge to [bus 07-38]
[ 1666.264277] pci 0000:05:04.0:   bridge window [mem 0xc4000000-0xd9efffff]
[ 1666.264288] pci 0000:05:04.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[ 1666.264372] pci_bus 0000:02: Allocating resources
[ 1666.264442] pci 0000:05:04.0: bridge window [io  0x1000-0x0fff] to [bus 07-38] add_size 1000
[ 1666.264459] pci 0000:04:00.0: bridge window [io  0x1000-0x0fff] to [bus 05-38] add_size 1000
[ 1666.264487] pci 0000:04:00.0: BAR 13: assigned [io  0x2000-0x2fff]
[ 1666.264489] pci 0000:05:04.0: BAR 13: assigned [io  0x2000-0x2fff]
[ 1666.264490] pci 0000:05:01.0: PCI bridge to [bus 06]
[ 1666.264515] pci 0000:05:04.0: PCI bridge to [bus 07-38]
[ 1666.264518] pci 0000:05:04.0:   bridge window [io  0x2000-0x2fff]
[ 1666.264527] pci 0000:05:04.0:   bridge window [mem 0xc4000000-0xd9efffff]
[ 1666.264533] pci 0000:05:04.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[ 1666.264544] pci 0000:04:00.0: PCI bridge to [bus 05-38]
[ 1666.264548] pci 0000:04:00.0:   bridge window [io  0x2000-0x2fff]
[ 1666.264556] pci 0000:04:00.0:   bridge window [mem 0xc4000000-0xd9efffff]
[ 1666.264562] pci 0000:04:00.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[ 1666.264621] pcieport 0000:04:00.0: enabling device (0006 -> 0007)
[ 1666.266164] pcieport 0000:05:04.0: enabling device (0006 -> 0007)
[ 1667.831384] thunderbolt 0000:03:00.0: 301: reading drom (length: 0x75)
[ 1668.000509] thunderbolt 0000:03:00.0: 301: uid: 0x80864f418ce17510
[ 1668.000582] thunderbolt 0000:03:00.0:  Port 0: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1668.000587] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
[ 1668.000591] thunderbolt 0000:03:00.0:   Max counters: 8
[ 1668.000595] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[ 1668.000807] thunderbolt 0000:03:00.0:  Port 1: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1668.000812] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[ 1668.000815] thunderbolt 0000:03:00.0:   Max counters: 16
[ 1668.000819] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
[ 1668.001033] thunderbolt 0000:03:00.0:  Port 2: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1668.001037] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[ 1668.001041] thunderbolt 0000:03:00.0:   Max counters: 16
[ 1668.001044] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
[ 1668.005369] thunderbolt 0000:03:00.0:  Port 3: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1668.005378] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[ 1668.005384] thunderbolt 0000:03:00.0:   Max counters: 16
[ 1668.005391] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
[ 1668.005655] thunderbolt 0000:03:00.0:  Port 4: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[ 1668.005661] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[ 1668.005666] thunderbolt 0000:03:00.0:   Max counters: 16
[ 1668.005672] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
[ 1668.005679] thunderbolt 0000:03:00.0: 301:5: disabled by eeprom
[ 1668.005757] thunderbolt 0000:03:00.0:  Port 6: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100102))
[ 1668.005763] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[ 1668.005768] thunderbolt 0000:03:00.0:   Max counters: 2
[ 1668.005774] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[ 1668.005848] thunderbolt 0000:03:00.0:  Port 7: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
[ 1668.005854] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[ 1668.005859] thunderbolt 0000:03:00.0:   Max counters: 2
[ 1668.005865] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[ 1668.005940] thunderbolt 0000:03:00.0:  Port 8: 8086:1578 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0102))
[ 1668.005946] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
[ 1668.005952] thunderbolt 0000:03:00.0:   Max counters: 2
[ 1668.005957] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[ 1668.005963] thunderbolt 0000:03:00.0: 301:9: disabled by eeprom
[ 1668.006041] thunderbolt 0000:03:00.0:  Port 10: 8086:1578 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0101))
[ 1668.006047] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
[ 1668.006052] thunderbolt 0000:03:00.0:   Max counters: 2
[ 1668.006058] thunderbolt 0000:03:00.0:   NFC Credits: 0x1000000
[ 1668.006064] thunderbolt 0000:03:00.0: 301:b: disabled by eeprom
[ 1671.717837] pci 0000:07:00.0: [8086:1578] type 01 class 0x060400
[ 1671.718339] pci 0000:07:00.0: supports D1 D2
[ 1671.718343] pci 0000:07:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1671.719133] pci 0000:08:01.0: [8086:1578] type 01 class 0x060400
[ 1671.719656] pci 0000:08:01.0: supports D1 D2
[ 1671.719660] pci 0000:08:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1671.720022] pci 0000:08:04.0: [8086:1578] type 01 class 0x060400
[ 1671.720512] pci 0000:08:04.0: supports D1 D2
[ 1671.720517] pci 0000:08:04.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1671.720925] pci 0000:07:00.0: PCI bridge to [bus 08-38]
[ 1671.720952] pci 0000:07:00.0:   bridge window [io  0x2000-0x2fff]
[ 1671.720966] pci 0000:07:00.0:   bridge window [mem 0xc4000000-0xd9efffff]
[ 1671.720986] pci 0000:07:00.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[ 1671.721402] pci 0000:09:00.0: [1b21:1142] type 00 class 0x0c0330
[ 1671.721548] pci 0000:09:00.0: reg 0x10: [mem 0xc4000000-0xc4007fff 64bit]
[ 1671.722373] pci 0000:09:00.0: PME# supported from D3cold
[ 1671.723086] pci 0000:08:01.0: PCI bridge to [bus 09]
[ 1671.723110] pci 0000:08:01.0:   bridge window [io  0x2000-0x2fff]
[ 1671.723121] pci 0000:08:01.0:   bridge window [mem 0xc4000000-0xc40fffff]
[ 1671.723391] pci 0000:08:04.0: PCI bridge to [bus 0a-38]
[ 1671.723422] pci 0000:08:04.0:   bridge window [mem 0xc4100000-0xd9efffff]
[ 1671.723441] pci 0000:08:04.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[ 1671.723636] pci_bus 0000:02: Allocating resources
[ 1671.723805] pci 0000:08:04.0: bridge window [io  0x1000-0x0fff] to [bus 0a-38] add_size 1000
[ 1671.723918] pci 0000:08:04.0: BAR 13: no space for [io  size 0x1000]
[ 1671.723921] pci 0000:08:04.0: BAR 13: failed to assign [io  size 0x1000]
[ 1671.723923] pci 0000:08:04.0: BAR 13: no space for [io  size 0x1000]
[ 1671.723924] pci 0000:08:04.0: BAR 13: failed to assign [io  size 0x1000]
[ 1671.723927] pci 0000:08:01.0: PCI bridge to [bus 09]
[ 1671.723933] pci 0000:08:01.0:   bridge window [io  0x2000-0x2fff]
[ 1671.723949] pci 0000:08:01.0:   bridge window [mem 0xc4000000-0xc40fffff]
[ 1671.723978] pci 0000:08:04.0: PCI bridge to [bus 0a-38]
[ 1671.723992] pci 0000:08:04.0:   bridge window [mem 0xc4100000-0xd9efffff]
[ 1671.724003] pci 0000:08:04.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[ 1671.724021] pci 0000:07:00.0: PCI bridge to [bus 08-38]
[ 1671.724027] pci 0000:07:00.0:   bridge window [io  0x2000-0x2fff]
[ 1671.724042] pci 0000:07:00.0:   bridge window [mem 0xc4000000-0xd9efffff]
[ 1671.724052] pci 0000:07:00.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[ 1671.729214] xhci_hcd 0000:09:00.0: xHCI Host Controller
[ 1671.729219] xhci_hcd 0000:09:00.0: new USB bus registered, assigned bus number 3
[ 1671.790011] xhci_hcd 0000:09:00.0: hcc params 0x0200e081 hci version 0x100 quirks 0x00000010
[ 1671.790376] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002
[ 1671.790378] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1671.790379] usb usb3: Product: xHCI Host Controller
[ 1671.790379] usb usb3: Manufacturer: Linux 4.12.0-rc1+ xhci-hcd
[ 1671.790380] usb usb3: SerialNumber: 0000:09:00.0
[ 1671.790536] hub 3-0:1.0: USB hub found
[ 1671.790552] hub 3-0:1.0: 2 ports detected
[ 1671.790682] xhci_hcd 0000:09:00.0: xHCI Host Controller
[ 1671.790688] xhci_hcd 0000:09:00.0: new USB bus registered, assigned bus number 4
[ 1671.790761] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[ 1671.790779] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003
[ 1671.790780] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1671.790781] usb usb4: Product: xHCI Host Controller
[ 1671.790782] usb usb4: Manufacturer: Linux 4.12.0-rc1+ xhci-hcd
[ 1671.790783] usb usb4: SerialNumber: 0000:09:00.0
[ 1671.791243] hub 4-0:1.0: USB hub found
[ 1671.791255] hub 4-0:1.0: 2 ports detected
[ 1672.177422] usb 3-1: new high-speed USB device number 2 using xhci_hcd
[ 1672.378504] usb 3-1: New USB device found, idVendor=0424, idProduct=2137
[ 1672.378507] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 1672.378510] usb 3-1: Product: USB2137B
[ 1672.378511] usb 3-1: Manufacturer: SMSC
[ 1672.380179] hub 3-1:1.0: USB hub found
[ 1672.380272] hub 3-1:1.0: 7 ports detected
[ 1672.497889] usb 4-1: new SuperSpeed USB device number 2 using xhci_hcd
[ 1672.523006] usb 4-1: New USB device found, idVendor=0424, idProduct=5537
[ 1672.523014] usb 4-1: New USB device strings: Mfr=2, Product=3, SerialNumber=0
[ 1672.523018] usb 4-1: Product: USB5537B
[ 1672.523023] usb 4-1: Manufacturer: SMSC
[ 1672.525130] hub 4-1:1.0: USB hub found
[ 1672.525678] hub 4-1:1.0: 7 ports detected
[ 1672.669558] usb 3-1.5: new high-speed USB device number 3 using xhci_hcd
[ 1672.854006] usb 3-1.5: New USB device found, idVendor=0bda, idProduct=4014
[ 1672.854014] usb 3-1.5: New USB device strings: Mfr=3, Product=1, SerialNumber=2
[ 1672.854018] usb 3-1.5: Product: USB Audio
[ 1672.854022] usb 3-1.5: Manufacturer: Generic
[ 1672.854025] usb 3-1.5: SerialNumber: 200901010001
[ 1672.913951] usb 4-1.2: new SuperSpeed USB device number 3 using xhci_hcd
[ 1672.939202] usb 4-1.2: New USB device found, idVendor=0bda, idProduct=8153
[ 1672.939209] usb 4-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=6
[ 1672.939214] usb 4-1.2: Product: USB 10/100/1000 LAN
[ 1672.939218] usb 4-1.2: Manufacturer: Realtek
[ 1672.939223] usb 4-1.2: SerialNumber: 000001000000
[ 1673.641329] usbcore: registered new interface driver r8152
[ 1673.645860] usbcore: registered new interface driver cdc_ether
[ 1674.275052] usb 4-1.2: reset SuperSpeed USB device number 3 using xhci_hcd
[ 1674.364596] usbcore: registered new interface driver snd-usb-audio
[ 1674.376658] r8152 4-1.2:1.0 (unnamed net_device) (uninitialized): Using pass-thru MAC addr 84:7b:eb:59:92:d2
[ 1674.440410] r8152 4-1.2:1.0 eth0: v1.08.9
[ 1674.670704] r8152 4-1.2:1.0 enx847beb5992d2: renamed from eth0
[ 1674.722446] IPv6: ADDRCONF(NETDEV_UP): enx847beb5992d2: link is not ready
[ 1674.763848] IPv6: ADDRCONF(NETDEV_UP): enx847beb5992d2: link is not ready

Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Mika Westerberg]

On Fri, May 19, 2017 at 04:35:01PM +0000, Mario.Limonciello@dell.com wrote:
> Mika,
> 
> Thanks for submitting this series.  
> I've tested security level stuff a little bit, 
> but I'm running into what I think is some odd behavior.

Thanks for testing.

> Here's my setup:
> System: I'm using is an XPS 9350 (Has Alpine Ridge).  It's got NVM 16.0.  BIOS 1.4.13
> TBT Device: Dell TB16 (which has AR in the cable and in dock - both NVM 16.0).
> 
> I created a udev rule that will automatically authorize the dock and cable.
> #dell cable
> ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{vendor}=="0xd4", ATTR{device}=="0xb051", ATTR{authorized}="1"
> #dell dock
> ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{vendor}=="0xd4", ATTR{device}=="0xb054", ATTR{authorized}="1"
> 
> If I boot the system with the dock connected the cable shows up and authorizes but the dock doesn't.
> ---
> [    6.916669] thunderbolt 0000:03:00.0: current switch config:
> [    6.916671] thunderbolt 0000:03:00.0:  Switch: 8086:1576 (Revision: 4, TB Version: 2)
> [    6.916673] thunderbolt 0000:03:00.0:   Max Port Number: 11
> [    6.916673] thunderbolt 0000:03:00.0:   Config:
> [    6.916675] thunderbolt 0000:03:00.0:    Upstream Port Number: 5 Depth: 0 Route String: 0x0 Enabled: 1, PlugEventsDelay: 254ms
> [    6.916676] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
> [    6.927293] thunderbolt 0000:03:00.0: 0: uid: 0x8086a3422131ea10
> [    6.927632] thunderbolt 0000:03:00.0:  Port 0: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [    6.927633] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
> [    6.927634] thunderbolt 0000:03:00.0:   Max counters: 8
> [    6.927635] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> [    6.927760] thunderbolt 0000:03:00.0:  Port 1: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [    6.927761] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [    6.927762] thunderbolt 0000:03:00.0:   Max counters: 16
> [    6.927763] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
> [    6.927890] thunderbolt 0000:03:00.0:  Port 2: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [    6.927895] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [    6.927896] thunderbolt 0000:03:00.0:   Max counters: 16
> [    6.927897] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
> [    6.928091] thunderbolt 0000:03:00.0:  Port 3: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [    6.928092] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [    6.928092] thunderbolt 0000:03:00.0:   Max counters: 16
> [    6.928093] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
> [    6.928217] thunderbolt 0000:03:00.0:  Port 4: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [    6.928219] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [    6.928219] thunderbolt 0000:03:00.0:   Max counters: 16
> [    6.928220] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
> [    6.928222] thunderbolt 0000:03:00.0: 0:5: disabled by eeprom
> [    6.928261] thunderbolt 0000:03:00.0:  Port 6: 8086:1576 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
> [    6.928262] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> [    6.928262] thunderbolt 0000:03:00.0:   Max counters: 2
> [    6.928263] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> [    6.928304] thunderbolt 0000:03:00.0:  Port 7: 8086:1576 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
> [    6.928305] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> [    6.928306] thunderbolt 0000:03:00.0:   Max counters: 2
> [    6.928306] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> [    6.930860] thunderbolt 0000:03:00.0:  Port 8: 8086:1576 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0102))
> [    6.930862] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
> [    6.930863] thunderbolt 0000:03:00.0:   Max counters: 2
> [    6.930864] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> [    6.930865] thunderbolt 0000:03:00.0: 0:9: disabled by eeprom
> [    6.930908] thunderbolt 0000:03:00.0:  Port 10: 8086:1576 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0101))
> [    6.930909] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
> [    6.930910] thunderbolt 0000:03:00.0:   Max counters: 2
> [    6.930911] thunderbolt 0000:03:00.0:   NFC Credits: 0x1000000
> [    6.930912] thunderbolt 0000:03:00.0: 0:b: disabled by eeprom
> [    6.932249] thunderbolt 0000:03:00.0: current switch config:
> [    6.932252] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB Version: 2)
> [    6.932253] thunderbolt 0000:03:00.0:   Max Port Number: 11
> [    6.932254] thunderbolt 0000:03:00.0:   Config:
> [    6.932255] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 1 Route String: 0x1 Enabled: 1, PlugEventsDelay: 254ms
> [    6.932256] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
> [    6.940642] thunderbolt 0000:03:00.0: 1: reading drom (length: 0x6e)
> [    7.040048] thunderbolt 0000:03:00.0: 1: drom data crc32 mismatch (expected: 0xaf438340, got: 0xaf4383c0), continuing
> [    7.040247] thunderbolt 0000:03:00.0: 1: uid: 0xd40f7a7928c300
> [    7.040296] thunderbolt 0000:03:00.0:  Port 0: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [    7.040297] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
> [    7.040298] thunderbolt 0000:03:00.0:   Max counters: 8
> [    7.040299] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> [    7.040574] thunderbolt 0000:03:00.0:  Port 1: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [    7.040575] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [    7.040576] thunderbolt 0000:03:00.0:   Max counters: 16
> [    7.040577] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
> [    7.040711] thunderbolt 0000:03:00.0:  Port 2: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [    7.040712] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [    7.040713] thunderbolt 0000:03:00.0:   Max counters: 16
> [    7.040714] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
> [    7.040848] thunderbolt 0000:03:00.0:  Port 3: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [    7.040849] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [    7.040850] thunderbolt 0000:03:00.0:   Max counters: 16
> [    7.040851] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
> [    7.040987] thunderbolt 0000:03:00.0:  Port 4: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [    7.040988] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [    7.040990] thunderbolt 0000:03:00.0:   Max counters: 16
> [    7.040991] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
> [    7.040992] thunderbolt 0000:03:00.0: 1:5: disabled by eeprom
> [    7.041033] thunderbolt 0000:03:00.0:  Port 6: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100102))
> [    7.041034] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> [    7.041035] thunderbolt 0000:03:00.0:   Max counters: 2
> [    7.041036] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> [    7.041078] thunderbolt 0000:03:00.0:  Port 7: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
> [    7.041080] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> [    7.041080] thunderbolt 0000:03:00.0:   Max counters: 2
> [    7.041081] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> [    7.041083] thunderbolt 0000:03:00.0: 1:8: disabled by eeprom
> [    7.041084] thunderbolt 0000:03:00.0: 1:9: disabled by eeprom
> [    7.041085] thunderbolt 0000:03:00.0: 1:a: disabled by eeprom
> [    7.041085] thunderbolt 0000:03:00.0: 1:b: disabled by eeprom
> [    7.041827] thunderbolt 0000:03:00.0: current switch config:
> [    7.041829] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB Version: 2)
> [    7.041829] thunderbolt 0000:03:00.0:   Max Port Number: 11
> [    7.041830] thunderbolt 0000:03:00.0:   Config:
> [    7.041831] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 2 Route String: 0x301 Enabled: 1, PlugEventsDelay: 254ms
> [    7.041832] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
> [    7.416678] pci 0000:04:00.0: [8086:1578] type 01 class 0x060400
> [    7.416925] pci 0000:04:00.0: supports D1 D2
> [    7.416926] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold

> [    7.428433] pci_bus 0000:05: [bus 05] partially hidden behind bridge 0000:04 [bus 04]
> [    7.428500] pci_bus 0000:39: busn_res: can not insert [bus 39] under [bus 02-05] (conflicts with (null) [bus 02-05])

These two I've seen before.

> [    7.428503] pcieport 0000:02:02.0: PCI bridge to [bus 39]
> [    7.428512] pcieport 0000:02:02.0:   bridge window [mem 0xd9f00000-0xd9ffffff]
> [    7.428519] pci_bus 0000:39: [bus 39] partially hidden behind bridge 0000:02 [bus 02-05]

And this.

It happens occasionally when you reboot the machine when a device is
connected but seems to be dependent on the BIOS version. Since it is the
BIOS who is supposed to enumerated these devices, I suspect that it is
either problem in BIOS or our PCI enumeration code does something wrong.

> [    7.428542] pci_bus 0000:02: Allocating resources
> [    7.428544] pcieport 0000:02:02.0: can't claim BAR 14 [mem 0xd9f00000-0xd9ffffff]: no compatible bridge window
> [    7.451018] Bluetooth: RFCOMM TTY layer initialized
> [    7.451022] Bluetooth: RFCOMM socket layer initialized
> [    7.451026] Bluetooth: RFCOMM ver 1.11
> [   11.958457] wlp58s0: authenticate with 50:6a:03:a7:1c:27
> [   11.969167] wlp58s0: send auth to 50:6a:03:a7:1c:27 (try 1/3)
> [   11.975768] wlp58s0: authenticated
> [   11.976386] wlp58s0: associate with 50:6a:03:a7:1c:27 (try 1/3)
> [   11.984817] wlp58s0: RX AssocResp from 50:6a:03:a7:1c:27 (capab=0x1411 status=0 aid=2)
> [   12.001864] wlp58s0: associated
> [   12.001913] IPv6: ADDRCONF(NETDEV_CHANGE): wlp58s0: link becomes ready
> [   27.616672] thunderbolt 0000:03:00.0: timeout reading config space 2 from 0x5
> [   27.616674] thunderbolt 0000:03:00.0: 301: cannot find TB_VSEC_CAP_PLUG_EVENTS aborting
> 
> If I then unplug the cable and plug it back in the dock does show up and authorize properly, but is a traceback
> along the way.
> ---
> [ 1653.226296] pcieport 0000:02:02.0: Refused to change power state, currently in D3
> [ 1653.729116] thunderbolt 0000:03:00.0: stopping RX ring 0
> [ 1653.729130] thunderbolt 0000:03:00.0: disabling interrupt at register 0x38200 bit 12 (0xffffffff -> 0xffffefff)
> [ 1653.729159] thunderbolt 0000:03:00.0: stopping TX ring 0
> [ 1653.729168] thunderbolt 0000:03:00.0: disabling interrupt at register 0x38200 bit 0 (0xffffffff -> 0xfffffffe)
> [ 1653.729195] thunderbolt 0000:03:00.0: control channel stopped
> [ 1653.729362] thunderbolt 0000:03:00.0: freeing RX ring 0
> [ 1653.729381] thunderbolt 0000:03:00.0: freeing TX ring 0
> [ 1653.729407] thunderbolt 0000:03:00.0: shutdown
> [ 1653.753091] pcieport 0000:02:00.0: Refused to change power state, currently in D3
> [ 1653.756383] pci_bus 0000:03: busn_res: [bus 03] is released
> [ 1653.756676] pci_bus 0000:04: busn_res: [bus 04] is released
> [ 1653.757479] pci_bus 0000:02: busn_res: [bus 02-05] is released
> [ 1660.846964] ACPI Error: [SPRT] Namespace lookup failure, AE_ALREADY_EXISTS (20170303/dswload2-330)
> [ 1660.846979] ACPI Exception: AE_ALREADY_EXISTS, During name lookup/catalog (20170303/psobject-241)
> [ 1660.846985] ACPI Error: Method parse/execution failed [\_GPE._E42] (Node ffff8ba73016b488), AE_ALREADY_EXISTS (20170303/psparse-543)
> [ 1660.846996] ACPI Error: Method parse/execution failed [\_GPE._E42] (Node ffff8ba73016b488), AE_ALREADY_EXISTS (20170303/psparse-543)
> [ 1660.847009] ACPI Exception: AE_ALREADY_EXISTS, while evaluating GPE method [_E42] (20170303/evgpe-646)
> [ 1660.893399] pci 0000:01:00.0: [8086:1576] type 01 class 0x060400
> [ 1660.893529] pci 0000:01:00.0: supports D1 D2
> [ 1660.893530] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> [ 1660.893625] pci 0000:01:00.0: System wakeup disabled by ACPI
> [ 1660.893776] pci 0000:02:00.0: [8086:1576] type 01 class 0x060400
> [ 1660.894047] pci 0000:02:00.0: supports D1 D2
> [ 1660.894049] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> [ 1660.894178] pci 0000:02:01.0: [8086:1576] type 01 class 0x060400
> [ 1660.894320] pci 0000:02:01.0: supports D1 D2
> [ 1660.894321] pci 0000:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
> [ 1660.894409] pci 0000:02:02.0: [8086:1576] type 01 class 0x060400
> [ 1660.894542] pci 0000:02:02.0: supports D1 D2
> [ 1660.894543] pci 0000:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
> [ 1660.894637] pci 0000:01:00.0: PCI bridge to [bus 02-39]
> [ 1660.894646] pci 0000:01:00.0:   bridge window [mem 0xc4000000-0xda0fffff]
> [ 1660.894652] pci 0000:01:00.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
> [ 1660.894716] pci 0000:03:00.0: [8086:1575] type 00 class 0x088000
> [ 1660.894744] pci 0000:03:00.0: reg 0x10: [mem 0xda000000-0xda03ffff]
> [ 1660.894756] pci 0000:03:00.0: reg 0x14: [mem 0xda040000-0xda040fff]
> [ 1660.894910] pci 0000:03:00.0: supports D1 D2
> [ 1660.894911] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> [ 1660.895079] pci 0000:02:00.0: PCI bridge to [bus 03]
> [ 1660.895089] pci 0000:02:00.0:   bridge window [mem 0xda000000-0xda0fffff]
> [ 1660.895152] pci 0000:02:01.0: PCI bridge to [bus 04-38]
> [ 1660.895161] pci 0000:02:01.0:   bridge window [mem 0xc4000000-0xd9efffff]
> [ 1660.895168] pci 0000:02:01.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
> [ 1660.895222] pci 0000:02:02.0: PCI bridge to [bus 39]
> [ 1660.895231] pci 0000:02:02.0:   bridge window [mem 0xd9f00000-0xd9ffffff]
> [ 1660.895263] pci_bus 0000:02: Allocating resources
> [ 1660.895296] pci 0000:02:01.0: bridge window [io  0x1000-0x0fff] to [bus 04-38] add_size 1000
> [ 1660.895314] pci 0000:01:00.0: bridge window [io  0x1000-0x0fff] to [bus 02-39] add_size 1000
> [ 1660.895318] pci 0000:01:00.0: BAR 13: assigned [io  0x2000-0x2fff]
> [ 1660.895321] pci 0000:02:01.0: BAR 13: assigned [io  0x2000-0x2fff]
> [ 1660.895322] pci 0000:02:00.0: PCI bridge to [bus 03]
> [ 1660.895329] pci 0000:02:00.0:   bridge window [mem 0xda000000-0xda0fffff]
> [ 1660.895338] pci 0000:02:01.0: PCI bridge to [bus 04-38]
> [ 1660.895341] pci 0000:02:01.0:   bridge window [io  0x2000-0x2fff]
> [ 1660.895346] pci 0000:02:01.0:   bridge window [mem 0xc4000000-0xd9efffff]
> [ 1660.895351] pci 0000:02:01.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
> [ 1660.895357] pci 0000:02:02.0: PCI bridge to [bus 39]
> [ 1660.895363] pci 0000:02:02.0:   bridge window [mem 0xd9f00000-0xd9ffffff]
> [ 1660.895372] pci 0000:01:00.0: PCI bridge to [bus 02-39]
> [ 1660.895374] pci 0000:01:00.0:   bridge window [io  0x2000-0x2fff]
> [ 1660.895379] pci 0000:01:00.0:   bridge window [mem 0xc4000000-0xda0fffff]
> [ 1660.895383] pci 0000:01:00.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
> [ 1661.283910] pcieport 0000:01:00.0: enabling device (0006 -> 0007)
> [ 1661.284659] pcieport 0000:02:01.0: enabling device (0006 -> 0007)
> [ 1661.285546] thunderbolt 0000:03:00.0: NHI initialized, starting thunderbolt
> [ 1661.285550] thunderbolt 0000:03:00.0: allocating TX ring 0 of size 10
> [ 1661.285565] thunderbolt 0000:03:00.0: allocating RX ring 0 of size 10
> [ 1661.285582] thunderbolt 0000:03:00.0: control channel created
> [ 1661.285583] thunderbolt 0000:03:00.0: control channel starting...
> [ 1661.285584] thunderbolt 0000:03:00.0: starting TX ring 0
> [ 1661.285591] thunderbolt 0000:03:00.0: enabling interrupt at register 0x38200 bit 0 (0x0 -> 0x1)
> [ 1661.285592] thunderbolt 0000:03:00.0: starting RX ring 0
> [ 1661.285599] thunderbolt 0000:03:00.0: enabling interrupt at register 0x38200 bit 12 (0x1 -> 0x1001)
> [ 1661.731122] ACPI Error: Cannot release Mutex [PATM], not acquired (20170303/exmutex-393)
> [ 1661.731143] ACPI Error: Method parse/execution failed [\_SB.PCI0.LPCB.ECDV._Q66] (Node ffff8ba73017bf00), AE_AML_MUTEX_NOT_ACQUIRED (20170303/psparse-543)
> [ 1662.538307] thunderbolt 0000:03:00.0: current switch config:
> [ 1662.538315] thunderbolt 0000:03:00.0:  Switch: 8086:1576 (Revision: 4, TB Version: 2)
> [ 1662.538319] thunderbolt 0000:03:00.0:   Max Port Number: 11
> [ 1662.538322] thunderbolt 0000:03:00.0:   Config:
> [ 1662.538329] thunderbolt 0000:03:00.0:    Upstream Port Number: 5 Depth: 0 Route String: 0x0 Enabled: 1, PlugEventsDelay: 254ms
> [ 1662.538333] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
> [ 1662.561266] thunderbolt 0000:03:00.0: 0: uid: 0x8086a3422131ea10
> [ 1662.561856] thunderbolt 0000:03:00.0:  Port 0: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [ 1662.561860] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
> [ 1662.561864] thunderbolt 0000:03:00.0:   Max counters: 8
> [ 1662.561868] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> [ 1662.562078] thunderbolt 0000:03:00.0:  Port 1: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [ 1662.562082] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [ 1662.562086] thunderbolt 0000:03:00.0:   Max counters: 16
> [ 1662.562090] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
> [ 1662.562323] thunderbolt 0000:03:00.0:  Port 2: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [ 1662.562328] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [ 1662.562331] thunderbolt 0000:03:00.0:   Max counters: 16
> [ 1662.562335] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
> [ 1662.562568] thunderbolt 0000:03:00.0:  Port 3: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [ 1662.562575] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [ 1662.562579] thunderbolt 0000:03:00.0:   Max counters: 16
> [ 1662.562583] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
> [ 1662.562797] thunderbolt 0000:03:00.0:  Port 4: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [ 1662.562801] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [ 1662.562805] thunderbolt 0000:03:00.0:   Max counters: 16
> [ 1662.562808] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
> [ 1662.562813] thunderbolt 0000:03:00.0: 0:5: disabled by eeprom
> [ 1662.562873] thunderbolt 0000:03:00.0:  Port 6: 8086:1576 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
> [ 1662.562877] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> [ 1662.562880] thunderbolt 0000:03:00.0:   Max counters: 2
> [ 1662.562884] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> [ 1662.562938] thunderbolt 0000:03:00.0:  Port 7: 8086:1576 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
> [ 1662.562942] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> [ 1662.562945] thunderbolt 0000:03:00.0:   Max counters: 2
> [ 1662.562949] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> [ 1662.563005] thunderbolt 0000:03:00.0:  Port 8: 8086:1576 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0102))
> [ 1662.563009] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
> [ 1662.563012] thunderbolt 0000:03:00.0:   Max counters: 2
> [ 1662.563016] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> [ 1662.563019] thunderbolt 0000:03:00.0: 0:9: disabled by eeprom
> [ 1662.563082] thunderbolt 0000:03:00.0:  Port 10: 8086:1576 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0101))
> [ 1662.563085] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
> [ 1662.563089] thunderbolt 0000:03:00.0:   Max counters: 2
> [ 1662.563092] thunderbolt 0000:03:00.0:   NFC Credits: 0x1000000
> [ 1662.563096] thunderbolt 0000:03:00.0: 0:b: disabled by eeprom
> [ 1662.564973] thunderbolt 0000:03:00.0: current switch config:
> [ 1662.564983] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB Version: 2)
> [ 1662.564989] thunderbolt 0000:03:00.0:   Max Port Number: 11
> [ 1662.564994] thunderbolt 0000:03:00.0:   Config:
> [ 1662.565004] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 1 Route String: 0x1 Enabled: 1, PlugEventsDelay: 254ms
> [ 1662.565011] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
> [ 1662.586185] thunderbolt 0000:03:00.0: 1: reading drom (length: 0x6e)
> [ 1662.685656] thunderbolt 0000:03:00.0: 1: drom data crc32 mismatch (expected: 0xaf438340, got: 0xaf4383c0), continuing
> [ 1662.685923] thunderbolt 0000:03:00.0: 1: uid: 0xd40f7a7928c300
> [ 1662.685970] thunderbolt 0000:03:00.0:  Port 0: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [ 1662.685972] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
> [ 1662.685974] thunderbolt 0000:03:00.0:   Max counters: 8
> [ 1662.685976] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> [ 1662.686163] thunderbolt 0000:03:00.0:  Port 1: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [ 1662.686165] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [ 1662.686167] thunderbolt 0000:03:00.0:   Max counters: 16
> [ 1662.686169] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
> [ 1662.686346] thunderbolt 0000:03:00.0:  Port 2: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [ 1662.686348] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [ 1662.686350] thunderbolt 0000:03:00.0:   Max counters: 16
> [ 1662.686351] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
> [ 1662.686520] thunderbolt 0000:03:00.0:  Port 3: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [ 1662.686522] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [ 1662.686523] thunderbolt 0000:03:00.0:   Max counters: 16
> [ 1662.686525] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
> [ 1662.686697] thunderbolt 0000:03:00.0:  Port 4: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
> [ 1662.686698] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> [ 1662.686700] thunderbolt 0000:03:00.0:   Max counters: 16
> [ 1662.686702] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
> [ 1662.686704] thunderbolt 0000:03:00.0: 1:5: disabled by eeprom
> [ 1662.686757] thunderbolt 0000:03:00.0:  Port 6: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100102))
> [ 1662.686758] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> [ 1662.686760] thunderbolt 0000:03:00.0:   Max counters: 2
> [ 1662.686762] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> [ 1662.686816] thunderbolt 0000:03:00.0:  Port 7: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
> [ 1662.686818] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> [ 1662.686820] thunderbolt 0000:03:00.0:   Max counters: 2
> [ 1662.686822] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> [ 1662.686825] thunderbolt 0000:03:00.0: 1:8: disabled by eeprom
> [ 1662.686827] thunderbolt 0000:03:00.0: 1:9: disabled by eeprom
> [ 1662.686830] thunderbolt 0000:03:00.0: 1:a: disabled by eeprom
> [ 1662.686832] thunderbolt 0000:03:00.0: 1:b: disabled by eeprom
> [ 1662.688262] thunderbolt 0000:03:00.0: current switch config:
> [ 1662.688271] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB Version: 2)
> [ 1662.688273] thunderbolt 0000:03:00.0:   Max Port Number: 11
> [ 1662.688275] thunderbolt 0000:03:00.0:   Config:
> [ 1662.688284] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 2 Route String: 0x301 Enabled: 1, PlugEventsDelay: 254ms
> [ 1662.688287] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
> [ 1666.261728] pci 0000:04:00.0: [8086:1578] type 01 class 0x060400
> [ 1666.262033] pci 0000:04:00.0: supports D1 D2
> [ 1666.262034] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> [ 1666.262442] sysfs: cannot create duplicate filename '/class/pci_bus/0000:05'

This is probably due the previous problem of the PCI device being hidden
behind a bridge.

If you shutdown the machine and boot it without devices connected and
then when the OS is up, connect the device, do you see the same issue?

Also can you try so that you power off the machine completely, connect
the device and power it on. Do you see the issue?

Thanks.

RE: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Mario.Limonciello]

> 
> It happens occasionally when you reboot the machine when a device is
> connected but seems to be dependent on the BIOS version. Since it is the
> BIOS who is supposed to enumerated these devices, I suspect that it is
> either problem in BIOS or our PCI enumeration code does something wrong.
> 

I'm fairly certain it's an issue somewhere with Linux PCI enumeration.  I took 
the exact same HW and switched it out the SSD to one w/ Win10 1607.
I set the dock and cable to "always allow" in TBT settings applet.

I don't reproduce any problems with enumeration with the dock plugged in
on cold boot.  All the devices hanging off the bridge show up properly.

> > [    7.428542] pci_bus 0000:02: Allocating resources
> > [    7.428544] pcieport 0000:02:02.0: can't claim BAR 14 [mem 0xd9f00000-
> 0xd9ffffff]: no compatible bridge window
> > [    7.451018] Bluetooth: RFCOMM TTY layer initialized
> > [    7.451022] Bluetooth: RFCOMM socket layer initialized
> > [    7.451026] Bluetooth: RFCOMM ver 1.11
> > [   11.958457] wlp58s0: authenticate with 50:6a:03:a7:1c:27
> > [   11.969167] wlp58s0: send auth to 50:6a:03:a7:1c:27 (try 1/3)
> > [   11.975768] wlp58s0: authenticated
> > [   11.976386] wlp58s0: associate with 50:6a:03:a7:1c:27 (try 1/3)
> > [   11.984817] wlp58s0: RX AssocResp from 50:6a:03:a7:1c:27 (capab=0x1411
> status=0 aid=2)
> > [   12.001864] wlp58s0: associated
> > [   12.001913] IPv6: ADDRCONF(NETDEV_CHANGE): wlp58s0: link becomes ready
> > [   27.616672] thunderbolt 0000:03:00.0: timeout reading config space 2 from 0x5
> > [   27.616674] thunderbolt 0000:03:00.0: 301: cannot find
> TB_VSEC_CAP_PLUG_EVENTS aborting
> >
> > If I then unplug the cable and plug it back in the dock does show up and authorize
> properly, but is a traceback
> > along the way.
> > ---
> > [ 1653.226296] pcieport 0000:02:02.0: Refused to change power state, currently
> in D3
> > [ 1653.729116] thunderbolt 0000:03:00.0: stopping RX ring 0
> > [ 1653.729130] thunderbolt 0000:03:00.0: disabling interrupt at register 0x38200
> bit 12 (0xffffffff -> 0xffffefff)
> > [ 1653.729159] thunderbolt 0000:03:00.0: stopping TX ring 0
> > [ 1653.729168] thunderbolt 0000:03:00.0: disabling interrupt at register 0x38200
> bit 0 (0xffffffff -> 0xfffffffe)
> > [ 1653.729195] thunderbolt 0000:03:00.0: control channel stopped
> > [ 1653.729362] thunderbolt 0000:03:00.0: freeing RX ring 0
> > [ 1653.729381] thunderbolt 0000:03:00.0: freeing TX ring 0
> > [ 1653.729407] thunderbolt 0000:03:00.0: shutdown
> > [ 1653.753091] pcieport 0000:02:00.0: Refused to change power state, currently
> in D3
> > [ 1653.756383] pci_bus 0000:03: busn_res: [bus 03] is released
> > [ 1653.756676] pci_bus 0000:04: busn_res: [bus 04] is released
> > [ 1653.757479] pci_bus 0000:02: busn_res: [bus 02-05] is released
> > [ 1660.846964] ACPI Error: [SPRT] Namespace lookup failure,
> AE_ALREADY_EXISTS (20170303/dswload2-330)
> > [ 1660.846979] ACPI Exception: AE_ALREADY_EXISTS, During name
> lookup/catalog (20170303/psobject-241)
> > [ 1660.846985] ACPI Error: Method parse/execution failed [\_GPE._E42] (Node
> ffff8ba73016b488), AE_ALREADY_EXISTS (20170303/psparse-543)
> > [ 1660.846996] ACPI Error: Method parse/execution failed [\_GPE._E42] (Node
> ffff8ba73016b488), AE_ALREADY_EXISTS (20170303/psparse-543)
> > [ 1660.847009] ACPI Exception: AE_ALREADY_EXISTS, while evaluating GPE
> method [_E42] (20170303/evgpe-646)
> > [ 1660.893399] pci 0000:01:00.0: [8086:1576] type 01 class 0x060400
> > [ 1660.893529] pci 0000:01:00.0: supports D1 D2
> > [ 1660.893530] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [ 1660.893625] pci 0000:01:00.0: System wakeup disabled by ACPI
> > [ 1660.893776] pci 0000:02:00.0: [8086:1576] type 01 class 0x060400
> > [ 1660.894047] pci 0000:02:00.0: supports D1 D2
> > [ 1660.894049] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [ 1660.894178] pci 0000:02:01.0: [8086:1576] type 01 class 0x060400
> > [ 1660.894320] pci 0000:02:01.0: supports D1 D2
> > [ 1660.894321] pci 0000:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [ 1660.894409] pci 0000:02:02.0: [8086:1576] type 01 class 0x060400
> > [ 1660.894542] pci 0000:02:02.0: supports D1 D2
> > [ 1660.894543] pci 0000:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [ 1660.894637] pci 0000:01:00.0: PCI bridge to [bus 02-39]
> > [ 1660.894646] pci 0000:01:00.0:   bridge window [mem 0xc4000000-0xda0fffff]
> > [ 1660.894652] pci 0000:01:00.0:   bridge window [mem 0xa0000000-0xc1ffffff
> 64bit pref]
> > [ 1660.894716] pci 0000:03:00.0: [8086:1575] type 00 class 0x088000
> > [ 1660.894744] pci 0000:03:00.0: reg 0x10: [mem 0xda000000-0xda03ffff]
> > [ 1660.894756] pci 0000:03:00.0: reg 0x14: [mem 0xda040000-0xda040fff]
> > [ 1660.894910] pci 0000:03:00.0: supports D1 D2
> > [ 1660.894911] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [ 1660.895079] pci 0000:02:00.0: PCI bridge to [bus 03]
> > [ 1660.895089] pci 0000:02:00.0:   bridge window [mem 0xda000000-0xda0fffff]
> > [ 1660.895152] pci 0000:02:01.0: PCI bridge to [bus 04-38]
> > [ 1660.895161] pci 0000:02:01.0:   bridge window [mem 0xc4000000-0xd9efffff]
> > [ 1660.895168] pci 0000:02:01.0:   bridge window [mem 0xa0000000-0xc1ffffff
> 64bit pref]
> > [ 1660.895222] pci 0000:02:02.0: PCI bridge to [bus 39]
> > [ 1660.895231] pci 0000:02:02.0:   bridge window [mem 0xd9f00000-0xd9ffffff]
> > [ 1660.895263] pci_bus 0000:02: Allocating resources
> > [ 1660.895296] pci 0000:02:01.0: bridge window [io  0x1000-0x0fff] to [bus 04-
> 38] add_size 1000
> > [ 1660.895314] pci 0000:01:00.0: bridge window [io  0x1000-0x0fff] to [bus 02-
> 39] add_size 1000
> > [ 1660.895318] pci 0000:01:00.0: BAR 13: assigned [io  0x2000-0x2fff]
> > [ 1660.895321] pci 0000:02:01.0: BAR 13: assigned [io  0x2000-0x2fff]
> > [ 1660.895322] pci 0000:02:00.0: PCI bridge to [bus 03]
> > [ 1660.895329] pci 0000:02:00.0:   bridge window [mem 0xda000000-0xda0fffff]
> > [ 1660.895338] pci 0000:02:01.0: PCI bridge to [bus 04-38]
> > [ 1660.895341] pci 0000:02:01.0:   bridge window [io  0x2000-0x2fff]
> > [ 1660.895346] pci 0000:02:01.0:   bridge window [mem 0xc4000000-0xd9efffff]
> > [ 1660.895351] pci 0000:02:01.0:   bridge window [mem 0xa0000000-0xc1ffffff
> 64bit pref]
> > [ 1660.895357] pci 0000:02:02.0: PCI bridge to [bus 39]
> > [ 1660.895363] pci 0000:02:02.0:   bridge window [mem 0xd9f00000-0xd9ffffff]
> > [ 1660.895372] pci 0000:01:00.0: PCI bridge to [bus 02-39]
> > [ 1660.895374] pci 0000:01:00.0:   bridge window [io  0x2000-0x2fff]
> > [ 1660.895379] pci 0000:01:00.0:   bridge window [mem 0xc4000000-0xda0fffff]
> > [ 1660.895383] pci 0000:01:00.0:   bridge window [mem 0xa0000000-0xc1ffffff
> 64bit pref]
> > [ 1661.283910] pcieport 0000:01:00.0: enabling device (0006 -> 0007)
> > [ 1661.284659] pcieport 0000:02:01.0: enabling device (0006 -> 0007)
> > [ 1661.285546] thunderbolt 0000:03:00.0: NHI initialized, starting thunderbolt
> > [ 1661.285550] thunderbolt 0000:03:00.0: allocating TX ring 0 of size 10
> > [ 1661.285565] thunderbolt 0000:03:00.0: allocating RX ring 0 of size 10
> > [ 1661.285582] thunderbolt 0000:03:00.0: control channel created
> > [ 1661.285583] thunderbolt 0000:03:00.0: control channel starting...
> > [ 1661.285584] thunderbolt 0000:03:00.0: starting TX ring 0
> > [ 1661.285591] thunderbolt 0000:03:00.0: enabling interrupt at register 0x38200
> bit 0 (0x0 -> 0x1)
> > [ 1661.285592] thunderbolt 0000:03:00.0: starting RX ring 0
> > [ 1661.285599] thunderbolt 0000:03:00.0: enabling interrupt at register 0x38200
> bit 12 (0x1 -> 0x1001)
> > [ 1661.731122] ACPI Error: Cannot release Mutex [PATM], not acquired
> (20170303/exmutex-393)
> > [ 1661.731143] ACPI Error: Method parse/execution failed
> [\_SB.PCI0.LPCB.ECDV._Q66] (Node ffff8ba73017bf00),
> AE_AML_MUTEX_NOT_ACQUIRED (20170303/psparse-543)
> > [ 1662.538307] thunderbolt 0000:03:00.0: current switch config:
> > [ 1662.538315] thunderbolt 0000:03:00.0:  Switch: 8086:1576 (Revision: 4, TB
> Version: 2)
> > [ 1662.538319] thunderbolt 0000:03:00.0:   Max Port Number: 11
> > [ 1662.538322] thunderbolt 0000:03:00.0:   Config:
> > [ 1662.538329] thunderbolt 0000:03:00.0:    Upstream Port Number: 5 Depth: 0
> Route String: 0x0 Enabled: 1, PlugEventsDelay: 254ms
> > [ 1662.538333] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
> > [ 1662.561266] thunderbolt 0000:03:00.0: 0: uid: 0x8086a3422131ea10
> > [ 1662.561856] thunderbolt 0000:03:00.0:  Port 0: 8086:1576 (Revision: 4, TB
> Version: 1, Type: Port (0x1))
> > [ 1662.561860] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
> > [ 1662.561864] thunderbolt 0000:03:00.0:   Max counters: 8
> > [ 1662.561868] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> > [ 1662.562078] thunderbolt 0000:03:00.0:  Port 1: 8086:1576 (Revision: 4, TB
> Version: 1, Type: Port (0x1))
> > [ 1662.562082] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > [ 1662.562086] thunderbolt 0000:03:00.0:   Max counters: 16
> > [ 1662.562090] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
> > [ 1662.562323] thunderbolt 0000:03:00.0:  Port 2: 8086:1576 (Revision: 4, TB
> Version: 1, Type: Port (0x1))
> > [ 1662.562328] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > [ 1662.562331] thunderbolt 0000:03:00.0:   Max counters: 16
> > [ 1662.562335] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
> > [ 1662.562568] thunderbolt 0000:03:00.0:  Port 3: 8086:1576 (Revision: 4, TB
> Version: 1, Type: Port (0x1))
> > [ 1662.562575] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > [ 1662.562579] thunderbolt 0000:03:00.0:   Max counters: 16
> > [ 1662.562583] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
> > [ 1662.562797] thunderbolt 0000:03:00.0:  Port 4: 8086:1576 (Revision: 4, TB
> Version: 1, Type: Port (0x1))
> > [ 1662.562801] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > [ 1662.562805] thunderbolt 0000:03:00.0:   Max counters: 16
> > [ 1662.562808] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
> > [ 1662.562813] thunderbolt 0000:03:00.0: 0:5: disabled by eeprom
> > [ 1662.562873] thunderbolt 0000:03:00.0:  Port 6: 8086:1576 (Revision: 4, TB
> Version: 1, Type: PCIe (0x100101))
> > [ 1662.562877] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> > [ 1662.562880] thunderbolt 0000:03:00.0:   Max counters: 2
> > [ 1662.562884] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> > [ 1662.562938] thunderbolt 0000:03:00.0:  Port 7: 8086:1576 (Revision: 4, TB
> Version: 1, Type: PCIe (0x100101))
> > [ 1662.562942] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> > [ 1662.562945] thunderbolt 0000:03:00.0:   Max counters: 2
> > [ 1662.562949] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> > [ 1662.563005] thunderbolt 0000:03:00.0:  Port 8: 8086:1576 (Revision: 4, TB
> Version: 1, Type: DP/HDMI (0xe0102))
> > [ 1662.563009] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
> > [ 1662.563012] thunderbolt 0000:03:00.0:   Max counters: 2
> > [ 1662.563016] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> > [ 1662.563019] thunderbolt 0000:03:00.0: 0:9: disabled by eeprom
> > [ 1662.563082] thunderbolt 0000:03:00.0:  Port 10: 8086:1576 (Revision: 4, TB
> Version: 1, Type: DP/HDMI (0xe0101))
> > [ 1662.563085] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
> > [ 1662.563089] thunderbolt 0000:03:00.0:   Max counters: 2
> > [ 1662.563092] thunderbolt 0000:03:00.0:   NFC Credits: 0x1000000
> > [ 1662.563096] thunderbolt 0000:03:00.0: 0:b: disabled by eeprom
> > [ 1662.564973] thunderbolt 0000:03:00.0: current switch config:
> > [ 1662.564983] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB
> Version: 2)
> > [ 1662.564989] thunderbolt 0000:03:00.0:   Max Port Number: 11
> > [ 1662.564994] thunderbolt 0000:03:00.0:   Config:
> > [ 1662.565004] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 1
> Route String: 0x1 Enabled: 1, PlugEventsDelay: 254ms
> > [ 1662.565011] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
> > [ 1662.586185] thunderbolt 0000:03:00.0: 1: reading drom (length: 0x6e)
> > [ 1662.685656] thunderbolt 0000:03:00.0: 1: drom data crc32 mismatch
> (expected: 0xaf438340, got: 0xaf4383c0), continuing
> > [ 1662.685923] thunderbolt 0000:03:00.0: 1: uid: 0xd40f7a7928c300
> > [ 1662.685970] thunderbolt 0000:03:00.0:  Port 0: 8086:1578 (Revision: 4, TB
> Version: 1, Type: Port (0x1))
> > [ 1662.685972] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
> > [ 1662.685974] thunderbolt 0000:03:00.0:   Max counters: 8
> > [ 1662.685976] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> > [ 1662.686163] thunderbolt 0000:03:00.0:  Port 1: 8086:1578 (Revision: 4, TB
> Version: 1, Type: Port (0x1))
> > [ 1662.686165] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > [ 1662.686167] thunderbolt 0000:03:00.0:   Max counters: 16
> > [ 1662.686169] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
> > [ 1662.686346] thunderbolt 0000:03:00.0:  Port 2: 8086:1578 (Revision: 4, TB
> Version: 1, Type: Port (0x1))
> > [ 1662.686348] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > [ 1662.686350] thunderbolt 0000:03:00.0:   Max counters: 16
> > [ 1662.686351] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
> > [ 1662.686520] thunderbolt 0000:03:00.0:  Port 3: 8086:1578 (Revision: 4, TB
> Version: 1, Type: Port (0x1))
> > [ 1662.686522] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > [ 1662.686523] thunderbolt 0000:03:00.0:   Max counters: 16
> > [ 1662.686525] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
> > [ 1662.686697] thunderbolt 0000:03:00.0:  Port 4: 8086:1578 (Revision: 4, TB
> Version: 1, Type: Port (0x1))
> > [ 1662.686698] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > [ 1662.686700] thunderbolt 0000:03:00.0:   Max counters: 16
> > [ 1662.686702] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
> > [ 1662.686704] thunderbolt 0000:03:00.0: 1:5: disabled by eeprom
> > [ 1662.686757] thunderbolt 0000:03:00.0:  Port 6: 8086:1578 (Revision: 4, TB
> Version: 1, Type: PCIe (0x100102))
> > [ 1662.686758] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> > [ 1662.686760] thunderbolt 0000:03:00.0:   Max counters: 2
> > [ 1662.686762] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> > [ 1662.686816] thunderbolt 0000:03:00.0:  Port 7: 8086:1578 (Revision: 4, TB
> Version: 1, Type: PCIe (0x100101))
> > [ 1662.686818] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> > [ 1662.686820] thunderbolt 0000:03:00.0:   Max counters: 2
> > [ 1662.686822] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> > [ 1662.686825] thunderbolt 0000:03:00.0: 1:8: disabled by eeprom
> > [ 1662.686827] thunderbolt 0000:03:00.0: 1:9: disabled by eeprom
> > [ 1662.686830] thunderbolt 0000:03:00.0: 1:a: disabled by eeprom
> > [ 1662.686832] thunderbolt 0000:03:00.0: 1:b: disabled by eeprom
> > [ 1662.688262] thunderbolt 0000:03:00.0: current switch config:
> > [ 1662.688271] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB
> Version: 2)
> > [ 1662.688273] thunderbolt 0000:03:00.0:   Max Port Number: 11
> > [ 1662.688275] thunderbolt 0000:03:00.0:   Config:
> > [ 1662.688284] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 2
> Route String: 0x301 Enabled: 1, PlugEventsDelay: 254ms
> > [ 1662.688287] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
> > [ 1666.261728] pci 0000:04:00.0: [8086:1578] type 01 class 0x060400
> > [ 1666.262033] pci 0000:04:00.0: supports D1 D2
> > [ 1666.262034] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [ 1666.262442] sysfs: cannot create duplicate filename '/class/pci_bus/0000:05'
> 
> This is probably due the previous problem of the PCI device being hidden
> behind a bridge.
> 
> If you shutdown the machine and boot it without devices connected and
> then when the OS is up, connect the device, do you see the same issue?
> 

The kernel traceback doesn't happen in this scenario, but if I follow these steps:
1) Boot
2) Plugin dock, make sure fully enumerated
3) Unplug
4) Replug

I get the trackeback warning about duplicate filenames again.

The second time I unplug I also get tracebacks related to a NULL pointer dereference
In kernfs_find_ns.

> Also can you try so that you power off the machine completely, connect
> the device and power it on. Do you see the issue?
> 

Yes, I do see error from the first syslog snippet about being stuck behind a host bridge.

Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Mika Westerberg]

On Fri, May 19, 2017 at 08:19:48PM +0300, Mika Westerberg wrote:
> These two I've seen before.
> 
> > [    7.428503] pcieport 0000:02:02.0: PCI bridge to [bus 39]
> > [    7.428512] pcieport 0000:02:02.0:   bridge window [mem 0xd9f00000-0xd9ffffff]
> > [    7.428519] pci_bus 0000:39: [bus 39] partially hidden behind bridge 0000:02 [bus 02-05]
> 
> And this.
> 
> It happens occasionally when you reboot the machine when a device is
> connected but seems to be dependent on the BIOS version. Since it is the
> BIOS who is supposed to enumerated these devices, I suspect that it is
> either problem in BIOS or our PCI enumeration code does something wrong.

I tried on Intel Skull Canyon NUC so that I downgraded the NVM firmware
from 25 to 18. With that I see the exactly same issue. Upgrading it back
to 25 seems to fix it.

Even with version 18 if I plug devices after boot or if the machine is
completely shut down from power button with devices connected, it works.
The problem happens only when the machine is warn booted.

However, since I'm able to reproduce this - I'll try to investigate what
might be the root cause.

Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Mika Westerberg]

On Fri, May 19, 2017 at 05:54:37PM +0000, Mario.Limonciello@dell.com wrote:
> > 
> > It happens occasionally when you reboot the machine when a device is
> > connected but seems to be dependent on the BIOS version. Since it is the
> > BIOS who is supposed to enumerated these devices, I suspect that it is
> > either problem in BIOS or our PCI enumeration code does something wrong.
> > 
> 
> I'm fairly certain it's an issue somewhere with Linux PCI enumeration.  I took 
> the exact same HW and switched it out the SSD to one w/ Win10 1607.
> I set the dock and cable to "always allow" in TBT settings applet.
> 
> I don't reproduce any problems with enumeration with the dock plugged in
> on cold boot.  All the devices hanging off the bridge show up properly.

You mean in Windows you don't reproduce the problem, right? Even when
you reboot the machine with devices connected.

I tried on Intel Skull Canyon NUC so that I disabled the thunderbolt
driver and after reboot (warm) I can see the PCI scan error about bus
being partially hidden behind a bridge.

When this happens PCIe ports of the thunderbolt device/host seem to be
unconfigured and Linux then decides to reconfigure them which leads to
the problem. When it works we get ACPI hotplug event to the PCIe root
port and the PCIe upstream/downstream ports are properly configured by
the BIOS.

I guess Windows does something differently here than what we do when PCI
devices are enumerated.

> > > [    7.428542] pci_bus 0000:02: Allocating resources
> > > [    7.428544] pcieport 0000:02:02.0: can't claim BAR 14 [mem 0xd9f00000-
> > 0xd9ffffff]: no compatible bridge window
> > > [    7.451018] Bluetooth: RFCOMM TTY layer initialized
> > > [    7.451022] Bluetooth: RFCOMM socket layer initialized
> > > [    7.451026] Bluetooth: RFCOMM ver 1.11
> > > [   11.958457] wlp58s0: authenticate with 50:6a:03:a7:1c:27
> > > [   11.969167] wlp58s0: send auth to 50:6a:03:a7:1c:27 (try 1/3)
> > > [   11.975768] wlp58s0: authenticated
> > > [   11.976386] wlp58s0: associate with 50:6a:03:a7:1c:27 (try 1/3)
> > > [   11.984817] wlp58s0: RX AssocResp from 50:6a:03:a7:1c:27 (capab=0x1411
> > status=0 aid=2)
> > > [   12.001864] wlp58s0: associated
> > > [   12.001913] IPv6: ADDRCONF(NETDEV_CHANGE): wlp58s0: link becomes ready
> > > [   27.616672] thunderbolt 0000:03:00.0: timeout reading config space 2 from 0x5
> > > [   27.616674] thunderbolt 0000:03:00.0: 301: cannot find
> > TB_VSEC_CAP_PLUG_EVENTS aborting
> > >
> > > If I then unplug the cable and plug it back in the dock does show up and authorize
> > properly, but is a traceback
> > > along the way.
> > > ---
> > > [ 1653.226296] pcieport 0000:02:02.0: Refused to change power state, currently
> > in D3
> > > [ 1653.729116] thunderbolt 0000:03:00.0: stopping RX ring 0
> > > [ 1653.729130] thunderbolt 0000:03:00.0: disabling interrupt at register 0x38200
> > bit 12 (0xffffffff -> 0xffffefff)
> > > [ 1653.729159] thunderbolt 0000:03:00.0: stopping TX ring 0
> > > [ 1653.729168] thunderbolt 0000:03:00.0: disabling interrupt at register 0x38200
> > bit 0 (0xffffffff -> 0xfffffffe)
> > > [ 1653.729195] thunderbolt 0000:03:00.0: control channel stopped
> > > [ 1653.729362] thunderbolt 0000:03:00.0: freeing RX ring 0
> > > [ 1653.729381] thunderbolt 0000:03:00.0: freeing TX ring 0
> > > [ 1653.729407] thunderbolt 0000:03:00.0: shutdown
> > > [ 1653.753091] pcieport 0000:02:00.0: Refused to change power state, currently
> > in D3
> > > [ 1653.756383] pci_bus 0000:03: busn_res: [bus 03] is released
> > > [ 1653.756676] pci_bus 0000:04: busn_res: [bus 04] is released
> > > [ 1653.757479] pci_bus 0000:02: busn_res: [bus 02-05] is released
> > > [ 1660.846964] ACPI Error: [SPRT] Namespace lookup failure,
> > AE_ALREADY_EXISTS (20170303/dswload2-330)
> > > [ 1660.846979] ACPI Exception: AE_ALREADY_EXISTS, During name
> > lookup/catalog (20170303/psobject-241)
> > > [ 1660.846985] ACPI Error: Method parse/execution failed [\_GPE._E42] (Node
> > ffff8ba73016b488), AE_ALREADY_EXISTS (20170303/psparse-543)
> > > [ 1660.846996] ACPI Error: Method parse/execution failed [\_GPE._E42] (Node
> > ffff8ba73016b488), AE_ALREADY_EXISTS (20170303/psparse-543)
> > > [ 1660.847009] ACPI Exception: AE_ALREADY_EXISTS, while evaluating GPE
> > method [_E42] (20170303/evgpe-646)
> > > [ 1660.893399] pci 0000:01:00.0: [8086:1576] type 01 class 0x060400
> > > [ 1660.893529] pci 0000:01:00.0: supports D1 D2
> > > [ 1660.893530] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> > > [ 1660.893625] pci 0000:01:00.0: System wakeup disabled by ACPI
> > > [ 1660.893776] pci 0000:02:00.0: [8086:1576] type 01 class 0x060400
> > > [ 1660.894047] pci 0000:02:00.0: supports D1 D2
> > > [ 1660.894049] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> > > [ 1660.894178] pci 0000:02:01.0: [8086:1576] type 01 class 0x060400
> > > [ 1660.894320] pci 0000:02:01.0: supports D1 D2
> > > [ 1660.894321] pci 0000:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
> > > [ 1660.894409] pci 0000:02:02.0: [8086:1576] type 01 class 0x060400
> > > [ 1660.894542] pci 0000:02:02.0: supports D1 D2
> > > [ 1660.894543] pci 0000:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
> > > [ 1660.894637] pci 0000:01:00.0: PCI bridge to [bus 02-39]
> > > [ 1660.894646] pci 0000:01:00.0:   bridge window [mem 0xc4000000-0xda0fffff]
> > > [ 1660.894652] pci 0000:01:00.0:   bridge window [mem 0xa0000000-0xc1ffffff
> > 64bit pref]
> > > [ 1660.894716] pci 0000:03:00.0: [8086:1575] type 00 class 0x088000
> > > [ 1660.894744] pci 0000:03:00.0: reg 0x10: [mem 0xda000000-0xda03ffff]
> > > [ 1660.894756] pci 0000:03:00.0: reg 0x14: [mem 0xda040000-0xda040fff]
> > > [ 1660.894910] pci 0000:03:00.0: supports D1 D2
> > > [ 1660.894911] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> > > [ 1660.895079] pci 0000:02:00.0: PCI bridge to [bus 03]
> > > [ 1660.895089] pci 0000:02:00.0:   bridge window [mem 0xda000000-0xda0fffff]
> > > [ 1660.895152] pci 0000:02:01.0: PCI bridge to [bus 04-38]
> > > [ 1660.895161] pci 0000:02:01.0:   bridge window [mem 0xc4000000-0xd9efffff]
> > > [ 1660.895168] pci 0000:02:01.0:   bridge window [mem 0xa0000000-0xc1ffffff
> > 64bit pref]
> > > [ 1660.895222] pci 0000:02:02.0: PCI bridge to [bus 39]
> > > [ 1660.895231] pci 0000:02:02.0:   bridge window [mem 0xd9f00000-0xd9ffffff]
> > > [ 1660.895263] pci_bus 0000:02: Allocating resources
> > > [ 1660.895296] pci 0000:02:01.0: bridge window [io  0x1000-0x0fff] to [bus 04-
> > 38] add_size 1000
> > > [ 1660.895314] pci 0000:01:00.0: bridge window [io  0x1000-0x0fff] to [bus 02-
> > 39] add_size 1000
> > > [ 1660.895318] pci 0000:01:00.0: BAR 13: assigned [io  0x2000-0x2fff]
> > > [ 1660.895321] pci 0000:02:01.0: BAR 13: assigned [io  0x2000-0x2fff]
> > > [ 1660.895322] pci 0000:02:00.0: PCI bridge to [bus 03]
> > > [ 1660.895329] pci 0000:02:00.0:   bridge window [mem 0xda000000-0xda0fffff]
> > > [ 1660.895338] pci 0000:02:01.0: PCI bridge to [bus 04-38]
> > > [ 1660.895341] pci 0000:02:01.0:   bridge window [io  0x2000-0x2fff]
> > > [ 1660.895346] pci 0000:02:01.0:   bridge window [mem 0xc4000000-0xd9efffff]
> > > [ 1660.895351] pci 0000:02:01.0:   bridge window [mem 0xa0000000-0xc1ffffff
> > 64bit pref]
> > > [ 1660.895357] pci 0000:02:02.0: PCI bridge to [bus 39]
> > > [ 1660.895363] pci 0000:02:02.0:   bridge window [mem 0xd9f00000-0xd9ffffff]
> > > [ 1660.895372] pci 0000:01:00.0: PCI bridge to [bus 02-39]
> > > [ 1660.895374] pci 0000:01:00.0:   bridge window [io  0x2000-0x2fff]
> > > [ 1660.895379] pci 0000:01:00.0:   bridge window [mem 0xc4000000-0xda0fffff]
> > > [ 1660.895383] pci 0000:01:00.0:   bridge window [mem 0xa0000000-0xc1ffffff
> > 64bit pref]
> > > [ 1661.283910] pcieport 0000:01:00.0: enabling device (0006 -> 0007)
> > > [ 1661.284659] pcieport 0000:02:01.0: enabling device (0006 -> 0007)
> > > [ 1661.285546] thunderbolt 0000:03:00.0: NHI initialized, starting thunderbolt
> > > [ 1661.285550] thunderbolt 0000:03:00.0: allocating TX ring 0 of size 10
> > > [ 1661.285565] thunderbolt 0000:03:00.0: allocating RX ring 0 of size 10
> > > [ 1661.285582] thunderbolt 0000:03:00.0: control channel created
> > > [ 1661.285583] thunderbolt 0000:03:00.0: control channel starting...
> > > [ 1661.285584] thunderbolt 0000:03:00.0: starting TX ring 0
> > > [ 1661.285591] thunderbolt 0000:03:00.0: enabling interrupt at register 0x38200
> > bit 0 (0x0 -> 0x1)
> > > [ 1661.285592] thunderbolt 0000:03:00.0: starting RX ring 0
> > > [ 1661.285599] thunderbolt 0000:03:00.0: enabling interrupt at register 0x38200
> > bit 12 (0x1 -> 0x1001)
> > > [ 1661.731122] ACPI Error: Cannot release Mutex [PATM], not acquired
> > (20170303/exmutex-393)
> > > [ 1661.731143] ACPI Error: Method parse/execution failed
> > [\_SB.PCI0.LPCB.ECDV._Q66] (Node ffff8ba73017bf00),
> > AE_AML_MUTEX_NOT_ACQUIRED (20170303/psparse-543)
> > > [ 1662.538307] thunderbolt 0000:03:00.0: current switch config:
> > > [ 1662.538315] thunderbolt 0000:03:00.0:  Switch: 8086:1576 (Revision: 4, TB
> > Version: 2)
> > > [ 1662.538319] thunderbolt 0000:03:00.0:   Max Port Number: 11
> > > [ 1662.538322] thunderbolt 0000:03:00.0:   Config:
> > > [ 1662.538329] thunderbolt 0000:03:00.0:    Upstream Port Number: 5 Depth: 0
> > Route String: 0x0 Enabled: 1, PlugEventsDelay: 254ms
> > > [ 1662.538333] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
> > > [ 1662.561266] thunderbolt 0000:03:00.0: 0: uid: 0x8086a3422131ea10
> > > [ 1662.561856] thunderbolt 0000:03:00.0:  Port 0: 8086:1576 (Revision: 4, TB
> > Version: 1, Type: Port (0x1))
> > > [ 1662.561860] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
> > > [ 1662.561864] thunderbolt 0000:03:00.0:   Max counters: 8
> > > [ 1662.561868] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> > > [ 1662.562078] thunderbolt 0000:03:00.0:  Port 1: 8086:1576 (Revision: 4, TB
> > Version: 1, Type: Port (0x1))
> > > [ 1662.562082] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > > [ 1662.562086] thunderbolt 0000:03:00.0:   Max counters: 16
> > > [ 1662.562090] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
> > > [ 1662.562323] thunderbolt 0000:03:00.0:  Port 2: 8086:1576 (Revision: 4, TB
> > Version: 1, Type: Port (0x1))
> > > [ 1662.562328] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > > [ 1662.562331] thunderbolt 0000:03:00.0:   Max counters: 16
> > > [ 1662.562335] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
> > > [ 1662.562568] thunderbolt 0000:03:00.0:  Port 3: 8086:1576 (Revision: 4, TB
> > Version: 1, Type: Port (0x1))
> > > [ 1662.562575] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > > [ 1662.562579] thunderbolt 0000:03:00.0:   Max counters: 16
> > > [ 1662.562583] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
> > > [ 1662.562797] thunderbolt 0000:03:00.0:  Port 4: 8086:1576 (Revision: 4, TB
> > Version: 1, Type: Port (0x1))
> > > [ 1662.562801] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > > [ 1662.562805] thunderbolt 0000:03:00.0:   Max counters: 16
> > > [ 1662.562808] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
> > > [ 1662.562813] thunderbolt 0000:03:00.0: 0:5: disabled by eeprom
> > > [ 1662.562873] thunderbolt 0000:03:00.0:  Port 6: 8086:1576 (Revision: 4, TB
> > Version: 1, Type: PCIe (0x100101))
> > > [ 1662.562877] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> > > [ 1662.562880] thunderbolt 0000:03:00.0:   Max counters: 2
> > > [ 1662.562884] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> > > [ 1662.562938] thunderbolt 0000:03:00.0:  Port 7: 8086:1576 (Revision: 4, TB
> > Version: 1, Type: PCIe (0x100101))
> > > [ 1662.562942] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> > > [ 1662.562945] thunderbolt 0000:03:00.0:   Max counters: 2
> > > [ 1662.562949] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> > > [ 1662.563005] thunderbolt 0000:03:00.0:  Port 8: 8086:1576 (Revision: 4, TB
> > Version: 1, Type: DP/HDMI (0xe0102))
> > > [ 1662.563009] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
> > > [ 1662.563012] thunderbolt 0000:03:00.0:   Max counters: 2
> > > [ 1662.563016] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> > > [ 1662.563019] thunderbolt 0000:03:00.0: 0:9: disabled by eeprom
> > > [ 1662.563082] thunderbolt 0000:03:00.0:  Port 10: 8086:1576 (Revision: 4, TB
> > Version: 1, Type: DP/HDMI (0xe0101))
> > > [ 1662.563085] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
> > > [ 1662.563089] thunderbolt 0000:03:00.0:   Max counters: 2
> > > [ 1662.563092] thunderbolt 0000:03:00.0:   NFC Credits: 0x1000000
> > > [ 1662.563096] thunderbolt 0000:03:00.0: 0:b: disabled by eeprom
> > > [ 1662.564973] thunderbolt 0000:03:00.0: current switch config:
> > > [ 1662.564983] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB
> > Version: 2)
> > > [ 1662.564989] thunderbolt 0000:03:00.0:   Max Port Number: 11
> > > [ 1662.564994] thunderbolt 0000:03:00.0:   Config:
> > > [ 1662.565004] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 1
> > Route String: 0x1 Enabled: 1, PlugEventsDelay: 254ms
> > > [ 1662.565011] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
> > > [ 1662.586185] thunderbolt 0000:03:00.0: 1: reading drom (length: 0x6e)
> > > [ 1662.685656] thunderbolt 0000:03:00.0: 1: drom data crc32 mismatch
> > (expected: 0xaf438340, got: 0xaf4383c0), continuing
> > > [ 1662.685923] thunderbolt 0000:03:00.0: 1: uid: 0xd40f7a7928c300
> > > [ 1662.685970] thunderbolt 0000:03:00.0:  Port 0: 8086:1578 (Revision: 4, TB
> > Version: 1, Type: Port (0x1))
> > > [ 1662.685972] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
> > > [ 1662.685974] thunderbolt 0000:03:00.0:   Max counters: 8
> > > [ 1662.685976] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> > > [ 1662.686163] thunderbolt 0000:03:00.0:  Port 1: 8086:1578 (Revision: 4, TB
> > Version: 1, Type: Port (0x1))
> > > [ 1662.686165] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > > [ 1662.686167] thunderbolt 0000:03:00.0:   Max counters: 16
> > > [ 1662.686169] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
> > > [ 1662.686346] thunderbolt 0000:03:00.0:  Port 2: 8086:1578 (Revision: 4, TB
> > Version: 1, Type: Port (0x1))
> > > [ 1662.686348] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > > [ 1662.686350] thunderbolt 0000:03:00.0:   Max counters: 16
> > > [ 1662.686351] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
> > > [ 1662.686520] thunderbolt 0000:03:00.0:  Port 3: 8086:1578 (Revision: 4, TB
> > Version: 1, Type: Port (0x1))
> > > [ 1662.686522] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > > [ 1662.686523] thunderbolt 0000:03:00.0:   Max counters: 16
> > > [ 1662.686525] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
> > > [ 1662.686697] thunderbolt 0000:03:00.0:  Port 4: 8086:1578 (Revision: 4, TB
> > Version: 1, Type: Port (0x1))
> > > [ 1662.686698] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
> > > [ 1662.686700] thunderbolt 0000:03:00.0:   Max counters: 16
> > > [ 1662.686702] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
> > > [ 1662.686704] thunderbolt 0000:03:00.0: 1:5: disabled by eeprom
> > > [ 1662.686757] thunderbolt 0000:03:00.0:  Port 6: 8086:1578 (Revision: 4, TB
> > Version: 1, Type: PCIe (0x100102))
> > > [ 1662.686758] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> > > [ 1662.686760] thunderbolt 0000:03:00.0:   Max counters: 2
> > > [ 1662.686762] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> > > [ 1662.686816] thunderbolt 0000:03:00.0:  Port 7: 8086:1578 (Revision: 4, TB
> > Version: 1, Type: PCIe (0x100101))
> > > [ 1662.686818] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
> > > [ 1662.686820] thunderbolt 0000:03:00.0:   Max counters: 2
> > > [ 1662.686822] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
> > > [ 1662.686825] thunderbolt 0000:03:00.0: 1:8: disabled by eeprom
> > > [ 1662.686827] thunderbolt 0000:03:00.0: 1:9: disabled by eeprom
> > > [ 1662.686830] thunderbolt 0000:03:00.0: 1:a: disabled by eeprom
> > > [ 1662.686832] thunderbolt 0000:03:00.0: 1:b: disabled by eeprom
> > > [ 1662.688262] thunderbolt 0000:03:00.0: current switch config:
> > > [ 1662.688271] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB
> > Version: 2)
> > > [ 1662.688273] thunderbolt 0000:03:00.0:   Max Port Number: 11
> > > [ 1662.688275] thunderbolt 0000:03:00.0:   Config:
> > > [ 1662.688284] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 2
> > Route String: 0x301 Enabled: 1, PlugEventsDelay: 254ms
> > > [ 1662.688287] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
> > > [ 1666.261728] pci 0000:04:00.0: [8086:1578] type 01 class 0x060400
> > > [ 1666.262033] pci 0000:04:00.0: supports D1 D2
> > > [ 1666.262034] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> > > [ 1666.262442] sysfs: cannot create duplicate filename '/class/pci_bus/0000:05'
> > 
> > This is probably due the previous problem of the PCI device being hidden
> > behind a bridge.
> > 
> > If you shutdown the machine and boot it without devices connected and
> > then when the OS is up, connect the device, do you see the same issue?
> > 
> 
> The kernel traceback doesn't happen in this scenario, but if I follow these steps:
> 1) Boot
> 2) Plugin dock, make sure fully enumerated
> 3) Unplug
> 4) Replug
> 
> I get the trackeback warning about duplicate filenames again.

Can you send me full dmesg when you do the above steps?

> The second time I unplug I also get tracebacks related to a NULL pointer dereference
> In kernfs_find_ns.

Yes, it starts to break down when the PCI device is left there and tried
to be added back after hotplug.

RE: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Levy, Amir (Jer)]

On Fri, May 19 2017, 07:35 PM, Mario.Limonciello@dell.com wrote:
> Here's my setup:
> System: I'm using is an XPS 9350 (Has Alpine Ridge).  It's got NVM 16.0.  BIOS
> 1.4.13 TBT Device: Dell TB16 (which has AR in the cable and in dock - both
> NVM 16.0).
> 

Is it BIOS assist or native enumeration?

> I created a udev rule that will automatically authorize the dock and cable.
> #dell cable
> ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0",
> ATTR{vendor}=="0xd4", ATTR{device}=="0xb051", ATTR{authorized}="1"
> #dell dock
> ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0",
> ATTR{vendor}=="0xd4", ATTR{device}=="0xb054", ATTR{authorized}="1"
> 

Note that the udev rule should authorize the cable first and then the dock.

> If I boot the system with the dock connected the cable shows up and authorizes
> but the dock doesn't.

I assume it works in Linux with SL0, right?

Re: [PATCH 18/24] thunderbolt: Store Thunderbolt generation in the switch structure

[Lukas Wunner]

On Thu, May 18, 2017 at 05:39:08PM +0300, Mika Westerberg wrote:
> In some cases it is useful to know what is the Thunderbolt generation
> the switch supports. This introduces a new field to struct switch that
> stores the generation of the switch based on the device ID.
> 
> Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
> Reviewed-by: Michael Jamet <michael.jamet@intel.com>
> ---
>  drivers/thunderbolt/switch.c | 24 ++++++++++++++++++++++++
>  drivers/thunderbolt/tb.h     |  2 ++
>  2 files changed, 26 insertions(+)
> 
> diff --git a/drivers/thunderbolt/switch.c b/drivers/thunderbolt/switch.c
> index 396e00ab7723..9c91d397d3b3 100644
> --- a/drivers/thunderbolt/switch.c
> +++ b/drivers/thunderbolt/switch.c
> @@ -382,6 +382,28 @@ struct device_type tb_switch_type = {
>  	.release = tb_switch_release,
>  };
>  
> +static void tb_switch_set_generation(struct tb_switch *sw)
> +{
> +	switch (sw->config.device_id) {
> +	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_LP_BRIDGE:
> +	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_2C_BRIDGE:
> +	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_4C_BRIDGE:
> +	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_2C_BRIDGE:
> +	case PCI_DEVICE_ID_INTEL_ALPINE_RIDGE_C_4C_BRIDGE:
> +		sw->generation = 3;
> +		break;
> +
> +	case PCI_DEVICE_ID_INTEL_FALCON_RIDGE_2C_BRIDGE:
> +	case PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_BRIDGE:
> +		sw->generation = 2;
> +		break;

This is missing PCI_DEVICE_ID_INTEL_WIN_RIDGE_2C_BRIDGE (0x157e).


> +
> +	default:
> +		sw->generation = 1;
> +		break;

If someone adds an entry for, say, a new TB3 controller to nhi_ids[]
but forgets to update this function, the controller is assigned the
wrong generation number.  It might be better to make TB3 the default
and list each TB1 controller instead since it's less likely for Intel
to introduce an older gen chip.

Generally I think it's problematic to require that multiple files
are touched whenever a new controller is added.  Isn't the generation
number or link speed (10/20/40) stored in some register in PCI config
space (VSEC 0x1234) or TB config space?

Thanks,

Lukas

> +	}
> +}
> +
>  /**
>   * tb_switch_alloc() - allocate a switch
>   * @tb: Pointer to the owning domain
> @@ -442,6 +464,8 @@ struct tb_switch *tb_switch_alloc(struct tb *tb, struct device *parent,
>  	}
>  	sw->cap_plug_events = cap;
>  
> +	tb_switch_set_generation(sw);
> +
>  	device_initialize(&sw->dev);
>  	sw->dev.parent = parent;
>  	sw->dev.bus = &tb_bus_type;
> diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
> index 0be989069941..b3cda7605619 100644
> --- a/drivers/thunderbolt/tb.h
> +++ b/drivers/thunderbolt/tb.h
> @@ -25,6 +25,7 @@
>   * @device: Device ID of the switch
>   * @vendor_name: Name of the vendor (or %NULL if not known)
>   * @device_name: Name of the device (or %NULL if not known)
> + * @generation: Switch Thunderbolt generation
>   * @cap_plug_events: Offset to the plug events capability (%0 if not found)
>   * @is_unplugged: The switch is going away
>   * @drom: DROM of the switch (%NULL if not found)
> @@ -40,6 +41,7 @@ struct tb_switch {
>  	u16 device;
>  	const char *vendor_name;
>  	const char *device_name;
> +	unsigned int generation;
>  	int cap_plug_events;
>  	bool is_unplugged;
>  	u8 *drom;
> -- 
> 2.11.0
> 

RE: [PATCH 18/24] thunderbolt: Store Thunderbolt generation in the switch structure

[Levy, Amir (Jer)]

On Sun, May 21 2017, 07:47 AM, Lukas Wunner wrote:
> On Thu, May 18, 2017 at 05:39:08PM +0300, Mika Westerberg wrote:
> > +
> > +	default:
> > +		sw->generation = 1;
> > +		break;
> 
> If someone adds an entry for, say, a new TB3 controller to nhi_ids[] but forgets
> to update this function, the controller is assigned the wrong generation
> number.  It might be better to make TB3 the default and list each TB1
> controller instead since it's less likely for Intel to introduce an older gen chip.
> 
> Generally I think it's problematic to require that multiple files are touched
> whenever a new controller is added.  Isn't the generation number or link speed
> (10/20/40) stored in some register in PCI config space (VSEC 0x1234) or TB
> config space?
> 
> Thanks,
> 
> Lukas
> 

How about setting information, that isn't available from PCI, in pci_device_id.driver_data when initializing nhi_ids[]?

Re: [PATCH 10/24] thunderbolt: Read vendor and device name from DROM

[Lukas Wunner]

On Fri, May 19, 2017 at 01:28:36PM +0300, Mika Westerberg wrote:
> On Fri, May 19, 2017 at 12:07:10PM +0200, Lukas Wunner wrote:
> > Apple uses 0x30 to store a
> > serial number.  Is this attribute number assigned by Intel to Apple
> > or is it reserved for vendor use or did they arbitrarily choose it?
> 
> It is part of the DROM specification. The 0x30 - 0x3e are vendor
> specific entries.

Ah, so I have to qualify the vendor number with Apple's ID before I know
that it's a serial number.  Thanks.


> > If there can be many attributes, should they be stored in a list
> > rather than adding a char* pointer for each one to struct tb_switch?
> > The latter doesn't scale.
> 
> I don't think we need other attributes (well, at least right now). The
> device/vendor name is useful because that's what we expose to the
> userspace for device identification along with the device/vendor ID.

Okay.  It might be worth to log additional attributes with info level.


> > > +static void tb_drom_parse_generic_entry(struct tb_switch *sw,
> > > +		struct tb_drom_entry_generic *entry)
> > > +{
> > > +	if (entry->header.index == 1)
> > > +		sw->vendor_name = kstrdup((char *)entry->data, GFP_KERNEL);
> > > +	else if (entry->header.index == 2)
> > > +		sw->device_name = kstrdup((char *)entry->data, GFP_KERNEL);
> > > +}
> > 
> > This assumes that these are properly null-terminated strings, but the DROM
> > may contain complete garbage.  The existing drom parser is very careful
> > to validate and sanitize everything.
> 
> The DROM specification says they must be null-terminated but I yes, it
> is possible that some of the devices have it wrong. The generic entry
> includes length field so I suppose we can use that + kmemdup() instead
> here?

Yes, as long as you check that the last character is null.

Thanks,

Lukas

Re: [PATCH 18/24] thunderbolt: Store Thunderbolt generation in the switch structure

[Lukas Wunner]

On Sun, May 21, 2017 at 05:29:47AM +0000, Levy, Amir (Jer) wrote:
> On Sun, May 21 2017, 07:47 AM, Lukas Wunner wrote:
> > On Thu, May 18, 2017 at 05:39:08PM +0300, Mika Westerberg wrote:
> > > +
> > > +	default:
> > > +		sw->generation = 1;
> > > +		break;
> > 
> > If someone adds an entry for, say, a new TB3 controller to nhi_ids[] but
> > forgets to update this function, the controller is assigned the wrong
> > generation number.  It might be better to make TB3 the default and list
> > each TB1 controller instead since it's less likely for Intel to introduce
> > an older gen chip.
> > 
> > Generally I think it's problematic to require that multiple files are
> > touched whenever a new controller is added.  Isn't the generation number
> > or link speed (10/20/40) stored in some register in PCI config space
> > (VSEC 0x1234) or TB config space?
> 
> How about setting information, that isn't available from PCI, in
> pci_device_id.driver_data when initializing nhi_ids[]?

Right, that would also be possible, though reading the generation number
from a register would be more elegant, if such a register exists.

Thanks,

Lukas

Re: [PATCH 18/24] thunderbolt: Store Thunderbolt generation in the switch structure

[Mika Westerberg]

On Sun, May 21, 2017 at 07:35:21AM +0200, Lukas Wunner wrote:
> On Sun, May 21, 2017 at 05:29:47AM +0000, Levy, Amir (Jer) wrote:
> > On Sun, May 21 2017, 07:47 AM, Lukas Wunner wrote:
> > > On Thu, May 18, 2017 at 05:39:08PM +0300, Mika Westerberg wrote:
> > > > +
> > > > +	default:
> > > > +		sw->generation = 1;
> > > > +		break;
> > > 
> > > If someone adds an entry for, say, a new TB3 controller to nhi_ids[] but
> > > forgets to update this function, the controller is assigned the wrong
> > > generation number.  It might be better to make TB3 the default and list
> > > each TB1 controller instead since it's less likely for Intel to introduce
> > > an older gen chip.
> > > 
> > > Generally I think it's problematic to require that multiple files are
> > > touched whenever a new controller is added.  Isn't the generation number
> > > or link speed (10/20/40) stored in some register in PCI config space
> > > (VSEC 0x1234) or TB config space?
> > 
> > How about setting information, that isn't available from PCI, in
> > pci_device_id.driver_data when initializing nhi_ids[]?
> 
> Right, that would also be possible, though reading the generation number
> from a register would be more elegant, if such a register exists.

I don't think there is such register but I can put this information to
the driver_data instead.

Re: [PATCH 10/24] thunderbolt: Read vendor and device name from DROM

[Mika Westerberg]

On Sun, May 21, 2017 at 07:31:14AM +0200, Lukas Wunner wrote:
> On Fri, May 19, 2017 at 01:28:36PM +0300, Mika Westerberg wrote:
> > On Fri, May 19, 2017 at 12:07:10PM +0200, Lukas Wunner wrote:
> > > Apple uses 0x30 to store a
> > > serial number.  Is this attribute number assigned by Intel to Apple
> > > or is it reserved for vendor use or did they arbitrarily choose it?
> > 
> > It is part of the DROM specification. The 0x30 - 0x3e are vendor
> > specific entries.
> 
> Ah, so I have to qualify the vendor number with Apple's ID before I know
> that it's a serial number.  Thanks.

Yes, something like that works.

> > > If there can be many attributes, should they be stored in a list
> > > rather than adding a char* pointer for each one to struct tb_switch?
> > > The latter doesn't scale.
> > 
> > I don't think we need other attributes (well, at least right now). The
> > device/vendor name is useful because that's what we expose to the
> > userspace for device identification along with the device/vendor ID.
> 
> Okay.  It might be worth to log additional attributes with info level.

I don't think we want to log anything with info level to be honest. The
driver currently already is pretty noisy so adding even more information
there just makes it worse ;-)

I would rather convert debugging information to use tracepoints and get
rid of the tb_*_info() things completely.

The whole DROM content is already available through nvm_active/nvmem
file under each device (well starting with Alpine Ridge) so the
userspace can investigate it as much as it likes without spamming the
kernel dmesg :)

> > > > +static void tb_drom_parse_generic_entry(struct tb_switch *sw,
> > > > +		struct tb_drom_entry_generic *entry)
> > > > +{
> > > > +	if (entry->header.index == 1)
> > > > +		sw->vendor_name = kstrdup((char *)entry->data, GFP_KERNEL);
> > > > +	else if (entry->header.index == 2)
> > > > +		sw->device_name = kstrdup((char *)entry->data, GFP_KERNEL);
> > > > +}
> > > 
> > > This assumes that these are properly null-terminated strings, but the DROM
> > > may contain complete garbage.  The existing drom parser is very careful
> > > to validate and sanitize everything.
> > 
> > The DROM specification says they must be null-terminated but I yes, it
> > is possible that some of the devices have it wrong. The generic entry
> > includes length field so I suppose we can use that + kmemdup() instead
> > here?
> 
> Yes, as long as you check that the last character is null.

OK, I'll do that then. Thanks.

Re: [PATCH 18/24] thunderbolt: Store Thunderbolt generation in the switch structure

[Mika Westerberg]

On Sun, May 21, 2017 at 10:40:41AM +0300, Mika Westerberg wrote:
> On Sun, May 21, 2017 at 07:35:21AM +0200, Lukas Wunner wrote:
> > On Sun, May 21, 2017 at 05:29:47AM +0000, Levy, Amir (Jer) wrote:
> > > On Sun, May 21 2017, 07:47 AM, Lukas Wunner wrote:
> > > > On Thu, May 18, 2017 at 05:39:08PM +0300, Mika Westerberg wrote:
> > > > > +
> > > > > +	default:
> > > > > +		sw->generation = 1;
> > > > > +		break;
> > > > 
> > > > If someone adds an entry for, say, a new TB3 controller to nhi_ids[] but
> > > > forgets to update this function, the controller is assigned the wrong
> > > > generation number.  It might be better to make TB3 the default and list
> > > > each TB1 controller instead since it's less likely for Intel to introduce
> > > > an older gen chip.
> > > > 
> > > > Generally I think it's problematic to require that multiple files are
> > > > touched whenever a new controller is added.  Isn't the generation number
> > > > or link speed (10/20/40) stored in some register in PCI config space
> > > > (VSEC 0x1234) or TB config space?
> > > 
> > > How about setting information, that isn't available from PCI, in
> > > pci_device_id.driver_data when initializing nhi_ids[]?
> > 
> > Right, that would also be possible, though reading the generation number
> > from a register would be more elegant, if such a register exists.
> 
> I don't think there is such register but I can put this information to
> the driver_data instead.

Actually these are Thunderbolt switch IDs, not NHI PCI IDs so I don't
think driver_data is the right place after all. So if no objections,
I'll update the function to default to TBT3 but keep the switch case and
add the TBT1 IDs + Win Ridge there.

RE: [PATCH 18/24] thunderbolt: Store Thunderbolt generation in the switch structure

[Levy, Amir (Jer)]

On Sun, May 21 2017, 11:00 AM, Mika Westerberg wrote:
> On Sun, May 21, 2017 at 10:40:41AM +0300, Mika Westerberg wrote:
> > On Sun, May 21, 2017 at 07:35:21AM +0200, Lukas Wunner wrote:
> > > On Sun, May 21, 2017 at 05:29:47AM +0000, Levy, Amir (Jer) wrote:
> > > > On Sun, May 21 2017, 07:47 AM, Lukas Wunner wrote:
> > > > > On Thu, May 18, 2017 at 05:39:08PM +0300, Mika Westerberg wrote:
> > > > > > +
> > > > > > +	default:
> > > > > > +		sw->generation = 1;
> > > > > > +		break;
> > > > >
> > > > > If someone adds an entry for, say, a new TB3 controller to
> > > > > nhi_ids[] but forgets to update this function, the controller is
> > > > > assigned the wrong generation number.  It might be better to
> > > > > make TB3 the default and list each TB1 controller instead since
> > > > > it's less likely for Intel to introduce an older gen chip.
> > > > >
> > > > > Generally I think it's problematic to require that multiple
> > > > > files are touched whenever a new controller is added.  Isn't the
> > > > > generation number or link speed (10/20/40) stored in some
> > > > > register in PCI config space (VSEC 0x1234) or TB config space?
> > > >
> > > > How about setting information, that isn't available from PCI, in
> > > > pci_device_id.driver_data when initializing nhi_ids[]?
> > >
> > > Right, that would also be possible, though reading the generation
> > > number from a register would be more elegant, if such a register exists.
> >
> > I don't think there is such register but I can put this information to
> > the driver_data instead.
> 
> Actually these are Thunderbolt switch IDs, not NHI PCI IDs so I don't think
> driver_data is the right place after all. So if no objections, I'll update the
> function to default to TBT3 but keep the switch case and add the TBT1 IDs +
> Win Ridge there.

There is correlation between switch ID to NHI ID.

Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[mika.westerberg]

On Sat, May 20, 2017 at 09:15:17AM +0000, Levy, Amir (Jer) wrote:
> > I created a udev rule that will automatically authorize the dock and cable.
> > #dell cable
> > ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0",
> > ATTR{vendor}=="0xd4", ATTR{device}=="0xb051", ATTR{authorized}="1"
> > #dell dock
> > ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0",
> > ATTR{vendor}=="0xd4", ATTR{device}=="0xb054", ATTR{authorized}="1"
> > 
> 
> Note that the udev rule should authorize the cable first and then the dock.

That should be fine, the devices appear in order closest to the host and
get added to the system in that order so udev should see them in that
order as well. Also the cable device will be parent to the dock.

Re: [PATCH 10/24] thunderbolt: Read vendor and device name from DROM

[Lukas Wunner]

On Sun, May 21, 2017 at 10:48:19AM +0300, Mika Westerberg wrote:
> On Sun, May 21, 2017 at 07:31:14AM +0200, Lukas Wunner wrote:
> > On Fri, May 19, 2017 at 01:28:36PM +0300, Mika Westerberg wrote:
> > > On Fri, May 19, 2017 at 12:07:10PM +0200, Lukas Wunner wrote:
> > > > If there can be many attributes, should they be stored in a list
> > > > rather than adding a char* pointer for each one to struct tb_switch?
> > > > The latter doesn't scale.
> > > 
> > > I don't think we need other attributes (well, at least right now). The
> > > device/vendor name is useful because that's what we expose to the
> > > userspace for device identification along with the device/vendor ID.
> > 
> > Okay.  It might be worth to log additional attributes with info level.
> 
> I don't think we want to log anything with info level to be honest. The
> driver currently already is pretty noisy so adding even more information
> there just makes it worse ;-)
> 
> I would rather convert debugging information to use tracepoints and get
> rid of the tb_*_info() things completely.

The noisiness has value in that it helps with reverse-engineering:
Just google for dmesg output and check what other machines are
reporting for unknown registers. :-)

If there was public documentation available or Intel would be okay
with answering specific questions (as you've done with the 0x30
attribute id), then the value obviously diminishes.

Can't say anything about converting to tracepoints, that's Andreas'
call.


> The whole DROM content is already available through nvm_active/nvmem
> file under each device (well starting with Alpine Ridge) so the
> userspace can investigate it as much as it likes without spamming the
> kernel dmesg :)

Okay, fair enough, that should indeed suffice.

Thanks,

Lukas

RE: [PATCH 18/24] thunderbolt: Store Thunderbolt generation in the switch structure

[Bernat, Yehezkel]

> -----Original Message-----
> From: Levy, Amir (Jer)
> Sent: Sunday, May 21, 2017 11:07
> To: Mika Westerberg <mika.westerberg@linux.intel.com>; Lukas Wunner
> <lukas@wunner.de>
> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>; Andreas Noever
> <andreas.noever@gmail.com>; Jamet, Michael <michael.jamet@intel.com>;
> Bernat, Yehezkel <yehezkel.bernat@intel.com>; Andy Lutomirski
> <luto@kernel.org>; Mario.Limonciello@dell.com;
> Jared.Dominguez@dell.com; Andy Shevchenko
> <andriy.shevchenko@linux.intel.com>; linux-kernel@vger.kernel.org
> Subject: RE: [PATCH 18/24] thunderbolt: Store Thunderbolt generation in the
> switch structure
> 
> On Sun, May 21 2017, 11:00 AM, Mika Westerberg wrote:
> > On Sun, May 21, 2017 at 10:40:41AM +0300, Mika Westerberg wrote:
> > > On Sun, May 21, 2017 at 07:35:21AM +0200, Lukas Wunner wrote:
> > > > On Sun, May 21, 2017 at 05:29:47AM +0000, Levy, Amir (Jer) wrote:
> > > > > On Sun, May 21 2017, 07:47 AM, Lukas Wunner wrote:
> > > > > > On Thu, May 18, 2017 at 05:39:08PM +0300, Mika Westerberg wrote:
> > > > > > > +
> > > > > > > +	default:
> > > > > > > +		sw->generation = 1;
> > > > > > > +		break;
> > > > > >
> > > > > > If someone adds an entry for, say, a new TB3 controller to
> > > > > > nhi_ids[] but forgets to update this function, the controller
> > > > > > is assigned the wrong generation number.  It might be better
> > > > > > to make TB3 the default and list each TB1 controller instead
> > > > > > since it's less likely for Intel to introduce an older gen chip.
> > > > > >
> > > > > > Generally I think it's problematic to require that multiple
> > > > > > files are touched whenever a new controller is added.  Isn't
> > > > > > the generation number or link speed (10/20/40) stored in some
> > > > > > register in PCI config space (VSEC 0x1234) or TB config space?
> > > > >
> > > > > How about setting information, that isn't available from PCI, in
> > > > > pci_device_id.driver_data when initializing nhi_ids[]?
> > > >
> > > > Right, that would also be possible, though reading the generation
> > > > number from a register would be more elegant, if such a register exists.
> > >
> > > I don't think there is such register but I can put this information
> > > to the driver_data instead.
> >
> > Actually these are Thunderbolt switch IDs, not NHI PCI IDs so I don't
> > think driver_data is the right place after all. So if no objections,
> > I'll update the function to default to TBT3 but keep the switch case
> > and add the TBT1 IDs + Win Ridge there.
> 
> There is correlation between switch ID to NHI ID.

I'm not sure defaulting to TBT3 is a good idea. ICM message format can be changed, DMA port can be different, nothing guaranties correct operation of the driver with newer unknown controllers.

Re: [PATCH 18/24] thunderbolt: Store Thunderbolt generation in the switch structure

[Mika Westerberg]

On Sun, May 21, 2017 at 08:07:08AM +0000, Levy, Amir (Jer) wrote:
> On Sun, May 21 2017, 11:00 AM, Mika Westerberg wrote:
> > On Sun, May 21, 2017 at 10:40:41AM +0300, Mika Westerberg wrote:
> > > On Sun, May 21, 2017 at 07:35:21AM +0200, Lukas Wunner wrote:
> > > > On Sun, May 21, 2017 at 05:29:47AM +0000, Levy, Amir (Jer) wrote:
> > > > > On Sun, May 21 2017, 07:47 AM, Lukas Wunner wrote:
> > > > > > On Thu, May 18, 2017 at 05:39:08PM +0300, Mika Westerberg wrote:
> > > > > > > +
> > > > > > > +	default:
> > > > > > > +		sw->generation = 1;
> > > > > > > +		break;
> > > > > >
> > > > > > If someone adds an entry for, say, a new TB3 controller to
> > > > > > nhi_ids[] but forgets to update this function, the controller is
> > > > > > assigned the wrong generation number.  It might be better to
> > > > > > make TB3 the default and list each TB1 controller instead since
> > > > > > it's less likely for Intel to introduce an older gen chip.
> > > > > >
> > > > > > Generally I think it's problematic to require that multiple
> > > > > > files are touched whenever a new controller is added.  Isn't the
> > > > > > generation number or link speed (10/20/40) stored in some
> > > > > > register in PCI config space (VSEC 0x1234) or TB config space?
> > > > >
> > > > > How about setting information, that isn't available from PCI, in
> > > > > pci_device_id.driver_data when initializing nhi_ids[]?
> > > >
> > > > Right, that would also be possible, though reading the generation
> > > > number from a register would be more elegant, if such a register exists.
> > >
> > > I don't think there is such register but I can put this information to
> > > the driver_data instead.
> > 
> > Actually these are Thunderbolt switch IDs, not NHI PCI IDs so I don't think
> > driver_data is the right place after all. So if no objections, I'll update the
> > function to default to TBT3 but keep the switch case and add the TBT1 IDs +
> > Win Ridge there.
> 
> There is correlation between switch ID to NHI ID.

Indeed but what if you have a device with a Port Ridge TBT controller
(switch) connected to the host? Here we want to get the generation of
whatever Thunderbolt switch, not just root switch (where NHI ID would
indeed suffice).

Re: [PATCH 18/24] thunderbolt: Store Thunderbolt generation in the switch structure

[Mika Westerberg]

On Sun, May 21, 2017 at 09:55:55AM +0000, Bernat, Yehezkel wrote:
> 
> 
> > -----Original Message-----
> > From: Levy, Amir (Jer)
> > Sent: Sunday, May 21, 2017 11:07
> > To: Mika Westerberg <mika.westerberg@linux.intel.com>; Lukas Wunner
> > <lukas@wunner.de>
> > Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>; Andreas Noever
> > <andreas.noever@gmail.com>; Jamet, Michael <michael.jamet@intel.com>;
> > Bernat, Yehezkel <yehezkel.bernat@intel.com>; Andy Lutomirski
> > <luto@kernel.org>; Mario.Limonciello@dell.com;
> > Jared.Dominguez@dell.com; Andy Shevchenko
> > <andriy.shevchenko@linux.intel.com>; linux-kernel@vger.kernel.org
> > Subject: RE: [PATCH 18/24] thunderbolt: Store Thunderbolt generation in the
> > switch structure
> > 
> > On Sun, May 21 2017, 11:00 AM, Mika Westerberg wrote:
> > > On Sun, May 21, 2017 at 10:40:41AM +0300, Mika Westerberg wrote:
> > > > On Sun, May 21, 2017 at 07:35:21AM +0200, Lukas Wunner wrote:
> > > > > On Sun, May 21, 2017 at 05:29:47AM +0000, Levy, Amir (Jer) wrote:
> > > > > > On Sun, May 21 2017, 07:47 AM, Lukas Wunner wrote:
> > > > > > > On Thu, May 18, 2017 at 05:39:08PM +0300, Mika Westerberg wrote:
> > > > > > > > +
> > > > > > > > +	default:
> > > > > > > > +		sw->generation = 1;
> > > > > > > > +		break;
> > > > > > >
> > > > > > > If someone adds an entry for, say, a new TB3 controller to
> > > > > > > nhi_ids[] but forgets to update this function, the controller
> > > > > > > is assigned the wrong generation number.  It might be better
> > > > > > > to make TB3 the default and list each TB1 controller instead
> > > > > > > since it's less likely for Intel to introduce an older gen chip.
> > > > > > >
> > > > > > > Generally I think it's problematic to require that multiple
> > > > > > > files are touched whenever a new controller is added.  Isn't
> > > > > > > the generation number or link speed (10/20/40) stored in some
> > > > > > > register in PCI config space (VSEC 0x1234) or TB config space?
> > > > > >
> > > > > > How about setting information, that isn't available from PCI, in
> > > > > > pci_device_id.driver_data when initializing nhi_ids[]?
> > > > >
> > > > > Right, that would also be possible, though reading the generation
> > > > > number from a register would be more elegant, if such a register exists.
> > > >
> > > > I don't think there is such register but I can put this information
> > > > to the driver_data instead.
> > >
> > > Actually these are Thunderbolt switch IDs, not NHI PCI IDs so I don't
> > > think driver_data is the right place after all. So if no objections,
> > > I'll update the function to default to TBT3 but keep the switch case
> > > and add the TBT1 IDs + Win Ridge there.
> > 
> > There is correlation between switch ID to NHI ID.
> 
> I'm not sure defaulting to TBT3 is a good idea. ICM message format can
> be changed, DMA port can be different, nothing guaranties correct
> operation of the driver with newer unknown controllers.

Fair enough :) Then we just need to remember to update the function here
as new generations get added and tested.

I suppose you don't know either if we could use the revision or similar
field in the switch config space to determine generation somehow?

RE: [PATCH 18/24] thunderbolt: Store Thunderbolt generation in the switch structure

[Bernat, Yehezkel]

> -----Original Message-----
> From: Mika Westerberg [mailto:mika.westerberg@linux.intel.com]
> Sent: Sunday, May 21, 2017 13:47
> To: Bernat, Yehezkel <yehezkel.bernat@intel.com>
> Cc: Levy, Amir (Jer) <amir.jer.levy@intel.com>; Lukas Wunner
> <lukas@wunner.de>; Greg Kroah-Hartman <gregkh@linuxfoundation.org>;
> Andreas Noever <andreas.noever@gmail.com>; Jamet, Michael
> <michael.jamet@intel.com>; Andy Lutomirski <luto@kernel.org>;
> Mario.Limonciello@dell.com; Jared.Dominguez@dell.com; Andy Shevchenko
> <andriy.shevchenko@linux.intel.com>; linux-kernel@vger.kernel.org
> Subject: Re: [PATCH 18/24] thunderbolt: Store Thunderbolt generation in the
> switch structure
> 
> On Sun, May 21, 2017 at 09:55:55AM +0000, Bernat, Yehezkel wrote:
> >
> >
> > > -----Original Message-----
> > > From: Levy, Amir (Jer)
> > > Sent: Sunday, May 21, 2017 11:07
> > > To: Mika Westerberg <mika.westerberg@linux.intel.com>; Lukas Wunner
> > > <lukas@wunner.de>
> > > Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>; Andreas
> Noever
> > > <andreas.noever@gmail.com>; Jamet, Michael
> > > <michael.jamet@intel.com>; Bernat, Yehezkel
> > > <yehezkel.bernat@intel.com>; Andy Lutomirski <luto@kernel.org>;
> > > Mario.Limonciello@dell.com; Jared.Dominguez@dell.com; Andy
> > > Shevchenko <andriy.shevchenko@linux.intel.com>;
> > > linux-kernel@vger.kernel.org
> > > Subject: RE: [PATCH 18/24] thunderbolt: Store Thunderbolt generation
> > > in the switch structure
> > >
> > > On Sun, May 21 2017, 11:00 AM, Mika Westerberg wrote:
> > > > On Sun, May 21, 2017 at 10:40:41AM +0300, Mika Westerberg wrote:
> > > > > On Sun, May 21, 2017 at 07:35:21AM +0200, Lukas Wunner wrote:
> > > > > > On Sun, May 21, 2017 at 05:29:47AM +0000, Levy, Amir (Jer) wrote:
> > > > > > > On Sun, May 21 2017, 07:47 AM, Lukas Wunner wrote:
> > > > > > > > On Thu, May 18, 2017 at 05:39:08PM +0300, Mika Westerberg
> wrote:
> > > > > > > > > +
> > > > > > > > > +	default:
> > > > > > > > > +		sw->generation = 1;
> > > > > > > > > +		break;
> > > > > > > >
> > > > > > > > If someone adds an entry for, say, a new TB3 controller to
> > > > > > > > nhi_ids[] but forgets to update this function, the
> > > > > > > > controller is assigned the wrong generation number.  It
> > > > > > > > might be better to make TB3 the default and list each TB1
> > > > > > > > controller instead since it's less likely for Intel to introduce an
> older gen chip.
> > > > > > > >
> > > > > > > > Generally I think it's problematic to require that
> > > > > > > > multiple files are touched whenever a new controller is
> > > > > > > > added.  Isn't the generation number or link speed
> > > > > > > > (10/20/40) stored in some register in PCI config space (VSEC
> 0x1234) or TB config space?
> > > > > > >
> > > > > > > How about setting information, that isn't available from
> > > > > > > PCI, in pci_device_id.driver_data when initializing nhi_ids[]?
> > > > > >
> > > > > > Right, that would also be possible, though reading the
> > > > > > generation number from a register would be more elegant, if such a
> register exists.
> > > > >
> > > > > I don't think there is such register but I can put this
> > > > > information to the driver_data instead.
> > > >
> > > > Actually these are Thunderbolt switch IDs, not NHI PCI IDs so I
> > > > don't think driver_data is the right place after all. So if no
> > > > objections, I'll update the function to default to TBT3 but keep
> > > > the switch case and add the TBT1 IDs + Win Ridge there.
> > >
> > > There is correlation between switch ID to NHI ID.
> >
> > I'm not sure defaulting to TBT3 is a good idea. ICM message format can
> > be changed, DMA port can be different, nothing guaranties correct
> > operation of the driver with newer unknown controllers.
> 
> Fair enough :) Then we just need to remember to update the function here
> as new generations get added and tested.
> 
> I suppose you don't know either if we could use the revision or similar field in
> the switch config space to determine generation somehow?

Nothing that I'm aware of.
Still, I like the idea of having a table-like construct somewhere in the code to centralize the handling of various controller-specific info (generation, DMA port, maybe more things in the future). 

Re: [PATCH 18/24] thunderbolt: Store Thunderbolt generation in the switch structure

[Mika Westerberg]

On Sun, May 21, 2017 at 11:18:01AM +0000, Bernat, Yehezkel wrote:
> Nothing that I'm aware of.

OK.

> Still, I like the idea of having a table-like construct somewhere in
> the code to centralize the handling of various controller-specific
> info (generation, DMA port, maybe more things in the future). 

Yup, I guess that's something we could try out in the next version of
the series.

Re: [PATCH 02/24] thunderbolt: Do not try to read UID if DROM offset is read as 0

[Andreas Noever]

On Thu, May 18, 2017 at 4:38 PM, Mika Westerberg
<mika.westerberg@linux.intel.com> wrote:
> At least Falcon Ridge when in host mode does not have any kind of DROM
> available and reading DROM offset returns 0 for these. Do not try to
> read DROM any further in that case.
>
> Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
> Reviewed-by: Michael Jamet <michael.jamet@intel.com>
> ---
>  drivers/thunderbolt/eeprom.c | 3 +++
>  1 file changed, 3 insertions(+)
>

Hi Mika,

nice work, it is nice to see Intel contribute to the Thunderbolt
driver (I can second Lukas's 'jaw drop' comment)!

I will try to read through everything today, but maybe the last few
patches will get pushed back to next weekend.

> diff --git a/drivers/thunderbolt/eeprom.c b/drivers/thunderbolt/eeprom.c
> index 6392990c984d..e4e64b130514 100644
> --- a/drivers/thunderbolt/eeprom.c
> +++ b/drivers/thunderbolt/eeprom.c
> @@ -276,6 +276,9 @@ int tb_drom_read_uid_only(struct tb_switch *sw, u64 *uid)
>         if (res)
>                 return res;
>
> +       if (drom_offset == 0)
> +               return -ENODEV;
> +
I think that this will make tb_switch_resume bail out on the root
switch, which is not good. Since the uid is only used to detect
whether a different device was plugged in while the system was
suspended I think that we can safely ignore the uid on the root
switch:
 - don't read it in tb_drom_read (route == 0 is already special cased anyways)
 - add a special case for the root switch to tb_switch_resume and
don't read the uid - just assume that it did not change (should be
impossible anyways)

What do you think?

Andreas

>         /* read uid */
>         res = tb_eeprom_read_n(sw, drom_offset, data, 9);
>         if (res)
> --
> 2.11.0
>

Re: [PATCH 04/24] thunderbolt: Add MSI-X support

[Andreas Noever]

On Thu, May 18, 2017 at 4:38 PM, Mika Westerberg
<mika.westerberg@linux.intel.com> wrote:
> Intel Thunderbolt controllers support up to 16 MSI-X vectors. Using
Is that true for all generations? If so can we remove the legacy path?

> MSI-X is preferred over MSI or legacy interrupt and may bring additional
> performance because there is no need to check the status registers which
> interrupt was triggered.
>
> While there we convert comments in structs tb_ring and tb_nhi to follow
> kernel-doc format more closely.
>
> This code is based on the work done by Amir Levy and Michael Jamet.
>
> Signed-off-by: Michael Jamet <michael.jamet@intel.com>
> Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
> ---
>  drivers/thunderbolt/ctl.c      |   4 +-
>  drivers/thunderbolt/nhi.c      | 159 ++++++++++++++++++++++++++++++++++++-----
>  drivers/thunderbolt/nhi.h      |  56 +++++++++++----
>  drivers/thunderbolt/nhi_regs.h |   9 +++
>  4 files changed, 196 insertions(+), 32 deletions(-)
>
> diff --git a/drivers/thunderbolt/ctl.c b/drivers/thunderbolt/ctl.c
> index 1031d97407a8..889a32dd21e7 100644
> --- a/drivers/thunderbolt/ctl.c
> +++ b/drivers/thunderbolt/ctl.c
> @@ -488,11 +488,11 @@ struct tb_ctl *tb_ctl_alloc(struct tb_nhi *nhi, hotplug_cb cb, void *cb_data)
>         if (!ctl->frame_pool)
>                 goto err;
>
> -       ctl->tx = ring_alloc_tx(nhi, 0, 10);
> +       ctl->tx = ring_alloc_tx(nhi, 0, 10, RING_FLAG_NO_SUSPEND);
>         if (!ctl->tx)
>                 goto err;
>
> -       ctl->rx = ring_alloc_rx(nhi, 0, 10);
> +       ctl->rx = ring_alloc_rx(nhi, 0, 10, RING_FLAG_NO_SUSPEND);
>         if (!ctl->rx)
>                 goto err;
>
> diff --git a/drivers/thunderbolt/nhi.c b/drivers/thunderbolt/nhi.c
> index a8c20413dbda..17f3b1bdb7da 100644
> --- a/drivers/thunderbolt/nhi.c
> +++ b/drivers/thunderbolt/nhi.c
> @@ -21,6 +21,12 @@
>
>  #define RING_TYPE(ring) ((ring)->is_tx ? "TX ring" : "RX ring")
>
> +/*
> + * Minimal number of vectors when we use MSI-X. Two for control channel
> + * Rx/Tx and the rest four are for cross domain DMA paths.
> + */
> +#define MSIX_MIN_VECS          6
> +#define MSIX_MAX_VECS          16
>
>  static int ring_interrupt_index(struct tb_ring *ring)
>  {
> @@ -239,8 +245,82 @@ int __ring_enqueue(struct tb_ring *ring, struct ring_frame *frame)
>         return ret;
>  }
>
> +static irqreturn_t ring_msix(int irq, void *data)
> +{
> +       struct tb_ring *ring = data;
> +
> +       schedule_work(&ring->work);
> +       return IRQ_HANDLED;
> +}
> +
> +static void ring_map_unmap_msix(struct tb_ring *ring, bool map)
> +{
This function does the same as ring_interrupt_active just for msix,
right? Can we rename one (or both) to express that?
For example rename ring_interrput_active -> ring_map_unmap_msi. Or
just roll this one into ring_interrupt_active and do the right thing
internally.

> +       u32 step, shift, ivr, misc;
> +       int index;
> +
> +       if (ring->irq <= 0)
> +               return;
> +
> +       if (ring->is_tx)
> +               index = ring->hop;
> +       else
> +               index = ring->hop + ring->nhi->hop_count;
> +
> +       /*
> +        * Ask the hardware to clear interrupt status bits automatically
> +        * since we already know which interrupt was triggered.
> +        */
> +       misc = ioread32(ring->nhi->iobase + REG_DMA_MISC);
> +       if (!(misc & REG_DMA_MISC_INT_AUTO_CLEAR)) {
> +               misc |= REG_DMA_MISC_INT_AUTO_CLEAR;
> +               iowrite32(misc, ring->nhi->iobase + REG_DMA_MISC);
> +       }
> +
> +       step = index / REG_INT_VEC_ALLOC_REGS * REG_INT_VEC_ALLOC_BITS;
> +       shift = index % REG_INT_VEC_ALLOC_REGS * REG_INT_VEC_ALLOC_BITS;
> +       ivr = ioread32(ring->nhi->iobase + REG_INT_VEC_ALLOC_BASE + step);
> +       ivr &= ~(REG_INT_VEC_ALLOC_MASK << shift);
> +       if (map)
> +               ivr |= ring->vector << shift;
> +       iowrite32(ivr, ring->nhi->iobase + REG_INT_VEC_ALLOC_BASE + step);
> +}
> +
> +static int ring_request_msix(struct tb_ring *ring, bool no_suspend)
> +{
> +       struct tb_nhi *nhi = ring->nhi;
> +       unsigned long irqflags;
> +       int ret;
> +
> +       if (!nhi->pdev->msix_enabled)
> +               return 0;
> +
> +       ret = ida_simple_get(&nhi->msix_ida, 0, MSIX_MAX_VECS, GFP_KERNEL);
> +       if (ret < 0)
> +               return ret;
> +
> +       ring->vector = ret;
> +
> +       ring->irq = pci_irq_vector(ring->nhi->pdev, ring->vector);
> +       if (ring->irq < 0)
> +               return ring->irq;
> +
> +       irqflags = no_suspend ? IRQF_NO_SUSPEND : 0;
> +       return request_irq(ring->irq, ring_msix, irqflags, "thunderbolt", ring);
> +}
> +
> +static void ring_release_msix(struct tb_ring *ring)
> +{
> +       if (ring->irq <= 0)
> +               return;
> +
> +       free_irq(ring->irq, ring);
> +       ida_simple_remove(&ring->nhi->msix_ida, ring->vector);
> +       ring->vector = 0;
> +       ring->irq = 0;
> +}
> +
>  static struct tb_ring *ring_alloc(struct tb_nhi *nhi, u32 hop, int size,
> -                                 bool transmit)
> +                                 bool transmit, unsigned int flags)
>  {
>         struct tb_ring *ring = NULL;
>         dev_info(&nhi->pdev->dev, "allocating %s ring %d of size %d\n",
> @@ -271,9 +351,14 @@ static struct tb_ring *ring_alloc(struct tb_nhi *nhi, u32 hop, int size,
>         ring->hop = hop;
>         ring->is_tx = transmit;
>         ring->size = size;
> +       ring->flags = flags;
>         ring->head = 0;
>         ring->tail = 0;
>         ring->running = false;
> +
> +       if (ring_request_msix(ring, flags & RING_FLAG_NO_SUSPEND))
> +               goto err;
> +
>         ring->descriptors = dma_alloc_coherent(&ring->nhi->pdev->dev,
>                         size * sizeof(*ring->descriptors),
>                         &ring->descriptors_dma, GFP_KERNEL | __GFP_ZERO);
> @@ -295,14 +380,16 @@ static struct tb_ring *ring_alloc(struct tb_nhi *nhi, u32 hop, int size,
>         return NULL;
>  }
>
> -struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size)
> +struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size,
> +                             unsigned int flags)
>  {
> -       return ring_alloc(nhi, hop, size, true);
> +       return ring_alloc(nhi, hop, size, true, flags);
>  }
>
> -struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size)
> +struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size,
> +                             unsigned int flags)
>  {
> -       return ring_alloc(nhi, hop, size, false);
> +       return ring_alloc(nhi, hop, size, false, flags);
>  }
>
>  /**
> @@ -334,6 +421,7 @@ void ring_start(struct tb_ring *ring)
>                 ring_iowrite32options(ring,
>                                       RING_FLAG_ENABLE | RING_FLAG_RAW, 0);
>         }
> +       ring_map_unmap_msix(ring, true);
>         ring_interrupt_active(ring, true);
>         ring->running = true;
>  err:
> @@ -366,6 +454,7 @@ void ring_stop(struct tb_ring *ring)
>                 goto err;
>         }
>         ring_interrupt_active(ring, false);
> +       ring_map_unmap_msix(ring, false);
>
>         ring_iowrite32options(ring, 0, 0);
>         ring_iowrite64desc(ring, 0, 0);
> @@ -413,6 +502,8 @@ void ring_free(struct tb_ring *ring)
>                          RING_TYPE(ring), ring->hop);
>         }
>
> +       ring_release_msix(ring);
> +
>         dma_free_coherent(&ring->nhi->pdev->dev,
>                           ring->size * sizeof(*ring->descriptors),
>                           ring->descriptors, ring->descriptors_dma);
There is a comment a couple lines below that states that ring->work is
only scheduled by nhi_interrupt_work and ring_stop. Please add
ring_msix to the list.

> @@ -528,9 +619,51 @@ static void nhi_shutdown(struct tb_nhi *nhi)
>          * We have to release the irq before calling flush_work. Otherwise an
>          * already executing IRQ handler could call schedule_work again.
>          */
> -       devm_free_irq(&nhi->pdev->dev, nhi->pdev->irq, nhi);
> +       if (!nhi->pdev->msix_enabled)
> +               devm_free_irq(&nhi->pdev->dev, nhi->pdev->irq, nhi);
>         flush_work(&nhi->interrupt_work);
>         mutex_destroy(&nhi->lock);
> +       ida_destroy(&nhi->msix_ida);
> +}
> +
> +static int nhi_init_msi(struct tb_nhi *nhi)
> +{
> +       struct pci_dev *pdev = nhi->pdev;
> +       int res, irq, nvec;
> +
> +       /* In case someone left them on. */
> +       nhi_disable_interrupts(nhi);
> +
> +       ida_init(&nhi->msix_ida);
> +
> +       /*
> +        * The NHI has 16 MSI-X vectors or a single MSI. We first try to
> +        * get all MSI-X vectors and if we succeed, each ring will have
> +        * one MSI-X. If for some reason that does not work out, we
> +        * fallback to a single MSI.
> +        */
> +       nvec = pci_alloc_irq_vectors(pdev, MSIX_MIN_VECS, MSIX_MAX_VECS,
> +                                    PCI_IRQ_MSIX);
> +       if (nvec < 0) {
> +               nvec = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> +               if (nvec < 0)
> +                       return nvec;
> +
> +               INIT_WORK(&nhi->interrupt_work, nhi_interrupt_work);
As far as I can see INIT_WORK is only called here in the fallback
case. But flush_work is still called unconditionally in nhi_shutdown.
> +
> +               irq = pci_irq_vector(nhi->pdev, 0);
> +               if (irq < 0)
> +                       return irq;
> +
> +               res = devm_request_irq(&pdev->dev, irq, nhi_msi,
> +                                      IRQF_NO_SUSPEND, "thunderbolt", nhi);
> +               if (res) {
> +                       dev_err(&pdev->dev, "request_irq failed, aborting\n");
> +                       return res;
> +               }
> +       }
> +
> +       return 0;
>  }
>
>  static int nhi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> @@ -545,12 +678,6 @@ static int nhi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>                 return res;
>         }
>
> -       res = pci_enable_msi(pdev);
> -       if (res) {
> -               dev_err(&pdev->dev, "cannot enable MSI, aborting\n");
> -               return res;
> -       }
> -
>         res = pcim_iomap_regions(pdev, 1 << 0, "thunderbolt");
>         if (res) {
>                 dev_err(&pdev->dev, "cannot obtain PCI resources, aborting\n");
> @@ -568,7 +695,6 @@ static int nhi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>         if (nhi->hop_count != 12 && nhi->hop_count != 32)
>                 dev_warn(&pdev->dev, "unexpected hop count: %d\n",
>                          nhi->hop_count);
> -       INIT_WORK(&nhi->interrupt_work, nhi_interrupt_work);
>
>         nhi->tx_rings = devm_kcalloc(&pdev->dev, nhi->hop_count,
>                                      sizeof(*nhi->tx_rings), GFP_KERNEL);
> @@ -577,12 +703,9 @@ static int nhi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>         if (!nhi->tx_rings || !nhi->rx_rings)
>                 return -ENOMEM;
>
> -       nhi_disable_interrupts(nhi); /* In case someone left them on. */
> -       res = devm_request_irq(&pdev->dev, pdev->irq, nhi_msi,
> -                              IRQF_NO_SUSPEND, /* must work during _noirq */
> -                              "thunderbolt", nhi);
> +       res = nhi_init_msi(nhi);
>         if (res) {
> -               dev_err(&pdev->dev, "request_irq failed, aborting\n");
> +               dev_err(&pdev->dev, "cannot enable MSI, aborting\n");
>                 return res;
>         }
>
> diff --git a/drivers/thunderbolt/nhi.h b/drivers/thunderbolt/nhi.h
> index 317242939b31..630f44140530 100644
> --- a/drivers/thunderbolt/nhi.h
> +++ b/drivers/thunderbolt/nhi.h
> @@ -7,45 +7,75 @@
>  #ifndef DSL3510_H_
>  #define DSL3510_H_
>
> +#include <linux/idr.h>
>  #include <linux/mutex.h>
>  #include <linux/workqueue.h>
>
>  /**
>   * struct tb_nhi - thunderbolt native host interface
> + * @lock: Must be held during ring creation/destruction. Is acquired by
> + *       interrupt_work when dispatching interrupts to individual rings.
> + * @pdev: Pointer to the PCI device
> + * @iobase: MMIO space of the NHI
> + * @tx_rings: All Tx rings available on this host controller
> + * @rx_rings: All Rx rings available on this host controller
> + * @msix_ida: Used to allocate MSI-X vectors for rings
> + * @interrupt_work: Work scheduled to handle ring interrupt when no
> + *                 MSI-X is used.
> + * @hop_count: Number of rings (end point hops) supported by NHI.
>   */
>  struct tb_nhi {
> -       struct mutex lock; /*
> -                           * Must be held during ring creation/destruction.
> -                           * Is acquired by interrupt_work when dispatching
> -                           * interrupts to individual rings.
> -                           **/
> +       struct mutex lock;
>         struct pci_dev *pdev;
>         void __iomem *iobase;
>         struct tb_ring **tx_rings;
>         struct tb_ring **rx_rings;
> +       struct ida msix_ida;
>         struct work_struct interrupt_work;
> -       u32 hop_count; /* Number of rings (end point hops) supported by NHI. */
> +       u32 hop_count;
>  };
>
>  /**
>   * struct tb_ring - thunderbolt TX or RX ring associated with a NHI
> + * @lock: Lock serializing actions to this ring. Must be acquired after
> + *       nhi->lock.
> + * @nhi: Pointer to the native host controller interface
> + * @size: Size of the ring
> + * @hop: Hop (DMA channel) associated with this ring
> + * @head: Head of the ring (write next descriptor here)
> + * @tail: Tail of the ring (complete next descriptor here)
> + * @descriptors: Allocated descriptors for this ring
> + * @queue: Queue holding frames to be transferred over this ring
> + * @in_flight: Queue holding frames that are currently in flight
> + * @work: Interrupt work structure
> + * @is_tx: Is the ring Tx or Rx
> + * @running: Is the ring running
> + * @irq: MSI-X irq number if the ring uses MSI-X. %0 otherwise.
> + * @vector: MSI-X vector number the ring uses (only set if @irq is > 0)
> + * @flags: Ring specific flags
>   */
>  struct tb_ring {
> -       struct mutex lock; /* must be acquired after nhi->lock */
> +       struct mutex lock;
>         struct tb_nhi *nhi;
>         int size;
>         int hop;
> -       int head; /* write next descriptor here */
> -       int tail; /* complete next descriptor here */
> +       int head;
> +       int tail;
>         struct ring_desc *descriptors;
>         dma_addr_t descriptors_dma;
>         struct list_head queue;
>         struct list_head in_flight;
>         struct work_struct work;
> -       bool is_tx:1; /* rx otherwise */
> +       bool is_tx:1;
>         bool running:1;
> +       int irq;
> +       u8 vector;
> +       unsigned int flags;
>  };
>
> +/* Leave ring interrupt enabled on suspend */
> +#define RING_FLAG_NO_SUSPEND   BIT(0)
> +
>  struct ring_frame;
>  typedef void (*ring_cb)(struct tb_ring*, struct ring_frame*, bool canceled);
>
> @@ -64,8 +94,10 @@ struct ring_frame {
>
>  #define TB_FRAME_SIZE 0x100    /* minimum size for ring_rx */
>
> -struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size);
> -struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size);
> +struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size,
> +                             unsigned int flags);
> +struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size,
> +                             unsigned int flags);
>  void ring_start(struct tb_ring *ring);
>  void ring_stop(struct tb_ring *ring);
>  void ring_free(struct tb_ring *ring);
> diff --git a/drivers/thunderbolt/nhi_regs.h b/drivers/thunderbolt/nhi_regs.h
> index 75cf0691e6c5..48b98d3c7e6a 100644
> --- a/drivers/thunderbolt/nhi_regs.h
> +++ b/drivers/thunderbolt/nhi_regs.h
> @@ -95,7 +95,16 @@ struct ring_desc {
>  #define REG_RING_INTERRUPT_BASE        0x38200
>  #define RING_INTERRUPT_REG_COUNT(nhi) ((31 + 2 * nhi->hop_count) / 32)
>
> +/* Interrupt Vector Allocation */
> +#define REG_INT_VEC_ALLOC_BASE 0x38c40
> +#define REG_INT_VEC_ALLOC_BITS 4
> +#define REG_INT_VEC_ALLOC_MASK GENMASK(3, 0)
> +#define REG_INT_VEC_ALLOC_REGS (32 / REG_INT_VEC_ALLOC_BITS)
> +
>  /* The last 11 bits contain the number of hops supported by the NHI port. */
>  #define REG_HOP_COUNT          0x39640
>
> +#define REG_DMA_MISC                   0x39864
> +#define REG_DMA_MISC_INT_AUTO_CLEAR     BIT(2)
> +
>  #endif
> --
> 2.11.0
>

Re: [PATCH 05/24] thunderbolt: Rework capability handling

[Andreas Noever]

On Thu, May 18, 2017 at 4:38 PM, Mika Westerberg
<mika.westerberg@linux.intel.com> wrote:
> Organization of the capabilities in switches and ports is not so random
> after all. Rework the capability handling functionality so that it
> follows how capabilities are organized and provide two new functions
> (tb_switch_find_vsec_cap() and tb_port_find_cap()) which can be used to
> extract capabilities for ports and switches. Then convert the current
> users over these.

Hm, could you explain the "official" logic a bit? What I could gather
from reading the code:
 - Port capabilities are always "basic" and the list is terminated
with next=0x00
 - Switch capabilities start "basic" then if cap==VSEC they turn into
"extendend_short" and after offset=0xff they turn into
"extended_long". The list is terminated by either next=0x00 or
next=0xffff.
 - There is always an extended cap whose offset is < 0xff  (otherwise
tb_switch_find_cap won't find the first vsec cap).
 - If there are "long" caps then there is always one at exactly 0xff
(because short caps have single byte next pointers).
Are these observations correct? I must say I found the old logic to
detect long capabilities (short.next == short.length == 0) more
elegant (and I'm quite sure that is the logic that the Mac driver
implemented at some point), but we can of course change it to match
the way it is written in the spec (and of course I have to request
this at least once: you should definitely release the spec - I highly
doubt that Intel's competitive advantage depends on keeping this
linked list technology secret ;) ).

Two more inline comments:
> Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
> Reviewed-by: Michael Jamet <michael.jamet@intel.com>
> ---
>  drivers/thunderbolt/cap.c        | 169 +++++++++++++++++++++------------------
>  drivers/thunderbolt/switch.c     |   6 +-
>  drivers/thunderbolt/tb.c         |   8 +-
>  drivers/thunderbolt/tb.h         |   3 +-
>  drivers/thunderbolt/tb_regs.h    |  31 ++++---
>  drivers/thunderbolt/tunnel_pci.c |   8 +-
>  6 files changed, 124 insertions(+), 101 deletions(-)
>
> diff --git a/drivers/thunderbolt/cap.c b/drivers/thunderbolt/cap.c
> index a7b47e7cddbd..0dd852c3df8d 100644
> --- a/drivers/thunderbolt/cap.c
> +++ b/drivers/thunderbolt/cap.c
> @@ -9,6 +9,8 @@
>
>  #include "tb.h"
>
> +#define CAP_OFFSET_MAX         0xff
> +#define VSEC_CAP_OFFSET_MAX    0xffff
>
>  struct tb_cap_any {
>         union {
> @@ -18,99 +20,110 @@ struct tb_cap_any {
>         };
>  } __packed;
>
> -static bool tb_cap_is_basic(struct tb_cap_any *cap)
> -{
> -       /* basic.cap is u8. This checks only the lower 8 bit of cap. */
> -       return cap->basic.cap != 5;
> -}
> -
> -static bool tb_cap_is_long(struct tb_cap_any *cap)
> +/**
> + * tb_port_find_cap() - Find port capability
> + * @port: Port to find the capability for
> + * @cap: Capability to look
> + *
> + * Returns offset to start of capability or %-ENOENT if no such
> + * capability was found. Negative errno is returned if there was an
> + * error.
> + */
> +int tb_port_find_cap(struct tb_port *port, enum tb_port_cap cap)
>  {
> -       return !tb_cap_is_basic(cap)
> -              && cap->extended_short.next == 0
> -              && cap->extended_short.length == 0;
> -}
> +       u32 offset;
>
> -static enum tb_cap tb_cap(struct tb_cap_any *cap)
> -{
> -       if (tb_cap_is_basic(cap))
> -               return cap->basic.cap;
> +       /*
> +        * DP out adapters claim to implement TMU capability but in
> +        * reality they do not so we hard code the adapter specific
> +        * capability offset here.
> +        */
> +       if (port->config.type == TB_TYPE_DP_HDMI_OUT)
> +               offset = 0x39;
>         else
> -               /* extended_short/long have cap at the same offset. */
> -               return cap->extended_short.cap;
> +               offset = 0x1;
When I was reverse engineering this the problem was not that some
capability was advertised (what does "TMU" stand for?), but that a
capabilities's next pointer (the one at 0xa) was pointing ..
*somewhere*. The fix was to redirect it to 0x39 (which looked like it
was the next valid capability). Your code makes the iteration start at
0x39. Which I guess is equivalent if the first capability is always
the faulty one?

Also can we make this quirk specific to some rom version or is it
expected that HDMI_OUT ports will forever have their first capability
at 0x39?

> +
> +       do {
> +               struct tb_cap_any header;
> +               int ret;
> +
> +               ret = tb_port_read(port, &header, TB_CFG_PORT, offset, 1);
> +               if (ret)
> +                       return ret;
> +
> +               if (header.basic.cap == cap)
> +                       return offset;
> +
> +               offset = header.basic.next;
> +       } while (offset);
> +
> +       return -ENOENT;
>  }
>
> -static u32 tb_cap_next(struct tb_cap_any *cap, u32 offset)
> +static int tb_switch_find_cap(struct tb_switch *sw, enum tb_switch_cap cap)
>  {
> -       int next;
> -       if (offset == 1) {
> -               /*
> -                * The first pointer is part of the switch header and always
> -                * a simple pointer.
> -                */
> -               next = cap->basic.next;
> -       } else {
> -               /*
> -                * Somehow Intel decided to use 3 different types of capability
> -                * headers. It is not like anyone could have predicted that
> -                * single byte offsets are not enough...
> -                */
> -               if (tb_cap_is_basic(cap))
> -                       next = cap->basic.next;
> -               else if (!tb_cap_is_long(cap))
> -                       next = cap->extended_short.next;
> -               else
> -                       next = cap->extended_long.next;
> +       int offset = sw->config.first_cap_offset;
> +
> +       while (offset > 0 && offset < CAP_OFFSET_MAX) {
> +               struct tb_cap_any header;
> +               int ret;
> +
> +               ret = tb_sw_read(sw, &header, TB_CFG_SWITCH, offset, 1);
> +               if (ret)
> +                       return ret;
> +
> +               if (header.basic.cap == cap)
> +                       return offset;
> +
> +               offset = header.basic.next;
>         }
> -       /*
> -        * "Hey, we could terminate some capability lists with a null offset
> -        *  and others with a pointer to the last element." - "Great idea!"
> -        */
> -       if (next == offset)
> -               return 0;
> -       return next;
> +
> +       return -ENOENT;
>  }
>
>  /**
> - * tb_find_cap() - find a capability
> + * tb_switch_find_vsec_cap() - Find switch vendor specific capability
> + * @sw: Switch to find the capability for
> + * @vsec: Vendor specific capability to look
>   *
> - * Return: Returns a positive offset if the capability was found and 0 if not.
> - * Returns an error code on failure.
> + * Functions enumerates vendor specific (VSEC) capabilities of a switch
> + * and returns offset when capability matching @vsed is found. If no
> + * such capability is found returns %-ENOENT. In case of error returns
> + * negative errno.
>   */
> -int tb_find_cap(struct tb_port *port, enum tb_cfg_space space, enum tb_cap cap)
> +int tb_switch_find_vsec_cap(struct tb_switch *sw, enum tb_switch_vsec_cap vsec)
>  {
> -       u32 offset = 1;
>         struct tb_cap_any header;
> -       int res;
> -       int retries = 10;
> -       while (retries--) {
> -               res = tb_port_read(port, &header, space, offset, 1);
> -               if (res) {
> -                       /* Intel needs some help with linked lists. */
> -                       if (space == TB_CFG_PORT && offset == 0xa
> -                           && port->config.type == TB_TYPE_DP_HDMI_OUT) {
> -                               offset = 0x39;
> -                               continue;
> -                       }
> -                       return res;
> -               }
> -               if (offset != 1) {
> -                       if (tb_cap(&header) == cap)
> +       int offset;
> +
> +       offset = tb_switch_find_cap(sw, TB_SWITCH_CAP_VSEC);
> +       if (offset < 0)
> +               return offset;
> +
> +       while (offset > 0 && offset < VSEC_CAP_OFFSET_MAX) {
> +               int ret;
> +
> +               ret = tb_sw_read(sw, &header, TB_CFG_SWITCH, offset, 2);
> +               if (ret)
> +                       return ret;
> +
> +               /*
> +                * Extended vendor specific capabilities come in two
> +                * flavors: short and long. The latter is used when
> +                * offset is over 0xff.
> +                */
> +               if (offset >= CAP_OFFSET_MAX) {
> +                       if (header.extended_long.vsec_id == vsec)
>                                 return offset;
> -                       if (tb_cap_is_long(&header)) {
> -                               /* tb_cap_extended_long is 2 dwords */
> -                               res = tb_port_read(port, &header, space,
> -                                                  offset, 2);
> -                               if (res)
> -                                       return res;
> -                       }
> +                       offset = header.extended_long.next;
> +               } else {
> +                       if (header.extended_short.vsec_id == vsec)
> +                               return offset;
> +                       if (!header.extended_short.length)
> +                               return -ENOENT;
> +                       offset = header.extended_short.next;
>                 }
> -               offset = tb_cap_next(&header, offset);
> -               if (!offset)
> -                       return 0;
>         }
> -       tb_port_WARN(port,
> -                    "run out of retries while looking for cap %#x in config space %d, last offset: %#x\n",
> -                    cap, space, offset);
> -       return -EIO;
> +
> +       return -ENOENT;
>  }
> diff --git a/drivers/thunderbolt/switch.c b/drivers/thunderbolt/switch.c
> index c6f30b1695a9..5bae49d72d2c 100644
> --- a/drivers/thunderbolt/switch.c
> +++ b/drivers/thunderbolt/switch.c
> @@ -192,7 +192,7 @@ static int tb_init_port(struct tb_port *port)
>
>         /* Port 0 is the switch itself and has no PHY. */
>         if (port->config.type == TB_TYPE_PORT && port->port != 0) {
> -               cap = tb_find_cap(port, TB_CFG_PORT, TB_CAP_PHY);
> +               cap = tb_port_find_cap(port, TB_PORT_CAP_PHY);
>
>                 if (cap > 0)
>                         port->cap_phy = cap;
> @@ -394,9 +394,9 @@ struct tb_switch *tb_switch_alloc(struct tb *tb, u64 route)
>                 sw->ports[i].port = i;
>         }
>
> -       cap = tb_find_cap(&sw->ports[0], TB_CFG_SWITCH, TB_CAP_PLUG_EVENTS);
> +       cap = tb_switch_find_vsec_cap(sw, TB_VSEC_CAP_PLUG_EVENTS);
>         if (cap < 0) {
> -               tb_sw_warn(sw, "cannot find TB_CAP_PLUG_EVENTS aborting\n");
> +               tb_sw_warn(sw, "cannot find TB_VSEC_CAP_PLUG_EVENTS aborting\n");
>                 goto err;
>         }
>         sw->cap_plug_events = cap;
> diff --git a/drivers/thunderbolt/tb.c b/drivers/thunderbolt/tb.c
> index 24b6d30c3c86..6b44076e1380 100644
> --- a/drivers/thunderbolt/tb.c
> +++ b/drivers/thunderbolt/tb.c
> @@ -121,8 +121,8 @@ static struct tb_port *tb_find_unused_down_port(struct tb_switch *sw)
>                         continue;
>                 if (sw->ports[i].config.type != TB_TYPE_PCIE_DOWN)
>                         continue;
> -               cap = tb_find_cap(&sw->ports[i], TB_CFG_PORT, TB_CAP_PCIE);
> -               if (cap <= 0)
> +               cap = tb_port_find_cap(&sw->ports[i], TB_PORT_CAP_ADAP);
> +               if (cap < 0)
>                         continue;
>                 res = tb_port_read(&sw->ports[i], &data, TB_CFG_PORT, cap, 1);
>                 if (res < 0)
> @@ -165,8 +165,8 @@ static void tb_activate_pcie_devices(struct tb *tb)
>                 }
>
>                 /* check whether port is already activated */
> -               cap = tb_find_cap(up_port, TB_CFG_PORT, TB_CAP_PCIE);
> -               if (cap <= 0)
> +               cap = tb_port_find_cap(up_port, TB_PORT_CAP_ADAP);
> +               if (cap < 0)
>                         continue;
>                 if (tb_port_read(up_port, &data, TB_CFG_PORT, cap, 1))
>                         continue;
> diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
> index ba2b85750335..9b60e71562dc 100644
> --- a/drivers/thunderbolt/tb.h
> +++ b/drivers/thunderbolt/tb.h
> @@ -233,7 +233,8 @@ int tb_wait_for_port(struct tb_port *port, bool wait_if_unplugged);
>  int tb_port_add_nfc_credits(struct tb_port *port, int credits);
>  int tb_port_clear_counter(struct tb_port *port, int counter);
>
> -int tb_find_cap(struct tb_port *port, enum tb_cfg_space space, enum tb_cap cap);
> +int tb_switch_find_vsec_cap(struct tb_switch *sw, enum tb_switch_vsec_cap vsec);
> +int tb_port_find_cap(struct tb_port *port, enum tb_port_cap cap);
>
>  struct tb_path *tb_path_alloc(struct tb *tb, int num_hops);
>  void tb_path_free(struct tb_path *path);
> diff --git a/drivers/thunderbolt/tb_regs.h b/drivers/thunderbolt/tb_regs.h
> index 1e2a4a8046be..57fdfa61e136 100644
> --- a/drivers/thunderbolt/tb_regs.h
> +++ b/drivers/thunderbolt/tb_regs.h
> @@ -23,15 +23,22 @@
>   */
>  #define TB_MAX_CONFIG_RW_LENGTH 60
>
> -enum tb_cap {
> -       TB_CAP_PHY              = 0x0001,
> -       TB_CAP_TIME1            = 0x0003,
> -       TB_CAP_PCIE             = 0x0004,
> -       TB_CAP_I2C              = 0x0005,
> -       TB_CAP_PLUG_EVENTS      = 0x0105, /* also EEPROM */
> -       TB_CAP_TIME2            = 0x0305,
> -       TB_CAP_IECS             = 0x0405,
> -       TB_CAP_LINK_CONTROLLER  = 0x0605, /* also IECS */
> +enum tb_switch_cap {
> +       TB_SWITCH_CAP_VSEC              = 0x05,
> +};
> +
> +enum tb_switch_vsec_cap {
> +       TB_VSEC_CAP_PLUG_EVENTS         = 0x01, /* also EEPROM */
> +       TB_VSEC_CAP_TIME2               = 0x03,
> +       TB_VSEC_CAP_IECS                = 0x04,
> +       TB_VSEC_CAP_LINK_CONTROLLER     = 0x06, /* also IECS */
> +};
> +
> +enum tb_port_cap {
> +       TB_PORT_CAP_PHY                 = 0x01,
> +       TB_PORT_CAP_TIME1               = 0x03,
> +       TB_PORT_CAP_ADAP                = 0x04,
> +       TB_PORT_CAP_VSEC                = 0x05,
>  };
>
>  enum tb_port_state {
> @@ -51,13 +58,15 @@ struct tb_cap_basic {
>
>  struct tb_cap_extended_short {
>         u8 next; /* if next and length are zero then we have a long cap */
> -       enum tb_cap cap:16;
> +       u8 cap;
> +       u8 vsec_id;
Nit: can we make this enum tb_switch_cap:8; enum tb_switch_vsec_cap:8?
(also below) Or at least add a comment linking these fields to the
enums.

Thanks,
Andreas
>         u8 length;
>  } __packed;
>
>  struct tb_cap_extended_long {
>         u8 zero1;
> -       enum tb_cap cap:16;
> +       u8 cap;
> +       u8 vsec_id;
>         u8 zero2;
>         u16 next;
>         u16 length;
> diff --git a/drivers/thunderbolt/tunnel_pci.c b/drivers/thunderbolt/tunnel_pci.c
> index baf1cd370446..f4ce9845e42a 100644
> --- a/drivers/thunderbolt/tunnel_pci.c
> +++ b/drivers/thunderbolt/tunnel_pci.c
> @@ -147,10 +147,10 @@ bool tb_pci_is_invalid(struct tb_pci_tunnel *tunnel)
>  static int tb_pci_port_active(struct tb_port *port, bool active)
>  {
>         u32 word = active ? 0x80000000 : 0x0;
> -       int cap = tb_find_cap(port, TB_CFG_PORT, TB_CAP_PCIE);
> -       if (cap <= 0) {
> -               tb_port_warn(port, "TB_CAP_PCIE not found: %d\n", cap);
> -               return cap ? cap : -ENXIO;
> +       int cap = tb_port_find_cap(port, TB_PORT_CAP_ADAP);
> +       if (cap < 0) {
> +               tb_port_warn(port, "TB_PORT_CAP_ADAP not found: %d\n", cap);
> +               return cap;
>         }
>         return tb_port_write(port, &word, TB_CFG_PORT, cap, 1);
>  }
> --
> 2.11.0
>

Re: [PATCH 02/24] thunderbolt: Do not try to read UID if DROM offset is read as 0

[Mika Westerberg]

On Sun, May 21, 2017 at 03:46:20PM +0200, Andreas Noever wrote:
> On Thu, May 18, 2017 at 4:38 PM, Mika Westerberg
> <mika.westerberg@linux.intel.com> wrote:
> > At least Falcon Ridge when in host mode does not have any kind of DROM
> > available and reading DROM offset returns 0 for these. Do not try to
> > read DROM any further in that case.
> >
> > Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> > Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
> > Reviewed-by: Michael Jamet <michael.jamet@intel.com>
> > ---
> >  drivers/thunderbolt/eeprom.c | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> 
> Hi Mika,
> 
> nice work, it is nice to see Intel contribute to the Thunderbolt
> driver (I can second Lukas's 'jaw drop' comment)!
> 
> I will try to read through everything today, but maybe the last few
> patches will get pushed back to next weekend.

Thanks :)

> > diff --git a/drivers/thunderbolt/eeprom.c b/drivers/thunderbolt/eeprom.c
> > index 6392990c984d..e4e64b130514 100644
> > --- a/drivers/thunderbolt/eeprom.c
> > +++ b/drivers/thunderbolt/eeprom.c
> > @@ -276,6 +276,9 @@ int tb_drom_read_uid_only(struct tb_switch *sw, u64 *uid)
> >         if (res)
> >                 return res;
> >
> > +       if (drom_offset == 0)
> > +               return -ENODEV;
> > +
> I think that this will make tb_switch_resume bail out on the root
> switch, which is not good. Since the uid is only used to detect
> whether a different device was plugged in while the system was
> suspended I think that we can safely ignore the uid on the root
> switch:
>  - don't read it in tb_drom_read (route == 0 is already special cased anyways)
>  - add a special case for the root switch to tb_switch_resume and
> don't read the uid - just assume that it did not change (should be
> impossible anyways)
> 
> What do you think?

I think there actually is such check already in tb_switch_resume() where
we special case the root switch ignoring its UID. Unless I'm missing
something.

I'm testing this on a Mac with Cactus Ridge and the root switch resume
does not fail :)

Re: [PATCH 04/24] thunderbolt: Add MSI-X support

[Mika Westerberg]

On Sun, May 21, 2017 at 07:51:09PM +0200, Andreas Noever wrote:
> On Thu, May 18, 2017 at 4:38 PM, Mika Westerberg
> <mika.westerberg@linux.intel.com> wrote:
> > Intel Thunderbolt controllers support up to 16 MSI-X vectors. Using
> Is that true for all generations? If so can we remove the legacy path?

Yes, I think it has been the case from the beginning. I have here one
Mac Mini with Light Ridge and even it has working MSI-X.

Yehezkel, Michael, Amir, do you know if that's the case for all other
controller as well?

> > MSI-X is preferred over MSI or legacy interrupt and may bring additional
> > performance because there is no need to check the status registers which
> > interrupt was triggered.
> >
> > While there we convert comments in structs tb_ring and tb_nhi to follow
> > kernel-doc format more closely.
> >
> > This code is based on the work done by Amir Levy and Michael Jamet.
> >
> > Signed-off-by: Michael Jamet <michael.jamet@intel.com>
> > Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> > Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
> > ---
> >  drivers/thunderbolt/ctl.c      |   4 +-
> >  drivers/thunderbolt/nhi.c      | 159 ++++++++++++++++++++++++++++++++++++-----
> >  drivers/thunderbolt/nhi.h      |  56 +++++++++++----
> >  drivers/thunderbolt/nhi_regs.h |   9 +++
> >  4 files changed, 196 insertions(+), 32 deletions(-)
> >
> > diff --git a/drivers/thunderbolt/ctl.c b/drivers/thunderbolt/ctl.c
> > index 1031d97407a8..889a32dd21e7 100644
> > --- a/drivers/thunderbolt/ctl.c
> > +++ b/drivers/thunderbolt/ctl.c
> > @@ -488,11 +488,11 @@ struct tb_ctl *tb_ctl_alloc(struct tb_nhi *nhi, hotplug_cb cb, void *cb_data)
> >         if (!ctl->frame_pool)
> >                 goto err;
> >
> > -       ctl->tx = ring_alloc_tx(nhi, 0, 10);
> > +       ctl->tx = ring_alloc_tx(nhi, 0, 10, RING_FLAG_NO_SUSPEND);
> >         if (!ctl->tx)
> >                 goto err;
> >
> > -       ctl->rx = ring_alloc_rx(nhi, 0, 10);
> > +       ctl->rx = ring_alloc_rx(nhi, 0, 10, RING_FLAG_NO_SUSPEND);
> >         if (!ctl->rx)
> >                 goto err;
> >
> > diff --git a/drivers/thunderbolt/nhi.c b/drivers/thunderbolt/nhi.c
> > index a8c20413dbda..17f3b1bdb7da 100644
> > --- a/drivers/thunderbolt/nhi.c
> > +++ b/drivers/thunderbolt/nhi.c
> > @@ -21,6 +21,12 @@
> >
> >  #define RING_TYPE(ring) ((ring)->is_tx ? "TX ring" : "RX ring")
> >
> > +/*
> > + * Minimal number of vectors when we use MSI-X. Two for control channel
> > + * Rx/Tx and the rest four are for cross domain DMA paths.
> > + */
> > +#define MSIX_MIN_VECS          6
> > +#define MSIX_MAX_VECS          16
> >
> >  static int ring_interrupt_index(struct tb_ring *ring)
> >  {
> > @@ -239,8 +245,82 @@ int __ring_enqueue(struct tb_ring *ring, struct ring_frame *frame)
> >         return ret;
> >  }
> >
> > +static irqreturn_t ring_msix(int irq, void *data)
> > +{
> > +       struct tb_ring *ring = data;
> > +
> > +       schedule_work(&ring->work);
> > +       return IRQ_HANDLED;
> > +}
> > +
> > +static void ring_map_unmap_msix(struct tb_ring *ring, bool map)
> > +{
> This function does the same as ring_interrupt_active just for msix,
> right? Can we rename one (or both) to express that?
> For example rename ring_interrput_active -> ring_map_unmap_msi. Or
> just roll this one into ring_interrupt_active and do the right thing
> internally.

Yes, I think we can do that.

> > +       u32 step, shift, ivr, misc;
> > +       int index;
> > +
> > +       if (ring->irq <= 0)
> > +               return;
> > +
> > +       if (ring->is_tx)
> > +               index = ring->hop;
> > +       else
> > +               index = ring->hop + ring->nhi->hop_count;
> > +
> > +       /*
> > +        * Ask the hardware to clear interrupt status bits automatically
> > +        * since we already know which interrupt was triggered.
> > +        */
> > +       misc = ioread32(ring->nhi->iobase + REG_DMA_MISC);
> > +       if (!(misc & REG_DMA_MISC_INT_AUTO_CLEAR)) {
> > +               misc |= REG_DMA_MISC_INT_AUTO_CLEAR;
> > +               iowrite32(misc, ring->nhi->iobase + REG_DMA_MISC);
> > +       }
> > +
> > +       step = index / REG_INT_VEC_ALLOC_REGS * REG_INT_VEC_ALLOC_BITS;
> > +       shift = index % REG_INT_VEC_ALLOC_REGS * REG_INT_VEC_ALLOC_BITS;
> > +       ivr = ioread32(ring->nhi->iobase + REG_INT_VEC_ALLOC_BASE + step);
> > +       ivr &= ~(REG_INT_VEC_ALLOC_MASK << shift);
> > +       if (map)
> > +               ivr |= ring->vector << shift;
> > +       iowrite32(ivr, ring->nhi->iobase + REG_INT_VEC_ALLOC_BASE + step);
> > +}
> > +
> > +static int ring_request_msix(struct tb_ring *ring, bool no_suspend)
> > +{
> > +       struct tb_nhi *nhi = ring->nhi;
> > +       unsigned long irqflags;
> > +       int ret;
> > +
> > +       if (!nhi->pdev->msix_enabled)
> > +               return 0;
> > +
> > +       ret = ida_simple_get(&nhi->msix_ida, 0, MSIX_MAX_VECS, GFP_KERNEL);
> > +       if (ret < 0)
> > +               return ret;
> > +
> > +       ring->vector = ret;
> > +
> > +       ring->irq = pci_irq_vector(ring->nhi->pdev, ring->vector);
> > +       if (ring->irq < 0)
> > +               return ring->irq;
> > +
> > +       irqflags = no_suspend ? IRQF_NO_SUSPEND : 0;
> > +       return request_irq(ring->irq, ring_msix, irqflags, "thunderbolt", ring);
> > +}
> > +
> > +static void ring_release_msix(struct tb_ring *ring)
> > +{
> > +       if (ring->irq <= 0)
> > +               return;
> > +
> > +       free_irq(ring->irq, ring);
> > +       ida_simple_remove(&ring->nhi->msix_ida, ring->vector);
> > +       ring->vector = 0;
> > +       ring->irq = 0;
> > +}
> > +
> >  static struct tb_ring *ring_alloc(struct tb_nhi *nhi, u32 hop, int size,
> > -                                 bool transmit)
> > +                                 bool transmit, unsigned int flags)
> >  {
> >         struct tb_ring *ring = NULL;
> >         dev_info(&nhi->pdev->dev, "allocating %s ring %d of size %d\n",
> > @@ -271,9 +351,14 @@ static struct tb_ring *ring_alloc(struct tb_nhi *nhi, u32 hop, int size,
> >         ring->hop = hop;
> >         ring->is_tx = transmit;
> >         ring->size = size;
> > +       ring->flags = flags;
> >         ring->head = 0;
> >         ring->tail = 0;
> >         ring->running = false;
> > +
> > +       if (ring_request_msix(ring, flags & RING_FLAG_NO_SUSPEND))
> > +               goto err;
> > +
> >         ring->descriptors = dma_alloc_coherent(&ring->nhi->pdev->dev,
> >                         size * sizeof(*ring->descriptors),
> >                         &ring->descriptors_dma, GFP_KERNEL | __GFP_ZERO);
> > @@ -295,14 +380,16 @@ static struct tb_ring *ring_alloc(struct tb_nhi *nhi, u32 hop, int size,
> >         return NULL;
> >  }
> >
> > -struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size)
> > +struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size,
> > +                             unsigned int flags)
> >  {
> > -       return ring_alloc(nhi, hop, size, true);
> > +       return ring_alloc(nhi, hop, size, true, flags);
> >  }
> >
> > -struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size)
> > +struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size,
> > +                             unsigned int flags)
> >  {
> > -       return ring_alloc(nhi, hop, size, false);
> > +       return ring_alloc(nhi, hop, size, false, flags);
> >  }
> >
> >  /**
> > @@ -334,6 +421,7 @@ void ring_start(struct tb_ring *ring)
> >                 ring_iowrite32options(ring,
> >                                       RING_FLAG_ENABLE | RING_FLAG_RAW, 0);
> >         }
> > +       ring_map_unmap_msix(ring, true);
> >         ring_interrupt_active(ring, true);
> >         ring->running = true;
> >  err:
> > @@ -366,6 +454,7 @@ void ring_stop(struct tb_ring *ring)
> >                 goto err;
> >         }
> >         ring_interrupt_active(ring, false);
> > +       ring_map_unmap_msix(ring, false);
> >
> >         ring_iowrite32options(ring, 0, 0);
> >         ring_iowrite64desc(ring, 0, 0);
> > @@ -413,6 +502,8 @@ void ring_free(struct tb_ring *ring)
> >                          RING_TYPE(ring), ring->hop);
> >         }
> >
> > +       ring_release_msix(ring);
> > +
> >         dma_free_coherent(&ring->nhi->pdev->dev,
> >                           ring->size * sizeof(*ring->descriptors),
> >                           ring->descriptors, ring->descriptors_dma);
> There is a comment a couple lines below that states that ring->work is
> only scheduled by nhi_interrupt_work and ring_stop. Please add
> ring_msix to the list.

OK.

> > @@ -528,9 +619,51 @@ static void nhi_shutdown(struct tb_nhi *nhi)
> >          * We have to release the irq before calling flush_work. Otherwise an
> >          * already executing IRQ handler could call schedule_work again.
> >          */
> > -       devm_free_irq(&nhi->pdev->dev, nhi->pdev->irq, nhi);
> > +       if (!nhi->pdev->msix_enabled)
> > +               devm_free_irq(&nhi->pdev->dev, nhi->pdev->irq, nhi);
> >         flush_work(&nhi->interrupt_work);
> >         mutex_destroy(&nhi->lock);
> > +       ida_destroy(&nhi->msix_ida);
> > +}
> > +
> > +static int nhi_init_msi(struct tb_nhi *nhi)
> > +{
> > +       struct pci_dev *pdev = nhi->pdev;
> > +       int res, irq, nvec;
> > +
> > +       /* In case someone left them on. */
> > +       nhi_disable_interrupts(nhi);
> > +
> > +       ida_init(&nhi->msix_ida);
> > +
> > +       /*
> > +        * The NHI has 16 MSI-X vectors or a single MSI. We first try to
> > +        * get all MSI-X vectors and if we succeed, each ring will have
> > +        * one MSI-X. If for some reason that does not work out, we
> > +        * fallback to a single MSI.
> > +        */
> > +       nvec = pci_alloc_irq_vectors(pdev, MSIX_MIN_VECS, MSIX_MAX_VECS,
> > +                                    PCI_IRQ_MSIX);
> > +       if (nvec < 0) {
> > +               nvec = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> > +               if (nvec < 0)
> > +                       return nvec;
> > +
> > +               INIT_WORK(&nhi->interrupt_work, nhi_interrupt_work);
> As far as I can see INIT_WORK is only called here in the fallback
> case. But flush_work is still called unconditionally in nhi_shutdown.

Good point. I should make it conditional as well (or INIT_WORK()
unconditional).

Re: [PATCH 05/24] thunderbolt: Rework capability handling

[Mika Westerberg]

On Sun, May 21, 2017 at 09:09:37PM +0200, Andreas Noever wrote:
> On Thu, May 18, 2017 at 4:38 PM, Mika Westerberg
> <mika.westerberg@linux.intel.com> wrote:
> > Organization of the capabilities in switches and ports is not so random
> > after all. Rework the capability handling functionality so that it
> > follows how capabilities are organized and provide two new functions
> > (tb_switch_find_vsec_cap() and tb_port_find_cap()) which can be used to
> > extract capabilities for ports and switches. Then convert the current
> > users over these.
> 
> Hm, could you explain the "official" logic a bit? What I could gather
> from reading the code:
>  - Port capabilities are always "basic" and the list is terminated
> with next=0x00

Yes.

It is pretty much (in both cases) following normal PCI capabilities with
a couple of special cases.

>  - Switch capabilities start "basic" then if cap==VSEC they turn into
> "extendend_short" and after offset=0xff they turn into
> "extended_long". The list is terminated by either next=0x00 or
> next=0xffff.

Vendor specific (VSEC) capabilities use the "extended_short" format
unless they point past 256 registers (0xff) in which case they use the
"extended_long" extension format instead. That register space ends at
0xffff.

>  - There is always an extended cap whose offset is < 0xff  (otherwise
> tb_switch_find_cap won't find the first vsec cap).
>  - If there are "long" caps then there is always one at exactly 0xff
> (because short caps have single byte next pointers).

At offset 0xff there is always link controller vendor specific
capability and that is using the "extended_long" format.

> Are these observations correct? I must say I found the old logic to
> detect long capabilities (short.next == short.length == 0) more
> elegant (and I'm quite sure that is the logic that the Mac driver
> implemented at some point), but we can of course change it to match
> the way it is written in the spec (and of course I have to request
> this at least once: you should definitely release the spec - I highly
> doubt that Intel's competitive advantage depends on keeping this
> linked list technology secret ;) ).

Basically what I wanted to do here is to make the capability walking
routines to follow how those are organized separating switch and port
capabilities from each other.

I agree with you that we should release these specs but unfortunately it
is not my call. But we can try to make the driver better with the
knowledge we have ;-)

If you want to keep the old logic, I can just drop this patch. The rest
of the functionality should still work.

> Two more inline comments:
> > Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> > Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
> > Reviewed-by: Michael Jamet <michael.jamet@intel.com>
> > ---
> >  drivers/thunderbolt/cap.c        | 169 +++++++++++++++++++++------------------
> >  drivers/thunderbolt/switch.c     |   6 +-
> >  drivers/thunderbolt/tb.c         |   8 +-
> >  drivers/thunderbolt/tb.h         |   3 +-
> >  drivers/thunderbolt/tb_regs.h    |  31 ++++---
> >  drivers/thunderbolt/tunnel_pci.c |   8 +-
> >  6 files changed, 124 insertions(+), 101 deletions(-)
> >
> > diff --git a/drivers/thunderbolt/cap.c b/drivers/thunderbolt/cap.c
> > index a7b47e7cddbd..0dd852c3df8d 100644
> > --- a/drivers/thunderbolt/cap.c
> > +++ b/drivers/thunderbolt/cap.c
> > @@ -9,6 +9,8 @@
> >
> >  #include "tb.h"
> >
> > +#define CAP_OFFSET_MAX         0xff
> > +#define VSEC_CAP_OFFSET_MAX    0xffff
> >
> >  struct tb_cap_any {
> >         union {
> > @@ -18,99 +20,110 @@ struct tb_cap_any {
> >         };
> >  } __packed;
> >
> > -static bool tb_cap_is_basic(struct tb_cap_any *cap)
> > -{
> > -       /* basic.cap is u8. This checks only the lower 8 bit of cap. */
> > -       return cap->basic.cap != 5;
> > -}
> > -
> > -static bool tb_cap_is_long(struct tb_cap_any *cap)
> > +/**
> > + * tb_port_find_cap() - Find port capability
> > + * @port: Port to find the capability for
> > + * @cap: Capability to look
> > + *
> > + * Returns offset to start of capability or %-ENOENT if no such
> > + * capability was found. Negative errno is returned if there was an
> > + * error.
> > + */
> > +int tb_port_find_cap(struct tb_port *port, enum tb_port_cap cap)
> >  {
> > -       return !tb_cap_is_basic(cap)
> > -              && cap->extended_short.next == 0
> > -              && cap->extended_short.length == 0;
> > -}
> > +       u32 offset;
> >
> > -static enum tb_cap tb_cap(struct tb_cap_any *cap)
> > -{
> > -       if (tb_cap_is_basic(cap))
> > -               return cap->basic.cap;
> > +       /*
> > +        * DP out adapters claim to implement TMU capability but in
> > +        * reality they do not so we hard code the adapter specific
> > +        * capability offset here.
> > +        */
> > +       if (port->config.type == TB_TYPE_DP_HDMI_OUT)
> > +               offset = 0x39;
> >         else
> > -               /* extended_short/long have cap at the same offset. */
> > -               return cap->extended_short.cap;
> > +               offset = 0x1;
> When I was reverse engineering this the problem was not that some
> capability was advertised (what does "TMU" stand for?), but that a
> capabilities's next pointer (the one at 0xa) was pointing ..
> *somewhere*.

I think TMU stands for Time Management Unit.

> The fix was to redirect it to 0x39 (which looked like it
> was the next valid capability). Your code makes the iteration start at
> 0x39. Which I guess is equivalent if the first capability is always
> the faulty one?

Yes for some reason (this is not in the hardware spec either) the DP out
capabilities actually starts from 0x39.

> Also can we make this quirk specific to some rom version or is it
> expected that HDMI_OUT ports will forever have their first capability
> at 0x39?

Yes, it is always like that so I don't think we need to quirk that.

> > +
> > +       do {
> > +               struct tb_cap_any header;
> > +               int ret;
> > +
> > +               ret = tb_port_read(port, &header, TB_CFG_PORT, offset, 1);
> > +               if (ret)
> > +                       return ret;
> > +
> > +               if (header.basic.cap == cap)
> > +                       return offset;
> > +
> > +               offset = header.basic.next;
> > +       } while (offset);
> > +
> > +       return -ENOENT;
> >  }
> >
> > -static u32 tb_cap_next(struct tb_cap_any *cap, u32 offset)
> > +static int tb_switch_find_cap(struct tb_switch *sw, enum tb_switch_cap cap)
> >  {
> > -       int next;
> > -       if (offset == 1) {
> > -               /*
> > -                * The first pointer is part of the switch header and always
> > -                * a simple pointer.
> > -                */
> > -               next = cap->basic.next;
> > -       } else {
> > -               /*
> > -                * Somehow Intel decided to use 3 different types of capability
> > -                * headers. It is not like anyone could have predicted that
> > -                * single byte offsets are not enough...
> > -                */
> > -               if (tb_cap_is_basic(cap))
> > -                       next = cap->basic.next;
> > -               else if (!tb_cap_is_long(cap))
> > -                       next = cap->extended_short.next;
> > -               else
> > -                       next = cap->extended_long.next;
> > +       int offset = sw->config.first_cap_offset;
> > +
> > +       while (offset > 0 && offset < CAP_OFFSET_MAX) {
> > +               struct tb_cap_any header;
> > +               int ret;
> > +
> > +               ret = tb_sw_read(sw, &header, TB_CFG_SWITCH, offset, 1);
> > +               if (ret)
> > +                       return ret;
> > +
> > +               if (header.basic.cap == cap)
> > +                       return offset;
> > +
> > +               offset = header.basic.next;
> >         }
> > -       /*
> > -        * "Hey, we could terminate some capability lists with a null offset
> > -        *  and others with a pointer to the last element." - "Great idea!"
> > -        */
> > -       if (next == offset)
> > -               return 0;
> > -       return next;
> > +
> > +       return -ENOENT;
> >  }
> >
> >  /**
> > - * tb_find_cap() - find a capability
> > + * tb_switch_find_vsec_cap() - Find switch vendor specific capability
> > + * @sw: Switch to find the capability for
> > + * @vsec: Vendor specific capability to look
> >   *
> > - * Return: Returns a positive offset if the capability was found and 0 if not.
> > - * Returns an error code on failure.
> > + * Functions enumerates vendor specific (VSEC) capabilities of a switch
> > + * and returns offset when capability matching @vsed is found. If no
> > + * such capability is found returns %-ENOENT. In case of error returns
> > + * negative errno.
> >   */
> > -int tb_find_cap(struct tb_port *port, enum tb_cfg_space space, enum tb_cap cap)
> > +int tb_switch_find_vsec_cap(struct tb_switch *sw, enum tb_switch_vsec_cap vsec)
> >  {
> > -       u32 offset = 1;
> >         struct tb_cap_any header;
> > -       int res;
> > -       int retries = 10;
> > -       while (retries--) {
> > -               res = tb_port_read(port, &header, space, offset, 1);
> > -               if (res) {
> > -                       /* Intel needs some help with linked lists. */
> > -                       if (space == TB_CFG_PORT && offset == 0xa
> > -                           && port->config.type == TB_TYPE_DP_HDMI_OUT) {
> > -                               offset = 0x39;
> > -                               continue;
> > -                       }
> > -                       return res;
> > -               }
> > -               if (offset != 1) {
> > -                       if (tb_cap(&header) == cap)
> > +       int offset;
> > +
> > +       offset = tb_switch_find_cap(sw, TB_SWITCH_CAP_VSEC);
> > +       if (offset < 0)
> > +               return offset;
> > +
> > +       while (offset > 0 && offset < VSEC_CAP_OFFSET_MAX) {
> > +               int ret;
> > +
> > +               ret = tb_sw_read(sw, &header, TB_CFG_SWITCH, offset, 2);
> > +               if (ret)
> > +                       return ret;
> > +
> > +               /*
> > +                * Extended vendor specific capabilities come in two
> > +                * flavors: short and long. The latter is used when
> > +                * offset is over 0xff.
> > +                */
> > +               if (offset >= CAP_OFFSET_MAX) {
> > +                       if (header.extended_long.vsec_id == vsec)
> >                                 return offset;
> > -                       if (tb_cap_is_long(&header)) {
> > -                               /* tb_cap_extended_long is 2 dwords */
> > -                               res = tb_port_read(port, &header, space,
> > -                                                  offset, 2);
> > -                               if (res)
> > -                                       return res;
> > -                       }
> > +                       offset = header.extended_long.next;
> > +               } else {
> > +                       if (header.extended_short.vsec_id == vsec)
> > +                               return offset;
> > +                       if (!header.extended_short.length)
> > +                               return -ENOENT;
> > +                       offset = header.extended_short.next;
> >                 }
> > -               offset = tb_cap_next(&header, offset);
> > -               if (!offset)
> > -                       return 0;
> >         }
> > -       tb_port_WARN(port,
> > -                    "run out of retries while looking for cap %#x in config space %d, last offset: %#x\n",
> > -                    cap, space, offset);
> > -       return -EIO;
> > +
> > +       return -ENOENT;
> >  }
> > diff --git a/drivers/thunderbolt/switch.c b/drivers/thunderbolt/switch.c
> > index c6f30b1695a9..5bae49d72d2c 100644
> > --- a/drivers/thunderbolt/switch.c
> > +++ b/drivers/thunderbolt/switch.c
> > @@ -192,7 +192,7 @@ static int tb_init_port(struct tb_port *port)
> >
> >         /* Port 0 is the switch itself and has no PHY. */
> >         if (port->config.type == TB_TYPE_PORT && port->port != 0) {
> > -               cap = tb_find_cap(port, TB_CFG_PORT, TB_CAP_PHY);
> > +               cap = tb_port_find_cap(port, TB_PORT_CAP_PHY);
> >
> >                 if (cap > 0)
> >                         port->cap_phy = cap;
> > @@ -394,9 +394,9 @@ struct tb_switch *tb_switch_alloc(struct tb *tb, u64 route)
> >                 sw->ports[i].port = i;
> >         }
> >
> > -       cap = tb_find_cap(&sw->ports[0], TB_CFG_SWITCH, TB_CAP_PLUG_EVENTS);
> > +       cap = tb_switch_find_vsec_cap(sw, TB_VSEC_CAP_PLUG_EVENTS);
> >         if (cap < 0) {
> > -               tb_sw_warn(sw, "cannot find TB_CAP_PLUG_EVENTS aborting\n");
> > +               tb_sw_warn(sw, "cannot find TB_VSEC_CAP_PLUG_EVENTS aborting\n");
> >                 goto err;
> >         }
> >         sw->cap_plug_events = cap;
> > diff --git a/drivers/thunderbolt/tb.c b/drivers/thunderbolt/tb.c
> > index 24b6d30c3c86..6b44076e1380 100644
> > --- a/drivers/thunderbolt/tb.c
> > +++ b/drivers/thunderbolt/tb.c
> > @@ -121,8 +121,8 @@ static struct tb_port *tb_find_unused_down_port(struct tb_switch *sw)
> >                         continue;
> >                 if (sw->ports[i].config.type != TB_TYPE_PCIE_DOWN)
> >                         continue;
> > -               cap = tb_find_cap(&sw->ports[i], TB_CFG_PORT, TB_CAP_PCIE);
> > -               if (cap <= 0)
> > +               cap = tb_port_find_cap(&sw->ports[i], TB_PORT_CAP_ADAP);
> > +               if (cap < 0)
> >                         continue;
> >                 res = tb_port_read(&sw->ports[i], &data, TB_CFG_PORT, cap, 1);
> >                 if (res < 0)
> > @@ -165,8 +165,8 @@ static void tb_activate_pcie_devices(struct tb *tb)
> >                 }
> >
> >                 /* check whether port is already activated */
> > -               cap = tb_find_cap(up_port, TB_CFG_PORT, TB_CAP_PCIE);
> > -               if (cap <= 0)
> > +               cap = tb_port_find_cap(up_port, TB_PORT_CAP_ADAP);
> > +               if (cap < 0)
> >                         continue;
> >                 if (tb_port_read(up_port, &data, TB_CFG_PORT, cap, 1))
> >                         continue;
> > diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
> > index ba2b85750335..9b60e71562dc 100644
> > --- a/drivers/thunderbolt/tb.h
> > +++ b/drivers/thunderbolt/tb.h
> > @@ -233,7 +233,8 @@ int tb_wait_for_port(struct tb_port *port, bool wait_if_unplugged);
> >  int tb_port_add_nfc_credits(struct tb_port *port, int credits);
> >  int tb_port_clear_counter(struct tb_port *port, int counter);
> >
> > -int tb_find_cap(struct tb_port *port, enum tb_cfg_space space, enum tb_cap cap);
> > +int tb_switch_find_vsec_cap(struct tb_switch *sw, enum tb_switch_vsec_cap vsec);
> > +int tb_port_find_cap(struct tb_port *port, enum tb_port_cap cap);
> >
> >  struct tb_path *tb_path_alloc(struct tb *tb, int num_hops);
> >  void tb_path_free(struct tb_path *path);
> > diff --git a/drivers/thunderbolt/tb_regs.h b/drivers/thunderbolt/tb_regs.h
> > index 1e2a4a8046be..57fdfa61e136 100644
> > --- a/drivers/thunderbolt/tb_regs.h
> > +++ b/drivers/thunderbolt/tb_regs.h
> > @@ -23,15 +23,22 @@
> >   */
> >  #define TB_MAX_CONFIG_RW_LENGTH 60
> >
> > -enum tb_cap {
> > -       TB_CAP_PHY              = 0x0001,
> > -       TB_CAP_TIME1            = 0x0003,
> > -       TB_CAP_PCIE             = 0x0004,
> > -       TB_CAP_I2C              = 0x0005,
> > -       TB_CAP_PLUG_EVENTS      = 0x0105, /* also EEPROM */
> > -       TB_CAP_TIME2            = 0x0305,
> > -       TB_CAP_IECS             = 0x0405,
> > -       TB_CAP_LINK_CONTROLLER  = 0x0605, /* also IECS */
> > +enum tb_switch_cap {
> > +       TB_SWITCH_CAP_VSEC              = 0x05,
> > +};
> > +
> > +enum tb_switch_vsec_cap {
> > +       TB_VSEC_CAP_PLUG_EVENTS         = 0x01, /* also EEPROM */
> > +       TB_VSEC_CAP_TIME2               = 0x03,
> > +       TB_VSEC_CAP_IECS                = 0x04,
> > +       TB_VSEC_CAP_LINK_CONTROLLER     = 0x06, /* also IECS */
> > +};
> > +
> > +enum tb_port_cap {
> > +       TB_PORT_CAP_PHY                 = 0x01,
> > +       TB_PORT_CAP_TIME1               = 0x03,
> > +       TB_PORT_CAP_ADAP                = 0x04,
> > +       TB_PORT_CAP_VSEC                = 0x05,
> >  };
> >
> >  enum tb_port_state {
> > @@ -51,13 +58,15 @@ struct tb_cap_basic {
> >
> >  struct tb_cap_extended_short {
> >         u8 next; /* if next and length are zero then we have a long cap */
> > -       enum tb_cap cap:16;
> > +       u8 cap;
> > +       u8 vsec_id;
> Nit: can we make this enum tb_switch_cap:8; enum tb_switch_vsec_cap:8?
> (also below) Or at least add a comment linking these fields to the
> enums.

I'm not sure it is good idea to use bit fields here at all. I'm not an
expert in C but I remember reading somewhere that they are not suitable
for representing fields inside hardware registers. Maybe someone can
correct me here?

That is one reason I've tried to avoid using them in these patches and
in the new code being added to this driver.

I'll add a comment there linking the fields (given that this patch is
not dropped) :)

RE: [PATCH 05/24] thunderbolt: Rework capability handling

[Levy, Amir (Jer)]

On Mon, May 22 2017, 12:45 PM, Mika Westerberg wrote:
> I'm not sure it is good idea to use bit fields here at all. I'm not an expert in C
> but I remember reading somewhere that they are not suitable for representing
> fields inside hardware registers. 
> 

http://yarchive.net/comp/linux/bitfields.html

RE: [PATCH 04/24] thunderbolt: Add MSI-X support

[Bernat, Yehezkel]

> -----Original Message-----
> From: Mika Westerberg [mailto:mika.westerberg@linux.intel.com]
> Sent: Monday, May 22, 2017 11:52
> To: Andreas Noever <andreas.noever@gmail.com>
> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>; Jamet, Michael
> <michael.jamet@intel.com>; Bernat, Yehezkel
> <yehezkel.bernat@intel.com>; Lukas Wunner <lukas@wunner.de>; Levy,
> Amir (Jer) <amir.jer.levy@intel.com>; Andy Lutomirski <luto@kernel.org>;
> Mario.Limonciello@dell.com; Jared.Dominguez@dell.com; Andy Shevchenko
> <andriy.shevchenko@linux.intel.com>; linux-kernel@vger.kernel.org
> Subject: Re: [PATCH 04/24] thunderbolt: Add MSI-X support
> 
> On Sun, May 21, 2017 at 07:51:09PM +0200, Andreas Noever wrote:
> > On Thu, May 18, 2017 at 4:38 PM, Mika Westerberg
> > <mika.westerberg@linux.intel.com> wrote:
> > > Intel Thunderbolt controllers support up to 16 MSI-X vectors. Using
> > Is that true for all generations? If so can we remove the legacy path?
> 
> Yes, I think it has been the case from the beginning. I have here one
> Mac Mini with Light Ridge and even it has working MSI-X.
> 
> Yehezkel, Michael, Amir, do you know if that's the case for all other
> controller as well?

We are checking it, but what if we don't get the MSI-X we need?
Don't we have to support MSI as a fallback option or maybe we are sure this will never happen in modern kernels?

> 
> > > MSI-X is preferred over MSI or legacy interrupt and may bring additional
> > > performance because there is no need to check the status registers
> which
> > > interrupt was triggered.
> > >
> > > While there we convert comments in structs tb_ring and tb_nhi to follow
> > > kernel-doc format more closely.
> > >
> > > This code is based on the work done by Amir Levy and Michael Jamet.
> > >
> > > Signed-off-by: Michael Jamet <michael.jamet@intel.com>
> > > Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> > > Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
> > > ---
> > >  drivers/thunderbolt/ctl.c      |   4 +-
> > >  drivers/thunderbolt/nhi.c      | 159
> ++++++++++++++++++++++++++++++++++++-----
> > >  drivers/thunderbolt/nhi.h      |  56 +++++++++++----
> > >  drivers/thunderbolt/nhi_regs.h |   9 +++
> > >  4 files changed, 196 insertions(+), 32 deletions(-)
> > >
> > > diff --git a/drivers/thunderbolt/ctl.c b/drivers/thunderbolt/ctl.c
> > > index 1031d97407a8..889a32dd21e7 100644
> > > --- a/drivers/thunderbolt/ctl.c
> > > +++ b/drivers/thunderbolt/ctl.c
> > > @@ -488,11 +488,11 @@ struct tb_ctl *tb_ctl_alloc(struct tb_nhi *nhi,
> hotplug_cb cb, void *cb_data)
> > >         if (!ctl->frame_pool)
> > >                 goto err;
> > >
> > > -       ctl->tx = ring_alloc_tx(nhi, 0, 10);
> > > +       ctl->tx = ring_alloc_tx(nhi, 0, 10, RING_FLAG_NO_SUSPEND);
> > >         if (!ctl->tx)
> > >                 goto err;
> > >
> > > -       ctl->rx = ring_alloc_rx(nhi, 0, 10);
> > > +       ctl->rx = ring_alloc_rx(nhi, 0, 10, RING_FLAG_NO_SUSPEND);
> > >         if (!ctl->rx)
> > >                 goto err;
> > >
> > > diff --git a/drivers/thunderbolt/nhi.c b/drivers/thunderbolt/nhi.c
> > > index a8c20413dbda..17f3b1bdb7da 100644
> > > --- a/drivers/thunderbolt/nhi.c
> > > +++ b/drivers/thunderbolt/nhi.c
> > > @@ -21,6 +21,12 @@
> > >
> > >  #define RING_TYPE(ring) ((ring)->is_tx ? "TX ring" : "RX ring")
> > >
> > > +/*
> > > + * Minimal number of vectors when we use MSI-X. Two for control
> channel
> > > + * Rx/Tx and the rest four are for cross domain DMA paths.
> > > + */
> > > +#define MSIX_MIN_VECS          6
> > > +#define MSIX_MAX_VECS          16
> > >
> > >  static int ring_interrupt_index(struct tb_ring *ring)
> > >  {
> > > @@ -239,8 +245,82 @@ int __ring_enqueue(struct tb_ring *ring, struct
> ring_frame *frame)
> > >         return ret;
> > >  }
> > >
> > > +static irqreturn_t ring_msix(int irq, void *data)
> > > +{
> > > +       struct tb_ring *ring = data;
> > > +
> > > +       schedule_work(&ring->work);
> > > +       return IRQ_HANDLED;
> > > +}
> > > +
> > > +static void ring_map_unmap_msix(struct tb_ring *ring, bool map)
> > > +{
> > This function does the same as ring_interrupt_active just for msix,
> > right? Can we rename one (or both) to express that?
> > For example rename ring_interrput_active -> ring_map_unmap_msi. Or
> > just roll this one into ring_interrupt_active and do the right thing
> > internally.
> 
> Yes, I think we can do that.
> 
> > > +       u32 step, shift, ivr, misc;
> > > +       int index;
> > > +
> > > +       if (ring->irq <= 0)
> > > +               return;
> > > +
> > > +       if (ring->is_tx)
> > > +               index = ring->hop;
> > > +       else
> > > +               index = ring->hop + ring->nhi->hop_count;
> > > +
> > > +       /*
> > > +        * Ask the hardware to clear interrupt status bits automatically
> > > +        * since we already know which interrupt was triggered.
> > > +        */
> > > +       misc = ioread32(ring->nhi->iobase + REG_DMA_MISC);
> > > +       if (!(misc & REG_DMA_MISC_INT_AUTO_CLEAR)) {
> > > +               misc |= REG_DMA_MISC_INT_AUTO_CLEAR;
> > > +               iowrite32(misc, ring->nhi->iobase + REG_DMA_MISC);
> > > +       }
> > > +
> > > +       step = index / REG_INT_VEC_ALLOC_REGS *
> REG_INT_VEC_ALLOC_BITS;
> > > +       shift = index % REG_INT_VEC_ALLOC_REGS *
> REG_INT_VEC_ALLOC_BITS;
> > > +       ivr = ioread32(ring->nhi->iobase + REG_INT_VEC_ALLOC_BASE +
> step);
> > > +       ivr &= ~(REG_INT_VEC_ALLOC_MASK << shift);
> > > +       if (map)
> > > +               ivr |= ring->vector << shift;
> > > +       iowrite32(ivr, ring->nhi->iobase + REG_INT_VEC_ALLOC_BASE +
> step);
> > > +}
> > > +
> > > +static int ring_request_msix(struct tb_ring *ring, bool no_suspend)
> > > +{
> > > +       struct tb_nhi *nhi = ring->nhi;
> > > +       unsigned long irqflags;
> > > +       int ret;
> > > +
> > > +       if (!nhi->pdev->msix_enabled)
> > > +               return 0;
> > > +
> > > +       ret = ida_simple_get(&nhi->msix_ida, 0, MSIX_MAX_VECS,
> GFP_KERNEL);
> > > +       if (ret < 0)
> > > +               return ret;
> > > +
> > > +       ring->vector = ret;
> > > +
> > > +       ring->irq = pci_irq_vector(ring->nhi->pdev, ring->vector);
> > > +       if (ring->irq < 0)
> > > +               return ring->irq;
> > > +
> > > +       irqflags = no_suspend ? IRQF_NO_SUSPEND : 0;
> > > +       return request_irq(ring->irq, ring_msix, irqflags, "thunderbolt",
> ring);
> > > +}
> > > +
> > > +static void ring_release_msix(struct tb_ring *ring)
> > > +{
> > > +       if (ring->irq <= 0)
> > > +               return;
> > > +
> > > +       free_irq(ring->irq, ring);
> > > +       ida_simple_remove(&ring->nhi->msix_ida, ring->vector);
> > > +       ring->vector = 0;
> > > +       ring->irq = 0;
> > > +}
> > > +
> > >  static struct tb_ring *ring_alloc(struct tb_nhi *nhi, u32 hop, int size,
> > > -                                 bool transmit)
> > > +                                 bool transmit, unsigned int flags)
> > >  {
> > >         struct tb_ring *ring = NULL;
> > >         dev_info(&nhi->pdev->dev, "allocating %s ring %d of size %d\n",
> > > @@ -271,9 +351,14 @@ static struct tb_ring *ring_alloc(struct tb_nhi
> *nhi, u32 hop, int size,
> > >         ring->hop = hop;
> > >         ring->is_tx = transmit;
> > >         ring->size = size;
> > > +       ring->flags = flags;
> > >         ring->head = 0;
> > >         ring->tail = 0;
> > >         ring->running = false;
> > > +
> > > +       if (ring_request_msix(ring, flags & RING_FLAG_NO_SUSPEND))
> > > +               goto err;
> > > +
> > >         ring->descriptors = dma_alloc_coherent(&ring->nhi->pdev->dev,
> > >                         size * sizeof(*ring->descriptors),
> > >                         &ring->descriptors_dma, GFP_KERNEL | __GFP_ZERO);
> > > @@ -295,14 +380,16 @@ static struct tb_ring *ring_alloc(struct tb_nhi
> *nhi, u32 hop, int size,
> > >         return NULL;
> > >  }
> > >
> > > -struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size)
> > > +struct tb_ring *ring_alloc_tx(struct tb_nhi *nhi, int hop, int size,
> > > +                             unsigned int flags)
> > >  {
> > > -       return ring_alloc(nhi, hop, size, true);
> > > +       return ring_alloc(nhi, hop, size, true, flags);
> > >  }
> > >
> > > -struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size)
> > > +struct tb_ring *ring_alloc_rx(struct tb_nhi *nhi, int hop, int size,
> > > +                             unsigned int flags)
> > >  {
> > > -       return ring_alloc(nhi, hop, size, false);
> > > +       return ring_alloc(nhi, hop, size, false, flags);
> > >  }
> > >
> > >  /**
> > > @@ -334,6 +421,7 @@ void ring_start(struct tb_ring *ring)
> > >                 ring_iowrite32options(ring,
> > >                                       RING_FLAG_ENABLE | RING_FLAG_RAW, 0);
> > >         }
> > > +       ring_map_unmap_msix(ring, true);
> > >         ring_interrupt_active(ring, true);
> > >         ring->running = true;
> > >  err:
> > > @@ -366,6 +454,7 @@ void ring_stop(struct tb_ring *ring)
> > >                 goto err;
> > >         }
> > >         ring_interrupt_active(ring, false);
> > > +       ring_map_unmap_msix(ring, false);
> > >
> > >         ring_iowrite32options(ring, 0, 0);
> > >         ring_iowrite64desc(ring, 0, 0);
> > > @@ -413,6 +502,8 @@ void ring_free(struct tb_ring *ring)
> > >                          RING_TYPE(ring), ring->hop);
> > >         }
> > >
> > > +       ring_release_msix(ring);
> > > +
> > >         dma_free_coherent(&ring->nhi->pdev->dev,
> > >                           ring->size * sizeof(*ring->descriptors),
> > >                           ring->descriptors, ring->descriptors_dma);
> > There is a comment a couple lines below that states that ring->work is
> > only scheduled by nhi_interrupt_work and ring_stop. Please add
> > ring_msix to the list.
> 
> OK.
> 
> > > @@ -528,9 +619,51 @@ static void nhi_shutdown(struct tb_nhi *nhi)
> > >          * We have to release the irq before calling flush_work. Otherwise an
> > >          * already executing IRQ handler could call schedule_work again.
> > >          */
> > > -       devm_free_irq(&nhi->pdev->dev, nhi->pdev->irq, nhi);
> > > +       if (!nhi->pdev->msix_enabled)
> > > +               devm_free_irq(&nhi->pdev->dev, nhi->pdev->irq, nhi);
> > >         flush_work(&nhi->interrupt_work);
> > >         mutex_destroy(&nhi->lock);
> > > +       ida_destroy(&nhi->msix_ida);
> > > +}
> > > +
> > > +static int nhi_init_msi(struct tb_nhi *nhi)
> > > +{
> > > +       struct pci_dev *pdev = nhi->pdev;
> > > +       int res, irq, nvec;
> > > +
> > > +       /* In case someone left them on. */
> > > +       nhi_disable_interrupts(nhi);
> > > +
> > > +       ida_init(&nhi->msix_ida);
> > > +
> > > +       /*
> > > +        * The NHI has 16 MSI-X vectors or a single MSI. We first try to
> > > +        * get all MSI-X vectors and if we succeed, each ring will have
> > > +        * one MSI-X. If for some reason that does not work out, we
> > > +        * fallback to a single MSI.
> > > +        */
> > > +       nvec = pci_alloc_irq_vectors(pdev, MSIX_MIN_VECS,
> MSIX_MAX_VECS,
> > > +                                    PCI_IRQ_MSIX);
> > > +       if (nvec < 0) {
> > > +               nvec = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> > > +               if (nvec < 0)
> > > +                       return nvec;
> > > +
> > > +               INIT_WORK(&nhi->interrupt_work, nhi_interrupt_work);
> > As far as I can see INIT_WORK is only called here in the fallback
> > case. But flush_work is still called unconditionally in nhi_shutdown.
> 
> Good point. I should make it conditional as well (or INIT_WORK()
> unconditional).

Re: [PATCH 04/24] thunderbolt: Add MSI-X support

[Mika Westerberg]

On Mon, May 22, 2017 at 10:35:08AM +0000, Bernat, Yehezkel wrote:
> 
> 
> > -----Original Message-----
> > From: Mika Westerberg [mailto:mika.westerberg@linux.intel.com]
> > Sent: Monday, May 22, 2017 11:52
> > To: Andreas Noever <andreas.noever@gmail.com>
> > Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>; Jamet, Michael
> > <michael.jamet@intel.com>; Bernat, Yehezkel
> > <yehezkel.bernat@intel.com>; Lukas Wunner <lukas@wunner.de>; Levy,
> > Amir (Jer) <amir.jer.levy@intel.com>; Andy Lutomirski <luto@kernel.org>;
> > Mario.Limonciello@dell.com; Jared.Dominguez@dell.com; Andy Shevchenko
> > <andriy.shevchenko@linux.intel.com>; linux-kernel@vger.kernel.org
> > Subject: Re: [PATCH 04/24] thunderbolt: Add MSI-X support
> > 
> > On Sun, May 21, 2017 at 07:51:09PM +0200, Andreas Noever wrote:
> > > On Thu, May 18, 2017 at 4:38 PM, Mika Westerberg
> > > <mika.westerberg@linux.intel.com> wrote:
> > > > Intel Thunderbolt controllers support up to 16 MSI-X vectors. Using
> > > Is that true for all generations? If so can we remove the legacy path?
> > 
> > Yes, I think it has been the case from the beginning. I have here one
> > Mac Mini with Light Ridge and even it has working MSI-X.
> > 
> > Yehezkel, Michael, Amir, do you know if that's the case for all other
> > controller as well?
> 
> We are checking it, but what if we don't get the MSI-X we need?
> Don't we have to support MSI as a fallback option or maybe we are sure
> this will never happen in modern kernels?

Indeed, I think it makes sense to keep the fallback code just to be sure
it still works if for some reason we don't get the MSI-X vectors. I
think there is even a command line option to turn MSI-(X) off
completely.

Regarding this, I see that I'm missing a flag in the second call:

  pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);

should probably be 

  pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI | PCI_IRQ_LEGACY);

instead to make it work with legacy IRQs as well.

Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Mika Westerberg]

On Sat, May 20, 2017 at 11:24:12AM +0300, Mika Westerberg wrote:
> On Fri, May 19, 2017 at 05:54:37PM +0000, Mario.Limonciello@dell.com wrote:
> > > 
> > > It happens occasionally when you reboot the machine when a device is
> > > connected but seems to be dependent on the BIOS version. Since it is the
> > > BIOS who is supposed to enumerated these devices, I suspect that it is
> > > either problem in BIOS or our PCI enumeration code does something wrong.
> > > 
> > 
> > I'm fairly certain it's an issue somewhere with Linux PCI enumeration.  I took 
> > the exact same HW and switched it out the SSD to one w/ Win10 1607.
> > I set the dock and cable to "always allow" in TBT settings applet.
> > 
> > I don't reproduce any problems with enumeration with the dock plugged in
> > on cold boot.  All the devices hanging off the bridge show up properly.
> 
> You mean in Windows you don't reproduce the problem, right? Even when
> you reboot the machine with devices connected.
> 
> I tried on Intel Skull Canyon NUC so that I disabled the thunderbolt
> driver and after reboot (warm) I can see the PCI scan error about bus
> being partially hidden behind a bridge.
> 
> When this happens PCIe ports of the thunderbolt device/host seem to be
> unconfigured and Linux then decides to reconfigure them which leads to
> the problem. When it works we get ACPI hotplug event to the PCIe root
> port and the PCIe upstream/downstream ports are properly configured by
> the BIOS.
> 
> I guess Windows does something differently here than what we do when PCI
> devices are enumerated.

We discussed this with our BIOS/firmware people and there was a firmware
bug that caused many issues around hotplug and reboot flows. Is is
possible for you to try with the latest BIOS and see if the issue
reproduces (or are you already running the latest)?

Re: [PATCH 02/24] thunderbolt: Do not try to read UID if DROM offset is read as 0

[Andreas Noever]

On Mon, May 22, 2017 at 10:40 AM, Mika Westerberg
<mika.westerberg@linux.intel.com> wrote:
> On Sun, May 21, 2017 at 03:46:20PM +0200, Andreas Noever wrote:
>> On Thu, May 18, 2017 at 4:38 PM, Mika Westerberg
>> <mika.westerberg@linux.intel.com> wrote:
>> > At least Falcon Ridge when in host mode does not have any kind of DROM
>> > available and reading DROM offset returns 0 for these. Do not try to
>> > read DROM any further in that case.
>> >
>> > Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
>> > Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
>> > Reviewed-by: Michael Jamet <michael.jamet@intel.com>
>> > ---
>> >  drivers/thunderbolt/eeprom.c | 3 +++
>> >  1 file changed, 3 insertions(+)
>> >
>>
>> Hi Mika,
>>
>> nice work, it is nice to see Intel contribute to the Thunderbolt
>> driver (I can second Lukas's 'jaw drop' comment)!
>>
>> I will try to read through everything today, but maybe the last few
>> patches will get pushed back to next weekend.
>
> Thanks :)
>
>> > diff --git a/drivers/thunderbolt/eeprom.c b/drivers/thunderbolt/eeprom.c
>> > index 6392990c984d..e4e64b130514 100644
>> > --- a/drivers/thunderbolt/eeprom.c
>> > +++ b/drivers/thunderbolt/eeprom.c
>> > @@ -276,6 +276,9 @@ int tb_drom_read_uid_only(struct tb_switch *sw, u64 *uid)
>> >         if (res)
>> >                 return res;
>> >
>> > +       if (drom_offset == 0)
>> > +               return -ENODEV;
>> > +
>> I think that this will make tb_switch_resume bail out on the root
>> switch, which is not good. Since the uid is only used to detect
>> whether a different device was plugged in while the system was
>> suspended I think that we can safely ignore the uid on the root
>> switch:
>>  - don't read it in tb_drom_read (route == 0 is already special cased anyways)
>>  - add a special case for the root switch to tb_switch_resume and
>> don't read the uid - just assume that it did not change (should be
>> impossible anyways)
>>
>> What do you think?
>
> I think there actually is such check already in tb_switch_resume() where
> we special case the root switch ignoring its UID. Unless I'm missing
> something.
>
> I'm testing this on a Mac with Cactus Ridge and the root switch resume
> does not fail :)

Yes there is a check for the root switch, but also one that checks the
return code of tb_drom_read_uid_only :)

err = tb_drom_read_uid_only(sw, &uid);
if (err) {
    tb_sw_warn(sw, "uid read failed\n");
    return err;
}
if (sw != sw->tb->root_switch && sw->uid != uid) {


The reason it works on the Mac is because drom_offset is not 0, so the
new branch in tb_drom_read_uid_only is not taken. Probably this is the
case for all Cactus Ridge models and Alpine Ridge doesn't go there
since it uses the ICM? Still it wouldn't hurt to only read the uid if
sw != root_switch, the value is not used if sw == root_switch.

RE: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Mario.Limonciello]

> -----Original Message-----
> From: Mika Westerberg [mailto:mika.westerberg@linux.intel.com]
> Sent: Monday, May 22, 2017 6:37 AM
> To: Limonciello, Mario <Mario_Limonciello@Dell.com>
> Cc: gregkh@linuxfoundation.org; andreas.noever@gmail.com;
> michael.jamet@intel.com; yehezkel.bernat@intel.com; lukas@wunner.de;
> amir.jer.levy@intel.com; luto@kernel.org; Dominguez, Jared
> <Jared_Dominguez@DELL.com>; andriy.shevchenko@linux.intel.com; linux-
> kernel@vger.kernel.org
> Subject: Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade
> 
> On Sat, May 20, 2017 at 11:24:12AM +0300, Mika Westerberg wrote:
> > On Fri, May 19, 2017 at 05:54:37PM +0000, Mario.Limonciello@dell.com wrote:
> > > >
> > > > It happens occasionally when you reboot the machine when a device is
> > > > connected but seems to be dependent on the BIOS version. Since it is the
> > > > BIOS who is supposed to enumerated these devices, I suspect that it is
> > > > either problem in BIOS or our PCI enumeration code does something wrong.
> > > >
> > >
> > > I'm fairly certain it's an issue somewhere with Linux PCI enumeration.  I took
> > > the exact same HW and switched it out the SSD to one w/ Win10 1607.
> > > I set the dock and cable to "always allow" in TBT settings applet.
> > >
> > > I don't reproduce any problems with enumeration with the dock plugged in
> > > on cold boot.  All the devices hanging off the bridge show up properly.
> >
> > You mean in Windows you don't reproduce the problem, right? Even when
> > you reboot the machine with devices connected.
> >
> > I tried on Intel Skull Canyon NUC so that I disabled the thunderbolt
> > driver and after reboot (warm) I can see the PCI scan error about bus
> > being partially hidden behind a bridge.
> >
> > When this happens PCIe ports of the thunderbolt device/host seem to be
> > unconfigured and Linux then decides to reconfigure them which leads to
> > the problem. When it works we get ACPI hotplug event to the PCIe root
> > port and the PCIe upstream/downstream ports are properly configured by
> > the BIOS.
> >
> > I guess Windows does something differently here than what we do when PCI
> > devices are enumerated.
> 
> We discussed this with our BIOS/firmware people and there was a firmware
> bug that caused many issues around hotplug and reboot flows. Is is
> possible for you to try with the latest BIOS and see if the issue
> reproduces (or are you already running the latest)?

I was 1 version behind, but I double checked with the latest version (1.1.15)
and the same behavior exists on Linux (still works properly on Win10).

If you have some more details about what the FW guys changed, I can check
with my Dell FW team if they've picked up the same fix.  I'm guessing it's not
the same problem though considering it works properly on Win10?

RE: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Bernat, Yehezkel]

> -----Original Message-----
> From: Mario.Limonciello@dell.com [mailto:Mario.Limonciello@dell.com]
> Sent: Monday, May 22, 2017 23:08
> To: mika.westerberg@linux.intel.com
> Cc: gregkh@linuxfoundation.org; andreas.noever@gmail.com; Jamet,
> Michael <michael.jamet@intel.com>; Bernat, Yehezkel
> <yehezkel.bernat@intel.com>; lukas@wunner.de; Levy, Amir (Jer)
> <amir.jer.levy@intel.com>; luto@kernel.org; Jared.Dominguez@dell.com;
> andriy.shevchenko@linux.intel.com; linux-kernel@vger.kernel.org
> Subject: RE: [PATCH 00/24] Thunderbolt security levels and NVM firmware
> upgrade
> 
> > -----Original Message-----
> > From: Mika Westerberg [mailto:mika.westerberg@linux.intel.com]
> > Sent: Monday, May 22, 2017 6:37 AM
> > To: Limonciello, Mario <Mario_Limonciello@Dell.com>
> > Cc: gregkh@linuxfoundation.org; andreas.noever@gmail.com;
> > michael.jamet@intel.com; yehezkel.bernat@intel.com; lukas@wunner.de;
> > amir.jer.levy@intel.com; luto@kernel.org; Dominguez, Jared
> > <Jared_Dominguez@DELL.com>; andriy.shevchenko@linux.intel.com;
> linux-
> > kernel@vger.kernel.org
> > Subject: Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware
> upgrade
> >
> > On Sat, May 20, 2017 at 11:24:12AM +0300, Mika Westerberg wrote:
> > > On Fri, May 19, 2017 at 05:54:37PM +0000, Mario.Limonciello@dell.com
> wrote:
> > > > >
> > > > > It happens occasionally when you reboot the machine when a device
> is
> > > > > connected but seems to be dependent on the BIOS version. Since it is
> the
> > > > > BIOS who is supposed to enumerated these devices, I suspect that it
> is
> > > > > either problem in BIOS or our PCI enumeration code does something
> wrong.
> > > > >
> > > >
> > > > I'm fairly certain it's an issue somewhere with Linux PCI enumeration.  I
> took
> > > > the exact same HW and switched it out the SSD to one w/ Win10 1607.
> > > > I set the dock and cable to "always allow" in TBT settings applet.
> > > >
> > > > I don't reproduce any problems with enumeration with the dock
> plugged in
> > > > on cold boot.  All the devices hanging off the bridge show up properly.
> > >
> > > You mean in Windows you don't reproduce the problem, right? Even
> when
> > > you reboot the machine with devices connected.
> > >
> > > I tried on Intel Skull Canyon NUC so that I disabled the thunderbolt
> > > driver and after reboot (warm) I can see the PCI scan error about bus
> > > being partially hidden behind a bridge.
> > >
> > > When this happens PCIe ports of the thunderbolt device/host seem to
> be
> > > unconfigured and Linux then decides to reconfigure them which leads to
> > > the problem. When it works we get ACPI hotplug event to the PCIe root
> > > port and the PCIe upstream/downstream ports are properly configured
> by
> > > the BIOS.
> > >
> > > I guess Windows does something differently here than what we do when
> PCI
> > > devices are enumerated.
> >
> > We discussed this with our BIOS/firmware people and there was a
> firmware
> > bug that caused many issues around hotplug and reboot flows. Is is
> > possible for you to try with the latest BIOS and see if the issue
> > reproduces (or are you already running the latest)?
> 
> I was 1 version behind, but I double checked with the latest version (1.1.15)
> and the same behavior exists on Linux (still works properly on Win10).
> 
> If you have some more details about what the FW guys changed, I can check
> with my Dell FW team if they've picked up the same fix.  I'm guessing it's not
> the same problem though considering it works properly on Win10?

What about the TBT NVM version?
NVM 16 sounds a bit old to me.

Re: [PATCH 02/24] thunderbolt: Do not try to read UID if DROM offset is read as 0

[Mika Westerberg]

On Mon, May 22, 2017 at 08:41:22PM +0200, Andreas Noever wrote:
> Yes there is a check for the root switch, but also one that checks the
> return code of tb_drom_read_uid_only :)
> 
> err = tb_drom_read_uid_only(sw, &uid);
> if (err) {
>     tb_sw_warn(sw, "uid read failed\n");
>     return err;
> }
> if (sw != sw->tb->root_switch && sw->uid != uid) {
> 
> 
> The reason it works on the Mac is because drom_offset is not 0, so the
> new branch in tb_drom_read_uid_only is not taken. Probably this is the
> case for all Cactus Ridge models and Alpine Ridge doesn't go there
> since it uses the ICM? 

Yes in case of ICM we don't call the function at all.

> Still it wouldn't hurt to only read the uid if
> sw != root_switch, the value is not used if sw == root_switch.

I agree. I'll update the code so that it will only read and check UID
when we are not dealing with the root switch.

Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Mika Westerberg]

On Mon, May 22, 2017 at 08:07:54PM +0000, Mario.Limonciello@dell.com wrote:
> I was 1 version behind, but I double checked with the latest version (1.1.15)
> and the same behavior exists on Linux (still works properly on Win10).
> 
> If you have some more details about what the FW guys changed, I can check
> with my Dell FW team if they've picked up the same fix.  I'm guessing it's not
> the same problem though considering it works properly on Win10?

Can you send me full dmesg preferably so that you have acpiphp.dyndbg in
the kernel command line (you also need to have CONFIG_DYNAMIC_DEBUG=y)?

It is possible that Windows does ACPI hotplug differently, for example
it could add some delay somewhere and that is enough for the firmware to
get the bridges initialized properly where as in Linux we only see the
bridge when it is in the middle of the initialization or so. When it
works properly Linux should see all the PCI bridges configured properly
by the BIOS SMI handler.

Re: [PATCH 02/24] thunderbolt: Do not try to read UID if DROM offset is read as 0

[Andreas Noever]

On Mon, May 22, 2017 at 10:38 PM, Mika Westerberg
<mika.westerberg@linux.intel.com> wrote:
> On Mon, May 22, 2017 at 08:41:22PM +0200, Andreas Noever wrote:
>> Yes there is a check for the root switch, but also one that checks the
>> return code of tb_drom_read_uid_only :)
>>
>> err = tb_drom_read_uid_only(sw, &uid);
>> if (err) {
>>     tb_sw_warn(sw, "uid read failed\n");
>>     return err;
>> }
>> if (sw != sw->tb->root_switch && sw->uid != uid) {
>>
>>
>> The reason it works on the Mac is because drom_offset is not 0, so the
>> new branch in tb_drom_read_uid_only is not taken. Probably this is the
>> case for all Cactus Ridge models and Alpine Ridge doesn't go there
>> since it uses the ICM?
>
> Yes in case of ICM we don't call the function at all.
>
>> Still it wouldn't hurt to only read the uid if
>> sw != root_switch, the value is not used if sw == root_switch.
>
> I agree. I'll update the code so that it will only read and check UID
> when we are not dealing with the root switch.
Thanks,
Andreas

RE: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Mario.Limonciello]

> -----Original Message-----
> From: Bernat, Yehezkel [mailto:yehezkel.bernat@intel.com]
> Sent: Monday, May 22, 2017 3:11 PM
> To: Limonciello, Mario <Mario_Limonciello@Dell.com>;
> mika.westerberg@linux.intel.com
> Cc: gregkh@linuxfoundation.org; andreas.noever@gmail.com; Jamet, Michael
> <michael.jamet@intel.com>; lukas@wunner.de; Levy, Amir (Jer)
> <amir.jer.levy@intel.com>; luto@kernel.org; Dominguez, Jared
> <Jared_Dominguez@DELL.com>; andriy.shevchenko@linux.intel.com; linux-
> kernel@vger.kernel.org
> Subject: RE: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade
> 
> 
> 
> > -----Original Message-----
> > From: Mario.Limonciello@dell.com [mailto:Mario.Limonciello@dell.com]
> > Sent: Monday, May 22, 2017 23:08
> > To: mika.westerberg@linux.intel.com
> > Cc: gregkh@linuxfoundation.org; andreas.noever@gmail.com; Jamet,
> > Michael <michael.jamet@intel.com>; Bernat, Yehezkel
> > <yehezkel.bernat@intel.com>; lukas@wunner.de; Levy, Amir (Jer)
> > <amir.jer.levy@intel.com>; luto@kernel.org; Jared.Dominguez@dell.com;
> > andriy.shevchenko@linux.intel.com; linux-kernel@vger.kernel.org
> > Subject: RE: [PATCH 00/24] Thunderbolt security levels and NVM firmware
> > upgrade
> >
> > > -----Original Message-----
> > > From: Mika Westerberg [mailto:mika.westerberg@linux.intel.com]
> > > Sent: Monday, May 22, 2017 6:37 AM
> > > To: Limonciello, Mario <Mario_Limonciello@Dell.com>
> > > Cc: gregkh@linuxfoundation.org; andreas.noever@gmail.com;
> > > michael.jamet@intel.com; yehezkel.bernat@intel.com; lukas@wunner.de;
> > > amir.jer.levy@intel.com; luto@kernel.org; Dominguez, Jared
> > > <Jared_Dominguez@DELL.com>; andriy.shevchenko@linux.intel.com;
> > linux-
> > > kernel@vger.kernel.org
> > > Subject: Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware
> > upgrade
> > >
> > > On Sat, May 20, 2017 at 11:24:12AM +0300, Mika Westerberg wrote:
> > > > On Fri, May 19, 2017 at 05:54:37PM +0000, Mario.Limonciello@dell.com
> > wrote:
> > > > > >
> > > > > > It happens occasionally when you reboot the machine when a device
> > is
> > > > > > connected but seems to be dependent on the BIOS version. Since it is
> > the
> > > > > > BIOS who is supposed to enumerated these devices, I suspect that it
> > is
> > > > > > either problem in BIOS or our PCI enumeration code does something
> > wrong.
> > > > > >
> > > > >
> > > > > I'm fairly certain it's an issue somewhere with Linux PCI enumeration.  I
> > took
> > > > > the exact same HW and switched it out the SSD to one w/ Win10 1607.
> > > > > I set the dock and cable to "always allow" in TBT settings applet.
> > > > >
> > > > > I don't reproduce any problems with enumeration with the dock
> > plugged in
> > > > > on cold boot.  All the devices hanging off the bridge show up properly.
> > > >
> > > > You mean in Windows you don't reproduce the problem, right? Even
> > when
> > > > you reboot the machine with devices connected.
> > > >
> > > > I tried on Intel Skull Canyon NUC so that I disabled the thunderbolt
> > > > driver and after reboot (warm) I can see the PCI scan error about bus
> > > > being partially hidden behind a bridge.
> > > >
> > > > When this happens PCIe ports of the thunderbolt device/host seem to
> > be
> > > > unconfigured and Linux then decides to reconfigure them which leads to
> > > > the problem. When it works we get ACPI hotplug event to the PCIe root
> > > > port and the PCIe upstream/downstream ports are properly configured
> > by
> > > > the BIOS.
> > > >
> > > > I guess Windows does something differently here than what we do when
> > PCI
> > > > devices are enumerated.
> > >
> > > We discussed this with our BIOS/firmware people and there was a
> > firmware
> > > bug that caused many issues around hotplug and reboot flows. Is is
> > > possible for you to try with the latest BIOS and see if the issue
> > > reproduces (or are you already running the latest)?
> >
> > I was 1 version behind, but I double checked with the latest version (1.1.15)
> > and the same behavior exists on Linux (still works properly on Win10).
> >
> > If you have some more details about what the FW guys changed, I can check
> > with my Dell FW team if they've picked up the same fix.  I'm guessing it's not
> > the same problem though considering it works properly on Win10?
> 
> What about the TBT NVM version?
> NVM 16 sounds a bit old to me.

16 is the latest available for the XPS 9350.  I'd guess it's got to do with the stepping
of AR, but I don't know for sure.

Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Andy Shevchenko]

On Thu, 2017-05-18 at 17:38 +0300, Mika Westerberg wrote:
> Hi all,
> 
> This patch series adds support for Thunderbolt security levels, which
> were
> first introduced in Intel Falcon Ridge Thunderbolt controller, to
> prevent
> DMA attacks when PCIe is tunneled over Thunderbolt fabric. This is
> needed
> if there is no IOMMU available for various reasons.
> 
> Most PCs out there having Falcon Ridge or newer have security level
> set to
> "user" which means that user authorization is needed before PCIe
> tunnel is
> creaded (the PCIe device appears). This effectively means that without
> driver support the user needs to configure security level from BIOS to
> "none" to get Thunderbolt devices connected. With these patches the
> user
> can authorize devices using sysfs attributes like:
> 
>   # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized
> 
> In addition these patches add support for upgrading NVM firmware
> running on
> a host or device by running something like:
> 
>   # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-
> 0/nvm_non_active0/nvmem
>   # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate
> 
> This is documented with more details in patch [23/24].
> 
> This series is based on Amir's networking patches [1] but instead of
> splitting the functionality between kernel driver and userspace
> daemon, we
> take advantage of Linux driver core by converting the existing driver
> to
> expose a Linux bus (domain) and devices (switches). Notifications to
> the
> userspace about plugged/unplugged devices is handled by standard
> uevents
> when a device is added to/removed from the Thunderbolt bus.
> 
> Since thunderbolt device identification and authorization can be done
> directly through sysfs attributes there is no need for userspace
> daemon.
> However, there still should be an application that promps user for
> unknown
> devices and allows selecting between "single connect" and "connect
> always"
> keeping this information in a database or similar persistent storage.
> This
> patch series only provides mechanism for userspace applications to
> achieve
> that.
> 
> Where Internal Connection Manager (ICM) firmware is available and
> usable,
> we use it in the driver. This also includes newer Apple Macbooks with
> Alpine Ridge. For older Macbooks the driver works as before but in
> addition
> the Thunderbolt bus is available there as well (including possibility
> to
> upgrade NVM firmware of connected devices).
> 
> We are also in works of porting Amir's networking driver to work on
> top of
> the new Thunderbolt bus pretty much the same way firewire networking
> is
> currently done. In addition this makes is possible to introduce other
> protocols like a char device that allows userspace directly to
> communicate
> accross Thunderbolt domains.
> 
> Note for Macs the Linux native PCIe hotplug support does not work well
> with
> the Thunderbolt PCIe topologies where there is need to put all
> available
> resources to the PCIe downstream port where the PCIe chain is
> extended.
> This is something we need to fix. In the mean time is a way to work it
> around by passing "pci=hpbussize=10,hpmemsize=2M" or so to the kernel
> command line.
> 
> These patches use uuid_be from uuid.h but I've learned that there is a
> work
> to remove the type completely in favor of new uuid_t [2]. I'm not sure
> what
> to do regarding that because those patches are not yet in the
> mainline.

Looks like we may use uuid_be for now, though having a patch to switch
to uuid_t eventually.

I have commented few patches (some minor comments), other than that,
FWIW:
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>

> 
> [1] https://lkml.org/lkml/2016/11/9/341
> [2] http://git.infradead.org/users/hch/vfs.git/shortlog/refs/heads/uui
> d-types
> 
> Mika Westerberg (24):
>   thunderbolt: Use const buffer pointer in write operations
>   thunderbolt: Do not try to read UID if DROM offset is read as 0
>   thunderbolt: Do not warn about newer DROM versions
>   thunderbolt: Add MSI-X support
>   thunderbolt: Rework capability handling
>   thunderbolt: Introduce thunderbolt bus and connection manager
>   thunderbolt: Convert switch to a device
>   thunderbolt: Fail switch adding operation if reading DROM fails
>   thunderbolt: Do not fail if DROM data CRC32 is invalid
>   thunderbolt: Read vendor and device name from DROM
>   thunderbolt: Move control channel messages to tb_msgs.h
>   thunderbolt: Expose get_route() to other files
>   thunderbolt: Expose make_header() to other files
>   thunderbolt: Let the connection manager handle all notifications
>   thunderbolt: Rework control channel to be more reliable
>   thunderbolt: Add Thunderbolt 3 PCI IDs
>   thunderbolt: Add support for NHI mailbox
>   thunderbolt: Store Thunderbolt generation in the switch structure
>   thunderbolt: Add support for DMA configuration based mailbox
>   thunderbolt: Do not touch the hardware if the NHI is gone on resume
>   thunderbolt: Add support for Internal Connection Manager (ICM)
>   thunderbolt: Add support for host and device NVM firmware upgrade
>   thunderbolt: Add documentation how Thunderbolt bus can be used
>   MAINTAINERS: Add maintainers for Thunderbolt driver
> 
>  Documentation/ABI/testing/sysfs-bus-thunderbolt |  108 +++
>  Documentation/admin-guide/index.rst             |    1 +
>  Documentation/admin-guide/thunderbolt.rst       |  197 ++++
>  MAINTAINERS                                     |    3 +
>  drivers/thunderbolt/Kconfig                     |   13 +-
>  drivers/thunderbolt/Makefile                    |    2 +-
>  drivers/thunderbolt/cap.c                       |  169 ++--
>  drivers/thunderbolt/ctl.c                       |  655 +++++++++----
>  drivers/thunderbolt/ctl.h                       |  105 ++-
>  drivers/thunderbolt/dma_port.c                  |  524 +++++++++++
>  drivers/thunderbolt/dma_port.h                  |   34 +
>  drivers/thunderbolt/domain.c                    |  455 ++++++++++
>  drivers/thunderbolt/eeprom.c                    |   84 +-
>  drivers/thunderbolt/icm.c                       | 1098
> ++++++++++++++++++++++
>  drivers/thunderbolt/nhi.c                       |  302 +++++-
>  drivers/thunderbolt/nhi.h                       |   91 +-
>  drivers/thunderbolt/nhi_regs.h                  |   27 +
>  drivers/thunderbolt/switch.c                    | 1109
> +++++++++++++++++++++--
>  drivers/thunderbolt/tb.c                        |  237 ++---
>  drivers/thunderbolt/tb.h                        |  242 ++++-
>  drivers/thunderbolt/tb_msgs.h                   |  260 ++++++
>  drivers/thunderbolt/tb_regs.h                   |   31 +-
>  drivers/thunderbolt/tunnel_pci.c                |   17 +-
>  23 files changed, 5213 insertions(+), 551 deletions(-)
>  create mode 100644 Documentation/ABI/testing/sysfs-bus-thunderbolt
>  create mode 100644 Documentation/admin-guide/thunderbolt.rst
>  create mode 100644 drivers/thunderbolt/dma_port.c
>  create mode 100644 drivers/thunderbolt/dma_port.h
>  create mode 100644 drivers/thunderbolt/domain.c
>  create mode 100644 drivers/thunderbolt/icm.c
>  create mode 100644 drivers/thunderbolt/tb_msgs.h
> 

-- 
Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Intel Finland Oy

RE: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Mario.Limonciello]

> -----Original Message-----
> From: Mika Westerberg [mailto:mika.westerberg@linux.intel.com]
> Sent: Monday, May 22, 2017 3:49 PM
> To: Limonciello, Mario <Mario_Limonciello@Dell.com>
> Cc: gregkh@linuxfoundation.org; andreas.noever@gmail.com;
> michael.jamet@intel.com; yehezkel.bernat@intel.com; lukas@wunner.de;
> amir.jer.levy@intel.com; luto@kernel.org; Dominguez, Jared
> <Jared_Dominguez@DELL.com>; andriy.shevchenko@linux.intel.com; linux-
> kernel@vger.kernel.org
> Subject: Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade
> 
> On Mon, May 22, 2017 at 08:07:54PM +0000, Mario.Limonciello@dell.com wrote:
> > I was 1 version behind, but I double checked with the latest version (1.1.15)
> > and the same behavior exists on Linux (still works properly on Win10).
> >
> > If you have some more details about what the FW guys changed, I can check
> > with my Dell FW team if they've picked up the same fix.  I'm guessing it's not
> > the same problem though considering it works properly on Win10?
> 
> Can you send me full dmesg preferably so that you have acpiphp.dyndbg in
> the kernel command line (you also need to have CONFIG_DYNAMIC_DEBUG=y)?
> 
> It is possible that Windows does ACPI hotplug differently, for example
> it could add some delay somewhere and that is enough for the firmware to
> get the bridges initialized properly where as in Linux we only see the
> bridge when it is in the middle of the initialization or so. When it
> works properly Linux should see all the PCI bridges configured properly
> by the BIOS SMI handler.

(Sorry my email client is not going to wrap these at 80 columns)

system booted with dock plugged in from a cold boot
[    0.000000] Linux version 4.12.0-rc1+ (test@test-XPS-13-9350) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #6 SMP Fri May 19 03:18:11 CST 2017
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.12.0-rc1+ root=UUID=68ee3977-2c7b-4b1a-bd43-7ec5d9ceb631 ro quiet splash acpiphp.dyndbg=+p vt.handoff=7
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: xstate_offset[3]:  832, xstate_sizes[3]:   64
[    0.000000] x86/fpu: xstate_offset[4]:  896, xstate_sizes[4]:   64
[    0.000000] x86/fpu: Enabled xstate features 0x1f, context size is 960 bytes, using 'compacted' format.
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000000fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000001000-0x0000000000028fff] usable
[    0.000000] BIOS-e820: [mem 0x0000000000029000-0x000000000002bfff] reserved
[    0.000000] BIOS-e820: [mem 0x000000000002c000-0x0000000000057fff] usable
[    0.000000] BIOS-e820: [mem 0x0000000000058000-0x0000000000058fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000059000-0x000000000009ffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000039129fff] usable
[    0.000000] BIOS-e820: [mem 0x000000003912a000-0x000000003a8a4fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000003a8a5000-0x000000003b05cfff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000003b05d000-0x000000003b8e6fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000003b8e7000-0x000000005b057fff] usable
[    0.000000] BIOS-e820: [mem 0x000000005b058000-0x000000005b058fff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000005b059000-0x000000005b082fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000005b083000-0x000000005b0ddfff] usable
[    0.000000] BIOS-e820: [mem 0x000000005b0de000-0x000000005b8cefff] reserved
[    0.000000] BIOS-e820: [mem 0x000000005b8cf000-0x000000006fffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000070000000-0x0000000077ffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000078000000-0x00000000784e2fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000784e3000-0x0000000078511fff] ACPI data
[    0.000000] BIOS-e820: [mem 0x0000000078512000-0x0000000078585fff] type 20
[    0.000000] BIOS-e820: [mem 0x0000000078586000-0x0000000078586fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000078587000-0x0000000078589fff] usable
[    0.000000] BIOS-e820: [mem 0x000000007858a000-0x000000007858afff] reserved
[    0.000000] BIOS-e820: [mem 0x000000007858b000-0x00000000785fffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000480ffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] efi: EFI v2.40 by American Megatrends
[    0.000000] efi:  ACPI=0x784e3000  ACPI 2.0=0x784e3000  SMBIOS=0xf0000  ESRT=0x3b6ac598 
[    0.000000] SMBIOS 2.8 present.
[    0.000000] DMI: Dell Inc. XPS 13 9350/09JHRY, BIOS 1.4.17 05/10/2017
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0x481000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: write-back
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 0080000000 mask 7F80000000 uncachable
[    0.000000]   1 base 007C000000 mask 7FFC000000 uncachable
[    0.000000]   2 base 007A000000 mask 7FFE000000 uncachable
[    0.000000]   3 base 0079800000 mask 7FFF800000 uncachable
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000]   8 disabled
[    0.000000]   9 disabled
[    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- WT  
[    0.000000] e820: last_pfn = 0x78600 max_arch_pfn = 0x400000000
[    0.000000] esrt: Reserving ESRT space from 0x000000003b6ac598 to 0x000000003b6ac5d0.
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff9af1c0023000] 23000 size 24576
[    0.000000] Using GB pages for direct mapping
[    0.000000] BRK [0x2d3247000, 0x2d3247fff] PGTABLE
[    0.000000] BRK [0x2d3248000, 0x2d3248fff] PGTABLE
[    0.000000] BRK [0x2d3249000, 0x2d3249fff] PGTABLE
[    0.000000] BRK [0x2d324a000, 0x2d324afff] PGTABLE
[    0.000000] BRK [0x2d324b000, 0x2d324bfff] PGTABLE
[    0.000000] BRK [0x2d324c000, 0x2d324cfff] PGTABLE
[    0.000000] BRK [0x2d324d000, 0x2d324dfff] PGTABLE
[    0.000000] BRK [0x2d324e000, 0x2d324efff] PGTABLE
[    0.000000] BRK [0x2d324f000, 0x2d324ffff] PGTABLE
[    0.000000] BRK [0x2d3250000, 0x2d3250fff] PGTABLE
[    0.000000] BRK [0x2d3251000, 0x2d3251fff] PGTABLE
[    0.000000] Secure boot could not be determined
[    0.000000] RAMDISK: [mem 0x3048b000-0x32a5bfff]
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x00000000784E3000 000024 (v02 DELL  )
[    0.000000] ACPI: XSDT 0x00000000784E30B0 0000DC (v01 DELL   CBX3     01072009 AMI  00010013)
[    0.000000] ACPI: FACP 0x0000000078504170 00010C (v05 DELL   CBX3     01072009 AMI  00010013)
[    0.000000] ACPI: DSDT 0x00000000784E3218 020F53 (v02 DELL   CBX3     01072009 INTL 20120913)
[    0.000000] ACPI: FACS 0x000000003B05BF80 000040
[    0.000000] ACPI: APIC 0x0000000078504280 000084 (v03 DELL   CBX3     01072009 AMI  00010013)
[    0.000000] ACPI: FPDT 0x0000000078504308 000044 (v01 DELL   CBX3     01072009 AMI  00010013)
[    0.000000] ACPI: FIDT 0x0000000078504350 00009C (v01 DELL   CBX3     01072009 AMI  00010013)
[    0.000000] ACPI: MCFG 0x00000000785043F0 00003C (v01 DELL   CBX3     01072009 MSFT 00000097)
[    0.000000] ACPI: HPET 0x0000000078504430 000038 (v01 DELL   CBX3     01072009 AMI. 0005000B)
[    0.000000] ACPI: SSDT 0x0000000078504468 000315 (v01 SataRe SataTabl 00001000 INTL 20120913)
[    0.000000] ACPI: LPIT 0x0000000078504780 000094 (v01 INTEL  SKL-ULT  00000000 MSFT 0000005F)
[    0.000000] ACPI: SSDT 0x0000000078504818 000248 (v02 INTEL  sensrhub 00000000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x0000000078504A60 002BAE (v02 INTEL  PtidDevc 00001000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x0000000078507610 000BE3 (v02 INTEL  Ther_Rvp 00001000 INTL 20120913)
[    0.000000] ACPI: DBGP 0x00000000785081F8 000034 (v01 INTEL           00000000 MSFT 0000005F)
[    0.000000] ACPI: DBG2 0x0000000078508230 000054 (v00 INTEL           00000000 MSFT 0000005F)
[    0.000000] ACPI: SSDT 0x0000000078508288 000737 (v02 INTEL  xh_rvp07 00000000 INTL 20120913)
[    0.000000] ACPI: BOOT 0x00000000785089C0 000028 (v01 DELL   CBX3     01072009 AMI  00010013)
[    0.000000] ACPI: SSDT 0x00000000785089E8 0035AE (v02 SaSsdt SaSsdt   00003000 INTL 20120913)
[    0.000000] ACPI: UEFI 0x000000007850BF98 000042 (v01                 00000000      00000000)
[    0.000000] ACPI: SSDT 0x000000007850BFE0 000E73 (v02 CpuRef CpuSsdt  00003000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x000000007850CE58 003B57 (v02 DptfTa DptfTabl 00001000 INTL 20120913)
[    0.000000] ACPI: BGRT 0x00000000785109B0 000038 (v00 ?\xffffffadqy            01072009 AMI  00010013)
[    0.000000] ACPI: DMAR 0x00000000785109E8 0000F0 (v01 INTEL  SKL      00000001 INTL 00000001)
[    0.000000] ACPI: TPM2 0x0000000078510AD8 000034 (v03        Tpm2Tabl 00000001 AMI  00000000)
[    0.000000] ACPI: ASF! 0x0000000078510B10 0000A5 (v32 INTEL   HCG     00000001 TFSM 000F4240)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x0000000480ffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x480ffb000-0x480ffffff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x0000000480ffffff]
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x0000000000028fff]
[    0.000000]   node   0: [mem 0x000000000002c000-0x0000000000057fff]
[    0.000000]   node   0: [mem 0x0000000000059000-0x000000000009ffff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x0000000039129fff]
[    0.000000]   node   0: [mem 0x000000003b8e7000-0x000000005b057fff]
[    0.000000]   node   0: [mem 0x000000005b083000-0x000000005b0ddfff]
[    0.000000]   node   0: [mem 0x000000005b8cf000-0x000000006fffffff]
[    0.000000]   node   0: [mem 0x0000000078000000-0x00000000784e2fff]
[    0.000000]   node   0: [mem 0x0000000078587000-0x0000000078589fff]
[    0.000000]   node   0: [mem 0x000000007858b000-0x00000000785fffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x0000000480ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000480ffffff]
[    0.000000] On node 0 totalpages: 4121885
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 136 pages reserved
[    0.000000]   DMA zone: 3995 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 6935 pages used for memmap
[    0.000000]   DMA32 zone: 443778 pages, LIFO batch:31
[    0.000000]   Normal zone: 57408 pages used for memmap
[    0.000000]   Normal zone: 3674112 pages, LIFO batch:31
[    0.000000] Reserving Intel graphics memory at 0x000000007a000000-0x000000007dffffff
[    0.000000] ACPI: PM-Timer IO Port: 0x1808
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
[    0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-119
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[    0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.000000] PM: Registered nosave memory: [mem 0x00029000-0x0002bfff]
[    0.000000] PM: Registered nosave memory: [mem 0x00058000-0x00058fff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]
[    0.000000] PM: Registered nosave memory: [mem 0x3912a000-0x3a8a4fff]
[    0.000000] PM: Registered nosave memory: [mem 0x3a8a5000-0x3b05cfff]
[    0.000000] PM: Registered nosave memory: [mem 0x3b05d000-0x3b8e6fff]
[    0.000000] PM: Registered nosave memory: [mem 0x5b058000-0x5b058fff]
[    0.000000] PM: Registered nosave memory: [mem 0x5b059000-0x5b082fff]
[    0.000000] PM: Registered nosave memory: [mem 0x5b0de000-0x5b8cefff]
[    0.000000] PM: Registered nosave memory: [mem 0x70000000-0x77ffffff]
[    0.000000] PM: Registered nosave memory: [mem 0x784e3000-0x78511fff]
[    0.000000] PM: Registered nosave memory: [mem 0x78512000-0x78585fff]
[    0.000000] PM: Registered nosave memory: [mem 0x78586000-0x78586fff]
[    0.000000] PM: Registered nosave memory: [mem 0x7858a000-0x7858afff]
[    0.000000] PM: Registered nosave memory: [mem 0x78600000-0x79ffffff]
[    0.000000] PM: Registered nosave memory: [mem 0x7a000000-0x7dffffff]
[    0.000000] PM: Registered nosave memory: [mem 0x7e000000-0xdfffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xe0000000-0xefffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xf0000000-0xfdffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfe000000-0xfe010fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfe011000-0xfebfffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec00fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfec01000-0xfedfffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfee01000-0xfeffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xff000000-0xffffffff]
[    0.000000] e820: [mem 0x7e000000-0xdfffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:4 nr_node_ids:1
[    0.000000] percpu: Embedded 38 pages/cpu @ffff9af640c00000 s118424 r8192 d29032 u524288
[    0.000000] pcpu-alloc: s118424 r8192 d29032 u524288 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 2 3 
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 4057342
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.12.0-rc1+ root=UUID=68ee3977-2c7b-4b1a-bd43-7ec5d9ceb631 ro quiet splash acpiphp.dyndbg=+p vt.handoff=7
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Calgary: detecting Calgary via BIOS EBDA area
[    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[    0.000000] Memory: 16011668K/16487540K available (8877K kernel code, 1487K rwdata, 3748K rodata, 1628K init, 1148K bss, 475872K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] ftrace: allocating 36224 entries in 142 pages
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS:16640 nr_irqs:1024 16
[    0.000000] spurious 8259A interrupt: IRQ7.
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns
[    0.000000] hpet clockevent registered
[    0.004000] tsc: Detected 2200.000 MHz processor
[    0.004000] Calibrating delay loop (skipped), value calculated using timer frequency.. 4416.00 BogoMIPS (lpj=8832000)
[    0.004000] pid_max: default: 32768 minimum: 301
[    0.004000] ACPI: Core revision 20170303
[    0.038985] ACPI: 9 ACPI AML tables successfully acquired and loaded
[    0.039786] Security Framework initialized
[    0.039787] Yama: becoming mindful.
[    0.039800] AppArmor: AppArmor initialized
[    0.040660] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[    0.043850] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.045331] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.045344] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.045584] CPU: Physical Processor ID: 0
[    0.045585] CPU: Processor Core ID: 0
[    0.045589] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.045589] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.045594] mce: CPU supports 9 MCE banks
[    0.045604] CPU0: Thermal monitoring enabled (TM1)
[    0.045624] process: using mwait in idle threads
[    0.045626] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[    0.045627] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[    0.046002] Freeing SMP alternatives memory: 32K
[    0.050366] smpboot: Max logical packages: 2
[    0.050370] DMAR: Host address width 39
[    0.050371] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.050378] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[    0.050379] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.050383] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.050384] DMAR: RMRR base: 0x0000003a637000 end: 0x0000003a656fff
[    0.050386] DMAR: RMRR base: 0x00000079800000 end: 0x0000007dffffff
[    0.050387] DMAR: ANDD device: 1 name: \_SB.PCI0.I2C0
[    0.050387] DMAR: ANDD device: 2 name: \_SB.PCI0.I2C1
[    0.050389] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.050390] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.050390] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.050391] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.053247] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.053248] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.057184] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.096900] TSC deadline timer enabled
[    0.096904] smpboot: CPU0: Intel(R) Core(TM) i7-6560U CPU @ 2.20GHz (family: 0x6, model: 0x4e, stepping: 0x3)
[    0.096964] Performance Events: PEBS fmt3+, Skylake events, 32-deep LBR, full-width counters, Intel PMU driver.
[    0.096996] ... version:                4
[    0.096996] ... bit width:              48
[    0.096997] ... generic registers:      4
[    0.096998] ... value mask:             0000ffffffffffff
[    0.096998] ... max period:             00007fffffffffff
[    0.096999] ... fixed-purpose events:   3
[    0.096999] ... event mask:             000000070000000f
[    0.097774] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
[    0.097787] smp: Bringing up secondary CPUs ...
[    0.097849] x86: Booting SMP configuration:
[    0.097850] .... node  #0, CPUs:      #1 #2 #3
[    0.336037] smp: Brought up 1 node, 4 CPUs
[    0.336037] smpboot: Total of 4 processors activated (17667.62 BogoMIPS)
[    0.340105] sched_clock: Marking stable (340000000, 0)->(338107310, 1892690)
[    0.340525] devtmpfs: initialized
[    0.340571] x86/mm: Memory block size: 128MB
[    0.343214] evm: security.selinux
[    0.343215] evm: security.SMACK64
[    0.343216] evm: security.SMACK64EXEC
[    0.343216] evm: security.SMACK64TRANSMUTE
[    0.343217] evm: security.SMACK64MMAP
[    0.343217] evm: security.ima
[    0.343218] evm: security.capability
[    0.343287] PM: Registering ACPI NVS region [mem 0x3a8a5000-0x3b05cfff] (8093696 bytes)
[    0.343380] PM: Registering ACPI NVS region [mem 0x5b058000-0x5b058fff] (4096 bytes)
[    0.343433] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.343439] futex hash table entries: 1024 (order: 4, 65536 bytes)
[    0.343495] pinctrl core: initialized pinctrl subsystem
[    0.343659] RTC time: 17:22:36, date: 05/23/17
[    0.344884] NET: Registered protocol family 16
[    0.345080] cpuidle: using governor ladder
[    0.345110] cpuidle: using governor menu
[    0.345111] PCCT header not found.
[    0.345171] Simple Boot Flag at 0x47 set to 0x80
[    0.345201] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    0.345203] ACPI: bus type PCI registered
[    0.345204] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.345261] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[    0.345262] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820
[    0.345274] PCI: Using configuration type 1 for base access
[    0.345280] dmi type 0xB1 record - unknown flag
[    0.346590] HugeTLB registered 1 GB page size, pre-allocated 0 pages
[    0.346590] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.346785] ACPI: Added _OSI(Module Device)
[    0.346786] ACPI: Added _OSI(Processor Device)
[    0.346787] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.346787] ACPI: Added _OSI(Processor Aggregator Device)
[    0.347896] ACPI: Executed 24 blocks of module-level executable AML code
[    0.358829] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[    0.428675] ACPI: Dynamic OEM Table Load:
[    0.428682] ACPI: SSDT 0xFFFF9AF62E35B000 0005DC (v02 PmRef  Cpu0Ist  00003000 INTL 20120913)
[    0.428922] ACPI: \_PR_.CPU0: _OSC native thermal LVT Acked
[    0.430197] ACPI: Dynamic OEM Table Load:
[    0.430202] ACPI: SSDT 0xFFFF9AF62DD94C00 00037F (v02 PmRef  Cpu0Cst  00003001 INTL 20120913)
[    0.430480] ACPI: Dynamic OEM Table Load:
[    0.430484] ACPI: SSDT 0xFFFF9AF62DDBB3C0 00008E (v02 PmRef  Cpu0Hwp  00003000 INTL 20120913)
[    0.430672] ACPI: Dynamic OEM Table Load:
[    0.430676] ACPI: SSDT 0xFFFF9AF62DDF2000 000130 (v02 PmRef  HwpLvt   00003000 INTL 20120913)
[    0.431484] ACPI: Dynamic OEM Table Load:
[    0.431490] ACPI: SSDT 0xFFFF9AF62E35B800 0005AA (v02 PmRef  ApIst    00003000 INTL 20120913)
[    0.431970] ACPI: Dynamic OEM Table Load:
[    0.431974] ACPI: SSDT 0xFFFF9AF62DDF2200 000119 (v02 PmRef  ApHwp    00003000 INTL 20120913)
[    0.432245] ACPI: Dynamic OEM Table Load:
[    0.432249] ACPI: SSDT 0xFFFF9AF62DDF2400 000119 (v02 PmRef  ApCst    00003000 INTL 20120913)
[    0.434429] ACPI : EC: EC started
[    0.434430] ACPI : EC: interrupt blocked
[    0.437146] ACPI: \_SB_.PCI0.LPCB.ECDV: Used as first EC
[    0.437148] ACPI: \_SB_.PCI0.LPCB.ECDV: GPE=0x14, EC_CMD/EC_SC=0x934, EC_DATA=0x930
[    0.437150] ACPI: \_SB_.PCI0.LPCB.ECDV: Used as boot DSDT EC to handle transactions
[    0.437150] ACPI: Interpreter enabled
[    0.437191] ACPI: (supports S0 S3 S4 S5)
[    0.437192] ACPI: Using IOAPIC for interrupt routing
[    0.437227] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.439254] ACPI: Power Resource [PG00] (on)
[    0.439588] ACPI: Power Resource [PG01] (on)
[    0.439908] ACPI: Power Resource [PG02] (on)
[    0.441952] ACPI: Power Resource [WRST] (off)
[    0.442291] ACPI: Power Resource [WRST] (off)
[    0.442629] ACPI: Power Resource [WRST] (off)
[    0.442960] ACPI: Power Resource [WRST] (off)
[    0.443297] ACPI: Power Resource [WRST] (off)
[    0.443636] ACPI: Power Resource [WRST] (off)
[    0.443971] ACPI: Power Resource [WRST] (off)
[    0.444304] ACPI: Power Resource [WRST] (off)
[    0.444635] ACPI: Power Resource [WRST] (off)
[    0.444995] ACPI: Power Resource [WRST] (off)
[    0.445335] ACPI: Power Resource [WRST] (off)
[    0.445667] ACPI: Power Resource [WRST] (off)
[    0.445998] ACPI: Power Resource [WRST] (off)
[    0.446330] ACPI: Power Resource [WRST] (off)
[    0.446662] ACPI: Power Resource [WRST] (off)
[    0.446994] ACPI: Power Resource [WRST] (off)
[    0.447336] ACPI: Power Resource [WRST] (off)
[    0.447673] ACPI: Power Resource [WRST] (off)
[    0.448006] ACPI: Power Resource [WRST] (off)
[    0.448339] ACPI: Power Resource [WRST] (off)
[    0.461462] ACPI: Power Resource [FN00] (off)
[    0.461549] ACPI: Power Resource [FN01] (off)
[    0.461634] ACPI: Power Resource [FN02] (off)
[    0.461716] ACPI: Power Resource [FN03] (off)
[    0.461802] ACPI: Power Resource [FN04] (off)
[    0.462944] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-fe])
[    0.462949] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    0.462982] acpi PNP0A08:00: _OSC failed (AE_ERROR); disabling ASPM
[    0.463561] PCI host bridge to bus 0000:00
[    0.463563] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.463564] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.463565] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.463566] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000c3fff window]
[    0.463567] pci_bus 0000:00: root bus resource [mem 0x000c4000-0x000c7fff window]
[    0.463568] pci_bus 0000:00: root bus resource [mem 0x000c8000-0x000cbfff window]
[    0.463569] pci_bus 0000:00: root bus resource [mem 0x000cc000-0x000cffff window]
[    0.463570] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff window]
[    0.463571] pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff window]
[    0.463572] pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff window]
[    0.463573] pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff window]
[    0.463574] pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff window]
[    0.463575] pci_bus 0000:00: root bus resource [mem 0x000e8000-0x000ebfff window]
[    0.463576] pci_bus 0000:00: root bus resource [mem 0x000ec000-0x000effff window]
[    0.463577] pci_bus 0000:00: root bus resource [mem 0x7e000000-0xdfffffff window]
[    0.463578] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]
[    0.463579] pci_bus 0000:00: root bus resource [bus 00-fe]
[    0.463587] pci 0000:00:00.0: [8086:1904] type 00 class 0x060000
[    0.463706] pci 0000:00:02.0: [8086:1926] type 00 class 0x030000
[    0.463716] pci 0000:00:02.0: reg 0x10: [mem 0xdb000000-0xdbffffff 64bit]
[    0.463723] pci 0000:00:02.0: reg 0x18: [mem 0x90000000-0x9fffffff 64bit pref]
[    0.463727] pci 0000:00:02.0: reg 0x20: [io  0xf000-0xf03f]
[    0.463862] pci 0000:00:04.0: [8086:1903] type 00 class 0x118000
[    0.463873] pci 0000:00:04.0: reg 0x10: [mem 0xdc320000-0xdc327fff 64bit]
[    0.464071] pci 0000:00:14.0: [8086:9d2f] type 00 class 0x0c0330
[    0.464090] pci 0000:00:14.0: reg 0x10: [mem 0xdc310000-0xdc31ffff 64bit]
[    0.464163] pci 0000:00:14.0: PME# supported from D3hot D3cold
[    0.464269] pci 0000:00:14.0: System wakeup disabled by ACPI
[    0.464300] pci 0000:00:14.2: [8086:9d31] type 00 class 0x118000
[    0.464320] pci 0000:00:14.2: reg 0x10: [mem 0xdc338000-0xdc338fff 64bit]
[    0.464571] pci 0000:00:15.0: [8086:9d60] type 00 class 0x118000
[    0.464779] pci 0000:00:15.0: reg 0x10: [mem 0xdc337000-0xdc337fff 64bit]
[    0.465740] pci 0000:00:15.1: [8086:9d61] type 00 class 0x118000
[    0.465948] pci 0000:00:15.1: reg 0x10: [mem 0xdc336000-0xdc336fff 64bit]
[    0.466843] pci 0000:00:16.0: [8086:9d3a] type 00 class 0x078000
[    0.466866] pci 0000:00:16.0: reg 0x10: [mem 0xdc335000-0xdc335fff 64bit]
[    0.466934] pci 0000:00:16.0: PME# supported from D3hot
[    0.467072] pci 0000:00:17.0: [8086:9d03] type 00 class 0x010601
[    0.467089] pci 0000:00:17.0: reg 0x10: [mem 0xdc330000-0xdc331fff]
[    0.467098] pci 0000:00:17.0: reg 0x14: [mem 0xdc334000-0xdc3340ff]
[    0.467106] pci 0000:00:17.0: reg 0x18: [io  0xf090-0xf097]
[    0.467116] pci 0000:00:17.0: reg 0x1c: [io  0xf080-0xf083]
[    0.467126] pci 0000:00:17.0: reg 0x20: [io  0xf060-0xf07f]
[    0.467135] pci 0000:00:17.0: reg 0x24: [mem 0xdc333000-0xdc3337ff]
[    0.467182] pci 0000:00:17.0: PME# supported from D3hot
[    0.467319] pci 0000:00:1c.0: [8086:9d10] type 01 class 0x060400
[    0.467389] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[    0.467513] pci 0000:00:1c.0: System wakeup disabled by ACPI
[    0.467551] pci 0000:00:1c.4: [8086:9d14] type 01 class 0x060400
[    0.467622] pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold
[    0.467743] pci 0000:00:1c.4: System wakeup disabled by ACPI
[    0.467775] pci 0000:00:1c.5: [8086:9d15] type 01 class 0x060400
[    0.467844] pci 0000:00:1c.5: PME# supported from D0 D3hot D3cold
[    0.467963] pci 0000:00:1c.5: System wakeup disabled by ACPI
[    0.467999] pci 0000:00:1d.0: [8086:9d18] type 01 class 0x060400
[    0.468076] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
[    0.468198] pci 0000:00:1d.0: System wakeup disabled by ACPI
[    0.468255] pci 0000:00:1f.0: [8086:9d48] type 00 class 0x060100
[    0.468481] pci 0000:00:1f.2: [8086:9d21] type 00 class 0x058000
[    0.468491] pci 0000:00:1f.2: reg 0x10: [mem 0xdc32c000-0xdc32ffff]
[    0.468651] pci 0000:00:1f.3: [8086:9d70] type 00 class 0x040380
[    0.468674] pci 0000:00:1f.3: reg 0x10: [mem 0xdc328000-0xdc32bfff 64bit]
[    0.468700] pci 0000:00:1f.3: reg 0x20: [mem 0xdc300000-0xdc30ffff 64bit]
[    0.468750] pci 0000:00:1f.3: PME# supported from D3hot D3cold
[    0.468912] pci 0000:00:1f.3: System wakeup disabled by ACPI
[    0.468945] pci 0000:00:1f.4: [8086:9d23] type 00 class 0x0c0500
[    0.468992] pci 0000:00:1f.4: reg 0x10: [mem 0xdc332000-0xdc3320ff 64bit]
[    0.469063] pci 0000:00:1f.4: reg 0x20: [io  0xf040-0xf05f]
[    0.469363] pci 0000:01:00.0: [8086:1576] type 01 class 0x060400
[    0.469483] pci 0000:01:00.0: supports D1 D2
[    0.469484] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.469570] pci 0000:01:00.0: System wakeup disabled by ACPI
[    0.469609] pci 0000:00:1c.0: PCI bridge to [bus 01-39]
[    0.469614] pci 0000:00:1c.0:   bridge window [mem 0xc4000000-0xda0fffff]
[    0.469618] pci 0000:00:1c.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[    0.469621] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.469670] pci_bus 0000:02: busn_res: can not insert [bus 02-ff] under [bus 01-39] (conflicts with (null) [bus 01-39])
[    0.469688] pci 0000:02:00.0: [8086:1576] type 01 class 0x060400
[    0.469809] pci 0000:02:00.0: supports D1 D2
[    0.469810] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.469877] pci 0000:02:01.0: [8086:1576] type 01 class 0x060400
[    0.470000] pci 0000:02:01.0: supports D1 D2
[    0.470001] pci 0000:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.470067] pci 0000:02:02.0: [8086:1576] type 01 class 0x060400
[    0.470188] pci 0000:02:02.0: supports D1 D2
[    0.470189] pci 0000:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.470277] pci 0000:01:00.0: PCI bridge to [bus 02-ff]
[    0.470283] pci 0000:01:00.0:   bridge window [io  0x0000-0x0fff]
[    0.470287] pci 0000:01:00.0:   bridge window [mem 0x00000000-0x000fffff]
[    0.470294] pci 0000:01:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[    0.470296] pci 0000:02:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.470304] pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.470312] pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.470376] pci 0000:03:00.0: [8086:1575] type 00 class 0x088000
[    0.470401] pci 0000:03:00.0: reg 0x10: [mem 0x00000000-0x0003ffff]
[    0.470410] pci 0000:03:00.0: reg 0x14: [mem 0x00000000-0x00000fff]
[    0.470552] pci 0000:03:00.0: supports D1 D2
[    0.470552] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.470667] pci 0000:02:00.0: PCI bridge to [bus 03-ff]
[    0.470674] pci 0000:02:00.0:   bridge window [io  0x0000-0x0fff]
[    0.470677] pci 0000:02:00.0:   bridge window [mem 0x00000000-0x000fffff]
[    0.470683] pci 0000:02:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[    0.470685] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
[    0.470733] pci 0000:02:01.0: PCI bridge to [bus 04-ff]
[    0.470740] pci 0000:02:01.0:   bridge window [io  0x0000-0x0fff]
[    0.470743] pci 0000:02:01.0:   bridge window [mem 0x00000000-0x000fffff]
[    0.470750] pci 0000:02:01.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[    0.470751] pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
[    0.470797] pci 0000:02:02.0: PCI bridge to [bus 05-ff]
[    0.470803] pci 0000:02:02.0:   bridge window [io  0x0000-0x0fff]
[    0.470806] pci 0000:02:02.0:   bridge window [mem 0x00000000-0x000fffff]
[    0.470812] pci 0000:02:02.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[    0.470813] pci_bus 0000:05: busn_res: [bus 05-ff] end is updated to 05
[    0.470817] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 05
[    0.470950] pci 0000:3a:00.0: [8086:24f3] type 00 class 0x028000
[    0.471003] pci 0000:3a:00.0: reg 0x10: [mem 0xdc200000-0xdc201fff 64bit]
[    0.471192] pci 0000:3a:00.0: PME# supported from D0 D3hot D3cold
[    0.471306] pci 0000:3a:00.0: System wakeup disabled by ACPI
[    0.481202] pci 0000:00:1c.4: PCI bridge to [bus 3a]
[    0.481207] pci 0000:00:1c.4:   bridge window [mem 0xdc200000-0xdc2fffff]
[    0.481320] pci 0000:3b:00.0: [10ec:525a] type 00 class 0xff0000
[    0.481354] pci 0000:3b:00.0: reg 0x14: [mem 0xdc100000-0xdc100fff]
[    0.481475] pci 0000:3b:00.0: supports D1 D2
[    0.481476] pci 0000:3b:00.0: PME# supported from D1 D2 D3hot D3cold
[    0.481554] pci 0000:3b:00.0: System wakeup disabled by ACPI
[    0.493194] pci 0000:00:1c.5: PCI bridge to [bus 3b]
[    0.493199] pci 0000:00:1c.5:   bridge window [mem 0xdc100000-0xdc1fffff]
[    0.494026] pci 0000:3c:00.0: [1179:010f] type 00 class 0x010802
[    0.494049] pci 0000:3c:00.0: reg 0x10: [mem 0xdc000000-0xdc003fff 64bit]
[    0.494220] pci 0000:3c:00.0: System wakeup disabled by ACPI
[    0.509479] pci 0000:00:1d.0: PCI bridge to [bus 3c]
[    0.509483] pci 0000:00:1d.0:   bridge window [mem 0xdc000000-0xdc0fffff]
[    0.511880] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
[    0.511931] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
[    0.511979] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 10 *11 12 14 15)
[    0.512027] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
[    0.512074] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 10 *11 12 14 15)
[    0.512122] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
[    0.512169] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 10 *11 12 14 15)
[    0.512216] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 10 *11 12 14 15)
[    0.514209] ACPI: Enabled 6 GPEs in block 00 to 7F
[    0.514331] ACPI : EC: interrupt unblocked
[    0.514343] ACPI : EC: event unblocked
[    0.514368] ACPI: \_SB_.PCI0.LPCB.ECDV: GPE=0x14, EC_CMD/EC_SC=0x934, EC_DATA=0x930
[    0.514370] ACPI: \_SB_.PCI0.LPCB.ECDV: Used as boot DSDT EC to handle transactions and events
[    0.514467] pci 0000:00:02.0: vgaarb: setting as boot VGA device
[    0.514469] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    0.514473] pci 0000:00:02.0: vgaarb: bridge control possible
[    0.514474] vgaarb: loaded
[    0.514756] SCSI subsystem initialized
[    0.514796] libata version 3.00 loaded.
[    0.514810] ACPI: bus type USB registered
[    0.514822] usbcore: registered new interface driver usbfs
[    0.514828] usbcore: registered new interface driver hub
[    0.514841] usbcore: registered new device driver usb
[    0.514880] EDAC MC: Ver: 3.0.0
[    0.516064] Registered efivars operations
[    0.524119] PCI: Using ACPI for IRQ routing
[    0.548741] PCI: pci_cache_line_size set to 64 bytes
[    0.549854] e820: reserve RAM buffer [mem 0x00029000-0x0002ffff]
[    0.549855] e820: reserve RAM buffer [mem 0x00058000-0x0005ffff]
[    0.549856] e820: reserve RAM buffer [mem 0x3912a000-0x3bffffff]
[    0.549857] e820: reserve RAM buffer [mem 0x5b058000-0x5bffffff]
[    0.549858] e820: reserve RAM buffer [mem 0x5b0de000-0x5bffffff]
[    0.549859] e820: reserve RAM buffer [mem 0x784e3000-0x7bffffff]
[    0.549861] e820: reserve RAM buffer [mem 0x7858a000-0x7bffffff]
[    0.549862] e820: reserve RAM buffer [mem 0x78600000-0x7bffffff]
[    0.549863] e820: reserve RAM buffer [mem 0x481000000-0x483ffffff]
[    0.549942] NetLabel: Initializing
[    0.549942] NetLabel:  domain hash size = 128
[    0.549943] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    0.549960] NetLabel:  unlabeled traffic allowed by default
[    0.550082] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[    0.550086] hpet0: 8 comparators, 64-bit 24.000000 MHz counter
[    0.552139] clocksource: Switched to clocksource hpet
[    0.560545] VFS: Disk quotas dquot_6.6.0
[    0.560559] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.560630] AppArmor: AppArmor Filesystem Enabled
[    0.560662] pnp: PnP ACPI init
[    0.560897] system 00:00: [io  0x0680-0x069f] has been reserved
[    0.560898] system 00:00: [io  0xffff] has been reserved
[    0.560899] system 00:00: [io  0xffff] has been reserved
[    0.560900] system 00:00: [io  0xffff] has been reserved
[    0.560902] system 00:00: [io  0x1800-0x18fe] has been reserved
[    0.560903] system 00:00: [io  0x164e-0x164f] has been reserved
[    0.560906] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.560997] pnp 00:01: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.561034] system 00:02: [io  0x1854-0x1857] has been reserved
[    0.561036] system 00:02: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active)
[    0.561123] pnp 00:03: Plug and Play ACPI device, IDs PNP0303 (active)
[    0.561139] pnp 00:04: Plug and Play ACPI device, IDs DLL0704 PNP0f13 (active)
[    0.561349] system 00:05: [mem 0xfed10000-0xfed17fff] has been reserved
[    0.561350] system 00:05: [mem 0xfed18000-0xfed18fff] has been reserved
[    0.561351] system 00:05: [mem 0xfed19000-0xfed19fff] has been reserved
[    0.561353] system 00:05: [mem 0xe0000000-0xefffffff] has been reserved
[    0.561354] system 00:05: [mem 0xfed20000-0xfed3ffff] has been reserved
[    0.561355] system 00:05: [mem 0xfed90000-0xfed93fff] could not be reserved
[    0.561357] system 00:05: [mem 0xfed45000-0xfed8ffff] has been reserved
[    0.561358] system 00:05: [mem 0xff000000-0xffffffff] has been reserved
[    0.561359] system 00:05: [mem 0xfee00000-0xfeefffff] could not be reserved
[    0.561361] system 00:05: [mem 0xdffe0000-0xdfffffff] has been reserved
[    0.561363] system 00:05: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.561403] system 00:06: [mem 0xfd000000-0xfdabffff] has been reserved
[    0.561404] system 00:06: [mem 0xfdad0000-0xfdadffff] has been reserved
[    0.561405] system 00:06: [mem 0xfdb00000-0xfdffffff] has been reserved
[    0.561407] system 00:06: [mem 0xfe000000-0xfe01ffff] could not be reserved
[    0.561408] system 00:06: [mem 0xfe036000-0xfe03bfff] has been reserved
[    0.561409] system 00:06: [mem 0xfe03d000-0xfe3fffff] has been reserved
[    0.561411] system 00:06: [mem 0xfe410000-0xfe7fffff] has been reserved
[    0.561412] system 00:06: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.561697] system 00:07: [io  0xff00-0xfffe] has been reserved
[    0.561699] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.562750] system 00:08: [mem 0xfe029000-0xfe029fff] has been reserved
[    0.562751] system 00:08: [mem 0xfe028000-0xfe028fff] has been reserved
[    0.562753] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.565930] pnp: PnP ACPI: found 9 devices
[    0.573478] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.573503] pci 0000:02:01.0: bridge window [io  0x1000-0x0fff] to [bus 04] add_size 1000
[    0.573505] pci 0000:02:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 04] add_size 200000 add_align 100000
[    0.573507] pci 0000:02:01.0: bridge window [mem 0x00100000-0x000fffff] to [bus 04] add_size 200000 add_align 100000
[    0.573525] pci 0000:01:00.0: bridge window [io  0x1000-0x0fff] to [bus 02-05] add_size 1000
[    0.573526] pci 0000:01:00.0: bridge window [mem 0x00100000-0x001fffff 64bit pref] to [bus 02-05] add_size 200000 add_align 100000
[    0.573528] pci 0000:01:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 02-05] add_size 200000 add_align 100000
[    0.573535] pci 0000:00:1c.0: bridge window [io  0x1000-0x0fff] to [bus 01-39] add_size 1000
[    0.573557] pci 0000:00:1c.0: BAR 13: assigned [io  0x2000-0x2fff]
[    0.573561] pci 0000:01:00.0: BAR 14: assigned [mem 0xc4000000-0xc43fffff]
[    0.573563] pci 0000:01:00.0: BAR 15: assigned [mem 0xa0000000-0xa02fffff 64bit pref]
[    0.573564] pci 0000:01:00.0: BAR 13: assigned [io  0x2000-0x2fff]
[    0.573567] pci 0000:02:00.0: BAR 14: assigned [mem 0xc4000000-0xc40fffff]
[    0.573568] pci 0000:02:01.0: BAR 14: assigned [mem 0xc4100000-0xc42fffff]
[    0.573570] pci 0000:02:01.0: BAR 15: assigned [mem 0xa0000000-0xa01fffff 64bit pref]
[    0.573571] pci 0000:02:01.0: BAR 13: assigned [io  0x2000-0x2fff]
[    0.573573] pci 0000:03:00.0: BAR 0: assigned [mem 0xc4000000-0xc403ffff]
[    0.573578] pci 0000:03:00.0: BAR 1: assigned [mem 0xc4040000-0xc4040fff]
[    0.573582] pci 0000:02:00.0: PCI bridge to [bus 03]
[    0.573588] pci 0000:02:00.0:   bridge window [mem 0xc4000000-0xc40fffff]
[    0.573597] pci 0000:02:01.0: PCI bridge to [bus 04]
[    0.573599] pci 0000:02:01.0:   bridge window [io  0x2000-0x2fff]
[    0.573604] pci 0000:02:01.0:   bridge window [mem 0xc4100000-0xc42fffff]
[    0.573608] pci 0000:02:01.0:   bridge window [mem 0xa0000000-0xa01fffff 64bit pref]
[    0.573614] pci 0000:02:02.0: PCI bridge to [bus 05]
[    0.573627] pci 0000:01:00.0: PCI bridge to [bus 02-05]
[    0.573629] pci 0000:01:00.0:   bridge window [io  0x2000-0x2fff]
[    0.573634] pci 0000:01:00.0:   bridge window [mem 0xc4000000-0xc43fffff]
[    0.573638] pci 0000:01:00.0:   bridge window [mem 0xa0000000-0xa02fffff 64bit pref]
[    0.573644] pci 0000:00:1c.0: PCI bridge to [bus 01-39]
[    0.573646] pci 0000:00:1c.0:   bridge window [io  0x2000-0x2fff]
[    0.573649] pci 0000:00:1c.0:   bridge window [mem 0xc4000000-0xda0fffff]
[    0.573652] pci 0000:00:1c.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[    0.573656] pci 0000:00:1c.4: PCI bridge to [bus 3a]
[    0.573660] pci 0000:00:1c.4:   bridge window [mem 0xdc200000-0xdc2fffff]
[    0.573666] pci 0000:00:1c.5: PCI bridge to [bus 3b]
[    0.573669] pci 0000:00:1c.5:   bridge window [mem 0xdc100000-0xdc1fffff]
[    0.573676] pci 0000:00:1d.0: PCI bridge to [bus 3c]
[    0.573679] pci 0000:00:1d.0:   bridge window [mem 0xdc000000-0xdc0fffff]
[    0.573687] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    0.573688] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.573689] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    0.573690] pci_bus 0000:00: resource 7 [mem 0x000c0000-0x000c3fff window]
[    0.573691] pci_bus 0000:00: resource 8 [mem 0x000c4000-0x000c7fff window]
[    0.573692] pci_bus 0000:00: resource 9 [mem 0x000c8000-0x000cbfff window]
[    0.573693] pci_bus 0000:00: resource 10 [mem 0x000cc000-0x000cffff window]
[    0.573694] pci_bus 0000:00: resource 11 [mem 0x000d0000-0x000d3fff window]
[    0.573695] pci_bus 0000:00: resource 12 [mem 0x000d4000-0x000d7fff window]
[    0.573696] pci_bus 0000:00: resource 13 [mem 0x000d8000-0x000dbfff window]
[    0.573697] pci_bus 0000:00: resource 14 [mem 0x000dc000-0x000dffff window]
[    0.573698] pci_bus 0000:00: resource 15 [mem 0x000e4000-0x000e7fff window]
[    0.573699] pci_bus 0000:00: resource 16 [mem 0x000e8000-0x000ebfff window]
[    0.573700] pci_bus 0000:00: resource 17 [mem 0x000ec000-0x000effff window]
[    0.573701] pci_bus 0000:00: resource 18 [mem 0x7e000000-0xdfffffff window]
[    0.573702] pci_bus 0000:00: resource 19 [mem 0xfd000000-0xfe7fffff window]
[    0.573703] pci_bus 0000:01: resource 0 [io  0x2000-0x2fff]
[    0.573704] pci_bus 0000:01: resource 1 [mem 0xc4000000-0xda0fffff]
[    0.573705] pci_bus 0000:01: resource 2 [mem 0xa0000000-0xc1ffffff 64bit pref]
[    0.573706] pci_bus 0000:02: resource 0 [io  0x2000-0x2fff]
[    0.573707] pci_bus 0000:02: resource 1 [mem 0xc4000000-0xc43fffff]
[    0.573708] pci_bus 0000:02: resource 2 [mem 0xa0000000-0xa02fffff 64bit pref]
[    0.573709] pci_bus 0000:03: resource 1 [mem 0xc4000000-0xc40fffff]
[    0.573710] pci_bus 0000:04: resource 0 [io  0x2000-0x2fff]
[    0.573711] pci_bus 0000:04: resource 1 [mem 0xc4100000-0xc42fffff]
[    0.573712] pci_bus 0000:04: resource 2 [mem 0xa0000000-0xa01fffff 64bit pref]
[    0.573713] pci_bus 0000:3a: resource 1 [mem 0xdc200000-0xdc2fffff]
[    0.573714] pci_bus 0000:3b: resource 1 [mem 0xdc100000-0xdc1fffff]
[    0.573715] pci_bus 0000:3c: resource 1 [mem 0xdc000000-0xdc0fffff]
[    0.573853] NET: Registered protocol family 2
[    0.574011] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.574200] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.574306] TCP: Hash tables configured (established 131072 bind 65536)
[    0.574333] UDP hash table entries: 8192 (order: 6, 262144 bytes)
[    0.574383] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
[    0.574443] NET: Registered protocol family 1
[    0.574455] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    0.575736] PCI: CLS 0 bytes, default 64
[    0.575779] Trying to unpack rootfs image as initramfs...
[    1.239415] Freeing initrd memory: 38724K
[    1.239442] DMAR: ACPI device "device:64" under DMAR at fed91000 as 00:15.0
[    1.239445] DMAR: ACPI device "device:65" under DMAR at fed91000 as 00:15.1
[    1.239462] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.239465] software IO TLB [mem 0x6c000000-0x70000000] (64MB) mapped at [ffff9af22c000000-ffff9af22fffffff]
[    1.239603] RAPL PMU: API unit is 2^-32 Joules, 5 fixed counters, 655360 ms ovfl timer
[    1.239603] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules
[    1.239604] RAPL PMU: hw unit of domain package 2^-14 Joules
[    1.239605] RAPL PMU: hw unit of domain dram 2^-14 Joules
[    1.239605] RAPL PMU: hw unit of domain pp1-gpu 2^-14 Joules
[    1.239606] RAPL PMU: hw unit of domain psys 2^-14 Joules
[    1.239826] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1fd3b81b95f, max_idle_ns: 440795257325 ns
[    1.239935] Scanning for low memory corruption every 60 seconds
[    1.240271] audit: initializing netlink subsys (disabled)
[    1.240345] audit: type=2000 audit(1495560157.236:1): state=initialized audit_enabled=0 res=1
[    1.240736] Initialise system trusted keyrings
[    1.240802] workingset: timestamp_bits=40 max_order=22 bucket_order=0
[    1.241762] zbud: loaded
[    1.242121] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.242266] fuse init (API version 7.26)
[    1.243589] Key type asymmetric registered
[    1.243590] Asymmetric key parser 'x509' registered
[    1.243603] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    1.243655] io scheduler noop registered
[    1.243656] io scheduler deadline registered (default)
[    1.243683] io scheduler cfq registered
[    1.243684] io scheduler mq-deadline registered
[    1.243685] io scheduler kyber registered
[    1.244395] pcieport 0000:01:00.0: enabling device (0000 -> 0003)
[    1.244568] pcieport 0000:02:00.0: enabling device (0000 -> 0002)
[    1.244725] pcieport 0000:02:01.0: enabling device (0000 -> 0003)
[    1.245089] efifb: probing for efifb
[    1.245098] efifb: framebuffer at 0x90000000, using 22528k, total 22528k
[    1.245099] efifb: mode is 3200x1800x32, linelength=12800, pages=1
[    1.245100] efifb: scrolling: redraw
[    1.245101] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[    1.256945] Console: switching to colour frame buffer device 400x112
[    1.268631] fb0: EFI VGA frame buffer device
[    1.268637] intel_idle: MWAIT substates: 0x11142120
[    1.268638] intel_idle: v0.4.1 model 0x4E
[    1.268837] intel_idle: lapic_timer_reliable_states 0xffffffff
[    1.269453] ACPI: AC Adapter [AC] (on-line)
[    1.270030] input: Lid Switch as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0D:00/input/input0
[    1.270059] ACPI: Lid Switch [LID0]
[    1.270085] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input1
[    1.270094] ACPI: Power Button [PBTN]
[    1.270118] input: Sleep Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0E:00/input/input2
[    1.270126] ACPI: Sleep Button [SBTN]
[    1.270152] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3
[    1.270159] ACPI: Power Button [PWRF]
[    1.330529] (NULL device *): hwmon_device_register() is deprecated. Please convert the driver to use hwmon_device_register_with_info().
[    1.331081] thermal LNXTHERM:00: registered as thermal_zone0
[    1.331082] ACPI: Thermal Zone [THM] (25 C)
[    1.388530] thermal LNXTHERM:01: registered as thermal_zone1
[    1.388532] ACPI: Thermal Zone [TZ00] (28 C)
[    1.388695] thermal LNXTHERM:02: registered as thermal_zone2
[    1.388696] ACPI: Thermal Zone [TZ01] (30 C)
[    1.388759] GHES: HEST is not enabled!
[    1.388917] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    1.391461] Linux agpgart interface v0.103
[    1.411498] ACPI: Battery Slot [BAT0] (battery present)
[    1.440354] tpm_tis MSFT0101:00: 2.0 TPM (device-id 0xFE, rev-id 2)
[    2.240673] clocksource: Switched to clocksource tsc
[    2.532420] brd: module loaded
[    2.533569] loop: module loaded
[    2.533748] libphy: Fixed MDIO Bus: probed
[    2.533749] tun: Universal TUN/TAP device driver, 1.6
[    2.533781] PPP generic driver version 2.4.2
[    2.533810] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.533812] ehci-pci: EHCI PCI platform driver
[    2.533821] ehci-platform: EHCI generic platform driver
[    2.533832] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    2.533833] ohci-pci: OHCI PCI platform driver
[    2.533840] ohci-platform: OHCI generic platform driver
[    2.533845] uhci_hcd: USB Universal Host Controller Interface driver
[    2.534021] xhci_hcd 0000:00:14.0: xHCI Host Controller
[    2.534026] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1
[    2.535120] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x00109810
[    2.535124] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported
[    2.535210] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    2.535212] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.535213] usb usb1: Product: xHCI Host Controller
[    2.535214] usb usb1: Manufacturer: Linux 4.12.0-rc1+ xhci-hcd
[    2.535215] usb usb1: SerialNumber: 0000:00:14.0
[    2.535291] hub 1-0:1.0: USB hub found
[    2.535305] hub 1-0:1.0: 12 ports detected
[    2.543039] xhci_hcd 0000:00:14.0: xHCI Host Controller
[    2.543042] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2
[    2.543068] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003
[    2.543069] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.543070] usb usb2: Product: xHCI Host Controller
[    2.543071] usb usb2: Manufacturer: Linux 4.12.0-rc1+ xhci-hcd
[    2.543072] usb usb2: SerialNumber: 0000:00:14.0
[    2.543173] hub 2-0:1.0: USB hub found
[    2.543184] hub 2-0:1.0: 6 ports detected
[    2.545743] usb: port power management may be unreliable
[    2.547140] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
[    2.548832] i8042: Warning: Keylock active
[    2.551163] serio: i8042 KBD port at 0x60,0x64 irq 1
[    2.551166] serio: i8042 AUX port at 0x60,0x64 irq 12
[    2.551319] mousedev: PS/2 mouse device common for all mice
[    2.551598] rtc_cmos 00:01: RTC can wake from S4
[    2.552022] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
[    2.552137] rtc_cmos 00:01: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
[    2.552144] i2c /dev entries driver
[    2.552183] device-mapper: uevent: version 1.0.3
[    2.552258] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com
[    2.552261] intel_pstate: Intel P-state driver initializing
[    2.552594] intel_pstate: HWP enabled
[    2.552613] ledtrig-cpu: registered to indicate activity on CPUs
[    2.552614] EFI Variables Facility v0.08 2004-May-17
[    2.552948] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input4
[    2.565266] NET: Registered protocol family 10
[    2.565416] Segment Routing with IPv6
[    2.565424] NET: Registered protocol family 17
[    2.565428] Key type dns_resolver registered
[    2.565613] microcode: sig=0x406e3, pf=0x40, revision=0x9e
[    2.565639] microcode: Microcode Update Driver: v2.2.
[    2.565695] registered taskstats version 1
[    2.565697] Loading compiled-in X.509 certificates
[    2.567386] Loaded X.509 cert 'Build time autogenerated kernel key: e2ba32e95617902b08eb6094213f0c2a1bd3f0b5'
[    2.567405] zswap: loaded using pool lzo/zbud
[    2.568439] Key type trusted registered
[    2.570524] Key type encrypted registered
[    2.570526] AppArmor: AppArmor sha1 policy hashing enabled
[    2.868321] usb 1-3: new full-speed USB device number 2 using xhci_hcd
[    2.888395] evm: HMAC attrs: 0x1
[    2.890103]   Magic number: 1:349:391
[    2.890122] thermal cooling_device6: hash matches
[    2.890147] acpi XXXX0000:01: hash matches
[    2.890307] rtc_cmos 00:01: setting system clock to 2017-05-23 17:22:39 UTC (1495560159)
[    2.890527] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[    2.890528] EDD information not available.
[    2.890564] PM: Hibernation image not present or could not be loaded.
[    2.891653] Freeing unused kernel memory: 1628K
[    2.891654] Write protecting the kernel read-only data: 14336k
[    2.891916] Freeing unused kernel memory: 1348K
[    2.892408] Freeing unused kernel memory: 348K
[    2.901477] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    2.901528] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    2.901535] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    2.901544] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    2.901849] random: udevadm: uninitialized urandom read (16 bytes read)
[    2.901867] random: udevadm: uninitialized urandom read (16 bytes read)
[    2.902824] random: udevadm: uninitialized urandom read (16 bytes read)
[    2.902857] random: udevadm: uninitialized urandom read (16 bytes read)
[    2.902864] random: udevadm: uninitialized urandom read (16 bytes read)
[    2.903154] random: udevadm: uninitialized urandom read (16 bytes read)
[    2.945041] wmi: Mapper loaded
[    2.945437] hidraw: raw HID events driver (C) Jiri Kosina
[    2.951660] ahci 0000:00:17.0: version 3.0
[    2.951877] ahci 0000:00:17.0: AHCI 0001.0301 32 slots 8 ports 6 Gbps 0x0 impl SATA mode
[    2.951879] ahci 0000:00:17.0: flags: 64bit ncq pm led clo only pio slum part deso sadm sds apst 
[    2.954418] scsi host0: ahci
[    2.954500] scsi host1: ahci
[    2.956089] scsi host2: ahci
[    2.956169] scsi host3: ahci
[    2.960267] scsi host4: ahci
[    2.960961] scsi host5: ahci
[    2.964145] scsi host6: ahci
[    2.968281] scsi host7: ahci
[    2.968324] ata1: DUMMY
[    2.968325] ata2: DUMMY
[    2.968326] ata3: DUMMY
[    2.968326] ata4: DUMMY
[    2.968327] ata5: DUMMY
[    2.968328] ata6: DUMMY
[    2.968328] ata7: DUMMY
[    2.968329] ata8: DUMMY
[    2.968380] nvme nvme0: pci function 0000:3c:00.0
[    2.984012] [drm] Found 64MB of eDRAM
[    2.986121] [drm] Memory usable by graphics device = 4096M
[    2.986123] checking generic (90000000 1600000) vs hw (90000000 10000000)
[    2.986123] fb: switching to inteldrmfb from EFI VGA
[    2.986192] Console: switching to colour dummy device 80x25
[    2.986288] [drm] Replacing VGA console driver
[    2.992242] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    2.992243] [drm] Driver supports precise vblank timestamp query.
[    3.001074] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[    3.001170] [drm] Finished loading DMC firmware i915/skl_dmc_ver1_26.bin (v1.26)
[    3.007601] [drm] Initialized i915 1.6.0 20170403 for 0000:00:02.0 on minor 0
[    3.009114] usb 1-3: New USB device found, idVendor=8087, idProduct=0a2b
[    3.009115] usb 1-3: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    3.014417] ACPI: Video Device [GFX0] (multi-head: yes  rom: no  post: no)
[    3.015181] input: Video Bus as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input7
[    3.015256] [drm] Initialized i915 1.6.0 20170403 for 0000:00:02.0 on minor 0
[    3.052627] fbcon: inteldrmfb (fb0) is primary device
[    3.128337] usb 1-4: new full-speed USB device number 3 using xhci_hcd
[    3.194808]  nvme0n1: p1 p2 p3 p4
[    3.200343] random: fast init done
[    3.784090] psmouse serio1: synaptics: queried max coordinates: x [..5666], y [..4734]
[    3.815680] psmouse serio1: synaptics: queried min coordinates: x [1276..], y [1118..]
[    3.875470] psmouse serio1: synaptics: Touchpad model: 1, fw: 8.2, id: 0x1e2a1, caps: 0xf00223/0x840300/0x12e800/0x36b7c98, board id: 3038, fw id: 2011643
[    3.914619] input: SynPS/2 Synaptics TouchPad as /devices/platform/i8042/serio1/input/input6
[    4.354596] Console: switching to colour frame buffer device 400x112
[    4.380093] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
[    4.833323] [drm] RC6 on
[    8.417912] usb 1-4: New USB device found, idVendor=04f3, idProduct=20d0
[    8.417914] usb 1-4: New USB device strings: Mfr=4, Product=14, SerialNumber=0
[    8.417915] usb 1-4: Product: Touchscreen
[    8.417916] usb 1-4: Manufacturer: ELAN
[    8.536344] usb 1-5: new high-speed USB device number 4 using xhci_hcd
[    8.738129] usb 1-5: New USB device found, idVendor=0c45, idProduct=670c
[    8.738130] usb 1-5: New USB device strings: Mfr=2, Product=1, SerialNumber=0
[    8.738131] usb 1-5: Product: Integrated_Webcam_HD
[    8.738132] usb 1-5: Manufacturer: CN09GTFM724875CFB3SYA01
[    8.753946] usbcore: registered new interface driver usbhid
[    8.753947] usbhid: USB HID core driver
[    8.960139] raid6: sse2x1   gen()  9215 MB/s
[    9.028136] raid6: sse2x1   xor()  5897 MB/s
[    9.096131] raid6: sse2x2   gen() 11101 MB/s
[    9.164133] raid6: sse2x2   xor()  7640 MB/s
[    9.232135] raid6: sse2x4   gen() 12159 MB/s
[    9.300131] raid6: sse2x4   xor()  8214 MB/s
[    9.368132] raid6: avx2x1   gen() 21859 MB/s
[    9.436132] raid6: avx2x1   xor() 15908 MB/s
[    9.504133] raid6: avx2x2   gen() 25045 MB/s
[    9.572132] raid6: avx2x2   xor() 17406 MB/s
[    9.640133] raid6: avx2x4   gen() 27531 MB/s
[    9.708131] raid6: avx2x4   xor() 18877 MB/s
[    9.708132] raid6: using algorithm avx2x4 gen() 27531 MB/s
[    9.708132] raid6: .... xor() 18877 MB/s, rmw enabled
[    9.708133] raid6: using avx2x2 recovery algorithm
[    9.708944] xor: automatically using best checksumming function   avx       
[    9.721380] Btrfs loaded, crc32c=crc32c-intel
[    9.769190] EXT4-fs (nvme0n1p3): mounted filesystem with ordered data mode. Opts: (null)
[    9.836439] systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
[    9.836524] systemd[1]: Detected architecture x86-64.
[    9.836912] systemd[1]: Set hostname to <test-XPS-13-9350>.
[    9.877033] systemd[1]: Reached target Remote File Systems (Pre).
[    9.877070] systemd[1]: Reached target Encrypted Volumes.
[    9.877090] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    9.877117] systemd[1]: Listening on Journal Socket.
[    9.877127] systemd[1]: Reached target Remote File Systems.
[    9.877140] systemd[1]: Listening on udev Kernel Socket.
[    9.877155] systemd[1]: Listening on udev Control Socket.
[    9.892524] lp: driver loaded but no devices found
[    9.897523] ppdev: user-space parallel port driver
[    9.941255] EXT4-fs (nvme0n1p3): re-mounted. Opts: errors=remount-ro
[    9.948180] systemd-journald[323]: Received request to flush runtime journal from PID 1
[   10.009841] acpi INT33D5:00: intel-hid: created platform device
[   10.010707] ACPI Warning: \_SB.IETM._ART: Return Package type mismatch at index 0 - found Integer, expected Reference (20170303/nspredef-297)
[   10.010712] ACPI: Invalid package element [0]: got number, expecting [R]
[   10.010712] _ART package 0 is invalid, ignored
[   10.010906] input: Intel HID events as /devices/platform/INT33D5:00/input/input8
[   10.028222] int3403 thermal: probe of INT3403:03 failed with error -22
[   10.037479] Bluetooth: Core ver 2.22
[   10.037487] NET: Registered protocol family 31
[   10.037487] Bluetooth: HCI device and connection manager initialized
[   10.037489] Bluetooth: HCI socket layer initialized
[   10.037491] Bluetooth: L2CAP socket layer initialized
[   10.037493] Bluetooth: SCO socket layer initialized
[   10.043477] Bluetooth: HCI UART driver ver 2.3
[   10.043478] Bluetooth: HCI UART protocol H4 registered
[   10.043478] Bluetooth: HCI UART protocol BCSP registered
[   10.043479] Bluetooth: HCI UART protocol LL registered
[   10.043479] Bluetooth: HCI UART protocol ATH3K registered
[   10.043480] Bluetooth: HCI UART protocol Three-wire (H5) registered
[   10.043502] Bluetooth: HCI UART protocol Intel registered
[   10.043511] Bluetooth: HCI UART protocol Broadcom registered
[   10.043512] Bluetooth: HCI UART protocol QCA registered
[   10.062838] intel-lpss 0000:00:15.0: enabling device (0000 -> 0002)
[   10.069471] idma64 idma64.0: Found Intel integrated DMA 64-bit
[   10.070841] intel-lpss 0000:00:15.1: enabling device (0000 -> 0002)
[   10.071044] idma64 idma64.1: Found Intel integrated DMA 64-bit
[   10.071295] (NULL device *): hwmon_device_register() is deprecated. Please convert the driver to use hwmon_device_register_with_info().
[   10.071320] proc_thermal 0000:00:04.0: enabling device (0000 -> 0002)
[   10.073851] mei_me 0000:00:16.0: enabling device (0000 -> 0002)
[   10.089004] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[   10.096824] AVX2 version of gcm_enc/dec engaged.
[   10.096824] AES CTR mode by8 optimization enabled
[   10.106617] Intel(R) Wireless WiFi driver for Linux
[   10.106618] Copyright(c) 2003- 2015 Intel Corporation
[   10.106690] iwlwifi 0000:3a:00.0: enabling device (0000 -> 0002)
[   10.106988] thunderbolt 0000:03:00.0: enabling device (0000 -> 0002)
[   10.107340] thunderbolt 0000:03:00.0: NHI initialized, starting thunderbolt
[   10.107343] thunderbolt 0000:03:00.0: allocating TX ring 0 of size 10
[   10.107376] thunderbolt 0000:03:00.0: allocating RX ring 0 of size 10
[   10.107382] thunderbolt 0000:03:00.0: control channel created
[   10.107383] thunderbolt 0000:03:00.0: control channel starting...
[   10.107383] thunderbolt 0000:03:00.0: starting TX ring 0
[   10.107403] thunderbolt 0000:03:00.0: enabling interrupt at register 0x38200 bit 0 (0x0 -> 0x1)
[   10.107403] thunderbolt 0000:03:00.0: starting RX ring 0
[   10.107409] thunderbolt 0000:03:00.0: enabling interrupt at register 0x38200 bit 12 (0x1 -> 0x1001)
[   10.108048] iwlwifi 0000:3a:00.0: Direct firmware load for iwlwifi-8000C-30.ucode failed with error -2
[   10.108144] iwlwifi 0000:3a:00.0: Direct firmware load for iwlwifi-8000C-29.ucode failed with error -2
[   10.108202] iwlwifi 0000:3a:00.0: Direct firmware load for iwlwifi-8000C-28.ucode failed with error -2
[   10.108303] iwlwifi 0000:3a:00.0: Direct firmware load for iwlwifi-8000C-27.ucode failed with error -2
[   10.108313] iwlwifi 0000:3a:00.0: Direct firmware load for iwlwifi-8000C-26.ucode failed with error -2
[   10.108321] iwlwifi 0000:3a:00.0: Direct firmware load for iwlwifi-8000C-25.ucode failed with error -2
[   10.108328] iwlwifi 0000:3a:00.0: Direct firmware load for iwlwifi-8000C-24.ucode failed with error -2
[   10.108341] iwlwifi 0000:3a:00.0: Direct firmware load for iwlwifi-8000C-23.ucode failed with error -2
[   10.108352] iwlwifi 0000:3a:00.0: Direct firmware load for iwlwifi-8000C-22.ucode failed with error -2
[   10.111066] iwlwifi 0000:3a:00.0: loaded firmware version 21.302800.0 op_mode iwlmvm
[   10.139921] iwlwifi 0000:3a:00.0: Detected Intel(R) Dual Band Wireless AC 8260, REV=0x208
[   10.141980] iwlwifi 0000:3a:00.0: L1 Enabled - LTR Disabled
[   10.142632] iwlwifi 0000:3a:00.0: L1 Enabled - LTR Disabled
[   10.173208] intel_rapl: Found RAPL domain package
[   10.173209] intel_rapl: Found RAPL domain core
[   10.173210] intel_rapl: Found RAPL domain uncore
[   10.173210] intel_rapl: Found RAPL domain dram
[   10.209753] snd_hda_intel 0000:00:1f.3: enabling device (0000 -> 0002)
[   10.209912] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[   10.249573] Adding 32974844k swap on /dev/nvme0n1p4.  Priority:-1 extents:1 across:32974844k SSFS
[   10.284637] snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC3246: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:speaker
[   10.284639] snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[   10.284640] snd_hda_codec_realtek hdaudioC0D0:    hp_outs=1 (0x21/0x0/0x0/0x0/0x0)
[   10.284641] snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0
[   10.284642] snd_hda_codec_realtek hdaudioC0D0:    inputs:
[   10.284644] snd_hda_codec_realtek hdaudioC0D0:      Headset Mic=0x19
[   10.284645] snd_hda_codec_realtek hdaudioC0D0:      Headphone Mic=0x1a
[   10.284646] snd_hda_codec_realtek hdaudioC0D0:      Internal Mic=0x12
[   10.316846] ieee80211 phy0: Selected rate control algorithm 'iwl-mvm-rs'
[   10.316978] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2)
[   10.317109] (NULL device *): hwmon_device_register() is deprecated. Please convert the driver to use hwmon_device_register_with_info().
[   10.317125] thermal thermal_zone10: failed to read out thermal zone (-5)
[   10.325669] input: HDA Intel PCH Headphone Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input9
[   10.325726] input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input10
[   10.325778] input: HDA Intel PCH HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input11
[   10.325823] input: HDA Intel PCH HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input12
[   10.325866] input: HDA Intel PCH HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input13
[   10.325906] input: HDA Intel PCH HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input14
[   10.345573] random: crng init done
[   10.359357] audit: type=1400 audit(1495560166.969:2): apparmor="STATUS" operation="profile_load" name="/usr/lib/lightdm/lightdm-guest-session" pid=805 comm="apparmor_parser"
[   10.359358] audit: type=1400 audit(1495560166.969:3): apparmor="STATUS" operation="profile_load" name="/usr/lib/lightdm/lightdm-guest-session//chromium" pid=805 comm="apparmor_parser"
[   10.360481] audit: type=1400 audit(1495560166.970:4): apparmor="STATUS" operation="profile_load" name="/sbin/dhclient" pid=806 comm="apparmor_parser"
[   10.360482] audit: type=1400 audit(1495560166.970:5): apparmor="STATUS" operation="profile_load" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=806 comm="apparmor_parser"
[   10.360483] audit: type=1400 audit(1495560166.970:6): apparmor="STATUS" operation="profile_load" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=806 comm="apparmor_parser"
[   10.360483] audit: type=1400 audit(1495560166.970:7): apparmor="STATUS" operation="profile_load" name="/usr/lib/connman/scripts/dhclient-script" pid=806 comm="apparmor_parser"
[   10.366367] audit: type=1400 audit(1495560166.976:8): apparmor="STATUS" operation="profile_load" name="/usr/lib/snapd/snap-confine" pid=810 comm="apparmor_parser"
[   10.366369] audit: type=1400 audit(1495560166.976:9): apparmor="STATUS" operation="profile_load" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=810 comm="apparmor_parser"
[   10.369480] audit: type=1400 audit(1495560166.979:10): apparmor="STATUS" operation="profile_load" name="/usr/sbin/cups-browsed" pid=811 comm="apparmor_parser"
[   10.371536] audit: type=1400 audit(1495560166.981:11): apparmor="STATUS" operation="profile_load" name="/usr/bin/evince" pid=807 comm="apparmor_parser"
[   10.383687] iwlwifi 0000:3a:00.0 wlp58s0: renamed from wlan0
[   10.420481] ACPI: Invalid package element [0]: got number, expecting [R]
[   10.420483] _ART package 0 is invalid, ignored
[   10.420495] ACPI: Invalid package element [0]: got number, expecting [R]
[   10.420496] _ART package 0 is invalid, ignored
[   10.480837] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[   10.480838] Bluetooth: BNEP filters: protocol multicast
[   10.480841] Bluetooth: BNEP socket layer initialized
[   10.602565] i2c_hid i2c-DLL0704:01: i2c-DLL0704:01 supply vdd not found, using dummy regulator
[   10.638786] input: ELAN Touchscreen as /devices/pci0000:00/0000:00:14.0/usb1/1-4/1-4:1.0/0003:04F3:20D0.0001/input/input15
[   10.653104] hid-multitouch 0003:04F3:20D0.0001: input,hiddev0,hidraw0: USB HID v1.10 Device [ELAN Touchscreen] on usb-0000:00:14.0-4/input0
[   10.662207] input: DLL0704:01 06CB:76AE Touchpad as /devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-7/i2c-DLL0704:01/0018:06CB:76AE.0002/input/input20
[   10.662305] hid-multitouch 0018:06CB:76AE.0002: input,hidraw1: I2C HID v1.00 Mouse [DLL0704:01 06CB:76AE] on i2c-DLL0704:01
[   10.673803] IPv6: ADDRCONF(NETDEV_UP): wlp58s0: link is not ready
[   10.675897] iwlwifi 0000:3a:00.0: L1 Enabled - LTR Disabled
[   10.676621] iwlwifi 0000:3a:00.0: L1 Enabled - LTR Disabled
[   10.816287] iwlwifi 0000:3a:00.0: L1 Enabled - LTR Disabled
[   10.816564] iwlwifi 0000:3a:00.0: L1 Enabled - LTR Disabled
[   10.896838] IPv6: ADDRCONF(NETDEV_UP): wlp58s0: link is not ready
[   10.979441] IPv6: ADDRCONF(NETDEV_UP): wlp58s0: link is not ready
[   11.009231] Non-volatile memory driver v1.3
[   11.100994] media: Linux media interface: v0.10
[   11.105692] Linux video capture interface: v2.00
[   11.240504] usbcore: registered new interface driver btusb
[   11.241889] Bluetooth: hci0: Bootloader revision 0.0 build 2 week 52 2014
[   11.246898] Bluetooth: hci0: Device revision is 5
[   11.246899] Bluetooth: hci0: Secure boot is enabled
[   11.246900] Bluetooth: hci0: OTP lock is enabled
[   11.246901] Bluetooth: hci0: API lock is enabled
[   11.246901] Bluetooth: hci0: Debug lock is disabled
[   11.246902] Bluetooth: hci0: Minimum firmware build 1 week 10 2014
[   11.247974] Bluetooth: hci0: Found device firmware: intel/ibt-11-5.sfi
[   11.251786] uvcvideo: Found UVC 1.00 device Integrated_Webcam_HD (0c45:670c)
[   11.291646] uvcvideo 1-5:1.0: Entity type for entity Extension 4 was not initialized!
[   11.291648] uvcvideo 1-5:1.0: Entity type for entity Extension 3 was not initialized!
[   11.291650] uvcvideo 1-5:1.0: Entity type for entity Processing 2 was not initialized!
[   11.291651] uvcvideo 1-5:1.0: Entity type for entity Camera 1 was not initialized!
[   11.291832] input: Integrated_Webcam_HD as /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5:1.0/input/input25
[   11.291929] usbcore: registered new interface driver uvcvideo
[   11.291929] USB Video Class driver (1.1.1)
[   11.368504] thunderbolt 0000:03:00.0: current switch config:
[   11.368506] thunderbolt 0000:03:00.0:  Switch: 8086:1576 (Revision: 4, TB Version: 2)
[   11.368507] thunderbolt 0000:03:00.0:   Max Port Number: 11
[   11.368508] thunderbolt 0000:03:00.0:   Config:
[   11.368510] thunderbolt 0000:03:00.0:    Upstream Port Number: 5 Depth: 0 Route String: 0x0 Enabled: 1, PlugEventsDelay: 254ms
[   11.368511] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
[   11.380254] thunderbolt 0000:03:00.0: 0: uid: 0x8086a3422131ea10
[   11.380659] thunderbolt 0000:03:00.0:  Port 0: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   11.380660] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
[   11.380661] thunderbolt 0000:03:00.0:   Max counters: 8
[   11.380662] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   11.380812] thunderbolt 0000:03:00.0:  Port 1: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   11.380813] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   11.380814] thunderbolt 0000:03:00.0:   Max counters: 16
[   11.380815] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
[   11.380986] thunderbolt 0000:03:00.0:  Port 2: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   11.380987] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   11.380989] thunderbolt 0000:03:00.0:   Max counters: 16
[   11.380989] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
[   11.381126] thunderbolt 0000:03:00.0:  Port 3: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   11.381128] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   11.381128] thunderbolt 0000:03:00.0:   Max counters: 16
[   11.381129] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
[   11.381277] thunderbolt 0000:03:00.0:  Port 4: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   11.381278] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   11.381279] thunderbolt 0000:03:00.0:   Max counters: 16
[   11.381280] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
[   11.381281] thunderbolt 0000:03:00.0: 0:5: disabled by eeprom
[   11.381320] thunderbolt 0000:03:00.0:  Port 6: 8086:1576 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
[   11.381321] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[   11.381322] thunderbolt 0000:03:00.0:   Max counters: 2
[   11.381323] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   11.381364] thunderbolt 0000:03:00.0:  Port 7: 8086:1576 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
[   11.381365] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[   11.381365] thunderbolt 0000:03:00.0:   Max counters: 2
[   11.381366] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   11.381407] thunderbolt 0000:03:00.0:  Port 8: 8086:1576 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0102))
[   11.381415] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
[   11.381416] thunderbolt 0000:03:00.0:   Max counters: 2
[   11.381417] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   11.381418] thunderbolt 0000:03:00.0: 0:9: disabled by eeprom
[   11.381462] thunderbolt 0000:03:00.0:  Port 10: 8086:1576 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0101))
[   11.381463] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
[   11.381464] thunderbolt 0000:03:00.0:   Max counters: 2
[   11.381465] thunderbolt 0000:03:00.0:   NFC Credits: 0x1000000
[   11.381466] thunderbolt 0000:03:00.0: 0:b: disabled by eeprom
[   11.383554] thunderbolt 0000:03:00.0: current switch config:
[   11.383557] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB Version: 2)
[   11.383558] thunderbolt 0000:03:00.0:   Max Port Number: 11
[   11.383559] thunderbolt 0000:03:00.0:   Config:
[   11.383560] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 1 Route String: 0x1 Enabled: 1, PlugEventsDelay: 254ms
[   11.383562] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
[   11.388107] thunderbolt 0000:03:00.0: 1: reading drom (length: 0x6e)
[   11.494850] thunderbolt 0000:03:00.0: 1: drom data crc32 mismatch (expected: 0xaf438340, got: 0xaf4383c0), continuing
[   11.495040] thunderbolt 0000:03:00.0: 1: uid: 0xd40f7a7928c300
[   11.495075] thunderbolt 0000:03:00.0:  Port 0: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   11.495077] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
[   11.495077] thunderbolt 0000:03:00.0:   Max counters: 8
[   11.495078] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   11.495213] thunderbolt 0000:03:00.0:  Port 1: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   11.495214] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   11.495215] thunderbolt 0000:03:00.0:   Max counters: 16
[   11.495215] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
[   11.495350] thunderbolt 0000:03:00.0:  Port 2: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   11.495351] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   11.495352] thunderbolt 0000:03:00.0:   Max counters: 16
[   11.495353] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
[   11.495497] thunderbolt 0000:03:00.0:  Port 3: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   11.495498] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   11.495499] thunderbolt 0000:03:00.0:   Max counters: 16
[   11.495499] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
[   11.495636] thunderbolt 0000:03:00.0:  Port 4: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   11.495637] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   11.495638] thunderbolt 0000:03:00.0:   Max counters: 16
[   11.495639] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
[   11.495640] thunderbolt 0000:03:00.0: 1:5: disabled by eeprom
[   11.495681] thunderbolt 0000:03:00.0:  Port 6: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100102))
[   11.495682] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[   11.495683] thunderbolt 0000:03:00.0:   Max counters: 2
[   11.495684] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   11.495726] thunderbolt 0000:03:00.0:  Port 7: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
[   11.495727] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[   11.495728] thunderbolt 0000:03:00.0:   Max counters: 2
[   11.495729] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   11.495730] thunderbolt 0000:03:00.0: 1:8: disabled by eeprom
[   11.495731] thunderbolt 0000:03:00.0: 1:9: disabled by eeprom
[   11.495732] thunderbolt 0000:03:00.0: 1:a: disabled by eeprom
[   11.495733] thunderbolt 0000:03:00.0: 1:b: disabled by eeprom
[   11.872228] ACPI: \_SB_.PCI0.RP01: acpiphp_glue: Bus check in hotplug_event()
[   11.872385] pci 0000:04:00.0: [8086:1578] type 01 class 0x060400
[   11.872636] pci 0000:04:00.0: supports D1 D2
[   11.872637] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[   11.884199] pci_bus 0000:05: [bus 05] partially hidden behind bridge 0000:04 [bus 04]
[   11.884289] pci_bus 0000:39: busn_res: can not insert [bus 39] under [bus 02-05] (conflicts with (null) [bus 02-05])
[   11.884292] pcieport 0000:02:02.0: PCI bridge to [bus 39]
[   11.884302] pcieport 0000:02:02.0:   bridge window [mem 0xd9f00000-0xd9ffffff]
[   11.884309] pci_bus 0000:39: [bus 39] partially hidden behind bridge 0000:02 [bus 02-05]
[   11.884334] pci_bus 0000:02: Allocating resources
[   11.884336] pcieport 0000:02:02.0: can't claim BAR 14 [mem 0xd9f00000-0xd9ffffff]: no compatible bridge window
[   12.686659] Bluetooth: hci0: Waiting for firmware download to complete
[   12.686914] Bluetooth: hci0: Firmware loaded in 1412486 usecs
[   12.687025] Bluetooth: hci0: Waiting for device to boot
[   12.698961] Bluetooth: hci0: Device booted in 11712 usecs
[   12.699496] Bluetooth: hci0: Found Intel DDC parameters: intel/ibt-11-5.ddc
[   12.701060] Bluetooth: hci0: Applying Intel DDC parameters completed
[   12.702019] Bluetooth: hci0: Setting Intel event mask failed (-16)
[   12.761365] Bluetooth: RFCOMM TTY layer initialized
[   12.761374] Bluetooth: RFCOMM socket layer initialized
[   12.761379] Bluetooth: RFCOMM ver 1.11
[   19.630402] wlp58s0: authenticate with 50:6a:03:a7:1c:27
[   19.635387] wlp58s0: send auth to 50:6a:03:a7:1c:27 (try 1/3)
[   19.641967] wlp58s0: authenticated
[   19.644174] wlp58s0: associate with 50:6a:03:a7:1c:27 (try 1/3)
[   19.648610] wlp58s0: RX AssocResp from 50:6a:03:a7:1c:27 (capab=0x1411 status=0 aid=1)
[   19.656611] wlp58s0: associated
[   19.656688] IPv6: ADDRCONF(NETDEV_CHANGE): wlp58s0: link becomes ready
[   31.968481] thunderbolt 0000:03:00.0: timeout reading config space 2 from 0x0
boot is finished, no new messages

now unplug the dock, results:
[   38.160442] ACPI: \_SB_.PCI0.RP01: acpiphp_glue: Bus check in hotplug_event()
[   38.160487] pci_raw_set_power_state: 12 callbacks suppressed
[   38.160489] pcieport 0000:04:00.0: Refused to change power state, currently in D3
[   38.162357] pci_bus 0000:39: [bus 39] partially hidden behind bridge 0000:02 [bus 02-05]
[   38.162381] pci_bus 0000:02: Allocating resources
[   48.281873] ACPI: \_SB_.PCI0.RP01: acpiphp_glue: Bus check in hotplug_event()
[   48.281902] pcieport 0000:02:02.0: Refused to change power state, currently in D3
[   48.782701] thunderbolt 0000:03:00.0: stopping RX ring 0
[   48.782709] thunderbolt 0000:03:00.0: disabling interrupt at register 0x38200 bit 12 (0xffffffff -> 0xffffefff)
[   48.782714] thunderbolt 0000:03:00.0: stopping TX ring 0
[   48.782716] thunderbolt 0000:03:00.0: disabling interrupt at register 0x38200 bit 0 (0xffffffff -> 0xfffffffe)
[   48.782721] thunderbolt 0000:03:00.0: control channel stopped
[   48.782768] thunderbolt 0000:03:00.0: freeing RX ring 0
[   48.782776] thunderbolt 0000:03:00.0: freeing TX ring 0
[   48.782782] thunderbolt 0000:03:00.0: shutdown
[   48.804321] pcieport 0000:02:00.0: Refused to change power state, currently in D3
[   48.805988] pci_bus 0000:03: busn_res: [bus 03] is released
[   48.806042] pci_bus 0000:04: busn_res: [bus 04] is released
[   48.806148] pci_bus 0000:02: busn_res: [bus 02-05] is released
waited some time to confirm fully settled

now plug dock back in, results:
[   86.900174] ACPI Error: [SPRT] Namespace lookup failure, AE_ALREADY_EXISTS (20170303/dswload2-330)
[   86.900195] ACPI Exception: AE_ALREADY_EXISTS, During name lookup/catalog (20170303/psobject-241)
[   86.900204] ACPI Error: Method parse/execution failed [\_GPE._E42] (Node ffff9af63016b488), AE_ALREADY_EXISTS (20170303/psparse-543)
[   86.900220] ACPI Error: Method parse/execution failed [\_GPE._E42] (Node ffff9af63016b488), AE_ALREADY_EXISTS (20170303/psparse-543)
[   86.900239] ACPI Exception: AE_ALREADY_EXISTS, while evaluating GPE method [_E42] (20170303/evgpe-646)
[   86.945889] ACPI: \_SB_.PCI0.RP01: acpiphp_glue: Bus check in hotplug_event()
[   86.945976] pci 0000:01:00.0: [8086:1576] type 01 class 0x060400
[   86.946101] pci 0000:01:00.0: supports D1 D2
[   86.946102] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[   86.946196] pci 0000:01:00.0: System wakeup disabled by ACPI
[   86.946359] pci 0000:02:00.0: [8086:1576] type 01 class 0x060400
[   86.946508] pci 0000:02:00.0: supports D1 D2
[   86.946509] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[   86.946620] pci 0000:02:01.0: [8086:1576] type 01 class 0x060400
[   86.946764] pci 0000:02:01.0: supports D1 D2
[   86.946765] pci 0000:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[   86.946868] pci 0000:02:02.0: [8086:1576] type 01 class 0x060400
[   86.946998] pci 0000:02:02.0: supports D1 D2
[   86.946999] pci 0000:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[   86.947106] pci 0000:01:00.0: PCI bridge to [bus 02-39]
[   86.947116] pci 0000:01:00.0:   bridge window [mem 0xc4000000-0xda0fffff]
[   86.947123] pci 0000:01:00.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[   86.947194] pci 0000:03:00.0: [8086:1575] type 00 class 0x088000
[   86.947224] pci 0000:03:00.0: reg 0x10: [mem 0xda000000-0xda03ffff]
[   86.947236] pci 0000:03:00.0: reg 0x14: [mem 0xda040000-0xda040fff]
[   86.947393] pci 0000:03:00.0: supports D1 D2
[   86.947394] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[   86.947528] pci 0000:02:00.0: PCI bridge to [bus 03]
[   86.947536] pci 0000:02:00.0:   bridge window [mem 0xda000000-0xda0fffff]
[   86.947598] pci 0000:02:01.0: PCI bridge to [bus 04-38]
[   86.947607] pci 0000:02:01.0:   bridge window [mem 0xc4000000-0xd9efffff]
[   86.947613] pci 0000:02:01.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[   86.947670] pci 0000:02:02.0: PCI bridge to [bus 39]
[   86.947679] pci 0000:02:02.0:   bridge window [mem 0xd9f00000-0xd9ffffff]
[   86.947710] pci_bus 0000:02: Allocating resources
[   86.947742] pci 0000:02:01.0: bridge window [io  0x1000-0x0fff] to [bus 04-38] add_size 1000
[   86.947760] pci 0000:01:00.0: bridge window [io  0x1000-0x0fff] to [bus 02-39] add_size 1000
[   86.947764] pci 0000:01:00.0: BAR 13: assigned [io  0x2000-0x2fff]
[   86.947766] pci 0000:02:01.0: BAR 13: assigned [io  0x2000-0x2fff]
[   86.947768] pci 0000:02:00.0: PCI bridge to [bus 03]
[   86.947773] pci 0000:02:00.0:   bridge window [mem 0xda000000-0xda0fffff]
[   86.947782] pci 0000:02:01.0: PCI bridge to [bus 04-38]
[   86.947785] pci 0000:02:01.0:   bridge window [io  0x2000-0x2fff]
[   86.947790] pci 0000:02:01.0:   bridge window [mem 0xc4000000-0xd9efffff]
[   86.947794] pci 0000:02:01.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[   86.947800] pci 0000:02:02.0: PCI bridge to [bus 39]
[   86.947806] pci 0000:02:02.0:   bridge window [mem 0xd9f00000-0xd9ffffff]
[   86.947815] pci 0000:01:00.0: PCI bridge to [bus 02-39]
[   86.947817] pci 0000:01:00.0:   bridge window [io  0x2000-0x2fff]
[   86.947822] pci 0000:01:00.0:   bridge window [mem 0xc4000000-0xda0fffff]
[   86.947826] pci 0000:01:00.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[   86.947930] pcieport 0000:01:00.0: enabling device (0006 -> 0007)
[   86.948329] pcieport 0000:02:01.0: enabling device (0006 -> 0007)
[   86.949027] thunderbolt 0000:03:00.0: NHI initialized, starting thunderbolt
[   86.949031] thunderbolt 0000:03:00.0: allocating TX ring 0 of size 10
[   86.949045] thunderbolt 0000:03:00.0: allocating RX ring 0 of size 10
[   86.949054] thunderbolt 0000:03:00.0: control channel created
[   86.949055] thunderbolt 0000:03:00.0: control channel starting...
[   86.949056] thunderbolt 0000:03:00.0: starting TX ring 0
[   86.949063] thunderbolt 0000:03:00.0: enabling interrupt at register 0x38200 bit 0 (0x0 -> 0x1)
[   86.949065] thunderbolt 0000:03:00.0: starting RX ring 0
[   86.949071] thunderbolt 0000:03:00.0: enabling interrupt at register 0x38200 bit 12 (0x1 -> 0x1001)
[   88.203530] thunderbolt 0000:03:00.0: current switch config:
[   88.203541] thunderbolt 0000:03:00.0:  Switch: 8086:1576 (Revision: 4, TB Version: 2)
[   88.203547] thunderbolt 0000:03:00.0:   Max Port Number: 11
[   88.203553] thunderbolt 0000:03:00.0:   Config:
[   88.203562] thunderbolt 0000:03:00.0:    Upstream Port Number: 5 Depth: 0 Route String: 0x0 Enabled: 1, PlugEventsDelay: 254ms
[   88.203568] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
[   88.224413] thunderbolt 0000:03:00.0: 0: uid: 0x8086a3422131ea10
[   88.224978] thunderbolt 0000:03:00.0:  Port 0: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   88.224983] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
[   88.224986] thunderbolt 0000:03:00.0:   Max counters: 8
[   88.224990] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   88.225225] thunderbolt 0000:03:00.0:  Port 1: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   88.225229] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   88.225232] thunderbolt 0000:03:00.0:   Max counters: 16
[   88.225236] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
[   88.225476] thunderbolt 0000:03:00.0:  Port 2: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   88.225481] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   88.225491] thunderbolt 0000:03:00.0:   Max counters: 16
[   88.225496] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
[   88.225703] thunderbolt 0000:03:00.0:  Port 3: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   88.225707] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   88.225710] thunderbolt 0000:03:00.0:   Max counters: 16
[   88.225712] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
[   88.225892] thunderbolt 0000:03:00.0:  Port 4: 8086:1576 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   88.225896] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   88.225898] thunderbolt 0000:03:00.0:   Max counters: 16
[   88.225901] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
[   88.225905] thunderbolt 0000:03:00.0: 0:5: disabled by eeprom
[   88.225972] thunderbolt 0000:03:00.0:  Port 6: 8086:1576 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
[   88.225975] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[   88.225983] thunderbolt 0000:03:00.0:   Max counters: 2
[   88.225986] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   88.232116] thunderbolt 0000:03:00.0:  Port 7: 8086:1576 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
[   88.232120] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[   88.232123] thunderbolt 0000:03:00.0:   Max counters: 2
[   88.232125] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   88.232207] thunderbolt 0000:03:00.0:  Port 8: 8086:1576 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0102))
[   88.232210] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
[   88.232213] thunderbolt 0000:03:00.0:   Max counters: 2
[   88.232216] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   88.232219] thunderbolt 0000:03:00.0: 0:9: disabled by eeprom
[   88.232299] thunderbolt 0000:03:00.0:  Port 10: 8086:1576 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0101))
[   88.232303] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
[   88.232305] thunderbolt 0000:03:00.0:   Max counters: 2
[   88.232308] thunderbolt 0000:03:00.0:   NFC Credits: 0x1000000
[   88.232311] thunderbolt 0000:03:00.0: 0:b: disabled by eeprom
[   88.234610] thunderbolt 0000:03:00.0: current switch config:
[   88.234617] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB Version: 2)
[   88.234620] thunderbolt 0000:03:00.0:   Max Port Number: 11
[   88.234623] thunderbolt 0000:03:00.0:   Config:
[   88.234628] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 1 Route String: 0x1 Enabled: 1, PlugEventsDelay: 254ms
[   88.234632] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
[   88.241473] thunderbolt 0000:03:00.0: 1: reading drom (length: 0x6e)
[   88.359442] thunderbolt 0000:03:00.0: 1: drom data crc32 mismatch (expected: 0xaf438340, got: 0xaf4383c0), continuing
[   88.359722] thunderbolt 0000:03:00.0: 1: uid: 0xd40f7a7928c300
[   88.359775] thunderbolt 0000:03:00.0:  Port 0: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   88.359779] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
[   88.359781] thunderbolt 0000:03:00.0:   Max counters: 8
[   88.359784] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   88.359972] thunderbolt 0000:03:00.0:  Port 1: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   88.359977] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   88.359979] thunderbolt 0000:03:00.0:   Max counters: 16
[   88.359982] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
[   88.360157] thunderbolt 0000:03:00.0:  Port 2: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   88.360161] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   88.360163] thunderbolt 0000:03:00.0:   Max counters: 16
[   88.360166] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
[   88.360344] thunderbolt 0000:03:00.0:  Port 3: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   88.360347] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   88.360349] thunderbolt 0000:03:00.0:   Max counters: 16
[   88.360352] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
[   88.360559] thunderbolt 0000:03:00.0:  Port 4: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   88.360562] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   88.360564] thunderbolt 0000:03:00.0:   Max counters: 16
[   88.360566] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
[   88.360569] thunderbolt 0000:03:00.0: 1:5: disabled by eeprom
[   88.360624] thunderbolt 0000:03:00.0:  Port 6: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100102))
[   88.360627] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[   88.360629] thunderbolt 0000:03:00.0:   Max counters: 2
[   88.360632] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   88.360682] thunderbolt 0000:03:00.0:  Port 7: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
[   88.360685] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[   88.360687] thunderbolt 0000:03:00.0:   Max counters: 2
[   88.360689] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   88.360692] thunderbolt 0000:03:00.0: 1:8: disabled by eeprom
[   88.360695] thunderbolt 0000:03:00.0: 1:9: disabled by eeprom
[   88.360697] thunderbolt 0000:03:00.0: 1:a: disabled by eeprom
[   88.360699] thunderbolt 0000:03:00.0: 1:b: disabled by eeprom
[   92.285434] thunderbolt 0000:03:00.0: current switch config:
[   92.285436] thunderbolt 0000:03:00.0:  Switch: 8086:1578 (Revision: 4, TB Version: 2)
[   92.285437] thunderbolt 0000:03:00.0:   Max Port Number: 11
[   92.285438] thunderbolt 0000:03:00.0:   Config:
[   92.285439] thunderbolt 0000:03:00.0:    Upstream Port Number: 1 Depth: 2 Route String: 0x301 Enabled: 1, PlugEventsDelay: 254ms
[   92.285440] thunderbolt 0000:03:00.0:    unknown1: 0x0 unknown4: 0x0
[   92.290231] thunderbolt 0000:03:00.0: 301: reading drom (length: 0x75)
[   92.309004] ACPI: \_SB_.PCI0.RP01: acpiphp_glue: Bus check in hotplug_event()
[   92.309147] pci 0000:04:00.0: [8086:1578] type 01 class 0x060400
[   92.309393] pci 0000:04:00.0: supports D1 D2
[   92.309394] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[   92.325039] sysfs: cannot create duplicate filename '/class/pci_bus/0000:05'
[   92.325045] ------------[ cut here ]------------
[   92.325049] WARNING: CPU: 2 PID: 5 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x56/0x70
[   92.325049] Modules linked in: ctr ccm rfcomm uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core videodev media btusb btrtl nvram msr hid_multitouch bnep i2c_designware_platform i2c_designware_core snd_hda_codec_hdmi dcdbas arc4 binfmt_misc nls_iso8859_1 snd_soc_skl snd_soc_skl_ipc snd_hda_codec_realtek snd_hda_codec_generic snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_ext_core snd_soc_sst_match snd_soc_core snd_compress snd_pcm_dmaengine ac97_bus snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl x86_pkg_temp_thermal intel_powerclamp snd_seq_midi coretemp snd_seq_midi_event kvm_intel kvm irqbypass iwlmvm crct10dif_pclmul snd_rawmidi crc32_pclmul mac80211 ghash_clmulni_intel rtsx_pci_ms pcbc snd_seq memstick thunderbolt iwlwifi nvmem_core cfg80211 snd_seq_device
[   92.325076]  aesni_intel snd_timer aes_x86_64 crypto_simd glue_helper cryptd snd joydev input_leds serio_raw soundcore shpchp idma64 virt_dma mei_me processor_thermal_device mei intel_pch_thermal intel_lpss_pci intel_soc_dts_iosf hci_uart btbcm btqca btintel bluetooth intel_lpss_acpi ecdh_generic intel_lpss int3403_thermal int340x_thermal_zone intel_hid int3400_thermal sparse_keymap acpi_thermal_rel tpm_crb acpi_pad mac_hid acpi_als kfifo_buf industrialio parport_pc ppdev lp parport autofs4 btrfs xor raid6_pq usbhid rtsx_pci_sdmmc i915 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt nvme psmouse nvme_core fb_sys_fops rtsx_pci ahci drm libahci i2c_hid hid wmi pinctrl_sunrisepoint pinctrl_intel video
[   92.325103] CPU: 2 PID: 5 Comm: kworker/u8:0 Not tainted 4.12.0-rc1+ #6
[   92.325104] Hardware name: Dell Inc. XPS 13 9350/09JHRY, BIOS 1.4.17 05/10/2017
[   92.325107] Workqueue: kacpi_hotplug acpi_hotplug_work_fn
[   92.325108] task: ffff9af62e6bbb00 task.stack: ffffbb3a018f8000
[   92.325109] RIP: 0010:sysfs_warn_dup+0x56/0x70
[   92.325110] RSP: 0018:ffffbb3a018fba08 EFLAGS: 00010286
[   92.325111] RAX: 0000000000000040 RBX: ffff9af62ab02000 RCX: 0000000000000000
[   92.325111] RDX: 0000000000000000 RSI: ffff9af640d0dd48 RDI: ffff9af640d0dd48
[   92.325112] RBP: ffffbb3a018fba20 R08: 0000000000000001 R09: 000000000000052f
[   92.325112] R10: ffff9af62e6a2000 R11: 000000000000052f R12: ffff9af5f223bdc0
[   92.325113] R13: ffff9af62dcdd3c0 R14: 0000000000000001 R15: ffffffffffffffef
[   92.325113] FS:  0000000000000000(0000) GS:ffff9af640d00000(0000) knlGS:0000000000000000
[   92.325114] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   92.325115] CR2: 00007f4ddb4d0760 CR3: 000000046c775000 CR4: 00000000003406e0
[   92.325115] Call Trace:
[   92.325118]  sysfs_do_create_link_sd.isra.2+0x9e/0xb0
[   92.325119]  sysfs_create_link+0x25/0x40
[   92.325121]  device_add+0x2e8/0x670
[   92.325123]  device_register+0x1a/0x20
[   92.325124]  pci_add_new_bus+0x179/0x420
[   92.325126]  pci_scan_bridge+0x5fc/0x640
[   92.325128]  pci_scan_child_bus+0xa4/0x180
[   92.325129]  pci_scan_bridge+0x447/0x640
[   92.325130]  ? pci_scan_single_device+0x51/0xd0
[   92.325131]  pci_scan_child_bus+0xa4/0x180
[   92.325133]  pci_scan_bridge+0x447/0x640
[   92.325135]  enable_slot+0xc6/0x2f0
[   92.325136]  ? trim_stale_devices+0xab/0x150
[   92.325137]  acpiphp_check_bridge.part.7+0xff/0x140
[   92.325139]  acpiphp_hotplug_notify+0x173/0x200
[   92.325140]  ? free_bridge+0x130/0x130
[   92.325141]  acpi_device_hotplug+0x9d/0x480
[   92.325143]  acpi_hotplug_work_fn+0x1e/0x30
[   92.325145]  process_one_work+0x156/0x3f0
[   92.325146]  worker_thread+0x4b/0x410
[   92.325147]  kthread+0x109/0x140
[   92.325148]  ? process_one_work+0x3f0/0x3f0
[   92.325149]  ? kthread_create_on_node+0x70/0x70
[   92.325151]  ret_from_fork+0x2c/0x40
[   92.325152] Code: 85 c0 48 89 c3 74 12 b9 00 10 00 00 48 89 c2 31 f6 4c 89 ef e8 ac c8 ff ff 4c 89 e2 48 89 de 48 c7 c7 58 59 ca 97 e8 77 27 ee ff <0f> ff 48 89 df e8 a0 fb f4 ff 5b 41 5c 41 5d 5d c3 66 0f 1f 84 
[   92.325170] ---[ end trace 533975a4cf49194b ]---
[   92.325191] ------------[ cut here ]------------
[   92.325193] WARNING: CPU: 2 PID: 5 at drivers/pci/probe.c:894 pci_add_new_bus+0x394/0x420
[   92.325194] Modules linked in: ctr ccm rfcomm uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core videodev media btusb btrtl nvram msr hid_multitouch bnep i2c_designware_platform i2c_designware_core snd_hda_codec_hdmi dcdbas arc4 binfmt_misc nls_iso8859_1 snd_soc_skl snd_soc_skl_ipc snd_hda_codec_realtek snd_hda_codec_generic snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_ext_core snd_soc_sst_match snd_soc_core snd_compress snd_pcm_dmaengine ac97_bus snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl x86_pkg_temp_thermal intel_powerclamp snd_seq_midi coretemp snd_seq_midi_event kvm_intel kvm irqbypass iwlmvm crct10dif_pclmul snd_rawmidi crc32_pclmul mac80211 ghash_clmulni_intel rtsx_pci_ms pcbc snd_seq memstick thunderbolt iwlwifi nvmem_core cfg80211 snd_seq_device
[   92.325218]  aesni_intel snd_timer aes_x86_64 crypto_simd glue_helper cryptd snd joydev input_leds serio_raw soundcore shpchp idma64 virt_dma mei_me processor_thermal_device mei intel_pch_thermal intel_lpss_pci intel_soc_dts_iosf hci_uart btbcm btqca btintel bluetooth intel_lpss_acpi ecdh_generic intel_lpss int3403_thermal int340x_thermal_zone intel_hid int3400_thermal sparse_keymap acpi_thermal_rel tpm_crb acpi_pad mac_hid acpi_als kfifo_buf industrialio parport_pc ppdev lp parport autofs4 btrfs xor raid6_pq usbhid rtsx_pci_sdmmc i915 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt nvme psmouse nvme_core fb_sys_fops rtsx_pci ahci drm libahci i2c_hid hid wmi pinctrl_sunrisepoint pinctrl_intel video
[   92.325243] CPU: 2 PID: 5 Comm: kworker/u8:0 Tainted: G        W       4.12.0-rc1+ #6
[   92.325244] Hardware name: Dell Inc. XPS 13 9350/09JHRY, BIOS 1.4.17 05/10/2017
[   92.325246] Workqueue: kacpi_hotplug acpi_hotplug_work_fn
[   92.325247] task: ffff9af62e6bbb00 task.stack: ffffbb3a018f8000
[   92.325249] RIP: 0010:pci_add_new_bus+0x394/0x420
[   92.325250] RSP: 0018:ffffbb3a018fbae8 EFLAGS: 00010282
[   92.325251] RAX: 00000000ffffffef RBX: ffff9af62bd7e800 RCX: 0000000000000000
[   92.325251] RDX: 0000000000000001 RSI: ffff9af62bd7e968 RDI: 0000000000000001
[   92.325252] RBP: ffffbb3a018fbb28 R08: 000000000001efa0 R09: ffffffff975a0dcb
[   92.325253] R10: fffffacfd11e1c40 R11: 0000000000000001 R12: ffff9af627ba1000
[   92.325253] R13: ffff9af62ab01000 R14: ffff9af62ab01000 R15: ffff9af62bd7e920
[   92.325254] FS:  0000000000000000(0000) GS:ffff9af640d00000(0000) knlGS:0000000000000000
[   92.325255] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   92.325256] CR2: 00007f4ddb4d0760 CR3: 000000046c775000 CR4: 00000000003406e0
[   92.325256] Call Trace:
[   92.325258]  pci_scan_bridge+0x5fc/0x640
[   92.325260]  pci_scan_child_bus+0xa4/0x180
[   92.325261]  pci_scan_bridge+0x447/0x640
[   92.325263]  ? pci_scan_single_device+0x51/0xd0
[   92.325264]  pci_scan_child_bus+0xa4/0x180
[   92.325266]  pci_scan_bridge+0x447/0x640
[   92.325268]  enable_slot+0xc6/0x2f0
[   92.325269]  ? trim_stale_devices+0xab/0x150
[   92.325271]  acpiphp_check_bridge.part.7+0xff/0x140
[   92.325273]  acpiphp_hotplug_notify+0x173/0x200
[   92.325274]  ? free_bridge+0x130/0x130
[   92.325276]  acpi_device_hotplug+0x9d/0x480
[   92.325278]  acpi_hotplug_work_fn+0x1e/0x30
[   92.325279]  process_one_work+0x156/0x3f0
[   92.325280]  worker_thread+0x4b/0x410
[   92.325282]  kthread+0x109/0x140
[   92.325283]  ? process_one_work+0x3f0/0x3f0
[   92.325285]  ? kthread_create_on_node+0x70/0x70
[   92.325286]  ret_from_fork+0x2c/0x40
[   92.325287] Code: 0f 84 80 00 00 00 83 c2 02 48 63 d2 0f b6 82 79 be a6 97 eb 0b 83 f8 01 19 c0 83 e0 03 83 c0 0c 88 83 e3 00 00 00 e9 7d fe ff ff <0f> ff e9 e6 fd ff ff 0f ff 48 c7 c6 4a a4 cc 97 89 c2 4c 89 ff 
[   92.325311] ---[ end trace 533975a4cf49194c ]---
[   92.325348] pci 0000:05:01.0: [8086:1578] type 01 class 0x060400
[   92.325594] pci 0000:05:01.0: supports D1 D2
[   92.325596] pci 0000:05:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[   92.325737] pci 0000:05:04.0: [8086:1578] type 01 class 0x060400
[   92.325994] pci 0000:05:04.0: supports D1 D2
[   92.325995] pci 0000:05:04.0: PME# supported from D0 D1 D2 D3hot D3cold
[   92.326164] pci 0000:04:00.0: PCI bridge to [bus 05-38]
[   92.326181] pci 0000:04:00.0:   bridge window [mem 0xc4000000-0xd9efffff]
[   92.326192] pci 0000:04:00.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[   92.326277] pci 0000:05:01.0: PCI bridge to [bus 06]
[   92.326384] pci 0000:05:04.0: PCI bridge to [bus 07-38]
[   92.326401] pci 0000:05:04.0:   bridge window [mem 0xc4000000-0xd9efffff]
[   92.326412] pci 0000:05:04.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[   92.326480] pci_bus 0000:02: Allocating resources
[   92.326550] pci 0000:05:04.0: bridge window [io  0x1000-0x0fff] to [bus 07-38] add_size 1000
[   92.326568] pci 0000:04:00.0: bridge window [io  0x1000-0x0fff] to [bus 05-38] add_size 1000
[   92.326596] pci 0000:04:00.0: BAR 13: assigned [io  0x2000-0x2fff]
[   92.326598] pci 0000:05:04.0: BAR 13: assigned [io  0x2000-0x2fff]
[   92.326599] pci 0000:05:01.0: PCI bridge to [bus 06]
[   92.326624] pci 0000:05:04.0: PCI bridge to [bus 07-38]
[   92.326628] pci 0000:05:04.0:   bridge window [io  0x2000-0x2fff]
[   92.326636] pci 0000:05:04.0:   bridge window [mem 0xc4000000-0xd9efffff]
[   92.326643] pci 0000:05:04.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[   92.326654] pci 0000:04:00.0: PCI bridge to [bus 05-38]
[   92.326657] pci 0000:04:00.0:   bridge window [io  0x2000-0x2fff]
[   92.326666] pci 0000:04:00.0:   bridge window [mem 0xc4000000-0xd9efffff]
[   92.326672] pci 0000:04:00.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[   92.326732] pcieport 0000:04:00.0: enabling device (0006 -> 0007)
[   92.327330] pcieport 0000:05:04.0: enabling device (0006 -> 0007)
[   92.406055] thunderbolt 0000:03:00.0: 301: uid: 0x80864f418ce17510
[   92.406117] thunderbolt 0000:03:00.0:  Port 0: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   92.406120] thunderbolt 0000:03:00.0:   Max hop id (in/out): 7/7
[   92.406123] thunderbolt 0000:03:00.0:   Max counters: 8
[   92.406126] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   92.406326] thunderbolt 0000:03:00.0:  Port 1: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   92.406331] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   92.406334] thunderbolt 0000:03:00.0:   Max counters: 16
[   92.406337] thunderbolt 0000:03:00.0:   NFC Credits: 0x7800000
[   92.406525] thunderbolt 0000:03:00.0:  Port 2: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   92.406528] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   92.406531] thunderbolt 0000:03:00.0:   Max counters: 16
[   92.406533] thunderbolt 0000:03:00.0:   NFC Credits: 0x0
[   92.406728] thunderbolt 0000:03:00.0:  Port 3: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   92.406731] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   92.406733] thunderbolt 0000:03:00.0:   Max counters: 16
[   92.406736] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
[   92.406922] thunderbolt 0000:03:00.0:  Port 4: 8086:1578 (Revision: 4, TB Version: 1, Type: Port (0x1))
[   92.406924] thunderbolt 0000:03:00.0:   Max hop id (in/out): 15/15
[   92.406927] thunderbolt 0000:03:00.0:   Max counters: 16
[   92.406929] thunderbolt 0000:03:00.0:   NFC Credits: 0x3c00000
[   92.406932] thunderbolt 0000:03:00.0: 301:5: disabled by eeprom
[   92.406988] thunderbolt 0000:03:00.0:  Port 6: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100102))
[   92.406991] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[   92.406993] thunderbolt 0000:03:00.0:   Max counters: 2
[   92.406996] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   92.407057] thunderbolt 0000:03:00.0:  Port 7: 8086:1578 (Revision: 4, TB Version: 1, Type: PCIe (0x100101))
[   92.407060] thunderbolt 0000:03:00.0:   Max hop id (in/out): 8/8
[   92.407062] thunderbolt 0000:03:00.0:   Max counters: 2
[   92.407065] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   92.407122] thunderbolt 0000:03:00.0:  Port 8: 8086:1578 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0102))
[   92.407124] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
[   92.407127] thunderbolt 0000:03:00.0:   Max counters: 2
[   92.407129] thunderbolt 0000:03:00.0:   NFC Credits: 0x800000
[   92.407132] thunderbolt 0000:03:00.0: 301:9: disabled by eeprom
[   92.407187] thunderbolt 0000:03:00.0:  Port 10: 8086:1578 (Revision: 4, TB Version: 1, Type: DP/HDMI (0xe0101))
[   92.407190] thunderbolt 0000:03:00.0:   Max hop id (in/out): 9/9
[   92.407192] thunderbolt 0000:03:00.0:   Max counters: 2
[   92.407195] thunderbolt 0000:03:00.0:   NFC Credits: 0x1000000
[   92.407197] thunderbolt 0000:03:00.0: 301:b: disabled by eeprom
[   97.740817] ACPI: \_SB_.PCI0.RP01: acpiphp_glue: Bus check in hotplug_event()
[   97.761204] pci 0000:07:00.0: [8086:1578] type 01 class 0x060400
[   97.761590] pci 0000:07:00.0: supports D1 D2
[   97.761592] pci 0000:07:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[   97.762071] pci 0000:08:01.0: [8086:1578] type 01 class 0x060400
[   97.762470] pci 0000:08:01.0: supports D1 D2
[   97.762472] pci 0000:08:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[   97.762683] pci 0000:08:04.0: [8086:1578] type 01 class 0x060400
[   97.763083] pci 0000:08:04.0: supports D1 D2
[   97.763084] pci 0000:08:04.0: PME# supported from D0 D1 D2 D3hot D3cold
[   97.763347] pci 0000:07:00.0: PCI bridge to [bus 08-38]
[   97.763364] pci 0000:07:00.0:   bridge window [io  0x2000-0x2fff]
[   97.763373] pci 0000:07:00.0:   bridge window [mem 0xc4000000-0xd9efffff]
[   97.763391] pci 0000:07:00.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[   97.763577] pci 0000:09:00.0: [1b21:1142] type 00 class 0x0c0330
[   97.763666] pci 0000:09:00.0: reg 0x10: [mem 0xc4000000-0xc4007fff 64bit]
[   97.764086] pci 0000:09:00.0: PME# supported from D3cold
[   97.764350] pci 0000:08:01.0: PCI bridge to [bus 09]
[   97.764368] pci 0000:08:01.0:   bridge window [io  0x2000-0x2fff]
[   97.764377] pci 0000:08:01.0:   bridge window [mem 0xc4000000-0xc40fffff]
[   97.764740] pci 0000:08:04.0: PCI bridge to [bus 0a-38]
[   97.764767] pci 0000:08:04.0:   bridge window [mem 0xc4100000-0xd9efffff]
[   97.764785] pci 0000:08:04.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[   97.764924] pci_bus 0000:02: Allocating resources
[   97.765123] pci 0000:08:04.0: bridge window [io  0x1000-0x0fff] to [bus 0a-38] add_size 1000
[   97.765230] pci 0000:08:04.0: BAR 13: no space for [io  size 0x1000]
[   97.765232] pci 0000:08:04.0: BAR 13: failed to assign [io  size 0x1000]
[   97.765236] pci 0000:08:04.0: BAR 13: no space for [io  size 0x1000]
[   97.765237] pci 0000:08:04.0: BAR 13: failed to assign [io  size 0x1000]
[   97.765240] pci 0000:08:01.0: PCI bridge to [bus 09]
[   97.765245] pci 0000:08:01.0:   bridge window [io  0x2000-0x2fff]
[   97.765259] pci 0000:08:01.0:   bridge window [mem 0xc4000000-0xc40fffff]
[   97.765285] pci 0000:08:04.0: PCI bridge to [bus 0a-38]
[   97.765299] pci 0000:08:04.0:   bridge window [mem 0xc4100000-0xd9efffff]
[   97.765309] pci 0000:08:04.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[   97.765326] pci 0000:07:00.0: PCI bridge to [bus 08-38]
[   97.765332] pci 0000:07:00.0:   bridge window [io  0x2000-0x2fff]
[   97.765345] pci 0000:07:00.0:   bridge window [mem 0xc4000000-0xd9efffff]
[   97.765355] pci 0000:07:00.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
[   97.767210] xhci_hcd 0000:09:00.0: xHCI Host Controller
[   97.767218] xhci_hcd 0000:09:00.0: new USB bus registered, assigned bus number 3
[   97.827955] xhci_hcd 0000:09:00.0: hcc params 0x0200e081 hci version 0x100 quirks 0x00000010
[   97.828247] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002
[   97.828248] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   97.828248] usb usb3: Product: xHCI Host Controller
[   97.828249] usb usb3: Manufacturer: Linux 4.12.0-rc1+ xhci-hcd
[   97.828250] usb usb3: SerialNumber: 0000:09:00.0
[   97.828373] hub 3-0:1.0: USB hub found
[   97.828385] hub 3-0:1.0: 2 ports detected
[   97.828471] xhci_hcd 0000:09:00.0: xHCI Host Controller
[   97.828473] xhci_hcd 0000:09:00.0: new USB bus registered, assigned bus number 4
[   97.828540] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[   97.828554] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003
[   97.828555] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   97.828557] usb usb4: Product: xHCI Host Controller
[   97.828558] usb usb4: Manufacturer: Linux 4.12.0-rc1+ xhci-hcd
[   97.828559] usb usb4: SerialNumber: 0000:09:00.0
[   97.830106] hub 4-0:1.0: USB hub found
[   97.830120] hub 4-0:1.0: 2 ports detected
[   98.212766] usb 3-1: new high-speed USB device number 2 using xhci_hcd
[   98.414092] usb 3-1: New USB device found, idVendor=0424, idProduct=2137
[   98.414096] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   98.414098] usb 3-1: Product: USB2137B
[   98.414100] usb 3-1: Manufacturer: SMSC
[   98.415534] hub 3-1:1.0: USB hub found
[   98.415602] hub 3-1:1.0: 7 ports detected
[   98.533098] usb 4-1: new SuperSpeed USB device number 2 using xhci_hcd
[   98.558798] usb 4-1: New USB device found, idVendor=0424, idProduct=5537
[   98.558806] usb 4-1: New USB device strings: Mfr=2, Product=3, SerialNumber=0
[   98.558811] usb 4-1: Product: USB5537B
[   98.558815] usb 4-1: Manufacturer: SMSC
[   98.560717] hub 4-1:1.0: USB hub found
[   98.561167] hub 4-1:1.0: 7 ports detected
[   98.704710] usb 3-1.5: new high-speed USB device number 3 using xhci_hcd
[   98.886109] usb 3-1.5: New USB device found, idVendor=0bda, idProduct=4014
[   98.886117] usb 3-1.5: New USB device strings: Mfr=3, Product=1, SerialNumber=2
[   98.886122] usb 3-1.5: Product: USB Audio
[   98.886126] usb 3-1.5: Manufacturer: Generic
[   98.886130] usb 3-1.5: SerialNumber: 200901010001
[   98.952814] usb 4-1.2: new SuperSpeed USB device number 3 using xhci_hcd
[   98.979824] usb 4-1.2: New USB device found, idVendor=0bda, idProduct=8153
[   98.979831] usb 4-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=6
[   98.979837] usb 4-1.2: Product: USB 10/100/1000 LAN
[   98.979842] usb 4-1.2: Manufacturer: Realtek
[   98.979846] usb 4-1.2: SerialNumber: 000001000000
[  100.302938] usbcore: registered new interface driver r8152
[  100.311888] usbcore: registered new interface driver cdc_ether
[  100.378549] usbcore: registered new interface driver snd-usb-audio
[  100.454308] usb 4-1.2: reset SuperSpeed USB device number 3 using xhci_hcd
[  100.617455] r8152 4-1.2:1.0 (unnamed net_device) (uninitialized): Using pass-thru MAC addr 84:7b:eb:59:92:d2
[  100.693273] r8152 4-1.2:1.0 eth0: v1.08.9
[  101.432339] r8152 4-1.2:1.0 enx847beb5992d2: renamed from eth0
[  101.467115] IPv6: ADDRCONF(NETDEV_UP): enx847beb5992d2: link is not ready
[  101.511190] IPv6: ADDRCONF(NETDEV_UP): enx847beb5992d2: link is not ready

Re: [PATCH 06/24] thunderbolt: Introduce thunderbolt bus and connection manager

[Lukas Wunner]

On Thu, May 18, 2017 at 05:38:56PM +0300, Mika Westerberg wrote:
> Thunderbolt fabric consists of one or more switches. This fabric is
> called domain and it is controlled by an entity called connection
> manager. The connection manager can be either internal (driven by a
> firmware running on the host controller) or external (software driver).
> This driver currently implements support for the latter.
> 
> In order to manage switches and their properties more easily we model
> this domain structure as a Linux bus. Each host controller adds a domain
> device to this bus, and these devices are named as domainN where N
> stands for index or id of the current domain.

What is the rationale for using a bus?  Do you intend to attach a driver
to each domain and/or switch?  Unless you do, a class might be more
appropriate.

Thanks,

Lukas

Re: [PATCH 06/24] thunderbolt: Introduce thunderbolt bus and connection manager

[Mika Westerberg]

On Wed, May 24, 2017 at 12:28:21PM +0200, Lukas Wunner wrote:
> On Thu, May 18, 2017 at 05:38:56PM +0300, Mika Westerberg wrote:
> > Thunderbolt fabric consists of one or more switches. This fabric is
> > called domain and it is controlled by an entity called connection
> > manager. The connection manager can be either internal (driven by a
> > firmware running on the host controller) or external (software driver).
> > This driver currently implements support for the latter.
> > 
> > In order to manage switches and their properties more easily we model
> > this domain structure as a Linux bus. Each host controller adds a domain
> > device to this bus, and these devices are named as domainN where N
> > stands for index or id of the current domain.
> 
> What is the rationale for using a bus?  Do you intend to attach a driver
> to each domain and/or switch?  Unless you do, a class might be more
> appropriate.

It is used to make switches (and the host) identification available for
the userspace. This identification information is used to make a
decision whether or not a device should be authorized.

There will be (in subsequent patches, currently work in progress) also
other type of devices (XDomain) which XDomain service drivers bind to in
order to send data, like IP traffic over Thunderbolt link(s).

And last, Thunderbolt is actually a bus as far as I can tell so we model
it accordingly ;-)

Re: [PATCH 07/24] thunderbolt: Convert switch to a device

[Lukas Wunner]

On Thu, May 18, 2017 at 05:38:57PM +0300, Mika Westerberg wrote:
> Thunderbolt domain consists of switches that are connected to each
> other, forming a bus. This will convert each switch into a real Linux
> device structure and adds them to the domain. The advantage here is
> that we get all the goodies from the driver core, like reference
> counting and sysfs hierarchy for free.

I'm wondering, would it make sense to also model each port of a switch
as a device?

With your patches, the hierarchy looks something like this:
nhi_pci_dev/domain0/switch0/switch1/switch2

If each port is modeled as a device, we'd get something like this:
nhi_pci_dev/domain0/switch0/port1/switch1/port3/switch2

I think with the controllers that shipped, there can be up to two
child switches below a switch.  If ports are not modeled as devices,
you can't tell which port a switch is connected to.

Also, if ports are modeled as devices we'd be able to store attributes
such as port type in sysfs.  Of course we could also store those in
each switch directory as "port0", "port1", ... files.

I don't know if this makes sense, but a discussion about the pros
and cons is probably warranted.  IIUC once the patches go in, the
layout is set in stone because it's a user-space API.


> +What:		/sys/bus/thunderbolt/devices/.../device
> +Date:		Sep 2017
> +KernelVersion:	4.13
> +Contact:	thunderbolt-software@lists.01.org
> +Description:	This attribute contains id of this device extracted from
> +		the device DROM.

Hm, why did you choose to expose the DROM vendor + device ID instead of
the one in switch config space?

The vendor on my MacBookPro9,1 is just 0x0001 and the device is 0x000a.
Is this valid?  Does Intel assign the vendor IDs, i.e. 0x0001 = Apple?


> +	if (sw->config.vendor_id != PCI_VENDOR_ID_INTEL)
> +		tb_sw_warn(sw, "unknown switch vendor id %#x\n",
> +			   sw->config.vendor_id);
> +
> +	if (sw->config.device_id != PCI_DEVICE_ID_INTEL_LIGHT_RIDGE &&
> +	    sw->config.device_id != PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C &&
> +	    sw->config.device_id != PCI_DEVICE_ID_INTEL_PORT_RIDGE &&
> +	    sw->config.device_id != PCI_DEVICE_ID_INTEL_FALCON_RIDGE_2C_BRIDGE &&
> +	    sw->config.device_id != PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_BRIDGE)
> +		tb_sw_warn(sw, "unsupported switch device id %#x\n",
> +			   sw->config.device_id);

I'm wondering if this makes sense anymore, as said we should try to avoid
having to amend device lists in multiple places in the driver for each
newly introduced controller.


> +static void tb_switch_set_uuid(struct tb_switch *sw)
> +{
> +	u32 uuid[4];
> +	int cap;
> +
> +	if (sw->uuid)
> +		return;

When can this be nonzero?


> +
> +	/*
> +	 * By default the UUID will be based on UID where upper two
> +	 * dwords are filled with ones.
> +	 */
> +	uuid[0] = sw->uid & 0xffffffff;
> +	uuid[1] = (sw->uid >> 32) & 0xffffffff;
> +	uuid[2] = 0xffffffff;
> +	uuid[3] = 0xffffffff;
> +
> +	/*
> +	 * The newer controllers include fused UUID as part of link
> +	 * controller specific registers
> +	 */
> +	cap = tb_switch_find_vsec_cap(sw, TB_VSEC_CAP_LINK_CONTROLLER);
> +	if (cap > 0)
> +		tb_sw_read(sw, uuid, TB_CFG_SWITCH, cap + 3, 4);
> +
> +	sw->uuid = kmemdup(uuid, sizeof(uuid), GFP_KERNEL);

Hm, so on newer controller the uuid is calculated and the result is
subsequently overwritten?  Meh... :-/

On Macs, the UUID is conveyed in an EFI device property.

How about putting the VSEC code path first, then fall back to calculating
the default UUID?  Apart from being prettier, this would allow me to
easily add the Mac-specific code path.

BTW, why is there a uid for each switch and a UUID on top?
IIUC, if the switch is the root switch, then the UUID is used to
uniquely identify the domain starting at that switch, is that correct?
Actually just using the 64-bit uid would still suffice for that use case.
That is something I've never really comprehended.

Thanks,

Lukas

Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Mika Westerberg]

On Tue, May 23, 2017 at 05:30:43PM +0000, Mario.Limonciello@dell.com wrote:
> (Sorry my email client is not going to wrap these at 80 columns)o

That's fine. It is more readable this way :)

> [    0.467319] pci 0000:00:1c.0: [8086:9d10] type 01 class 0x060400
> [    0.467389] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
> [    0.467513] pci 0000:00:1c.0: System wakeup disabled by ACPI

[...]

> [    0.469363] pci 0000:01:00.0: [8086:1576] type 01 class 0x060400
> [    0.469483] pci 0000:01:00.0: supports D1 D2
> [    0.469484] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    0.469570] pci 0000:01:00.0: System wakeup disabled by ACPI
> [    0.469609] pci 0000:00:1c.0: PCI bridge to [bus 01-39]
> [    0.469614] pci 0000:00:1c.0:   bridge window [mem 0xc4000000-0xda0fffff]
> [    0.469618] pci 0000:00:1c.0:   bridge window [mem 0xa0000000-0xc1ffffff 64bit pref]
> [    0.469621] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring

This is the problem. Here the PCIe upstream port (0000:01:00.0) is
visible to Linux but it is not fully configured by the BIOS ->
(primary/secondary/subordinate) is set to 0.

At this point Linux decides to configure the port itself and goes wrong
since our allocation strategy tries to keep resource windows, including
reserved buses as small as possible so that everything we currently find
barely fits there.

This continues few lines below:

> [    0.469670] pci_bus 0000:02: busn_res: can not insert [bus 02-ff] under [bus 01-39] (conflicts with (null) [bus 01-39])
> [    0.469688] pci 0000:02:00.0: [8086:1576] type 01 class 0x060400
> [    0.469809] pci 0000:02:00.0: supports D1 D2
> [    0.469810] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    0.469877] pci 0000:02:01.0: [8086:1576] type 01 class 0x060400
> [    0.470000] pci 0000:02:01.0: supports D1 D2
> [    0.470001] pci 0000:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    0.470067] pci 0000:02:02.0: [8086:1576] type 01 class 0x060400
> [    0.470188] pci 0000:02:02.0: supports D1 D2
> [    0.470189] pci 0000:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    0.470277] pci 0000:01:00.0: PCI bridge to [bus 02-ff]
> [    0.470283] pci 0000:01:00.0:   bridge window [io  0x0000-0x0fff]
> [    0.470287] pci 0000:01:00.0:   bridge window [mem 0x00000000-0x000fffff]
> [    0.470294] pci 0000:01:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
> [    0.470296] pci 0000:02:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> [    0.470304] pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> [    0.470312] pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring

Here.

And ends up in failure when we create PCIe tunnels later on.

Now, this is probably where Windows does something else, like it may
skip re-configuring phase which could explain why it works. However, to
me this looks pretty much like a bug in the BIOS/firmware as we are
expecting the BIOS to configure the PCIe devices properly before the OS
is send ACPI hotplug event.

We need to handle this in Linux in the same way Windows does but
currently I have no idea. It is however, more related to our PCI
enumeration code than the patches in question, I think.

I also have a Dell 9350 here so I can reproduce the problem and I'm
going to investigate this further probably involving Linux PCI people.

My testing on the machine shows this behaviour only when the cable is
connected during boot.

If I connect the cable after OS is booted I don't see the problem, even
if I do unplug / plug cycle.

Can you try that also (again)? And if you see the problem, send me the
dmesg? I have the latest BIOS (1.4.17) and NVM 16 so this machine
configuration should match yours if I'm not mistaken.

Re: [PATCH 07/24] thunderbolt: Convert switch to a device

[Mika Westerberg]

On Wed, May 24, 2017 at 01:09:08PM +0200, Lukas Wunner wrote:
> On Thu, May 18, 2017 at 05:38:57PM +0300, Mika Westerberg wrote:
> > Thunderbolt domain consists of switches that are connected to each
> > other, forming a bus. This will convert each switch into a real Linux
> > device structure and adds them to the domain. The advantage here is
> > that we get all the goodies from the driver core, like reference
> > counting and sysfs hierarchy for free.
> 
> I'm wondering, would it make sense to also model each port of a switch
> as a device?
> 
> With your patches, the hierarchy looks something like this:
> nhi_pci_dev/domain0/switch0/switch1/switch2
> 
> If each port is modeled as a device, we'd get something like this:
> nhi_pci_dev/domain0/switch0/port1/switch1/port3/switch2
> 
> I think with the controllers that shipped, there can be up to two
> child switches below a switch.  If ports are not modeled as devices,
> you can't tell which port a switch is connected to.

Yes you can - it is part of the route string that we use here as bus
address. Each "hop" there is the port number through the switch is
accessed. For example I have here 4 devices connected:

$ ls -1 /sys/bus/thunderbolt/devices/
0-0
0-1
0-1040301
0-301
0-40301
domain0

> Also, if ports are modeled as devices we'd be able to store attributes
> such as port type in sysfs.  Of course we could also store those in
> each switch directory as "port0", "port1", ... files.

Is there any reason adding these? It will not help the user to identify
the connected device and I don't see how that information could be
useful.

That kind of information could go to debugfs, though.

> I don't know if this makes sense, but a discussion about the pros
> and cons is probably warranted.  IIUC once the patches go in, the
> layout is set in stone because it's a user-space API.

Yes, that's right. Also that is one of the reasons why this series tries
to keep the attribute amout minimum and their contents simple.

> > +What:		/sys/bus/thunderbolt/devices/.../device
> > +Date:		Sep 2017
> > +KernelVersion:	4.13
> > +Contact:	thunderbolt-software@lists.01.org
> > +Description:	This attribute contains id of this device extracted from
> > +		the device DROM.
> 
> Hm, why did you choose to expose the DROM vendor + device ID instead of
> the one in switch config space?

That is the actual vendor/device who made the thing and can be used to
identify the actual device as well.

> The vendor on my MacBookPro9,1 is just 0x0001 and the device is 0x000a.
> Is this valid?  Does Intel assign the vendor IDs, i.e. 0x0001 = Apple?

Apple vendor ID in Thunderbolt realm is 0x0001. Yes, Intel assigns
those.

> > +	if (sw->config.vendor_id != PCI_VENDOR_ID_INTEL)
> > +		tb_sw_warn(sw, "unknown switch vendor id %#x\n",
> > +			   sw->config.vendor_id);
> > +
> > +	if (sw->config.device_id != PCI_DEVICE_ID_INTEL_LIGHT_RIDGE &&
> > +	    sw->config.device_id != PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C &&
> > +	    sw->config.device_id != PCI_DEVICE_ID_INTEL_PORT_RIDGE &&
> > +	    sw->config.device_id != PCI_DEVICE_ID_INTEL_FALCON_RIDGE_2C_BRIDGE &&
> > +	    sw->config.device_id != PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_BRIDGE)
> > +		tb_sw_warn(sw, "unsupported switch device id %#x\n",
> > +			   sw->config.device_id);
> 
> I'm wondering if this makes sense anymore, as said we should try to avoid
> having to amend device lists in multiple places in the driver for each
> newly introduced controller.

I agree. In the next version I'm moving this to be part of the code
where we identify the generation. So at least it will be removed from
this place.

> > +static void tb_switch_set_uuid(struct tb_switch *sw)
> > +{
> > +	u32 uuid[4];
> > +	int cap;
> > +
> > +	if (sw->uuid)
> > +		return;
> 
> When can this be nonzero?

On Macs with older controller than AR (no ICM running).

> > +
> > +	/*
> > +	 * By default the UUID will be based on UID where upper two
> > +	 * dwords are filled with ones.
> > +	 */
> > +	uuid[0] = sw->uid & 0xffffffff;
> > +	uuid[1] = (sw->uid >> 32) & 0xffffffff;
> > +	uuid[2] = 0xffffffff;
> > +	uuid[3] = 0xffffffff;
> > +
> > +	/*
> > +	 * The newer controllers include fused UUID as part of link
> > +	 * controller specific registers
> > +	 */
> > +	cap = tb_switch_find_vsec_cap(sw, TB_VSEC_CAP_LINK_CONTROLLER);
> > +	if (cap > 0)
> > +		tb_sw_read(sw, uuid, TB_CFG_SWITCH, cap + 3, 4);
> > +
> > +	sw->uuid = kmemdup(uuid, sizeof(uuid), GFP_KERNEL);
> 
> Hm, so on newer controller the uuid is calculated and the result is
> subsequently overwritten?  Meh... :-/

ICM gives the UUID as part of the device connected message. This here to
make the UUID working also on systems withouth ICM (older Macs). The
special case comes from the older devices where there is no fused UUID
so we generate one based on UID instead following what ICM does.

This allows us to show "unique_id" attribute the same way on all
systems.

> On Macs, the UUID is conveyed in an EFI device property.

Yes, but only for the host and UUID is actually coming from the fuses
(hardware), not from EFI property.

> How about putting the VSEC code path first, then fall back to calculating
> the default UUID?  Apart from being prettier, this would allow me to
> easily add the Mac-specific code path.

I can change the ordering but you don't need to add Mac specific path
there - this code has been tested on Macs already and it should work
exactly the same there.

I have here Mac Mini (Light Ridge), Macbook (Cactus Ridge) and newer
Macbook with Alpine Ridge and I've tried to make sure it works the same.

Below is taken from the same devices connected to a Cactus Ridge based
Mac.

0-0/authorized:1
0-0/device:0xa
0-0/device_name:Macintosh
0-0/uevent:DEVTYPE=thunderbolt_device
0-0/unique_id:00000000-0000-0008-83b3-30099bc1194a
0-0/vendor:0x1
0-0/vendor_name:Apple, Inc.
0-1/authorized:1
0-1/device:0x5
0-1/device_name:34UC97
0-1/uevent:DEVTYPE=thunderbolt_device
0-1/unique_id:00000000-0000-0018-0022-c32682b35855
0-1/vendor:0x1e
0-1/vendor_name:LG Electronics
0-1030301/authorized:1
0-1030301/device:0x301
0-1030301/device_name:AKiTiO Thunder3 Duo Pro
0-1030301/nvm_authenticate:0x0
0-1030301/nvm_version:16.0
0-1030301/uevent:DEVTYPE=thunderbolt_device
0-1030301/unique_id:d3010000-0000-9518-a29c-9bc5cca35116
0-1030301/vendor:0x41
0-1030301/vendor_name:inXtron
0-301/authorized:1
0-301/device:0x3
0-301/device_name:eSata Hub
0-301/uevent:DEVTYPE=thunderbolt_device
0-301/unique_id:102f9d1e-0000-0300-ffff-ffffffffffff
0-301/vendor:0x3
0-301/vendor_name:LaCie
0-30301/authorized:1
0-30301/device:0x305
0-30301/device_name:AKiTiO Thunder3 PCIe Box
0-30301/nvm_authenticate:0x0
0-30301/nvm_version:19.0
0-30301/uevent:DEVTYPE=thunderbolt_device
0-30301/unique_id:dc010000-0000-8508-a22d-32ca6421cb16
0-30301/vendor:0x41
0-30301/vendor_name:inXtron
domain0/security:none
domain0/uevent:DEVTYPE=thunderbolt_domain

> BTW, why is there a uid for each switch and a UUID on top?

I think the UUID came with Redwood Ridge or so so before that there was
only UID.

> IIUC, if the switch is the root switch, then the UUID is used to
> uniquely identify the domain starting at that switch, is that correct?

Yes, that's correct.

> Actually just using the 64-bit uid would still suffice for that use case.
> That is something I've never really comprehended.

Well they decided to go with full UUID and I can't blame them ;-)

Re: [PATCH 07/24] thunderbolt: Convert switch to a device

[Lukas Wunner]

On Wed, May 24, 2017 at 02:43:22PM +0300, Mika Westerberg wrote:
> On Wed, May 24, 2017 at 01:09:08PM +0200, Lukas Wunner wrote:
> > On Thu, May 18, 2017 at 05:38:57PM +0300, Mika Westerberg wrote:
> > > Thunderbolt domain consists of switches that are connected to each
> > > other, forming a bus. This will convert each switch into a real Linux
> > > device structure and adds them to the domain. The advantage here is
> > > that we get all the goodies from the driver core, like reference
> > > counting and sysfs hierarchy for free.
> > 
> > I'm wondering, would it make sense to also model each port of a switch
> > as a device?
> > 
> > With your patches, the hierarchy looks something like this:
> > nhi_pci_dev/domain0/switch0/switch1/switch2
> > 
> > If each port is modeled as a device, we'd get something like this:
> > nhi_pci_dev/domain0/switch0/port1/switch1/port3/switch2
> > 
> > I think with the controllers that shipped, there can be up to two
> > child switches below a switch.  If ports are not modeled as devices,
> > you can't tell which port a switch is connected to.
> 
> Yes you can - it is part of the route string that we use here as bus
> address. Each "hop" there is the port number through the switch is
> accessed. For example I have here 4 devices connected:
> 
> $ ls -1 /sys/bus/thunderbolt/devices/
> 0-0
> 0-1
> 0-1040301
> 0-301
> 0-40301
> domain0

I see.

Is it possible to determine for a given port of type PCIe to which
downstream bridge it belongs?

E.g. on my Light Ridge, the PCIe ports are numbered 7, 8, 9, 10.
The downstream bridges on the controller are numbered 03, 04, 05, 06.
But the ordering seems to be mixed up, e.g. port 7 corresponds to
downstream bridge 0000:06:04.0.  Not to 03, as one would expect.
                          ^^

Can this perhaps be determined from config space?

If we knew the correlation between Thunderbolt PCIe port and downstream
bridge, we could provide a symlink in sysfs from the Thunderbolt bus
to the PCI bus.  E.g. in the switch directory for a plugged in device,
there would be a symlink to the corresponding upstream bridge.  For the
root switch, this would be the upstream bridge of the host controller.


> > Also, if ports are modeled as devices we'd be able to store attributes
> > such as port type in sysfs.  Of course we could also store those in
> > each switch directory as "port0", "port1", ... files.
> 
> Is there any reason adding these? It will not help the user to identify
> the connected device and I don't see how that information could be
> useful.
>
> That kind of information could go to debugfs, though.

Well, currently we print that stuff to syslog and it's valuable to
understand what's going on.  Being able to check e.g. the type of
a port on a running system would be even better.  I'm not saying
we should model ports as devices, I just want to have a discussion
about the pros and cons.


> > On Macs, the UUID is conveyed in an EFI device property.
> 
> Yes, but only for the host and UUID is actually coming from the fuses
> (hardware), not from EFI property.

Nope, on Macs the UUID is calculated by sha1-hashing a constant, then
extending that by sha1-hashing the uid, then truncating the result to
16 bytes.

The uid in turn is calculated by combining a 16-bit constant with a
24-bit model-specific number and a 24-bit serial number.

This is done by the EFI NHI driver.  No fuses involved.

(Okay the serial number is coming from an EEPROM, but not that of
the Thunderbolt controller).


> > > +
> > > +	/*
> > > +	 * By default the UUID will be based on UID where upper two
> > > +	 * dwords are filled with ones.
> > > +	 */
> > > +	uuid[0] = sw->uid & 0xffffffff;
> > > +	uuid[1] = (sw->uid >> 32) & 0xffffffff;
> > > +	uuid[2] = 0xffffffff;
> > > +	uuid[3] = 0xffffffff;
> > > +
> > > +	/*
> > > +	 * The newer controllers include fused UUID as part of link
> > > +	 * controller specific registers
> > > +	 */
> > > +	cap = tb_switch_find_vsec_cap(sw, TB_VSEC_CAP_LINK_CONTROLLER);
> > > +	if (cap > 0)
> > > +		tb_sw_read(sw, uuid, TB_CFG_SWITCH, cap + 3, 4);
> > > +
> > > +	sw->uuid = kmemdup(uuid, sizeof(uuid), GFP_KERNEL);
> > 
> > Hm, so on newer controller the uuid is calculated and the result is
> > subsequently overwritten?  Meh... :-/
> 
> ICM gives the UUID as part of the device connected message. This here to
> make the UUID working also on systems withouth ICM (older Macs). The
> special case comes from the older devices where there is no fused UUID
> so we generate one based on UID instead following what ICM does.
> 
> This allows us to show "unique_id" attribute the same way on all
> systems.

I'll have to double check if the macOS NHI driver calculates a UUID for
each switch, and how it does that.  We should try to be compatible.


> > How about putting the VSEC code path first, then fall back to calculating
> > the default UUID?  Apart from being prettier, this would allow me to
> > easily add the Mac-specific code path.
> 
> I can change the ordering but you don't need to add Mac specific path
> there - this code has been tested on Macs already and it should work
> exactly the same there.

I don't doubt that it works, the problem is that the UUID should be
identical to what macOS uses.


> Below is taken from the same devices connected to a Cactus Ridge based
> Mac.
> 
> 0-0/authorized:1
> 0-0/device:0xa
> 0-0/device_name:Macintosh
> 0-0/uevent:DEVTYPE=thunderbolt_device
> 0-0/unique_id:00000000-0000-0008-83b3-30099bc1194a
> 0-0/vendor:0x1
> 0-0/vendor_name:Apple, Inc.

Interesting, however this means that the device ID is identical across
all Macs.  (0xa, same as on my MacBookPro9,1 w/ Light Ridge.)


> > IIUC, if the switch is the root switch, then the UUID is used to
> > uniquely identify the domain starting at that switch, is that correct?
> 
> Yes, that's correct.

Why is a UUID calculated for each switch on the chain even if only the
UUID on the root switch is needed to give the domain a unique ID?

Thanks,

Lukas

Re: [PATCH 14/24] thunderbolt: Let the connection manager handle all notifications

[Lukas Wunner]

On Thu, May 18, 2017 at 05:39:04PM +0300, Mika Westerberg wrote:
> @@ -320,18 +330,42 @@ static void tb_ctl_rx_callback(struct tb_ring *ring, struct ring_frame *frame,
>  	}
>  
>  	frame->size -= 4; /* remove checksum */
> -	if (*(__be32 *) (pkg->buffer + frame->size)
> -			!= tb_crc(pkg->buffer, frame->size)) {
> -		tb_ctl_err(pkg->ctl,
> -			   "RX: checksum mismatch, dropping packet\n");
> -		goto rx;
> -	}
> +	crc32 = tb_crc(pkg->buffer, frame->size);
>  	be32_to_cpu_array(pkg->buffer, pkg->buffer, frame->size / 4);
>  
> -	if (frame->eof == TB_CFG_PKG_EVENT) {
> -		tb_ctl_handle_plug_event(pkg->ctl, pkg);
> +	switch (frame->eof) {
> +	case TB_CFG_PKG_READ:
> +	case TB_CFG_PKG_WRITE:
> +	case TB_CFG_PKG_ERROR:
> +	case TB_CFG_PKG_OVERRIDE:
> +	case TB_CFG_PKG_RESET:
> +		if (*(__be32 *)(pkg->buffer + frame->size) != crc32) {
> +			tb_ctl_err(pkg->ctl,
> +				   "RX: checksum mismatch, dropping packet\n");
> +			goto rx;
> +		}

Any harm keeping the crc32 check above the switch/case statement?
(And thus also execute it for unknown packets?)


> +	case TB_CFG_PKG_EVENT:
> +		if (*(__be32 *)(pkg->buffer + frame->size) != crc32) {
> +			tb_ctl_err(pkg->ctl,
> +				   "RX: checksum mismatch, dropping packet\n");
> +			goto rx;
> +		}
> +		tb_ctl_handle_event(pkg->ctl, frame->eof, pkg, frame->size);
> +		goto rx;
> +
> +	default:
> +		tb_ctl_dbg(pkg->ctl, "RX: unknown package %#x, dropping\n",

The packet / package terminology is a bit inconsistent here.
Andreas originally used package.  What's the term used by the TB spec?

Thanks,

Lukas

Re: [PATCH 20/24] thunderbolt: Do not touch the hardware if the NHI is gone on resume

[Lukas Wunner]

On Thu, May 18, 2017 at 05:39:10PM +0300, Mika Westerberg wrote:
> @@ -655,6 +659,16 @@ static int nhi_resume_noirq(struct device *dev)
>  {
>  	struct pci_dev *pdev = to_pci_dev(dev);
>  	struct tb *tb = pci_get_drvdata(pdev);
> +	u32 vid;
> +
> +	/*
> +	 * Check that the device is still there. It may be that the user
> +	 * unplugged last device which causes the host controller to go
> +	 * away on PCs.
> +	 */
> +	pci_read_config_dword(pdev, PCI_VENDOR_ID, &vid);
> +	if (vid == ~0)
> +		tb->nhi->going_away = true;

if (!pci_device_is_present(pdev))

Thanks,

Lukas

RE: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Mario.Limonciello]

> -----Original Message-----
> From: Mika Westerberg [mailto:mika.westerberg@linux.intel.com]
> Sent: Wednesday, May 24, 2017 6:11 AM
> To: Limonciello, Mario <Mario_Limonciello@Dell.com>
> Cc: gregkh@linuxfoundation.org; andreas.noever@gmail.com;
> michael.jamet@intel.com; yehezkel.bernat@intel.com; lukas@wunner.de;
> amir.jer.levy@intel.com; luto@kernel.org; Dominguez, Jared
> <Jared_Dominguez@DELL.com>; andriy.shevchenko@linux.intel.com; linux-
> kernel@vger.kernel.org
> Subject: Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade
> 
> On Tue, May 23, 2017 at 05:30:43PM +0000, Mario.Limonciello@dell.com wrote:
> > (Sorry my email client is not going to wrap these at 80 columns)o
> 
> That's fine. It is more readable this way :)
> 
> > [    0.467319] pci 0000:00:1c.0: [8086:9d10] type 01 class 0x060400
> > [    0.467389] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
> > [    0.467513] pci 0000:00:1c.0: System wakeup disabled by ACPI
> 
> [...]
> 
> > [    0.469363] pci 0000:01:00.0: [8086:1576] type 01 class 0x060400
> > [    0.469483] pci 0000:01:00.0: supports D1 D2
> > [    0.469484] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [    0.469570] pci 0000:01:00.0: System wakeup disabled by ACPI
> > [    0.469609] pci 0000:00:1c.0: PCI bridge to [bus 01-39]
> > [    0.469614] pci 0000:00:1c.0:   bridge window [mem 0xc4000000-0xda0fffff]
> > [    0.469618] pci 0000:00:1c.0:   bridge window [mem 0xa0000000-0xc1ffffff
> 64bit pref]
> > [    0.469621] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]),
> reconfiguring
> 
> This is the problem. Here the PCIe upstream port (0000:01:00.0) is
> visible to Linux but it is not fully configured by the BIOS ->
> (primary/secondary/subordinate) is set to 0.

So at least for me the other difference between a successful run (where you plug
in after boot instead) is that it shows up as instead:
PCI bridge to [bus 02-39]

Same bridge window though.

> 
> At this point Linux decides to configure the port itself and goes wrong
> since our allocation strategy tries to keep resource windows, including
> reserved buses as small as possible so that everything we currently find
> barely fits there.
> 
> This continues few lines below:
> 
> > [    0.469670] pci_bus 0000:02: busn_res: can not insert [bus 02-ff] under [bus 01-
> 39] (conflicts with (null) [bus 01-39])
> > [    0.469688] pci 0000:02:00.0: [8086:1576] type 01 class 0x060400
> > [    0.469809] pci 0000:02:00.0: supports D1 D2
> > [    0.469810] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [    0.469877] pci 0000:02:01.0: [8086:1576] type 01 class 0x060400
> > [    0.470000] pci 0000:02:01.0: supports D1 D2
> > [    0.470001] pci 0000:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [    0.470067] pci 0000:02:02.0: [8086:1576] type 01 class 0x060400
> > [    0.470188] pci 0000:02:02.0: supports D1 D2
> > [    0.470189] pci 0000:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [    0.470277] pci 0000:01:00.0: PCI bridge to [bus 02-ff]
> > [    0.470283] pci 0000:01:00.0:   bridge window [io  0x0000-0x0fff]
> > [    0.470287] pci 0000:01:00.0:   bridge window [mem 0x00000000-0x000fffff]
> > [    0.470294] pci 0000:01:00.0:   bridge window [mem 0x00000000-0x000fffff
> 64bit pref]
> > [    0.470296] pci 0000:02:00.0: bridge configuration invalid ([bus 00-00]),
> reconfiguring
> > [    0.470304] pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]),
> reconfiguring
> > [    0.470312] pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]),
> reconfiguring
> 
> Here.
> 
> And ends up in failure when we create PCIe tunnels later on.

For what it's worth the XPS 9365 which has a different BIOS core has these
exact same behaviors on Linux if booted with the TBT dock plugged in.

> 
> Now, this is probably where Windows does something else, like it may
> skip re-configuring phase which could explain why it works. However, to
> me this looks pretty much like a bug in the BIOS/firmware as we are
> expecting the BIOS to configure the PCIe devices properly before the OS
> is send ACPI hotplug event.
> 

I'll reach out to the BIOS guys to see if they can give some more comments
from their perspective.

I came across something interesting from browsing MSDN about this topic.
It hasn't been updated in a long time but I think should still be a relevant 
indication of the approach that Windows was taking and why the firmware 
is this way and expecting OS to reconfigure.

"The BIOS cannot preconfigure PCI-to-PCI (P2P) bridges on adapters during 
hot plug. Consequently, the operating system assigns resource windows of 
a default size to a bridge.

I/O window. The default size for the I/O window is 4 KB in Windows 2000, 
Windows XP, and Windows Server 2003.  
Memory window. The configuration for the memory window differs for 
Windows 2000, Windows XP, and Windows Server 2003:
*	For Windows 2000, the default size for the memory window is 2 MB.
*	For Windows XP and Windows Server 2003, the operating system 
first attempts to find a memory window of 32 MB. If it cannot find a 
window of that size, the operating system attempts to find a memory 
window of progressively smaller sizes (16, 8, 4, 2, and finally 1 MB) until
 it finds a size that works."

> We need to handle this in Linux in the same way Windows does but
> currently I have no idea. It is however, more related to our PCI
> enumeration code than the patches in question, I think.
> 

Come to think of it, I have seen the dock have troubles if plugged in at 
boot on Linux even with SL0 before this patch series.

> I also have a Dell 9350 here so I can reproduce the problem and I'm
> going to investigate this further probably involving Linux PCI people.
To clarify are you reproducing it with a TB16 or some other TBT device?

> 
> My testing on the machine shows this behaviour only when the cable is
> connected during boot.

Yep same.

> 
> If I connect the cable after OS is booted I don't see the problem, even
> if I do unplug / plug cycle.
> 
> Can you try that also (again)? And if you see the problem, send me the
> dmesg? I have the latest BIOS (1.4.17) and NVM 16 so this machine
> configuration should match yours if I'm not mistaken.

It does work properly if I boot no cable plugged in and then plug one in.	

RE: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Jamet, Michael]

> -----Original Message-----
> From: Mario.Limonciello@dell.com [mailto:Mario.Limonciello@dell.com]
> Sent: Wednesday, May 24, 2017 22:07
> To: mika.westerberg@linux.intel.com
> Cc: gregkh@linuxfoundation.org; andreas.noever@gmail.com; Jamet,
> Michael <michael.jamet@intel.com>; Bernat, Yehezkel
> <yehezkel.bernat@intel.com>; lukas@wunner.de; Levy, Amir (Jer)
> <amir.jer.levy@intel.com>; luto@kernel.org; Jared.Dominguez@dell.com;
> andriy.shevchenko@linux.intel.com; linux-kernel@vger.kernel.org
> Subject: RE: [PATCH 00/24] Thunderbolt security levels and NVM firmware
> upgrade
> 
> > -----Original Message-----
> > From: Mika Westerberg [mailto:mika.westerberg@linux.intel.com]
> > Sent: Wednesday, May 24, 2017 6:11 AM
> > To: Limonciello, Mario <Mario_Limonciello@Dell.com>
> > Cc: gregkh@linuxfoundation.org; andreas.noever@gmail.com;
> > michael.jamet@intel.com; yehezkel.bernat@intel.com; lukas@wunner.de;
> > amir.jer.levy@intel.com; luto@kernel.org; Dominguez, Jared
> > <Jared_Dominguez@DELL.com>; andriy.shevchenko@linux.intel.com;
> linux-
> > kernel@vger.kernel.org
> > Subject: Re: [PATCH 00/24] Thunderbolt security levels and NVM
> > firmware upgrade
> >
> > On Tue, May 23, 2017 at 05:30:43PM +0000, Mario.Limonciello@dell.com
> wrote:
> > > (Sorry my email client is not going to wrap these at 80 columns)o
> >
> > That's fine. It is more readable this way :)
> >
> > > [    0.467319] pci 0000:00:1c.0: [8086:9d10] type 01 class 0x060400
> > > [    0.467389] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
> > > [    0.467513] pci 0000:00:1c.0: System wakeup disabled by ACPI
> >
> > [...]
> >
> > > [    0.469363] pci 0000:01:00.0: [8086:1576] type 01 class 0x060400
> > > [    0.469483] pci 0000:01:00.0: supports D1 D2
> > > [    0.469484] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot
> D3cold
> > > [    0.469570] pci 0000:01:00.0: System wakeup disabled by ACPI
> > > [    0.469609] pci 0000:00:1c.0: PCI bridge to [bus 01-39]
> > > [    0.469614] pci 0000:00:1c.0:   bridge window [mem 0xc4000000-
> 0xda0fffff]
> > > [    0.469618] pci 0000:00:1c.0:   bridge window [mem 0xa0000000-
> 0xc1ffffff
> > 64bit pref]
> > > [    0.469621] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]),
> > reconfiguring
> >
> > This is the problem. Here the PCIe upstream port (0000:01:00.0) is
> > visible to Linux but it is not fully configured by the BIOS ->
> > (primary/secondary/subordinate) is set to 0.
> 
> So at least for me the other difference between a successful run (where you
> plug in after boot instead) is that it shows up as instead:
> PCI bridge to [bus 02-39]
> 
> Same bridge window though.

I talked to our BIOS expert today. Here is his advice to debugging further:

It looks like something may have been wrong from system (BIOS, FW, others...) perspective.
On reboot need to enter EFI shell and check resources of 
pci 0000:01:00.0: bridge.
At the EFI shell, this bridge MUST be either configured or absent.

I would start this way, once we have this info, we may circle back to him and look into next debugging step.

/Michael
---------------------------------------------------------------------
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: [PATCH 05/24] thunderbolt: Rework capability handling

[Lukas Wunner]

On Sun, May 21, 2017 at 09:09:37PM +0200, Andreas Noever wrote:
> (and of course I have to request
> this at least once: you should definitely release the spec - I highly
> doubt that Intel's competitive advantage depends on keeping this
> linked list technology secret ;) ).

"next year Intel plans to make the Thunderbolt protocol specification
 available to the industry under a nonexclusive, royalty-free license.
 Releasing the Thunderbolt protocol specification in this manner is
 expected to greatly increase Thunderbolt adoption by encouraging
 third-party chip makers to build Thunderbolt-compatible chips."

https://newsroom.intel.com/editorials/envision-world-thunderbolt-3-everywhere/

Lukas

Re: [PATCH 07/24] thunderbolt: Convert switch to a device

[Mika Westerberg]

On Wed, May 24, 2017 at 03:53:59PM +0200, Lukas Wunner wrote:
> On Wed, May 24, 2017 at 02:43:22PM +0300, Mika Westerberg wrote:
> > On Wed, May 24, 2017 at 01:09:08PM +0200, Lukas Wunner wrote:
> > > On Thu, May 18, 2017 at 05:38:57PM +0300, Mika Westerberg wrote:
> > > > Thunderbolt domain consists of switches that are connected to each
> > > > other, forming a bus. This will convert each switch into a real Linux
> > > > device structure and adds them to the domain. The advantage here is
> > > > that we get all the goodies from the driver core, like reference
> > > > counting and sysfs hierarchy for free.
> > > 
> > > I'm wondering, would it make sense to also model each port of a switch
> > > as a device?
> > > 
> > > With your patches, the hierarchy looks something like this:
> > > nhi_pci_dev/domain0/switch0/switch1/switch2
> > > 
> > > If each port is modeled as a device, we'd get something like this:
> > > nhi_pci_dev/domain0/switch0/port1/switch1/port3/switch2
> > > 
> > > I think with the controllers that shipped, there can be up to two
> > > child switches below a switch.  If ports are not modeled as devices,
> > > you can't tell which port a switch is connected to.
> > 
> > Yes you can - it is part of the route string that we use here as bus
> > address. Each "hop" there is the port number through the switch is
> > accessed. For example I have here 4 devices connected:
> > 
> > $ ls -1 /sys/bus/thunderbolt/devices/
> > 0-0
> > 0-1
> > 0-1040301
> > 0-301
> > 0-40301
> > domain0
> 
> I see.
> 
> Is it possible to determine for a given port of type PCIe to which
> downstream bridge it belongs?
> 
> E.g. on my Light Ridge, the PCIe ports are numbered 7, 8, 9, 10.
> The downstream bridges on the controller are numbered 03, 04, 05, 06.
> But the ordering seems to be mixed up, e.g. port 7 corresponds to
> downstream bridge 0000:06:04.0.  Not to 03, as one would expect.
>                           ^^
> 
> Can this perhaps be determined from config space?

I don't think it is in config space. I'm not sure if there is a way to
determine that actually. It may be part of the DROM entry for the PCIe
adapter.

> If we knew the correlation between Thunderbolt PCIe port and downstream
> bridge, we could provide a symlink in sysfs from the Thunderbolt bus
> to the PCI bus.  E.g. in the switch directory for a plugged in device,
> there would be a symlink to the corresponding upstream bridge.  For the
> root switch, this would be the upstream bridge of the host controller.

Yes.

> > > Also, if ports are modeled as devices we'd be able to store attributes
> > > such as port type in sysfs.  Of course we could also store those in
> > > each switch directory as "port0", "port1", ... files.
> > 
> > Is there any reason adding these? It will not help the user to identify
> > the connected device and I don't see how that information could be
> > useful.
> >
> > That kind of information could go to debugfs, though.
> 
> Well, currently we print that stuff to syslog and it's valuable to
> understand what's going on.  Being able to check e.g. the type of
> a port on a running system would be even better.  I'm not saying
> we should model ports as devices, I just want to have a discussion
> about the pros and cons.

Sure.

> > > On Macs, the UUID is conveyed in an EFI device property.
> > 
> > Yes, but only for the host and UUID is actually coming from the fuses
> > (hardware), not from EFI property.
> 
> Nope, on Macs the UUID is calculated by sha1-hashing a constant, then
> extending that by sha1-hashing the uid, then truncating the result to
> 16 bytes.
> 
> The uid in turn is calculated by combining a 16-bit constant with a
> 24-bit model-specific number and a 24-bit serial number.
> 
> This is done by the EFI NHI driver.  No fuses involved.
> 
> (Okay the serial number is coming from an EEPROM, but not that of
> the Thunderbolt controller).

As far as I can tell they read it from fuse but I haven't looked at the
Apple driver.

> > > > +
> > > > +	/*
> > > > +	 * By default the UUID will be based on UID where upper two
> > > > +	 * dwords are filled with ones.
> > > > +	 */
> > > > +	uuid[0] = sw->uid & 0xffffffff;
> > > > +	uuid[1] = (sw->uid >> 32) & 0xffffffff;
> > > > +	uuid[2] = 0xffffffff;
> > > > +	uuid[3] = 0xffffffff;
> > > > +
> > > > +	/*
> > > > +	 * The newer controllers include fused UUID as part of link
> > > > +	 * controller specific registers
> > > > +	 */
> > > > +	cap = tb_switch_find_vsec_cap(sw, TB_VSEC_CAP_LINK_CONTROLLER);
> > > > +	if (cap > 0)
> > > > +		tb_sw_read(sw, uuid, TB_CFG_SWITCH, cap + 3, 4);
> > > > +
> > > > +	sw->uuid = kmemdup(uuid, sizeof(uuid), GFP_KERNEL);
> > > 
> > > Hm, so on newer controller the uuid is calculated and the result is
> > > subsequently overwritten?  Meh... :-/
> > 
> > ICM gives the UUID as part of the device connected message. This here to
> > make the UUID working also on systems withouth ICM (older Macs). The
> > special case comes from the older devices where there is no fused UUID
> > so we generate one based on UID instead following what ICM does.
> > 
> > This allows us to show "unique_id" attribute the same way on all
> > systems.
> 
> I'll have to double check if the macOS NHI driver calculates a UUID for
> each switch, and how it does that.  We should try to be compatible.

OS X uses UID instead to show the unique id. Here we should really use
UUID so that we are compatible with Windows as well. Since that UUID is
derived from UID for older switches, we should show pretty much the same
information as on Macs.

> > > How about putting the VSEC code path first, then fall back to calculating
> > > the default UUID?  Apart from being prettier, this would allow me to
> > > easily add the Mac-specific code path.
> > 
> > I can change the ordering but you don't need to add Mac specific path
> > there - this code has been tested on Macs already and it should work
> > exactly the same there.
> 
> I don't doubt that it works, the problem is that the UUID should be
> identical to what macOS uses.

I agree and I think I have checked it on OS X as well but it would be
helpful if you could do the same :)

> > Below is taken from the same devices connected to a Cactus Ridge based
> > Mac.
> > 
> > 0-0/authorized:1
> > 0-0/device:0xa
> > 0-0/device_name:Macintosh
> > 0-0/uevent:DEVTYPE=thunderbolt_device
> > 0-0/unique_id:00000000-0000-0008-83b3-30099bc1194a
> > 0-0/vendor:0x1
> > 0-0/vendor_name:Apple, Inc.
> 
> Interesting, however this means that the device ID is identical across
> all Macs.  (0xa, same as on my MacBookPro9,1 w/ Light Ridge.)

That is fine because we are not authorizing root switches anyway. Only
the devices (switches) that get connected to the root switch.

> > > IIUC, if the switch is the root switch, then the UUID is used to
> > > uniquely identify the domain starting at that switch, is that correct?
> > 
> > Yes, that's correct.
> 
> Why is a UUID calculated for each switch on the chain even if only the
> UUID on the root switch is needed to give the domain a unique ID?

It is different thing. Domain UUID is used to identify another domain if
you have thunderbolt cable connected between two hosts. We will be using
that when XDomain stuff is added.

UUID of non-root switches is used to uniquely identify that particular
device so that userspace knows it is the same user has already
authorized. This way we can automatically authorized the device without
need to ask from user.

Re: [PATCH 14/24] thunderbolt: Let the connection manager handle all notifications

[Mika Westerberg]

On Wed, May 24, 2017 at 04:00:49PM +0200, Lukas Wunner wrote:
> On Thu, May 18, 2017 at 05:39:04PM +0300, Mika Westerberg wrote:
> > @@ -320,18 +330,42 @@ static void tb_ctl_rx_callback(struct tb_ring *ring, struct ring_frame *frame,
> >  	}
> >  
> >  	frame->size -= 4; /* remove checksum */
> > -	if (*(__be32 *) (pkg->buffer + frame->size)
> > -			!= tb_crc(pkg->buffer, frame->size)) {
> > -		tb_ctl_err(pkg->ctl,
> > -			   "RX: checksum mismatch, dropping packet\n");
> > -		goto rx;
> > -	}
> > +	crc32 = tb_crc(pkg->buffer, frame->size);
> >  	be32_to_cpu_array(pkg->buffer, pkg->buffer, frame->size / 4);
> >  
> > -	if (frame->eof == TB_CFG_PKG_EVENT) {
> > -		tb_ctl_handle_plug_event(pkg->ctl, pkg);
> > +	switch (frame->eof) {
> > +	case TB_CFG_PKG_READ:
> > +	case TB_CFG_PKG_WRITE:
> > +	case TB_CFG_PKG_ERROR:
> > +	case TB_CFG_PKG_OVERRIDE:
> > +	case TB_CFG_PKG_RESET:
> > +		if (*(__be32 *)(pkg->buffer + frame->size) != crc32) {
> > +			tb_ctl_err(pkg->ctl,
> > +				   "RX: checksum mismatch, dropping packet\n");
> > +			goto rx;
> > +		}
> 
> Any harm keeping the crc32 check above the switch/case statement?
> (And thus also execute it for unknown packets?)

Not all packets carry crc32, only these configuration space packets.

> > +	case TB_CFG_PKG_EVENT:
> > +		if (*(__be32 *)(pkg->buffer + frame->size) != crc32) {
> > +			tb_ctl_err(pkg->ctl,
> > +				   "RX: checksum mismatch, dropping packet\n");
> > +			goto rx;
> > +		}
> > +		tb_ctl_handle_event(pkg->ctl, frame->eof, pkg, frame->size);
> > +		goto rx;
> > +
> > +	default:
> > +		tb_ctl_dbg(pkg->ctl, "RX: unknown package %#x, dropping\n",
> 
> The packet / package terminology is a bit inconsistent here.
> Andreas originally used package.  What's the term used by the TB spec?

IIRC it uses packet.

Re: [PATCH 20/24] thunderbolt: Do not touch the hardware if the NHI is gone on resume

[Mika Westerberg]

On Wed, May 24, 2017 at 04:43:10PM +0200, Lukas Wunner wrote:
> On Thu, May 18, 2017 at 05:39:10PM +0300, Mika Westerberg wrote:
> > @@ -655,6 +659,16 @@ static int nhi_resume_noirq(struct device *dev)
> >  {
> >  	struct pci_dev *pdev = to_pci_dev(dev);
> >  	struct tb *tb = pci_get_drvdata(pdev);
> > +	u32 vid;
> > +
> > +	/*
> > +	 * Check that the device is still there. It may be that the user
> > +	 * unplugged last device which causes the host controller to go
> > +	 * away on PCs.
> > +	 */
> > +	pci_read_config_dword(pdev, PCI_VENDOR_ID, &vid);
> > +	if (vid == ~0)
> > +		tb->nhi->going_away = true;
> 
> if (!pci_device_is_present(pdev))

Sure.

Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[Mika Westerberg]

On Wed, May 24, 2017 at 07:06:33PM +0000, Mario.Limonciello@dell.com wrote:
> > -----Original Message-----
> > From: Mika Westerberg [mailto:mika.westerberg@linux.intel.com]
> > Sent: Wednesday, May 24, 2017 6:11 AM
> > To: Limonciello, Mario <Mario_Limonciello@Dell.com>
> > Cc: gregkh@linuxfoundation.org; andreas.noever@gmail.com;
> > michael.jamet@intel.com; yehezkel.bernat@intel.com; lukas@wunner.de;
> > amir.jer.levy@intel.com; luto@kernel.org; Dominguez, Jared
> > <Jared_Dominguez@DELL.com>; andriy.shevchenko@linux.intel.com; linux-
> > kernel@vger.kernel.org
> > Subject: Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade
> > 
> > On Tue, May 23, 2017 at 05:30:43PM +0000, Mario.Limonciello@dell.com wrote:
> > > (Sorry my email client is not going to wrap these at 80 columns)o
> > 
> > That's fine. It is more readable this way :)
> > 
> > > [    0.467319] pci 0000:00:1c.0: [8086:9d10] type 01 class 0x060400
> > > [    0.467389] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
> > > [    0.467513] pci 0000:00:1c.0: System wakeup disabled by ACPI
> > 
> > [...]
> > 
> > > [    0.469363] pci 0000:01:00.0: [8086:1576] type 01 class 0x060400
> > > [    0.469483] pci 0000:01:00.0: supports D1 D2
> > > [    0.469484] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> > > [    0.469570] pci 0000:01:00.0: System wakeup disabled by ACPI
> > > [    0.469609] pci 0000:00:1c.0: PCI bridge to [bus 01-39]
> > > [    0.469614] pci 0000:00:1c.0:   bridge window [mem 0xc4000000-0xda0fffff]
> > > [    0.469618] pci 0000:00:1c.0:   bridge window [mem 0xa0000000-0xc1ffffff
> > 64bit pref]
> > > [    0.469621] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]),
> > reconfiguring
> > 
> > This is the problem. Here the PCIe upstream port (0000:01:00.0) is
> > visible to Linux but it is not fully configured by the BIOS ->
> > (primary/secondary/subordinate) is set to 0.
> 
> So at least for me the other difference between a successful run (where you plug
> in after boot instead) is that it shows up as instead:
> PCI bridge to [bus 02-39]
> 
> Same bridge window though.
> 
> > 
> > At this point Linux decides to configure the port itself and goes wrong
> > since our allocation strategy tries to keep resource windows, including
> > reserved buses as small as possible so that everything we currently find
> > barely fits there.
> > 
> > This continues few lines below:
> > 
> > > [    0.469670] pci_bus 0000:02: busn_res: can not insert [bus 02-ff] under [bus 01-
> > 39] (conflicts with (null) [bus 01-39])
> > > [    0.469688] pci 0000:02:00.0: [8086:1576] type 01 class 0x060400
> > > [    0.469809] pci 0000:02:00.0: supports D1 D2
> > > [    0.469810] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> > > [    0.469877] pci 0000:02:01.0: [8086:1576] type 01 class 0x060400
> > > [    0.470000] pci 0000:02:01.0: supports D1 D2
> > > [    0.470001] pci 0000:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
> > > [    0.470067] pci 0000:02:02.0: [8086:1576] type 01 class 0x060400
> > > [    0.470188] pci 0000:02:02.0: supports D1 D2
> > > [    0.470189] pci 0000:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
> > > [    0.470277] pci 0000:01:00.0: PCI bridge to [bus 02-ff]
> > > [    0.470283] pci 0000:01:00.0:   bridge window [io  0x0000-0x0fff]
> > > [    0.470287] pci 0000:01:00.0:   bridge window [mem 0x00000000-0x000fffff]
> > > [    0.470294] pci 0000:01:00.0:   bridge window [mem 0x00000000-0x000fffff
> > 64bit pref]
> > > [    0.470296] pci 0000:02:00.0: bridge configuration invalid ([bus 00-00]),
> > reconfiguring
> > > [    0.470304] pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]),
> > reconfiguring
> > > [    0.470312] pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]),
> > reconfiguring
> > 
> > Here.
> > 
> > And ends up in failure when we create PCIe tunnels later on.
> 
> For what it's worth the XPS 9365 which has a different BIOS core has these
> exact same behaviors on Linux if booted with the TBT dock plugged in.
> 
> > 
> > Now, this is probably where Windows does something else, like it may
> > skip re-configuring phase which could explain why it works. However, to
> > me this looks pretty much like a bug in the BIOS/firmware as we are
> > expecting the BIOS to configure the PCIe devices properly before the OS
> > is send ACPI hotplug event.
> > 
> 
> I'll reach out to the BIOS guys to see if they can give some more comments
> from their perspective.
> 
> I came across something interesting from browsing MSDN about this topic.
> It hasn't been updated in a long time but I think should still be a relevant 
> indication of the approach that Windows was taking and why the firmware 
> is this way and expecting OS to reconfigure.
> 
> "The BIOS cannot preconfigure PCI-to-PCI (P2P) bridges on adapters during 
> hot plug. Consequently, the operating system assigns resource windows of 
> a default size to a bridge.
> 
> I/O window. The default size for the I/O window is 4 KB in Windows 2000, 
> Windows XP, and Windows Server 2003.  
> Memory window. The configuration for the memory window differs for 
> Windows 2000, Windows XP, and Windows Server 2003:
> *	For Windows 2000, the default size for the memory window is 2 MB.
> *	For Windows XP and Windows Server 2003, the operating system 
> first attempts to find a memory window of 32 MB. If it cannot find a 
> window of that size, the operating system attempts to find a memory 
> window of progressively smaller sizes (16, 8, 4, 2, and finally 1 MB) until
>  it finds a size that works."

I think the way current BIOS does it, is that it actually configures the
hotplugged bridge and assigns resources accordingly. Once that is done
it trigggers hotplug to the OS using ACPI event.

> > We need to handle this in Linux in the same way Windows does but
> > currently I have no idea. It is however, more related to our PCI
> > enumeration code than the patches in question, I think.
> > 
> 
> Come to think of it, I have seen the dock have troubles if plugged in at 
> boot on Linux even with SL0 before this patch series.
> 
> > I also have a Dell 9350 here so I can reproduce the problem and I'm
> > going to investigate this further probably involving Linux PCI people.
> To clarify are you reproducing it with a TB16 or some other TBT device?

I'm using a chain of 1 to 5 devices. I don't have TB16 here but I don't
think it matters here.

> > My testing on the machine shows this behaviour only when the cable is
> > connected during boot.
> 
> Yep same.
> 
> > 
> > If I connect the cable after OS is booted I don't see the problem, even
> > if I do unplug / plug cycle.
> > 
> > Can you try that also (again)? And if you see the problem, send me the
> > dmesg? I have the latest BIOS (1.4.17) and NVM 16 so this machine
> > configuration should match yours if I'm not mistaken.
> 
> It does work properly if I boot no cable plugged in and then plug one in.	

OK, so we see the same behavior.

To summarize: This happens only on boot when Thunderbolt device is
already connected.

Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[mika.westerberg]

On Wed, May 24, 2017 at 07:32:45PM +0000, Jamet, Michael wrote:
> I talked to our BIOS expert today. Here is his advice to debugging further:
> 
> It looks like something may have been wrong from system (BIOS, FW, others...) perspective.
> On reboot need to enter EFI shell and check resources of 
> pci 0000:01:00.0: bridge.
> At the EFI shell, this bridge MUST be either configured or absent.
> 
> I would start this way, once we have this info, we may circle back to
> him and look into next debugging step.

Thanks, I'll try this today.

Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[mika.westerberg]

On Thu, May 25, 2017 at 10:20:10AM +0300, mika.westerberg@linux.intel.com wrote:
> On Wed, May 24, 2017 at 07:32:45PM +0000, Jamet, Michael wrote:
> > I talked to our BIOS expert today. Here is his advice to debugging further:
> > 
> > It looks like something may have been wrong from system (BIOS, FW, others...) perspective.
> > On reboot need to enter EFI shell and check resources of 
> > pci 0000:01:00.0: bridge.
> > At the EFI shell, this bridge MUST be either configured or absent.
> > 
> > I would start this way, once we have this info, we may circle back to
> > him and look into next debugging step.
> 
> Thanks, I'll try this today.


This is the contents dumped directly from EFI shell when a device is
connected. It seems that the vendor_id/device_id is 0xffff but the rest
of the config seems to be present (although not fully configured):

  PCI Segment 00 Bus 01 Device 00 Func 00 [EFI 0001000000]
  00000000: FF FF FF FF 00 00 10 00-00 00 04 06 00 00 01 00  *................*
  00000010: 00 00 00 00 00 00 00 00-00 00 00 00 01 01 00 00  *................*
  00000020: 00 00 00 00 01 00 01 00-00 00 00 00 00 00 00 00  *................*
  00000030: 00 00 00 00 80 00 00 00-00 00 00 00 FF 01 00 00  *................*
  00000040: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
  00000050: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
  00000060: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
  00000070: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
  00000080: 01 88 C3 FF 08 00 00 00-05 AC 80 00 00 00 00 00  *................*
  00000090: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
  000000A0: 00 00 00 00 00 00 00 00-00 00 00 00 0D C0 00 00  *................*
  000000B0: 22 22 11 11 00 00 00 00-00 00 00 00 00 00 00 00  *""..............*
  000000C0: 10 00 52 00 20 80 E8 07-10 28 10 00 43 5C 45 00  *..R. ....(..C\E.*
  000000D0: 00 00 23 10 00 00 00 00-00 00 00 00 00 00 00 00  *..#.............*
  000000E0: 00 00 00 00 00 08 00 00-00 00 00 00 0E 00 00 00  *................*
  000000F0: 03 00 1E 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*

I wonder how Linux manages to find the device if vendor_id/device_id
reads 0xffff?

Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[mika.westerberg]

On Thu, May 25, 2017 at 11:04:08AM +0300, mika.westerberg@linux.intel.com wrote:
> On Thu, May 25, 2017 at 10:20:10AM +0300, mika.westerberg@linux.intel.com wrote:
> > On Wed, May 24, 2017 at 07:32:45PM +0000, Jamet, Michael wrote:
> > > I talked to our BIOS expert today. Here is his advice to debugging further:
> > > 
> > > It looks like something may have been wrong from system (BIOS, FW, others...) perspective.
> > > On reboot need to enter EFI shell and check resources of 
> > > pci 0000:01:00.0: bridge.
> > > At the EFI shell, this bridge MUST be either configured or absent.
> > > 
> > > I would start this way, once we have this info, we may circle back to
> > > him and look into next debugging step.
> > 
> > Thanks, I'll try this today.
> 
> 
> This is the contents dumped directly from EFI shell when a device is
> connected. It seems that the vendor_id/device_id is 0xffff but the rest
> of the config seems to be present (although not fully configured):
> 
>   PCI Segment 00 Bus 01 Device 00 Func 00 [EFI 0001000000]
>   00000000: FF FF FF FF 00 00 10 00-00 00 04 06 00 00 01 00  *................*
>   00000010: 00 00 00 00 00 00 00 00-00 00 00 00 01 01 00 00  *................*
>   00000020: 00 00 00 00 01 00 01 00-00 00 00 00 00 00 00 00  *................*
>   00000030: 00 00 00 00 80 00 00 00-00 00 00 00 FF 01 00 00  *................*
>   00000040: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
>   00000050: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
>   00000060: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
>   00000070: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
>   00000080: 01 88 C3 FF 08 00 00 00-05 AC 80 00 00 00 00 00  *................*
>   00000090: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
>   000000A0: 00 00 00 00 00 00 00 00-00 00 00 00 0D C0 00 00  *................*
>   000000B0: 22 22 11 11 00 00 00 00-00 00 00 00 00 00 00 00  *""..............*
>   000000C0: 10 00 52 00 20 80 E8 07-10 28 10 00 43 5C 45 00  *..R. ....(..C\E.*
>   000000D0: 00 00 23 10 00 00 00 00-00 00 00 00 00 00 00 00  *..#.............*
>   000000E0: 00 00 00 00 00 08 00 00-00 00 00 00 0E 00 00 00  *................*
>   000000F0: 03 00 1E 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
> 
> I wonder how Linux manages to find the device if vendor_id/device_id
> reads 0xffff?

OK, here's the explanation.

When Linux initializes ACPI (this happens before PCI initial scan), it
calls acpi_initialize_objects(). This in turn causes _INI methods of
devices to be executed. Now, the _SB.PCI0._INI() ends up calling
\_GPE.TINI() which executes Thunderbolt specific OSUP() method. Purpose
of this method is to overwrite vendor_id/device_id to the correct values
with the assumption that the OS has already done the initial PCI scan.

In case of Linux this is not true and that is the reason the upstream
port is found half-initialized leading to the failure.

Re: [PATCH 01/24] thunderbolt: Use const buffer pointer in write operations

[Greg Kroah-Hartman]

On Thu, May 18, 2017 at 05:38:51PM +0300, Mika Westerberg wrote:
> These functions should not (and do not) modify the argument in any way
> so make it const.
> 
> Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
> Reviewed-by: Michael Jamet <michael.jamet@intel.com>

Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

Re: [PATCH 06/24] thunderbolt: Introduce thunderbolt bus and connection manager

[Greg Kroah-Hartman]

On Thu, May 18, 2017 at 05:38:56PM +0300, Mika Westerberg wrote:
> Thunderbolt fabric consists of one or more switches. This fabric is
> called domain and it is controlled by an entity called connection
> manager. The connection manager can be either internal (driven by a
> firmware running on the host controller) or external (software driver).
> This driver currently implements support for the latter.
> 
> In order to manage switches and their properties more easily we model
> this domain structure as a Linux bus. Each host controller adds a domain
> device to this bus, and these devices are named as domainN where N
> stands for index or id of the current domain.
> 
> We then abstract connection manager specific operations into a new
> structure tb_cm_ops and convert the existing tb.c to fill those
> accordingly. This makes it easier to add support for the internal
> connection manager in subsequent patches.
> 
> Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
> Reviewed-by: Michael Jamet <michael.jamet@intel.com>
> ---
>  drivers/thunderbolt/Makefile     |   2 +-
>  drivers/thunderbolt/domain.c     | 230 +++++++++++++++++++++++++++++++++++++++
>  drivers/thunderbolt/nhi.c        |  31 ++++--
>  drivers/thunderbolt/tb.c         | 156 ++++++++++++--------------
>  drivers/thunderbolt/tb.h         |  70 +++++++++---
>  drivers/thunderbolt/tunnel_pci.c |   9 +-
>  6 files changed, 377 insertions(+), 121 deletions(-)
>  create mode 100644 drivers/thunderbolt/domain.c
> 
> diff --git a/drivers/thunderbolt/Makefile b/drivers/thunderbolt/Makefile
> index 5d1053cdfa54..e276a9a62261 100644
> --- a/drivers/thunderbolt/Makefile
> +++ b/drivers/thunderbolt/Makefile
> @@ -1,3 +1,3 @@
>  obj-${CONFIG_THUNDERBOLT} := thunderbolt.o
>  thunderbolt-objs := nhi.o ctl.o tb.o switch.o cap.o path.o tunnel_pci.o eeprom.o
> -
> +thunderbolt-objs += domain.o
> diff --git a/drivers/thunderbolt/domain.c b/drivers/thunderbolt/domain.c
> new file mode 100644
> index 000000000000..b5bfca95415d
> --- /dev/null
> +++ b/drivers/thunderbolt/domain.c
> @@ -0,0 +1,230 @@
> +/*
> + * Thunderbolt bus support
> + *
> + * Copyright (C) 2017, Intel Corporation
> + * Author:  Mika Westerberg <mika.westerberg@linux.intel.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + */
> +
> +#include <linux/device.h>
> +#include <linux/idr.h>
> +#include <linux/module.h>
> +#include <linux/slab.h>
> +
> +#include "tb.h"
> +
> +static DEFINE_IDA(tb_domain_ida);

You forgot to clean this up in your module exit code with a call to
ida_destroy(&tb_domain_ida);

Yeah, it's not obvious, I've made the same mistake before :)

Other than that minor thing, the bus logic looks good, nice job.

thanks,

greg k-h

Re: [PATCH 15/24] thunderbolt: Rework control channel to be more reliable

[Greg Kroah-Hartman]

On Thu, May 18, 2017 at 05:39:05PM +0300, Mika Westerberg wrote:
> If a request times out the response might arrive right after the request
> is failed. This response is pushed to the kfifo and next request will
> read it instead. Since it most likely will not pass our validation
> checks in parse_header() the next request will fail as well, and
> response to that request will be pushed to the kfifo, ad infinitum.
> 
> We end up in a situation where all requests fail and no devices can be
> added anymore until the driver is unloaded and reloaded again.
> 
> To overcome this, rework the control channel so that we will have a
> queue of outstanding requests. Each request will be handled in turn and
> the response is validated against what is expected. Unexpected packets
> (for example responses for requests that have been timed out) are
> dropped. This model is copied from Greybus implementation with small
> changes here and there to get it cope with Thunderbolt control packets.
> 
> In addition the configuration packets support sequence number which the
> switch is supposed to copy from the request to response. We use this to
> drop responses that are already timed out. Taking advantage of the
> sequence number, we automatically retry configuration read/write 4 times
> before giving up.
> 
> Also timeout is not a programming error so there is no need to trigger a
> scary backtrace (WARN), instead we just log a warning.  After all
> Thunderbolt devices are hot-pluggable by definition which means user can
> unplug a device any time and that is totally acceptable.
> 
> With this change there is no need to take the global domain lock when
> sending configuration packets anymore. This is useful when we add
> support for cross-domain (XDomain) communication later on.
> 
> Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
> Reviewed-by: Michael Jamet <michael.jamet@intel.com>
> ---
>  drivers/thunderbolt/ctl.c | 471 +++++++++++++++++++++++++++++++++++++++-------
>  drivers/thunderbolt/ctl.h |  65 +++++++
>  drivers/thunderbolt/tb.h  |   2 +-
>  3 files changed, 467 insertions(+), 71 deletions(-)
> 
> diff --git a/drivers/thunderbolt/ctl.c b/drivers/thunderbolt/ctl.c
> index b3ee755fdb37..2b1255cbf3c9 100644
> --- a/drivers/thunderbolt/ctl.c
> +++ b/drivers/thunderbolt/ctl.c
> @@ -5,22 +5,17 @@
>   */
>  
>  #include <linux/crc32.h>
> +#include <linux/delay.h>
>  #include <linux/slab.h>
>  #include <linux/pci.h>
>  #include <linux/dmapool.h>
>  #include <linux/workqueue.h>
> -#include <linux/kfifo.h>
>  
>  #include "ctl.h"
>  
>  
> -struct ctl_pkg {
> -	struct tb_ctl *ctl;
> -	void *buffer;
> -	struct ring_frame frame;
> -};
> -
> -#define TB_CTL_RX_PKG_COUNT 10
> +#define TB_CTL_RX_PKG_COUNT	10
> +#define TB_CTL_RETRIES		4
>  
>  /**
>   * struct tb_cfg - thunderbolt control channel
> @@ -32,8 +27,9 @@ struct tb_ctl {
>  
>  	struct dma_pool *frame_pool;
>  	struct ctl_pkg *rx_packets[TB_CTL_RX_PKG_COUNT];
> -	DECLARE_KFIFO(response_fifo, struct ctl_pkg*, 16);
> -	struct completion response_ready;
> +	struct mutex request_lock;
> +	struct list_head request_queue;
> +	bool running;
>  
>  	event_cb callback;
>  	void *callback_data;
> @@ -55,10 +51,115 @@ struct tb_ctl {
>  #define tb_ctl_dbg(ctl, format, arg...) \
>  	dev_dbg(&(ctl)->nhi->pdev->dev, format, ## arg)
>  
> +static DECLARE_WAIT_QUEUE_HEAD(tb_cfg_request_cancel_queue);
> +
> +/**
> + * tb_cfg_request_alloc() - Allocates a new config request
> + *
> + * This is refcounted object so when you are done with this, call
> + * tb_cfg_request_put() to it.
> + */
> +struct tb_cfg_request *tb_cfg_request_alloc(void)
> +{
> +	struct tb_cfg_request *req;
> +
> +	req = kzalloc(sizeof(*req), GFP_KERNEL);
> +	if (!req)
> +		return NULL;
> +
> +	kref_init(&req->kref);
> +
> +	return req;
> +}
> +
> +/**
> + * tb_cfg_request_get() - Increase refcount of a request
> + * @req: Request whose refcount is increased
> + */
> +void tb_cfg_request_get(struct tb_cfg_request *req)
> +{
> +	kref_get(&req->kref);
> +}
> +
> +static void tb_cfg_request_destroy(struct kref *kref)
> +{
> +	struct tb_cfg_request *req = container_of(kref, typeof(*req), kref);
> +
> +	kfree(req);
> +}
> +
> +/**
> + * tb_cfg_request_put() - Decrease refcount and possibly release the request
> + * @req: Request whose refcount is decreased
> + *
> + * Call this function when you are done with the request. When refcount
> + * goes to %0 the object is released.
> + */
> +void tb_cfg_request_put(struct tb_cfg_request *req)
> +{
> +	kref_put(&req->kref, tb_cfg_request_destroy);
> +}

What prevents this call from being called twice on the same object from
different threads at the same time?  You still need a lock somewhere to
protect yourself from that, am I just missing where that lock is?

thanks,

greg k-h

Re: [PATCH 22/24] thunderbolt: Add support for host and device NVM firmware upgrade

[Greg Kroah-Hartman]

On Thu, May 18, 2017 at 05:39:12PM +0300, Mika Westerberg wrote:
> +static int tb_switch_nvm_add(struct tb_switch *sw)
> +{
> +	struct nvmem_device *nvm_dev;
> +	struct tb_switch_nvm *nvm;
> +	u32 val;
> +	int ret;
> +
> +	if (!sw->dma_port)
> +		return 0;
> +
> +	nvm = kzalloc(sizeof(*nvm), GFP_KERNEL);
> +	if (!nvm)
> +		return -ENOMEM;
> +
> +	nvm->id = ida_simple_get(&nvm_ida, 0, 0, GFP_KERNEL);
> +
> +	/*
> +	 * If the switch is in safe-mode the only accessible portion of
> +	 * the NVM is the non-active one where userspace is expected to
> +	 * write new functional NVM.
> +	 */
> +	if (!sw->safe_mode) {
> +		u32 nvm_size, hdr_size;
> +
> +		ret = dma_port_flash_read(sw->dma_port, NVM_FLASH_SIZE, &val,
> +					  sizeof(val));
> +		if (ret)
> +			goto err_ida;
> +
> +		hdr_size = sw->generation < 3 ? SZ_8K : SZ_16K;
> +		nvm_size = (SZ_1M << (val & 7)) / 8;
> +		nvm_size = (nvm_size - hdr_size) / 2;
> +
> +		ret = dma_port_flash_read(sw->dma_port, NVM_VERSION, &val,
> +					  sizeof(val));
> +		if (ret)
> +			goto err_ida;
> +
> +		nvm->major = val >> 16 & 0xff;
> +		nvm->minor = val >> 8 & 0xff;
> +
> +		nvm_dev = register_nvmem(sw, nvm->id, nvm_size, true);
> +		if (IS_ERR(nvm_dev)) {
> +			ret = PTR_ERR(nvm_dev);
> +			goto err_ida;
> +		}
> +		nvm->active = nvm_dev;
> +	}
> +
> +	nvm_dev = register_nvmem(sw, nvm->id, NVM_MAX_SIZE, false);
> +	if (IS_ERR(nvm_dev)) {
> +		ret = PTR_ERR(nvm_dev);
> +		goto err_nvm_active;
> +	}
> +	nvm->non_active = nvm_dev;
> +
> +	sw->nvm = nvm;
> +
> +	ret = sysfs_create_group(&sw->dev.kobj, &nvm_group);

Why are you adding this to the sw device?  And doing so _after_ it was
announced to userspace?  Why can't you make it part of the device's
default groups so that the driver core can handle it properly?

Hint, if you ever have to call sysfs_* from within a driver, something
might really be wrong :)

thanks,

greg k-h

Re: [PATCH 15/24] thunderbolt: Rework control channel to be more reliable

[Mika Westerberg]

On Thu, May 25, 2017 at 03:25:46PM +0200, Greg Kroah-Hartman wrote:
> > +/**
> > + * tb_cfg_request_put() - Decrease refcount and possibly release the request
> > + * @req: Request whose refcount is decreased
> > + *
> > + * Call this function when you are done with the request. When refcount
> > + * goes to %0 the object is released.
> > + */
> > +void tb_cfg_request_put(struct tb_cfg_request *req)
> > +{
> > +	kref_put(&req->kref, tb_cfg_request_destroy);
> > +}
> 
> What prevents this call from being called twice on the same object from
> different threads at the same time?  You still need a lock somewhere to
> protect yourself from that, am I just missing where that lock is?

No, you are right - it is missing a lock. I will add it there in the
next version.

Thanks.

Re: [PATCH 22/24] thunderbolt: Add support for host and device NVM firmware upgrade

[Mika Westerberg]

On Thu, May 25, 2017 at 03:28:34PM +0200, Greg Kroah-Hartman wrote:
> On Thu, May 18, 2017 at 05:39:12PM +0300, Mika Westerberg wrote:
> > +static int tb_switch_nvm_add(struct tb_switch *sw)
> > +{
> > +	struct nvmem_device *nvm_dev;
> > +	struct tb_switch_nvm *nvm;
> > +	u32 val;
> > +	int ret;
> > +
> > +	if (!sw->dma_port)
> > +		return 0;
> > +
> > +	nvm = kzalloc(sizeof(*nvm), GFP_KERNEL);
> > +	if (!nvm)
> > +		return -ENOMEM;
> > +
> > +	nvm->id = ida_simple_get(&nvm_ida, 0, 0, GFP_KERNEL);
> > +
> > +	/*
> > +	 * If the switch is in safe-mode the only accessible portion of
> > +	 * the NVM is the non-active one where userspace is expected to
> > +	 * write new functional NVM.
> > +	 */
> > +	if (!sw->safe_mode) {
> > +		u32 nvm_size, hdr_size;
> > +
> > +		ret = dma_port_flash_read(sw->dma_port, NVM_FLASH_SIZE, &val,
> > +					  sizeof(val));
> > +		if (ret)
> > +			goto err_ida;
> > +
> > +		hdr_size = sw->generation < 3 ? SZ_8K : SZ_16K;
> > +		nvm_size = (SZ_1M << (val & 7)) / 8;
> > +		nvm_size = (nvm_size - hdr_size) / 2;
> > +
> > +		ret = dma_port_flash_read(sw->dma_port, NVM_VERSION, &val,
> > +					  sizeof(val));
> > +		if (ret)
> > +			goto err_ida;
> > +
> > +		nvm->major = val >> 16 & 0xff;
> > +		nvm->minor = val >> 8 & 0xff;
> > +
> > +		nvm_dev = register_nvmem(sw, nvm->id, nvm_size, true);
> > +		if (IS_ERR(nvm_dev)) {
> > +			ret = PTR_ERR(nvm_dev);
> > +			goto err_ida;
> > +		}
> > +		nvm->active = nvm_dev;
> > +	}
> > +
> > +	nvm_dev = register_nvmem(sw, nvm->id, NVM_MAX_SIZE, false);
> > +	if (IS_ERR(nvm_dev)) {
> > +		ret = PTR_ERR(nvm_dev);
> > +		goto err_nvm_active;
> > +	}
> > +	nvm->non_active = nvm_dev;
> > +
> > +	sw->nvm = nvm;
> > +
> > +	ret = sysfs_create_group(&sw->dev.kobj, &nvm_group);
> 
> Why are you adding this to the sw device?  And doing so _after_ it was
> announced to userspace?  Why can't you make it part of the device's
> default groups so that the driver core can handle it properly?

I was thinking those attributes should show up only when we have
successfully created the two NVMem devices. But maybe I can add those
conditionally to the device default groups and make the attributes
return error if the NVM device creation fails.

> Hint, if you ever have to call sysfs_* from within a driver, something
> might really be wrong :)

Understood, thanks.

Re: [PATCH 06/24] thunderbolt: Introduce thunderbolt bus and connection manager

[Mika Westerberg]

On Thu, May 25, 2017 at 03:23:39PM +0200, Greg Kroah-Hartman wrote:
> On Thu, May 18, 2017 at 05:38:56PM +0300, Mika Westerberg wrote:
> > Thunderbolt fabric consists of one or more switches. This fabric is
> > called domain and it is controlled by an entity called connection
> > manager. The connection manager can be either internal (driven by a
> > firmware running on the host controller) or external (software driver).
> > This driver currently implements support for the latter.
> > 
> > In order to manage switches and their properties more easily we model
> > this domain structure as a Linux bus. Each host controller adds a domain
> > device to this bus, and these devices are named as domainN where N
> > stands for index or id of the current domain.
> > 
> > We then abstract connection manager specific operations into a new
> > structure tb_cm_ops and convert the existing tb.c to fill those
> > accordingly. This makes it easier to add support for the internal
> > connection manager in subsequent patches.
> > 
> > Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> > Reviewed-by: Yehezkel Bernat <yehezkel.bernat@intel.com>
> > Reviewed-by: Michael Jamet <michael.jamet@intel.com>
> > ---
> >  drivers/thunderbolt/Makefile     |   2 +-
> >  drivers/thunderbolt/domain.c     | 230 +++++++++++++++++++++++++++++++++++++++
> >  drivers/thunderbolt/nhi.c        |  31 ++++--
> >  drivers/thunderbolt/tb.c         | 156 ++++++++++++--------------
> >  drivers/thunderbolt/tb.h         |  70 +++++++++---
> >  drivers/thunderbolt/tunnel_pci.c |   9 +-
> >  6 files changed, 377 insertions(+), 121 deletions(-)
> >  create mode 100644 drivers/thunderbolt/domain.c
> > 
> > diff --git a/drivers/thunderbolt/Makefile b/drivers/thunderbolt/Makefile
> > index 5d1053cdfa54..e276a9a62261 100644
> > --- a/drivers/thunderbolt/Makefile
> > +++ b/drivers/thunderbolt/Makefile
> > @@ -1,3 +1,3 @@
> >  obj-${CONFIG_THUNDERBOLT} := thunderbolt.o
> >  thunderbolt-objs := nhi.o ctl.o tb.o switch.o cap.o path.o tunnel_pci.o eeprom.o
> > -
> > +thunderbolt-objs += domain.o
> > diff --git a/drivers/thunderbolt/domain.c b/drivers/thunderbolt/domain.c
> > new file mode 100644
> > index 000000000000..b5bfca95415d
> > --- /dev/null
> > +++ b/drivers/thunderbolt/domain.c
> > @@ -0,0 +1,230 @@
> > +/*
> > + * Thunderbolt bus support
> > + *
> > + * Copyright (C) 2017, Intel Corporation
> > + * Author:  Mika Westerberg <mika.westerberg@linux.intel.com>
> > + *
> > + * This program is free software; you can redistribute it and/or modify
> > + * it under the terms of the GNU General Public License version 2 as
> > + * published by the Free Software Foundation.
> > + */
> > +
> > +#include <linux/device.h>
> > +#include <linux/idr.h>
> > +#include <linux/module.h>
> > +#include <linux/slab.h>
> > +
> > +#include "tb.h"
> > +
> > +static DEFINE_IDA(tb_domain_ida);
> 
> You forgot to clean this up in your module exit code with a call to
> ida_destroy(&tb_domain_ida);

Ah, I see. OK, I'll add that missing call to tb_domain_exit().

> Yeah, it's not obvious, I've made the same mistake before :)
> 
> Other than that minor thing, the bus logic looks good, nice job.

Thanks :-)

Re: [PATCH 22/24] thunderbolt: Add support for host and device NVM firmware upgrade

[Greg Kroah-Hartman]

On Thu, May 25, 2017 at 05:39:40PM +0300, Mika Westerberg wrote:
> On Thu, May 25, 2017 at 03:28:34PM +0200, Greg Kroah-Hartman wrote:
> > On Thu, May 18, 2017 at 05:39:12PM +0300, Mika Westerberg wrote:
> > > +static int tb_switch_nvm_add(struct tb_switch *sw)
> > > +{
> > > +	struct nvmem_device *nvm_dev;
> > > +	struct tb_switch_nvm *nvm;
> > > +	u32 val;
> > > +	int ret;
> > > +
> > > +	if (!sw->dma_port)
> > > +		return 0;
> > > +
> > > +	nvm = kzalloc(sizeof(*nvm), GFP_KERNEL);
> > > +	if (!nvm)
> > > +		return -ENOMEM;
> > > +
> > > +	nvm->id = ida_simple_get(&nvm_ida, 0, 0, GFP_KERNEL);
> > > +
> > > +	/*
> > > +	 * If the switch is in safe-mode the only accessible portion of
> > > +	 * the NVM is the non-active one where userspace is expected to
> > > +	 * write new functional NVM.
> > > +	 */
> > > +	if (!sw->safe_mode) {
> > > +		u32 nvm_size, hdr_size;
> > > +
> > > +		ret = dma_port_flash_read(sw->dma_port, NVM_FLASH_SIZE, &val,
> > > +					  sizeof(val));
> > > +		if (ret)
> > > +			goto err_ida;
> > > +
> > > +		hdr_size = sw->generation < 3 ? SZ_8K : SZ_16K;
> > > +		nvm_size = (SZ_1M << (val & 7)) / 8;
> > > +		nvm_size = (nvm_size - hdr_size) / 2;
> > > +
> > > +		ret = dma_port_flash_read(sw->dma_port, NVM_VERSION, &val,
> > > +					  sizeof(val));
> > > +		if (ret)
> > > +			goto err_ida;
> > > +
> > > +		nvm->major = val >> 16 & 0xff;
> > > +		nvm->minor = val >> 8 & 0xff;
> > > +
> > > +		nvm_dev = register_nvmem(sw, nvm->id, nvm_size, true);
> > > +		if (IS_ERR(nvm_dev)) {
> > > +			ret = PTR_ERR(nvm_dev);
> > > +			goto err_ida;
> > > +		}
> > > +		nvm->active = nvm_dev;
> > > +	}
> > > +
> > > +	nvm_dev = register_nvmem(sw, nvm->id, NVM_MAX_SIZE, false);
> > > +	if (IS_ERR(nvm_dev)) {
> > > +		ret = PTR_ERR(nvm_dev);
> > > +		goto err_nvm_active;
> > > +	}
> > > +	nvm->non_active = nvm_dev;
> > > +
> > > +	sw->nvm = nvm;
> > > +
> > > +	ret = sysfs_create_group(&sw->dev.kobj, &nvm_group);
> > 
> > Why are you adding this to the sw device?  And doing so _after_ it was
> > announced to userspace?  Why can't you make it part of the device's
> > default groups so that the driver core can handle it properly?
> 
> I was thinking those attributes should show up only when we have
> successfully created the two NVMem devices. But maybe I can add those
> conditionally to the device default groups and make the attributes
> return error if the NVM device creation fails.

Or have the device files only show up if there is an nvm device (however
this is hooked up, I didn't spend the time to look at it deeply.)  The
is_visible() callback in the attribute is there just for that.

good luck!

greg k-h

Re: [PATCH 00/24] Thunderbolt security levels and NVM firmware upgrade

[mika.westerberg]

On Thu, May 25, 2017 at 03:03:07PM +0300, mika.westerberg@linux.intel.com wrote:
> On Thu, May 25, 2017 at 11:04:08AM +0300, mika.westerberg@linux.intel.com wrote:
> > On Thu, May 25, 2017 at 10:20:10AM +0300, mika.westerberg@linux.intel.com wrote:
> > > On Wed, May 24, 2017 at 07:32:45PM +0000, Jamet, Michael wrote:
> > > > I talked to our BIOS expert today. Here is his advice to debugging further:
> > > > 
> > > > It looks like something may have been wrong from system (BIOS, FW, others...) perspective.
> > > > On reboot need to enter EFI shell and check resources of 
> > > > pci 0000:01:00.0: bridge.
> > > > At the EFI shell, this bridge MUST be either configured or absent.
> > > > 
> > > > I would start this way, once we have this info, we may circle back to
> > > > him and look into next debugging step.
> > > 
> > > Thanks, I'll try this today.
> > 
> > 
> > This is the contents dumped directly from EFI shell when a device is
> > connected. It seems that the vendor_id/device_id is 0xffff but the rest
> > of the config seems to be present (although not fully configured):
> > 
> >   PCI Segment 00 Bus 01 Device 00 Func 00 [EFI 0001000000]
> >   00000000: FF FF FF FF 00 00 10 00-00 00 04 06 00 00 01 00  *................*
> >   00000010: 00 00 00 00 00 00 00 00-00 00 00 00 01 01 00 00  *................*
> >   00000020: 00 00 00 00 01 00 01 00-00 00 00 00 00 00 00 00  *................*
> >   00000030: 00 00 00 00 80 00 00 00-00 00 00 00 FF 01 00 00  *................*
> >   00000040: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
> >   00000050: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
> >   00000060: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
> >   00000070: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
> >   00000080: 01 88 C3 FF 08 00 00 00-05 AC 80 00 00 00 00 00  *................*
> >   00000090: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
> >   000000A0: 00 00 00 00 00 00 00 00-00 00 00 00 0D C0 00 00  *................*
> >   000000B0: 22 22 11 11 00 00 00 00-00 00 00 00 00 00 00 00  *""..............*
> >   000000C0: 10 00 52 00 20 80 E8 07-10 28 10 00 43 5C 45 00  *..R. ....(..C\E.*
> >   000000D0: 00 00 23 10 00 00 00 00-00 00 00 00 00 00 00 00  *..#.............*
> >   000000E0: 00 00 00 00 00 08 00 00-00 00 00 00 0E 00 00 00  *................*
> >   000000F0: 03 00 1E 00 00 00 00 00-00 00 00 00 00 00 00 00  *................*
> > 
> > I wonder how Linux manages to find the device if vendor_id/device_id
> > reads 0xffff?
> 
> OK, here's the explanation.
> 
> When Linux initializes ACPI (this happens before PCI initial scan), it
> calls acpi_initialize_objects(). This in turn causes _INI methods of
> devices to be executed. Now, the _SB.PCI0._INI() ends up calling
> \_GPE.TINI() which executes Thunderbolt specific OSUP() method. Purpose
> of this method is to overwrite vendor_id/device_id to the correct values
> with the assumption that the OS has already done the initial PCI scan.
> 
> In case of Linux this is not true and that is the reason the upstream
> port is found half-initialized leading to the failure.

We finally found out what the problem is. In short the above
_SB.PCI0._INI() (and OSUP()) gets called correctly but this is only
part of the story. When OSUP() is called it rewrites the
vendor/deviceid and then signals that certain GPE handler can continue
to trigger the SMI handler which should enumerate all the devices
before PCI scan happens.

In Linux we enable GPEs later, after PCI scan so we only see partially
configured PCI bridges (as the SMI handler has not run yet).

Rafael's patch series here:

https://lkml.org/lkml/2017/8/9/1017

addresses this so that we enable GPEs earlier among other things.