LKML Archive on lore.kernel.org
 help / color / Atom feed
* [Patch v2 00/16] Refine IR initialization flow and fixes bugs related to X2APIC
@ 2015-01-07  7:31 Jiang Liu
  2015-01-07  7:31 ` [Patch v2 01/16] iommu, x86: Restructure setup of the irq remapping feature Jiang Liu
                   ` (16 more replies)
  0 siblings, 17 replies; 56+ messages in thread
From: Jiang Liu @ 2015-01-07  7:31 UTC (permalink / raw)
  To: Thomas Gleixner, Joerg Roedel, Benjamin Herrenschmidt,
	Ingo Molnar, H. Peter Anvin, Yinghai Lu, Borislav Petkov
  Cc: Jiang Liu, Tony Luck, x86, linux-kernel, iommu

When converting x86 to new hierarchy irqdoamin framework, Thomas noticed
that the interrupt remapping initialization flow is a little complex and
has troubles in memory allocation. Then there is a joint force to
simplify IR initialization flow, please refer to related threads at:
https://lkml.org/lkml/2014/12/5/114
https://lkml.org/lkml/2014/12/10/20
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg788792.html

This patch set is based on v3.19-rc3. And you may access it at:
https://github.com/jiangliu/linux.git ir_init_v2

This patch set combimes above three patches to simplify IR initalization
flow and solves memory allocation issues. While at it, this patch set
also refines CPU X2APIC initialization code for maintenance. It also
fixes two bugs related to X2APIC support.
1) System hangs or panics if BIOS enables CPU X2APIC mode but kernel
   doesn't support X2APIC.
2) System livelocks if BIOS enables CPU X2APIC but opt-outs IR X2APIC.

This patch set has been tested with on an Intel 4-socket system with
following configuration:
---------------------------------------------------------------------------
[CPU X2APIC]	[IR X2APIC]	[Linux IR]	[Linux X2APIC]	[Result]
1     D		    /		    D		    D		  OK
2     D		    /		    E		    D		  OK
3     D		    /		    E		    E		  OK
4     P		    /		    E		    E		  OK
5     P		    /		    D		    D		  Panic(expected)
6     P		    /		    E		    D		  Panic(expected)
7     P		    H		    E		    E		  OK
----------------------------------------------------------------------------
CPU X2APIC: whether CPU X2APIC is enabled by hardware and BIOS
IR X2APIC: whether interrupt remapping hardware supports X2APIC mode
Linux IR: whether interrupt remapping is enabled by Linux kernel
Linux X2APIC: whether X2APIC is supported by Linux kernel
D: disabled
E: enabled
/: Not care
P: CPU X2APIC pre-enabled by BIOS
H: Hard-coded to opt-out X2APIC support in interrupt remapping hardware

The patch set changes the behevior of the last three rows:
1) Row 5 and 6 panics with clear messages instead of random hang or panic.
2) Row 7 boots successfully instead of livelocking.

The patch set also passes Fengguang's 0day test suites.

Due to lack of hardware platforms for tests, tests on AMD platform are
welcomed!

V1->V2:
1) Rebase onto v3.19-rc3

Jiang Liu (11):
  x86/apic: Panic if kernel doesn't support x2apic but BIOS has enabled
    x2apic
  x86/apic: Kill useless variable x2apic_enabled in function
    enable_IR_x2apic()
  x86/apic: Correctly detect X2APIC status in function enable_IR()
  x86/apic: Refine enable_IR_x2apic() and related functions
  iommu/vt-d: Prepare for killing function irq_remapping_supported()
  iommu/vt-d: Allow IR works in XAPIC mode though CPU works in X2APIC
    mode
  x86/apic: Only disable CPU x2apic mode when necessary
  iommu/irq_remapping: Kill function irq_remapping_supported() and
    related code
  iommu/irq_remapping: Refine function irq_remapping_prepare() for
    maintenance
  iommu/irq_remapping: Change variable disable_irq_remap to be static
  iommu/irq_remapping: Normailize the way to detect whether IR is
    enabled

Joerg Roedel (2):
  iommu/vt-d: Allocate IRQ remapping data structures only for all
    IOMMUs
  iommu/amd: Check for irq-remap support amd_iommu_prepare()

Thomas Gleixner (3):
  iommu, x86: Restructure setup of the irq remapping feature
  iommu/vt-d: Move iommu preparatory allocations to
    irq_remap_ops.prepare
  iommu/vt-d: Convert allocations to GFP_KERNEL

 arch/x86/include/asm/irq_remapping.h |    4 --
 arch/x86/kernel/apic/apic.c          |  104 ++++++++++++++++------------------
 drivers/iommu/amd_iommu.c            |    1 -
 drivers/iommu/amd_iommu_init.c       |   10 +---
 drivers/iommu/amd_iommu_proto.h      |    1 -
 drivers/iommu/intel_irq_remapping.c  |   96 ++++++++++++++++---------------
 drivers/iommu/irq_remapping.c        |   74 ++++++++----------------
 drivers/iommu/irq_remapping.h        |    5 --
 8 files changed, 128 insertions(+), 167 deletions(-)

-- 
1.7.10.4


^ permalink raw reply	[flat|nested] 56+ messages in thread
* [patch 0/5] x86/iommu: Bootup stage cleanups
@ 2014-12-05  8:48 Thomas Gleixner
  2014-12-05  8:48 ` [patch 1/5] x86, smpboot: Remove pointless preempt_disable() in native_smp_prepare_cpus() Thomas Gleixner
                   ` (5 more replies)
  0 siblings, 6 replies; 56+ messages in thread
From: Thomas Gleixner @ 2014-12-05  8:48 UTC (permalink / raw)
  To: LKML; +Cc: x86, Jiang Liu, Joerg Roedel, Borislav Petkov

Boris reported that the conversion to stacked irqdomains causes a
GFP_KERNEL allocation from atomic context splat.

The callchain is:

kernel_init_freeable
 smp_prepare_cpus
  default_setup_apic_routing
   enable_IR_x2apic
    irq_remapping_enable
     intel_enable_irq_remapping
       intel_setup_irq_remapping
         irq_domain_add_hierarchy
	   __irq_domain_add

The reason for the splat is that smp_prepare_cpus() is pointlessly
disabling preemption and enable_IR_x2apic is calling
irq_remapping_enable with interrupts disabled.

Removing the completely useless preemption disable from
smp_prepare_cpus() is a no brainer, but resolving the interrupt
disabled issue in enable_IR_x2apic/irq_remapping_enable forced me to
look deeper into the iommu setup code.

Convoluted is probably an euphemism for that maze.

The early detection of iommus happens in mem_init() which looks for
the availability of ACPI tables describing iommus.

The actual parsing of the tables happens for AMD in
setup_irq_remapping_ops() which calls remap_ops.prepare() and for
Intel in irq_remapping_enable().

The AMD parsing allocates data structures, memory pages and the irq
domain from the prepare callback, which makes a lot of sense as this
has nothing to do with the actual enablement. That AMD code uses
GFP_KERNEL allocations inside of a preempt disabled region. That issue
is solved by removing the silly preempt_disable() from
smp_prepare_cpus().

For Intel the actual parsing and allocation is moved from the enable
to the prepare callback. That cures the reported splat and allows to
use GFP_KERNEL allocations for the iommu internal stuff as well. While
in the early bootup it does not matter much, for physical hotplug
GFP_ATOMIC allocations are not desired at all.

That solves the issue at hand, but the iommu setup stays a tangled
maze with completely non obvious setup mechanisms depending on the
options (iommu on/off - irq remapping on/off).

Thanks,

	tglx
---
 arch/x86/include/asm/irq_remapping.h |    2 -
 arch/x86/kernel/apic/apic.c          |    3 -
 arch/x86/kernel/smpboot.c            |    8 ----
 drivers/iommu/intel_irq_remapping.c  |   68 +++++++++++++++++++++++------------
 drivers/iommu/irq_remapping.c        |   19 +++------
 kernel/irq/irqdomain.c               |   21 +---------
 6 files changed, 56 insertions(+), 65 deletions(-)






^ permalink raw reply	[flat|nested] 56+ messages in thread

end of thread, back to index

Thread overview: 56+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-01-07  7:31 [Patch v2 00/16] Refine IR initialization flow and fixes bugs related to X2APIC Jiang Liu
2015-01-07  7:31 ` [Patch v2 01/16] iommu, x86: Restructure setup of the irq remapping feature Jiang Liu
2015-01-15 12:58   ` [tip:x86/apic] " tip-bot for Thomas Gleixner
2015-01-07  7:31 ` [Patch v2 02/16] iommu/vt-d: Move iommu preparatory allocations to irq_remap_ops.prepare Jiang Liu
2015-01-15 12:58   ` [tip:x86/apic] " tip-bot for Thomas Gleixner
2015-01-07  7:31 ` [Patch v2 03/16] iommu/vt-d: Convert allocations to GFP_KERNEL Jiang Liu
2015-01-15 12:59   ` [tip:x86/apic] " tip-bot for Thomas Gleixner
2015-01-07  7:31 ` [Patch v2 04/16] x86/apic: Panic if kernel doesn't support x2apic but BIOS has enabled x2apic Jiang Liu
2015-01-15 12:59   ` [tip:x86/apic] x86/apic: Panic if kernel doesn' t " tip-bot for Jiang Liu
2015-01-07  7:31 ` [Patch v2 05/16] x86/apic: Kill useless variable x2apic_enabled in function enable_IR_x2apic() Jiang Liu
2015-01-15 12:59   ` [tip:x86/apic] " tip-bot for Jiang Liu
2015-01-07  7:31 ` [Patch v2 06/16] x86/apic: Correctly detect X2APIC status in function enable_IR() Jiang Liu
2015-01-15 13:00   ` [tip:x86/apic] " tip-bot for Jiang Liu
2015-01-07  7:31 ` [Patch v2 07/16] x86/apic: Refine enable_IR_x2apic() and related functions Jiang Liu
2015-01-15  9:24   ` Thomas Gleixner
2015-01-15 13:00   ` [tip:x86/apic] " tip-bot for Jiang Liu
2015-01-15 13:00   ` [tip:x86/apic] x86/apic: Handle XAPIC remap mode proper tip-bot for Jiang Liu
2015-01-07  7:31 ` [Patch v2 08/16] iommu/vt-d: Prepare for killing function irq_remapping_supported() Jiang Liu
2015-01-15 13:01   ` [tip:x86/apic] " tip-bot for Jiang Liu
2015-01-07  7:31 ` [Patch v2 09/16] iommu/vt-d: Allocate IRQ remapping data structures only for all IOMMUs Jiang Liu
2015-01-15 13:01   ` [tip:x86/apic] " tip-bot for Joerg Roedel
2015-01-07  7:31 ` [Patch v2 10/16] iommu/vt-d: Allow IR works in XAPIC mode though CPU works in X2APIC mode Jiang Liu
2015-01-15 13:01   ` [tip:x86/apic] " tip-bot for Jiang Liu
2015-01-07  7:31 ` [Patch v2 11/16] x86/apic: Only disable CPU x2apic mode when necessary Jiang Liu
2015-01-15 13:02   ` [tip:x86/apic] " tip-bot for Jiang Liu
2015-01-07  7:31 ` [Patch v2 12/16] iommu/amd: Check for irq-remap support amd_iommu_prepare() Jiang Liu
2015-01-15 13:02   ` [tip:x86/apic] " tip-bot for Joerg Roedel
2015-01-07  7:31 ` [Patch v2 13/16] iommu/irq_remapping: Kill function irq_remapping_supported() and related code Jiang Liu
2015-01-15 13:02   ` [tip:x86/apic] " tip-bot for Jiang Liu
2015-01-07  7:31 ` [Patch v2 14/16] iommu/irq_remapping: Refine function irq_remapping_prepare() for maintenance Jiang Liu
2015-01-15 13:03   ` [tip:x86/apic] " tip-bot for Jiang Liu
2015-01-07  7:31 ` [Patch v2 15/16] iommu/irq_remapping: Change variable disable_irq_remap to be static Jiang Liu
2015-01-15 13:03   ` [tip:x86/apic] " tip-bot for Jiang Liu
2015-01-07  7:31 ` [Patch v2 16/16] iommu/irq_remapping: Normailize the way to detect whether IR is enabled Jiang Liu
2015-01-15 13:03   ` [tip:x86/apic] " tip-bot for Jiang Liu
2015-01-09 12:03 ` [Patch v2 00/16] Refine IR initialization flow and fixes bugs related to X2APIC Joerg Roedel
  -- strict thread matches above, loose matches on Subject: below --
2014-12-05  8:48 [patch 0/5] x86/iommu: Bootup stage cleanups Thomas Gleixner
2014-12-05  8:48 ` [patch 1/5] x86, smpboot: Remove pointless preempt_disable() in native_smp_prepare_cpus() Thomas Gleixner
2014-12-05 23:25   ` [tip:x86/apic] " tip-bot for Thomas Gleixner
2014-12-19 14:02   ` tip-bot for Thomas Gleixner
2014-12-05  8:48 ` [patch 2/5] iommu, x86: Restructure setup of the irq remapping feature Thomas Gleixner
2014-12-05 23:25   ` [tip:x86/apic] " tip-bot for Thomas Gleixner
2014-12-05  8:48 ` [patch 3/5] iommu/vt-d: Move iommu preparatory allocations to irq_remap_ops.prepare Thomas Gleixner
2014-12-05 23:26   ` [tip:x86/apic] " tip-bot for Thomas Gleixner
2014-12-11  7:35     ` Yinghai Lu
2014-12-11 14:33       ` Jiang Liu
2014-12-11 17:57         ` Yinghai Lu
2014-12-11 20:30           ` Thomas Gleixner
2014-12-12  2:04             ` Yinghai Lu
2015-04-27 22:46               ` Yinghai Lu
2015-04-29  8:15                 ` Jiang Liu
2014-12-05  8:48 ` [patch 4/5] irqdomain: Revert gfp hackery Thomas Gleixner
2014-12-05 23:26   ` [tip:x86/apic] " tip-bot for Thomas Gleixner
2014-12-05  8:48 ` [patch 5/5] iommu/vt-d: Convert allocations to GFP_KERNEL Thomas Gleixner
2014-12-05 23:26   ` [tip:x86/apic] " tip-bot for Thomas Gleixner
2014-12-05 12:22 ` [patch 0/5] x86/iommu: Bootup stage cleanups Joerg Roedel

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git