Re: ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

Re: ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Jonathan Nieder]

Hi Dan,

Thomas and William (cc-ed) have been having trouble loading the
ioatdma driver on a 32-bit Xen dom0.  The module loads automatically
at boot time and trips

	BUG_ON(active && !seen_current); /* no active descs have written a completion? */

from drivers/dma/ioat/dma_v2.c.  That check has been present since
5cbafa65b92e (ioat2,3: convert to a true ring buffer, 2009-08-26).
The bug is probably in Xen code and seems to be a regression (the bug
is present in 3.2 but not 3.1.8).

Thomas Goirand wrote:
> On 03/01/2012 11:53 PM, Bastian Blank wrote:
>> On Thu, Mar 01, 2012 at 06:02:15PM +0800, Thomas Goirand wrote:

>>>                     Any clue why I don't see crashes without Xen, with a
>>> 64 bits kernel, or with earlier versions of Linux (eg: 3.1 for example)?
>>
>> xen/i386 uses a different memory model to anything else, this may be a
>> problem.
[...]
> Replacing BUG_ON by a WARN_ON, and adding #define DEBUG 1 on top of
> dma_v2.c, my kernel booted, and I had the attached dmesg output.
>
> Blacklisting the ioatdma kernel module of course, solved the issue.
>
> I hope that helps, please let me know if I should do more to help. If
> you need access to my server, that's possible (I use it only for
> packaging XCP and some tests...).

I don't expect you to debug this Xen-specific bug, but I'm wondering:
is there any reason this check has to be a BUG_ON instead of a
WARN_ON?  If there is some way to recover when the impossible happens,
that would make using and debugging the kernel a little easier.

Curious,
Jonathan

Re: [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Bastian Blank]

On Thu, Mar 01, 2012 at 10:42:44PM -0800, Dan Williams wrote:
> WARN_ON may work, but then kernel may be subject random hangs from missed
> i/o completions.

Why is that? Currently it just dies if was triggered via interrupt and
for some reason no active descriptor was found.

>                   Is xen32 using vt-d?

Yes. Xen can use VT-D.

>                                         Just wondering if writes from ioat
> device are getting misdirected.

How do VT-D and ioat interact?

Bastian

-- 
Too much of anything, even love, isn't necessarily a good thing.
		-- Kirk, "The Trouble with Tribbles", stardate 4525.6

Re: [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Dan Williams]

On Fri, Mar 2, 2012 at 8:21 AM, Bastian Blank <waldi@debian.org> wrote:
> On Thu, Mar 01, 2012 at 10:42:44PM -0800, Dan Williams wrote:
>> WARN_ON may work, but then kernel may be subject random hangs from missed
>> i/o completions.

...actually descriptors completing too early.

>
> Why is that? Currently it just dies if was triggered via interrupt and
> for some reason no active descriptor was found.

No, it's not the case that "no active descriptor was found".

The channel is walking through the submitted descriptor chain to catch
up with what was last posted to '
phys_complete'.  It expects to stop when seeing phys_complete, but if
it never finds it the driver ends up completing the entire pending
ring.  The BUG_ON is there because the driver has just completed every
descriptor in the chain, and if the kernel was depending on proper
descriptor ordering it may have just violated it.

So I take it back, we can't go to WARN_ON, because the state of the
system is compromised and we need to bring it to a halt.

That said the code is likely failing in the self test, so the system
is probably fine, but if this happened in the network or raid layer it
is potentially fatal.

>>                   Is xen32 using vt-d?
>
> Yes. Xen can use VT-D.
>
>>                                         Just wondering if writes from ioat
>> device are getting misdirected.
>
> How do VT-D and ioat interact?
>

Same as any other pci bus mastering device, via dma_map to get a
io-virtual  address.

Re: [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Bastian Blank]

On Fri, Mar 02, 2012 at 08:44:00AM -0800, Dan Williams wrote:
> On Fri, Mar 2, 2012 at 8:21 AM, Bastian Blank <waldi@debian.org> wrote:
> > On Thu, Mar 01, 2012 at 10:42:44PM -0800, Dan Williams wrote:
> >> WARN_ON may work, but then kernel may be subject random hangs from missed
> >> i/o completions.
> ...actually descriptors completing too early.

The interrupt happens while the module is still loading, so most likely
directly after enabling them. There should be no request in flight yet.

What puzzles me is the mix of different data types in the ioatdma
driver:

| u64 completion = *chan->completion;
| unsigned long phys_complete = completion & ~0x3f;

The state is 64bit long, but is down converted to a 32bit value without
anything.

phys_complete (a 32 bit value) gets compared to struct
dma_async_tx_descriptor.phys, which is defined as dma_addr_t, a _64_ bit
value.

Bastian

-- 
... The prejudices people feel about each other disappear when they get
to know each other.
		-- Kirk, "Elaan of Troyius", stardate 4372.5

Re: [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Dan Williams]

On Fri, Mar 2, 2012 at 9:57 AM, Bastian Blank <waldi@debian.org> wrote:
> On Fri, Mar 02, 2012 at 08:44:00AM -0800, Dan Williams wrote:
>> On Fri, Mar 2, 2012 at 8:21 AM, Bastian Blank <waldi@debian.org> wrote:
>> > On Thu, Mar 01, 2012 at 10:42:44PM -0800, Dan Williams wrote:
>> >> WARN_ON may work, but then kernel may be subject random hangs from missed
>> >> i/o completions.
>> ...actually descriptors completing too early.
>
> The interrupt happens while the module is still loading, so most likely
> directly after enabling them. There should be no request in flight yet.
>
> What puzzles me is the mix of different data types in the ioatdma
> driver:
>
> | u64 completion = *chan->completion;
> | unsigned long phys_complete = completion & ~0x3f;
>
> The state is 64bit long, but is down converted to a 32bit value without
> anything.
>
> phys_complete (a 32 bit value) gets compared to struct
> dma_async_tx_descriptor.phys, which is defined as dma_addr_t, a _64_ bit
> value.

The assumption is that the driver's control structures are not in high
memory so all address values will only have 32-bits of valid data, but
maybe xen32 changes that assumption?

Can you send the log of the driver load with debug enabled?

diff --git a/drivers/dma/ioat/dma.c b/drivers/dma/ioat/dma.c
index a4d6cb0..82472de 100644
--- a/drivers/dma/ioat/dma.c
+++ b/drivers/dma/ioat/dma.c
@@ -24,7 +24,7 @@
  * This driver supports an Intel I/OAT DMA engine, which does asynchronous
  * copy operations.
  */
-
+#define DEBUG
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/slab.h>
diff --git a/drivers/dma/ioat/dma_v2.c b/drivers/dma/ioat/dma_v2.c
index 5d65f83..da337e7 100644
--- a/drivers/dma/ioat/dma_v2.c
+++ b/drivers/dma/ioat/dma_v2.c
@@ -24,7 +24,7 @@
  * This driver supports an Intel I/OAT DMA engine (versions >= 2), which
  * does asynchronous data movement and checksumming operations.
  */
-
+#define DEBUG
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/slab.h>

Re: [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Bastian Blank]

On Fri, Mar 02, 2012 at 11:31:56AM -0800, Dan Williams wrote:
> On Fri, Mar 2, 2012 at 9:57 AM, Bastian Blank <waldi@debian.org> wrote:
> > phys_complete (a 32 bit value) gets compared to struct
> > dma_async_tx_descriptor.phys, which is defined as dma_addr_t, a _64_ bit
> > value.
> The assumption is that the driver's control structures are not in high
> memory so all address values will only have 32-bits of valid data,

Can you back that up by some kernel documentation? There is a reason why
pci_alloc_pool uses dma_addr_t to store the address and _not_ unsigned
long. This are physical addresses, nothing the kernel can access
directly without a mapping.

>                                                                    but
> maybe xen32 changes that assumption?

Xen changes a lot of things in the memory management. This includes that
physical != machine addresses, where i915 failed horrible.

> Can you send the log of the driver load with debug enabled?

No, I don't have that hardware.

Bastian

-- 
Each kiss is as the first.
		-- Miramanee, Kirk's wife, "The Paradise Syndrome",
		   stardate 4842.6

Re: [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Dan Williams]

On Fri, Mar 2, 2012 at 12:08 PM, Bastian Blank <waldi@debian.org> wrote:
> On Fri, Mar 02, 2012 at 11:31:56AM -0800, Dan Williams wrote:
>> On Fri, Mar 2, 2012 at 9:57 AM, Bastian Blank <waldi@debian.org> wrote:
>> > phys_complete (a 32 bit value) gets compared to struct
>> > dma_async_tx_descriptor.phys, which is defined as dma_addr_t, a _64_ bit
>> > value.
>> The assumption is that the driver's control structures are not in high
>> memory so all address values will only have 32-bits of valid data,
>
> Can you back that up by some kernel documentation? There is a reason why
> pci_alloc_pool uses dma_addr_t to store the address and _not_ unsigned
> long. This are physical addresses, nothing the kernel can access
> directly without a mapping.

High memory can only be accessed with kmap(), so the assumption is
that dma_alloc never gives a buffer address above 32-bits on a 32-bit
build.  Yes, if HIGHMEM64G is set dma_addr_t becomes 64-bit, but that
is only to access high memory mapped application buffers via dma_map.
I'm not aware of any documentation in this area.

I don't mind bumping up the size if xen32 is changing the above
assumptions, but I'd want confirmation that this is the failure
scenario.

Re: [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Bastian Blank]

On Fri, Mar 02, 2012 at 12:16:47PM -0800, Dan Williams wrote:
> On Fri, Mar 2, 2012 at 12:08 PM, Bastian Blank <waldi@debian.org> wrote:
> > On Fri, Mar 02, 2012 at 11:31:56AM -0800, Dan Williams wrote:
> >> On Fri, Mar 2, 2012 at 9:57 AM, Bastian Blank <waldi@debian.org> wrote:
> >> > phys_complete (a 32 bit value) gets compared to struct
> >> > dma_async_tx_descriptor.phys, which is defined as dma_addr_t, a _64_ bit
> >> > value.
> >> The assumption is that the driver's control structures are not in high
> >> memory so all address values will only have 32-bits of valid data,
> > Can you back that up by some kernel documentation? There is a reason why
> > pci_alloc_pool uses dma_addr_t to store the address and _not_ unsigned
> > long. This are physical addresses, nothing the kernel can access
> > directly without a mapping.
> High memory can only be accessed with kmap(), so the assumption is
> that dma_alloc never gives a buffer address above 32-bits on a 32-bit
> build.  Yes, if HIGHMEM64G is set dma_addr_t becomes 64-bit, but that
> is only to access high memory mapped application buffers via dma_map.

All memory needs to be mapped.  Linux just have a default mapping of 1GiB
of the memory handy. However this is irrelevant for the physical DMA
addresses we talk about.

A assume this devices have a DMA mask of 2^64, so they can address
memory above the 4GiB.  And the kernel will happily assign this memory
if necessary or usefull.

> I'm not aware of any documentation in this area.

There is; the header files qualifies as documentation.

> I don't mind bumping up the size if xen32 is changing the above
> assumptions, but I'd want confirmation that this is the failure
> scenario.

At least it looks pretty wrong to remove four bits from a given address
just for fun.

Bastian

-- 
Respect is a rational process
		-- McCoy, "The Galileo Seven", stardate 2822.3

Re: [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Dan Williams]

On Fri, Mar 2, 2012 at 12:56 PM, Bastian Blank <waldi@debian.org> wrote:
> On Fri, Mar 02, 2012 at 12:16:47PM -0800, Dan Williams wrote:
>> On Fri, Mar 2, 2012 at 12:08 PM, Bastian Blank <waldi@debian.org> wrote:
>> > On Fri, Mar 02, 2012 at 11:31:56AM -0800, Dan Williams wrote:
>> >> On Fri, Mar 2, 2012 at 9:57 AM, Bastian Blank <waldi@debian.org> wrote:
>> >> > phys_complete (a 32 bit value) gets compared to struct
>> >> > dma_async_tx_descriptor.phys, which is defined as dma_addr_t, a _64_ bit
>> >> > value.
>> >> The assumption is that the driver's control structures are not in high
>> >> memory so all address values will only have 32-bits of valid data,
>> > Can you back that up by some kernel documentation? There is a reason why
>> > pci_alloc_pool uses dma_addr_t to store the address and _not_ unsigned
>> > long. This are physical addresses, nothing the kernel can access
>> > directly without a mapping.
>> High memory can only be accessed with kmap(), so the assumption is
>> that dma_alloc never gives a buffer address above 32-bits on a 32-bit
>> build.  Yes, if HIGHMEM64G is set dma_addr_t becomes 64-bit, but that
>> is only to access high memory mapped application buffers via dma_map.
>
> All memory needs to be mapped.  Linux just have a default mapping of 1GiB
> of the memory handy. However this is irrelevant for the physical DMA
> addresses we talk about.

I'm not sure you understand how himem works or we're talking past each other.

Re: [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Thomas Goirand]

On 03/03/2012 03:31 AM, Dan Williams wrote:
> Can you send the log of the driver load with debug enabled?
> 
> diff --git a/drivers/dma/ioat/dma.c b/drivers/dma/ioat/dma.c
> index a4d6cb0..82472de 100644
> --- a/drivers/dma/ioat/dma.c
> +++ b/drivers/dma/ioat/dma.c
> @@ -24,7 +24,7 @@
>   * This driver supports an Intel I/OAT DMA engine, which does asynchronous
>   * copy operations.
>   */
> -
> +#define DEBUG
>  #include <linux/init.h>
>  #include <linux/module.h>
>  #include <linux/slab.h>
> diff --git a/drivers/dma/ioat/dma_v2.c b/drivers/dma/ioat/dma_v2.c
> index 5d65f83..da337e7 100644
> --- a/drivers/dma/ioat/dma_v2.c
> +++ b/drivers/dma/ioat/dma_v2.c
> @@ -24,7 +24,7 @@
>   * This driver supports an Intel I/OAT DMA engine (versions >= 2), which
>   * does asynchronous data movement and checksumming operations.
>   */
> -
> +#define DEBUG
>  #include <linux/init.h>
>  #include <linux/module.h>
>  #include <linux/slab.h>

I will do my best to provide it ASAP. Should I compile with BUG_ON so
you see it crashing, as per the original code, or just with WARN_ON, so
you also see further things in dmesg?

Thomas

Re: [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Dan Williams]

On Mon, Mar 5, 2012 at 7:26 AM, Thomas Goirand <zigo@debian.org> wrote:
> I will do my best to provide it ASAP. Should I compile with BUG_ON so
> you see it crashing, as per the original code, or just with WARN_ON, so
> you also see further things in dmesg?

Yes, replacing with a WARN_ON might allow it to skid after the crash
and give a bit more information.

Thank you for grabbing this info.

--
Dan

Re: [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Thomas Goirand]

[-- Attachment #1: Type: text/plain, Size: 919 bytes --]

On 03/05/2012 11:38 PM, Dan Williams wrote:
> On Mon, Mar 5, 2012 at 7:26 AM, Thomas Goirand <zigo@debian.org> wrote:
>> I will do my best to provide it ASAP. Should I compile with BUG_ON so
>> you see it crashing, as per the original code, or just with WARN_ON, so
>> you also see further things in dmesg?
> 
> Yes, replacing with a WARN_ON might allow it to skid after the crash
> and give a bit more information.
> 
> Thank you for grabbing this info.
> 
> --
> Dan

Hi Dan,

Please find attached the log that you asked me, with WARN_ON instead of
BUG_ON, and with the 2 #define DEBUG in dma.c and dma_v2.c.

Let me know if you want me to do more, or if you want to have access to
my server (in which case, provide me a public ssh key and sign your
email with PGP).

Thomas

P.S: I compressed the dmesg.txt because on debian lists if a message is
>= 40K, it requires administrator moderation, which I want to avoid.

[-- Attachment #2: dmesg.txt.gz --]
[-- Type: application/x-gzip, Size: 20336 bytes --]

Re: [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Bastian Blank]

On Tue, Mar 06, 2012 at 05:20:54PM +0800, Thomas Goirand wrote:
> Please find attached the log that you asked me, with WARN_ON instead of
> BUG_ON, and with the 2 #define DEBUG in dma.c and dma_v2.c.

| ioatdma 0000:00:16.1: desc[1]: (0x27ea6d040->0x27ea6d080) cookie: 0 flags: 0x0 ctl: 0x0 (op: 0 int_en: 0 compl: 0)
| ioatdma 0000:00:16.1: desc[1]: (0x27ea6d040->0x27ea6d080) cookie: 0 flags: 0x31 ctl: 0x9 (op: 0 int_en: 1 compl: 1)

*counting* 9 hex digest, aka > 2^32. What did I say?

Bastian

-- 
The joys of love made her human and the agonies of love destroyed her.
		-- Spock, "Requiem for Methuselah", stardate 5842.8

Re: [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Dan Williams]

On Tue, Mar 6, 2012 at 1:20 AM, Thomas Goirand <zigo@debian.org> wrote:
> On 03/05/2012 11:38 PM, Dan Williams wrote:
>> On Mon, Mar 5, 2012 at 7:26 AM, Thomas Goirand <zigo@debian.org> wrote:
>>> I will do my best to provide it ASAP. Should I compile with BUG_ON so
>>> you see it crashing, as per the original code, or just with WARN_ON, so
>>> you also see further things in dmesg?
>>
>> Yes, replacing with a WARN_ON might allow it to skid after the crash
>> and give a bit more information.
>>
>> Thank you for grabbing this info.
>>
>> --
>> Dan
>
> Hi Dan,
>
> Please find attached the log that you asked me, with WARN_ON instead of
> BUG_ON, and with the 2 #define DEBUG in dma.c and dma_v2.c.
>

[    9.276817] ioatdma 0000:00:16.4: desc[0]:
(0x300cc7000->0x300cc7040) cookie: 0 flags: 0x2 ctl: 0x29 (op: 0
int_en: 1 compl: 1)
...
[    9.276832] ioatdma 0000:00:16.4: ioat_get_current_completion:
phys_complete: 0xcc7000

Thanks, this clearly shows that our descriptors are above 4GB and that
the driver truncates the completion word.

Is this new behavior for xen?

Before you had mentioned that non-xen 32-bit builds don't fail.  Can
you send me the .config from those two cases (offlist if they are too
large)?  I'm looking for what config option enables this so I can
quote it in the patch to increase the size of phys_complete.
Certainly this changes my assumptions of what address ranges
GFP_KERNEL memory will be located.

--
Dan

Re: [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Ian Campbell]

On Tue, 2012-03-06 at 06:14 -0800, Dan Williams wrote:
> [    9.276817] ioatdma 0000:00:16.4: desc[0]:
> (0x300cc7000->0x300cc7040) cookie: 0 flags: 0x2 ctl: 0x29 (op: 0
> int_en: 1 compl: 1)
> ...
> [    9.276832] ioatdma 0000:00:16.4: ioat_get_current_completion:
> phys_complete: 0xcc7000
> 
> Thanks, this clearly shows that our descriptors are above 4GB and that
> the driver truncates the completion word.
> 
> Is this new behavior for xen?

Xen makes a distinction between physical addresses and DMA addresses and
the latter can potentially be anywhere in the machine's real address
space while the former is what GFP_KERNEL etc controls.

You are using pci_pool_alloc which is the correct API to use for these
things since it's purpose is to handle cases where PHYS != DMA addr by
exposing the DMA address to the caller. As part of that you should also
be using dma_addr_t for DMA addresses since that is the type which is
defined to handle the appropriate DMA address size on the platform.

I think this DMA!=PHYS can also be true of some non-x86 architectures
without Xen too but I guess ioat is quite x86 specific? In any case it
is wrong, or at least non-portable, to use unsigned long for these
addresses even though it happens on x86 that physaddr == dma addr
(usually).

Ian.

-- 
Ian Campbell

Start every day off with a smile and get it over with.
		-- W. C. Fields

Re: ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Jonathan Nieder]

Hi Dan,

Dan Williams wrote:

> Before you had mentioned that non-xen 32-bit builds don't fail.  Can
> you send me the .config from those two cases (offlist if they are too
> large)?

The failing and non-failing kernels are identical.  It is the
environment in which they are run that is different.

Running the kernel on bare metal works fine, while booting as a dom0
from the xen hypervisor triggers the assertion failure.[1]

.config: [2]

Hope that helps,
Jonathan

[1] http://thread.gmane.org/gmane.comp.emulators.xen.devel/121604/focus=121615
[2] http://alioth.debian.org/~jrnieder-guest/temp/config-3.2.0-2-686-pae

Re: [Xen-devel] [Pkg-xen-devel] ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Konrad Rzeszutek Wilk]

On Tue, Mar 06, 2012 at 06:39:12AM -0800, Ian Campbell wrote:
> On Tue, 2012-03-06 at 06:14 -0800, Dan Williams wrote:
> > [    9.276817] ioatdma 0000:00:16.4: desc[0]:
> > (0x300cc7000->0x300cc7040) cookie: 0 flags: 0x2 ctl: 0x29 (op: 0
> > int_en: 1 compl: 1)
> > ...
> > [    9.276832] ioatdma 0000:00:16.4: ioat_get_current_completion:
> > phys_complete: 0xcc7000
> > 
> > Thanks, this clearly shows that our descriptors are above 4GB and that
> > the driver truncates the completion word.
> > 
> > Is this new behavior for xen?
> 
> Xen makes a distinction between physical addresses and DMA addresses and
> the latter can potentially be anywhere in the machine's real address
> space while the former is what GFP_KERNEL etc controls.
> 
> You are using pci_pool_alloc which is the correct API to use for these
> things since it's purpose is to handle cases where PHYS != DMA addr by
> exposing the DMA address to the caller. As part of that you should also
> be using dma_addr_t for DMA addresses since that is the type which is
> defined to handle the appropriate DMA address size on the platform.
> 
> I think this DMA!=PHYS can also be true of some non-x86 architectures

Especially SPARC.
> without Xen too but I guess ioat is quite x86 specific? In any case it
> is wrong, or at least non-portable, to use unsigned long for these
> addresses even though it happens on x86 that physaddr == dma addr
> (usually).

I think with the Intel VT-d that can be different. The bus addresses returned
do seem different.

Re: ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Dan Williams]

Subject: ioat: fix size of 'completion' for Xen

From: Dan Williams <dan.j.williams@intel.com>

Starting with v3.2 Jonathan reports that Xen crashes loading the ioatdma
driver.  A debug run shows:

  ioatdma 0000:00:16.4: desc[0]: (0x300cc7000->0x300cc7040) cookie: 0 flags: 0x2 ctl: 0x29 (op: 0 int_en: 1 compl: 1)
  ...
  ioatdma 0000:00:16.4: ioat_get_current_completion: phys_complete: 0xcc7000

...which shows that in this environment GFP_KERNEL memory may be backed
by a 64-bit dma address.  This breaks the driver's assumption that an
unsigned long should be able to contain the physical address for
descriptor memory.  Switch to dma_addr_t which beyond being the right
size, is the true type for the data i.e. an io-virtual address
indicating the engine's last processed descriptor.

[stable: 3.2+]
Cc: <stable@vger.kernel.org>
Reported-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---

On Sun, 2012-03-11 at 17:06 -0500, Jonathan Nieder wrote:
Hi Dan,
> 
> Dan Williams wrote:
> 
> > Before you had mentioned that non-xen 32-bit builds don't fail.  Can
> > you send me the .config from those two cases (offlist if they are too
> > large)?
> 
> The failing and non-failing kernels are identical.  It is the
> environment in which they are run that is different.
> 
> Running the kernel on bare metal works fine, while booting as a dom0
> from the xen hypervisor triggers the assertion failure.[1]
> 
> .config: [2]
> 
> Hope that helps,
> Jonathan

Thanks for the debug help, does this patch fix the issue for you?

 drivers/dma/ioat/dma.c    |   16 ++++++++--------
 drivers/dma/ioat/dma.h    |    6 +++---
 drivers/dma/ioat/dma_v2.c |    8 ++++----
 drivers/dma/ioat/dma_v3.c |    8 ++++----
 4 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/drivers/dma/ioat/dma.c b/drivers/dma/ioat/dma.c
index a4d6cb0..6595180 100644
--- a/drivers/dma/ioat/dma.c
+++ b/drivers/dma/ioat/dma.c
@@ -548,9 +548,9 @@ void ioat_dma_unmap(struct ioat_chan_common *chan, enum dma_ctrl_flags flags,
 			   PCI_DMA_TODEVICE, flags, 0);
 }
 
-unsigned long ioat_get_current_completion(struct ioat_chan_common *chan)
+dma_addr_t ioat_get_current_completion(struct ioat_chan_common *chan)
 {
-	unsigned long phys_complete;
+	dma_addr_t phys_complete;
 	u64 completion;
 
 	completion = *chan->completion;
@@ -571,7 +571,7 @@ unsigned long ioat_get_current_completion(struct ioat_chan_common *chan)
 }
 
 bool ioat_cleanup_preamble(struct ioat_chan_common *chan,
-			   unsigned long *phys_complete)
+			   dma_addr_t *phys_complete)
 {
 	*phys_complete = ioat_get_current_completion(chan);
 	if (*phys_complete == chan->last_completion)
@@ -582,14 +582,14 @@ bool ioat_cleanup_preamble(struct ioat_chan_common *chan,
 	return true;
 }
 
-static void __cleanup(struct ioat_dma_chan *ioat, unsigned long phys_complete)
+static void __cleanup(struct ioat_dma_chan *ioat, dma_addr_t phys_complete)
 {
 	struct ioat_chan_common *chan = &ioat->base;
 	struct list_head *_desc, *n;
 	struct dma_async_tx_descriptor *tx;
 
-	dev_dbg(to_dev(chan), "%s: phys_complete: %lx\n",
-		 __func__, phys_complete);
+	dev_dbg(to_dev(chan), "%s: phys_complete: %llx\n",
+		 __func__, (unsigned long long) phys_complete);
 	list_for_each_safe(_desc, n, &ioat->used_desc) {
 		struct ioat_desc_sw *desc;
 
@@ -655,7 +655,7 @@ static void __cleanup(struct ioat_dma_chan *ioat, unsigned long phys_complete)
 static void ioat1_cleanup(struct ioat_dma_chan *ioat)
 {
 	struct ioat_chan_common *chan = &ioat->base;
-	unsigned long phys_complete;
+	dma_addr_t phys_complete;
 
 	prefetch(chan->completion);
 
@@ -701,7 +701,7 @@ static void ioat1_timer_event(unsigned long data)
 		mod_timer(&chan->timer, jiffies + COMPLETION_TIMEOUT);
 		spin_unlock_bh(&ioat->desc_lock);
 	} else if (test_bit(IOAT_COMPLETION_PENDING, &chan->state)) {
-		unsigned long phys_complete;
+		dma_addr_t phys_complete;
 
 		spin_lock_bh(&ioat->desc_lock);
 		/* if we haven't made progress and we have already
diff --git a/drivers/dma/ioat/dma.h b/drivers/dma/ioat/dma.h
index 5216c8a..8bebddd 100644
--- a/drivers/dma/ioat/dma.h
+++ b/drivers/dma/ioat/dma.h
@@ -88,7 +88,7 @@ struct ioatdma_device {
 struct ioat_chan_common {
 	struct dma_chan common;
 	void __iomem *reg_base;
-	unsigned long last_completion;
+	dma_addr_t last_completion;
 	spinlock_t cleanup_lock;
 	dma_cookie_t completed_cookie;
 	unsigned long state;
@@ -333,7 +333,7 @@ int __devinit ioat_dma_self_test(struct ioatdma_device *device);
 void __devexit ioat_dma_remove(struct ioatdma_device *device);
 struct dca_provider * __devinit ioat_dca_init(struct pci_dev *pdev,
 					      void __iomem *iobase);
-unsigned long ioat_get_current_completion(struct ioat_chan_common *chan);
+dma_addr_t ioat_get_current_completion(struct ioat_chan_common *chan);
 void ioat_init_channel(struct ioatdma_device *device,
 		       struct ioat_chan_common *chan, int idx);
 enum dma_status ioat_dma_tx_status(struct dma_chan *c, dma_cookie_t cookie,
@@ -341,7 +341,7 @@ enum dma_status ioat_dma_tx_status(struct dma_chan *c, dma_cookie_t cookie,
 void ioat_dma_unmap(struct ioat_chan_common *chan, enum dma_ctrl_flags flags,
 		    size_t len, struct ioat_dma_descriptor *hw);
 bool ioat_cleanup_preamble(struct ioat_chan_common *chan,
-			   unsigned long *phys_complete);
+			   dma_addr_t *phys_complete);
 void ioat_kobject_add(struct ioatdma_device *device, struct kobj_type *type);
 void ioat_kobject_del(struct ioatdma_device *device);
 extern const struct sysfs_ops ioat_sysfs_ops;
diff --git a/drivers/dma/ioat/dma_v2.c b/drivers/dma/ioat/dma_v2.c
index 5d65f83..cb8864d 100644
--- a/drivers/dma/ioat/dma_v2.c
+++ b/drivers/dma/ioat/dma_v2.c
@@ -126,7 +126,7 @@ static void ioat2_start_null_desc(struct ioat2_dma_chan *ioat)
 	spin_unlock_bh(&ioat->prep_lock);
 }
 
-static void __cleanup(struct ioat2_dma_chan *ioat, unsigned long phys_complete)
+static void __cleanup(struct ioat2_dma_chan *ioat, dma_addr_t phys_complete)
 {
 	struct ioat_chan_common *chan = &ioat->base;
 	struct dma_async_tx_descriptor *tx;
@@ -178,7 +178,7 @@ static void __cleanup(struct ioat2_dma_chan *ioat, unsigned long phys_complete)
 static void ioat2_cleanup(struct ioat2_dma_chan *ioat)
 {
 	struct ioat_chan_common *chan = &ioat->base;
-	unsigned long phys_complete;
+	dma_addr_t phys_complete;
 
 	spin_lock_bh(&chan->cleanup_lock);
 	if (ioat_cleanup_preamble(chan, &phys_complete))
@@ -259,7 +259,7 @@ int ioat2_reset_sync(struct ioat_chan_common *chan, unsigned long tmo)
 static void ioat2_restart_channel(struct ioat2_dma_chan *ioat)
 {
 	struct ioat_chan_common *chan = &ioat->base;
-	unsigned long phys_complete;
+	dma_addr_t phys_complete;
 
 	ioat2_quiesce(chan, 0);
 	if (ioat_cleanup_preamble(chan, &phys_complete))
@@ -274,7 +274,7 @@ void ioat2_timer_event(unsigned long data)
 	struct ioat_chan_common *chan = &ioat->base;
 
 	if (test_bit(IOAT_COMPLETION_PENDING, &chan->state)) {
-		unsigned long phys_complete;
+		dma_addr_t phys_complete;
 		u64 status;
 
 		status = ioat_chansts(chan);
diff --git a/drivers/dma/ioat/dma_v3.c b/drivers/dma/ioat/dma_v3.c
index f519c93..2dbf32b 100644
--- a/drivers/dma/ioat/dma_v3.c
+++ b/drivers/dma/ioat/dma_v3.c
@@ -256,7 +256,7 @@ static bool desc_has_ext(struct ioat_ring_ent *desc)
  * The difference from the dma_v2.c __cleanup() is that this routine
  * handles extended descriptors and dma-unmapping raid operations.
  */
-static void __cleanup(struct ioat2_dma_chan *ioat, unsigned long phys_complete)
+static void __cleanup(struct ioat2_dma_chan *ioat, dma_addr_t phys_complete)
 {
 	struct ioat_chan_common *chan = &ioat->base;
 	struct ioat_ring_ent *desc;
@@ -314,7 +314,7 @@ static void __cleanup(struct ioat2_dma_chan *ioat, unsigned long phys_complete)
 static void ioat3_cleanup(struct ioat2_dma_chan *ioat)
 {
 	struct ioat_chan_common *chan = &ioat->base;
-	unsigned long phys_complete;
+	dma_addr_t phys_complete;
 
 	spin_lock_bh(&chan->cleanup_lock);
 	if (ioat_cleanup_preamble(chan, &phys_complete))
@@ -333,7 +333,7 @@ static void ioat3_cleanup_event(unsigned long data)
 static void ioat3_restart_channel(struct ioat2_dma_chan *ioat)
 {
 	struct ioat_chan_common *chan = &ioat->base;
-	unsigned long phys_complete;
+	dma_addr_t phys_complete;
 
 	ioat2_quiesce(chan, 0);
 	if (ioat_cleanup_preamble(chan, &phys_complete))
@@ -348,7 +348,7 @@ static void ioat3_timer_event(unsigned long data)
 	struct ioat_chan_common *chan = &ioat->base;
 
 	if (test_bit(IOAT_COMPLETION_PENDING, &chan->state)) {
-		unsigned long phys_complete;
+		dma_addr_t phys_complete;
 		u64 status;
 
 		status = ioat_chansts(chan);

Re: ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[William Dauchy]

On Sat, Mar 24, 2012 at 12:55 AM, Dan Williams <dan.j.williams@intel.com> wrote:
> Starting with v3.2 Jonathan reports that Xen crashes loading the ioatdma
> driver.  A debug run shows:

Please note that I reported the crash a bit earlier
http://lists.xen.org/archives/html/xen-devel/2012-01/msg02408.html
I will test this patch as soon as possible. Thanks for your work.

Reported-by: William Dauchy <wdauchy@gmail.com>

Regards,
-- 
William

Re: ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[William Dauchy]

Hi Dan,

On Sat, Mar 24, 2012 at 12:55 AM, Dan Williams <dan.j.williams@intel.com> wrote:
> Thanks for the debug help, does this patch fix the issue for you?

I successfully tested your patch and it works fine. Thanks again for your work.

Reported-by: William Dauchy <wdauchy@gmail.com>
Tested-by: William Dauchy <wdauchy@gmail.com>

Best regards,
-- 
William

Re: ioatdma: Boot process hangs then reboots when using Xen + Linux 3.2

[Williams, Dan J]

On Fri, Mar 23, 2012 at 7:25 PM, William Dauchy <wdauchy@gmail.com> wrote:
> Hi Dan,
>
> On Sat, Mar 24, 2012 at 12:55 AM, Dan Williams <dan.j.williams@intel.com> wrote:
>> Thanks for the debug help, does this patch fix the issue for you?
>
> I successfully tested your patch and it works fine. Thanks again for your work.
>
> Reported-by: William Dauchy <wdauchy@gmail.com>
> Tested-by: William Dauchy <wdauchy@gmail.com>

Great, thanks for the test.

--
Dan