From: Arnd Bergmann <arnd@arndb.de>
To: Mikulas Patocka <mpatocka@redhat.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>,
Richard.Earnshaw@arm.com,
Thomas Petazzoni <thomas.petazzoni@free-electrons.com>,
Joao Pinto <Joao.Pinto@synopsys.com>,
GNU C Library <libc-alpha@sourceware.org>,
Ard Biesheuvel <ard.biesheuvel@linaro.org>,
Jingoo Han <jingoohan1@gmail.com>,
Will Deacon <will.deacon@arm.com>,
Russell King - ARM Linux <linux@armlinux.org.uk>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
neko@bakuhatsu.net, linux-pci <linux-pci@vger.kernel.org>,
Linux ARM <linux-arm-kernel@lists.infradead.org>
Subject: Re: framebuffer corruption due to overlapping stp instructions on arm64
Date: Wed, 8 Aug 2018 23:51:53 +0200 [thread overview]
Message-ID: <CAK8P3a1CdL5ZVvWPMriOknTCgkfbR4F-a6dRZ7ap9+NiD8H5cg@mail.gmail.com> (raw)
In-Reply-To: <alpine.LRH.2.02.1808081347180.18198@file01.intranet.prod.int.rdu2.redhat.com>
On Wed, Aug 8, 2018 at 8:25 PM Mikulas Patocka <mpatocka@redhat.com> wrote:
> On Wed, 8 Aug 2018, Arnd Bergmann wrote:
>
> > On Wed, Aug 8, 2018 at 5:15 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > >
> > > On Wed, Aug 08, 2018 at 04:01:12PM +0100, Richard Earnshaw wrote:
> > > > On 08/08/18 15:12, Mikulas Patocka wrote:
> > > > > On Wed, 8 Aug 2018, Catalin Marinas wrote:
> > > > >> On Fri, Aug 03, 2018 at 01:09:02PM -0400, Mikulas Patocka wrote:
> > > - failing to write a few bytes
> > > - writing a few bytes that were written 16 bytes before
> > > - writing a few bytes that were written 16 bytes after
> > >
> > > > The overlapping writes in memcpy never write different values to the
> > > > same location, so I still feel this must be some sort of HW issue, not a
> > > > SW one.
> > >
> > > So do I (my interpretation is that it combines or rather skips some of
> > > the writes to the same 16-byte address as it ignores the data strobes).
> >
> > Maybe it just always writes to the wrong location, 16 bytes apart for one of
> > the stp instructions. Since we are usually dealing with a pair of overlapping
> > 'stp', both unaligned, that could explain both the missing bytes (we write
> > data to the wrong place, but overwrite it with the correct data right away)
> > and the extra copy (we write it to the wrong place, but then write the correct
> > data to the correct place as well).
> >
> > This sounds a bit like what the original ARM CPUs did on unaligned
> > memory access, where a single aligned 4-byte location was accessed,
> > but the bytes swapped around.
> >
> > There may be a few more things worth trying out or analysing from
> > the recorded past failures to understand more about how it goes
> > wrong:
> >
> > - For which data lengths does it fail? Having two overlapping
> > unaligned stp is something that only happens for 16..96 byte
> > memcpy.
>
> If you want to research the corruptions in detail, I uploaded a file
> containing 7k corruptions here:
> http://people.redhat.com/~mpatocka/testcases/arm-pcie-corruption/
Nice!
I already found a couple of things:
- Failure to copy always happens at the *end* of a 16 byte aligned
physical address, it misses between 1 and 6 bytes, never 7 or more,
and it's more likely to be fewer bytes that are affected.
279 7
389 6
484 5
683 4
741 3
836 2
946 1
- The first byte that fails to get copied is always 16 bytes after the
memcpy target. Since we only observe it at the end of the 16 byte
range, it means this happens specifically for addresses ending in
0x9 (7 bytes missed) to 0xf (1 byte missed).
- Out of 7445 corruptions, 4358 were of the kind that misses a copy at the
end of a 16-byte area, they were for copies between 41 and 64 bytes,
more to the larger end of the scale (note that with your test program,
smaller memcpys happen more frequenly than larger ones).
47 0x29
36 0x2a
47 0x2b
23 0x2c
29 0x2d
31 0x2e
36 0x2f
46 0x30
45 0x31
51 0x32
62 0x33
64 0x34
77 0x35
91 0x36
90 0x37
100 0x38
100 0x39
209 0x3a
279 0x3b
366 0x3c
498 0x3d
602 0x3e
682 0x3f
747 0x40
- All corruption with data copied to the wrong place happened for copies
between 33 and 47 bytes, mostly to the smaller end of the scale:
391 0x21
360 0x22
319 0x23
273 0x24
273 0x25
241 0x26
224 0x27
221 0x28
231 0x29
208 0x2a
163 0x2b
86 0x2c
63 0x2d
33 0x2e
1 0x2f
- One common (but not the only, still investigating) case for data getting
written to the wrong place is:
* corruption starts 16 bytes after the memcpy start
* corrupt bytes are the same as the bytes written to the start
* start address ends in 0x1 through 0x7
* length of corruption is at most memcpy length- 32, always
between 1 and 7.
Arnd
next prev parent reply other threads:[~2018-08-08 21:52 UTC|newest]
Thread overview: 95+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-08-02 19:31 framebuffer corruption due to overlapping stp instructions on arm64 Mikulas Patocka
[not found] ` <CAHCPf3tFGqkYEcWNN4LaWThw_rVqT316pzLv6T7RfxwO-eZ0EA@mail.gmail.com>
2018-08-03 6:35 ` Mikulas Patocka
2018-08-03 7:16 ` Ard Biesheuvel
2018-08-03 9:41 ` Will Deacon
2018-08-03 17:09 ` Mikulas Patocka
2018-08-03 17:32 ` Sinan Kaya
2018-08-03 17:33 ` Ard Biesheuvel
2018-08-03 18:25 ` Mikulas Patocka
2018-08-03 20:44 ` Matt Sealey
2018-08-03 21:20 ` Ard Biesheuvel
2018-08-06 10:25 ` Mikulas Patocka
2018-08-06 12:42 ` Robin Murphy
2018-08-06 12:53 ` Ard Biesheuvel
2018-08-06 13:41 ` Marcin Wojtas
2018-08-06 13:48 ` Ard Biesheuvel
2018-08-06 14:07 ` Marcin Wojtas
2018-08-06 14:13 ` Mikulas Patocka
2018-08-06 15:47 ` Ard Biesheuvel
2018-08-06 17:09 ` Mikulas Patocka
2018-08-06 17:21 ` Ard Biesheuvel
2018-08-06 19:54 ` Mikulas Patocka
2018-08-06 20:11 ` Ard Biesheuvel
2018-08-06 20:31 ` Mikulas Patocka
2018-08-07 16:40 ` Marcin Wojtas
2018-08-07 17:39 ` Mikulas Patocka
2018-08-07 18:07 ` Ard Biesheuvel
2018-08-07 18:17 ` Mikulas Patocka
[not found] ` <CAPv3WKcKoEe=Qysp6Oac2C=G9bUhUQf1twSRCY+_qJ6XEC-iag@mail.gmail.com>
2018-08-08 14:10 ` Mikulas Patocka
2018-08-06 17:13 ` Catalin Marinas
2018-08-06 17:19 ` Mikulas Patocka
2018-08-08 18:31 ` Mikulas Patocka
2018-08-04 13:29 ` Mikulas Patocka
2018-08-08 12:16 ` Catalin Marinas
2018-08-08 13:02 ` David Laight
2018-08-08 13:46 ` Mikulas Patocka
2018-08-08 14:26 ` David Laight
2018-08-08 14:50 ` Catalin Marinas
2018-08-08 16:21 ` Mikulas Patocka
2018-08-08 16:31 ` Arnd Bergmann
2018-08-08 16:43 ` David Laight
2018-08-08 18:56 ` Mikulas Patocka
2018-08-08 18:37 ` Mikulas Patocka
2018-08-08 11:39 ` Catalin Marinas
2018-08-08 14:12 ` Mikulas Patocka
2018-08-08 14:28 ` Catalin Marinas
2018-08-08 18:40 ` Mikulas Patocka
2018-08-08 15:01 ` Richard Earnshaw (lists)
2018-08-08 15:14 ` Catalin Marinas
2018-08-08 16:01 ` Arnd Bergmann
2018-08-08 18:25 ` Mikulas Patocka
2018-08-08 21:51 ` Arnd Bergmann [this message]
2018-08-09 15:29 ` Arnd Bergmann
2018-08-03 7:11 ` Andrew Pinski
2018-08-03 7:53 ` Florian Weimer
2018-08-03 9:12 ` Szabolcs Nagy
2018-08-03 9:15 ` Ramana Radhakrishnan
2018-08-03 9:29 ` Ard Biesheuvel
2018-08-03 9:37 ` Ramana Radhakrishnan
2018-08-03 9:42 ` Richard Earnshaw (lists)
2018-08-04 0:58 ` Mikulas Patocka
2018-08-04 1:13 ` Andrew Pinski
2018-08-04 11:04 ` Mikulas Patocka
2018-08-05 18:33 ` Florian Weimer
2018-08-06 8:02 ` Mikulas Patocka
2018-08-06 8:10 ` Ard Biesheuvel
2018-08-06 10:31 ` Mikulas Patocka
2018-08-06 10:37 ` Ard Biesheuvel
2018-08-06 10:42 ` Mikulas Patocka
2018-08-06 10:48 ` Ard Biesheuvel
2018-08-06 12:09 ` Mikulas Patocka
2018-08-06 12:19 ` Ard Biesheuvel
2018-08-06 12:22 ` Ard Biesheuvel
2018-08-07 14:14 ` Mikulas Patocka
2018-08-07 14:40 ` Ard Biesheuvel
2018-08-08 19:15 ` Mikulas Patocka
2018-08-06 11:19 ` Siddhesh Poyarekar
2018-08-06 11:29 ` Ard Biesheuvel
2018-08-06 14:26 ` Tulio Magno Quites Machado Filho
2018-08-05 21:51 ` Pavel Machek
2018-08-06 14:30 ` Mikulas Patocka
2018-08-03 11:24 ` David Laight
2018-08-03 12:04 ` Mikulas Patocka
2018-08-03 13:04 ` David Laight
2018-08-05 14:36 ` Mikulas Patocka
2018-08-06 10:18 ` David Laight
2018-08-07 14:07 ` Mikulas Patocka
2018-08-07 14:33 ` David Laight
2018-08-08 14:21 ` Mikulas Patocka
2018-08-03 13:20 ` Mikulas Patocka
2018-08-03 13:31 ` Mikulas Patocka
2018-08-03 14:17 ` Richard Earnshaw (lists)
2018-08-05 21:36 ` Pavel Machek
2018-08-06 8:04 ` Ramana Radhakrishnan
2018-08-06 8:44 ` Pavel Machek
2018-08-06 9:11 ` Ard Biesheuvel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAK8P3a1CdL5ZVvWPMriOknTCgkfbR4F-a6dRZ7ap9+NiD8H5cg@mail.gmail.com \
--to=arnd@arndb.de \
--cc=Joao.Pinto@synopsys.com \
--cc=Richard.Earnshaw@arm.com \
--cc=ard.biesheuvel@linaro.org \
--cc=catalin.marinas@arm.com \
--cc=jingoohan1@gmail.com \
--cc=libc-alpha@sourceware.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux@armlinux.org.uk \
--cc=mpatocka@redhat.com \
--cc=neko@bakuhatsu.net \
--cc=thomas.petazzoni@free-electrons.com \
--cc=will.deacon@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).