[RFC] algorithm for handling bad cachelines

[RFC] algorithm for handling bad cachelines

[Ben Widawsky]

I wanted to run this by folks before I start doing any actual work.

This is primarily for GPGPU, or perhaps *really* accurate rendering
requirements.

IVB+ has an interrupt to tell us when a cacheline seems to be going bad.
There is also a mechanism to remap the bad cachelines. The
implementation details aren't quite clear to me yet, but I'd like to
enable this feature for userspace.

Here is my current plan, but it involves filesystem access, so it's
probably going to get a lot of flames.

1. Handle cache line going bad interrupt.
<After n number of these interrupts to the same line,>
2. send a uevent
2.5 reset the GPU (docs tell us to)
<On module load>
3. Read  a module parameter with a path in the filesystem
of the list of bad lines. It's not clear to me yet exactly what I need
to store, but it should be a relatively simple list.
4. Parse list on driver load, and handle as necessary.
5. goto 1.

Probably the biggest unanswered question is exactly when in the HW
loading do we have to finish remapping. If it can happen at any time
while the card is running, I don't need the filesystem stuff, but I
believe I need to remap the lines quite early in the device bootstrap.

The only alternative I have is a huge comma separated string for a
module parameter, but I kind of like reading the file better.

Any feedback is highly appreciated. I couldn't really find much
precedent for doing this in other drivers, so pointers to similar
things would also be highly welcome.

Ben

Re: [RFC] algorithm for handling bad cachelines

[Chris Wilson]

On Tue, 27 Mar 2012 07:19:43 -0700, Ben Widawsky <ben@bwidawsk.net> wrote:
> Any feedback is highly appreciated. I couldn't really find much
> precedent for doing this in other drivers, so pointers to similar
> things would also be highly welcome.

badblocks and badram (memmap) both seem to be similar situations. A
comma-separated list works when the number is quite small, otherwise an
ioctl. Of course that requires us to be able to reprogram the cache on
the fly.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre

Re: [RFC] algorithm for handling bad cachelines

[Daniel Vetter]

On Tue, Mar 27, 2012 at 07:19:43AM -0700, Ben Widawsky wrote:
> I wanted to run this by folks before I start doing any actual work.
> 
> This is primarily for GPGPU, or perhaps *really* accurate rendering
> requirements.
> 
> IVB+ has an interrupt to tell us when a cacheline seems to be going bad.
> There is also a mechanism to remap the bad cachelines. The
> implementation details aren't quite clear to me yet, but I'd like to
> enable this feature for userspace.
> 
> Here is my current plan, but it involves filesystem access, so it's
> probably going to get a lot of flames.
> 
> 1. Handle cache line going bad interrupt.
> <After n number of these interrupts to the same line,>
> 2. send a uevent
> 2.5 reset the GPU (docs tell us to)
> <On module load>
> 3. Read  a module parameter with a path in the filesystem
> of the list of bad lines. It's not clear to me yet exactly what I need
> to store, but it should be a relatively simple list.

.... path in filesystem is no-go for kernel interface. So bad cachelines
need to go into the modele parameter itself. Or we add a sysfs interface
and reset the gpu (because if my understanding is right, we can't disable
cachelines once the gpu has used them).

> 4. Parse list on driver load, and handle as necessary.
> 5. goto 1.
> 
> Probably the biggest unanswered question is exactly when in the HW
> loading do we have to finish remapping. If it can happen at any time
> while the card is running, I don't need the filesystem stuff, but I
> believe I need to remap the lines quite early in the device bootstrap.

I believe so, too ;-)

> The only alternative I have is a huge comma separated string for a
> module parameter, but I kind of like reading the file better.

Well, you can't read a file from the kernel because we might init the
driver without any userspace present (when the driver is built-in).

> Any feedback is highly appreciated. I couldn't really find much
> precedent for doing this in other drivers, so pointers to similar
> things would also be highly welcome.
-Daniel
-- 
Daniel Vetter
Mail: daniel@ffwll.ch
Mobile: +41 (0)79 365 57 48

Re: [RFC] algorithm for handling bad cachelines

[Ben Widawsky]

On Tue, 27 Mar 2012 16:50:39 +0200
Daniel Vetter <daniel@ffwll.ch> wrote:

> On Tue, Mar 27, 2012 at 07:19:43AM -0700, Ben Widawsky wrote:
> > I wanted to run this by folks before I start doing any actual work.
> > 
> > This is primarily for GPGPU, or perhaps *really* accurate rendering
> > requirements.
> > 
> > IVB+ has an interrupt to tell us when a cacheline seems to be going
> > bad. There is also a mechanism to remap the bad cachelines. The
> > implementation details aren't quite clear to me yet, but I'd like to
> > enable this feature for userspace.
> > 
> > Here is my current plan, but it involves filesystem access, so it's
> > probably going to get a lot of flames.
> > 
> > 1. Handle cache line going bad interrupt.
> > <After n number of these interrupts to the same line,>
> > 2. send a uevent
> > 2.5 reset the GPU (docs tell us to)
> > <On module load>
> > 3. Read  a module parameter with a path in the filesystem
> > of the list of bad lines. It's not clear to me yet exactly what I
> > need to store, but it should be a relatively simple list.
> 
> .... path in filesystem is no-go for kernel interface. So bad
> cachelines need to go into the modele parameter itself. Or we add a
> sysfs interface and reset the gpu (because if my understanding is
> right, we can't disable cachelines once the gpu has used them).

I think we have to assume the list could get quite long. So long in
fact, I imagine the user may often want to reset it and try his/her
luck again with some lines.

Could you elaborate more on why it's a no-go? The module parameter
setting itself is limited to root. I was trying to clearly understand
exactly why this can't be done, and some of the lore behind why file
access in the kernel is such a bad thing (assuming the files being
accessed are set at module load time). I wouldn't want to go the route
of loading an arbitrary path - which seems like a terrible idea;
though it works for firmware blobs, and I half thought we could load
this like a firmware blob.

Anyway, assuming a gpu reset is sufficient to remap (docs only clearly
state reset works for disabling, iirc) then I would like to do that.
What is the appropriate interface for that? The dev node? Sysfs?

> 
> > 4. Parse list on driver load, and handle as necessary.
> > 5. goto 1.
> > 
> > Probably the biggest unanswered question is exactly when in the HW
> > loading do we have to finish remapping. If it can happen at any time
> > while the card is running, I don't need the filesystem stuff, but I
> > believe I need to remap the lines quite early in the device
> > bootstrap.
> 
> I believe so, too ;-)
> 
> > The only alternative I have is a huge comma separated string for a
> > module parameter, but I kind of like reading the file better.
> 
> Well, you can't read a file from the kernel because we might init the
> driver without any userspace present (when the driver is built-in).

Userspace should still be present in this case, right? The kernel
command line should suffice, I think.

> 
> > Any feedback is highly appreciated. I couldn't really find much
> > precedent for doing this in other drivers, so pointers to similar
> > things would also be highly welcome.
> -Daniel

Thanks

Re: [RFC] algorithm for handling bad cachelines

[Daniel Vetter]

On Tue, Mar 27, 2012 at 08:09:31AM -0700, Ben Widawsky wrote:
> On Tue, 27 Mar 2012 16:50:39 +0200
> Daniel Vetter <daniel@ffwll.ch> wrote:
> 
> > On Tue, Mar 27, 2012 at 07:19:43AM -0700, Ben Widawsky wrote:
> > > I wanted to run this by folks before I start doing any actual work.
> > > 
> > > This is primarily for GPGPU, or perhaps *really* accurate rendering
> > > requirements.
> > > 
> > > IVB+ has an interrupt to tell us when a cacheline seems to be going
> > > bad. There is also a mechanism to remap the bad cachelines. The
> > > implementation details aren't quite clear to me yet, but I'd like to
> > > enable this feature for userspace.
> > > 
> > > Here is my current plan, but it involves filesystem access, so it's
> > > probably going to get a lot of flames.
> > > 
> > > 1. Handle cache line going bad interrupt.
> > > <After n number of these interrupts to the same line,>
> > > 2. send a uevent
> > > 2.5 reset the GPU (docs tell us to)
> > > <On module load>
> > > 3. Read  a module parameter with a path in the filesystem
> > > of the list of bad lines. It's not clear to me yet exactly what I
> > > need to store, but it should be a relatively simple list.
> > 
> > .... path in filesystem is no-go for kernel interface. So bad
> > cachelines need to go into the modele parameter itself. Or we add a
> > sysfs interface and reset the gpu (because if my understanding is
> > right, we can't disable cachelines once the gpu has used them).
> 
> I think we have to assume the list could get quite long. So long in
> fact, I imagine the user may often want to reset it and try his/her
> luck again with some lines.
> 
> Could you elaborate more on why it's a no-go? The module parameter
> setting itself is limited to root. I was trying to clearly understand
> exactly why this can't be done, and some of the lore behind why file
> access in the kernel is such a bad thing (assuming the files being
> accessed are set at module load time). I wouldn't want to go the route
> of loading an arbitrary path - which seems like a terrible idea;
> though it works for firmware blobs, and I half thought we could load
> this like a firmware blob.
> 
> Anyway, assuming a gpu reset is sufficient to remap (docs only clearly
> state reset works for disabling, iirc) then I would like to do that.
> What is the appropriate interface for that? The dev node? Sysfs?

I personally prefer sysfs for this. Albeit you might have some issues with
the one value per file limit ... I guess a list of hex values is ok
though.

> > > 4. Parse list on driver load, and handle as necessary.
> > > 5. goto 1.
> > > 
> > > Probably the biggest unanswered question is exactly when in the HW
> > > loading do we have to finish remapping. If it can happen at any time
> > > while the card is running, I don't need the filesystem stuff, but I
> > > believe I need to remap the lines quite early in the device
> > > bootstrap.
> > 
> > I believe so, too ;-)
> > 
> > > The only alternative I have is a huge comma separated string for a
> > > module parameter, but I kind of like reading the file better.
> > 
> > Well, you can't read a file from the kernel because we might init the
> > driver without any userspace present (when the driver is built-in).
> 
> Userspace should still be present in this case, right? The kernel
> command line should suffice, I think.

Somewhen later on, but only after the hw is intialized. But if you're
going the runtime interface route anyway, it doesn't matter.
-Daniel
-- 
Daniel Vetter
Mail: daniel@ffwll.ch
Mobile: +41 (0)79 365 57 48

Re: [RFC] algorithm for handling bad cachelines

[Jesse Barnes]

[-- Attachment #1.1: Type: text/plain, Size: 2796 bytes --]

On Tue, 27 Mar 2012 07:19:43 -0700
Ben Widawsky <ben@bwidawsk.net> wrote:

> I wanted to run this by folks before I start doing any actual work.
> 
> This is primarily for GPGPU, or perhaps *really* accurate rendering
> requirements.
> 
> IVB+ has an interrupt to tell us when a cacheline seems to be going bad.
> There is also a mechanism to remap the bad cachelines. The
> implementation details aren't quite clear to me yet, but I'd like to
> enable this feature for userspace.
> 
> Here is my current plan, but it involves filesystem access, so it's
> probably going to get a lot of flames.
> 
> 1. Handle cache line going bad interrupt.
> <After n number of these interrupts to the same line,>
> 2. send a uevent
> 2.5 reset the GPU (docs tell us to)
> <On module load>
> 3. Read  a module parameter with a path in the filesystem
> of the list of bad lines. It's not clear to me yet exactly what I need
> to store, but it should be a relatively simple list.
> 4. Parse list on driver load, and handle as necessary.
> 5. goto 1.
> 
> Probably the biggest unanswered question is exactly when in the HW
> loading do we have to finish remapping. If it can happen at any time
> while the card is running, I don't need the filesystem stuff, but I
> believe I need to remap the lines quite early in the device bootstrap.
> 
> The only alternative I have is a huge comma separated string for a
> module parameter, but I kind of like reading the file better.
> 
> Any feedback is highly appreciated. I couldn't really find much
> precedent for doing this in other drivers, so pointers to similar
> things would also be highly welcome.

I think the main thing here is to make sure we handle the L3 parity
check interrupts.  I don't think "lines going bad" will be very common
in practice (maybe if you really abuse your CPU by putting it in the
freezer and then into an oven or something), so having a fancy
interface for it probably isn't too important.

Also, the behavior should be configurable.  For the vast majority of
users, an L3 parity interrupt is no big deal, and resetting the GPU
when we see one is more than we want.  So it should probably be off by
default and be controlled with a module parameter and maybe a config
option.

As for the interface for feeding in bad lines (useful for testing if
nothing else), I'd prefer an ioctl over a module parameter.  Some as yet
uncoded userspace service can load a set based on previously collected
uevents at boot time before any real GPU stuff runs...

Reading a file at load time is definitely a non-starter; we have no
idea whether the filesystem will be available to the module based on
mount points, hiding, chroots, etc.

-- 
Jesse Barnes, Intel Open Source Technology Center

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [RFC] algorithm for handling bad cachelines

[Ben Widawsky]

On Wed, 28 Mar 2012 10:26:52 -0700
Jesse Barnes <jbarnes@virtuousgeek.org> wrote:

> On Tue, 27 Mar 2012 07:19:43 -0700
> Ben Widawsky <ben@bwidawsk.net> wrote:
> 
> > I wanted to run this by folks before I start doing any actual work.
> > 
> > This is primarily for GPGPU, or perhaps *really* accurate rendering
> > requirements.
> > 
> > IVB+ has an interrupt to tell us when a cacheline seems to be going
> > bad. There is also a mechanism to remap the bad cachelines. The
> > implementation details aren't quite clear to me yet, but I'd like to
> > enable this feature for userspace.
> > 
> > Here is my current plan, but it involves filesystem access, so it's
> > probably going to get a lot of flames.
> > 
> > 1. Handle cache line going bad interrupt.
> > <After n number of these interrupts to the same line,>
> > 2. send a uevent
> > 2.5 reset the GPU (docs tell us to)
> > <On module load>
> > 3. Read  a module parameter with a path in the filesystem
> > of the list of bad lines. It's not clear to me yet exactly what I
> > need to store, but it should be a relatively simple list.
> > 4. Parse list on driver load, and handle as necessary.
> > 5. goto 1.
> > 
> > Probably the biggest unanswered question is exactly when in the HW
> > loading do we have to finish remapping. If it can happen at any time
> > while the card is running, I don't need the filesystem stuff, but I
> > believe I need to remap the lines quite early in the device
> > bootstrap.
> > 
> > The only alternative I have is a huge comma separated string for a
> > module parameter, but I kind of like reading the file better.
> > 
> > Any feedback is highly appreciated. I couldn't really find much
> > precedent for doing this in other drivers, so pointers to similar
> > things would also be highly welcome.
> 
> I think the main thing here is to make sure we handle the L3 parity
> check interrupts.  I don't think "lines going bad" will be very common
> in practice (maybe if you really abuse your CPU by putting it in the
> freezer and then into an oven or something), so having a fancy
> interface for it probably isn't too important.

I'd like to clarify my statement a bit. Thanks for pointing this out.
I've yet to enable this interrupt and see what happens, so this is all
speculative based on that document:

It seems like more lines may be susepitble to parity errors via cosmic
rays. Since upon learning this information requires a GPU reset (to
Andi's point which I'll address seperately), I think it's desirable for
users to be able to choose early on whether or not the lines ever get
enabled. For the graphics case however, learning the information every
time is totally acceptible, and as you say below is probably a nop.

> 
> Also, the behavior should be configurable.  For the vast majority of
> users, an L3 parity interrupt is no big deal, and resetting the GPU
> when we see one is more than we want.  So it should probably be off by
> default and be controlled with a module parameter and maybe a config
> option.
> 
> As for the interface for feeding in bad lines (useful for testing if
> nothing else), I'd prefer an ioctl over a module parameter.  Some as
> yet uncoded userspace service can load a set based on previously
> collected uevents at boot time before any real GPU stuff runs...

It's still not clear to me if we can remap after the GPU is fully up
and running and handling the IOCTLs. If so, I believe that is
definitely the consesus.

> 
> Reading a file at load time is definitely a non-starter; we have no
> idea whether the filesystem will be available to the module based on
> mount points, hiding, chroots, etc.
> 

Re: [RFC] algorithm for handling bad cachelines

[Ben Widawsky]

On Wed, 28 Mar 2012 02:59:26 -0700
Andi Kleen <andi@firstfloor.org> wrote:

> Ben Widawsky <ben@bwidawsk.net> writes:
> >
> > 1. Handle cache line going bad interrupt.
> > <After n number of these interrupts to the same line,>
> 
> Never use global n without timeout for corrected errors, you would 
> need a leaky bucket with a suitable timeout.

As I understand electrons (which is not very well) parity errors happen
all the time and are transparently corrected by our HW. So I suppose
'n' is still interesting information, but your point is noted. It is
probably better to let userspace decide that n value.

Take this with a grain of salt because the number of interrupts we get
is speculative as I haven't actually tried to enable this.

> 
> > 2. send a uevent
> > 2.5 reset the GPU (docs tell us to)
> > <On module load>
> 
> Persistent lists on disk usually suffer from all kinds of problems,
> e.g. you need to detect when the board or CPU has changed.
> Also when the problem is temporary you do not really want
> to save such information permanent.
> 
> Usually it's better to rediscover such state each time and handle
> it again. Then you also don't need the uevent or complicated
> user interfaces.

It seems nice to have information stored non-volatility. It doesn't have
to be used by the user, but assuming they want to load the option to
actually detect these events, it's probably also beneficial to give the
known bad cachelines since this requires a GPU reset once detected. The
reset both takes time, and may do more damage (that is based on past
experience/products only and I hope IVB can magnificently recover from
our bad GPU programming).

> 
> > Any feedback is highly appreciated. I couldn't really find much
> > precedent for doing this in other drivers, so pointers to similar
> > things would also be highly welcome.
> 
> http://mcelog.org
> 
> -Andi

Thanks.