Re: pin OABUFFER to GGTT

Re: pin OABUFFER to GGTT

[Mateo Lozano, Oscar]

Submitting again (this time copying the mailing list correctly):

The bo_pin ioctl has been discarded in GEN6+ with this patch:

    drm/i915: Reject the pin ioctl on gen6+
    
    Especially with ppgtt this kinda stopped making sense. And if we
    indeed need this to hack around an issue, we need something that also
    works for non-root.
    
    Signed-off-by: Daniel Vetter daniel.vetter@ffwll.ch

The thing is, the performance team used this call to pin the OABUFFER to GGTT and then mapping it to userspace. This OABUFFER cannot be in PPGTT because: "When each context has its own Per Process GTT, this field should be always set to GGTT." (BSpec dixit).

Can we re-enable it? or should we find an alternative for this case?

-- Oscar

Re: pin OABUFFER to GGTT

[Chris Wilson]

On Tue, Jul 01, 2014 at 04:24:56PM +0000, Mateo Lozano, Oscar wrote:
> Submitting again (this time copying the mailing list correctly):
> 
> The bo_pin ioctl has been discarded in GEN6+ with this patch:
> 
>     drm/i915: Reject the pin ioctl on gen6+
>     
>     Especially with ppgtt this kinda stopped making sense. And if we
>     indeed need this to hack around an issue, we need something that also
>     works for non-root.
>     
>     Signed-off-by: Daniel Vetter daniel.vetter@ffwll.ch
> 
> The thing is, the performance team used this call to pin the OABUFFER to GGTT and then mapping it to userspace. This OABUFFER cannot be in PPGTT because: "When each context has its own Per Process GTT, this field should be always set to GGTT." (BSpec dixit).
> 
> Can we re-enable it? or should we find an alternative for this case?

EXEC_OBJECT_NEEDS_GTT?
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre

Re: pin OABUFFER to GGTT

[Mateo Lozano, Oscar]

> -----Original Message-----
> From: Chris Wilson [mailto:chris@chris-wilson.co.uk]
> Sent: Tuesday, July 01, 2014 5:30 PM
> To: Mateo Lozano, Oscar
> Cc: Intel-gfx@lists.freedesktop.org; Madajczak, Tomasz
> Subject: Re: [Intel-gfx] pin OABUFFER to GGTT
> 
> On Tue, Jul 01, 2014 at 04:24:56PM +0000, Mateo Lozano, Oscar wrote:
> > Submitting again (this time copying the mailing list correctly):
> >
> > The bo_pin ioctl has been discarded in GEN6+ with this patch:
> >
> >     drm/i915: Reject the pin ioctl on gen6+
> >
> >     Especially with ppgtt this kinda stopped making sense. And if we
> >     indeed need this to hack around an issue, we need something that also
> >     works for non-root.
> >
> >     Signed-off-by: Daniel Vetter daniel.vetter@ffwll.ch
> >
> > The thing is, the performance team used this call to pin the OABUFFER to
> GGTT and then mapping it to userspace. This OABUFFER cannot be in PPGTT
> because: "When each context has its own Per Process GTT, this field should
> be always set to GGTT." (BSpec dixit).
> >
> > Can we re-enable it? or should we find an alternative for this case?
> 
> EXEC_OBJECT_NEEDS_GTT?
> -Chris

The object (AFAICT, please Tomasz correct me if I am wrong) is not really used inside any batchbuffer.
Also:

		if (exec[i].flags & EXEC_OBJECT_NEEDS_GTT &&
		    USES_FULL_PPGTT(vm->dev)) {
			ret = -EINVAL;
			goto err;
		}

Re: pin OABUFFER to GGTT

[Chris Wilson]

On Tue, Jul 01, 2014 at 04:34:48PM +0000, Mateo Lozano, Oscar wrote:
> > -----Original Message-----
> > From: Chris Wilson [mailto:chris@chris-wilson.co.uk]
> > Sent: Tuesday, July 01, 2014 5:30 PM
> > To: Mateo Lozano, Oscar
> > Cc: Intel-gfx@lists.freedesktop.org; Madajczak, Tomasz
> > Subject: Re: [Intel-gfx] pin OABUFFER to GGTT
> > 
> > On Tue, Jul 01, 2014 at 04:24:56PM +0000, Mateo Lozano, Oscar wrote:
> > > Submitting again (this time copying the mailing list correctly):
> > >
> > > The bo_pin ioctl has been discarded in GEN6+ with this patch:
> > >
> > >     drm/i915: Reject the pin ioctl on gen6+
> > >
> > >     Especially with ppgtt this kinda stopped making sense. And if we
> > >     indeed need this to hack around an issue, we need something that also
> > >     works for non-root.
> > >
> > >     Signed-off-by: Daniel Vetter daniel.vetter@ffwll.ch
> > >
> > > The thing is, the performance team used this call to pin the OABUFFER to
> > GGTT and then mapping it to userspace. This OABUFFER cannot be in PPGTT
> > because: "When each context has its own Per Process GTT, this field should
> > be always set to GGTT." (BSpec dixit).
> > >
> > > Can we re-enable it? or should we find an alternative for this case?
> > 
> > EXEC_OBJECT_NEEDS_GTT?
> > -Chris
> 
> The object (AFAICT, please Tomasz correct me if I am wrong) is not really used inside any batchbuffer.

Then what's the issue? If you only use it as via a global gtt mapping it
only exists in the ggtt.

> Also:
> 
> 		if (exec[i].flags & EXEC_OBJECT_NEEDS_GTT &&
> 		    USES_FULL_PPGTT(vm->dev)) {
> 			ret = -EINVAL;
> 			goto err;
> 		}
> 

Yeah, that's just full-ppgtt not quite being ready yet.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre

Re: pin OABUFFER to GGTT

[Mateo Lozano, Oscar]

> -----Original Message-----
> From: Chris Wilson [mailto:chris@chris-wilson.co.uk]
> Sent: Tuesday, July 01, 2014 5:52 PM
> To: Mateo Lozano, Oscar
> Cc: Intel-gfx@lists.freedesktop.org; Madajczak, Tomasz; Rutkowski, Adam J;
> Jesse Barnes (jbarnes@virtuousgeek.org)
> Subject: Re: [Intel-gfx] pin OABUFFER to GGTT
> 
> On Tue, Jul 01, 2014 at 04:34:48PM +0000, Mateo Lozano, Oscar wrote:
> > > -----Original Message-----
> > > From: Chris Wilson [mailto:chris@chris-wilson.co.uk]
> > > Sent: Tuesday, July 01, 2014 5:30 PM
> > > To: Mateo Lozano, Oscar
> > > Cc: Intel-gfx@lists.freedesktop.org; Madajczak, Tomasz
> > > Subject: Re: [Intel-gfx] pin OABUFFER to GGTT
> > >
> > > On Tue, Jul 01, 2014 at 04:24:56PM +0000, Mateo Lozano, Oscar wrote:
> > > > Submitting again (this time copying the mailing list correctly):
> > > >
> > > > The bo_pin ioctl has been discarded in GEN6+ with this patch:
> > > >
> > > >     drm/i915: Reject the pin ioctl on gen6+
> > > >
> > > >     Especially with ppgtt this kinda stopped making sense. And if
> > > > we
> > > >     indeed need this to hack around an issue, we need something
> > > > that also
> > > >     works for non-root.
> > > >
> > > >     Signed-off-by: Daniel Vetter daniel.vetter@ffwll.ch
> > > >
> > > > The thing is, the performance team used this call to pin the
> > > > OABUFFER to
> > > GGTT and then mapping it to userspace. This OABUFFER cannot be in
> > > PPGTT
> > > because: "When each context has its own Per Process GTT, this field
> > > should be always set to GGTT." (BSpec dixit).
> > > >
> > > > Can we re-enable it? or should we find an alternative for this case?
> > >
> > > EXEC_OBJECT_NEEDS_GTT?
> > > -Chris
> >
> > The object (AFAICT, please Tomasz correct me if I am wrong) is not really
> used inside any batchbuffer.
> 
> Then what's the issue? If you only use it as via a global gtt mapping it only
> exists in the ggtt.

The issue is they need:

A) A buffer object.
B) Bound to GGTT.
C) That userspace knows the GGTT offset of, so that they can program OABUFFER with it.
D) That userspace can map so that they can read the reported counters.

They used to create a bo, call bo_pin on it, use args->offset to program OABUFFER (via MI_LOAD_REGISTER_IMM, I imagine), map it and read the counter values. They cannot do this anymore.

Re: pin OABUFFER to GGTT

[Mateo Lozano, Oscar]

> -----Original Message-----
> From: Mateo Lozano, Oscar
> Sent: Tuesday, July 01, 2014 6:14 PM
> To: 'Chris Wilson'
> Cc: Intel-gfx@lists.freedesktop.org; Madajczak, Tomasz; Rutkowski, Adam J;
> Jesse Barnes (jbarnes@virtuousgeek.org)
> Subject: RE: [Intel-gfx] pin OABUFFER to GGTT
> 
> > -----Original Message-----
> > From: Chris Wilson [mailto:chris@chris-wilson.co.uk]
> > Sent: Tuesday, July 01, 2014 5:52 PM
> > To: Mateo Lozano, Oscar
> > Cc: Intel-gfx@lists.freedesktop.org; Madajczak, Tomasz; Rutkowski,
> > Adam J; Jesse Barnes (jbarnes@virtuousgeek.org)
> > Subject: Re: [Intel-gfx] pin OABUFFER to GGTT
> >
> > On Tue, Jul 01, 2014 at 04:34:48PM +0000, Mateo Lozano, Oscar wrote:
> > > > -----Original Message-----
> > > > From: Chris Wilson [mailto:chris@chris-wilson.co.uk]
> > > > Sent: Tuesday, July 01, 2014 5:30 PM
> > > > To: Mateo Lozano, Oscar
> > > > Cc: Intel-gfx@lists.freedesktop.org; Madajczak, Tomasz
> > > > Subject: Re: [Intel-gfx] pin OABUFFER to GGTT
> > > >
> > > > On Tue, Jul 01, 2014 at 04:24:56PM +0000, Mateo Lozano, Oscar
> wrote:
> > > > > Submitting again (this time copying the mailing list correctly):
> > > > >
> > > > > The bo_pin ioctl has been discarded in GEN6+ with this patch:
> > > > >
> > > > >     drm/i915: Reject the pin ioctl on gen6+
> > > > >
> > > > >     Especially with ppgtt this kinda stopped making sense. And
> > > > > if we
> > > > >     indeed need this to hack around an issue, we need something
> > > > > that also
> > > > >     works for non-root.
> > > > >
> > > > >     Signed-off-by: Daniel Vetter daniel.vetter@ffwll.ch
> > > > >
> > > > > The thing is, the performance team used this call to pin the
> > > > > OABUFFER to
> > > > GGTT and then mapping it to userspace. This OABUFFER cannot be in
> > > > PPGTT
> > > > because: "When each context has its own Per Process GTT, this
> > > > field should be always set to GGTT." (BSpec dixit).
> > > > >
> > > > > Can we re-enable it? or should we find an alternative for this case?
> > > >
> > > > EXEC_OBJECT_NEEDS_GTT?
> > > > -Chris
> > >
> > > The object (AFAICT, please Tomasz correct me if I am wrong) is not
> > > really
> > used inside any batchbuffer.
> >
> > Then what's the issue? If you only use it as via a global gtt mapping
> > it only exists in the ggtt.
> 
> The issue is they need:
> 
> A) A buffer object.
> B) Bound to GGTT.
> C) That userspace knows the GGTT offset of, so that they can program
> OABUFFER with it.
> D) That userspace can map so that they can read the reported counters.
> 
> They used to create a bo, call bo_pin on it, use args->offset to program
> OABUFFER (via MI_LOAD_REGISTER_IMM, I imagine), map it and read the
> counter values. They cannot do this anymore.

The answer might be that all of this needs to be done by the kernel itself, but then we need to provide an interface to userspace...

Re: pin OABUFFER to GGTT

[Chris Wilson]

On Tue, Jul 01, 2014 at 05:16:30PM +0000, Mateo Lozano, Oscar wrote:
> > The issue is they need:
> > 
> > A) A buffer object.
> > B) Bound to GGTT.
> > C) That userspace knows the GGTT offset of, so that they can program
> > OABUFFER with it.
> > D) That userspace can map so that they can read the reported counters.
> > 
> > They used to create a bo, call bo_pin on it, use args->offset to program
> > OABUFFER (via MI_LOAD_REGISTER_IMM, I imagine), map it and read the
> > counter values. They cannot do this anymore.
> 
> The answer might be that all of this needs to be done by the kernel itself, but then we need to provide an interface to userspace...

Yes. If you need to pin a buffer for a register, then it needs to be
handled by the kernel. Especially one that provides information about
other users.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre

Re: pin OABUFFER to GGTT

[Ben Widawsky]

On Tue, Jul 01, 2014 at 08:54:27PM +0100, Chris Wilson wrote:
> On Tue, Jul 01, 2014 at 05:16:30PM +0000, Mateo Lozano, Oscar wrote:
> > > The issue is they need:
> > > 
> > > A) A buffer object.
> > > B) Bound to GGTT.
> > > C) That userspace knows the GGTT offset of, so that they can program
> > > OABUFFER with it.
> > > D) That userspace can map so that they can read the reported counters.
> > > 
> > > They used to create a bo, call bo_pin on it, use args->offset to program
> > > OABUFFER (via MI_LOAD_REGISTER_IMM, I imagine), map it and read the
> > > counter values. They cannot do this anymore.
> > 
> > The answer might be that all of this needs to be done by the kernel itself, but then we need to provide an interface to userspace...
> 
> Yes. If you need to pin a buffer for a register, then it needs to be
> handled by the kernel. Especially one that provides information about
> other users.
> -Chris
> 

I'm unclear why they need the offset. Can it not work like any other
relocation, except with the requirement that it's in the GGTT?

I'd also argue that they need to be able to map it (they need the
contents, which may or may not be mapped). However, I think this is a
very minor point.

With the command validator this should be a pretty reasonable thing to
accomplish. I think we can just give a flag for the reloc that it needs
to be in the GGTT, and then pass a check to the command scanner that
only that one offset is allowed, and only for OA.

-- 
Ben Widawsky, Intel Open Source Technology Center

Re: pin OABUFFER to GGTT

[Chris Wilson]

On Tue, Jul 01, 2014 at 11:40:52PM -0700, Ben Widawsky wrote:
> On Tue, Jul 01, 2014 at 08:54:27PM +0100, Chris Wilson wrote:
> > On Tue, Jul 01, 2014 at 05:16:30PM +0000, Mateo Lozano, Oscar wrote:
> > > > The issue is they need:
> > > > 
> > > > A) A buffer object.
> > > > B) Bound to GGTT.
> > > > C) That userspace knows the GGTT offset of, so that they can program
> > > > OABUFFER with it.
> > > > D) That userspace can map so that they can read the reported counters.
> > > > 
> > > > They used to create a bo, call bo_pin on it, use args->offset to program
> > > > OABUFFER (via MI_LOAD_REGISTER_IMM, I imagine), map it and read the
> > > > counter values. They cannot do this anymore.
> > > 
> > > The answer might be that all of this needs to be done by the kernel itself, but then we need to provide an interface to userspace...
> > 
> > Yes. If you need to pin a buffer for a register, then it needs to be
> > handled by the kernel. Especially one that provides information about
> > other users.
> > -Chris
> > 
> 
> I'm unclear why they need the offset. Can it not work like any other
> relocation, except with the requirement that it's in the GGTT?
> 
> I'd also argue that they need to be able to map it (they need the
> contents, which may or may not be mapped). However, I think this is a
> very minor point.
> 
> With the command validator this should be a pretty reasonable thing to
> accomplish. I think we can just give a flag for the reloc that it needs
> to be in the GGTT, and then pass a check to the command scanner that
> only that one offset is allowed, and only for OA.

If the intention was to only measure within a batch and the command
parser ensured that the register was cleared before the end of the
batch, fine. That's not an information leak nor do we keep the hardware
pointing to an unpinned buffer afterwards.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre

Re: pin OABUFFER to GGTT

[Mateo Lozano, Oscar]

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

> -----Original Message-----
> From: Chris Wilson [mailto:chris@chris-wilson.co.uk]
> Sent: Wednesday, July 02, 2014 7:56 AM
> To: Ben Widawsky
> Cc: Mateo Lozano, Oscar; Intel-gfx@lists.freedesktop.org; Madajczak,
> Tomasz; Rutkowski, Adam J; Jesse Barnes (jbarnes@virtuousgeek.org)
> Subject: Re: [Intel-gfx] pin OABUFFER to GGTT
> 
> On Tue, Jul 01, 2014 at 11:40:52PM -0700, Ben Widawsky wrote:
> > On Tue, Jul 01, 2014 at 08:54:27PM +0100, Chris Wilson wrote:
> > > On Tue, Jul 01, 2014 at 05:16:30PM +0000, Mateo Lozano, Oscar wrote:
> > > > > The issue is they need:
> > > > >
> > > > > A) A buffer object.
> > > > > B) Bound to GGTT.
> > > > > C) That userspace knows the GGTT offset of, so that they can
> > > > > program OABUFFER with it.
> > > > > D) That userspace can map so that they can read the reported
> counters.
> > > > >
> > > > > They used to create a bo, call bo_pin on it, use args->offset to
> > > > > program OABUFFER (via MI_LOAD_REGISTER_IMM, I imagine), map it
> > > > > and read the counter values. They cannot do this anymore.
> > > >
> > > > The answer might be that all of this needs to be done by the kernel
> itself, but then we need to provide an interface to userspace...
> > >
> > > Yes. If you need to pin a buffer for a register, then it needs to be
> > > handled by the kernel. Especially one that provides information
> > > about other users.
> > > -Chris
> > >
> >
> > I'm unclear why they need the offset. Can it not work like any other
> > relocation, except with the requirement that it's in the GGTT?
> >
> > I'd also argue that they need to be able to map it (they need the
> > contents, which may or may not be mapped). However, I think this is a
> > very minor point.
> >
> > With the command validator this should be a pretty reasonable thing to
> > accomplish. I think we can just give a flag for the reloc that it
> > needs to be in the GGTT, and then pass a check to the command scanner
> > that only that one offset is allowed, and only for OA.
> 
> If the intention was to only measure within a batch and the command parser
> ensured that the register was cleared before the end of the batch, fine.
> That's not an information leak nor do we keep the hardware pointing to an
> unpinned buffer afterwards.
> Chris Wilson, Intel Open Source Technology Centre

Tomasz: is the intention to only measure within a batch? My impression is that you wanted to maintain the OABUFFER programmed and then collect performance reports for a period of time (probably for several batchbuffers). If that´s the case, the relocation approach is not possible.

Chris: please notice that the bo_pin ioctl they were using until now required root, so the information leak about other users was not that bad. Still, I tend to agree that bo_pin for this is overkill and the programming of OABUFFER should be carried out by the kernel (with some interface to userspace so that they can retrieve the actual contents of the buffer... root only of course). 

-- Oscar

Re: pin OABUFFER to GGTT

[Madajczak, Tomasz]

[-- Attachment #1.1: Type: text/plain, Size: 5040 bytes --]

Ad>Tomasz: is the intention to only measure within a batch? My impression is
that you wanted to maintain the OABUFFER programmed and then collect
performance reports for a period of time (probably for several
batchbuffers). If that´s the case, the relocation approach is not possible.

I confirm that relocation approach is not possible.

OA buffer collect performance data for many hw contexts, and thus many batch
buffers, it is a hardware reporting mechanism not bound to any specific GPU
command send within a BB.

The report is triggered by hw when:
- hw context switches, that action can be caused by any level GPU
preemption, GUC direct context submission, and regular kernel exec list
submission
- GPU frequency change - that is trigger by power management
- RC6 transition
- time interval expiration (if programmed so for collect performance data at
regular time intervals)

Each instance of UMD maps then the OA buffer to examine it within the
Performance Query measurement window. Window is learned with Query::Begin
and Query::End timestamps and reported OABufferTail pointer on Query::End.
As OA buffer is large (up to 16MB), any KMD assist in conveying that data to
UMD on each Query::End is not applicable - queries might be done per each
Draw. There isn't any secret or privacy in that OA buffer data - just
results of performance counters, shown by tools such as GPA/ Vtune.

Thanks,
Tomasz






-----Original Message-----
From: Mateo Lozano, Oscar 
Sent: Wednesday, July 02, 2014 10:50 AM
To: Chris Wilson; Ben Widawsky
Cc: Intel-gfx@lists.freedesktop.org; Madajczak, Tomasz; Rutkowski, Adam J;
Jesse Barnes (jbarnes@virtuousgeek.org)
Subject: RE: [Intel-gfx] pin OABUFFER to GGTT



---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47

> -----Original Message-----
> From: Chris Wilson [mailto:chris@chris-wilson.co.uk]
> Sent: Wednesday, July 02, 2014 7:56 AM
> To: Ben Widawsky
> Cc: Mateo Lozano, Oscar; Intel-gfx@lists.freedesktop.org; Madajczak, 
> Tomasz; Rutkowski, Adam J; Jesse Barnes (jbarnes@virtuousgeek.org)
> Subject: Re: [Intel-gfx] pin OABUFFER to GGTT
> 
> On Tue, Jul 01, 2014 at 11:40:52PM -0700, Ben Widawsky wrote:
> > On Tue, Jul 01, 2014 at 08:54:27PM +0100, Chris Wilson wrote:
> > > On Tue, Jul 01, 2014 at 05:16:30PM +0000, Mateo Lozano, Oscar wrote:
> > > > > The issue is they need:
> > > > >
> > > > > A) A buffer object.
> > > > > B) Bound to GGTT.
> > > > > C) That userspace knows the GGTT offset of, so that they can 
> > > > > program OABUFFER with it.
> > > > > D) That userspace can map so that they can read the reported
> counters.
> > > > >
> > > > > They used to create a bo, call bo_pin on it, use args->offset 
> > > > > to program OABUFFER (via MI_LOAD_REGISTER_IMM, I imagine), map 
> > > > > it and read the counter values. They cannot do this anymore.
> > > >
> > > > The answer might be that all of this needs to be done by the 
> > > > kernel
> itself, but then we need to provide an interface to userspace...
> > >
> > > Yes. If you need to pin a buffer for a register, then it needs to 
> > > be handled by the kernel. Especially one that provides information 
> > > about other users.
> > > -Chris
> > >
> >
> > I'm unclear why they need the offset. Can it not work like any other 
> > relocation, except with the requirement that it's in the GGTT?
> >
> > I'd also argue that they need to be able to map it (they need the 
> > contents, which may or may not be mapped). However, I think this is 
> > a very minor point.
> >
> > With the command validator this should be a pretty reasonable thing 
> > to accomplish. I think we can just give a flag for the reloc that it 
> > needs to be in the GGTT, and then pass a check to the command 
> > scanner that only that one offset is allowed, and only for OA.
> 
> If the intention was to only measure within a batch and the command 
> parser ensured that the register was cleared before the end of the batch,
fine.
> That's not an information leak nor do we keep the hardware pointing to 
> an unpinned buffer afterwards.
> Chris Wilson, Intel Open Source Technology Centre

Tomasz: is the intention to only measure within a batch? My impression is
that you wanted to maintain the OABUFFER programmed and then collect
performance reports for a period of time (probably for several
batchbuffers). If that´s the case, the relocation approach is not possible.

Chris: please notice that the bo_pin ioctl they were using until now
required root, so the information leak about other users was not that bad.
Still, I tend to agree that bo_pin for this is overkill and the programming
of OABUFFER should be carried out by the kernel (with some interface to
userspace so that they can retrieve the actual contents of the buffer...
root only of course). 

-- Oscar

[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 7228 bytes --]

[-- Attachment #2.1: Type: text/plain, Size: 846 bytes --]

--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.

[-- Attachment #2.2: Type: text/html, Size: 1553 bytes --]

[-- Attachment #3: Type: text/plain, Size: 159 bytes --]

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: pin OABUFFER to GGTT

[Damien Lespiau]

On Wed, Jul 02, 2014 at 10:31:45AM +0000, Madajczak, Tomasz wrote:
> There isn't any secret or privacy in that OA buffer data - just
> results of performance counters, shown by tools such as GPA/ Vtune.

Chris alludes to the fact that if there's a way for one application to
gather data about other applications (whatever kind of data), that's an
automatic security concern.

One may think that's only very theoretical, but the side channel attacks
are (surpringly) real and effective. It cannot be proven easily that
exposing data about other contexes is totally secure.

There are ways around that however. If only root can gather that data, I
think we've contained the issue enough for the case at hand?

-- 
Damien

Re: pin OABUFFER to GGTT

[Madajczak, Tomasz]

[-- Attachment #1.1: Type: text/plain, Size: 1157 bytes --]

bo_pin had so far root privileges - they were sufficient and it would be
enough to get bo_pin back the same way. This also implies that root can only
start the OA buffer measurements.

-Tomasz

-----Original Message-----
From: Lespiau, Damien 
Sent: Wednesday, July 02, 2014 12:49 PM
To: Madajczak, Tomasz
Cc: Mateo Lozano, Oscar; Chris Wilson; Ben Widawsky;
Intel-gfx@lists.freedesktop.org
Subject: Re: [Intel-gfx] pin OABUFFER to GGTT

On Wed, Jul 02, 2014 at 10:31:45AM +0000, Madajczak, Tomasz wrote:
> There isn't any secret or privacy in that OA buffer data - just 
> results of performance counters, shown by tools such as GPA/ Vtune.

Chris alludes to the fact that if there's a way for one application to
gather data about other applications (whatever kind of data), that's an
automatic security concern.

One may think that's only very theoretical, but the side channel attacks are
(surpringly) real and effective. It cannot be proven easily that exposing
data about other contexes is totally secure.

There are ways around that however. If only root can gather that data, I
think we've contained the issue enough for the case at hand?

--
Damien

[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 7228 bytes --]

[-- Attachment #2.1: Type: text/plain, Size: 846 bytes --]

--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.

[-- Attachment #2.2: Type: text/html, Size: 1553 bytes --]

[-- Attachment #3: Type: text/plain, Size: 159 bytes --]

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: pin OABUFFER to GGTT

[Rutkowski, Adam J]

Hi

> Chris alludes to the fact that if there's a way for one application to gather
> data about other applications (whatever kind of data), that's an automatic
> security concern.

We have a few use cases for OA buffer and "one application to gather data about other applications" is basically a definition of at least some of them. 

> I'm unclear why they need the offset. Can it not work like any other relocation, except with the requirement that it's in the GGTT?
We read OA_TAIL_PTR register via batch buffer (SRM) and we need to subtract OA buffer offset from it to get the relative tail.

Currently in most cases we write to OACONTROL, OABUFFER and other OA registers via MMIO in user mode.

I suppose your answer is to move OA ownership to kernel and provide new API for this. This is fine for us as long as multiple non-root applications are able to freely access contents of OA buffer in an efficient manner. We are ok with limiting other functionalities (like setting up / programming OA) to root only. 

One more thing. Assuming usermode has OA buffer mapped I don't see how to avoid exposing of OA buffer offset to user mode in real life scenarios. To profile selected commands usermode driver must read OA_TAIL_PTR via SRM (so that it is written to memory together with perf counters). And it needs to know how the value read translates to offset in OA buffer. This implies usermode needs to know OA buffer offset (since the address in OA_BUFFER_PTR is not relative to the buffer boundary). 

Having said all this, how about restoring the pin_ioctl? At least for some time? We do have a use case and moving to other - better - solution would take time. I think backward compatibility is something that you take into consideration as well.

Thanks
Adam

> -----Original Message-----
> From: Intel-gfx [mailto:intel-gfx-bounces@lists.freedesktop.org] On Behalf
> Of Madajczak, Tomasz
> Sent: Wednesday, July 2, 2014 1:12 PM
> To: Lespiau, Damien
> Cc: Ben Widawsky; Intel-gfx@lists.freedesktop.org
> Subject: Re: [Intel-gfx] pin OABUFFER to GGTT
> 
> bo_pin had so far root privileges - they were sufficient and it would be
> enough to get bo_pin back the same way. This also implies that root can only
> start the OA buffer measurements.
> 
> -Tomasz
> 
> -----Original Message-----
> From: Lespiau, Damien
> Sent: Wednesday, July 02, 2014 12:49 PM
> To: Madajczak, Tomasz
> Cc: Mateo Lozano, Oscar; Chris Wilson; Ben Widawsky; Intel-
> gfx@lists.freedesktop.org
> Subject: Re: [Intel-gfx] pin OABUFFER to GGTT
> 
> On Wed, Jul 02, 2014 at 10:31:45AM +0000, Madajczak, Tomasz wrote:
> > There isn't any secret or privacy in that OA buffer data - just
> > results of performance counters, shown by tools such as GPA/ Vtune.
> 
> Chris alludes to the fact that if there's a way for one application to gather
> data about other applications (whatever kind of data), that's an automatic
> security concern.
> 
> One may think that's only very theoretical, but the side channel attacks are
> (surpringly) real and effective. It cannot be proven easily that exposing data
> about other contexes is totally secure.
> 
> There are ways around that however. If only root can gather that data, I think
> we've contained the issue enough for the case at hand?
> 
> --
> Damien
--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.

Re: pin OABUFFER to GGTT

[Ben Widawsky]

On Wed, Jul 02, 2014 at 10:31:45AM +0000, Madajczak, Tomasz wrote:
> Ad>Tomasz: is the intention to only measure within a batch? My impression is
> that you wanted to maintain the OABUFFER programmed and then collect
> performance reports for a period of time (probably for several
> batchbuffers). If that´s the case, the relocation approach is not possible.
> 
> I confirm that relocation approach is not possible.

I'm not sure it's not possible. As long as you only need to start and
stop the OA counters from one context, only one context needs to emit
the reloc (with the GGTT pin flag). Nobody else needs to know the flag
is present, and indeed it prevents other contexts from snooping the
data.

OTOH, later in the thread you say root + PIN is okay for you. That is
probably easiest.

> 
> OA buffer collect performance data for many hw contexts, and thus many batch
> buffers, it is a hardware reporting mechanism not bound to any specific GPU
> command send within a BB.
> 
> The report is triggered by hw when:
> - hw context switches, that action can be caused by any level GPU
> preemption, GUC direct context submission, and regular kernel exec list
> submission
> - GPU frequency change - that is trigger by power management
> - RC6 transition
> - time interval expiration (if programmed so for collect performance data at
> regular time intervals)
> 
> Each instance of UMD maps then the OA buffer to examine it within the
> Performance Query measurement window. Window is learned with Query::Begin
> and Query::End timestamps and reported OABufferTail pointer on Query::End.
> As OA buffer is large (up to 16MB), any KMD assist in conveying that data to
> UMD on each Query::End is not applicable - queries might be done per each
> Draw. There isn't any secret or privacy in that OA buffer data - just
> results of performance counters, shown by tools such as GPA/ Vtune.
> 
> Thanks,
> Tomasz
> 
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Mateo Lozano, Oscar 
> Sent: Wednesday, July 02, 2014 10:50 AM
> To: Chris Wilson; Ben Widawsky
> Cc: Intel-gfx@lists.freedesktop.org; Madajczak, Tomasz; Rutkowski, Adam J;
> Jesse Barnes (jbarnes@virtuousgeek.org)
> Subject: RE: [Intel-gfx] pin OABUFFER to GGTT
> 
> 
> 
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
> 
> > -----Original Message-----
> > From: Chris Wilson [mailto:chris@chris-wilson.co.uk]
> > Sent: Wednesday, July 02, 2014 7:56 AM
> > To: Ben Widawsky
> > Cc: Mateo Lozano, Oscar; Intel-gfx@lists.freedesktop.org; Madajczak, 
> > Tomasz; Rutkowski, Adam J; Jesse Barnes (jbarnes@virtuousgeek.org)
> > Subject: Re: [Intel-gfx] pin OABUFFER to GGTT
> > 
> > On Tue, Jul 01, 2014 at 11:40:52PM -0700, Ben Widawsky wrote:
> > > On Tue, Jul 01, 2014 at 08:54:27PM +0100, Chris Wilson wrote:
> > > > On Tue, Jul 01, 2014 at 05:16:30PM +0000, Mateo Lozano, Oscar wrote:
> > > > > > The issue is they need:
> > > > > >
> > > > > > A) A buffer object.
> > > > > > B) Bound to GGTT.
> > > > > > C) That userspace knows the GGTT offset of, so that they can 
> > > > > > program OABUFFER with it.
> > > > > > D) That userspace can map so that they can read the reported
> > counters.
> > > > > >
> > > > > > They used to create a bo, call bo_pin on it, use args->offset 
> > > > > > to program OABUFFER (via MI_LOAD_REGISTER_IMM, I imagine), map 
> > > > > > it and read the counter values. They cannot do this anymore.
> > > > >
> > > > > The answer might be that all of this needs to be done by the 
> > > > > kernel
> > itself, but then we need to provide an interface to userspace...
> > > >
> > > > Yes. If you need to pin a buffer for a register, then it needs to 
> > > > be handled by the kernel. Especially one that provides information 
> > > > about other users.
> > > > -Chris
> > > >
> > >
> > > I'm unclear why they need the offset. Can it not work like any other 
> > > relocation, except with the requirement that it's in the GGTT?
> > >
> > > I'd also argue that they need to be able to map it (they need the 
> > > contents, which may or may not be mapped). However, I think this is 
> > > a very minor point.
> > >
> > > With the command validator this should be a pretty reasonable thing 
> > > to accomplish. I think we can just give a flag for the reloc that it 
> > > needs to be in the GGTT, and then pass a check to the command 
> > > scanner that only that one offset is allowed, and only for OA.
> > 
> > If the intention was to only measure within a batch and the command 
> > parser ensured that the register was cleared before the end of the batch,
> fine.
> > That's not an information leak nor do we keep the hardware pointing to 
> > an unpinned buffer afterwards.
> > Chris Wilson, Intel Open Source Technology Centre
> 
> Tomasz: is the intention to only measure within a batch? My impression is
> that you wanted to maintain the OABUFFER programmed and then collect
> performance reports for a period of time (probably for several
> batchbuffers). If that´s the case, the relocation approach is not possible.
> 
> Chris: please notice that the bo_pin ioctl they were using until now
> required root, so the information leak about other users was not that bad.
> Still, I tend to agree that bo_pin for this is overkill and the programming
> of OABUFFER should be carried out by the kernel (with some interface to
> userspace so that they can retrieve the actual contents of the buffer...
> root only of course). 
> 
> -- Oscar



> --------------------------------------------------------------------
> 
> Intel Technology Poland sp. z o.o.
> ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
> 
> Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
> przegladanie lub rozpowszechnianie jest zabronione.
> This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
> others is strictly prohibited.


-- 
Ben Widawsky, Intel Open Source Technology Center
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: pin OABUFFER to GGTT

[Damien Lespiau]

On Wed, Jul 02, 2014 at 02:19:42PM +0100, Rutkowski, Adam J wrote:
> Having said all this, how about restoring the pin_ioctl? At least for
> some time? We do have a use case and moving to other - better -
> solution would take time. I think backward compatibility is something
> that you take into consideration as well.

So, I just sent a patch reverting the change. Daniel will have an
opinion about this I'm sure, being the original author. Let see what
happens when he's back from holidays.

Cheers,

-- 
Damien

Re: pin OABUFFER to GGTT

[Ben Widawsky]

On Thu, Jul 03, 2014 at 08:17:32AM +0100, Damien Lespiau wrote:
> On Wed, Jul 02, 2014 at 02:19:42PM +0100, Rutkowski, Adam J wrote:
> > Having said all this, how about restoring the pin_ioctl? At least for
> > some time? We do have a use case and moving to other - better -
> > solution would take time. I think backward compatibility is something
> > that you take into consideration as well.
> 
> So, I just sent a patch reverting the change. Daniel will have an
> opinion about this I'm sure, being the original author. Let see what
> happens when he's back from holidays.
> 
> Cheers,
> 
> -- 
> Damien

Just a note for a future ppgtt people - this adds another way to get
multiple VMAs for a single BO. To this point, it had only been flink,
and dmabuf. IIRC there are few unhandled corner cases for this. Also
note that if the BO is still referenced within a batch, we need the flag
to tell us it needs global binding.

FWIW, I remain in favor of the relocation idea unless someone already
expressed why we need multiple processes to have the relocation info.

-- 
Ben Widawsky, Intel Open Source Technology Center

Re: pin OABUFFER to GGTT

[Ben Widawsky]

On Thu, Jul 03, 2014 at 10:10:48AM -0700, Ben Widawsky wrote:
> On Thu, Jul 03, 2014 at 08:17:32AM +0100, Damien Lespiau wrote:
> > On Wed, Jul 02, 2014 at 02:19:42PM +0100, Rutkowski, Adam J wrote:
> > > Having said all this, how about restoring the pin_ioctl? At least for
> > > some time? We do have a use case and moving to other - better -
> > > solution would take time. I think backward compatibility is something
> > > that you take into consideration as well.
> > 
> > So, I just sent a patch reverting the change. Daniel will have an
> > opinion about this I'm sure, being the original author. Let see what
> > happens when he's back from holidays.
> > 
> > Cheers,
> > 
> > -- 
> > Damien
> 
> Just a note for a future ppgtt people - this adds another way to get
> multiple VMAs for a single BO. To this point, it had only been flink,
> and dmabuf. IIRC there are few unhandled corner cases for this. Also
> note that if the BO is still referenced within a batch, we need the flag
> to tell us it needs global binding.
> 
> FWIW, I remain in favor of the relocation idea unless someone already
> expressed why we need multiple processes to have the relocation info.
> 

I just realized there isn't really a good way to make the buffer persist
at the same offset if we use the relocation method. So I take back the
statement that it's a good idea.

-- 
Ben Widawsky, Intel Open Source Technology Center

Re: pin OABUFFER to GGTT

[Daniel Vetter]

On Tue, Jul 01, 2014 at 08:54:27PM +0100, Chris Wilson wrote:
> On Tue, Jul 01, 2014 at 05:16:30PM +0000, Mateo Lozano, Oscar wrote:
> > > The issue is they need:
> > > 
> > > A) A buffer object.
> > > B) Bound to GGTT.
> > > C) That userspace knows the GGTT offset of, so that they can program
> > > OABUFFER with it.
> > > D) That userspace can map so that they can read the reported counters.
> > > 
> > > They used to create a bo, call bo_pin on it, use args->offset to program
> > > OABUFFER (via MI_LOAD_REGISTER_IMM, I imagine), map it and read the
> > > counter values. They cannot do this anymore.
> > 
> > The answer might be that all of this needs to be done by the kernel
> > itself, but then we need to provide an interface to userspace...
> 
> Yes. If you need to pin a buffer for a register, then it needs to be
> handled by the kernel. Especially one that provides information about
> other users.

Short-circuiting the entire discussion here. Afaik there's two OA modes:
- inline with the batch with MI_REPORT_PERF
- global with the ringbuffer setup with the OABUFFER registers

The later should indeed be fully controlled by the kernel as Chris
suggested and exposed as an off-cpu performance monitoring unit through
the perf subsystem. Chris has rfc patches floating somewhere to do this
for other gpu perf data.

One fun thing here is the coordination between these two OA modes since
iirc they both use the same setup registers for the performance counter
configuration. No idea yet how to solve this.

But really userspace shouldn't program ggtt offset, not even
debug/performance measuring tools.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: pin OABUFFER to GGTT

[Bragg, Robert]

[-- Attachment #1.1: Type: text/plain, Size: 3183 bytes --]

On Mon, Jul 7, 2014 at 9:43 PM, Daniel Vetter <daniel@ffwll.ch> wrote:

> On Tue, Jul 01, 2014 at 08:54:27PM +0100, Chris Wilson wrote:
> > On Tue, Jul 01, 2014 at 05:16:30PM +0000, Mateo Lozano, Oscar wrote:
> > > > The issue is they need:
> > > >
> > > > A) A buffer object.
> > > > B) Bound to GGTT.
> > > > C) That userspace knows the GGTT offset of, so that they can program
> > > > OABUFFER with it.
> > > > D) That userspace can map so that they can read the reported
> counters.
> > > >
> > > > They used to create a bo, call bo_pin on it, use args->offset to
> program
> > > > OABUFFER (via MI_LOAD_REGISTER_IMM, I imagine), map it and read the
> > > > counter values. They cannot do this anymore.
> > >
> > > The answer might be that all of this needs to be done by the kernel
> > > itself, but then we need to provide an interface to userspace...
> >
> > Yes. If you need to pin a buffer for a register, then it needs to be
> > handled by the kernel. Especially one that provides information about
> > other users.
>
> Short-circuiting the entire discussion here. Afaik there's two OA modes:
> - inline with the batch with MI_REPORT_PERF
> - global with the ringbuffer setup with the OABUFFER registers
>

There's one other; we can read the counters via mmio for HSW+ too. I've
found that quite convenient for experimenting with capturing OA counters
from userspace. A notable disadvantage with reading via mmio though is that
there's no latch and hold mechanism to make sure the counters are frozen
while they are read back-to-back.


> The later should indeed be fully controlled by the kernel as Chris
> suggested and exposed as an off-cpu performance monitoring unit through
> the perf subsystem. Chris has rfc patches floating somewhere to do this
> for other gpu perf data.
>

Just to let folks know; I've recently been starting to play around with
Chris' perf patch plus a couple of small patches on top and was planning on
experimenting with exposing the oa counters this way soon.

I was hoping to try and pick up the discussion of this patch soon too and
give some comments, but I'd also like to collect some concrete data as a
reference point first, to be more confident in my own understanding of how
things are behaving.


> One fun thing here is the coordination between these two OA modes since
> iirc they both use the same setup registers for the performance counter
> configuration. No idea yet how to solve this.
>

> But really userspace shouldn't program ggtt offset, not even
> debug/performance measuring tools.
>

I just wanted to pop my head up here just so others are aware that I'm
another person looking at this area, aiming to understand how best to make
this data available to both GL and tools; initially considering Mesa's
performance query extensions (that don't always report reliable data
currently) and tools like intel-gpu-top.

--
Regards,
Robert


> -Daniel
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> +41 (0) 79 365 57 48 - http://blog.ffwll.ch
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx
>

[-- Attachment #1.2: Type: text/html, Size: 4812 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx