Preventing zero GPU virtual address allocation

Preventing zero GPU virtual address allocation

[Song, Ruiling]

[-- Attachment #1: Type: text/plain, Size: 479 bytes --]

Hi Daniel,

OpenCL language support NULL pointer, using zero as the NULL pointer is the obvious way. That is zero will be treated as invalid address.
Then it requires drm won't allocate zero to drm buffer. And David in CC list has help us make a patch, please see attached. The logic is only for
ppgtt, and he said zero offset is used under ggtt. My question is what is offset zero used under ggtt? Will it make sure zero is not allocatable to drm buffer object?

Ruiling

[-- Attachment #2: nozerooffset2.patch --]
[-- Type: application/octet-stream, Size: 583 bytes --]

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 996f60fae212..72657aa40319 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -3530,6 +3530,12 @@ i915_gem_object_bind_to_vm(struct drm_i915_gem_object *obj,
 	struct i915_vma *vma;
 	int ret;
 
+	/* We reserve address 0 when using ppgtt,
+	 * since it's used by opencl as the NULL pointer
+	 */
+	if (start == 0 && !i915_is_ggtt(vm))
+		start = 1;
+
 	fence_size = i915_gem_get_gtt_size(dev,
 					   obj->base.size,
 					   obj->tiling_mode);

[-- Attachment #3: Type: text/plain, Size: 153 bytes --]

_______________________________________________
Beignet mailing list
Beignet@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/beignet

Re: [Intel-gfx] Preventing zero GPU virtual address allocation

[Daniel Vetter]

On Thu, Mar 05, 2015 at 02:56:52AM +0000, Song, Ruiling wrote:
> Hi Daniel,
> 
> OpenCL language support NULL pointer, using zero as the NULL pointer is
> the obvious way. That is zero will be treated as invalid address.  Then
> it requires drm won't allocate zero to drm buffer. And David in CC
> list has help us make a patch, please see attached. The logic is only
> for ppgtt, and he said zero offset is used under ggtt. My question is
> what is offset zero used under ggtt? Will it make sure zero is not
> allocatable to drm buffer object?

The code in i915_gem_execbuf.c already supports an optional bias to avoid
putting a buffer into the first few kb. See __EXEC_OBJECT_NEEDS_BIAS. I
suggest you expose this to userspace, which also address your issue that
you didn't add an abi revision flag.

And since this is abi I need open-source userspace and all that.

Thanks, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Beignet mailing list
Beignet@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/beignet

Re: Preventing zero GPU virtual address allocation

[Chris Wilson]

On Thu, Mar 05, 2015 at 01:52:51PM +0100, Daniel Vetter wrote:
> On Thu, Mar 05, 2015 at 02:56:52AM +0000, Song, Ruiling wrote:
> > Hi Daniel,
> > 
> > OpenCL language support NULL pointer, using zero as the NULL pointer is
> > the obvious way. That is zero will be treated as invalid address.  Then
> > it requires drm won't allocate zero to drm buffer. And David in CC
> > list has help us make a patch, please see attached. The logic is only
> > for ppgtt, and he said zero offset is used under ggtt. My question is
> > what is offset zero used under ggtt? Will it make sure zero is not
> > allocatable to drm buffer object?
> 
> The code in i915_gem_execbuf.c already supports an optional bias to avoid
> putting a buffer into the first few kb. See __EXEC_OBJECT_NEEDS_BIAS. I
> suggest you expose this to userspace, which also address your issue that
> you didn't add an abi revision flag.

A better API would be to allow userspace to request a buffer to place at
a specific point in the VM and fail if that is not possible aka
soft-pinning. Then OCL could assign a bo to offset 0 and detect writes
to the NULL address if it so desired. With full-ppgtt, userspace can be
sure of being able to evict any location in its VM and so also allows
graceful detection of scenarios under which it cannot provide the NULL
address safety feature (and opt not to run, or just bury its head in the
sand).
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Intel-gfx] Preventing zero GPU virtual address allocation

[Daniel Vetter]

On Thu, Mar 05, 2015 at 01:01:21PM +0000, Chris Wilson wrote:
> On Thu, Mar 05, 2015 at 01:52:51PM +0100, Daniel Vetter wrote:
> > On Thu, Mar 05, 2015 at 02:56:52AM +0000, Song, Ruiling wrote:
> > > Hi Daniel,
> > > 
> > > OpenCL language support NULL pointer, using zero as the NULL pointer is
> > > the obvious way. That is zero will be treated as invalid address.  Then
> > > it requires drm won't allocate zero to drm buffer. And David in CC
> > > list has help us make a patch, please see attached. The logic is only
> > > for ppgtt, and he said zero offset is used under ggtt. My question is
> > > what is offset zero used under ggtt? Will it make sure zero is not
> > > allocatable to drm buffer object?
> > 
> > The code in i915_gem_execbuf.c already supports an optional bias to avoid
> > putting a buffer into the first few kb. See __EXEC_OBJECT_NEEDS_BIAS. I
> > suggest you expose this to userspace, which also address your issue that
> > you didn't add an abi revision flag.
> 
> A better API would be to allow userspace to request a buffer to place at
> a specific point in the VM and fail if that is not possible aka
> soft-pinning. Then OCL could assign a bo to offset 0 and detect writes
> to the NULL address if it so desired. With full-ppgtt, userspace can be
> sure of being able to evict any location in its VM and so also allows
> graceful detection of scenarios under which it cannot provide the NULL
> address safety feature (and opt not to run, or just bury its head in the
> sand).

I recommended exposing the PIN_BIAS since that will work without full
ppgtt too. And yeah for full ppgtt we could just use svm where userspace
controls the address, but since that's still a bit out we might need a
quick interim solution?
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Beignet mailing list
Beignet@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/beignet

Re: [Intel-gfx] Preventing zero GPU virtual address allocation

[Chris Wilson]

On Thu, Mar 05, 2015 at 04:27:59PM +0100, Daniel Vetter wrote:
> I recommended exposing the PIN_BIAS since that will work without full
> ppgtt too. And yeah for full ppgtt we could just use svm where userspace
> controls the address, but since that's still a bit out we might need a
> quick interim solution?

Letting userspace control the address of bo used in a batch is about 2
patches each of ~100 lines. And it could be used will full-ppgtt before
svm if mesa wants to take complete control of its layout. I think it is
one of those useful tools that is likely to find uses far beyond the
initial justification.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Beignet mailing list
Beignet@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/beignet

Re: [Intel-gfx] Preventing zero GPU virtual address allocation

[Zou, Nanhai]

I don't understand why we need a complex solution when there is already a simple solution with patch.
What is the drawback of reserving page 0?
Before we going to that complex solution, could we just reserve page zero?
It is simple and straight forward.

Thanks
Zou Nanhai

> -----Original Message-----
> From: Beignet [mailto:beignet-bounces@lists.freedesktop.org] On Behalf Of
> Daniel Vetter
> Sent: Thursday, March 05, 2015 11:28 PM
> To: Chris Wilson; Daniel Vetter; Song, Ruiling; Vetter, Daniel;
> intel-gfx@lists.freedesktop.org; Yang, Rong R; beignet@lists.freedesktop.org;
> Weinehall, David
> Subject: Re: [Beignet] [Intel-gfx] Preventing zero GPU virtual address allocation
> 
> On Thu, Mar 05, 2015 at 01:01:21PM +0000, Chris Wilson wrote:
> > On Thu, Mar 05, 2015 at 01:52:51PM +0100, Daniel Vetter wrote:
> > > On Thu, Mar 05, 2015 at 02:56:52AM +0000, Song, Ruiling wrote:
> > > > Hi Daniel,
> > > >
> > > > OpenCL language support NULL pointer, using zero as the NULL
> > > > pointer is the obvious way. That is zero will be treated as
> > > > invalid address.  Then it requires drm won't allocate zero to drm
> > > > buffer. And David in CC list has help us make a patch, please see
> > > > attached. The logic is only for ppgtt, and he said zero offset is
> > > > used under ggtt. My question is what is offset zero used under
> > > > ggtt? Will it make sure zero is not allocatable to drm buffer object?
> > >
> > > The code in i915_gem_execbuf.c already supports an optional bias to
> > > avoid putting a buffer into the first few kb. See
> > > __EXEC_OBJECT_NEEDS_BIAS. I suggest you expose this to userspace,
> > > which also address your issue that you didn't add an abi revision flag.
> >
> > A better API would be to allow userspace to request a buffer to place
> > at a specific point in the VM and fail if that is not possible aka
> > soft-pinning. Then OCL could assign a bo to offset 0 and detect writes
> > to the NULL address if it so desired. With full-ppgtt, userspace can
> > be sure of being able to evict any location in its VM and so also
> > allows graceful detection of scenarios under which it cannot provide
> > the NULL address safety feature (and opt not to run, or just bury its
> > head in the sand).
> 
> I recommended exposing the PIN_BIAS since that will work without full ppgtt
> too. And yeah for full ppgtt we could just use svm where userspace controls the
> address, but since that's still a bit out we might need a quick interim solution?
> -Daniel
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> +41 (0) 79 365 57 48 - http://blog.ffwll.ch
> _______________________________________________
> Beignet mailing list
> Beignet@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/beignet
_______________________________________________
Beignet mailing list
Beignet@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/beignet

Re: [Beignet] Preventing zero GPU virtual address allocation

[Chris Wilson]

On Fri, Mar 06, 2015 at 02:11:18AM +0000, Zou, Nanhai wrote:
> I don't understand why we need a complex solution when there is already a simple solution with patch.
> What is the drawback of reserving page 0?
> Before we going to that complex solution, could we just reserve page zero?
> It is simple and straight forward.

Because it is a nonsense ABI constraint. If you want the equivalent
of MAP_FIXED, we should give you MAP_FIXED.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Beignet] Preventing zero GPU virtual address allocation

[Zou, Nanhai]

We don't need MAP_FIXED, we just want to avoid address 0 to be allocated.

Though I think using MAP_FIXED is overkill, will bring much unnecessary complexity on both kernel and beignet side.
I don't mind if people can provide stable MAP_FIXED patches to resolve this problem a few months or years later.

At that time, kernel driver can revert the reserve page 0 patch.
Before that reserve page 0 can benefit all the Beignet user without breaking anything.

As I know, on CPU side, many arches with flexible address space like IA64 have reserved virtual page 0 to address this problem,
I don't see why this is non sense.

Thanks
Zou Nanhai



> -----Original Message-----
> From: Chris Wilson [mailto:chris@chris-wilson.co.uk]
> Sent: Friday, March 06, 2015 4:39 PM
> To: Zou, Nanhai
> Cc: Daniel Vetter; Song, Ruiling; Vetter, Daniel; intel-gfx@lists.freedesktop.org;
> Yang, Rong R; beignet@lists.freedesktop.org; Weinehall, David
> Subject: Re: [Beignet] [Intel-gfx] Preventing zero GPU virtual address allocation
> 
> On Fri, Mar 06, 2015 at 02:11:18AM +0000, Zou, Nanhai wrote:
> > I don't understand why we need a complex solution when there is already a
> simple solution with patch.
> > What is the drawback of reserving page 0?
> > Before we going to that complex solution, could we just reserve page zero?
> > It is simple and straight forward.
> 
> Because it is a nonsense ABI constraint. If you want the equivalent of
> MAP_FIXED, we should give you MAP_FIXED.
> -Chris
> 
> --
> Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Intel-gfx] Preventing zero GPU virtual address allocation

[Chris Wilson]

On Mon, Mar 09, 2015 at 02:34:46AM +0000, Zou, Nanhai wrote:
> We don't need MAP_FIXED, we just want to avoid address 0 to be allocated.
> 
> Though I think using MAP_FIXED is overkill, will bring much unnecessary complexity on both kernel and beignet side.
> I don't mind if people can provide stable MAP_FIXED patches to resolve this problem a few months or years later.
> 
> At that time, kernel driver can revert the reserve page 0 patch.
> Before that reserve page 0 can benefit all the Beignet user without breaking anything.

The point is that is becomes ABI. So no the kernel can't just revert it.
There is nothing special about address 0 in ether GTT or virtual memory.
If you require a special object allocated at address 0, allocate a
special object at address 0.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Beignet mailing list
Beignet@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/beignet

Re: Preventing zero GPU virtual address allocation

[Jesse Barnes]

On 03/05/2015 01:07 PM, Chris Wilson wrote:
> On Thu, Mar 05, 2015 at 04:27:59PM +0100, Daniel Vetter wrote:
>> I recommended exposing the PIN_BIAS since that will work without full
>> ppgtt too. And yeah for full ppgtt we could just use svm where userspace
>> controls the address, but since that's still a bit out we might need a
>> quick interim solution?
> 
> Letting userspace control the address of bo used in a batch is about 2
> patches each of ~100 lines. And it could be used will full-ppgtt before
> svm if mesa wants to take complete control of its layout. I think it is
> one of those useful tools that is likely to find uses far beyond the
> initial justification.

Well we need someone to pick it up and do it; we've already shafted
userspace for several years due to foot dragging on the command
parser...  I hope something as simple as this doesn't stall out.

Jesse

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: Preventing zero GPU virtual address allocation

[Jesse Barnes]

On 03/09/2015 08:46 AM, Jesse Barnes wrote:
> On 03/05/2015 01:07 PM, Chris Wilson wrote:
>> On Thu, Mar 05, 2015 at 04:27:59PM +0100, Daniel Vetter wrote:
>>> I recommended exposing the PIN_BIAS since that will work without full
>>> ppgtt too. And yeah for full ppgtt we could just use svm where userspace
>>> controls the address, but since that's still a bit out we might need a
>>> quick interim solution?
>>
>> Letting userspace control the address of bo used in a batch is about 2
>> patches each of ~100 lines. And it could be used will full-ppgtt before
>> svm if mesa wants to take complete control of its layout. I think it is
>> one of those useful tools that is likely to find uses far beyond the
>> initial justification.
> 
> Well we need someone to pick it up and do it; we've already shafted
> userspace for several years due to foot dragging on the command
> parser...  I hope something as simple as this doesn't stall out.

Nevermind, I see you already did it.  Ruiling and Nanhai, can you make
sure Chris's proposed interface will work for you and provide a review
if so?

Thanks,
Jesse

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Intel-gfx] Preventing zero GPU virtual address allocation

[Zou, Nanhai]

> -----Original Message-----
> From: Chris Wilson [mailto:chris@chris-wilson.co.uk]
> Sent: Monday, March 09, 2015 8:02 PM
> To: Zou, Nanhai
> Cc: Daniel Vetter; Song, Ruiling; Vetter, Daniel; intel-gfx@lists.freedesktop.org;
> Yang, Rong R; beignet@lists.freedesktop.org; Weinehall, David
> Subject: Re: [Beignet] [Intel-gfx] Preventing zero GPU virtual address allocation
> 
> On Mon, Mar 09, 2015 at 02:34:46AM +0000, Zou, Nanhai wrote:
> > We don't need MAP_FIXED, we just want to avoid address 0 to be allocated.
> >
> > Though I think using MAP_FIXED is overkill, will bring much unnecessary
> complexity on both kernel and beignet side.
> > I don't mind if people can provide stable MAP_FIXED patches to resolve this
> problem a few months or years later.
> >
> > At that time, kernel driver can revert the reserve page 0 patch.
> > Before that reserve page 0 can benefit all the Beignet user without breaking
> anything.
> 
> The point is that is becomes ABI. So no the kernel can't just revert it.
> There is nothing special about address 0 in ether GTT or virtual memory.
> If you require a special object allocated at address 0, allocate a special object
> at address 0.
> -Chris


Hi,
	Zero page is not an ABI, It is only a strategy of virtual space allocation.
	Nobody would need exactly a page of virtual address 0. So there is no dependency.
	
	Zero page is a very common used method to avoid the NULL pointer issue.
	See http://en.wikipedia.org/wiki/Zero_page

	I can see a lot of issue to do that with MAP_FIXED zero page in user space.

	Say if a program combine libva, beignet and mesa or other graphics components together, 
	which component should be responsible of allocating this zero page?

	What if a component happened to allocate a page in offset 0 before beignet and release it later? 
	Using a zero page just remove the unnecessary mess and resolve problem in a clean explicit way.


Thanks
Zou Nanhai


> --
> Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Beignet mailing list
Beignet@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/beignet

Re: [Beignet] Preventing zero GPU virtual address allocation

[David Weinehall]

On 2015-03-09 14:02, Chris Wilson wrote:
> On Mon, Mar 09, 2015 at 02:34:46AM +0000, Zou, Nanhai wrote:
>> We don't need MAP_FIXED, we just want to avoid address 0 to be allocated.
>>
>> Though I think using MAP_FIXED is overkill, will bring much unnecessary complexity on both kernel and beignet side.
>> I don't mind if people can provide stable MAP_FIXED patches to resolve this problem a few months or years later.
>>
>> At that time, kernel driver can revert the reserve page 0 patch.
>> Before that reserve page 0 can benefit all the Beignet user without breaking anything.
>
> The point is that is becomes ABI. So no the kernel can't just revert it.
> There is nothing special about address 0 in ether GTT or virtual memory.
> If you require a special object allocated at address 0, allocate a
> special object at address 0.

I've explained the ABI issue in a separate e-mail discussion, and I 
believe that they now fully understand what you meant.

That said, their main chain of reasoning makes some sense -- there is a
race condition if we rely on using MAP_FIXED, at least on systems that
do not support ppgtt. Ending up in a situation where opencl applications 
work on other hw, but fails when run on an i915-system would, at least 
in my opinion, not be ideal, no matter if it's due to an unfortunate design.

*If* a MAP_FIXED solution is decided upon, how can userland be sure that 
the GTT page mapped to 0 is indeed usable as the NULL pointer?
ON a PPGTT system that would be easy enough -- it's per process, so 
we'll be the only process allocating a page at 0, but if allocations
use a global address space that won't be possible to guarantee.

I realise that the first submitted patch didn't cover the GTT case, 
since the first indication I got was that not only was there a special 
case for GTT to need page 0 for other things, but also that this was 
"good enough" for opencl, but it seems that a full solution would be needed.

Since this is a memory area we're talking about it's not uncommon to 
have the 0th page represent the NULL pointer and impossible for 
applications to reserve, so it would hardly be an unusual and 
inexplicable solution.

All this said, how do "the other two" (NVidia, ATI) deal with this?
Implicit NULL-page, explicit MAP_FIXED allocation, or something else?


Kind regards, David
---------------------------------------------------------------------
Intel Finland Oy
Registered Address: PL 281, 00181 Helsinki 
Business Identity Code: 0357606 - 4 
Domiciled in Helsinki 

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Intel-gfx] Preventing zero GPU virtual address allocation

[Chris Wilson]

On Fri, Mar 13, 2015 at 11:10:56AM +0200, David Weinehall wrote:
> On 2015-03-09 14:02, Chris Wilson wrote:
> >On Mon, Mar 09, 2015 at 02:34:46AM +0000, Zou, Nanhai wrote:
> >>We don't need MAP_FIXED, we just want to avoid address 0 to be allocated.
> >>
> >>Though I think using MAP_FIXED is overkill, will bring much unnecessary complexity on both kernel and beignet side.
> >>I don't mind if people can provide stable MAP_FIXED patches to resolve this problem a few months or years later.
> >>
> >>At that time, kernel driver can revert the reserve page 0 patch.
> >>Before that reserve page 0 can benefit all the Beignet user without breaking anything.
> >
> >The point is that is becomes ABI. So no the kernel can't just revert it.
> >There is nothing special about address 0 in ether GTT or virtual memory.
> >If you require a special object allocated at address 0, allocate a
> >special object at address 0.
> 
> I've explained the ABI issue in a separate e-mail discussion, and I
> believe that they now fully understand what you meant.
> 
> That said, their main chain of reasoning makes some sense -- there is a
> race condition if we rely on using MAP_FIXED, at least on systems that
> do not support ppgtt. Ending up in a situation where opencl
> applications work on other hw, but fails when run on an i915-system
> would, at least in my opinion, not be ideal, no matter if it's due
> to an unfortunate design.
> 
> *If* a MAP_FIXED solution is decided upon, how can userland be sure
> that the GTT page mapped to 0 is indeed usable as the NULL pointer?
> ON a PPGTT system that would be easy enough -- it's per process, so
> we'll be the only process allocating a page at 0, but if allocations
> use a global address space that won't be possible to guarantee.

It's simple, userspace has no control over allocation of the GGTT.
Espcially something like address 0. They can make a request to have
their object at a certain location, but the kernel is entirely likely to
reject their request because it is pinned to hardware.

> I realise that the first submitted patch didn't cover the GTT case,
> since the first indication I got was that not only was there a
> special case for GTT to need page 0 for other things, but also that
> this was "good enough" for opencl, but it seems that a full solution
> would be needed.

Exactly.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Beignet mailing list
Beignet@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/beignet

Re: [Intel-gfx] Preventing zero GPU virtual address allocation

[Daniel Vetter]

On Fri, Mar 13, 2015 at 11:10:56AM +0200, David Weinehall wrote:
> On 2015-03-09 14:02, Chris Wilson wrote:
> >On Mon, Mar 09, 2015 at 02:34:46AM +0000, Zou, Nanhai wrote:
> >>We don't need MAP_FIXED, we just want to avoid address 0 to be allocated.
> >>
> >>Though I think using MAP_FIXED is overkill, will bring much unnecessary complexity on both kernel and beignet side.
> >>I don't mind if people can provide stable MAP_FIXED patches to resolve this problem a few months or years later.
> >>
> >>At that time, kernel driver can revert the reserve page 0 patch.
> >>Before that reserve page 0 can benefit all the Beignet user without breaking anything.
> >
> >The point is that is becomes ABI. So no the kernel can't just revert it.
> >There is nothing special about address 0 in ether GTT or virtual memory.
> >If you require a special object allocated at address 0, allocate a
> >special object at address 0.
> 
> I've explained the ABI issue in a separate e-mail discussion, and I believe
> that they now fully understand what you meant.
> 
> That said, their main chain of reasoning makes some sense -- there is a
> race condition if we rely on using MAP_FIXED, at least on systems that
> do not support ppgtt. Ending up in a situation where opencl applications
> work on other hw, but fails when run on an i915-system would, at least in my
> opinion, not be ideal, no matter if it's due to an unfortunate design.
> 
> *If* a MAP_FIXED solution is decided upon, how can userland be sure that the
> GTT page mapped to 0 is indeed usable as the NULL pointer?
> ON a PPGTT system that would be easy enough -- it's per process, so we'll be
> the only process allocating a page at 0, but if allocations
> use a global address space that won't be possible to guarantee.
> 
> I realise that the first submitted patch didn't cover the GTT case, since
> the first indication I got was that not only was there a special case for
> GTT to need page 0 for other things, but also that this was "good enough"
> for opencl, but it seems that a full solution would be needed.
> 
> Since this is a memory area we're talking about it's not uncommon to have
> the 0th page represent the NULL pointer and impossible for applications to
> reserve, so it would hardly be an unusual and inexplicable solution.
> 
> All this said, how do "the other two" (NVidia, ATI) deal with this?
> Implicit NULL-page, explicit MAP_FIXED allocation, or something else?

Afaik there's no opensource ocl implementation yet that needs this. So no
idea what they do.

If supporting systems without full ppgtt is a requirement for you (still
wonky on gen8 a bit, so might be a good strategy) then imo it's the
PIN_BIAS idea I've laid out earlier in this thread. That one will work
everywhere. softpin can unexpectedly fail without full ppgtt if the kernel
decides to put something at a given spot, which imo means we should only
expose it on full ppgtt systems.

And PIN_BIAS should be fairly easy to wire up since the internal logic is
all there already. So "just" needs an execbuf flag, igt test and
appropriate userspace to set that new bit.

> Kind regards, David
> ---------------------------------------------------------------------
> Intel Finland Oy
> Registered Address: PL 281, 00181 Helsinki Business Identity Code: 0357606 -
> 4 Domiciled in Helsinki
> 
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.

You want to remove this when submitting to public mailing lists. Either
apply for an exception or grab a 2nd mail address for mailing list
communication.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Beignet mailing list
Beignet@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/beignet

Re: [Beignet] Preventing zero GPU virtual address allocation

[Chris Wilson]

On Fri, Mar 13, 2015 at 10:27:38AM +0100, Daniel Vetter wrote:
> If supporting systems without full ppgtt is a requirement for you (still
> wonky on gen8 a bit, so might be a good strategy) then imo it's the
> PIN_BIAS idea I've laid out earlier in this thread. That one will work
> everywhere. softpin can unexpectedly fail without full ppgtt if the kernel
> decides to put something at a given spot, which imo means we should only
> expose it on full ppgtt systems.
> 
> And PIN_BIAS should be fairly easy to wire up since the internal logic is
> all there already. So "just" needs an execbuf flag, igt test and
> appropriate userspace to set that new bit.

It doesn't though. To provide the guarantee userspace is asking for
(which is that address 0 goes to a special, preferrably inaccessible,
page), you have to evict the first N pages in the GGTT. That is just as
likely to fail with an execbuffer flag as it would with an execobject flag.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Intel-gfx] Preventing zero GPU virtual address allocation

[Daniel Vetter]

On Fri, Mar 13, 2015 at 04:58:47PM +0000, Chris Wilson wrote:
> On Fri, Mar 13, 2015 at 10:27:38AM +0100, Daniel Vetter wrote:
> > If supporting systems without full ppgtt is a requirement for you (still
> > wonky on gen8 a bit, so might be a good strategy) then imo it's the
> > PIN_BIAS idea I've laid out earlier in this thread. That one will work
> > everywhere. softpin can unexpectedly fail without full ppgtt if the kernel
> > decides to put something at a given spot, which imo means we should only
> > expose it on full ppgtt systems.
> > 
> > And PIN_BIAS should be fairly easy to wire up since the internal logic is
> > all there already. So "just" needs an execbuf flag, igt test and
> > appropriate userspace to set that new bit.
> 
> It doesn't though. To provide the guarantee userspace is asking for
> (which is that address 0 goes to a special, preferrably inaccessible,
> page), you have to evict the first N pages in the GGTT. That is just as
> likely to fail with an execbuffer flag as it would with an execobject flag.

Afaiui userspace only needs the guarantee that NULL is never a valid
address. Which means it's never a valid address for its own buffer
objects. I don't think it cares one bit what's actually there, it's not
mandatory to fault apparently. And faulting is what's not possible.

I guess the standard is like normal C: If you access a NULL pointer,
anything can happen (including garbage on the frontbuffer), the only
guarantee you need to make is that NULL is never a valid address. At least
if no one plays tricks ;-)
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Beignet mailing list
Beignet@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/beignet

Re: [Beignet] Preventing zero GPU virtual address allocation

[Chris Wilson]

On Fri, Mar 13, 2015 at 06:13:39PM +0100, Daniel Vetter wrote:
> On Fri, Mar 13, 2015 at 04:58:47PM +0000, Chris Wilson wrote:
> > On Fri, Mar 13, 2015 at 10:27:38AM +0100, Daniel Vetter wrote:
> > > If supporting systems without full ppgtt is a requirement for you (still
> > > wonky on gen8 a bit, so might be a good strategy) then imo it's the
> > > PIN_BIAS idea I've laid out earlier in this thread. That one will work
> > > everywhere. softpin can unexpectedly fail without full ppgtt if the kernel
> > > decides to put something at a given spot, which imo means we should only
> > > expose it on full ppgtt systems.
> > > 
> > > And PIN_BIAS should be fairly easy to wire up since the internal logic is
> > > all there already. So "just" needs an execbuf flag, igt test and
> > > appropriate userspace to set that new bit.
> > 
> > It doesn't though. To provide the guarantee userspace is asking for
> > (which is that address 0 goes to a special, preferrably inaccessible,
> > page), you have to evict the first N pages in the GGTT. That is just as
> > likely to fail with an execbuffer flag as it would with an execobject flag.
> 
> Afaiui userspace only needs the guarantee that NULL is never a valid
> address. Which means it's never a valid address for its own buffer
> objects. I don't think it cares one bit what's actually there, it's not
> mandatory to fault apparently. And faulting is what's not possible.

You are bending ABI to allow userspace to use absolute addressing (no
relocations). The kernel has to make sure that nothing else is at that
address.
 
> I guess the standard is like normal C: If you access a NULL pointer,
> anything can happen (including garbage on the frontbuffer), the only
> guarantee you need to make is that NULL is never a valid address. At least
> if no one plays tricks ;-)

No. The kernel is quite strict about *NULL. It certainly doesn't allow
trivial information leakage.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Intel-gfx] Preventing zero GPU virtual address allocation

[Daniel Vetter]

On Fri, Mar 13, 2015 at 05:34:22PM +0000, Chris Wilson wrote:
> On Fri, Mar 13, 2015 at 06:13:39PM +0100, Daniel Vetter wrote:
> > On Fri, Mar 13, 2015 at 04:58:47PM +0000, Chris Wilson wrote:
> > > On Fri, Mar 13, 2015 at 10:27:38AM +0100, Daniel Vetter wrote:
> > > > If supporting systems without full ppgtt is a requirement for you (still
> > > > wonky on gen8 a bit, so might be a good strategy) then imo it's the
> > > > PIN_BIAS idea I've laid out earlier in this thread. That one will work
> > > > everywhere. softpin can unexpectedly fail without full ppgtt if the kernel
> > > > decides to put something at a given spot, which imo means we should only
> > > > expose it on full ppgtt systems.
> > > > 
> > > > And PIN_BIAS should be fairly easy to wire up since the internal logic is
> > > > all there already. So "just" needs an execbuf flag, igt test and
> > > > appropriate userspace to set that new bit.
> > > 
> > > It doesn't though. To provide the guarantee userspace is asking for
> > > (which is that address 0 goes to a special, preferrably inaccessible,
> > > page), you have to evict the first N pages in the GGTT. That is just as
> > > likely to fail with an execbuffer flag as it would with an execobject flag.
> > 
> > Afaiui userspace only needs the guarantee that NULL is never a valid
> > address. Which means it's never a valid address for its own buffer
> > objects. I don't think it cares one bit what's actually there, it's not
> > mandatory to fault apparently. And faulting is what's not possible.
> 
> You are bending ABI to allow userspace to use absolute addressing (no
> relocations). The kernel has to make sure that nothing else is at that
> address.
>  
> > I guess the standard is like normal C: If you access a NULL pointer,
> > anything can happen (including garbage on the frontbuffer), the only
> > guarantee you need to make is that NULL is never a valid address. At least
> > if no one plays tricks ;-)
> 
> No. The kernel is quite strict about *NULL. It certainly doesn't allow
> trivial information leakage.

We already have that trivial information leakage anyway. Just because we
make it a notch more likely that a special address will leak information
that doesn't make it different that without full ppgtt you can read any
other gpu clients stuff easily. And also write to it if you feel like.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Beignet mailing list
Beignet@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/beignet

Re: [Beignet] Preventing zero GPU virtual address allocation

[Song, Ruiling]

> -----Original Message-----
> From: Daniel Vetter [mailto:daniel.vetter@ffwll.ch] On Behalf Of Daniel
> Vetter
> Sent: Saturday, March 14, 2015 1:14 AM
> To: Chris Wilson; Daniel Vetter; Weinehall, David; Zou, Nanhai; Song, Ruiling;
> Vetter, Daniel; intel-gfx@lists.freedesktop.org; Yang, Rong R;
> beignet@lists.freedesktop.org
> Subject: Re: [Beignet] [Intel-gfx] Preventing zero GPU virtual address
> allocation
> 
> On Fri, Mar 13, 2015 at 04:58:47PM +0000, Chris Wilson wrote:
> > On Fri, Mar 13, 2015 at 10:27:38AM +0100, Daniel Vetter wrote:
> > > If supporting systems without full ppgtt is a requirement for you
> > > (still wonky on gen8 a bit, so might be a good strategy) then imo
> > > it's the PIN_BIAS idea I've laid out earlier in this thread. That
> > > one will work everywhere. softpin can unexpectedly fail without full
> > > ppgtt if the kernel decides to put something at a given spot, which
> > > imo means we should only expose it on full ppgtt systems.
> > >
> > > And PIN_BIAS should be fairly easy to wire up since the internal
> > > logic is all there already. So "just" needs an execbuf flag, igt
> > > test and appropriate userspace to set that new bit.
> >
> > It doesn't though. To provide the guarantee userspace is asking for
> > (which is that address 0 goes to a special, preferrably inaccessible,
> > page), you have to evict the first N pages in the GGTT. That is just
> > as likely to fail with an execbuffer flag as it would with an execobject flag.
> 
> Afaiui userspace only needs the guarantee that NULL is never a valid address.
> Which means it's never a valid address for its own buffer objects. I don't
> think it cares one bit what's actually there, it's not mandatory to fault
> apparently. And faulting is what's not possible.
Yes, This is what exactly what we need, that is make NULL as an invalid address. It's just like C language.
But I have some more comment. The buffer object used in opencl may be allocated in libva/opengl and shared for opencl usage through some opencl extension.
Afaiui, this would implicitly require libva/mesa also avoid zero-address buffer object allocation.
Will libdrm re-bind such kind of shared buffer object to a new graphics virtual address?
So that PIN_BIAS is also effective on the shared buffer, right?

Thanks!
Ruiling
> I guess the standard is like normal C: If you access a NULL pointer, anything
> can happen (including garbage on the frontbuffer), the only guarantee you
> need to make is that NULL is never a valid address. At least if no one plays
> tricks ;-) -Daniel
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> +41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Beignet] Preventing zero GPU virtual address allocation

[Daniel Vetter]

On Mon, Mar 16, 2015 at 02:29:24AM +0000, Song, Ruiling wrote:
> 
> 
> > -----Original Message-----
> > From: Daniel Vetter [mailto:daniel.vetter@ffwll.ch] On Behalf Of Daniel
> > Vetter
> > Sent: Saturday, March 14, 2015 1:14 AM
> > To: Chris Wilson; Daniel Vetter; Weinehall, David; Zou, Nanhai; Song, Ruiling;
> > Vetter, Daniel; intel-gfx@lists.freedesktop.org; Yang, Rong R;
> > beignet@lists.freedesktop.org
> > Subject: Re: [Beignet] [Intel-gfx] Preventing zero GPU virtual address
> > allocation
> > 
> > On Fri, Mar 13, 2015 at 04:58:47PM +0000, Chris Wilson wrote:
> > > On Fri, Mar 13, 2015 at 10:27:38AM +0100, Daniel Vetter wrote:
> > > > If supporting systems without full ppgtt is a requirement for you
> > > > (still wonky on gen8 a bit, so might be a good strategy) then imo
> > > > it's the PIN_BIAS idea I've laid out earlier in this thread. That
> > > > one will work everywhere. softpin can unexpectedly fail without full
> > > > ppgtt if the kernel decides to put something at a given spot, which
> > > > imo means we should only expose it on full ppgtt systems.
> > > >
> > > > And PIN_BIAS should be fairly easy to wire up since the internal
> > > > logic is all there already. So "just" needs an execbuf flag, igt
> > > > test and appropriate userspace to set that new bit.
> > >
> > > It doesn't though. To provide the guarantee userspace is asking for
> > > (which is that address 0 goes to a special, preferrably inaccessible,
> > > page), you have to evict the first N pages in the GGTT. That is just
> > > as likely to fail with an execbuffer flag as it would with an execobject flag.
> > 
> > Afaiui userspace only needs the guarantee that NULL is never a valid address.
> > Which means it's never a valid address for its own buffer objects. I don't
> > think it cares one bit what's actually there, it's not mandatory to fault
> > apparently. And faulting is what's not possible.
> Yes, This is what exactly what we need, that is make NULL as an invalid address. It's just like C language.
> But I have some more comment. The buffer object used in opencl may be allocated in libva/opengl and shared for opencl usage through some opencl extension.
> Afaiui, this would implicitly require libva/mesa also avoid zero-address buffer object allocation.
> Will libdrm re-bind such kind of shared buffer object to a new graphics virtual address?
> So that PIN_BIAS is also effective on the shared buffer, right?

Yeah we'll rebind if needed. We can make this an execbuf or context flag,
in either case anything that gets executed by ocl will be moved around if
it accidentally ended up at the wrong place. The only exception is if a
buffer is pinned already, i.e. if you're doing direct rendering to the
frontbuffer. That will give you an EBUSY, but otoh that also shouldn't
ever happen really.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Beignet] Preventing zero GPU virtual address allocation

[Jesse Barnes]

On 03/16/2015 01:52 AM, Daniel Vetter wrote:
> On Mon, Mar 16, 2015 at 02:29:24AM +0000, Song, Ruiling wrote:
>>
>>
>>> -----Original Message-----
>>> From: Daniel Vetter [mailto:daniel.vetter@ffwll.ch] On Behalf Of Daniel
>>> Vetter
>>> Sent: Saturday, March 14, 2015 1:14 AM
>>> To: Chris Wilson; Daniel Vetter; Weinehall, David; Zou, Nanhai; Song, Ruiling;
>>> Vetter, Daniel; intel-gfx@lists.freedesktop.org; Yang, Rong R;
>>> beignet@lists.freedesktop.org
>>> Subject: Re: [Beignet] [Intel-gfx] Preventing zero GPU virtual address
>>> allocation
>>>
>>> On Fri, Mar 13, 2015 at 04:58:47PM +0000, Chris Wilson wrote:
>>>> On Fri, Mar 13, 2015 at 10:27:38AM +0100, Daniel Vetter wrote:
>>>>> If supporting systems without full ppgtt is a requirement for you
>>>>> (still wonky on gen8 a bit, so might be a good strategy) then imo
>>>>> it's the PIN_BIAS idea I've laid out earlier in this thread. That
>>>>> one will work everywhere. softpin can unexpectedly fail without full
>>>>> ppgtt if the kernel decides to put something at a given spot, which
>>>>> imo means we should only expose it on full ppgtt systems.
>>>>>
>>>>> And PIN_BIAS should be fairly easy to wire up since the internal
>>>>> logic is all there already. So "just" needs an execbuf flag, igt
>>>>> test and appropriate userspace to set that new bit.
>>>>
>>>> It doesn't though. To provide the guarantee userspace is asking for
>>>> (which is that address 0 goes to a special, preferrably inaccessible,
>>>> page), you have to evict the first N pages in the GGTT. That is just
>>>> as likely to fail with an execbuffer flag as it would with an execobject flag.
>>>
>>> Afaiui userspace only needs the guarantee that NULL is never a valid address.
>>> Which means it's never a valid address for its own buffer objects. I don't
>>> think it cares one bit what's actually there, it's not mandatory to fault
>>> apparently. And faulting is what's not possible.
>> Yes, This is what exactly what we need, that is make NULL as an invalid address. It's just like C language.
>> But I have some more comment. The buffer object used in opencl may be allocated in libva/opengl and shared for opencl usage through some opencl extension.
>> Afaiui, this would implicitly require libva/mesa also avoid zero-address buffer object allocation.
>> Will libdrm re-bind such kind of shared buffer object to a new graphics virtual address?
>> So that PIN_BIAS is also effective on the shared buffer, right?
> 
> Yeah we'll rebind if needed. We can make this an execbuf or context flag,
> in either case anything that gets executed by ocl will be moved around if
> it accidentally ended up at the wrong place. The only exception is if a
> buffer is pinned already, i.e. if you're doing direct rendering to the
> frontbuffer. That will give you an EBUSY, but otoh that also shouldn't
> ever happen really.

Ruiling, are you working on this or someone from your team, presumably
based on the patch Chris posted earlier?  The zero page reservation
certainly seems simpler to me, but the MAP_FIXED approach is a lot more
flexible, and can be used for other types of debug and usages as well
(we'll need something like it for OCL pointer sharing for example), so
seems like a good thing to pursue regardless.

Thanks,
Jesse

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Beignet] Preventing zero GPU virtual address allocation

[Jesse Barnes]

On 03/16/2015 01:52 AM, Daniel Vetter wrote:
> On Mon, Mar 16, 2015 at 02:29:24AM +0000, Song, Ruiling wrote:
>>
>>
>>> -----Original Message-----
>>> From: Daniel Vetter [mailto:daniel.vetter@ffwll.ch] On Behalf Of Daniel
>>> Vetter
>>> Sent: Saturday, March 14, 2015 1:14 AM
>>> To: Chris Wilson; Daniel Vetter; Weinehall, David; Zou, Nanhai; Song, Ruiling;
>>> Vetter, Daniel; intel-gfx@lists.freedesktop.org; Yang, Rong R;
>>> beignet@lists.freedesktop.org
>>> Subject: Re: [Beignet] [Intel-gfx] Preventing zero GPU virtual address
>>> allocation
>>>
>>> On Fri, Mar 13, 2015 at 04:58:47PM +0000, Chris Wilson wrote:
>>>> On Fri, Mar 13, 2015 at 10:27:38AM +0100, Daniel Vetter wrote:
>>>>> If supporting systems without full ppgtt is a requirement for you
>>>>> (still wonky on gen8 a bit, so might be a good strategy) then imo
>>>>> it's the PIN_BIAS idea I've laid out earlier in this thread. That
>>>>> one will work everywhere. softpin can unexpectedly fail without full
>>>>> ppgtt if the kernel decides to put something at a given spot, which
>>>>> imo means we should only expose it on full ppgtt systems.
>>>>>
>>>>> And PIN_BIAS should be fairly easy to wire up since the internal
>>>>> logic is all there already. So "just" needs an execbuf flag, igt
>>>>> test and appropriate userspace to set that new bit.
>>>>
>>>> It doesn't though. To provide the guarantee userspace is asking for
>>>> (which is that address 0 goes to a special, preferrably inaccessible,
>>>> page), you have to evict the first N pages in the GGTT. That is just
>>>> as likely to fail with an execbuffer flag as it would with an execobject flag.
>>>
>>> Afaiui userspace only needs the guarantee that NULL is never a valid address.
>>> Which means it's never a valid address for its own buffer objects. I don't
>>> think it cares one bit what's actually there, it's not mandatory to fault
>>> apparently. And faulting is what's not possible.
>> Yes, This is what exactly what we need, that is make NULL as an invalid address. It's just like C language.
>> But I have some more comment. The buffer object used in opencl may be allocated in libva/opengl and shared for opencl usage through some opencl extension.
>> Afaiui, this would implicitly require libva/mesa also avoid zero-address buffer object allocation.
>> Will libdrm re-bind such kind of shared buffer object to a new graphics virtual address?
>> So that PIN_BIAS is also effective on the shared buffer, right?
> 
> Yeah we'll rebind if needed. We can make this an execbuf or context flag,
> in either case anything that gets executed by ocl will be moved around if
> it accidentally ended up at the wrong place. The only exception is if a
> buffer is pinned already, i.e. if you're doing direct rendering to the
> frontbuffer. That will give you an EBUSY, but otoh that also shouldn't
> ever happen really.

Oh and we may need to re-introduce the ppgtt zero page reservation
anyway due to bugs, but that's another topic...

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Intel-gfx] Preventing zero GPU virtual address allocation

[Zhigang Gong]

On Mon, Mar 16, 2015 at 01:10:28PM -0700, Jesse Barnes wrote:
> On 03/16/2015 01:52 AM, Daniel Vetter wrote:
> > On Mon, Mar 16, 2015 at 02:29:24AM +0000, Song, Ruiling wrote:
> >>
> >>
> >>> -----Original Message-----
> >>> From: Daniel Vetter [mailto:daniel.vetter@ffwll.ch] On Behalf Of Daniel
> >>> Vetter
> >>> Sent: Saturday, March 14, 2015 1:14 AM
> >>> To: Chris Wilson; Daniel Vetter; Weinehall, David; Zou, Nanhai; Song, Ruiling;
> >>> Vetter, Daniel; intel-gfx@lists.freedesktop.org; Yang, Rong R;
> >>> beignet@lists.freedesktop.org
> >>> Subject: Re: [Beignet] [Intel-gfx] Preventing zero GPU virtual address
> >>> allocation
> >>>
> >>> On Fri, Mar 13, 2015 at 04:58:47PM +0000, Chris Wilson wrote:
> >>>> On Fri, Mar 13, 2015 at 10:27:38AM +0100, Daniel Vetter wrote:
> >>>>> If supporting systems without full ppgtt is a requirement for you
> >>>>> (still wonky on gen8 a bit, so might be a good strategy) then imo
> >>>>> it's the PIN_BIAS idea I've laid out earlier in this thread. That
> >>>>> one will work everywhere. softpin can unexpectedly fail without full
> >>>>> ppgtt if the kernel decides to put something at a given spot, which
> >>>>> imo means we should only expose it on full ppgtt systems.
> >>>>>
> >>>>> And PIN_BIAS should be fairly easy to wire up since the internal
> >>>>> logic is all there already. So "just" needs an execbuf flag, igt
> >>>>> test and appropriate userspace to set that new bit.
> >>>>
> >>>> It doesn't though. To provide the guarantee userspace is asking for
> >>>> (which is that address 0 goes to a special, preferrably inaccessible,
> >>>> page), you have to evict the first N pages in the GGTT. That is just
> >>>> as likely to fail with an execbuffer flag as it would with an execobject flag.
> >>>
> >>> Afaiui userspace only needs the guarantee that NULL is never a valid address.
> >>> Which means it's never a valid address for its own buffer objects. I don't
> >>> think it cares one bit what's actually there, it's not mandatory to fault
> >>> apparently. And faulting is what's not possible.
> >> Yes, This is what exactly what we need, that is make NULL as an invalid address. It's just like C language.
> >> But I have some more comment. The buffer object used in opencl may be allocated in libva/opengl and shared for opencl usage through some opencl extension.
> >> Afaiui, this would implicitly require libva/mesa also avoid zero-address buffer object allocation.
> >> Will libdrm re-bind such kind of shared buffer object to a new graphics virtual address?
> >> So that PIN_BIAS is also effective on the shared buffer, right?
> > 
> > Yeah we'll rebind if needed. We can make this an execbuf or context flag,
> > in either case anything that gets executed by ocl will be moved around if
> > it accidentally ended up at the wrong place. The only exception is if a
> > buffer is pinned already, i.e. if you're doing direct rendering to the
> > frontbuffer. That will give you an EBUSY, but otoh that also shouldn't
> > ever happen really.
> 
> Ruiling, are you working on this or someone from your team, presumably
> based on the patch Chris posted earlier?  The zero page reservation
> certainly seems simpler to me, but the MAP_FIXED approach is a lot more
> flexible, and can be used for other types of debug and usages as well
> (we'll need something like it for OCL pointer sharing for example), so
> seems like a good thing to pursue regardless.

Jesse, here is my 2 cent comment:

IMO, the start of this discussion is from a very simple issue:
1. Make sure any valid buffer object will not be mapped to zero page.
   OCL wants to avoid 0 virtual address for any valid buffer object. Thus
   the following type of code will work as expected.
   __kernel foo(__global int *src, __global int* dst) {
     if (dst == NULL)
       return;
   }
   Although the above type code may not make much sense, but it should
   be supported as well.
   This is a simple requirement for all the bo's binding operations from
   CL application. And we think that simply skip the zero-page in KMD is
   a good enough solution and don't see any side-effect till now.

Then let's talk about the MAP_FIXED approach:
2. About the MAP_FIXED approach.
  If my understanding is correct, this is a intermediate solution towards FULL
  SVM function.
  From OCL's view, it is usefull for the 2.0's. There are three types of SVM
  options. Even for the most basic "Coarse-Grained buffer SVM", something like
  MAP_FIXED is useful. Becuase it needs to pass a linked list object to the
  OCL kernel directly. If we can use MAP_FIXED to map the bo to the VM address
  which is used by host, then we can access such a linked list easily on OCL
  kernel side, otherwise, we may need some tricks to do extra address adjustment.
  So this feature is useful for Beignet before we get full SVM support.

My conclusion is:
* The 1st is a totally *passive* requirement and need to be applied to each
  buffer binding. And it's better to be transparent to userspace.
* The 2nd is an *active* requirement and need the kernel to provide SVM like
  interface to the user space. And only those SVM buffers (for the first 2 OCL
  SVM options) need to call the new interfaces explicitly.

Thanks,
Zhigang Gong.

> 
> Thanks,
> Jesse
> 
> _______________________________________________
> Beignet mailing list
> Beignet@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/beignet
_______________________________________________
Beignet mailing list
Beignet@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/beignet

Re: [Intel-gfx] Preventing zero GPU virtual address allocation

[Zou, Nanhai]

> -----Original Message-----
> From: Intel-gfx [mailto:intel-gfx-bounces@lists.freedesktop.org] On Behalf Of
> Jesse Barnes
> Sent: Tuesday, March 17, 2015 4:10 AM
> To: Daniel Vetter; Song, Ruiling
> Cc: Vetter, Daniel; intel-gfx@lists.freedesktop.org; Yang, Rong R;
> beignet@lists.freedesktop.org; Weinehall, David
> Subject: Re: [Intel-gfx] [Beignet] Preventing zero GPU virtual address allocation
> 
> On 03/16/2015 01:52 AM, Daniel Vetter wrote:
> > On Mon, Mar 16, 2015 at 02:29:24AM +0000, Song, Ruiling wrote:
> >>
> >>
> >>> -----Original Message-----
> >>> From: Daniel Vetter [mailto:daniel.vetter@ffwll.ch] On Behalf Of
> >>> Daniel Vetter
> >>> Sent: Saturday, March 14, 2015 1:14 AM
> >>> To: Chris Wilson; Daniel Vetter; Weinehall, David; Zou, Nanhai;
> >>> Song, Ruiling; Vetter, Daniel; intel-gfx@lists.freedesktop.org;
> >>> Yang, Rong R; beignet@lists.freedesktop.org
> >>> Subject: Re: [Beignet] [Intel-gfx] Preventing zero GPU virtual
> >>> address allocation
> >>>
> >>> On Fri, Mar 13, 2015 at 04:58:47PM +0000, Chris Wilson wrote:
> >>>> On Fri, Mar 13, 2015 at 10:27:38AM +0100, Daniel Vetter wrote:
> >>>>> If supporting systems without full ppgtt is a requirement for you
> >>>>> (still wonky on gen8 a bit, so might be a good strategy) then imo
> >>>>> it's the PIN_BIAS idea I've laid out earlier in this thread. That
> >>>>> one will work everywhere. softpin can unexpectedly fail without
> >>>>> full ppgtt if the kernel decides to put something at a given spot,
> >>>>> which imo means we should only expose it on full ppgtt systems.
> >>>>>
> >>>>> And PIN_BIAS should be fairly easy to wire up since the internal
> >>>>> logic is all there already. So "just" needs an execbuf flag, igt
> >>>>> test and appropriate userspace to set that new bit.
> >>>>
> >>>> It doesn't though. To provide the guarantee userspace is asking for
> >>>> (which is that address 0 goes to a special, preferrably
> >>>> inaccessible, page), you have to evict the first N pages in the
> >>>> GGTT. That is just as likely to fail with an execbuffer flag as it would with
> an execobject flag.
> >>>
> >>> Afaiui userspace only needs the guarantee that NULL is never a valid
> address.
> >>> Which means it's never a valid address for its own buffer objects. I
> >>> don't think it cares one bit what's actually there, it's not
> >>> mandatory to fault apparently. And faulting is what's not possible.
> >> Yes, This is what exactly what we need, that is make NULL as an invalid
> address. It's just like C language.
> >> But I have some more comment. The buffer object used in opencl may be
> allocated in libva/opengl and shared for opencl usage through some opencl
> extension.
> >> Afaiui, this would implicitly require libva/mesa also avoid zero-address buffer
> object allocation.
> >> Will libdrm re-bind such kind of shared buffer object to a new graphics virtual
> address?
> >> So that PIN_BIAS is also effective on the shared buffer, right?
> >
> > Yeah we'll rebind if needed. We can make this an execbuf or context
> > flag, in either case anything that gets executed by ocl will be moved
> > around if it accidentally ended up at the wrong place. The only
> > exception is if a buffer is pinned already, i.e. if you're doing
> > direct rendering to the frontbuffer. That will give you an EBUSY, but
> > otoh that also shouldn't ever happen really.
> 
> Ruiling, are you working on this or someone from your team, presumably based
> on the patch Chris posted earlier?  The zero page reservation certainly seems
> simpler to me, but the MAP_FIXED approach is a lot more flexible, and can be
> used for other types of debug and usages as well (we'll need something like it
> for OCL pointer sharing for example), so seems like a good thing to pursue
> regardless.
> 
Hi Jesse,
	MAP_FIXED cannot solve this issue. You may see my previous comments for this topic, 
There could be many components in on single process, Beignet cannot be guaranteed to be the first one who has allocated address 0.

Thanks
Zou Nanhai

> Thanks,
> Jesse
> 
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Beignet mailing list
Beignet@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/beignet

Re: [Beignet] Preventing zero GPU virtual address allocation

[Daniel Vetter]

On Mon, Mar 16, 2015 at 01:10:28PM -0700, Jesse Barnes wrote:
> On 03/16/2015 01:52 AM, Daniel Vetter wrote:
> > On Mon, Mar 16, 2015 at 02:29:24AM +0000, Song, Ruiling wrote:
> >>
> >>
> >>> -----Original Message-----
> >>> From: Daniel Vetter [mailto:daniel.vetter@ffwll.ch] On Behalf Of Daniel
> >>> Vetter
> >>> Sent: Saturday, March 14, 2015 1:14 AM
> >>> To: Chris Wilson; Daniel Vetter; Weinehall, David; Zou, Nanhai; Song, Ruiling;
> >>> Vetter, Daniel; intel-gfx@lists.freedesktop.org; Yang, Rong R;
> >>> beignet@lists.freedesktop.org
> >>> Subject: Re: [Beignet] [Intel-gfx] Preventing zero GPU virtual address
> >>> allocation
> >>>
> >>> On Fri, Mar 13, 2015 at 04:58:47PM +0000, Chris Wilson wrote:
> >>>> On Fri, Mar 13, 2015 at 10:27:38AM +0100, Daniel Vetter wrote:
> >>>>> If supporting systems without full ppgtt is a requirement for you
> >>>>> (still wonky on gen8 a bit, so might be a good strategy) then imo
> >>>>> it's the PIN_BIAS idea I've laid out earlier in this thread. That
> >>>>> one will work everywhere. softpin can unexpectedly fail without full
> >>>>> ppgtt if the kernel decides to put something at a given spot, which
> >>>>> imo means we should only expose it on full ppgtt systems.
> >>>>>
> >>>>> And PIN_BIAS should be fairly easy to wire up since the internal
> >>>>> logic is all there already. So "just" needs an execbuf flag, igt
> >>>>> test and appropriate userspace to set that new bit.
> >>>>
> >>>> It doesn't though. To provide the guarantee userspace is asking for
> >>>> (which is that address 0 goes to a special, preferrably inaccessible,
> >>>> page), you have to evict the first N pages in the GGTT. That is just
> >>>> as likely to fail with an execbuffer flag as it would with an execobject flag.
> >>>
> >>> Afaiui userspace only needs the guarantee that NULL is never a valid address.
> >>> Which means it's never a valid address for its own buffer objects. I don't
> >>> think it cares one bit what's actually there, it's not mandatory to fault
> >>> apparently. And faulting is what's not possible.
> >> Yes, This is what exactly what we need, that is make NULL as an invalid address. It's just like C language.
> >> But I have some more comment. The buffer object used in opencl may be allocated in libva/opengl and shared for opencl usage through some opencl extension.
> >> Afaiui, this would implicitly require libva/mesa also avoid zero-address buffer object allocation.
> >> Will libdrm re-bind such kind of shared buffer object to a new graphics virtual address?
> >> So that PIN_BIAS is also effective on the shared buffer, right?
> > 
> > Yeah we'll rebind if needed. We can make this an execbuf or context flag,
> > in either case anything that gets executed by ocl will be moved around if
> > it accidentally ended up at the wrong place. The only exception is if a
> > buffer is pinned already, i.e. if you're doing direct rendering to the
> > frontbuffer. That will give you an EBUSY, but otoh that also shouldn't
> > ever happen really.
> 
> Ruiling, are you working on this or someone from your team, presumably
> based on the patch Chris posted earlier?  The zero page reservation
> certainly seems simpler to me, but the MAP_FIXED approach is a lot more
> flexible, and can be used for other types of debug and usages as well
> (we'll need something like it for OCL pointer sharing for example), so
> seems like a good thing to pursue regardless.

I prefer not to merge fix MAP_FIXED with the justification that ocl will
need the full power of it for buffered svm, without the pieces really
being ready. And I don't think we should block this little fix for current
ocl on the svm work either. Hence why I suggested to just expose the
already existing PIN_BIAS support somehow. That also has the upside of
working without full ppgtt (i.e. on hsw and earlier).
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Beignet] Preventing zero GPU virtual address allocation

[Jesse Barnes]

>>> Yeah we'll rebind if needed. We can make this an execbuf or context
>>> flag, in either case anything that gets executed by ocl will be moved
>>> around if it accidentally ended up at the wrong place. The only
>>> exception is if a buffer is pinned already, i.e. if you're doing
>>> direct rendering to the frontbuffer. That will give you an EBUSY, but
>>> otoh that also shouldn't ever happen really.
>>
>> Ruiling, are you working on this or someone from your team, presumably based
>> on the patch Chris posted earlier?  The zero page reservation certainly seems
>> simpler to me, but the MAP_FIXED approach is a lot more flexible, and can be
>> used for other types of debug and usages as well (we'll need something like it
>> for OCL pointer sharing for example), so seems like a good thing to pursue
>> regardless.
>>
> Hi Jesse,
> 	MAP_FIXED cannot solve this issue. You may see my previous comments for this topic, 
> There could be many components in on single process, Beignet cannot be guaranteed to be the first one who has allocated address 0.

Yeah, MAP_FIXED sounds a bit more ambitious and though I think it would
work for OCL 2.0 pointer sharing, it's a little different than we were
planning.  To summarize, we have three possible approaches, each with
its own problems:
  1) simple patch to avoid binding at address 0 in PPGTT:
     does impact the ABI (though generally not in a harmful way), and
     may not be possible with aliasing PPGTT with e.g. framebuffers
     bound at offset 0
  2) exposing PIN_BIAS to userspace
     Would allow userspace to avoid pinning any buffers at offset 0 at
     execbuf time, but still has the problem with previously bound buffers
     and aliasing PPGTT
  3) MAP_FIXED interface
     Flexible approach allowing userspace to manage its own virtual
     memory, but still has the same issues with aliasing PPGTT, and with
     shared contexts, which would have to negotiate between libraries how to
     handle the zero page

For (1) and (2) the kernel pieces are really already in place, the main
thing we need is a new flag to userspace to indicate behavior.  I'd
prefer (1) with a context creation flag to indicate "don't bind at 0".
Execbuf would try to honor this, and userspace could check if any
buffers ended up at 0 in the aliasing PPGTT case by checking the
resulting offsets following the call.  I expect in most cases this would
be fine.

It should be pretty easy to extend Ruiling's patch to use a context flag
to determine the behavior; is that something you can do?  Any objections
to this approach?

It does mean that shared contexts need to be handled specially, or won't
get the 0 page protection, but I think Mesa wants this behavior too, and
libva probably wouldn't mind, so you could just require new versions of
those that set this flag when telling people what's supported for proper
NULL pointer handling.

Any objections to that approach?

Thanks,
Jesse
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Intel-gfx] Preventing zero GPU virtual address allocation

[Song, Ruiling]

> Yeah, MAP_FIXED sounds a bit more ambitious and though I think it would
> work for OCL 2.0 pointer sharing, it's a little different than we were planning.
> To summarize, we have three possible approaches, each with its own
> problems:
>   1) simple patch to avoid binding at address 0 in PPGTT:
>      does impact the ABI (though generally not in a harmful way), and
>      may not be possible with aliasing PPGTT with e.g. framebuffers
>      bound at offset 0
>   2) exposing PIN_BIAS to userspace
>      Would allow userspace to avoid pinning any buffers at offset 0 at
>      execbuf time, but still has the problem with previously bound buffers
>      and aliasing PPGTT
>   3) MAP_FIXED interface
>      Flexible approach allowing userspace to manage its own virtual
>      memory, but still has the same issues with aliasing PPGTT, and with
>      shared contexts, which would have to negotiate between libraries
> how to
>      handle the zero page
> 
> For (1) and (2) the kernel pieces are really already in place, the main thing we
> need is a new flag to userspace to indicate behavior.  I'd prefer (1) with a
> context creation flag to indicate "don't bind at 0".
> Execbuf would try to honor this, and userspace could check if any buffers
> ended up at 0 in the aliasing PPGTT case by checking the resulting offsets
> following the call.  I expect in most cases this would be fine.
> 
> It should be pretty easy to extend Ruiling's patch to use a context flag to
> determine the behavior; is that something you can do?  Any objections to
> this approach?

I am ok with adding a context flag to indicate "don't bind at 0". Any objections from others?
The patch is not from me, it is from David. I am not familiar with KMD. David, could you help on this patch?

> It does mean that shared contexts need to be handled specially, or won't get
> the 0 page protection, but I think Mesa wants this behavior too, and libva
> probably wouldn't mind, so you could just require new versions of those that
> set this flag when telling people what's supported for proper NULL pointer
> handling.
> 
> Any objections to that approach?
> 
> Thanks,
> Jesse
_______________________________________________
Beignet mailing list
Beignet@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/beignet

Re: [Beignet] Preventing zero GPU virtual address allocation

[David Weinehall]

On Thu, Mar 19, 2015 at 03:22:42AM +0000, Song, Ruiling wrote:
> 
> > Yeah, MAP_FIXED sounds a bit more ambitious and though I think it would
> > work for OCL 2.0 pointer sharing, it's a little different than we were planning.
> > To summarize, we have three possible approaches, each with its own
> > problems:
> >   1) simple patch to avoid binding at address 0 in PPGTT:
> >      does impact the ABI (though generally not in a harmful way), and
> >      may not be possible with aliasing PPGTT with e.g. framebuffers
> >      bound at offset 0
> >   2) exposing PIN_BIAS to userspace
> >      Would allow userspace to avoid pinning any buffers at offset 0 at
> >      execbuf time, but still has the problem with previously bound buffers
> >      and aliasing PPGTT
> >   3) MAP_FIXED interface
> >      Flexible approach allowing userspace to manage its own virtual
> >      memory, but still has the same issues with aliasing PPGTT, and with
> >      shared contexts, which would have to negotiate between libraries
> > how to
> >      handle the zero page
> > 
> > For (1) and (2) the kernel pieces are really already in place, the main thing we
> > need is a new flag to userspace to indicate behavior.  I'd prefer (1) with a
> > context creation flag to indicate "don't bind at 0".
> > Execbuf would try to honor this, and userspace could check if any buffers
> > ended up at 0 in the aliasing PPGTT case by checking the resulting offsets
> > following the call.  I expect in most cases this would be fine.
> > 
> > It should be pretty easy to extend Ruiling's patch to use a context flag to
> > determine the behavior; is that something you can do?  Any objections to
> > this approach?
> 
> I am ok with adding a context flag to indicate "don't bind at 0". Any objections from others?
> The patch is not from me, it is from David. I am not familiar with KMD. David, could you help on this patch?

Yup, assuming, of course, that such an approach is acceptable.


Kind regards, David
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Beignet] Preventing zero GPU virtual address allocation

[Daniel Vetter]

On Thu, Mar 19, 2015 at 12:09:53PM +0200, David Weinehall wrote:
> On Thu, Mar 19, 2015 at 03:22:42AM +0000, Song, Ruiling wrote:
> > 
> > > Yeah, MAP_FIXED sounds a bit more ambitious and though I think it would
> > > work for OCL 2.0 pointer sharing, it's a little different than we were planning.
> > > To summarize, we have three possible approaches, each with its own
> > > problems:
> > >   1) simple patch to avoid binding at address 0 in PPGTT:
> > >      does impact the ABI (though generally not in a harmful way), and
> > >      may not be possible with aliasing PPGTT with e.g. framebuffers
> > >      bound at offset 0
> > >   2) exposing PIN_BIAS to userspace
> > >      Would allow userspace to avoid pinning any buffers at offset 0 at
> > >      execbuf time, but still has the problem with previously bound buffers
> > >      and aliasing PPGTT
> > >   3) MAP_FIXED interface
> > >      Flexible approach allowing userspace to manage its own virtual
> > >      memory, but still has the same issues with aliasing PPGTT, and with
> > >      shared contexts, which would have to negotiate between libraries
> > > how to
> > >      handle the zero page
> > > 
> > > For (1) and (2) the kernel pieces are really already in place, the main thing we
> > > need is a new flag to userspace to indicate behavior.  I'd prefer (1) with a
> > > context creation flag to indicate "don't bind at 0".
> > > Execbuf would try to honor this, and userspace could check if any buffers
> > > ended up at 0 in the aliasing PPGTT case by checking the resulting offsets
> > > following the call.  I expect in most cases this would be fine.
> > > 
> > > It should be pretty easy to extend Ruiling's patch to use a context flag to
> > > determine the behavior; is that something you can do?  Any objections to
> > > this approach?
> > 
> > I am ok with adding a context flag to indicate "don't bind at 0". Any objections from others?
> > The patch is not from me, it is from David. I am not familiar with KMD. David, could you help on this patch?
> 
> Yup, assuming, of course, that such an approach is acceptable.

Yeah my big concern was with not making this opt-in like the old patch or
adding an interface which does a lot more than what we need right now
(Chris' patch). Just a bitflag to ask for this seems best and is fine with
me.

And for the implementation I think we should reuse the PIN_BIAS logic
since that'll work in all places where it's possible. One open from my
side is how we should handle failures to move buffers (in case they ended
up at 0 somehow) - we can either silently fail or return an error to
userspace.

Note that this is only possible if you render to an elg image from ocl,
and if that egl image is a pinned frontbuffer and if we don't have full
ppgtt support. I don't know what the spec requires us to do here, or
whether we should care at all.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Beignet] Preventing zero GPU virtual address allocation

[Song, Ruiling]

> Yeah my big concern was with not making this opt-in like the old patch or
> adding an interface which does a lot more than what we need right now
> (Chris' patch). Just a bitflag to ask for this seems best and is fine with me.
> 
> And for the implementation I think we should reuse the PIN_BIAS logic since
> that'll work in all places where it's possible. One open from my side is how
> we should handle failures to move buffers (in case they ended up at 0
> somehow) - we can either silently fail or return an error to userspace.
> 
> Note that this is only possible if you render to an elg image from ocl, and if
> that egl image is a pinned frontbuffer and if we don't have full ppgtt support.
> I don't know what the spec requires us to do here, or whether we should
> care at all.
So the situation you mentioned only comes up when a pinned buffer under global gtt?
Under global gtt, a buffer would rarely binded at offset 0, in fact the most often cases are under ppgtt.
So, I think silent ignore moving pinned buffer under gtt is acceptable. I will add the check for the binded at zero case in beignet.

Ruiling
> -Daniel
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch
> _______________________________________________
> Beignet mailing list
> Beignet@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/beignet
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx