RFC: Driver for Oracle Data Analytics Accelerator

RFC: Driver for Oracle Data Analytics Accelerator

[Rob Gardner]

Recent Oracle Sparc processors (M7 and M8) have a coprocessor which
lives on the cpu chip. The coprocessor is called DAX, and is
controlled via sun4v hypercalls. The programmatic interface to the
coprocessor is somewhat unorthodox, and is described in detail in the
documentation file. The machine descriptor identifies the device as
"dax", and all internal documentation refers to it as "dax". But since
the term "dax" already has other meanings and uses, we propose to call
this driver "oradax"; other suggestions are welcome.

To answer the most obvious question "who will use this?", there is a
companion userspace library which will be published under UPL
shortly. This is a comprehensive collection of higher level functions
along with tests and documentation. We'll include a link to this before
the code is submitted as a patch.

Apart from general comments about the code, style, formatting, logic,
etc., we seek suggestions about where the source files ought to
live. Right now we've got the main driver source file in
drivers/sbus/char, but seeing as "sbus" is a very old Sparc feature
and not used anymore, this doesn't seem to be a great place for it.

Any and all feedback welcome and appreciated.

Thanks,
Rob Gardner
Sanath Kumar
Jonathan Helman

Re: RFC: Driver for Oracle Data Analytics Accelerator

[David Miller]

From: Rob Gardner <rob.gardner@oracle.com>
Date: Tue, 29 Aug 2017 17:36:01 -0600

> Any and all feedback welcome and appreciated.

So basically binary blobs will be passed into this driver, and there
will be no verification or documentation of the various DAX
operations, their parameters, what operation they perform, how they
work, and their how their references to the user's buffers operate?

Right?

Sorry, no way.

Re: RFC: Driver for Oracle Data Analytics Accelerator

[Rob Gardner]

On 08/29/2017 07:07 PM, David Miller wrote:
> From: Rob Gardner <rob.gardner@oracle.com>
> Date: Tue, 29 Aug 2017 17:36:01 -0600
>
>> Any and all feedback welcome and appreciated.
> So basically binary blobs will be passed into this driver, and there
> will be no verification or documentation of the various DAX
> operations, their parameters, what operation they perform, how they
> work, and their how their references to the user's buffers operate?
>
> Right?
>

No, that is not at all how it works. Full documentation of the command 
interface will be provided in the library. The driver is just a 
transport to connect user space with the hypervisor calls.

Rob

Re: RFC: Driver for Oracle Data Analytics Accelerator

[David Miller]

From: Rob Gardner <rob.gardner@oracle.com>
Date: Tue, 29 Aug 2017 19:47:23 -0600

> On 08/29/2017 07:07 PM, David Miller wrote:
>> From: Rob Gardner <rob.gardner@oracle.com>
>> Date: Tue, 29 Aug 2017 17:36:01 -0600
>>
>>> Any and all feedback welcome and appreciated.
>> So basically binary blobs will be passed into this driver, and there
>> will be no verification or documentation of the various DAX
>> operations, their parameters, what operation they perform, how they
>> work, and their how their references to the user's buffers operate?
>>
>> Right?
>>
> 
> No, that is not at all how it works. Full documentation of the command
> interface will be provided in the library. The driver is just a
> transport to connect user space with the hypervisor calls.

Again, my point is that what the kernel processes is basically
semanticless.

The kernel doesn't validate the DAX commands, it doesn't validate
the parameters given to the commands, etc.

It's a block box, accepting binary blobs, and passing them on
as-is to the hypervisor.

That's what I don't like about this interface.

Re: RFC: Driver for Oracle Data Analytics Accelerator

[Rob Gardner]

On 08/29/2017 08:33 PM, David Miller wrote:
> From: Rob Gardner <rob.gardner@oracle.com>
> Date: Tue, 29 Aug 2017 19:47:23 -0600
>
>> On 08/29/2017 07:07 PM, David Miller wrote:
>>> From: Rob Gardner <rob.gardner@oracle.com>
>>> Date: Tue, 29 Aug 2017 17:36:01 -0600
>>>
>>>> Any and all feedback welcome and appreciated.
>>> So basically binary blobs will be passed into this driver, and there
>>> will be no verification or documentation of the various DAX
>>> operations, their parameters, what operation they perform, how they
>>> work, and their how their references to the user's buffers operate?
>>>
>>> Right?
>>>
>> No, that is not at all how it works. Full documentation of the command
>> interface will be provided in the library. The driver is just a
>> transport to connect user space with the hypervisor calls.
> Again, my point is that what the kernel processes is basically
> semanticless.
>
> The kernel doesn't validate the DAX commands, it doesn't validate
> the parameters given to the commands, etc.
>
> It's a block box, accepting binary blobs, and passing them on
> as-is to the hypervisor.

Not quite as-is, since the driver does check address types to make sure 
the user is not passing in a real address. But yes, apart from that, it 
is a pass through driver, much like a scsi pass through driver; a user 
app can construct a command block and use the driver to get it to the 
hardware. A scsi pass through driver does not validate that the scsi ccb 
has valid commands, parameters, etc, but allows the hardware to do that 
and decide to accept it or reject it. As long as we are sure that the 
user cannot accomplish anything malicious, we don't need to duplicate 
validation that the hardware already does for us.

Perhaps it would seem less like a black box with further explanation of 
the operations, or perhaps some example programs that use the driver 
interface. Would that help? The library that goes along with this driver 
has extensive documentation and it will be open source very soon, but we 
can certainly provide further detail before that happens.

Thank you for your feedback. I am already revising the documentation to 
better explain the interface, and eager to hear more of your thinking on 
this.

Rob

Re: RFC: Driver for Oracle Data Analytics Accelerator

[John Paul Adrian Glaubitz]

Hi Rob!

On 08/30/2017 01:36 AM, Rob Gardner wrote:
> To answer the most obvious question "who will use this?", there is a
> companion userspace library which will be published under UPL
> shortly. This is a comprehensive collection of higher level functions
> along with tests and documentation. We'll include a link to this before
> the code is submitted as a patch.

Out of curiosity. Do you know what the current progress with the publication
of the patches for LDOM control support on Linux are? I have seen that Oracle
Linux now ships the LDOM utilities, so I assume that it is possible to run
Oracle Linux in the control domain. But since the sources are missing for
the LDOM packages in OL, it's not yet possible to package it for Debian.

It would be really nice to have Debian running in the control domain,
currently we're still running Solaris there on the SPARC T5 we have.

Adrian

-- 
  .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
   `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: RFC: Driver for Oracle Data Analytics Accelerator

[Rob Gardner]

On 08/30/2017 01:22 AM, John Paul Adrian Glaubitz wrote:
> Hi Rob!
>
> On 08/30/2017 01:36 AM, Rob Gardner wrote:
>> To answer the most obvious question "who will use this?", there is a
>> companion userspace library which will be published under UPL
>> shortly. This is a comprehensive collection of higher level functions
>> along with tests and documentation. We'll include a link to this before
>> the code is submitted as a patch.
>
> Out of curiosity. Do you know what the current progress with the 
> publication
> of the patches for LDOM control support on Linux are? I have seen that 
> Oracle
> Linux now ships the LDOM utilities, so I assume that it is possible to 
> run
> Oracle Linux in the control domain. But since the sources are missing for
> the LDOM packages in OL, it's not yet possible to package it for Debian.
>
> It would be really nice to have Debian running in the control domain,
> currently we're still running Solaris there on the SPARC T5 we have.
>
> Adrian
>

I do not know offhand the state of any ldom control support patches. It 
is definitely possible to run linux in the control domain, I've seen it 
being done. I hope they plan on sharing the sources for that.

Rob

Re: RFC: Driver for Oracle Data Analytics Accelerator

[Alexandre Chartre]

On 08/31/2017 05:50 AM, Rob Gardner wrote:
> On 08/30/2017 01:22 AM, John Paul Adrian Glaubitz wrote:
>> Hi Rob!
>>
>> On 08/30/2017 01:36 AM, Rob Gardner wrote:
>>> To answer the most obvious question "who will use this?", there is a
>>> companion userspace library which will be published under UPL
>>> shortly. This is a comprehensive collection of higher level functions
>>> along with tests and documentation. We'll include a link to this before
>>> the code is submitted as a patch.
>>
>> Out of curiosity. Do you know what the current progress with the publication
>> of the patches for LDOM control support on Linux are? I have seen that Oracle
>> Linux now ships the LDOM utilities, so I assume that it is possible to run
>> Oracle Linux in the control domain. But since the sources are missing for
>> the LDOM packages in OL, it's not yet possible to package it for Debian.
>>
>> It would be really nice to have Debian running in the control domain,
>> currently we're still running Solaris there on the SPARC T5 we have.
>>
>> Adrian
>>
>
> I do not know offhand the state of any ldom control support patches.
> It is definitely possible to run linux in the control domain, I've
> seen it being done. I hope they plan on sharing the sources for
> that.
>

It's possible to run Linux as a LDoms control domain with the Oracle SPARC UEK kernel.

See here: https://docs.oracle.com/cd/E37670_01/E86243/html/InstallControlDomTask.html

However, all required drivers are not upstream yet. In particular, we have
changes to the ds driver, and new vldc and vlds driver. We are working on
upstreaming those drivers which are required to run the LDoms Manager.

In addition, you need the ldoms and the ldomsmanager packages which are available here:

Binary: http://yum.oracle.com/repo/OracleLinux/OL6/latest/sparc64/index.html
Source: http://yum.oracle.com/repo/OracleLinux/OL6/latest/source/index_src.html

alex.