Users of SBE FIFO kernel driver

Users of SBE FIFO kernel driver

[Christopher Bostic]

I've not seen this discussed in the group but if this has already been 
determined than any pointers to the latest documentation would be 
appreciated.

Who are all the potential users of the OpenBMC SBE FIFO device driver?   
I understand it will need to export its general submit interface to 
other kernel drivers.  Only one I know of for sure would be the OCC 
driver Eddie James is working on.

Any others?

What about user space access of the SBE FIFO engine?  What apps would 
require it?  Cronus I assume will need some means to directly access it.


Thanks

Re: Users of SBE FIFO kernel driver

[Joel Stanley]

Hi Chris,

On Thu, Feb 2, 2017 at 5:58 AM, Christopher Bostic
<cbostic@linux.vnet.ibm.com> wrote:
> I've not seen this discussed in the group but if this has already been
> determined than any pointers to the latest documentation would be
> appreciated.
>
> Who are all the potential users of the OpenBMC SBE FIFO device driver?   I
> understand it will need to export its general submit interface to other
> kernel drivers.  Only one I know of for sure would be the OCC driver Eddie
> James is working on.

In addition to Eddie's driver, in userspace we will have the code that
performs the power on sequence.

> Any others?
>
> What about user space access of the SBE FIFO engine?  What apps would
> require it?  Cronus I assume will need some means to directly access it.

We want userspace API and an in-kernel API.

The driver you design will need to take into account that there will
be multiple users from each of these APIs. For instance, there will be
Eddie's OCC hwmon driver and the userspace code that kicks off the
boot sequence.

Cheers,

Joel

Re: Users of SBE FIFO kernel driver

[Benjamin Herrenschmidt]

On Thu, 2017-02-02 at 13:59 +1030, Joel Stanley wrote:
> The driver you design will need to take into account that there will
> be multiple users from each of these APIs. For instance, there will be
> Eddie's OCC hwmon driver and the userspace code that kicks off the
> boot sequence.

In addition, we expect host debugging tools running on the BMC using
that interface.

Cheers,
Ben.

Re: Users of SBE FIFO kernel driver

[Alistair Popple]

On Thu, 2 Feb 2017 01:59:08 PM Joel Stanley wrote:
> Hi Chris,
> 
> On Thu, Feb 2, 2017 at 5:58 AM, Christopher Bostic
> <cbostic@linux.vnet.ibm.com> wrote:
> > I've not seen this discussed in the group but if this has already been
> > determined than any pointers to the latest documentation would be
> > appreciated.
> >
> > Who are all the potential users of the OpenBMC SBE FIFO device driver?   I
> > understand it will need to export its general submit interface to other
> > kernel drivers.  Only one I know of for sure would be the OCC driver Eddie
> > James is working on.
> 
> In addition to Eddie's driver, in userspace we will have the code that
> performs the power on sequence.
> 
> > Any others?
> > What about user space access of the SBE FIFO engine?  What apps would
> > require it?  Cronus I assume will need some means to directly access it.
> 
> We want userspace API and an in-kernel API.

Yep. For example as Ben mentioned the chip-ops also use the SBE FIFO. However 
we certainly don't want to put chip-ops directly in the kernel. It makes more 
sense to use a userspace SBE FIFO API to do the chips-ops, power on sequence 
etc. from userspace.
 
> The driver you design will need to take into account that there will
> be multiple users from each of these APIs. For instance, there will be
> Eddie's OCC hwmon driver and the userspace code that kicks off the
> boot sequence.
> 
> Cheers,
> 
> Joel

Re: Users of SBE FIFO kernel driver

[Venkatesh Sainath]

I believe there are 2 drivers here. SBE FIFO driver and SBEI protocol 
driver. The SBEI protocol driver is the only user for the submit 
operation of SBE FIFO driver (as it requires the protocol knowledge to 
communicate with SBE). The SBEI protocol driver will be used by OCC and 
any user-space application that may want to perform a direct HW access 
over FSI  ( for eg any scom operation).


On 02/02/17 8:59 AM, Joel Stanley wrote:
> Hi Chris,
>
> On Thu, Feb 2, 2017 at 5:58 AM, Christopher Bostic
> <cbostic@linux.vnet.ibm.com> wrote:
>> I've not seen this discussed in the group but if this has already been
>> determined than any pointers to the latest documentation would be
>> appreciated.
>>
>> Who are all the potential users of the OpenBMC SBE FIFO device driver?   I
>> understand it will need to export its general submit interface to other
>> kernel drivers.  Only one I know of for sure would be the OCC driver Eddie
>> James is working on.
> In addition to Eddie's driver, in userspace we will have the code that
> performs the power on sequence.
>
>> Any others?
>>
>> What about user space access of the SBE FIFO engine?  What apps would
>> require it?  Cronus I assume will need some means to directly access it.
> We want userspace API and an in-kernel API.
>
> The driver you design will need to take into account that there will
> be multiple users from each of these APIs. For instance, there will be
> Eddie's OCC hwmon driver and the userspace code that kicks off the
> boot sequence.
>
> Cheers,
>
> Joel
> _______________________________________________
> openbmc mailing list
> openbmc@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/openbmc

Re: Users of SBE FIFO kernel driver

[Joel Stanley]

On Thu, Feb 2, 2017 at 3:18 PM, Alistair Popple <alistair@popple.id.au> wrote:
> On Thu, 2 Feb 2017 01:59:08 PM Joel Stanley wrote:
>> Hi Chris,
>>
>> On Thu, Feb 2, 2017 at 5:58 AM, Christopher Bostic
>> <cbostic@linux.vnet.ibm.com> wrote:
>> > I've not seen this discussed in the group but if this has already been
>> > determined than any pointers to the latest documentation would be
>> > appreciated.
>> >
>> > Who are all the potential users of the OpenBMC SBE FIFO device driver?   I
>> > understand it will need to export its general submit interface to other
>> > kernel drivers.  Only one I know of for sure would be the OCC driver Eddie
>> > James is working on.
>>
>> In addition to Eddie's driver, in userspace we will have the code that
>> performs the power on sequence.
>>
>> > Any others?
>> > What about user space access of the SBE FIFO engine?  What apps would
>> > require it?  Cronus I assume will need some means to directly access it.
>>
>> We want userspace API and an in-kernel API.
>
> Yep. For example as Ben mentioned the chip-ops also use the SBE FIFO. However
> we certainly don't want to put chip-ops directly in the kernel. It makes more
> sense to use a userspace SBE FIFO API to do the chips-ops, power on sequence
> etc. from userspace.
>
>> The driver you design will need to take into account that there will
>> be multiple users from each of these APIs. For instance, there will be
>> Eddie's OCC hwmon driver and the userspace code that kicks off the
>> boot sequence.

Agreed. This driver is to control access to the FIFO. We don't want to
have to modify the kernel every time a new operation is required.

The programs sending the messages will know how to encode their own
operations, such as pdbg, the occ hwmon driver, etc.

Cheers,

Joel

Re: Users of SBE FIFO kernel driver

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1282 bytes --]

On Thu, Feb 02, 2017 at 03:48:02PM +1100, Alistair Popple wrote:
> On Thu, 2 Feb 2017 01:59:08 PM Joel Stanley wrote:
> > On Thu, Feb 2, 2017 at 5:58 AM, Christopher Bostic
> > <cbostic@linux.vnet.ibm.com> wrote:
> > In addition to Eddie's driver, in userspace we will have the code that
> > performs the power on sequence.
> 
> Yep. For example as Ben mentioned the chip-ops also use the SBE FIFO. However 
> we certainly don't want to put chip-ops directly in the kernel. It makes more 
> sense to use a userspace SBE FIFO API to do the chips-ops, power on sequence 
> etc. from userspace.

What is being referred to here as the "power on sequence" that goes over
the SBE FIFO?  We do putcfams to start the power on sequence and there
is no interaction with the FIFO to the best of my knowledge.

On another topic, when we are under secureboot, get/putscom operations
need to go through the SBE FIFO.  Should there be a kernel driver for
SCOM with the same userspace API no matter which path we should go?
This certainly makes an application like pdbg easier.  One hang-up I can
think of is I am not sure how we are suppose to know and inform the
kernel if we can do direct SCOM or SBE-FIFO SCOM.  Maybe we always do
SBE-FIFO SCOM.

-- 
Patrick Williams

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: Users of SBE FIFO kernel driver

[Alistair Popple]

Hi,

On Wed, 1 Feb 2017 11:26:42 PM Patrick Williams wrote:
> On Thu, Feb 02, 2017 at 03:48:02PM +1100, Alistair Popple wrote:
> > On Thu, 2 Feb 2017 01:59:08 PM Joel Stanley wrote:
> > > On Thu, Feb 2, 2017 at 5:58 AM, Christopher Bostic
> > > <cbostic@linux.vnet.ibm.com> wrote:
> > > In addition to Eddie's driver, in userspace we will have the code that
> > > performs the power on sequence.
> > 
> > Yep. For example as Ben mentioned the chip-ops also use the SBE FIFO. 
However 
> > we certainly don't want to put chip-ops directly in the kernel. It makes 
more 
> > sense to use a userspace SBE FIFO API to do the chips-ops, power on 
sequence 
> > etc. from userspace.
> 
> What is being referred to here as the "power on sequence" that goes over
> the SBE FIFO?  We do putcfams to start the power on sequence and there
> is no interaction with the FIFO to the best of my knowledge.
> 
> On another topic, when we are under secureboot, get/putscom operations
> need to go through the SBE FIFO.  Should there be a kernel driver for
> SCOM with the same userspace API no matter which path we should go?

I'd be inclined to expose both the SBE-FIFO and direct SCOM access to 
userspace using different instances of the same API (eg. /dev/scomsbe 
/dev/scomdirect - although I'm sure there are better names). That way 
userspace programs can choose which method if required.

> This certainly makes an application like pdbg easier.  One hang-up I can
> think of is I am not sure how we are suppose to know and inform the
> kernel if we can do direct SCOM or SBE-FIFO SCOM.  Maybe we always do
> SBE-FIFO SCOM.

Exactly - everything can just use /dev/scomsbe by default as that should 
always work, secure boot or not. If tools like cronus or pdbg need direct 
access they can just switch to using /dev/scomdirect if needed/configured to.

- Alistair

Re: Users of SBE FIFO kernel driver

[Christopher Bostic]

On 2/1/17 10:48 PM, Venkatesh Sainath wrote:
> I believe there are 2 drivers here. SBE FIFO driver and SBEI protocol 
> driver. The SBEI protocol driver is the only user for the submit 
> operation of SBE FIFO driver (as it requires the protocol knowledge to 
> communicate with SBE). The SBEI protocol driver will be used by OCC 
> and any user-space application that may want to perform a direct HW 
> access over FSI  ( for eg any scom operation).
>
If I understand then the lowest level device driver in the kernel is the 
sbe_fifo driver running on OpenFSI.
The only thing that should have direct access to this is the sbei 
protocol driver - via sbe_fifo submit operations.

Any other code wishing to access the SBE FIFO engine must go through the 
sbei protocol driver.  This would include
OCC hwmon driver, SCOM device driver, as well as any user space 
applications like host debugger/chip ops (potentially boot sequence if 
in fact sbe fifo access is required).

sbei protocol driver must then provide a means for user space to access it.

Where do we stand with the sbei protocol device driver?  Who owns this 
and is it complete?

>
> On 02/02/17 8:59 AM, Joel Stanley wrote:
>> Hi Chris,
>>
>> On Thu, Feb 2, 2017 at 5:58 AM, Christopher Bostic
>> <cbostic@linux.vnet.ibm.com> wrote:
>>> I've not seen this discussed in the group but if this has already been
>>> determined than any pointers to the latest documentation would be
>>> appreciated.
>>>
>>> Who are all the potential users of the OpenBMC SBE FIFO device 
>>> driver?   I
>>> understand it will need to export its general submit interface to other
>>> kernel drivers.  Only one I know of for sure would be the OCC driver 
>>> Eddie
>>> James is working on.
>> In addition to Eddie's driver, in userspace we will have the code that
>> performs the power on sequence.
>>
>>> Any others?
>>>
>>> What about user space access of the SBE FIFO engine?  What apps would
>>> require it?  Cronus I assume will need some means to directly access 
>>> it.
>> We want userspace API and an in-kernel API.
>>
>> The driver you design will need to take into account that there will
>> be multiple users from each of these APIs. For instance, there will be
>> Eddie's OCC hwmon driver and the userspace code that kicks off the
>> boot sequence.
>>
>> Cheers,
>>
>> Joel
>> _______________________________________________
>> openbmc mailing list
>> openbmc@lists.ozlabs.org
>> https://lists.ozlabs.org/listinfo/openbmc
>
> _______________________________________________
> openbmc mailing list
> openbmc@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/openbmc

Re: Users of SBE FIFO kernel driver

[Venkatesh Sainath]

On 04/02/17 12:52 AM, Christopher Bostic wrote:
>
>
> On 2/1/17 10:48 PM, Venkatesh Sainath wrote:
>> I believe there are 2 drivers here. SBE FIFO driver and SBEI protocol 
>> driver. The SBEI protocol driver is the only user for the submit 
>> operation of SBE FIFO driver (as it requires the protocol knowledge 
>> to communicate with SBE). The SBEI protocol driver will be used by 
>> OCC and any user-space application that may want to perform a direct 
>> HW access over FSI  ( for eg any scom operation).
>>
> If I understand then the lowest level device driver in the kernel is 
> the sbe_fifo driver running on OpenFSI.
> The only thing that should have direct access to this is the sbei 
> protocol driver - via sbe_fifo submit operations.
>
> Any other code wishing to access the SBE FIFO engine must go through 
> the sbei protocol driver.  This would include
> OCC hwmon driver, SCOM device driver, as well as any user space 
> applications like host debugger/chip ops (potentially boot sequence if 
> in fact sbe fifo access is required).
>
> sbei protocol driver must then provide a means for user space to 
> access it.
I agree.
>
> Where do we stand with the sbei protocol device driver?  Who owns this 
> and is it complete?
>
The plan was to port the sbei protocol driver developed on FSP based 
systems. Chris: I believe Dhruvaraj discussed with you about this driver 
when he was in Austin. I can share the latest version of this driver.
>>
>> On 02/02/17 8:59 AM, Joel Stanley wrote:
>>> Hi Chris,
>>>
>>> On Thu, Feb 2, 2017 at 5:58 AM, Christopher Bostic
>>> <cbostic@linux.vnet.ibm.com> wrote:
>>>> I've not seen this discussed in the group but if this has already been
>>>> determined than any pointers to the latest documentation would be
>>>> appreciated.
>>>>
>>>> Who are all the potential users of the OpenBMC SBE FIFO device 
>>>> driver?   I
>>>> understand it will need to export its general submit interface to 
>>>> other
>>>> kernel drivers.  Only one I know of for sure would be the OCC 
>>>> driver Eddie
>>>> James is working on.
>>> In addition to Eddie's driver, in userspace we will have the code that
>>> performs the power on sequence.
>>>
>>>> Any others?
>>>>
>>>> What about user space access of the SBE FIFO engine?  What apps would
>>>> require it?  Cronus I assume will need some means to directly 
>>>> access it.
>>> We want userspace API and an in-kernel API.
>>>
>>> The driver you design will need to take into account that there will
>>> be multiple users from each of these APIs. For instance, there will be
>>> Eddie's OCC hwmon driver and the userspace code that kicks off the
>>> boot sequence.
>>>
>>> Cheers,
>>>
>>> Joel
>>> _______________________________________________
>>> openbmc mailing list
>>> openbmc@lists.ozlabs.org
>>> https://lists.ozlabs.org/listinfo/openbmc
>>
>> _______________________________________________
>> openbmc mailing list
>> openbmc@lists.ozlabs.org
>> https://lists.ozlabs.org/listinfo/openbmc
>
> _______________________________________________
> openbmc mailing list
> openbmc@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/openbmc

Re: Users of SBE FIFO kernel driver

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

On Sat, Feb 04, 2017 at 04:35:30PM +0530, Venkatesh Sainath wrote:
> > Where do we stand with the sbei protocol device driver?  Who owns this 
> > and is it complete?
> >
> The plan was to port the sbei protocol driver developed on FSP based 
> systems. Chris: I believe Dhruvaraj discussed with you about this driver 
> when he was in Austin. I can share the latest version of this driver.

Is this "driver" really a driver in the Linux sense or is it a shared
library that is used in FSP userspace?  Whatever code we end up using we
eventually need it to be upstream-able into the real Linux kernel.

Is this something that Dhruv can take on or is this going to be left to
Chris?  How do we handle divergence between the open-source code and the
IBM proprietary version (the one direction is an IBM-internal problem,
but I have concern that future development will only be happening on
the proprietary one and we'll have to keep playing catch up).

-- 
Patrick Williams

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Users of SBE FIFO kernel driver

[Venkatesh Sainath]

On 10/02/17 2:46 AM, Patrick Williams wrote:
> On Sat, Feb 04, 2017 at 04:35:30PM +0530, Venkatesh Sainath wrote:
>>> Where do we stand with the sbei protocol device driver?  Who owns this
>>> and is it complete?
>>>
>> The plan was to port the sbei protocol driver developed on FSP based
>> systems. Chris: I believe Dhruvaraj discussed with you about this driver
>> when he was in Austin. I can share the latest version of this driver.
> Is this "driver" really a driver in the Linux sense or is it a shared
> library that is used in FSP userspace?  Whatever code we end up using we
> eventually need it to be upstream-able into the real Linux kernel.
This is not really a driver in the Linux sense. It is a shared lib in 
FSP user space. As the OCC driver has to use this SBEI protocol, we  
have to implement this as a Linux driver in OpenBMC
> Is this something that Dhruv can take on or is this going to be left to
> Chris?  How do we handle divergence between the open-source code and the
> IBM proprietary version (the one direction is an IBM-internal problem,
> but I have concern that future development will only be happening on
> the proprietary one and we'll have to keep playing catch up).
>
The SBE interfaces are open-sourced and there may not be any proprietary 
implementation here. However, the code architecture between FSP and 
OpenBMC is different ( FSP - user space and OpenBMC - Linux driver ) and 
hence we will have to maintain two streams. The HWSV team can implement  
it as a Linux driver in OpenBMC and deliver it in stages. Any defect 
fixes in future can be done by the same team on both streams. I will ask 
Sachin Gupta to sync up with Chris. As of now, we need only Get/Put 
Memory, Get/Put Scom and Get/Put SRAM for enabling OCC and HW access. We 
can add the remaining interfaces subsequently. As a user of this SBEI 
protocol driver, we need OCC driver changes, Rest API to call OCC 
driver, SCOM drivers and user space app (Rest API / ??) to call the SCOM 
driver.

Re: Users of SBE FIFO kernel driver

[Brad Bishop]

Trying to keep this discussion going.

> On Feb 10, 2017, at 1:58 AM, Venkatesh Sainath <vsainath@linux.vnet.ibm.com> wrote:
> 
> 
> 
> On 10/02/17 2:46 AM, Patrick Williams wrote:
>> On Sat, Feb 04, 2017 at 04:35:30PM +0530, Venkatesh Sainath wrote:
>>>> Where do we stand with the sbei protocol device driver?  Who owns this
>>>> and is it complete?
>>>> 
>>> The plan was to port the sbei protocol driver developed on FSP based
>>> systems. Chris: I believe Dhruvaraj discussed with you about this driver
>>> when he was in Austin. I can share the latest version of this driver.
>> Is this "driver" really a driver in the Linux sense or is it a shared
>> library that is used in FSP userspace?  Whatever code we end up using we
>> eventually need it to be upstream-able into the real Linux kernel.
> This is not really a driver in the Linux sense. It is a shared lib in FSP user space. As the OCC driver has to use this SBEI protocol, we  have to implement this as a Linux driver in OpenBMC
>> Is this something that Dhruv can take on or is this going to be left to
>> Chris?  How do we handle divergence between the open-source code and the
>> IBM proprietary version (the one direction is an IBM-internal problem,
>> but I have concern that future development will only be happening on
>> the proprietary one and we'll have to keep playing catch up).
>> 
> The SBE interfaces are open-sourced and there may not be any proprietary implementation here. However, the code architecture between FSP and OpenBMC is different ( FSP - user space and OpenBMC - Linux driver ) and hence we will have to maintain two streams. The HWSV team can implement  it as a Linux driver in OpenBMC and deliver it in stages. Any defect fixes in future can be done by the same team on both streams. I will ask Sachin Gupta to sync up with Chris. As of now, we need only Get/Put Memory, Get/Put Scom and Get/Put SRAM for enabling OCC and HW access.

Are these things what were referred to previously in the thread as ‘chip-ops’ ?

If they are, then an SBEI protocol device driver sounds to me like chip-ops in the kernel but previously there were a number of statements from multiple people along these lines:

> However we certainly don't want to put chip-ops directly in the kernel.
> Agreed. This driver is to control access to the FIFO. We don't want to have to modify the kernel every time a new operation is required.
> The programs sending the messages will know how to encode their own operations, such as pdbg, the occ hwmon driver, etc.

Venkatesh - What role do you see the SBEI protocol driver serving?

Joel/Alistair/Ben - So far for in-kernel users we have scom and OCC.  I’m assuming that list will grow somewhat.  Obviously the sbescom driver will need to encode scom operations…but so does the occ driver.  If both drivers encode their own operations, won’t they be writing the same code twice?  Isn’t that going to happen over and over?

Everyone - please poke holes.

thx

-brad




> We can add the remaining interfaces subsequently. As a user of this SBEI protocol driver, we need OCC driver changes, Rest API to call OCC driver, SCOM drivers and user space app (Rest API / ??) to call the SCOM driver.