Question about NVMe MCTP in dbus-sensors

Question about NVMe MCTP in dbus-sensors

[Heyi Guo]

Hi,

We can see that NVMe sensors in dbus-sensors relies on MCTP to get 
hardware information. It is using libmctp interfaces to initialize MCTP 
and SMBus. However I don't find the code or component who is responsible 
as a bus owner, to discover endpoints, manager EID and update routing 
table. Isn't necessary for a central component to do such things? Will 
there be access conflict if non-NVMe devices (MCTP capable) are also 
connected to the same SMBus？

We also found another implementation from Intel: 
https://github.com/Intel-BMC/pmci. It implements mctpd to provide MCTP 
message transfer interfaces on D-Bus, while PLDM, NVME-MI and others can 
rely on the D-Bus interfaces instead of libmctp. In my opinion it seems 
to be a more mature solution for MCTP and higher PCMI management models. 
Is there any plan to accept such implementations into the mainline?

Thanks,

Heyi

Re: Question about NVMe MCTP in dbus-sensors

[Jeremy Kerr]

Hi Heyi,

> We can see that NVMe sensors in dbus-sensors relies on MCTP to get 
> hardware information. It is using libmctp interfaces to initialize
> MCTP and SMBus. However I don't find the code or component who is
> responsible as a bus owner, to discover endpoints, manager EID and
> update routing table.

So we're working on this at the moment.

The new design is here:

  https://github.com/openbmc/docs/blob/master/designs/mctp/mctp-kernel.md

The MCTP core patches have been accepted to the upstream kernel, and
we're finishing up the i2c/SMBus binding driver at the moment:

 https://github.com/CodeConstruct/linux/tree/dev/mctp-i2c-mux

The prototype userspace support code is here:

  https://github.com/CodeConstruct/mctp

- which includes the control protocol implementation (mctpd) for
endpoint discovery, routing table updates, etc.

The switch over from the demux-based approach to the kernel sockets
interface should be fairly straightforward, but Intel are working on a
library to abstract that.

Cheers,


Jeremy

Re: Question about NVMe MCTP in dbus-sensors

[Andrew Jeffery]

Hello Heyi!

On Fri, 6 Aug 2021, at 14:47, Heyi Guo wrote:
> Hi,
> 
> We can see that NVMe sensors in dbus-sensors relies on MCTP to get 
> hardware information. It is using libmctp interfaces to initialize MCTP 
> and SMBus. 

To be clear, it's using a fork of libmctp that implements an SMBus 
binding via a fork of the kernel that exposes a I2C API that isn't 
upstream.

For the SMBus binding to be merged in upstream libmctp I'd like to see 
the necessary kernel interfaces merged into the upstream kernel tree, 
but I don't expect that's going to happen any time soon. More on why 
below.

As an aside you may be interested in this patch which allows nvmesensor 
to use the basic management command to fetch temperature data:

https://gerrit.openbmc-project.xyz/c/openbmc/dbus-sensors/+/43665

> However I don't find the code or component who is responsible 
> as a bus owner, to discover endpoints, manager EID and update routing 
> table. Isn't necessary for a central component to do such things?

It's not strictly necessary in that it's valid for systems to be 
completely defined in terms of static endpoints. Doing so isn't covered 
by the MCTP spec, and it's also pretty limiting, but it is enough in 
some circumstances.

> Will 
> there be access conflict if non-NVMe devices (MCTP capable) are also 
> connected to the same SMBus？

No.

> 
> We also found another implementation from Intel: 
> https://github.com/Intel-BMC/pmci. It implements mctpd to provide MCTP 
> message transfer interfaces on D-Bus, while PLDM, NVME-MI and others can 
> rely on the D-Bus interfaces instead of libmctp.

This is a direction Intel chose to go however it's not the direction 
that upstream OpenBMC will be using. The use of a pure userspace 
solution such as libmctp went far enough to expose the need for a 
kernel-based solution. We will shortly have that, thanks to the efforts 
of Jeremy and Matt at Code Construct:

https://lore.kernel.org/netdev/20210729022053.134453-1-jk@codeconstruct.com.au/

This series is now in net-next, and Matt and Jeremy have also been 
developing the in-kernel hardware binding support and necessary 
userspace components[1].

[1] https://github.com/CodeConstruct/mctp

You can read more about the application of the socket syscalls here:

https://lore.kernel.org/netdev/20210729022053.134453-16-jk@codeconstruct.com.au/

and here:

https://github.com/openbmc/docs/blob/master/designs/mctp/mctp-kernel.md

Once we have AF_MCTP in the OpenBMC kernel tree with some binding 
implementations we'll be switching the userspace applications over to 
use it.

Hope that helps!

Andrew

Re: Question about NVMe MCTP in dbus-sensors

[Heyi Guo]

Hi Andrew, Jeremy,

Appreciate your helpful information. Let me take some time to look into 
all this document and patches :)

BTW, when will MCTP be ready in openbmc mainline, including SMBus 
binding and PCIe binding? Do you have any plan in the future openbmc 
release?

Thanks,

Heyi


On 2021/8/6 下午1:42, Andrew Jeffery wrote:
> Hello Heyi!
>
> On Fri, 6 Aug 2021, at 14:47, Heyi Guo wrote:
>> Hi,
>>
>> We can see that NVMe sensors in dbus-sensors relies on MCTP to get
>> hardware information. It is using libmctp interfaces to initialize MCTP
>> and SMBus.
> To be clear, it's using a fork of libmctp that implements an SMBus
> binding via a fork of the kernel that exposes a I2C API that isn't
> upstream.
>
> For the SMBus binding to be merged in upstream libmctp I'd like to see
> the necessary kernel interfaces merged into the upstream kernel tree,
> but I don't expect that's going to happen any time soon. More on why
> below.
>
> As an aside you may be interested in this patch which allows nvmesensor
> to use the basic management command to fetch temperature data:
>
> https://gerrit.openbmc-project.xyz/c/openbmc/dbus-sensors/+/43665
>
>> However I don't find the code or component who is responsible
>> as a bus owner, to discover endpoints, manager EID and update routing
>> table. Isn't necessary for a central component to do such things?
> It's not strictly necessary in that it's valid for systems to be
> completely defined in terms of static endpoints. Doing so isn't covered
> by the MCTP spec, and it's also pretty limiting, but it is enough in
> some circumstances.
>
>> Will
>> there be access conflict if non-NVMe devices (MCTP capable) are also
>> connected to the same SMBus？
> No.
>
>> We also found another implementation from Intel:
>> https://github.com/Intel-BMC/pmci. It implements mctpd to provide MCTP
>> message transfer interfaces on D-Bus, while PLDM, NVME-MI and others can
>> rely on the D-Bus interfaces instead of libmctp.
> This is a direction Intel chose to go however it's not the direction
> that upstream OpenBMC will be using. The use of a pure userspace
> solution such as libmctp went far enough to expose the need for a
> kernel-based solution. We will shortly have that, thanks to the efforts
> of Jeremy and Matt at Code Construct:
>
> https://lore.kernel.org/netdev/20210729022053.134453-1-jk@codeconstruct.com.au/
>
> This series is now in net-next, and Matt and Jeremy have also been
> developing the in-kernel hardware binding support and necessary
> userspace components[1].
>
> [1] https://github.com/CodeConstruct/mctp
>
> You can read more about the application of the socket syscalls here:
>
> https://lore.kernel.org/netdev/20210729022053.134453-16-jk@codeconstruct.com.au/
>
> and here:
>
> https://github.com/openbmc/docs/blob/master/designs/mctp/mctp-kernel.md
>
> Once we have AF_MCTP in the OpenBMC kernel tree with some binding
> implementations we'll be switching the userspace applications over to
> use it.
>
> Hope that helps!
>
> Andrew

Re: Question about NVMe MCTP in dbus-sensors

[Jeremy Kerr]

Hi Heyi,

> BTW, when will MCTP be ready in openbmc mainline, including SMBus 
> binding and PCIe binding? Do you have any plan in the future openbmc 
> release?

We're mostly working on the generic infrastructure components at the
moment; those are based on their relevant upstream projects rather than
as parts of OpenBMC specifically. However, the upstream work is going
well, and the OpenBMC integration shouldn't be too difficult there;
we'll get that done as necessary.

As for interfaces: I currently have i2c and serial transports working,
but have no specific plans for PCIe - no hardware to test on. If this
is something you'd be interested in working on yourself, I can provide
some pointers.

Regards,


Jeremy

Re: Question about NVMe MCTP in dbus-sensors

[Heyi Guo]

On 2021/8/6 下午5:46, Jeremy Kerr wrote:
> Hi Heyi,
>
>> BTW, when will MCTP be ready in openbmc mainline, including SMBus
>> binding and PCIe binding? Do you have any plan in the future openbmc
>> release?
> We're mostly working on the generic infrastructure components at the
> moment; those are based on their relevant upstream projects rather than
> as parts of OpenBMC specifically. However, the upstream work is going
> well, and the OpenBMC integration shouldn't be too difficult there;
> we'll get that done as necessary.
>
> As for interfaces: I currently have i2c and serial transports working,
> but have no specific plans for PCIe - no hardware to test on. If this
> is something you'd be interested in working on yourself, I can provide
> some pointers.

Yes, we are both interested in SMBus binding and PCIe VDM binding, and 
we have supporting hardware platforms (servers with Aspeed 2600 as BMC).

It's my pleasure to make contribution to the community :)

Thanks,

Heyi

>
> Regards,
>
>
> Jeremy

Re: Question about NVMe MCTP in dbus-sensors

[Heyi Guo]

Hi Andrew,


On 2021/8/6 下午1:42, Andrew Jeffery wrote:
> Hello Heyi!
>
> On Fri, 6 Aug 2021, at 14:47, Heyi Guo wrote:
>> Hi,
>>
>> We can see that NVMe sensors in dbus-sensors relies on MCTP to get
>> hardware information. It is using libmctp interfaces to initialize MCTP
>> and SMBus.
> To be clear, it's using a fork of libmctp that implements an SMBus
> binding via a fork of the kernel that exposes a I2C API that isn't
> upstream.

Could you point out where I can find these forks, including libmctp and 
kernel? So that we can do some initial test with the current 
implementation of NVMeSensor in dbus-sensors.

Thanks,

Heyi


>
> For the SMBus binding to be merged in upstream libmctp I'd like to see
> the necessary kernel interfaces merged into the upstream kernel tree,
> but I don't expect that's going to happen any time soon. More on why
> below.
>
> As an aside you may be interested in this patch which allows nvmesensor
> to use the basic management command to fetch temperature data:
>
> https://gerrit.openbmc-project.xyz/c/openbmc/dbus-sensors/+/43665
>
>> However I don't find the code or component who is responsible
>> as a bus owner, to discover endpoints, manager EID and update routing
>> table. Isn't necessary for a central component to do such things?
> It's not strictly necessary in that it's valid for systems to be
> completely defined in terms of static endpoints. Doing so isn't covered
> by the MCTP spec, and it's also pretty limiting, but it is enough in
> some circumstances.
>
>> Will
>> there be access conflict if non-NVMe devices (MCTP capable) are also
>> connected to the same SMBus？
> No.
>
>> We also found another implementation from Intel:
>> https://github.com/Intel-BMC/pmci. It implements mctpd to provide MCTP
>> message transfer interfaces on D-Bus, while PLDM, NVME-MI and others can
>> rely on the D-Bus interfaces instead of libmctp.
> This is a direction Intel chose to go however it's not the direction
> that upstream OpenBMC will be using. The use of a pure userspace
> solution such as libmctp went far enough to expose the need for a
> kernel-based solution. We will shortly have that, thanks to the efforts
> of Jeremy and Matt at Code Construct:
>
> https://lore.kernel.org/netdev/20210729022053.134453-1-jk@codeconstruct.com.au/
>
> This series is now in net-next, and Matt and Jeremy have also been
> developing the in-kernel hardware binding support and necessary
> userspace components[1].
>
> [1] https://github.com/CodeConstruct/mctp
>
> You can read more about the application of the socket syscalls here:
>
> https://lore.kernel.org/netdev/20210729022053.134453-16-jk@codeconstruct.com.au/
>
> and here:
>
> https://github.com/openbmc/docs/blob/master/designs/mctp/mctp-kernel.md
>
> Once we have AF_MCTP in the OpenBMC kernel tree with some binding
> implementations we'll be switching the userspace applications over to
> use it.
>
> Hope that helps!
>
> Andrew

Re: Question about NVMe MCTP in dbus-sensors

[Andrew Jeffery]

On Mon, 9 Aug 2021, at 16:03, Heyi Guo wrote:
> Hi Andrew,
> 
> 
> On 2021/8/6 下午1:42, Andrew Jeffery wrote:
> > Hello Heyi!
> >
> > On Fri, 6 Aug 2021, at 14:47, Heyi Guo wrote:
> >> Hi,
> >>
> >> We can see that NVMe sensors in dbus-sensors relies on MCTP to get
> >> hardware information. It is using libmctp interfaces to initialize MCTP
> >> and SMBus.
> > To be clear, it's using a fork of libmctp that implements an SMBus
> > binding via a fork of the kernel that exposes a I2C API that isn't
> > upstream.
> 
> Could you point out where I can find these forks, including libmctp and 
> kernel? So that we can do some initial test with the current 
> implementation of NVMeSensor in dbus-sensors.

The relevant forks are under the Intel-BMC Github org:

https://github.com/Intel-BMC/

Hopefully the Intel folks on Cc can give you pointers on how to get it going.

Cheers,

Andrew