Implement Ethernet Interface Properties

Implement Ethernet Interface Properties

[Spencer Ku (古世瑜)]

[-- Attachment #1: Type: text/plain, Size: 1231 bytes --]

Hi All,



We want to implement two network Ethernet interface properties “Bandwidth” and “Drop package rate”, and expose the information via Redfish. The goal is that clients can talk to bmc via Redfish and get those two information. It will improve the debuggability of BMCs.



About the implement method, we divide the work into two parts: phosphor-network(or other suitable service) part, and bmcweb part.

For phosphor-network part, we propose to use existing tool like ipert to get the Ethernet BW/Drop package rate. Or the other method is that read TX/RX from ethernet driver (/sys/class/net/{Ethernet_Name}), then calculate the BW/Drop package rate. After that, we will need to register the D-Bus interface and sync the property info to that D-Bus.



For bmcweb part, as I know so far, there has no suitable field under the EthernetInterface <https://redfish.dmtf.org/schemas/v1/EthernetInterface.v1_7_0.json> to set BW/Drop package rate. The idea is that read the value from D-bus then set the properties under the EthernetInterface OEM field. Please let me know if I have missed anything.



We are willing to see any suggestions and alternatives as well. Thanks!



Sincerely,

Spencer Ku

[-- Attachment #2: Type: text/html, Size: 4400 bytes --]

Re: Implement Ethernet Interface Properties

[Ed Tanous]

On Wed, Dec 1, 2021 at 1:59 AM Spencer Ku (古世瑜) <Spencer.Ku@quantatw.com> wrote:
>
> Hi All,
>
>
>
> We want to implement two network Ethernet interface properties “Bandwidth” and “Drop package rate”, and expose the information via Redfish. The goal is that clients can talk to bmc via Redfish and get those two information. It will improve the debuggability of BMCs.
>

The only thing I would note in your plan is that Redfish generally
doesn't expose anything as a "rate", it exposes it as a count (packet
count, drop count), and lets the client determine whatever sampling
rate they want.  I suspect you should do the same.

Also, I'm assuming "drop package rate" was a typo for "dropped packets rate"?

>
>
> About the implement method, we divide the work into two parts: phosphor-network(or other suitable service) part, and bmcweb part.
>
> For phosphor-network part, we propose to use existing tool like ipert to get the Ethernet BW/Drop package rate. Or the other method is that read TX/RX from ethernet driver (/sys/class/net/{Ethernet_Name}), then calculate the BW/Drop package rate. After that, we will need to register the D-Bus interface and sync the property info to that D-Bus.
>
>
>
> For bmcweb part, as I know so far, there has no suitable field under the EthernetInterface to set BW/Drop package rate. The idea is that read the value from D-bus then set the properties under the EthernetInterface OEM field. Please let me know if I have missed anything.

Considering that not all NICs will have this telemetry, it probably
makes sense to have it on a separate dbus interface.

>
>
>
> We are willing to see any suggestions and alternatives as well. Thanks!
>
>
>
> Sincerely,
>
> Spencer Ku

Re: Implement Ethernet Interface Properties

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 862 bytes --]

On Wed, Dec 01, 2021 at 09:58:41AM +0000, Spencer Ku (古世瑜) wrote:

> For phosphor-network part, we propose to use existing tool like ipert to get the Ethernet BW/Drop package rate. Or the other method is that read TX/RX from ethernet driver (/sys/class/net/{Ethernet_Name}), then calculate the BW/Drop package rate. After that, we will need to register the D-Bus interface and sync the property info to that D-Bus.

Is this something that systemd-networkd might be interested in exposing?  We
should ask this question upstream there to see if they would be interested in
the contribution.

I'd prefer using the sysfs interface direction rather than wrapping a tool in
`system()` calls.  

Did you mean 'iperf'?  How do you get the drop rate from that?  Don't you need
another system to act as the other side of iperf?

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: Implement Ethernet Interface Properties

[Spencer Ku (古世瑜)]

> On Wed, Dec 1, 2021 at 1:59 AM Spencer Ku (古世瑜)
> <Spencer.Ku@quantatw.com> wrote:
> >
> > Hi All,
> >
> >
> >
> > We want to implement two network Ethernet interface properties
> “Bandwidth” and “Drop package rate”, and expose the information via
> Redfish. The goal is that clients can talk to bmc via Redfish and get those two
> information. It will improve the debuggability of BMCs.
> >
> 
> The only thing I would note in your plan is that Redfish generally doesn't
> expose anything as a "rate", it exposes it as a count (packet count, drop count),
> and lets the client determine whatever sampling rate they want.  I suspect
> you should do the same.

Thanks for your explanation! I will follow the feedback to I implement this feature.

> 
> Also, I'm assuming "drop package rate" was a typo for "dropped packets rate"?

Yes , it should be "dropped packets rate", sorry for the typo.

> 
> >
> >
> > About the implement method, we divide the work into two parts:
> phosphor-network(or other suitable service) part, and bmcweb part.
> >
> > For phosphor-network part, we propose to use existing tool like ipert to get
> the Ethernet BW/Drop package rate. Or the other method is that read TX/RX
> from ethernet driver (/sys/class/net/{Ethernet_Name}), then calculate the
> BW/Drop package rate. After that, we will need to register the D-Bus interface
> and sync the property info to that D-Bus.
> >
> >
> >
> > For bmcweb part, as I know so far, there has no suitable field under the
> EthernetInterface to set BW/Drop package rate. The idea is that read the value
> from D-bus then set the properties under the EthernetInterface OEM field.
> Please let me know if I have missed anything.
> 
> Considering that not all NICs will have this telemetry, it probably makes sense
> to have it on a separate dbus interface.

Understand, I will notice this part when I register the D-Bus interface.

> 
> >
> >
> >
> > We are willing to see any suggestions and alternatives as well. Thanks!
> >
> >
> >
> > Sincerely,
> >
> > Spencer Ku

RE: Implement Ethernet Interface Properties

[Spencer Ku (古世瑜)]

> On Wed, Dec 01, 2021 at 09:58:41AM +0000, Spencer Ku (古世瑜) wrote:
> 
> > For phosphor-network part, we propose to use existing tool like ipert to get
> the Ethernet BW/Drop package rate. Or the other method is that read TX/RX
> from ethernet driver (/sys/class/net/{Ethernet_Name}), then calculate the
> BW/Drop package rate. After that, we will need to register the D-Bus interface
> and sync the property info to that D-Bus.
> 
> Is this something that systemd-networkd might be interested in exposing?
> We should ask this question upstream there to see if they would be interested
> in the contribution.

That will be good if systemd-networkd can expose those information. I will try to contact them asking about this question.

> 
> I'd prefer using the sysfs interface direction rather than wrapping a tool in
> `system()` calls.
> 
> Did you mean 'iperf'?  How do you get the drop rate from that?  Don't you
> need another system to act as the other side of iperf?

It should be "iperf" here. Yes, iperf need a client to act, so it may not be suitable in this implementation.
I will focus on the method of sysfs interface direction. Sorry for the confusion and typo.

> 
> --
> Patrick Williams

RE: Implement Ethernet Interface Properties

[Spencer Ku (古世瑜)]

Hi All,

We finally decide to implement six properties to Ethernet interface: 
RxPackets, TxPackets, RxBytes, TxBytes, RxDropped, and TxDropped.

Those information come from Ethernet statistics and we can read the file 
to get the property value directly. (under /sys/class/net/<Ethernet name>/statistics)

For the suitable property in EthernetInterface model, we got the feedback 
form Redfish forum and they said OEM properties would be the only other 
option at this time. For more discussion detail:
https://redfishforum.com/thread/579/additional-information-ethernet-interface

The following output is the result of my currently OEM implement:
https://${bmc}/redfish/v1/Managers/bmc/EhernetInterfaces/eth0
{
  "@odata.id": "/redfish/v1/Managers/bmc/EthernetInterfaces/eth0",
  ...
  "Oem": {
    "Statistics": [
      {
        "RxBytes": 48244026,
        "RxDropped": 222856,
        "RxPackets": 484315,
        "TxBytes": 825845,
        "TxDropped": 0,
        "TxPackets": 5472
      }
    ]
  },
  "SpeedMbps": 100,
  ...
}

But it cannot pass the Redfish validator, and here is the error messages:
ERROR - This complex object RxBytes should be a dictionary or None, but it's of type ...
ERROR - RxBytes: Could not finish check on this property ('int' object is not iterable)

Is there anything I didn't notice? Or has there any spec or example that I can follow to implement? 

I had already push the implementation to gerrit and mark as WIP. Here is the link:
https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/50522

We are willing to see any suggestions and alternatives as well. Thanks!

Sincerely,
Spencer Ku

Re: Implement Ethernet Interface Properties

[i.kononenko]

Hi Spencer!

On 20.01.2022 11:42, Spencer Ku (古世瑜) wrote:
> Hi All,
> 
> We finally decide to implement six properties to Ethernet interface: 
> RxPackets, TxPackets, RxBytes, TxBytes, RxDropped, and TxDropped.
> 
> Those information come from Ethernet statistics and we can read the file 
> to get the property value directly. (under /sys/class/net/<Ethernet name>/statistics)
> 
> For the suitable property in EthernetInterface model, we got the feedback 
> form Redfish forum and they said OEM properties would be the only other 
> option at this time. For more discussion detail:
> https://redfishforum.com/thread/579/additional-information-ethernet-interface
> 

I'd suggest referring to the [1] **9.7. Resource extensibility** to examine the proper 
way of OEM-properties definition.


> The following output is the result of my currently OEM implement:
> https://${bmc}/redfish/v1/Managers/bmc/EhernetInterfaces/eth0
> {
>   "@odata.id": "/redfish/v1/Managers/bmc/EthernetInterfaces/eth0",
>   ...
>   "Oem": {
>     "Statistics": [

Looks like "@odata.type" required field is missed. The **9.7.4. OEM property examples** 
contains the example of the right way to define OEM-properties.

>       {
>         "RxBytes": 48244026,
>         "RxDropped": 222856,
>         "RxPackets": 484315,
>         "TxBytes": 825845,
>         "TxDropped": 0,
>         "TxPackets": 5472
>       }
>     ]
>   },
>   "SpeedMbps": 100,
>   ...
> }
> 
> But it cannot pass the Redfish validator, and here is the error messages:
> ERROR - This complex object RxBytes should be a dictionary or None, but it's of type ...

Does it require to have the integer values? I guess each mentioned field might contain string values. 

> ERROR - RxBytes: Could not finish check on this property ('int' object is not iterable)
> 
> Is there anything I didn't notice? Or has there any spec or example that I can follow to implement? 
> 
> I had already push the implementation to gerrit and mark as WIP. Here is the link:
> https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/50522
> 
> We are willing to see any suggestions and alternatives as well. Thanks!
> 
> Sincerely,
> Spencer Ku
> 

Links:
[1] DSP0266 Redfish Specification: 
    https://www.dmtf.org/sites/default/files/standards/documents/DSP0266_1.8.0.pdf

-- 
Best regards,

Igor Kononenko

RE: Implement Ethernet Interface Properties

[Spencer Ku (古世瑜)]

Hi Kononenko, Thanks for your reply!

> -----Original Message-----
> From: i.kononenko <i.kononenko@yadro.com>
> Subject: Re: Implement Ethernet Interface Properties
> 
> Hi Spencer!
> 
> On 20.01.2022 11:42, Spencer Ku (古世瑜) wrote:
> > Hi All,
> >
> > We finally decide to implement six properties to Ethernet interface:
> > RxPackets, TxPackets, RxBytes, TxBytes, RxDropped, and TxDropped.
> >
> > Those information come from Ethernet statistics and we can read the
> > file to get the property value directly. (under
> > /sys/class/net/<Ethernet name>/statistics)
> >
> > For the suitable property in EthernetInterface model, we got the
> > feedback form Redfish forum and they said OEM properties would be the
> > only other option at this time. For more discussion detail:
> > https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fredf
> > ishforum.com%2Fthread%2F579%2Fadditional-information-ethernet-interfac
> >
> e&amp;data=04%7C01%7CSpencer.Ku%40quantatw.com%7Ce2120a7a2bb246
> 9f409c0
> >
> 8d9dbf46100%7C179b032707fc4973ac738de7313561b2%7C1%7C0%7C637782
> 6661318
> >
> 53103%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2lu
> MzIiLCJB
> >
> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=fIEODKRf1elAoepIdSbpG
> SL9nY
> > ME3FTEQm1IooJKOLs%3D&amp;reserved=0
> >
> 
> I'd suggest referring to the [1] **9.7. Resource extensibility** to examine the
> proper way of OEM-properties definition.
> 

As we discussed in the PR , I need to define and create the JSON schema "OemEthernetInterface".
I will push the schema to gerrit soon when I finish it.

> 
> > The following output is the result of my currently OEM implement:
> > https://${bmc}/redfish/v1/Managers/bmc/EhernetInterfaces/eth0
> > {
> >   "@odata.id": "/redfish/v1/Managers/bmc/EthernetInterfaces/eth0",
> >   ...
> >   "Oem": {
> >     "Statistics": [
> 
> Looks like "@odata.type" required field is missed. The **9.7.4. OEM property
> examples**
> contains the example of the right way to define OEM-properties.
> 
> >       {
> >         "RxBytes": 48244026,
> >         "RxDropped": 222856,
> >         "RxPackets": 484315,
> >         "TxBytes": 825845,
> >         "TxDropped": 0,
> >         "TxPackets": 5472
> >       }
> >     ]
> >   },
> >   "SpeedMbps": 100,
> >   ...
> > }
> >
> > But it cannot pass the Redfish validator, and here is the error messages:
> > ERROR - This complex object RxBytes should be a dictionary or None, but it's
> of type ...
> 
> Does it require to have the integer values? I guess each mentioned field might
> contain string values.
> 

I think the main reason which occurs the error is that I didn't define the odata ID and type. 
They cannot find the reference to analyze the property. I add the temp odata field (patch 3 in the PR), and the error messages has been changed to WRANING:

WARNING - Couldn't get schema for object, skipping OemObject OpenBmc
WARNING - Statistics not defined in schema Complex OpenBmc Resource.OemObject (check version, spelling and casing)

I will run the Redfish validator again when I finish the schema.

> > ERROR - RxBytes: Could not finish check on this property ('int' object is not
> iterable)
> >
> > Is there anything I didn't notice? Or has there any spec or example that I can
> follow to implement?
> >
> > I had already push the implementation to gerrit and mark as WIP. Here is the
> link:
> >
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgerrit.op
> enbmc-project.xyz%2Fc%2Fopenbmc%2Fbmcweb%2F%2B%2F50522&amp;data
> =04%7C01%7CSpencer.Ku%40quantatw.com%7Ce2120a7a2bb2469f409c08d9d
> bf46100%7C179b032707fc4973ac738de7313561b2%7C1%7C0%7C6377826661
> 31853103%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2
> luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=dv%2FXHFKTv
> IPnbHDgBFbE0UU%2BH8G6jzfs4XvGYVultDo%3D&amp;reserved=0
> >
> > We are willing to see any suggestions and alternatives as well. Thanks!
> >
> > Sincerely,
> > Spencer Ku
> >
> 
> Links:
> [1] DSP0266 Redfish Specification:
> 
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.dm
> tf.org%2Fsites%2Fdefault%2Ffiles%2Fstandards%2Fdocuments%2FDSP0266_1.8
> .0.pdf&amp;data=04%7C01%7CSpencer.Ku%40quantatw.com%7Ce2120a7a2bb
> 2469f409c08d9dbf46100%7C179b032707fc4973ac738de7313561b2%7C1%7C0
> %7C637782666131853103%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjA
> wMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sd
> ata=kP6kdPx7SGbRU2P8tMZCGfvq6MIhz8jOp%2BQf%2BSmOwTI%3D&amp;res
> erved=0
> 
> --
> Best regards,
> 
> Igor Kononenko

Sincerely,
Spencer Ku

Re: Implement Ethernet Interface Properties

[Ed Tanous]

On Thu, Jan 20, 2022 at 12:44 AM Spencer Ku (古世瑜)
<Spencer.Ku@quantatw.com> wrote:
>
> Hi All,
>
> We finally decide to implement six properties to Ethernet interface:
> RxPackets, TxPackets, RxBytes, TxBytes, RxDropped, and TxDropped.
>
> Those information come from Ethernet statistics and we can read the file
> to get the property value directly. (under /sys/class/net/<Ethernet name>/statistics)
>
> For the suitable property in EthernetInterface model, we got the feedback
> form Redfish forum and they said OEM properties would be the only other
> option at this time. For more discussion detail:

From their reply
"We may need to open discussions further for how to model these sorts
of metrics purely under EthernetInterface"

I think you've misinterpreted the response.  They asked for further
discussion on how to model it, and that OEM would be the alternative
if you weren't able to discuss.  Given our rules around OEM schemas,
please continue the discussion with the Redfish forum around this
topic to determine the right way to model this in the standard without
OEM.  There is nothing unique about OpenBMC with regards to these
telemetries, so it should be generic.  This might be a good candidate
for the newly formed "DiagnosticData" resource model, which is
initially exposing basically this same telemetry, but for i2c lanes
instead of network lanes.

> https://redfishforum.com/thread/579/additional-information-ethernet-interface
>
> The following output is the result of my currently OEM implement:
> https://${bmc}/redfish/v1/Managers/bmc/EhernetInterfaces/eth0
> {
>   "@odata.id": "/redfish/v1/Managers/bmc/EthernetInterfaces/eth0",
>   ...
>   "Oem": {
>     "Statistics": [
>       {
>         "RxBytes": 48244026,
>         "RxDropped": 222856,
>         "RxPackets": 484315,
>         "TxBytes": 825845,
>         "TxDropped": 0,
>         "TxPackets": 5472
>       }
>     ]
>   },
>   "SpeedMbps": 100,
>   ...
> }
>
> But it cannot pass the Redfish validator, and here is the error messages:
> ERROR - This complex object RxBytes should be a dictionary or None, but it's of type ...
> ERROR - RxBytes: Could not finish check on this property ('int' object is not iterable)
>
> Is there anything I didn't notice? Or has there any spec or example that I can follow to implement?
>
> I had already push the implementation to gerrit and mark as WIP. Here is the link:
> https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/50522


You haven't modeled your schema properly in the above.  This is one of
the reasons we try to avoid OEM schemas, as they're difficult to get
right without help from the standards body.  Please continue to engage
with them to get this in a standard interface.

>
> We are willing to see any suggestions and alternatives as well. Thanks!
>
> Sincerely,
> Spencer Ku

RE: Implement Ethernet Interface Properties

[Spencer Ku (古世瑜)]

Hi Ed, thank you for the feedback!

> -----Original Message-----
> From: openbmc
> <openbmc-bounces+spencer.ku=quantatw.com@lists.ozlabs.org> On Behalf Of
> Ed Tanous
> Sent: Tuesday, January 25, 2022 4:38 AM
> To: Spencer Ku (古世瑜) <Spencer.Ku@quantatw.com>
> Cc: Litzung Chen (陳利琮) <Litzung.Chen@quantatw.com>; Ed Tanous
> <edtanous@google.com>; Fran Hsu (徐誌謙) <Fran.Hsu@quantatw.com>;
> openbmc@lists.ozlabs.org; Nan Zhou <nanzhou@google.com>
> Subject: Re: Implement Ethernet Interface Properties
> 
> On Thu, Jan 20, 2022 at 12:44 AM Spencer Ku (古世瑜)
> <Spencer.Ku@quantatw.com> wrote:
> >
> > Hi All,
> >
> > We finally decide to implement six properties to Ethernet interface:
> > RxPackets, TxPackets, RxBytes, TxBytes, RxDropped, and TxDropped.
> >
> > Those information come from Ethernet statistics and we can read the
> > file to get the property value directly. (under
> > /sys/class/net/<Ethernet name>/statistics)
> >
> > For the suitable property in EthernetInterface model, we got the
> > feedback form Redfish forum and they said OEM properties would be the
> > only other option at this time. For more discussion detail:
> 
> From their reply
> "We may need to open discussions further for how to model these sorts of
> metrics purely under EthernetInterface"
> 
> I think you've misinterpreted the response.  They asked for further discussion
> on how to model it, and that OEM would be the alternative if you weren't able
> to discuss.  Given our rules around OEM schemas, please continue the
> discussion with the Redfish forum around this topic to determine the right way
> to model this in the standard without OEM.  There is nothing unique about
> OpenBMC with regards to these telemetries, so it should be generic.  This
> might be a good candidate for the newly formed "DiagnosticData" resource
> model, which is initially exposing basically this same telemetry, but for i2c
> lanes instead of network lanes.
> 

Understand, after looking for the DiagnosticData resource model, I think the model is actually more suitable field for the Ethernet statistic.
I will continue discuss with them and see if the Ethernet statistic can be the standard under the DiagnosticData or EthernetInterface model without OEM.

> > https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fredf
> > ishforum.com%2Fthread%2F579%2Fadditional-information-ethernet-interfac
> >
> e&amp;data=04%7C01%7CSpencer.Ku%40quantatw.com%7Ce1af9d9b5520475
> f00180
> >
> 8d9df797c1f%7C179b032707fc4973ac738de7313561b2%7C1%7C0%7C637786
> 5351607
> >
> 38828%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2lu
> MzIiLCJB
> >
> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=HP1lhG%2FtnuKvgskOes
> dwDrpJ
> > 7S56XmXAXJrFUwV%2F6%2BY%3D&amp;reserved=0
> >
> > The following output is the result of my currently OEM implement:
> > https://${bmc}/redfish/v1/Managers/bmc/EhernetInterfaces/eth0
> > {
> >   "@odata.id": "/redfish/v1/Managers/bmc/EthernetInterfaces/eth0",
> >   ...
> >   "Oem": {
> >     "Statistics": [
> >       {
> >         "RxBytes": 48244026,
> >         "RxDropped": 222856,
> >         "RxPackets": 484315,
> >         "TxBytes": 825845,
> >         "TxDropped": 0,
> >         "TxPackets": 5472
> >       }
> >     ]
> >   },
> >   "SpeedMbps": 100,
> >   ...
> > }
> >
> > But it cannot pass the Redfish validator, and here is the error messages:
> > ERROR - This complex object RxBytes should be a dictionary or None, but it's
> of type ...
> > ERROR - RxBytes: Could not finish check on this property ('int' object
> > is not iterable)
> >
> > Is there anything I didn't notice? Or has there any spec or example that I can
> follow to implement?
> >
> > I had already push the implementation to gerrit and mark as WIP. Here is the
> link:
> > https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgerr
> >
> it.openbmc-project.xyz%2Fc%2Fopenbmc%2Fbmcweb%2F%2B%2F50522&amp;
> data=0
> >
> 4%7C01%7CSpencer.Ku%40quantatw.com%7Ce1af9d9b5520475f001808d9df79
> 7c1f%
> >
> 7C179b032707fc4973ac738de7313561b2%7C1%7C0%7C637786535160738828
> %7CUnkn
> >
> own%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h
> aWwi
> >
> LCJXVCI6Mn0%3D%7C3000&amp;sdata=KlMfvE50FOX3DPqDoHfQ952XSNEMs
> EDUpO2tdF
> > j8%2BZQ%3D&amp;reserved=0
> 
> 
> You haven't modeled your schema properly in the above.  This is one of the
> reasons we try to avoid OEM schemas, as they're difficult to get right without
> help from the standards body.  Please continue to engage with them to get
> this in a standard interface.
> 

I see, here I missed the odata ID and odata type properties which OEM properties must have.
If we can let the properties become standard, this problem can be above. 

> >
> > We are willing to see any suggestions and alternatives as well. Thanks!
> >
> > Sincerely,
> > Spencer Ku

Sincerely,
Spencer Ku