* Re: apply for a new repo "openbmc/node-data-sync" [not found] ` <AF9887DB-F6CB-4CE0-90B1-17FEA2497013@fuzziesquirrel.com> @ 2021-04-12 6:38 ` yugang.chen 2021-04-12 11:29 ` Patrick Williams 2021-04-12 19:26 ` Ed Tanous 0 siblings, 2 replies; 7+ messages in thread From: yugang.chen @ 2021-04-12 6:38 UTC (permalink / raw) To: Brad Bishop, openbmc; +Cc: Bills, Jason M, chunhui.jia thanks, Brad. + openbmc@lists.ozlabs.org . Dear All, I'm Intel BMC engineer, we have a module is used to sync sensor data between BMC nodes in one Chassis system. In our Modular system, there are 2 nodes or 4 nodes, one node works as Primary node, the other nodes work as secondary nodes. 1. Some sensors are only visible for primary BMC, but secondary nodes need the sensors for Fan control. 2. Some sensor are in secondary nodes, they are needed to show them on primary node. So, we need a new repo to sync the sensor data between primary node and secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks. Best Regards Daniel(Yugang) On 4/10/2021 10:55 PM, Brad Bishop wrote: > Hi Daniel > > Can you please send this to the mailing list so everyone can be aware of the work you are doing? > > thanks - brad > >> On Apr 7, 2021, at 1:31 AM, yugang.chen <yugang.chen@linux.intel.com> wrote: >> >> Hi Brad, >> >> I'm Intel BMC engineer, we have a module is used to sync sensor data between BMC nodes in one Chassis system. In our Modular system, there are 2 nodes or 4 nodes, one node works as Primary node, the other nodes work as secondary nodes. >> >> 1. Some sensors are only visible for primary BMC, but secondary nodes need the sensors for Fan control. >> >> 2. Some sensor are in secondary nodes, they are needed to show them on primary node. >> >> So, we need a new repo to sync the sensor data between primary node and secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks. >> >> >> Best Regards >> >> Daniel(Yugang) >> ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: apply for a new repo "openbmc/node-data-sync" 2021-04-12 6:38 ` apply for a new repo "openbmc/node-data-sync" yugang.chen @ 2021-04-12 11:29 ` Patrick Williams 2021-04-12 19:26 ` Ed Tanous 1 sibling, 0 replies; 7+ messages in thread From: Patrick Williams @ 2021-04-12 11:29 UTC (permalink / raw) To: yugang.chen; +Cc: openbmc, Brad Bishop, Bills, Jason M, chunhui.jia [-- Attachment #1: Type: text/plain, Size: 1637 bytes --] On Mon, Apr 12, 2021 at 02:38:38PM +0800, yugang.chen wrote: > thanks, Brad. > > + openbmc@lists.ozlabs.org . > > Dear All, > > I'm Intel BMC engineer, we have a module is used to sync sensor data between BMC nodes in one Chassis system. In our Modular system, there are 2 nodes or 4 nodes, one node works as Primary node, the other nodes work as secondary nodes. > > 1. Some sensors are only visible for primary BMC, but secondary nodes need the sensors for Fan control. > > 2. Some sensor are in secondary nodes, they are needed to show them on primary node. > > So, we need a new repo to sync the sensor data between primary node and secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks. Do you have a design on how this would be accomplished? I'm curious what your proposal ends up looking like at a dbus-level. We should make sure this is documented at a high level and also aligns with how we are handling multi-host scenarios already in our dbus layout. How are the two BMC devices connected? How do you ensure you can safely / securely accept the data between the two BMCs? If done poorly this could open up a large security attack surface. I suspect your software design is somewhat driven by a particular hardware design as well, so we need to make sure it is clear the scoping of your solution and should probably even communicate that in the name. For instance, if your data transport is IPMB over i2c and you don't plan on this being a general daemon that could be expanded to a network connection, we shouldn't name it so generally. -- Patrick Williams [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: apply for a new repo "openbmc/node-data-sync" 2021-04-12 6:38 ` apply for a new repo "openbmc/node-data-sync" yugang.chen 2021-04-12 11:29 ` Patrick Williams @ 2021-04-12 19:26 ` Ed Tanous 2021-04-19 4:41 ` yugang.chen 1 sibling, 1 reply; 7+ messages in thread From: Ed Tanous @ 2021-04-12 19:26 UTC (permalink / raw) To: yugang.chen; +Cc: openbmc, Brad Bishop, Bills, Jason M, chunhui.jia On Sun, Apr 11, 2021 at 11:40 PM yugang.chen <yugang.chen@linux.intel.com> wrote: > > thanks, Brad. > > + openbmc@lists.ozlabs.org . > > Dear All, > > I'm Intel BMC engineer, we have a module is used to sync sensor data between BMC nodes in one Chassis system. In our Modular system, there are 2 nodes or 4 nodes, one node works as Primary node, the other nodes work as secondary nodes. > > 1. Some sensors are only visible for primary BMC, but secondary nodes need the sensors for Fan control. > > 2. Some sensor are in secondary nodes, they are needed to show them on primary node. > > So, we need a new repo to sync the sensor data between primary node and secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks. > +1 to wanting a design document for this. In absence of that, I'm going to make the possibly wrong assumption that you're planning on using IPMB. In dbus-sensors, we already have IPMB Sensor syncing, and patches are in progress to add IPMB version remoting. You might want to look into those to see if they meet your needs, or can be amended to meet your needs before trying to push a new project. > Best Regards > > Daniel(Yugang) > > > On 4/10/2021 10:55 PM, Brad Bishop wrote: > > Hi Daniel > > > > Can you please send this to the mailing list so everyone can be aware of the work you are doing? > > > > thanks - brad > > > >> On Apr 7, 2021, at 1:31 AM, yugang.chen <yugang.chen@linux.intel.com> wrote: > >> > >> Hi Brad, > >> > >> I'm Intel BMC engineer, we have a module is used to sync sensor data between BMC nodes in one Chassis system. In our Modular system, there are 2 nodes or 4 nodes, one node works as Primary node, the other nodes work as secondary nodes. > >> > >> 1. Some sensors are only visible for primary BMC, but secondary nodes need the sensors for Fan control. > >> > >> 2. Some sensor are in secondary nodes, they are needed to show them on primary node. > >> > >> So, we need a new repo to sync the sensor data between primary node and secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks. > >> > >> > >> Best Regards > >> > >> Daniel(Yugang) > >> ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: apply for a new repo "openbmc/node-data-sync" 2021-04-12 19:26 ` Ed Tanous @ 2021-04-19 4:41 ` yugang.chen 2021-04-19 14:26 ` Brad Bishop 0 siblings, 1 reply; 7+ messages in thread From: yugang.chen @ 2021-04-19 4:41 UTC (permalink / raw) To: Ed Tanous; +Cc: openbmc, Brad Bishop, Bills, Jason M, chunhui.jia [-- Attachment #1: Type: text/plain, Size: 2289 bytes --] attach the design document, please take a look. Best Regards Daniel(Yugang) On 4/13/2021 3:26 AM, Ed Tanous wrote: > On Sun, Apr 11, 2021 at 11:40 PM yugang.chen > <yugang.chen@linux.intel.com> wrote: >> thanks, Brad. >> >> + openbmc@lists.ozlabs.org . >> >> Dear All, >> >> I'm Intel BMC engineer, we have a module is used to sync sensor data between BMC nodes in one Chassis system. In our Modular system, there are 2 nodes or 4 nodes, one node works as Primary node, the other nodes work as secondary nodes. >> >> 1. Some sensors are only visible for primary BMC, but secondary nodes need the sensors for Fan control. >> >> 2. Some sensor are in secondary nodes, they are needed to show them on primary node. >> >> So, we need a new repo to sync the sensor data between primary node and secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks. >> > +1 to wanting a design document for this. In absence of that, I'm > going to make the possibly wrong assumption that you're planning on > using IPMB. In dbus-sensors, we already have IPMB Sensor syncing, and > patches are in progress to add IPMB version remoting. You might want > to look into those to see if they meet your needs, or can be amended > to meet your needs before trying to push a new project. > >> Best Regards >> >> Daniel(Yugang) >> >> >> On 4/10/2021 10:55 PM, Brad Bishop wrote: >>> Hi Daniel >>> >>> Can you please send this to the mailing list so everyone can be aware of the work you are doing? >>> >>> thanks - brad >>> >>>> On Apr 7, 2021, at 1:31 AM, yugang.chen <yugang.chen@linux.intel.com> wrote: >>>> >>>> Hi Brad, >>>> >>>> I'm Intel BMC engineer, we have a module is used to sync sensor data between BMC nodes in one Chassis system. In our Modular system, there are 2 nodes or 4 nodes, one node works as Primary node, the other nodes work as secondary nodes. >>>> >>>> 1. Some sensors are only visible for primary BMC, but secondary nodes need the sensors for Fan control. >>>> >>>> 2. Some sensor are in secondary nodes, they are needed to show them on primary node. >>>> >>>> So, we need a new repo to sync the sensor data between primary node and secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks. >>>> >>>> >>>> Best Regards >>>> >>>> Daniel(Yugang) >>>> [-- Attachment #2: node-data-sync.md --] [-- Type: text/plain, Size: 3573 bytes --] # Modular System Support - PCH IO Expander Author: Chen, Yugang ## Problem Description When multi modular 2 socket reference platforms (RP) are connected together, the BMCs on each RP shall support the ability to manage the system as either single 4S/8S system or standalone as 2S systems. ## Background and References In 4S/8S mode, only one PCH works at DMI (Direct Media Interface) mode, this PCH is called legacy PCH (PCH.L), the other PCHs are called PCH IO Expander (PCH.IO). The BMC attached to the PCH.L works as a primary and the other BMCs work as secondaries. ## Requirements Check a modular system as either 4S/8S or standalone mode by GPIOs on BMC each time system AC on. Each BMC works independently in standalone mode. In 4S/8S mode, only primary BMC is responsible for management of system level resources, secondary BMCs need to collect its local management resources and report them to primary BMC. ## Proposed Design Each BMC will have a property to reveal its work mode. Users can know if BMC is in standalone mode or 4S/8S mode. Each BMC has another property to show if the BMC mode is PCH Legacy BMC or Non-Legacy BMC. PCH Legacy BMC is working as primary and Non-Legacy BMC working as secondary. Each time a BMC reboot, The BMC needs to check 3 GPIO Pins: FM_STANDALONE_MONDE_N / FM_4S_8S_N_MODE / FM_NODE_ID to get working mode as standalone mode or primary /secondary(4S/8S) role according to the GPIO values. After confirming the mode and BMCs' role, BMCs should set properties according to the correct configuration. In 4S/8S mode only node id 0 will be primary BMC because only this node will be the PCH.L. Node id 1,2,3 will be the secondary nodes. Once a BMC gets mode is in 4S/8S, node roles are configured by node ID (GPIO Pins) and keep consistent once AC on. Once node role check is done, In 4S/8S mode, Primary BMC needs to broadcast its role to make sure there is only one primary BMC in the system. Need a new feature to make sure secondary BMCs send local redfish events to primary BMC. And primary BMC needs to add a tag to those events coming from secondary BMC so that user can know the event logs happened on which node. Even in 4S/8S mode, each BMC will collect its local management resources, including sensors, fans and do FSC according to the values of local sensors. PSU and Fans on each node will not be connected together. Configuration settings of each secondary node will remain the same, and won't be synced across the nodes. In 4S/8S mode, PECI will only be connected to primary node. Primary BMC needs to monitor all CPU and DIMM sensors and deliver the sensor values of the CPUs/DIMMs on secondary nodes to secondary BMCs. So that secondary BMCs can use the sensor values to control their own FSCs. Primary BMC also needs to have a way to find how many CPUs are in the whole system include Primary and Secondary nodes. ## Alternatives Considered Primary node monitors all the IPMI sensors in secondary nodes and creates redfish log by itself. Instead of BMC reboot, only AC cycle will make BMCs check GPIO pins and set Legacy BMC or Non-Legacy BMC mode. Only primary BMC broadcast its role and secondary only waiting for the broadcast from primary. ## Impacts Only on the motherboard where legacy PCH is located, POST code/Front Panel/KCS port/UART will work, while these interfaces on board with non-legacy PCH will not work due to BIOS and HW design. And this will cause non-functioning of SOL/KVM/Virtual media on secondary BMCs. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: apply for a new repo "openbmc/node-data-sync" 2021-04-19 4:41 ` yugang.chen @ 2021-04-19 14:26 ` Brad Bishop 2021-04-23 13:04 ` Patrick Williams 0 siblings, 1 reply; 7+ messages in thread From: Brad Bishop @ 2021-04-19 14:26 UTC (permalink / raw) To: yugang.chen; +Cc: Bills, Jason M, Ed Tanous, openbmc, chunhui.jia Hi Daniel Typically email attachments are not opened. To ensure your mail gets read, avoid attachments. Resending without the attachment... On Mon, Apr 19, 2021 at 12:41:00PM +0800, yugang.chen wrote: >attach the design document, please take a look. > >Best Regards >Daniel(Yugang) > ># Modular System Support - PCH IO Expander > >Author: Chen, Yugang > >## Problem Description >When multi modular 2 socket reference platforms (RP) are connected together, >the BMCs on each RP shall support the ability to manage the system as either >single 4S/8S system or standalone as 2S systems. > >## Background and References >In 4S/8S mode, only one PCH works at DMI (Direct Media Interface) mode, >this PCH is called legacy PCH (PCH.L), the other PCHs are called PCH IO Expander >(PCH.IO). The BMC attached to the PCH.L works as a primary and the other BMCs >work as secondaries. > >## Requirements >Check a modular system as either 4S/8S or standalone mode by GPIOs on BMC each >time system AC on. >Each BMC works independently in standalone mode. >In 4S/8S mode, only primary BMC is responsible for management of system level >resources, secondary BMCs need to collect its local management resources and report >them to primary BMC. > >## Proposed Design >Each BMC will have a property to reveal its work mode. Users can know if BMC is in standalone mode or 4S/8S mode. >Each BMC has another property to show if the BMC mode is PCH Legacy BMC or >Non-Legacy BMC. PCH Legacy BMC is working as primary and Non-Legacy BMC working >as secondary. > >Each time a BMC reboot, The BMC needs to check 3 GPIO Pins: FM_STANDALONE_MONDE_N / FM_4S_8S_N_MODE / FM_NODE_ID to get working mode as standalone mode or primary /secondary(4S/8S) role according to the GPIO values. >After confirming the mode and BMCs' role, BMCs should set properties according >to the correct configuration. >In 4S/8S mode only node id 0 will be primary BMC because only this node will be the PCH.L. Node id 1,2,3 will be the secondary nodes. > >Once a BMC gets mode is in 4S/8S, node roles are configured by node >ID (GPIO Pins) and keep consistent once AC on. Once node role check is done, > >In 4S/8S mode, Primary BMC needs to broadcast its role to make sure there is only >one primary BMC in the system. > >Need a new feature to make sure secondary BMCs send local redfish events to primary >BMC. And primary BMC needs to add a tag to those events coming from secondary BMC so >that user can know the event logs happened on which node. > >Even in 4S/8S mode, each BMC will collect its local management resources, >including sensors, fans and do FSC according to the values of local sensors. >PSU and Fans on each node will not be connected together. Configuration >settings of each secondary node will remain the same, and won't be synced across >the nodes. > >In 4S/8S mode, PECI will only be connected to primary node. Primary BMC needs to >monitor all CPU and DIMM sensors and deliver the sensor values of the CPUs/DIMMs >on secondary nodes to secondary BMCs. So that secondary BMCs can use the sensor values to control their own FSCs. Primary BMC also needs to have a way to find how many >CPUs are in the whole system include Primary and Secondary nodes. > >## Alternatives Considered >Primary node monitors all the IPMI sensors in secondary nodes and creates redfish log >by itself. > >Instead of BMC reboot, only AC cycle will make BMCs check GPIO pins and set >Legacy BMC or Non-Legacy BMC mode. > >Only primary BMC broadcast its role and secondary only waiting for the broadcast >from primary. > >## Impacts >Only on the motherboard where legacy PCH is located, POST code/Front Panel/KCS >port/UART will work, while these interfaces on board with non-legacy PCH will >not work due to BIOS and HW design. And this will cause non-functioning of >SOL/KVM/Virtual media on secondary BMCs. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: apply for a new repo "openbmc/node-data-sync" 2021-04-19 14:26 ` Brad Bishop @ 2021-04-23 13:04 ` Patrick Williams 0 siblings, 0 replies; 7+ messages in thread From: Patrick Williams @ 2021-04-23 13:04 UTC (permalink / raw) To: yugang.chen; +Cc: Bills, Jason M, Ed Tanous, chunhui.jia, openbmc [-- Attachment #1: Type: text/plain, Size: 7438 bytes --] Hello Daniel, I've read through this and feel like most of my original questions are still unanswered. I've pasted them here again for reference: --- > So, we need a new repo to sync the sensor data between primary node and +secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks. Do you have a design on how this would be accomplished? I'm curious what your proposal ends up looking like at a dbus-level. We should make sure this is documented at a high level and also aligns with how we are handling multi-host scenarios already in our dbus layout. How are the two BMC devices connected? How do you ensure you can safely / securely accept the data between the two BMCs? If done poorly this could open up a large security attack surface. I suspect your software design is somewhat driven by a particular hardware design as well, so we need to make sure it is clear the scoping of your solution and should probably even communicate that in the name. For instance, if your data transport is IPMB over i2c and you don't plan on this being a general daemon that could be expanded to a network connection, we shouldn't name it so generally. --- On Mon, Apr 19, 2021 at 10:26:31AM -0400, Brad Bishop wrote: > Hi Daniel > > Typically email attachments are not opened. To ensure your mail gets > read, avoid attachments. Resending without the attachment... > > On Mon, Apr 19, 2021 at 12:41:00PM +0800, yugang.chen wrote: > >attach the design document, please take a look. > > > >Best Regards > >Daniel(Yugang) > > > > >## Requirements > >resources, secondary BMCs need to collect its local management resources and report > >them to primary BMC. Why is this a requirement? What is driving it? You could alternatively have DC-level software gather data directly from both BMCs. Please put that in the alternative and give rationale on why this extra software is a better solution to the problem. > >## Proposed Design > >Each time a BMC reboot, The BMC needs to check 3 GPIO Pins: FM_STANDALONE_MONDE_N / FM_4S_8S_N_MODE / FM_NODE_ID to get working mode as standalone mode or primary /secondary(4S/8S) role according to the GPIO values. > >After confirming the mode and BMCs' role, BMCs should set properties according > >to the correct configuration. > >In 4S/8S mode only node id 0 will be primary BMC because only this node will be the PCH.L. Node id 1,2,3 will be the secondary nodes. Some of this is very specific to your hardware design / architecture being used (ex. "PCH" is an Intel term). We may want to put some of this into a background or architecture-specific section. I get two high-level design points from reading this in my mind: 1. You are not handling dynamic primary/secondary BMC selection; it is all assigned by physical position. This is not unreasonable but it isn't sufficient for some other designs, so we should spell this out. 2. Assignment of role will be done based on GPIOs for your particular implementation, but could be handled in a different manner for other systems. Based on #2, I would suggest the role-selection code is separated from any dbus/Redfish propagation code. The role should likely be a dbus property. Also, these GPIO names seem like they are schematic names. You might want to look at docs:designs/device-tree-gpio-naming.md. We typically want to see a logical name from a userspace application point of view. > > > >Once a BMC gets mode is in 4S/8S, node roles are configured by node > >ID (GPIO Pins) and keep consistent once AC on. I think I know what you meant by this, but the wording is very difficult for BMCs to accomplish. OCP systems don't even have "AC power" since the server is supplied DC 12 or 48v from the rack. The BMC has almost no way to know if it was reset in certain ways vs had standby power cycled. I would suggest at a hardware level that the node roles are kept static during the entire standby power cycle. Likely this is what you meant, but the wording here could be clearer. > >Once node role check is done, > > > >In 4S/8S mode, Primary BMC needs to broadcast its role to make sure there is only > >one primary BMC in the system. And what if not? > >Need a new feature to make sure secondary BMCs send local redfish events to primary > >BMC. And primary BMC needs to add a tag to those events coming from secondary BMC so > >that user can know the event logs happened on which node. Why Redfish and not dbus? I think this is the first time that Redfish is mentioned. Does the primary really just need "events" or does it really need all of the data necessary to create a Redfish model of the resources modeled by the secondary? I assume the latter, because if it was just events then something still needs to go to the secondary to get the rest of the model and to make changes to it. > >Even in 4S/8S mode, each BMC will collect its local management resources, > >including sensors, fans and do FSC according to the values of local sensors. > >PSU and Fans on each node will not be connected together. Configuration > >settings of each secondary node will remain the same, and won't be synced across > >the nodes. FSC? Fan Speed Control? Please don't use new and previously unused acronyms, unless they are a well-known / industry standard protocol. > >In 4S/8S mode, PECI will only be connected to primary node. Primary BMC needs to > >monitor all CPU and DIMM sensors and deliver the sensor values of the CPUs/DIMMs > >on secondary nodes to secondary BMCs. So that secondary BMCs can use the sensor values to control their own FSCs. Primary BMC also needs to have a way to find how many > >CPUs are in the whole system include Primary and Secondary nodes. This paragraph implies to me that the primary is also *pushing* sensor values to the secondary so that the secondary can make local decisions about fan speeds from information obtained over PECI on the primary? It really seems like Redfish "events" is not sufficient for what you're trying to accomplish, but you want some generic "sync dbus state between BMCs" design. > >## Alternatives Considered All of these alternatives need an explaination of why this alternative was not chosen. > >Primary node monitors all the IPMI sensors in secondary nodes and creates redfish log > >by itself. Is "IPMI sensors" really what you wanted to say here? > >Instead of BMC reboot, only AC cycle will make BMCs check GPIO pins and set > >Legacy BMC or Non-Legacy BMC mode. I didn't get the impression above that BMC reboot was when GPIO pins were checked (and part of the reason why I pointed that out above). > >Only primary BMC broadcast its role and secondary only waiting for the broadcast > >from primary. I'm not sure what this means... > >## Impacts > >Only on the motherboard where legacy PCH is located, POST code/Front Panel/KCS > >port/UART will work, while these interfaces on board with non-legacy PCH will > >not work due to BIOS and HW design. And this will cause non-functioning of > >SOL/KVM/Virtual media on secondary BMCs. We are interested in *software* impacts here. What code needs to change? How does this new design affect the way we think about existing software components? -- Patrick Williams [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: apply for a new repo "openbmc/node-data-sync"
@ 2021-04-19 5:10 Milton Miller II
0 siblings, 0 replies; 7+ messages in thread
From: Milton Miller II @ 2021-04-19 5:10 UTC (permalink / raw)
To: yugang.chen, Ed Tanous; +Cc: openbmc, Brad Bishop, Bills, Jason M, chunhui.jia
[-- Attachment #1.1: Type: text/plain, Size: 204 bytes --]
yugang.chen wrote:
> attach the design document, please take a look.
Please create a gerrit review placing this in an appropriate directory in the OpenBMC docs repository.
Thanks,
Milton
[-- Attachment #1.2: Type: text/html, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in threadend of thread, other threads:[~2021-04-23 13:11 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <cab2988f-76b8-6a30-5fa9-0ee8030af7f0@linux.intel.com>
[not found] ` <AF9887DB-F6CB-4CE0-90B1-17FEA2497013@fuzziesquirrel.com>
2021-04-12 6:38 ` apply for a new repo "openbmc/node-data-sync" yugang.chen
2021-04-12 11:29 ` Patrick Williams
2021-04-12 19:26 ` Ed Tanous
2021-04-19 4:41 ` yugang.chen
2021-04-19 14:26 ` Brad Bishop
2021-04-23 13:04 ` Patrick Williams
2021-04-19 5:10 Milton Miller II
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).