* CXL Fabric Manager (FM) architecture diagrams
@ 2023-03-06 18:59 Viacheslav Dubeyko
2023-03-07 17:21 ` Jonathan Cameron
0 siblings, 1 reply; 6+ messages in thread
From: Viacheslav Dubeyko @ 2023-03-06 18:59 UTC (permalink / raw)
To: Jonathan.Cameron
Cc: linux-cxl, a.manzanares, viacheslav.dubeyko, Viacheslav Dubeyko
Hi Jonathan,
You can find diagram online now:
(1) Diagram 1: single host with Fabric Manager (FM)
https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram1-cxl-fm-single-host-with-fm.pdf
(2) Diagram 2: single host with orchestrator
https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram2-cxl-fm-single-host-with-orhestrator.pdf
(3) Diagram3: Multi-headed device
https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram3-cxl-fm-multi-headed-device.pdf
(4) Diagram 4: Multi-headed device + Orchestrator
https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram4-cxl-fm-multi-headed-device-orchestrator.pdf
(5) Diagram5: Multiple hosts with Fabric Manager (FM)
https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram5-cxl-fm-multiple-hosts-with-fm.pdf
(6) Diagram 6: Multiple hosts with orhestrator
https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram6-cxl-fm-multiple-hosts-with-orhestrator.pdf
(7) Diagram 7: Distributed Fabric Manager (FM)
https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram7-cxl-fm-distributed-fm.pdf
(8) Diagram 8: Layered Fabric Manager (FM) and separate orchestrator
https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram8-cxl-fm-layered-fm-and-separate-orchestrator.pdf
(9) Diagram 9: BMC based layered Fabric Manager (FM)
https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram9-cxl-fm-bmc-based-layered-fm.pdf
So, have I missed something?
Should we correct something on diagrams?
Does it look good?
Thanks,
Slava.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: CXL Fabric Manager (FM) architecture diagrams
2023-03-06 18:59 CXL Fabric Manager (FM) architecture diagrams Viacheslav Dubeyko
@ 2023-03-07 17:21 ` Jonathan Cameron
2023-03-09 22:28 ` [External] " Viacheslav A.Dubeyko
0 siblings, 1 reply; 6+ messages in thread
From: Jonathan Cameron @ 2023-03-07 17:21 UTC (permalink / raw)
To: Viacheslav Dubeyko; +Cc: linux-cxl, a.manzanares, viacheslav.dubeyko
On Mon, 6 Mar 2023 10:59:13 -0800
Viacheslav Dubeyko <slava@dubeyko.com> wrote:
> Hi Jonathan,
>
> You can find diagram online now:
>
> (1) Diagram 1: single host with Fabric Manager (FM)
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram1-cxl-fm-single-host-with-fm.pdf
Whilst accurate that you might only be able to control the switch, I would add one MLD to this
diagram so that tunneling can be discussed. It's minimal in sense of near minimal number of
components that might exist in a switched system
Thanks to asciiflow.com + some editing of the resutl - fingers crossed this works.
Obviously these can't be as rich as your nice diagrams so I've
left out what the connections are, but having them inline has
it's advantages as well.
I've played fast and loose with some of the terminology and not checked
it against the spec naming. I've also left it vague how an application
talks to the orchestrator. That was via agent in your terms I think
┌───────────────────┐ ┌───────────────┐ ┌────────────────┐
│ │ │ │ │ │
│ │ │ ┌───────┐ │ │ ┌────────────┐ │
│ │ │ │ ├───┼───┼─►FM Owned LD │ │
│ ┌───┐ │ │ │Switch │ │ │ └────────────┘ │
│ │FM ├───┼────┼───► CCI │ ├───┤ │
│ └───┘ │ │ │ │ │ │ MLD 1 │
│ │ │ └──────┬┘ │ └────────────────┘
│ ┌────┤ │ │ │
│ │RP0 ├────┤ │ │ ┌────────────────┐
│ └────┤ │ │ │ │ │
│ │ │ │ │ │ ┌────────────┐ │
│ ┌────┤ │ └────┼───┼─►FM Owned LD │ │
│ │RP1 ├────┤ ├───┤ └────────────┘ │
│ └────┤ │ │ │ │
│ Host A │ │ SWITCH │ │ MLD 2 │
└───────────────────┘ └───────────────┘ └────────────────┘
>
> (2) Diagram 2: single host with orchestrator
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram2-cxl-fm-single-host-with-orhestrator.pdf
Actually having an FM in a switch might happen, but there is no spec defined way of doing it.
From a software architecture point of view it's no different from another host talking to the
switch - just think of sticking a BMC down next to the switch chip.
Having the orchestrator in the host is also rather odd though it could in theory happen.
Typically the orchestrator is considered a 'cloud' level thing floating way above individual host.
Without any loss of generality I'd always have the orchestrator as something on it's own
machine chatting over a network to the Agents and FM-API accessed devices.
Something like
┌──────────────────────┐ ┌────────┐
│ │ │ │
│ │ │ │
│ Orchestrator ──────┼────► FM │
│ │ │ │
└───▲──────────────────┘ └─┬──────┘
│ │
┌─────┼─────────────┐ ┌──────┼────────┐ ┌────────────────┐
│ │ │ │ │ │ │ │
│ │ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
│ │ │ │ │ Switch ├───┼───┼─►FM Owned LD │ │
│ ┌───┴──┐ │ │ │ CCI │ │ │ └────────────┘ │
│ │ APP │ │ │ │ or MCTP│ ├───┤ │
│ │ │ │ │ │ CCI │ │ │ MLD 1 │
│ └──────┘ │ │ └───────┬┘ │ └────────────────┘
│ ┌────┤ │ │ │
│ │RP0 ├────┤ │ │ ┌────────────────┐
│ └────┤ │ │ │ │ │
│ │ │ │ │ │ ┌────────────┐ │
│ ┌────┤ │ └────┼───┼─►FM Owned LD │ │
│ │RP1 ├────┤ ├───┤ └────────────┘ │
│ └────┤ │ │ │ │
│ Host A │ │ SWITCH │ │ MLD 2 │
└───────────────────┘ └───────────────┘ └────────────────┘
>
> (3) Diagram3: Multi-headed device
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram3-cxl-fm-multi-headed-device.pdf
Looks good though there is a simpler single host version.
┌───────────────────┐ ┌─────────────────────┐
│ ┌───┐ │ │ ┌────────────┐ │
│ │FM ├───┼────┼─► Mailbox CCI│ │
│ └───┘ │ │ └────────────┘ │
│ │ │ ┌─────▼───────┐ │
│ ┌────┤ │ │ LD Pool CCI │ │
│ │RP0 ├────┤ └─────────────┘ │
│ └────┤ │ │
│ ┌────┤ │ │
│ │RP1 ├────┤ │
│ └────┤ │ │
│ Host A │ │ MHD 1 │
└───────────────────┘ └─────────────────────┘
I'd jump from single host to the multi host with external FM and
orchestator.
┌──────────────────────┐ ┌────────┐
│ │ │ │
│ │ │ │
┌─────► Orchestrator ──────┼────► FM │
│ │ │ │ │
│ └───▲──────────────────┘ └────┬───┘
│ │ │
│ ┌─────┼─────────────┐ ┌───────┼─────────────┐
│ │ │ │ │ ┌─────▼───────┐ │
│ │ │ │ │ │ Mailbox CCI │ │
│ │ ┌───┴──┐ ┌───┤ │ └─────-───────┘ │
│ │ │ APP │ │RP0├──────┤ │ │
│ │ │ │ └───┤ │ ┌─────▼───────┐ │
│ │ └──────┘ │ │ │ LD Pool CCI │ │
│ │ │ │ └─────────────┘ │
│ │ Host A │ │ │
│ └───────────────────┘ │ │
│ ┌───────────────────┐ │ │
└───┼─┬──────┐ ┌───┤ │ │
│ │ APP │ │RP0├──────┤ │
│ │ │ └───┤ │ │
│ └──────┘ │ │ │
│ Host B │ │ MHD 1 │
└───────────────────┘ └─────────────────────┘
>
> (4) Diagram 4: Multi-headed device + Orchestrator
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram4-cxl-fm-multi-headed-device-orchestrator.pdf
I'd put the orchestrator in it's own 'host' as above... I've drawn it with a mailbox cci but
could be a mctp CCI.
>
> (5) Diagram5: Multiple hosts with Fabric Manager (FM)
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram5-cxl-fm-multiple-hosts-with-fm.pdf
Another one where I'd separate the FM from the switch. It may be near the switch but
it's talking fm-api to the switch and that's an interface that is well defined.
┌──────────────────────┐ ┌────────┐
┌─────► Orchestrator ──────┼────► FM │
│ └───▲──────────────────┘ └────┬───┘
│ ┌─────┼─────────────┐ │
│ │ │ │ ┌──────┼────────┐ ┌────────────────┐
│ │ │ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
│ │ ┌───┴──┐ ┌───┤ │ │ Switch ├───┼───┼─►FM Owned LD │ │
│ │ │ APP │ │RP0├───────┤ │ CCI │ │ │ └────────────┘ │
│ │ │ │ └───┤ │ │ or MCTP│ ├───┤ │
│ │ └──────┘ │ │ │ CCI │ │ │ MLD 1 │
│ │ │ │ └───────┬┘ │ └────────────────┘
│ │ Host A │ │ │ │
│ └───────────────────┘ │ │ │ ┌────────────────┐
│ ┌───────────────────┐ │ │ │ │ ┌────────────┐ │
│ │ │ │ └────┼───┼─►FM Owned LD │ │
│ │ │ │ ├───┤ └────────────┘ │
│ │ │ │ │ │ │
└───┼─┬──────┐ ┌───┤ │ SWITCH │ │ MLD 2 │
│ │ APP │ │RP0├───────┤ │ └────────────────┘
│ │ │ └───┤ │ │
│ └──────┘ │ └───────────────┘
│ Host B │
└───────────────────┘
>
> (6) Diagram 6: Multiple hosts with orhestrator
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram6-cxl-fm-multiple-hosts-with-orhestrator.pdf
I'm not sure there is a need to separate the case where there
is an orchestrator in the loop from when there isn't. Hence I just threw one on the diagram above.
>
> (7) Diagram 7: Distributed Fabric Manager (FM)
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram7-cxl-fm-distributed-fm.pdf
>
Looks like a set of FMs, not what I'd think of as a distributed FM.
Only makes sense to me if more like this...
┌──────────────────────┐
┌───► Orchestrator ──────┼─────────────────────────────────────────┐
│ │ ▲ │ │
│ └───┬──────────┬───────┘ │
│ │ │ │
│ │ │ ┌────────┐ │
│ │ └────────────► FM1 │ │
│ │ └────┬───┘ │
│ ┌─────┼─────────────┐ │ │
│ │ │ │ ┌──────┼────────┐ ┌────────────────┐ │
│ │ │ ┌───┤ │ │ │ │ │ │
│ │ │ │RP0├───────┤ ┌───▼────┐ │ │ ┌────────────┐ │ │
│ │ ┌───┴──┐ └───┤ │ │ Switch ├───┼───┼─►FM Owned LD │ │ │
│ │ │ APP │ │ │ │ CCI │ │ │ └────────────┘ │ │
│ │ │ │ ┌───┤ │ │ or MCTP│ ├───┤ │ │
│ │ └──────┘ │RP1├─────┐ │ │ CCI │ │ │ MLD 1 │ │
│ │ └───┤ │ │ └───────┬┘ │ └────────────────┘ │
│ │ Host A │ │ │ │ │ │
│ └───────────────────┘ │ │ │ │ ┌────────────────┐ │
│ │ │ │ │ │ │ │
│ ┌───────────────────┐ │ │ │ │ │ ┌────────────┐ │ │
│ │ │ │ │ └────┼───┼─►FM Owned LD │ │ │
│ │ ┌───┤ │ │ ├───┤ └────────────┘ │ │
│ │ │RP0├─────┼─┤ │ │ │ │
│ │ ┌──────┐ └───┤ │ │ SWITCH 1 │ │ MLD 2 │ │
└─┼─┤ APP │ │ │ │ │ └────────────────┘ │
│ │ │ ┌───┤ │ │ │ │
│ └──────┘ │RP1├──┐ │ └───────────────┘ │
│ └───┤ │ │ │
│ Host B │ │ │ │
└───────────────────┘ │ │ │
│ │ ┌────────┐ │
│ │ │ FM2 ◄───────────────────────────┘
│ │ └────┬───┘
│ │ ┌──────┼────────┐ ┌────────────────┐
│ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
│ │ │ │ Switch ├───┼───┼─►FM Owned LD │ │
│ └─┤ │ CCI │ │ │ └────────────┘ │
│ │ │ or MCTP│ ├───┤ │
│ │ │ CCI │ │ │ MLD 3 │
│ │ └───────┬┘ │ └────────────────┘
│ │ │ │ ┌────────────────┐
│ │ │ │ │ ┌────────────┐ │
│ │ └────┼───┼─►FM Owned LD │ │
│ │ ├───┤ └────────────┘ │
│ │ │ │ │
│ │ SWITCH 2 │ │ MLD 4 │
└────┤ │ └────────────────┘
└───────────────┘
> (8) Diagram 8: Layered Fabric Manager (FM) and separate orchestrator
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram8-cxl-fm-layered-fm-and-separate-orchestrator.pdf
I don't follow the layered part on this diagram. My interpretation would
be something like.
┌──────────────────────┐ ┌────────┐
┌─────► Orchestrator ──────┼────► TOP FM ├──────────────────────────────┐
│ │ ▲ │ │ │ │
│ └───┬──────────────────┘ └────┬───┘ │
│ │ │ │
│ │ ┌────┴───┐ │
│ │ │ BOTTOM │ │
│ │ │ FM1 │ │
│ │ └────┬───┘ │
│ ┌─────┼─────────────┐ │ │
│ │ │ │ ┌──────┼────────┐ ┌────────────────┐ │
│ │ │ ┌───┤ │ │ │ │ │ │
│ │ │ │RP0├───────┤ ┌───▼────┐ │ │ ┌────────────┐ │ │
│ │ ┌───┴──┐ └───┤ │ │ Switch ├───┼───┼─►FM Owned LD │ │ │
│ │ │ APP │ │ │ │ CCI │ │ │ └────────────┘ │ │
│ │ │ │ ┌───┤ │ │ or MCTP│ ├───┤ │ │
│ │ └──────┘ │RP1├─────┐ │ │ CCI │ │ │ MLD 1 │ │
│ │ └───┤ │ │ └───────┬┘ │ └────────────────┘ │
│ │ Host A │ │ │ │ │ │
│ └───────────────────┘ │ │ │ │ ┌────────────────┐ │
│ │ │ │ │ │ │ │
│ │ │ │ │ │ ┌────────────┐ │ │
│ ┌───────────────────┐ │ │ └────┼───┼─►FM Owned LD │ │ │
│ │ ┌───┤ │ │ ├───┤ └────────────┘ │ │
│ │ │RP0├─────┼─┤ │ │ │ │
│ │ ┌──────┐ └───┤ │ │ SWITCH 1 │ │ MLD 2 │ │
└───┼─┤ APP │ │ │ │ │ └────────────────┘ │
│ │ │ ┌───┤ │ │ │ │
│ └──────┘ │RP1├──┐ │ └───────────────┘ │
│ └───┤ │ │ │
│ Host B │ │ │ │
└───────────────────┘ │ │ │
│ │ ┌────────┐ │
│ │ │ BOTTOM │ │
│ │ │ FM2 ◄──────────────────────────────┘
│ │ └────┬───┘
│ │ ┌──────┼────────┐ ┌────────────────┐
│ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
│ │ │ │ Switch ├───┼───┼─►FM Owned LD │ │
│ └─┤ │ CCI │ │ │ └────────────┘ │
│ │ │ or MCTP│ ├───┤ │
│ │ │ CCI │ │ │ MLD 3 │
│ │ └───────┬┘ │ └────────────────┘
│ │ │ │ ┌────────────────┐
│ │ │ │ │ ┌────────────┐ │
│ │ └────┼───┼─►FM Owned LD │ │
│ │ ├───┤ └────────────┘ │
│ │ SWITCH 2 │ │ MLD 4 │
└────┤ │ └────────────────┘
└───────────────┘
>
> (9) Diagram 9: BMC based layered Fabric Manager (FM)
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram9-cxl-fm-bmc-based-layered-fm.pdf
I don't follow what the BMC is in this diagram.
The BMC is just a (cheap) host that happens to have some elements of the overall management
infrastructure on it. Let it be any of the FMs floating around on their own in the
diagrams above. The diagram immediately above might be built with 3 BMCs or
as a single BMC like this...
┌──────────────────────┐
┌───► Orchestrator │
│ └───▲──────────┬───────┘ ┌────────┐
│ │ │ │ ┌────┐ │
│ │ └────────────►─┤FM │ ├───────────────────────────┐
│ │ │ └──┬─┘ │ │
│ │ │BMC │ │ │
│ │ └────┼───┘ │
│ ┌─────┼─────────────┐ │ │
│ │ │ │ ┌──────┼────────┐ ┌────────────────┐ │
│ │ │ ┌───┤ │ │ │ │ │ │
│ │ │ │RP0├───────┤ ┌───▼────┐ │ │ ┌────────────┐ │ │
│ │ ┌───┴──┐ └───┤ │ │ Switch ├───┼───┼─►FM Owned LD │ │ │
│ │ │ APP │ │ │ │ CCI │ │ │ └────────────┘ │ │
│ │ │ │ ┌───┤ │ │ or MCTP│ ├───┤ │ │
│ │ └──────┘ │RP1├─────┐ │ │ CCI │ │ │ MLD 1 │ │
│ │ └───┤ │ │ └───────┬┘ │ └────────────────┘ │
│ │ Host A │ │ │ │ │ │
│ └───────────────────┘ │ │ │ │ ┌────────────────┐ │
│ │ │ │ │ │ │ │
│ ┌───────────────────┐ │ │ │ │ │ ┌────────────┐ │ │
│ │ │ │ │ └────┼───┼─►FM Owned LD │ │ │
│ │ ┌───┤ │ │ ├───┤ └────────────┘ │ │
│ │ │RP0├─────┼─┤ │ │ │ │
│ │ ┌──────┐ └───┤ │ │ SWITCH 1 │ │ MLD 2 │ │
└─┼─┤ APP │ │ │ │ │ └────────────────┘ │
│ │ │ ┌───┤ │ │ │ │
│ └──────┘ │RP1├──┐ │ └───────────────┘ │
│ └───┤ │ │ │
│ Host B │ │ │ │
└───────────────────┘ │ │ │
│ │ ┌───────────────────────────────┘
│ │ ┌──────┼────────┐ ┌────────────────┐
│ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
│ │ │ │ Switch ├───┼───┼─►FM Owned LD │ │
│ └─┤ │ CCI │ │ │ └────────────┘ │
│ │ │ or MCTP│ ├───┤ │
│ │ │ CCI │ │ │ MLD 3 │
│ │ └───────┬┘ │ └────────────────┘
│ │ │ │ ┌────────────────┐
│ │ │ │ │ ┌────────────┐ │
│ │ └────┼───┼─►FM Owned LD │ │
│ │ ├───┤ └────────────┘ │
│ │ SWITCH 2 │ │ MLD 4 │
└────┤ │ └────────────────┘
└───────────────┘
>
> So, have I missed something?
> Should we correct something on diagrams?
> Does it look good?
Thare are far too many things we should perhaps show on these diagrams, but
I suspect it will mostly fall out of any layered design.
We could draw one incredibly complex diagram that does everything :)
*crossed fingers the ascii art fun above looks ok*
Thanks for getting this started btw! I was completely failing to start
whereas once there was a proposal it became easier to have a go!
Jonathan
>
> Thanks,
> Slava.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [External] CXL Fabric Manager (FM) architecture diagrams
2023-03-07 17:21 ` Jonathan Cameron
@ 2023-03-09 22:28 ` Viacheslav A.Dubeyko
[not found] ` <20230310120126.000057b9@Huawei.com>
0 siblings, 1 reply; 6+ messages in thread
From: Viacheslav A.Dubeyko @ 2023-03-09 22:28 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: Viacheslav Dubeyko, linux-cxl, a.manzanares
> On Mar 7, 2023, at 9:21 AM, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>
> On Mon, 6 Mar 2023 10:59:13 -0800
> Viacheslav Dubeyko <slava@dubeyko.com> wrote:
>
>> Hi Jonathan,
>>
>> You can find diagram online now:
>>
>> (1) Diagram 1: single host with Fabric Manager (FM)
>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram1-cxl-fm-single-host-with-fm.pdf
>
> Whilst accurate that you might only be able to control the switch, I would add one MLD to this
> diagram so that tunneling can be discussed. It's minimal in sense of near minimal number of
> components that might exist in a switched system
>
> Thanks to asciiflow.com + some editing of the resutl - fingers crossed this works.
> Obviously these can't be as rich as your nice diagrams so I've
> left out what the connections are, but having them inline has
> it's advantages as well.
>
I’ve copied ascii diagrams into pure text document and I can see the diagrams properly aligned. :)
> I've played fast and loose with some of the terminology and not checked
> it against the spec naming. I've also left it vague how an application
> talks to the orchestrator. That was via agent in your terms I think
>
> ┌───────────────────┐ ┌───────────────┐ ┌────────────────┐
> │ │ │ │ │ │
> │ │ │ ┌───────┐ │ │ ┌────────────┐ │
> │ │ │ │ ├───┼───┼─►FM Owned LD │ │
> │ ┌───┐ │ │ │Switch │ │ │ └────────────┘ │
> │ │FM ├───┼────┼───► CCI │ ├───┤ │
> │ └───┘ │ │ │ │ │ │ MLD 1 │
> │ │ │ └──────┬┘ │ └────────────────┘
> │ ┌────┤ │ │ │
> │ │RP0 ├────┤ │ │ ┌────────────────┐
> │ └────┤ │ │ │ │ │
> │ │ │ │ │ │ ┌────────────┐ │
> │ ┌────┤ │ └────┼───┼─►FM Owned LD │ │
> │ │RP1 ├────┤ ├───┤ └────────────┘ │
> │ └────┤ │ │ │ │
> │ Host A │ │ SWITCH │ │ MLD 2 │
> └───────────────────┘ └───────────────┘ └────────────────┘
>
I assume that RP0/RP1 are Root Ports. Should RP0/RP1 be on switch side?
I think APP should be shown too on this diagram. What do you think?
>>
>> (2) Diagram 2: single host with orchestrator
>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram2-cxl-fm-single-host-with-orhestrator.pdf
>
> Actually having an FM in a switch might happen, but there is no spec defined way of doing it.
> From a software architecture point of view it's no different from another host talking to the
> switch - just think of sticking a BMC down next to the switch chip.
> Having the orchestrator in the host is also rather odd though it could in theory happen.
> Typically the orchestrator is considered a 'cloud' level thing floating way above individual host.
> Without any loss of generality I'd always have the orchestrator as something on it's own
> machine chatting over a network to the Agents and FM-API accessed devices.
>
I agree that orchestrator could be standalone system. But diagram 2 is an example of simple case
when FM is located outside of the host (CXL switch or something else). So, I mean here simple
subsystem (dedicated application or library) that can talk with FM. I believe that standalone
orchestrator could be overkill for such case. What do you think?
> Something like
>
>
> ┌──────────────────────┐ ┌────────┐
> │ │ │ │
> │ │ │ │
> │ Orchestrator ──────┼────► FM │
> │ │ │ │
> └───▲──────────────────┘ └─┬──────┘
> │ │
> ┌─────┼─────────────┐ ┌──────┼────────┐ ┌────────────────┐
> │ │ │ │ │ │ │ │
> │ │ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
> │ │ │ │ │ Switch ├───┼───┼─►FM Owned LD │ │
> │ ┌───┴──┐ │ │ │ CCI │ │ │ └────────────┘ │
> │ │ APP │ │ │ │ or MCTP│ ├───┤ │
> │ │ │ │ │ │ CCI │ │ │ MLD 1 │
> │ └──────┘ │ │ └───────┬┘ │ └────────────────┘
> │ ┌────┤ │ │ │
> │ │RP0 ├────┤ │ │ ┌────────────────┐
> │ └────┤ │ │ │ │ │
> │ │ │ │ │ │ ┌────────────┐ │
> │ ┌────┤ │ └────┼───┼─►FM Owned LD │ │
> │ │RP1 ├────┤ ├───┤ └────────────┘ │
> │ └────┤ │ │ │ │
> │ Host A │ │ SWITCH │ │ MLD 2 │
> └───────────────────┘ └───────────────┘ └────────────────┘
>
Most probably, FM outside of CXL switch will talk by means of Redfish protocol.
Do you mean that MCTP CCI is subprotocol of Redfish? Should we introduce some
CXL switch’s subsystem that can talk by means of Redfish?
>>
>> (3) Diagram3: Multi-headed device
>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram3-cxl-fm-multi-headed-device.pdf
>
> Looks good though there is a simpler single host version.
>
> ┌───────────────────┐ ┌─────────────────────┐
> │ ┌───┐ │ │ ┌────────────┐ │
> │ │FM ├───┼────┼─► Mailbox CCI│ │
> │ └───┘ │ │ └────────────┘ │
> │ │ │ ┌─────▼───────┐ │
> │ ┌────┤ │ │ LD Pool CCI │ │
> │ │RP0 ├────┤ └─────────────┘ │
> │ └────┤ │ │
> │ ┌────┤ │ │
> │ │RP1 ├────┤ │
> │ └────┤ │ │
> │ Host A │ │ MHD 1 │
> └───────────────────┘ └─────────────────────┘
>
> I'd jump from single host to the multi host with external FM and
> orchestator.
>
> ┌──────────────────────┐ ┌────────┐
> │ │ │ │
> │ │ │ │
> ┌─────► Orchestrator ──────┼────► FM │
> │ │ │ │ │
> │ └───▲──────────────────┘ └────┬───┘
> │ │ │
> │ ┌─────┼─────────────┐ ┌───────┼─────────────┐
> │ │ │ │ │ ┌─────▼───────┐ │
> │ │ │ │ │ │ Mailbox CCI │ │
> │ │ ┌───┴──┐ ┌───┤ │ └─────-───────┘ │
> │ │ │ APP │ │RP0├──────┤ │ │
> │ │ │ │ └───┤ │ ┌─────▼───────┐ │
> │ │ └──────┘ │ │ │ LD Pool CCI │ │
> │ │ │ │ └─────────────┘ │
> │ │ Host A │ │ │
> │ └───────────────────┘ │ │
> │ ┌───────────────────┐ │ │
> └───┼─┬──────┐ ┌───┤ │ │
> │ │ APP │ │RP0├──────┤ │
> │ │ │ └───┤ │ │
> │ └──────┘ │ │ │
> │ Host B │ │ MHD 1 │
> └───────────────────┘ └─────────────────────┘
>
>
It looks like FM needs to use network + Redfish to interact with Multi-Headed Device (MHD).
Could Multi-Headed Device support network protocol? I assume it sounds like overkill.
>
>>
>> (4) Diagram 4: Multi-headed device + Orchestrator
>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram4-cxl-fm-multi-headed-device-orchestrator.pdf
> I'd put the orchestrator in it's own 'host' as above... I've drawn it with a mailbox cci but
> could be a mctp CCI.
>>
>> (5) Diagram5: Multiple hosts with Fabric Manager (FM)
>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram5-cxl-fm-multiple-hosts-with-fm.pdf
>
> Another one where I'd separate the FM from the switch. It may be near the switch but
> it's talking fm-api to the switch and that's an interface that is well defined.
>
> ┌──────────────────────┐ ┌────────┐
> ┌─────► Orchestrator ──────┼────► FM │
> │ └───▲──────────────────┘ └────┬───┘
> │ ┌─────┼─────────────┐ │
> │ │ │ │ ┌──────┼────────┐ ┌────────────────┐
> │ │ │ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
> │ │ ┌───┴──┐ ┌───┤ │ │ Switch ├───┼───┼─►FM Owned LD │ │
> │ │ │ APP │ │RP0├───────┤ │ CCI │ │ │ └────────────┘ │
> │ │ │ │ └───┤ │ │ or MCTP│ ├───┤ │
> │ │ └──────┘ │ │ │ CCI │ │ │ MLD 1 │
> │ │ │ │ └───────┬┘ │ └────────────────┘
> │ │ Host A │ │ │ │
> │ └───────────────────┘ │ │ │ ┌────────────────┐
> │ ┌───────────────────┐ │ │ │ │ ┌────────────┐ │
> │ │ │ │ └────┼───┼─►FM Owned LD │ │
> │ │ │ │ ├───┤ └────────────┘ │
> │ │ │ │ │ │ │
> └───┼─┬──────┐ ┌───┤ │ SWITCH │ │ MLD 2 │
> │ │ APP │ │RP0├───────┤ │ └────────────────┘
> │ │ │ └───┤ │ │
> │ └──────┘ │ └───────────────┘
> │ Host B │
> └───────────────────┘
The same question here for interaction of FM with CXL switch by Redfish.
Is MCTP CCI compatible with Redfish?
>>
>> (6) Diagram 6: Multiple hosts with orhestrator
>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram6-cxl-fm-multiple-hosts-with-orhestrator.pdf
>
> I'm not sure there is a need to separate the case where there
> is an orchestrator in the loop from when there isn't. Hence I just threw one on the diagram above.
>
The same logic here. Application need to use some library or simple subsystem that can talk
with FM. And standalone Orchestrator could be overkill here.
>>
>> (7) Diagram 7: Distributed Fabric Manager (FM)
>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram7-cxl-fm-distributed-fm.pdf
>>
> Looks like a set of FMs, not what I'd think of as a distributed FM.
> Only makes sense to me if more like this...
I think I don’t follow here what you mean by distributed FM. I see the same set of FMs on your
version of diagram. What’s the difference? :)
>
> ┌──────────────────────┐
> ┌───► Orchestrator ──────┼─────────────────────────────────────────┐
> │ │ ▲ │ │
> │ └───┬──────────┬───────┘ │
> │ │ │ │
> │ │ │ ┌────────┐ │
> │ │ └────────────► FM1 │ │
> │ │ └────┬───┘ │
> │ ┌─────┼─────────────┐ │ │
> │ │ │ │ ┌──────┼────────┐ ┌────────────────┐ │
> │ │ │ ┌───┤ │ │ │ │ │ │
> │ │ │ │RP0├───────┤ ┌───▼────┐ │ │ ┌────────────┐ │ │
> │ │ ┌───┴──┐ └───┤ │ │ Switch ├───┼───┼─►FM Owned LD │ │ │
> │ │ │ APP │ │ │ │ CCI │ │ │ └────────────┘ │ │
> │ │ │ │ ┌───┤ │ │ or MCTP│ ├───┤ │ │
> │ │ └──────┘ │RP1├─────┐ │ │ CCI │ │ │ MLD 1 │ │
> │ │ └───┤ │ │ └───────┬┘ │ └────────────────┘ │
> │ │ Host A │ │ │ │ │ │
> │ └───────────────────┘ │ │ │ │ ┌────────────────┐ │
> │ │ │ │ │ │ │ │
> │ ┌───────────────────┐ │ │ │ │ │ ┌────────────┐ │ │
> │ │ │ │ │ └────┼───┼─►FM Owned LD │ │ │
> │ │ ┌───┤ │ │ ├───┤ └────────────┘ │ │
> │ │ │RP0├─────┼─┤ │ │ │ │
> │ │ ┌──────┐ └───┤ │ │ SWITCH 1 │ │ MLD 2 │ │
> └─┼─┤ APP │ │ │ │ │ └────────────────┘ │
> │ │ │ ┌───┤ │ │ │ │
> │ └──────┘ │RP1├──┐ │ └───────────────┘ │
> │ └───┤ │ │ │
> │ Host B │ │ │ │
> └───────────────────┘ │ │ │
> │ │ ┌────────┐ │
> │ │ │ FM2 ◄───────────────────────────┘
> │ │ └────┬───┘
> │ │ ┌──────┼────────┐ ┌────────────────┐
> │ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
> │ │ │ │ Switch ├───┼───┼─►FM Owned LD │ │
> │ └─┤ │ CCI │ │ │ └────────────┘ │
> │ │ │ or MCTP│ ├───┤ │
> │ │ │ CCI │ │ │ MLD 3 │
> │ │ └───────┬┘ │ └────────────────┘
> │ │ │ │ ┌────────────────┐
> │ │ │ │ │ ┌────────────┐ │
> │ │ └────┼───┼─►FM Owned LD │ │
> │ │ ├───┤ └────────────┘ │
> │ │ │ │ │
> │ │ SWITCH 2 │ │ MLD 4 │
> └────┤ │ └────────────────┘
> └───────────────┘
>
>> (8) Diagram 8: Layered Fabric Manager (FM) and separate orchestrator
>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram8-cxl-fm-layered-fm-and-separate-orchestrator.pdf
>
> I don't follow the layered part on this diagram. My interpretation would
> be something like.
>
Yeah, your version looks more logical. But top FM and bottom FM sounds slightly not
obvious. Maybe, we need to name top FM as root FM and bottom FM as leaf FM?
However, leaf FM sounds confusing too.
But why do we need top FM at all? I considered Orchestrator as root of hierarchy that
can keep the knowledge about all FMs. Am I wrong here?
>
> ┌──────────────────────┐ ┌────────┐
> ┌─────► Orchestrator ──────┼────► TOP FM ├──────────────────────────────┐
> │ │ ▲ │ │ │ │
> │ └───┬──────────────────┘ └────┬───┘ │
> │ │ │ │
> │ │ ┌────┴───┐ │
> │ │ │ BOTTOM │ │
> │ │ │ FM1 │ │
> │ │ └────┬───┘ │
> │ ┌─────┼─────────────┐ │ │
> │ │ │ │ ┌──────┼────────┐ ┌────────────────┐ │
> │ │ │ ┌───┤ │ │ │ │ │ │
> │ │ │ │RP0├───────┤ ┌───▼────┐ │ │ ┌────────────┐ │ │
> │ │ ┌───┴──┐ └───┤ │ │ Switch ├───┼───┼─►FM Owned LD │ │ │
> │ │ │ APP │ │ │ │ CCI │ │ │ └────────────┘ │ │
> │ │ │ │ ┌───┤ │ │ or MCTP│ ├───┤ │ │
> │ │ └──────┘ │RP1├─────┐ │ │ CCI │ │ │ MLD 1 │ │
> │ │ └───┤ │ │ └───────┬┘ │ └────────────────┘ │
> │ │ Host A │ │ │ │ │ │
> │ └───────────────────┘ │ │ │ │ ┌────────────────┐ │
> │ │ │ │ │ │ │ │
> │ │ │ │ │ │ ┌────────────┐ │ │
> │ ┌───────────────────┐ │ │ └────┼───┼─►FM Owned LD │ │ │
> │ │ ┌───┤ │ │ ├───┤ └────────────┘ │ │
> │ │ │RP0├─────┼─┤ │ │ │ │
> │ │ ┌──────┐ └───┤ │ │ SWITCH 1 │ │ MLD 2 │ │
> └───┼─┤ APP │ │ │ │ │ └────────────────┘ │
> │ │ │ ┌───┤ │ │ │ │
> │ └──────┘ │RP1├──┐ │ └───────────────┘ │
> │ └───┤ │ │ │
> │ Host B │ │ │ │
> └───────────────────┘ │ │ │
> │ │ ┌────────┐ │
> │ │ │ BOTTOM │ │
> │ │ │ FM2 ◄──────────────────────────────┘
> │ │ └────┬───┘
> │ │ ┌──────┼────────┐ ┌────────────────┐
> │ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
> │ │ │ │ Switch ├───┼───┼─►FM Owned LD │ │
> │ └─┤ │ CCI │ │ │ └────────────┘ │
> │ │ │ or MCTP│ ├───┤ │
> │ │ │ CCI │ │ │ MLD 3 │
> │ │ └───────┬┘ │ └────────────────┘
> │ │ │ │ ┌────────────────┐
> │ │ │ │ │ ┌────────────┐ │
> │ │ └────┼───┼─►FM Owned LD │ │
> │ │ ├───┤ └────────────┘ │
> │ │ SWITCH 2 │ │ MLD 4 │
> └────┤ │ └────────────────┘
> └───────────────┘
>>
>> (9) Diagram 9: BMC based layered Fabric Manager (FM)
>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram9-cxl-fm-bmc-based-layered-fm.pdf
>
> I don't follow what the BMC is in this diagram.
>
> The BMC is just a (cheap) host that happens to have some elements of the overall management
> infrastructure on it. Let it be any of the FMs floating around on their own in the
> diagrams above. The diagram immediately above might be built with 3 BMCs or
> as a single BMC like this...
>
> ┌──────────────────────┐
> ┌───► Orchestrator │
> │ └───▲──────────┬───────┘ ┌────────┐
> │ │ │ │ ┌────┐ │
> │ │ └────────────►─┤FM │ ├───────────────────────────┐
> │ │ │ └──┬─┘ │ │
> │ │ │BMC │ │ │
> │ │ └────┼───┘ │
> │ ┌─────┼─────────────┐ │ │
> │ │ │ │ ┌──────┼────────┐ ┌────────────────┐ │
> │ │ │ ┌───┤ │ │ │ │ │ │
> │ │ │ │RP0├───────┤ ┌───▼────┐ │ │ ┌────────────┐ │ │
> │ │ ┌───┴──┐ └───┤ │ │ Switch ├───┼───┼─►FM Owned LD │ │ │
> │ │ │ APP │ │ │ │ CCI │ │ │ └────────────┘ │ │
> │ │ │ │ ┌───┤ │ │ or MCTP│ ├───┤ │ │
> │ │ └──────┘ │RP1├─────┐ │ │ CCI │ │ │ MLD 1 │ │
> │ │ └───┤ │ │ └───────┬┘ │ └────────────────┘ │
> │ │ Host A │ │ │ │ │ │
> │ └───────────────────┘ │ │ │ │ ┌────────────────┐ │
> │ │ │ │ │ │ │ │
> │ ┌───────────────────┐ │ │ │ │ │ ┌────────────┐ │ │
> │ │ │ │ │ └────┼───┼─►FM Owned LD │ │ │
> │ │ ┌───┤ │ │ ├───┤ └────────────┘ │ │
> │ │ │RP0├─────┼─┤ │ │ │ │
> │ │ ┌──────┐ └───┤ │ │ SWITCH 1 │ │ MLD 2 │ │
> └─┼─┤ APP │ │ │ │ │ └────────────────┘ │
> │ │ │ ┌───┤ │ │ │ │
> │ └──────┘ │RP1├──┐ │ └───────────────┘ │
> │ └───┤ │ │ │
> │ Host B │ │ │ │
> └───────────────────┘ │ │ │
> │ │ ┌───────────────────────────────┘
> │ │ ┌──────┼────────┐ ┌────────────────┐
> │ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
> │ │ │ │ Switch ├───┼───┼─►FM Owned LD │ │
> │ └─┤ │ CCI │ │ │ └────────────┘ │
> │ │ │ or MCTP│ ├───┤ │
> │ │ │ CCI │ │ │ MLD 3 │
> │ │ └───────┬┘ │ └────────────────┘
> │ │ │ │ ┌────────────────┐
> │ │ │ │ │ ┌────────────┐ │
> │ │ └────┼───┼─►FM Owned LD │ │
> │ │ ├───┤ └────────────┘ │
> │ │ SWITCH 2 │ │ MLD 4 │
> └────┤ │ └────────────────┘
> └───────────────┘
>>
>> So, have I missed something?
>> Should we correct something on diagrams?
>> Does it look good?
>
> Thare are far too many things we should perhaps show on these diagrams, but
> I suspect it will mostly fall out of any layered design.
>
> We could draw one incredibly complex diagram that does everything :)
>
I believe several simple diagrams are better than one really complex one. :)
> *crossed fingers the ascii art fun above looks ok*
>
> Thanks for getting this started btw! I was completely failing to start
> whereas once there was a proposal it became easier to have a go!
>
My pleasure. :)
Thanks,
Slava.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [External] CXL Fabric Manager (FM) architecture diagrams
[not found] ` <20230310120126.000057b9@Huawei.com>
@ 2023-03-16 20:43 ` Viacheslav A.Dubeyko
[not found] ` <20230406204227.GA5971@bgt-140510-bm01>
0 siblings, 1 reply; 6+ messages in thread
From: Viacheslav A.Dubeyko @ 2023-03-16 20:43 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: Viacheslav Dubeyko, linux-cxl, a.manzanares
Hi Jonathan,
> On Mar 10, 2023, at 4:01 AM, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>
> On Thu, 9 Mar 2023 14:28:51 -0800
> "Viacheslav A.Dubeyko" <viacheslav.dubeyko@bytedance.com> wrote:
>
>>> On Mar 7, 2023, at 9:21 AM, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>>
>>> On Mon, 6 Mar 2023 10:59:13 -0800
>>> Viacheslav Dubeyko <slava@dubeyko.com> wrote:
>>>
>>>> Hi Jonathan,
>>>>
>>>> You can find diagram online now:
>>>>
>>>> (1) Diagram 1: single host with Fabric Manager (FM)
>>>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram1-cxl-fm-single-host-with-fm.pdf
>>>
Diagram 1 has been reworked.
>>
>>>>
>>>> (2) Diagram 2: single host with orchestrator
>>>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram2-cxl-fm-single-host-with-orhestrator.pdf
>>>
Diagram 2 has been reworked.
>>>>
>>>> (3) Diagram3: Multi-headed device
>>>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram3-cxl-fm-multi-headed-device.pdf
>>>
Diagram 3 has been reworked.
>>>> (4) Diagram 4: Multi-headed device + Orchestrator
>>>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram4-cxl-fm-multi-headed-device-orchestrator.pdf
Diagram 4 has been reworked.
>>>>
>>>> (5) Diagram5: Multiple hosts with Fabric Manager (FM)
>>>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram5-cxl-fm-multiple-hosts-with-fm.pdf
>>>
Diagram 5 has been reworked.
>>>>
>>>> (6) Diagram 6: Multiple hosts with orhestrator
>>>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram6-cxl-fm-multiple-hosts-with-orhestrator.pdf
>>>
Diagram 6 has been removed.
>>>>
>>>> (7) Diagram 7: Distributed Fabric Manager (FM)
>>>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram7-cxl-fm-distributed-fm.pdf
>>>>
Diagram 7 is Diagram 6 now. And it’s reworked.
>>>> (8) Diagram 8: Layered Fabric Manager (FM) and separate orchestrator
>>>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram8-cxl-fm-layered-fm-and-separate-orchestrator.pdf
>>>
Diagram 8 is Diagram 7 now. And it’s reworked.
>>>> (9) Diagram 9: BMC based layered Fabric Manager (FM)
>>>> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram9-cxl-fm-bmc-based-layered-fm.pdf
>>>
Diagram 9 is Diagram 8 now. And it’s reworked.
Please, check the online version of diagrams and
let me know if I need to rework any diagram.
Thanks,
Slava.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [External] CXL Fabric Manager (FM) architecture diagrams
[not found] ` <20230406204227.GA5971@bgt-140510-bm01>
@ 2023-04-07 18:18 ` Viacheslav A.Dubeyko
[not found] ` <90aea04c-dab2-4b7e-bfc6-09a1240793a9@nmtadam.samsung>
0 siblings, 1 reply; 6+ messages in thread
From: Viacheslav A.Dubeyko @ 2023-04-07 18:18 UTC (permalink / raw)
To: Adam Manzanares; +Cc: Jonathan Cameron, Viacheslav Dubeyko, linux-cxl
Hi Adam,
> On Apr 6, 2023, at 1:42 PM, Adam Manzanares <a.manzanares@samsung.com> wrote:
>
> On Thu, Mar 16, 2023 at 01:43:25PM -0700, Viacheslav A.Dubeyko wrote:
>> Hi Jonathan,
>>
>>> On Mar 10, 2023, at 4:01 AM, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>>
>>> On Thu, 9 Mar 2023 14:28:51 -0800
>>> "Viacheslav A.Dubeyko" <viacheslav.dubeyko@bytedance.com> wrote:
>>>
>>>>> On Mar 7, 2023, at 9:21 AM, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>>>>
>>>>> On Mon, 6 Mar 2023 10:59:13 -0800
>>>>> Viacheslav Dubeyko <slava@dubeyko.com> wrote:
>>>>>
>>>>>> Hi Jonathan,
>>>>>>
>>>>>> You can find diagram online now:
>
> Hello Slava,
>
> Thanks for putting these together. I think we need descriptive text to add
> context to the diagrams. I suggest adding a wiki.
I enabled the wiki. Let me know if you have troubles to access it.
>
>
>>>>>>
>>>>>> (1) Diagram 1: single host with Fabric Manager (FM)
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=6d42817d-0c3f6bfe-6d430a32-74fe48600158-e525dd835650a36b&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram1-cxl-fm-single-host-with-fm.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8TuzI6n8Hlk$
>>>>>
>>
>> Diagram 1 has been reworked.
>
> Once we have a wiki it would be great to document how we enable logical
> connections to the host. We also need to define all of the terms used in the
> diagram. I can post a v1 once the wiki is enabled.
>
Sounds great! Let's do it.
>>
>>>>
>>>>>>
>>>>>> (2) Diagram 2: single host with orchestrator
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=bfd57fd6-dea89555-bfd4f499-74fe48600158-f6b28a23672be168&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram2-cxl-fm-single-host-with-orhestrator.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8TuzeCJmksA$
>>>>>
>>
>> Diagram 2 has been reworked.
>
> I think the orchestrator does not always have to be over a network connection.
> In Diagram 1 the APP is acting as the orchestrator IMO because it directly
> interacts with the FM.
>
If we don’t need to go through network, then we don’t need orchestrator at all.
It means that local FM can manage all requests. The point of orchestrator is to manage
multiple FM instances through the network connection.
> Do you think it would be valuable to have a bit more detail about where the FM
> and orchestor reside. For example orchestrator could be another host and FM
> could be part of the switch. I'm trying to come up with more detail without
> being tied to a particular architecture. I suppose the wiki will help here.
>
I think that problem with going into more details is losing the generality of the diagram.
Also, more detailed diagram makes picture more complicated. Technically speaking, orchestrator
could reside on the same machine or on another dedicated machine. Showing the particular
placement of orchestrator and FM can initiate discussion related the feasibility to have FM
inside of switch, for example. And it’s not important from software point of view, currently,
as far as I can see.
>>
>>
>>>>>>
>>>>>> (3) Diagram3: Multi-headed device
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=626e6b92-03138111-626fe0dd-74fe48600158-4dacfa8033223c46&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram3-cxl-fm-multi-headed-device.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8TuzVSc7k_E$
>>>>>
>>
>> Diagram 3 has been reworked.
>>
>
> Can you remind me why we need both heads of the MHD to be connected to the
> same host?
>
I think it’s one to the potential case. Diagram 3 shows the case of connected both heads
to one host and diagram 4 shows the case of connection of heads to different hosts.
Technically speaking, we cannot exclude the case of connection of both heads to the same host.
>>>>>> (4) Diagram 4: Multi-headed device + Orchestrator
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=7775371c-1608dd9f-7774bc53-74fe48600158-eaab011af7366bc3&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram4-cxl-fm-multi-headed-device-orchestrator.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8TuzftBJkJc$
>>
>> Diagram 4 has been reworked.
>>
>>>>>>
>>>>>> (5) Diagram5: Multiple hosts with Fabric Manager (FM)
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=f03a7511-91479f92-f03bfe5e-74fe48600158-97054848fb228380&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram5-cxl-fm-multiple-hosts-with-fm.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8Tuz6UJQkqk$
>>>>>
>>
>> Diagram 5 has been reworked.
>
> I am thinking the multiple hosts and single host with the orchestrator off the
> hosts shown can be represented with a single diagram.
>
Do you mean diagram 2 and diagram 5? Possibly, yes. But I believe it makes sense to show
the both cases on independent diagrams.
>>
>>>>>>
>>>>>> (6) Diagram 6: Multiple hosts with orhestrator
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=55b499bc-34c9733f-55b512f3-74fe48600158-8fbd9efa3688d19c&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram6-cxl-fm-multiple-hosts-with-orhestrator.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8TuzaEytqFw$
>>>>>
>>
>>
>> Diagram 6 has been removed.
>>
>>>>>>
>>>>>> (7) Diagram 7: Distributed Fabric Manager (FM)
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=2f929e7e-4eef74fd-2f931531-74fe48600158-45f23959e990d9c3&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram7-cxl-fm-distributed-fm.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8Tuz_smZe_0$
>>>>>>
>>
>> Diagram 7 is Diagram 6 now. And it’s reworked.
>>
>
> The fun begins! Is there an implication that each switch will have an FM
> component? Once we have the wiki we should try to describe the most likely
> scenario that would produce this topology.
>
As far as I can see, not everybody is convinced that FM can be inside of switch at all.
We need to take such possibility into account. I assume that switch could contain simplified
FM implementation. But if you need more powerful FM, then it will be outside of the switch.
>>>>>> (8) Diagram 8: Layered Fabric Manager (FM) and separate orchestrator
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=f31a68c9-9267824a-f31be386-74fe48600158-2837aeae8831a63e&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram8-cxl-fm-layered-fm-and-separate-orchestrator.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8TuzGsyovOU$
>>>>>
>>
>> Diagram 8 is Diagram 7 now. And it’s reworked.
>
> What is the TOP FM and bottom FM responsible for?
>>
As far as I can see, it is the implementation of layered or hierarchical FM
infrastructure. The bottom FM represents leaf level of the hierarchy and
the top FM can be imagined like a index node of the hierarchy or tree.
So, the top FM can work like an index node of tree with the goal to find
the proper instance of bottom FM.
>>>>>> (9) Diagram 9: BMC based layered Fabric Manager (FM)
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=14514ee3-752ca460-1450c5ac-74fe48600158-0c34c12aaa752376&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram9-cxl-fm-bmc-based-layered-fm.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8Tuz1Yn7XZU$
>>>>>
>>
>> Diagram 9 is Diagram 8 now. And it’s reworked.
>
> Where is the layering here? It looks like the BMC has the entire FM.
>
Yeah, maybe the title needs to be reworked.
>>
>> Please, check the online version of diagrams and
>> let me know if I need to rework any diagram.
>
> Thanks for putting this together. Once we have a wiki in place I think it is
> a great spot to reference when we have discussions about open source code
> development.
>
I enabled the wiki. So, let’s start collaboration in wiki space.
Thanks,
Slava.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [External] CXL Fabric Manager (FM) architecture diagrams
[not found] ` <90aea04c-dab2-4b7e-bfc6-09a1240793a9@nmtadam.samsung>
@ 2023-04-07 23:24 ` Viacheslav A.Dubeyko
0 siblings, 0 replies; 6+ messages in thread
From: Viacheslav A.Dubeyko @ 2023-04-07 23:24 UTC (permalink / raw)
To: Adam Manzanares; +Cc: Jonathan Cameron, Viacheslav Dubeyko, linux-cxl
> On Apr 7, 2023, at 4:18 PM, Adam Manzanares <a.manzanares@samsung.com> wrote:
>
> On Fri, Apr 07, 2023 at 11:18:43AM -0700, Viacheslav A.Dubeyko wrote:
>> Hi Adam,
>>
>>> On Apr 6, 2023, at 1:42 PM, Adam Manzanares <a.manzanares@samsung.com> wrote:
>>>
>>> On Thu, Mar 16, 2023 at 01:43:25PM -0700, Viacheslav A.Dubeyko wrote:
>>>> Hi Jonathan,
>>>>
>>>>> On Mar 10, 2023, at 4:01 AM, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>>>>
>>>>> On Thu, 9 Mar 2023 14:28:51 -0800
>>>>> "Viacheslav A.Dubeyko" <viacheslav.dubeyko@bytedance.com> wrote:
>>>>>
>>>>>>> On Mar 7, 2023, at 9:21 AM, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>>>>>>
>>>>>>> On Mon, 6 Mar 2023 10:59:13 -0800
>>>>>>> Viacheslav Dubeyko <slava@dubeyko.com> wrote:
>>>>>>>
>>>>>>>> Hi Jonathan,
>>>>>>>>
>>>>>>>> You can find diagram online now:
>>>
>>> Hello Slava,
>>>
>>> Thanks for putting these together. I think we need descriptive text to add
>>> context to the diagrams. I suggest adding a wiki.
>>
>> I enabled the wiki. Let me know if you have troubles to access it.
>
> I can see it, but I am unable to edit pages ATM.
>
As far as I can see, you need to clone the wiki. And then you can follow
to regular git way. I mean you can edit locally, make commit and
send pull/merge requests.
Thanks,
Slava.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-04-07 23:25 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-03-06 18:59 CXL Fabric Manager (FM) architecture diagrams Viacheslav Dubeyko
2023-03-07 17:21 ` Jonathan Cameron
2023-03-09 22:28 ` [External] " Viacheslav A.Dubeyko
[not found] ` <20230310120126.000057b9@Huawei.com>
2023-03-16 20:43 ` Viacheslav A.Dubeyko
[not found] ` <20230406204227.GA5971@bgt-140510-bm01>
2023-04-07 18:18 ` Viacheslav A.Dubeyko
[not found] ` <90aea04c-dab2-4b7e-bfc6-09a1240793a9@nmtadam.samsung>
2023-04-07 23:24 ` Viacheslav A.Dubeyko
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.