Re: [External] CXL Fabric Manager (FM) architecture diagrams

["Viacheslav A.Dubeyko" <viacheslav.dubeyko@bytedance.com>]

Hi Adam,

> On Apr 6, 2023, at 1:42 PM, Adam Manzanares <a.manzanares@samsung.com> wrote:
> 
> On Thu, Mar 16, 2023 at 01:43:25PM -0700, Viacheslav A.Dubeyko wrote:
>> Hi Jonathan,
>> 
>>> On Mar 10, 2023, at 4:01 AM, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>> 
>>> On Thu, 9 Mar 2023 14:28:51 -0800
>>> "Viacheslav A.Dubeyko" <viacheslav.dubeyko@bytedance.com> wrote:
>>> 
>>>>> On Mar 7, 2023, at 9:21 AM, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>>>> 
>>>>> On Mon,  6 Mar 2023 10:59:13 -0800
>>>>> Viacheslav Dubeyko <slava@dubeyko.com> wrote:
>>>>> 
>>>>>> Hi Jonathan,
>>>>>> 
>>>>>> You can find diagram online now:
> 
> Hello Slava,
> 
> Thanks for putting these together. I think we need descriptive text to add 
> context to the diagrams. I suggest adding a wiki.

I enabled the wiki. Let me know if you have troubles to access it.

>  
> 
>>>>>> 
>>>>>> (1) Diagram 1: single host with Fabric Manager (FM)
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=6d42817d-0c3f6bfe-6d430a32-74fe48600158-e525dd835650a36b&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram1-cxl-fm-single-host-with-fm.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8TuzI6n8Hlk$   
>>>>> 
>> 
>> Diagram 1 has been reworked.
> 
> Once we have a wiki it would be great to document how we enable logical
> connections to the host. We also need to define all of the terms used in the 
> diagram. I can post a v1 once the wiki is enabled.
> 

Sounds great! Let's do it.

>> 
>>>> 
>>>>>> 
>>>>>> (2) Diagram 2: single host with orchestrator
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=bfd57fd6-dea89555-bfd4f499-74fe48600158-f6b28a23672be168&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram2-cxl-fm-single-host-with-orhestrator.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8TuzeCJmksA$   
>>>>> 
>> 
>> Diagram 2 has been reworked.
> 
> I think the orchestrator does not always have to be over a network connection.
> In Diagram 1 the APP is acting as the orchestrator IMO because it directly
> interacts with the FM. 
> 

If we don’t need to go through network, then we don’t need orchestrator at all.
It means that local FM can manage all requests. The point of orchestrator is to manage
multiple FM instances through the network connection.

> Do you think it would be valuable to have a bit more detail about where the FM
> and orchestor reside. For example orchestrator could be another host and FM
> could be part of the switch. I'm trying to come up with more detail without
> being tied to a particular architecture. I suppose the wiki will help here.
> 

I think that problem with going into more details is losing the generality of the diagram.
Also, more detailed diagram makes picture more complicated. Technically speaking, orchestrator
could reside on the same machine or on another dedicated machine. Showing the particular
placement of orchestrator and FM can initiate discussion related the feasibility to have FM
inside of switch, for example. And it’s not important from software point of view, currently,
as far as I can see.

>> 
>> 
>>>>>> 
>>>>>> (3) Diagram3: Multi-headed device
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=626e6b92-03138111-626fe0dd-74fe48600158-4dacfa8033223c46&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram3-cxl-fm-multi-headed-device.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8TuzVSc7k_E$   
>>>>> 
>> 
>> Diagram 3 has been reworked.
>> 
> 
> Can you remind me why we need both heads of the MHD to be connected to the 
> same host?
> 

I think it’s one to the potential case. Diagram 3 shows the case of connected both heads
to one host and diagram 4 shows the case of connection of heads to different hosts.
Technically speaking, we cannot exclude the case of connection of both heads to the same host.

>>>>>> (4) Diagram 4: Multi-headed device + Orchestrator
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=7775371c-1608dd9f-7774bc53-74fe48600158-eaab011af7366bc3&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram4-cxl-fm-multi-headed-device-orchestrator.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8TuzftBJkJc$   
>> 
>> Diagram 4 has been reworked.
>> 
>>>>>> 
>>>>>> (5) Diagram5: Multiple hosts with Fabric Manager (FM)
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=f03a7511-91479f92-f03bfe5e-74fe48600158-97054848fb228380&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram5-cxl-fm-multiple-hosts-with-fm.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8Tuz6UJQkqk$   
>>>>> 
>> 
>> Diagram 5 has been reworked.
> 
> I am thinking the multiple hosts and single host with the orchestrator off the
> hosts shown can be represented with a single diagram.
> 

Do you mean diagram 2 and diagram 5? Possibly, yes. But I believe it makes sense to show
the both cases on independent diagrams.

>> 
>>>>>> 
>>>>>> (6) Diagram 6: Multiple hosts with orhestrator
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=55b499bc-34c9733f-55b512f3-74fe48600158-8fbd9efa3688d19c&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram6-cxl-fm-multiple-hosts-with-orhestrator.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8TuzaEytqFw$  
>>>>> 
>> 
>> 
>> Diagram 6 has been removed.
>> 
>>>>>> 
>>>>>> (7) Diagram 7: Distributed Fabric Manager (FM)
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=2f929e7e-4eef74fd-2f931531-74fe48600158-45f23959e990d9c3&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram7-cxl-fm-distributed-fm.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8Tuz_smZe_0$ 
>>>>>> 
>> 
>> Diagram 7 is Diagram 6 now. And it’s reworked.
>> 
> 
> The fun begins! Is there an implication that each switch will have an FM
> component? Once we have the wiki we should try to describe the most likely
> scenario that would produce this topology.
> 

As far as I can see, not everybody is convinced that FM can be inside of switch at all.
We need to take such possibility into account. I assume that switch could contain simplified
FM implementation. But if you need more powerful FM, then it will be outside of the switch.

>>>>>> (8) Diagram 8: Layered Fabric Manager (FM) and separate orchestrator
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=f31a68c9-9267824a-f31be386-74fe48600158-2837aeae8831a63e&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram8-cxl-fm-layered-fm-and-separate-orchestrator.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8TuzGsyovOU$   
>>>>> 
>> 
>> Diagram 8 is Diagram 7 now. And it’s reworked.
> 
> What is the TOP FM and bottom FM responsible for? 
>> 

As far as I can see, it is the implementation of layered or hierarchical FM
infrastructure. The bottom FM represents leaf level of the hierarchy and
the top FM can be imagined like a index node of the hierarchy or tree.
So, the top FM can work like an index node of tree with the goal to find
the proper instance of bottom FM.

>>>>>> (9) Diagram 9: BMC based layered Fabric Manager (FM)
>>>>>> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=14514ee3-752ca460-1450c5ac-74fe48600158-0c34c12aaa752376&q=1&e=93cc7f88-264f-4371-963b-b0f922c6255f&u=https*3A*2F*2Fgithub.com*2Fcomputexpresslink*2Fcxl-fm-architecture*2Fblob*2Fmain*2Fdiagram9-cxl-fm-bmc-based-layered-fm.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!WOepCxwyOun3qkuCOUyGyroMSJFLjcvROfsCKFs1MLgQagV8Ijg_oyEnnH4md9RO3YgIouhzWUr6Xfi1RYJdR7wk8Tuz1Yn7XZU$   
>>>>> 
>> 
>> Diagram 9 is Diagram 8 now. And it’s reworked.
> 
> Where is the layering here? It looks like the BMC has the entire FM.
> 

Yeah, maybe the title needs to be reworked.

>> 
>> Please, check the online version of diagrams and
>> let me know if I need to rework any diagram.
> 
> Thanks for putting this together. Once we have a wiki in place I think it is
> a great spot to reference when we have discussions about open source code
> development.
> 

I enabled the wiki. So, let’s start collaboration in wiki space.

Thanks,
Slava.