From: Wen Gu <guwen@linux.alibaba.com>
To: Alexandra Winter <wintera@linux.ibm.com>,
wenjia@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com,
agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com,
kuba@kernel.org, pabeni@redhat.com, jaka@linux.ibm.com
Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com,
alibuda@linux.alibaba.com, tonylu@linux.alibaba.com,
linux-s390@vger.kernel.org, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH net-next 00/15] net/smc: implement loopback-ism used by SMC-D
Date: Mon, 19 Feb 2024 22:04:32 +0800 [thread overview]
Message-ID: <8a333c58-4a74-4a1c-9680-a0b9b4020a62@linux.alibaba.com> (raw)
In-Reply-To: <83b9d600-339c-4c9f-860d-ab4539a0ae6b@linux.ibm.com>
On 2024/2/6 20:18, Alexandra Winter wrote:
>
>
> On 11.01.24 13:00, Wen Gu wrote:
>> This patch set acts as the second part of the new version of [1] (The first
>> part can be referred from [2]), the updated things of this version are listed
>> at the end.
>>
>> # Background
>>
>> SMC-D is now used in IBM z with ISM function to optimize network interconnect
>> for intra-CPC communications. Inspired by this, we try to make SMC-D available
>> on the non-s390 architecture through a software-implemented virtual ISM device,
>> that is the loopback-ism device here, to accelerate inter-process or
>> inter-containers communication within the same OS instance.
>
>
> Hello Wen Gu,
>
> thank you very much for this patchset. I have been looking at it a bit.
> I installed in on a testserver, but did not yet excercise the loopback-ism device.
> I want to continue investigations, but daily work interferes, so I thought I
> send you some comments now. So this is not at all a code review, but some
> thoughts and observations about the general concept.
>
Thank you very much, Sandy.
>
> In [1] there was a discussion about an abstraction layer between smc-d and the
> ism devices.
> I am not sure what you are proposing now, is it an smc-d feature or independent of smc?
> In 3/15 you say it is part of the SMC module, but then it has its own device entry.
> Didn't you want to use it for other things as well? Or is it an SMC-D only feature?
> Is it a device (Config help: "kind of virtual device")? Or an SMC-D feature?
>
This patchset aims to propose an SMC feature, which is SMC-D loopback. The main work
to achieve this feature is to implement an Emulated-ISM, which is loopback-ism. The
loopback-ism is a 'built-in' dummy device of SMC and only serves SMC.
SMC-D protocol + 'built-in dummy device' (loopback-ism device) = SMC-D loopback feature.
To provide the runtime switch and statistics of loopback-ism, I need to find a sysfs
entry for it, since it doesn't belong to any class (e.g. pci_bus), I created an 'smc'
entry under /sys/devices/virtual/ and put loopback-ism under it.
The other SMC devices, such as RoCE, s390 ISM, virtio-ism will be in their own sysfs
entry, not under the /sys/devices/*virtual*/smc/.
The Config help is somewhat inaccurate. To be more precise, the SMC_LO config is used to
configure whether to enable this built-in dummy device for intra-OS communication.
> Will we have a class of ism devices (s390 ism, ism-loopback, virtio-ism)
> That share common properties (internal API?)
> and smc-d will work with any of those? > But they all can exist without smc ?! BTW: This is what we want for s390-ism.
> The client-registration interface [2] is currently the way to achieve this.
> But maybe we need a more general concept?
>
I didn't mean to create a class to cover all the ISM devices. It is only for
loopback-ism. Because loopback-ism can not be classified, so I create an entry
under /sys/devices/virtual/.
> Maybe first a preparation patchset that introduces a class/ism
> Together with an improved API?
> In case you want to use ISM devices for other purposes as well..
> But then the whole picture of ism-loopback in one patchset (RFC?)
> has its benefits as well.
>
Sorry for causing, I didn't mean to create a class to cover all the ISM devices.
They should be in their own sysfs entries (e.g. pci_bus), since they will be used
out of SMC. Only loopback-ism belongs only to SMC.
>
> Other points that I noticed:
>
> Naming: smc loopback, ism-loopback, loopback-ism ?
>
> config: why not tristate? Why under net/smc?
>
'SMC-D loopback' or 'SMC loopback' is used to indicate the feature or capability.
'loopback-ism' is the emulated-ISM device that 'SMC/SMC-D loopback' used.
('ism-loopback' doesn't seem to appear in my patchset)
If we all agree with these, I will check all the terms in the patch and unify them.
SMC_LO is used to configure whether SMC is allowed to use loopback-ism (CONFIG_SMC_LO),
it acts as a check in the code, so I defined it as a bool.
And loopback-ism only serves SMC-D loopback, as a feature of SMC, so the implementation
(net/smc/smc_loopback.{c|h}) is under net/smc.
> /sys/devices/virtual/smc does not initially show up in my installation!!!
> root@t35lp50:/sys/devices/virtual/> ls
> 3270/ bdi/ block/ graphics/ iommu/ mem/ memory_tiering/ misc/ net/ tty/ vc/ vtconsole/ workqueue/
> root@t35lp50:/sys/devices/virtual/> ls smc/loopback-ism
> active dmb_copy dmbs_cnt dmb_type subsystem@ uevent xfer_bytes
> root@t35lp50:/sys/devices/virtual/> ls
> 3270/ bdi/ block/ graphics/ iommu/ mem/ memory_tiering/ misc/ net/ smc/ tty/ vc/ vtconsole/ workqueue/
> Is that normal behaviour?
>
/sys/devices/virtual/smc is created after SMC module initialization.
During the SMC module initialization, smc_loopback_init() is called, and the
/sys/devices/virtual/smc entry is created.
> You introduced a class/smc
> Maybe class/ism would be better?
> The other smc interfaces do not show up in class/smc!! Not so good
>
Sorry for causing, I didn't mean to create a class to cover all the ISM devices.
They should be in their own sysfs entries (e.g. pci_bus), since they can be used
out of SMC. But loopback-ism is a SMC 'built-in' dummy device, it belongs only
to SMC and can't be classified to other entries.
> Why doesn't it show in smc_rnics?
> (Maybe some deficiency of smc_rnics?)
>
smc_rnics can't be used on the arch other than s390.
# ./smc_rnics -a
Error: s390/s390x supported only
> But then it shows in other smc-tools:
> root@t35lp50:/sys/> smcd device
> FID Type PCI-ID PCHID InUse #LGs PNET-ID
> 0000 0 loopback-ism ffff No 0
> 0029 ISM 0000:00:00.0 07c1 No 0 NET1
> Nice!
>
Yes, this is did on patch 01/15.
Best regards,
Wen Gu
> Kind regards
> Sandy
>
>
> [1] https://lore.kernel.org/lkml/e3819550-7b10-4f9c-7347-dcf1f97b8e6b@linux.alibaba.com/
> [2] 89e7d2ba61b7 ("net/ism: Add new API for client registration")
next prev parent reply other threads:[~2024-02-19 14:04 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-11 12:00 [PATCH net-next 00/15] net/smc: implement loopback-ism used by SMC-D Wen Gu
2024-01-11 12:00 ` [PATCH net-next 01/15] net/smc: improve SMC-D device dump for virtual ISM Wen Gu
2024-01-11 12:00 ` [PATCH net-next 02/15] net/smc: decouple specialized struct from SMC-D DMB registration Wen Gu
2024-01-11 12:00 ` [PATCH net-next 03/15] net/smc: introduce virtual ISM device loopback-ism Wen Gu
2024-02-16 14:11 ` Wenjia Zhang
2024-02-20 1:20 ` Wen Gu
2024-01-11 12:00 ` [PATCH net-next 04/15] net/smc: implement ID-related operations of loopback-ism Wen Gu
2024-01-11 12:00 ` [PATCH net-next 05/15] net/smc: implement some unsupported " Wen Gu
2024-01-11 12:00 ` [PATCH net-next 06/15] net/smc: implement DMB-related " Wen Gu
2024-02-16 14:13 ` Wenjia Zhang
2024-02-20 1:55 ` Wen Gu
2024-02-23 14:12 ` Wenjia Zhang
2024-02-26 3:04 ` Wen Gu
2024-01-11 12:00 ` [PATCH net-next 07/15] net/smc: register loopback-ism into SMC-D device list Wen Gu
2024-01-11 12:00 ` [PATCH net-next 08/15] net/smc: introduce loopback-ism runtime switch Wen Gu
2024-01-11 12:00 ` [PATCH net-next 09/15] net/smc: introduce loopback-ism statistics attributes Wen Gu
2024-02-16 14:24 ` Wenjia Zhang
2024-02-20 2:45 ` Wen Gu
2024-02-23 14:13 ` Wenjia Zhang
2024-02-26 12:58 ` Wen Gu
2024-01-11 12:00 ` [PATCH net-next 10/15] net/smc: add operations to merge sndbuf with peer DMB Wen Gu
2024-01-11 12:00 ` [PATCH net-next 11/15] net/smc: attach or detach ghost sndbuf to " Wen Gu
2024-01-11 12:00 ` [PATCH net-next 12/15] net/smc: adapt cursor update when sndbuf and peer DMB are merged Wen Gu
2024-01-11 12:00 ` [PATCH net-next 13/15] net/smc: introduce loopback-ism DMB type control Wen Gu
2024-02-16 14:25 ` Wenjia Zhang
2024-02-20 3:19 ` Wen Gu
2024-01-11 12:00 ` [PATCH net-next 14/15] net/smc: introduce loopback-ism DMB data copy control Wen Gu
2024-01-12 16:24 ` Niklas Schnelle
2024-01-13 7:12 ` Wen Gu
2024-02-16 14:25 ` Wenjia Zhang
2024-02-20 3:36 ` Wen Gu
2024-02-23 14:42 ` Wenjia Zhang
2024-01-11 12:00 ` [PATCH net-next 15/15] net/smc: implement DMB-merged operations of loopback-ism Wen Gu
2024-01-11 13:36 ` [PATCH net-next 00/15] net/smc: implement loopback-ism used by SMC-D Simon Horman
2024-01-12 2:54 ` Wen Gu
2024-01-11 14:50 ` Jiri Pirko
2024-01-12 8:29 ` Wen Gu
2024-01-12 9:10 ` Jiri Pirko
2024-01-12 12:32 ` Wen Gu
2024-01-12 15:50 ` Jiri Pirko
2024-01-13 9:22 ` Wen Gu
2024-01-15 14:11 ` Jiri Pirko
2024-01-18 8:27 ` Wen Gu
2024-01-18 13:59 ` Wenjia Zhang
2024-01-19 1:46 ` Wen Gu
2024-01-23 14:03 ` Alexandra Winter
2024-01-24 6:33 ` Wen Gu
2024-02-05 10:05 ` Wen Gu
2024-02-07 9:08 ` Wenjia Zhang
2024-02-06 12:18 ` Alexandra Winter
2024-02-08 16:12 ` Wen Gu
2024-02-19 14:04 ` Wen Gu [this message]
2024-02-16 14:09 ` Wenjia Zhang
2024-02-20 3:52 ` Wen Gu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8a333c58-4a74-4a1c-9680-a0b9b4020a62@linux.alibaba.com \
--to=guwen@linux.alibaba.com \
--cc=agordeev@linux.ibm.com \
--cc=alibuda@linux.alibaba.com \
--cc=borntraeger@linux.ibm.com \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=gor@linux.ibm.com \
--cc=hca@linux.ibm.com \
--cc=jaka@linux.ibm.com \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=svens@linux.ibm.com \
--cc=tonylu@linux.alibaba.com \
--cc=wenjia@linux.ibm.com \
--cc=wintera@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).