[LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Bart Van Assche]

The LIO and SCST SCSI target subsystems consist of the following components:
* A core that processes SCSI commands and that provides common
functionality like persistent reservations, LUN masking and an interface
that allows configuration from user space.
* Device handlers that allow this core to access SCSI devices, block
devices and files uniformly as SCSI devices.
* Target drivers that implement a storage protocol (iSCSI, FC, SRP,
iSER, FCoE, ...) and that realize the SCSI request and response
communication between the target system and an initiator system.

A significant amount of code is shared between several LIO target
drivers and the SCST target drivers that implement the same storage
protocol. Since there are two sets of these drivers this means that each
set has to be maintained, extended and tested separately. This means a
lot of redundant work. The main difference between these two sets of
drivers is the interface between the target drivers and the SCSI target
core.  Hence the proposal to discuss the unification of the API between
SCSI target core and SCSI target drivers. Implementing a single unified
API would have the following advantages:
* A single set of target drivers works for both projects which means a
reduction of the maintenance effort for those who maintain target
drivers for target driver developers and target driver users.
* This would increase the size of the user base for the unified target
drivers.
* This would reduce the workload for the storage target maintainers.
* This would motivate the SCST target driver maintainers to contribute
to the upstream target drivers and to bring the upstream SRP and FCoE
target drivers to the same feature and stability level as their SCST
counterparts. In other words, the LIO users would also benefit from this
work.
* This effort would also help SCST users by ensuring that all latest
target driver features are also available to SCST users. Some time ago
(but no longer today) the LIO QLogic target driver was ahead of the SCST
QLogic target driver. This motivated an SCST user to port the LIO QLogic
target driver to SCST. See also Greg Wettstein, New release of
SCST/Qlogic target interface driver, linux-scsi, April 2014,
http://marc.info/?l=linux-scsi&m=139649571807085).

During the first phase of this initiative the focus will be on the
QLogic FC, SRP and FCoE target drivers since a significant part of the
code of these drivers is shared between the two target frameworks.

For those who are not following the SCST project: I'm maintaining the
SCST SRP and FCoE target drivers.

Nic, in case it was not yet clear, you would be more than welcome during
this session :-)

Bart.

Re: [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Hannes Reinecke]

On 01/14/2015 11:05 AM, Bart Van Assche wrote:
> The LIO and SCST SCSI target subsystems consist of the following components:
> * A core that processes SCSI commands and that provides common
> functionality like persistent reservations, LUN masking and an interface
> that allows configuration from user space.
> * Device handlers that allow this core to access SCSI devices, block
> devices and files uniformly as SCSI devices.
> * Target drivers that implement a storage protocol (iSCSI, FC, SRP,
> iSER, FCoE, ...) and that realize the SCSI request and response
> communication between the target system and an initiator system.
> 
> A significant amount of code is shared between several LIO target
> drivers and the SCST target drivers that implement the same storage
> protocol. Since there are two sets of these drivers this means that each
> set has to be maintained, extended and tested separately. This means a
> lot of redundant work. The main difference between these two sets of
> drivers is the interface between the target drivers and the SCSI target
> core.  Hence the proposal to discuss the unification of the API between
> SCSI target core and SCSI target drivers. Implementing a single unified
> API would have the following advantages:
> * A single set of target drivers works for both projects which means a
> reduction of the maintenance effort for those who maintain target
> drivers for target driver developers and target driver users.
> * This would increase the size of the user base for the unified target
> drivers.
> * This would reduce the workload for the storage target maintainers.
> * This would motivate the SCST target driver maintainers to contribute
> to the upstream target drivers and to bring the upstream SRP and FCoE
> target drivers to the same feature and stability level as their SCST
> counterparts. In other words, the LIO users would also benefit from this
> work.
> * This effort would also help SCST users by ensuring that all latest
> target driver features are also available to SCST users. Some time ago
> (but no longer today) the LIO QLogic target driver was ahead of the SCST
> QLogic target driver. This motivated an SCST user to port the LIO QLogic
> target driver to SCST. See also Greg Wettstein, New release of
> SCST/Qlogic target interface driver, linux-scsi, April 2014,
> http://marc.info/?l=linux-scsi&m=139649571807085).
> 
> During the first phase of this initiative the focus will be on the
> QLogic FC, SRP and FCoE target drivers since a significant part of the
> code of these drivers is shared between the two target frameworks.
> 
> For those who are not following the SCST project: I'm maintaining the
> SCST SRP and FCoE target drivers.
> 
> Nic, in case it was not yet clear, you would be more than welcome during
> this session :-)
> 
I'd like to have this discussion, too.

It would be really good if we can make that work; after all, Linux
should be about _choice_. So if both parties agree to have this
discussion I'm all for it.
I can even act as a moderator if required :-)

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		               zSeries & Storage
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Sagi Grimberg]

On 1/14/2015 12:05 PM, Bart Van Assche wrote:
> The LIO and SCST SCSI target subsystems consist of the following components:
> * A core that processes SCSI commands and that provides common
> functionality like persistent reservations, LUN masking and an interface
> that allows configuration from user space.
> * Device handlers that allow this core to access SCSI devices, block
> devices and files uniformly as SCSI devices.
> * Target drivers that implement a storage protocol (iSCSI, FC, SRP,
> iSER, FCoE, ...) and that realize the SCSI request and response
> communication between the target system and an initiator system.
>
> A significant amount of code is shared between several LIO target
> drivers and the SCST target drivers that implement the same storage
> protocol. Since there are two sets of these drivers this means that each
> set has to be maintained, extended and tested separately. This means a
> lot of redundant work. The main difference between these two sets of
> drivers is the interface between the target drivers and the SCSI target
> core.  Hence the proposal to discuss the unification of the API between
> SCSI target core and SCSI target drivers. Implementing a single unified
> API would have the following advantages:
> * A single set of target drivers works for both projects which means a
> reduction of the maintenance effort for those who maintain target
> drivers for target driver developers and target driver users.
> * This would increase the size of the user base for the unified target
> drivers.
> * This would reduce the workload for the storage target maintainers.
> * This would motivate the SCST target driver maintainers to contribute
> to the upstream target drivers and to bring the upstream SRP and FCoE
> target drivers to the same feature and stability level as their SCST
> counterparts. In other words, the LIO users would also benefit from this
> work.
> * This effort would also help SCST users by ensuring that all latest
> target driver features are also available to SCST users. Some time ago
> (but no longer today) the LIO QLogic target driver was ahead of the SCST
> QLogic target driver. This motivated an SCST user to port the LIO QLogic
> target driver to SCST. See also Greg Wettstein, New release of
> SCST/Qlogic target interface driver, linux-scsi, April 2014,
> http://marc.info/?l=linux-scsi&m=139649571807085).
>

I would definitely like to talk about this. The upstream SRP target
driver definitely needs more active maintenance.

I still feel that unifying the fabric drivers would not reduce testing
efforts (at least not dramatically) as these drivers will need to work
with two different target core stacks. This even might introduce
additional challenges.

> During the first phase of this initiative the focus will be on the
> QLogic FC, SRP and FCoE target drivers since a significant part of the
> code of these drivers is shared between the two target frameworks.
>

I would like to discuss about unifying the iSCSI and iSER drivers
as well. As you said, This might be more challenging since the two
stacks evolved from different code base and have different architecture,
so it would be nice to discuss if we would really benefit from a unified
implementation in the long run.

Sagi.

Re: [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Quinn Tran]

[-- Attachment #1: Type: text/plain, Size: 3105 bytes --]



On 1/14/15, 2:05 AM, "Bart Van Assche" <bart.vanassche@sandisk.com> wrote:

>The LIO and SCST SCSI target subsystems consist of the following
>components:
>* A core that processes SCSI commands and that provides common
>functionality like persistent reservations, LUN masking and an interface
>that allows configuration from user space.
>* Device handlers that allow this core to access SCSI devices, block
>devices and files uniformly as SCSI devices.
>* Target drivers that implement a storage protocol (iSCSI, FC, SRP,
>iSER, FCoE, ...) and that realize the SCSI request and response
>communication between the target system and an initiator system.
>
>A significant amount of code is shared between several LIO target
>drivers and the SCST target drivers that implement the same storage
>protocol. Since there are two sets of these drivers this means that each
>set has to be maintained, extended and tested separately. This means a
>lot of redundant work. The main difference between these two sets of
>drivers is the interface between the target drivers and the SCSI target
>core.  Hence the proposal to discuss the unification of the API between
>SCSI target core and SCSI target drivers. Implementing a single unified
>API would have the following advantages:
>* A single set of target drivers works for both projects which means a
>reduction of the maintenance effort for those who maintain target
>drivers for target driver developers and target driver users.
>* This would increase the size of the user base for the unified target
>drivers.
>* This would reduce the workload for the storage target maintainers.
>* This would motivate the SCST target driver maintainers to contribute
>to the upstream target drivers and to bring the upstream SRP and FCoE
>target drivers to the same feature and stability level as their SCST
>counterparts. In other words, the LIO users would also benefit from this
>work.
>* This effort would also help SCST users by ensuring that all latest
>target driver features are also available to SCST users. Some time ago
>(but no longer today) the LIO QLogic target driver was ahead of the SCST
>QLogic target driver. This motivated an SCST user to port the LIO QLogic
>target driver to SCST. See also Greg Wettstein, New release of
>SCST/Qlogic target interface driver, linux-scsi, April 2014,
>http://marc.info/?l=linux-scsi&m=139649571807085).
>
>During the first phase of this initiative the focus will be on the
>QLogic FC, SRP and FCoE target drivers since a significant part of the
>code of these drivers is shared between the two target frameworks.
>
>For those who are not following the SCST project: I'm maintaining the
>SCST SRP and FCoE target drivers.
>
>Nic, in case it was not yet clear, you would be more than welcome during
>this session :-)
>
>Bart.

QT> +1.  This would be a plus for Qlogic to have 2 stacks under a unify
API.  Test resource & devlopment cycles are limited.  A lot of cycles are
loss in keeping patches in sync.

Would like to listen in at LSF to hear the discussion.


[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 5168 bytes --]

Re: [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Nicholas A. Bellinger]

On Wed, 2015-01-14 at 11:05 +0100, Bart Van Assche wrote:
> The LIO and SCST SCSI target subsystems consist of the following components:
> * A core that processes SCSI commands and that provides common
> functionality like persistent reservations, LUN masking and an interface
> that allows configuration from user space.
> * Device handlers that allow this core to access SCSI devices, block
> devices and files uniformly as SCSI devices.
> * Target drivers that implement a storage protocol (iSCSI, FC, SRP,
> iSER, FCoE, ...) and that realize the SCSI request and response
> communication between the target system and an initiator system.
> 
> A significant amount of code is shared between several LIO target
> drivers and the SCST target drivers that implement the same storage
> protocol. Since there are two sets of these drivers this means that each
> set has to be maintained, extended and tested separately. This means a
> lot of redundant work. The main difference between these two sets of
> drivers is the interface between the target drivers and the SCSI target
> core.  Hence the proposal to discuss the unification of the API between
> SCSI target core and SCSI target drivers. Implementing a single unified
> API would have the following advantages:
> * A single set of target drivers works for both projects which means a
> reduction of the maintenance effort for those who maintain target
> drivers for target driver developers and target driver users.
> * This would increase the size of the user base for the unified target
> drivers.
> * This would reduce the workload for the storage target maintainers.
> * This would motivate the SCST target driver maintainers to contribute
> to the upstream target drivers and to bring the upstream SRP and FCoE
> target drivers to the same feature and stability level as their SCST
> counterparts. In other words, the LIO users would also benefit from this
> work.
> * This effort would also help SCST users by ensuring that all latest
> target driver features are also available to SCST users. Some time ago
> (but no longer today) the LIO QLogic target driver was ahead of the SCST
> QLogic target driver. This motivated an SCST user to port the LIO QLogic
> target driver to SCST. See also Greg Wettstein, New release of
> SCST/Qlogic target interface driver, linux-scsi, April 2014,
> http://marc.info/?l=linux-scsi&m=139649571807085).
> 
> During the first phase of this initiative the focus will be on the
> QLogic FC, SRP and FCoE target drivers since a significant part of the
> code of these drivers is shared between the two target frameworks.
> 

As always, I'm open to discussion, but if the main selling point for
unification is a bag of out-of-tree drivers that LIO already has vendor
and enterprise distro support for in upstream code, the net result will
not be to bring in more developers, vendors, and users to contribute to
a single target-core subsystem.

Contributing to a single target-core is after all, my main interest as
target subsystem maintainer.

If anything, this proposal gives certain vendors continued incentive to
avoid supporting and improving upstream target-core.

--nab

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Christoph Hellwig]

I don't quite understand whay you're proposing.  We're obbiously not
adding hooks for out of tree code to the kernel.  If you can improve
the target core <-> driver interface (and there's plenty of opportunity
for improvement) and it also happens to help your cause it's all fine

But all that really is a matter of sending patches, and not some out
of the blue discussion.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Bart Van Assche]

On 01/15/15 10:08, Christoph Hellwig wrote:
> We're obviously not adding hooks for out of tree code to the kernel.

Hello Christoph,

My goal is to realize this proposal without adding hooks for out-of-tree
code in the upstream kernel. What I had in mind is to raise the
abstraction level of the API between LIO core and target drivers a
little bit (e.g. by using accessor functions where necessary instead of
accessing structure members directly) and to implement the translation
between this API and the SCST API in a translation header file.
#including that last header file should be sufficient to build a target
driver against the SCST core. Other changes might also be necessary,
e.g. general code cleanup, simplification or making the API between core
and target drivers more consistent.

Bart.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Christoph Hellwig]

On Thu, Jan 15, 2015 at 05:13:00PM +0100, Bart Van Assche wrote:
> My goal is to realize this proposal without adding hooks for out-of-tree
> code in the upstream kernel. What I had in mind is to raise the
> abstraction level of the API between LIO core and target drivers a
> little bit (e.g. by using accessor functions where necessary instead of
> accessing structure members directly)

That's very much a hook, althiugh a week one.

Either way I don't think bringing up a very much political topic
without even any code to discuss isn't a very valueable use of our time
slots.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Bart Van Assche]

On 01/19/15 10:22, Christoph Hellwig wrote:
> On Thu, Jan 15, 2015 at 05:13:00PM +0100, Bart Van Assche wrote:
>> My goal is to realize this proposal without adding hooks for out-of-tree
>> code in the upstream kernel. What I had in mind is to raise the
>> abstraction level of the API between LIO core and target drivers a
>> little bit (e.g. by using accessor functions where necessary instead of
>> accessing structure members directly)
> 
> That's very much a hook, althiugh a week one.
> 
> Either way I don't think bringing up a very much political topic
> without even any code to discuss isn't a very valueable use of our time
> slots.

A possible approach is that I start implementing a unified SRP target
driver and post that driver together with the necessary LIO and SCST
core changes before the LSF/MM starts. That could be a helpful starting
point for further discussions.

Bart.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Bart Van Assche]

On 01/19/15 10:36, Bart Van Assche wrote:
> On 01/19/15 10:22, Christoph Hellwig wrote:
>> On Thu, Jan 15, 2015 at 05:13:00PM +0100, Bart Van Assche wrote:
>>> My goal is to realize this proposal without adding hooks for out-of-tree
>>> code in the upstream kernel. What I had in mind is to raise the
>>> abstraction level of the API between LIO core and target drivers a
>>> little bit (e.g. by using accessor functions where necessary instead of
>>> accessing structure members directly)
>>
>> That's very much a hook, althiugh a week one.
>>
>> Either way I don't think bringing up a very much political topic
>> without even any code to discuss isn't a very valueable use of our time
>> slots.
> 
> A possible approach is that I start implementing a unified SRP target
> driver and post that driver together with the necessary LIO and SCST
> core changes before the LSF/MM starts. That could be a helpful starting
> point for further discussions.

(replying to my own e-mail)

Hello Christoph,

What I proposed myself consists of three steps:
1. Updating the LIO SRP target driver to a more recent version.
2. Target driver and LIO core refactoring such that the LIO and
   SCST APIs can be unified.
3. Adding support in SCST for the unified target driver API.

This work is taking a little more time than I had expected - I still
have to start with step (3). But I can already show in which direction I
would like to go for steps (1) and (2). The 43 patches I came up with so
far for steps (1) and (2) are available in the lio branch of the
https://github.com/bvanassche/linux git repository. As you can see in
that repository 42 of these 43 patches make sense even without knowing
that something like SCST exists.

Bart.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Nicholas A. Bellinger]

On Fri, 2015-02-20 at 11:49 +0100, Bart Van Assche wrote:
> On 01/19/15 10:36, Bart Van Assche wrote:
> > On 01/19/15 10:22, Christoph Hellwig wrote:
> >> On Thu, Jan 15, 2015 at 05:13:00PM +0100, Bart Van Assche wrote:
> >>> My goal is to realize this proposal without adding hooks for out-of-tree
> >>> code in the upstream kernel. What I had in mind is to raise the
> >>> abstraction level of the API between LIO core and target drivers a
> >>> little bit (e.g. by using accessor functions where necessary instead of
> >>> accessing structure members directly)
> >>
> >> That's very much a hook, althiugh a week one.
> >>
> >> Either way I don't think bringing up a very much political topic
> >> without even any code to discuss isn't a very valueable use of our time
> >> slots.
> > 
> > A possible approach is that I start implementing a unified SRP target
> > driver and post that driver together with the necessary LIO and SCST
> > core changes before the LSF/MM starts. That could be a helpful starting
> > point for further discussions.
> 
> (replying to my own e-mail)
> 
> Hello Christoph,
> 
> What I proposed myself consists of three steps:
> 1. Updating the LIO SRP target driver to a more recent version.
> 2. Target driver and LIO core refactoring such that the LIO and
>    SCST APIs can be unified.
> 3. Adding support in SCST for the unified target driver API.
> 
> This work is taking a little more time than I had expected - I still
> have to start with step (3). But I can already show in which direction I
> would like to go for steps (1) and (2). The 43 patches I came up with so
> far for steps (1) and (2) are available in the lio branch of the
> https://github.com/bvanassche/linux git repository. As you can see in
> that repository 42 of these 43 patches make sense even without knowing
> that something like SCST exists.
> 

Please go ahead and post the series for review with bug fixes at the
head of the series, with future improvements following after the fixes.

Doing a brief review from your tree, I can see there are some useful
fixes, but also there are some patches that have incorrect assumptions
and/or break the existing userspace APIs.

That said, I'm happy to review the full series, and very much appreciate
your efforts to improve upstream code.

However, keep in mind that I'll not be merging anything for target-core
that adds support for out-of-tree code, nor anything that changes
existing target configfs APIs to fit out-of-tree code requirements.

Thank you,

--nab

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Sagi Grimberg]

On 2/20/2015 12:49 PM, Bart Van Assche wrote:
> On 01/19/15 10:36, Bart Van Assche wrote:
>> On 01/19/15 10:22, Christoph Hellwig wrote:
>>> On Thu, Jan 15, 2015 at 05:13:00PM +0100, Bart Van Assche wrote:
>>>> My goal is to realize this proposal without adding hooks for out-of-tree
>>>> code in the upstream kernel. What I had in mind is to raise the
>>>> abstraction level of the API between LIO core and target drivers a
>>>> little bit (e.g. by using accessor functions where necessary instead of
>>>> accessing structure members directly)
>>>
>>> That's very much a hook, althiugh a week one.
>>>
>>> Either way I don't think bringing up a very much political topic
>>> without even any code to discuss isn't a very valueable use of our time
>>> slots.
>>
>> A possible approach is that I start implementing a unified SRP target
>> driver and post that driver together with the necessary LIO and SCST
>> core changes before the LSF/MM starts. That could be a helpful starting
>> point for further discussions.
>
> (replying to my own e-mail)
>
> Hello Christoph,
>
> What I proposed myself consists of three steps:
> 1. Updating the LIO SRP target driver to a more recent version.
> 2. Target driver and LIO core refactoring such that the LIO and
>     SCST APIs can be unified.
> 3. Adding support in SCST for the unified target driver API.
>
> This work is taking a little more time than I had expected - I still
> have to start with step (3). But I can already show in which direction I
> would like to go for steps (1) and (2). The 43 patches I came up with so
> far for steps (1) and (2) are available in the lio branch of the
> https://github.com/bvanassche/linux git repository. As you can see in
> that repository 42 of these 43 patches make sense even without knowing
> that something like SCST exists.

Hey Bart,

So other than the obvious srpt update (which is well needed..), the
other patches in the set look very useful. I'll go ahead and pull them
for tests.

Sagi.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Christoph Hellwig]

On Fri, Feb 20, 2015 at 11:49:19AM +0100, Bart Van Assche wrote:
> Hello Christoph,
> 
> What I proposed myself consists of three steps:
> 1. Updating the LIO SRP target driver to a more recent version.
> 2. Target driver and LIO core refactoring such that the LIO and
>    SCST APIs can be unified.
> 3. Adding support in SCST for the unified target driver API.
> 
> This work is taking a little more time than I had expected - I still
> have to start with step (3). But I can already show in which direction I
> would like to go for steps (1) and (2). The 43 patches I came up with so
> far for steps (1) and (2) are available in the lio branch of the
> https://github.com/bvanassche/linux git repository. As you can see in
> that repository 42 of these 43 patches make sense even without knowing
> that something like SCST exists.

The work in there looks really good and easily mergeable with a few
mostly cosmetic comments.

I still don't see the point of adding this to LSF, we have far more
important and wide ranging topics then cleaning up some target
internals.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Bart Van Assche]

On 02/21/15 01:00, Nicholas A. Bellinger wrote:
> On Fri, 2015-02-20 at 11:49 +0100, Bart Van Assche wrote:
>> What I proposed myself consists of three steps:
>> 1. Updating the LIO SRP target driver to a more recent version.
>> 2. Target driver and LIO core refactoring such that the LIO and
>>    SCST APIs can be unified.
>> 3. Adding support in SCST for the unified target driver API.
>>
>> This work is taking a little more time than I had expected - I still
>> have to start with step (3). But I can already show in which direction I
>> would like to go for steps (1) and (2). The 43 patches I came up with so
>> far for steps (1) and (2) are available in the lio branch of the
>> https://github.com/bvanassche/linux git repository. As you can see in
>> that repository 42 of these 43 patches make sense even without knowing
>> that something like SCST exists.
> 
> Please go ahead and post the series for review with bug fixes at the
> head of the series, with future improvements following after the fixes.
> 
> Doing a brief review from your tree, I can see there are some useful
> fixes, but also there are some patches that have incorrect assumptions
> and/or break the existing userspace APIs.
> 
> That said, I'm happy to review the full series, and very much appreciate
> your efforts to improve upstream code.
> 
> However, keep in mind that I'll not be merging anything for target-core
> that adds support for out-of-tree code, nor anything that changes
> existing target configfs APIs to fit out-of-tree code requirements.

Let's start with what we agree on and discuss the other topics during
the LSF/MM summit. I just sent one patch of that series to the
linux-scsi mailing list for review.

Bart.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Nicholas A. Bellinger]

On Wed, 2015-02-25 at 09:43 +0100, Bart Van Assche wrote:
> On 02/21/15 01:00, Nicholas A. Bellinger wrote:
> > On Fri, 2015-02-20 at 11:49 +0100, Bart Van Assche wrote:
> >> What I proposed myself consists of three steps:
> >> 1. Updating the LIO SRP target driver to a more recent version.
> >> 2. Target driver and LIO core refactoring such that the LIO and
> >>    SCST APIs can be unified.
> >> 3. Adding support in SCST for the unified target driver API.
> >>
> >> This work is taking a little more time than I had expected - I still
> >> have to start with step (3). But I can already show in which direction I
> >> would like to go for steps (1) and (2). The 43 patches I came up with so
> >> far for steps (1) and (2) are available in the lio branch of the
> >> https://github.com/bvanassche/linux git repository. As you can see in
> >> that repository 42 of these 43 patches make sense even without knowing
> >> that something like SCST exists.
> > 
> > Please go ahead and post the series for review with bug fixes at the
> > head of the series, with future improvements following after the fixes.
> > 
> > Doing a brief review from your tree, I can see there are some useful
> > fixes, but also there are some patches that have incorrect assumptions
> > and/or break the existing userspace APIs.
> > 
> > That said, I'm happy to review the full series, and very much appreciate
> > your efforts to improve upstream code.
> > 
> > However, keep in mind that I'll not be merging anything for target-core
> > that adds support for out-of-tree code, nor anything that changes
> > existing target configfs APIs to fit out-of-tree code requirements.
> 
> Let's start with what we agree on and discuss the other topics during
> the LSF/MM summit.
> 

Looking at how your attempting to drive creation + removal of struct
config_group from within kernel code here:

target: Add target port registration API
https://github.com/bvanassche/linux/commit/dbb8bf32db3428ede6ecc688ede1e5e01fc59d88

is the exactly the wrong approach to take.

The creation and deletion of struct config_group must be driven by
user-space, and by user-space only.  No exceptions will be made.

This was already proposed way back during the original LIO configfs
conversion in 2008, and was strongly rejected.  To quote Joel [1]:

  "That's precisely what you don't do with configfs.  It's a
   defined "not to be done" thing.  So there's no preferred way, there's
   no way at all."

What it fundamentally comes down to is that you need to change SCST to
be driven by user-space like LIO does using configfs to properly
reference count kernel data structures, and not try to hack upstream
target-core and target drivers to fit your out-of-tree sysfs design.

And you'll recall Greg already commented on this back in 2010 [2], and
his response to you was:

  "And use configfs for your configuration stuff, that's what it is 
   there for, and if it doesn't somehow work properly for you, please 
   work with the configfs developers to fix that up."

--nab

[1] http://marc.info/?l=linux-scsi&m=122109943422792&w=2
[2] http://marc.info/?l=linux-scsi&m=128985930727238&w=2

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Bart Van Assche]

On 02/27/15 22:58, Nicholas A. Bellinger wrote:
> Looking at how your attempting to drive creation + removal of struct
> config_group from within kernel code here:
> 
> target: Add target port registration API
> https://github.com/bvanassche/linux/commit/dbb8bf32db3428ede6ecc688ede1e5e01fc59d88
> 
> is the exactly the wrong approach to take.
> 
> The creation and deletion of struct config_group must be driven by
> user-space, and by user-space only.  No exceptions will be made.

There exists an approach that preserves the ABI of both SCST and LIO,
namely:
* Add empty transport_register_wwn() and transport_unregister_wwn()
  functions in the LIO core.
* Add calls to these functions at the appropriate place in the FC
  and SRP target drivers.
* In the SCST implementation of the unified target driver API, route
  calls to transport_register_wwn() and transport_unregister_wwn() to
  scst_register_target() and scst_unregister_target() respectively.

Bart.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Nicholas A. Bellinger]

On Sat, 2015-02-28 at 12:59 +0100, Bart Van Assche wrote:
> On 02/27/15 22:58, Nicholas A. Bellinger wrote:
> > Looking at how your attempting to drive creation + removal of struct
> > config_group from within kernel code here:
> > 
> > target: Add target port registration API
> > https://github.com/bvanassche/linux/commit/dbb8bf32db3428ede6ecc688ede1e5e01fc59d88
> > 
> > is the exactly the wrong approach to take.
> > 
> > The creation and deletion of struct config_group must be driven by
> > user-space, and by user-space only.  No exceptions will be made.
> 
> There exists an approach that preserves the ABI of both SCST and LIO,
> namely:
> * Add empty transport_register_wwn() and transport_unregister_wwn()
>   functions in the LIO core.
> * Add calls to these functions at the appropriate place in the FC
>   and SRP target drivers.
> * In the SCST implementation of the unified target driver API, route
>   calls to transport_register_wwn() and transport_unregister_wwn() to
>   scst_register_target() and scst_unregister_target() respectively.
> 

NAK.

I'll not consider any hooks in upstream target code, and certainly not
driver hooks for a control-plane approach unanimously rejected by both
configfs and sysfs maintainers in 2008 and 2010.

The whole point of target_core_fabric_configfs.c is to transparently
reference count fabric driver data structures populated underneath
/sys/kernel/config/target/$FABRIC/, who's design also has the added
bonus of providing module reference counting to drivers 'for free'.

So until you're prepared to evolve on this issue and stop pretending as
if the future didn't already happen, this type of discussion at LSF is a
waste of time.

--nab

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Bart Van Assche]

On 03/02/15 07:59, Nicholas A. Bellinger wrote:
> On Sat, 2015-02-28 at 12:59 +0100, Bart Van Assche wrote:
>> On 02/27/15 22:58, Nicholas A. Bellinger wrote:
>>> Looking at how your attempting to drive creation + removal of struct
>>> config_group from within kernel code here:
>>>
>>> target: Add target port registration API
>>> https://github.com/bvanassche/linux/commit/dbb8bf32db3428ede6ecc688ede1e5e01fc59d88
>>>
>>> is the exactly the wrong approach to take.
>>>
>>> The creation and deletion of struct config_group must be driven by
>>> user-space, and by user-space only.  No exceptions will be made.
>>
>> There exists an approach that preserves the ABI of both SCST and LIO,
>> namely:
>> * Add empty transport_register_wwn() and transport_unregister_wwn()
>>   functions in the LIO core.
>> * Add calls to these functions at the appropriate place in the FC
>>   and SRP target drivers.
>> * In the SCST implementation of the unified target driver API, route
>>   calls to transport_register_wwn() and transport_unregister_wwn() to
>>   scst_register_target() and scst_unregister_target() respectively.
> 
> NAK.
> 
> I'll not consider any hooks in upstream target code.

If you do not agree with what I proposed in my previous e-mail that's
fine. But the companies who maintain SCSI target drivers and I would
appreciate it if you would propose and alternative approach that allows
SCSI target driver unification. I think the LIO users would also
appreciate such a unification. As an example, the SCST FCoE target
driver works much more reliably than the LIO FCoE target driver. The
only way to motivate the SCST maintainers to improve the LIO FCoE target
driver is by unifying the target driver API.

Bart.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Christoph Hellwig]

This is exactly why I didn't want to put this topic onto the LSF agenda.

There are tons of very useful fixes and cleanus in Barts series, and I'd
really like to get them in ASAP.

As Nic pointed out (in a slightly unhelpful tone) we're not interested
in adding any hooks for out of tree code.  But if SCST needs hooks it
should eithe switch to the in-kerel way of doing it, or if has a clearly
better way of doing it merge that into the kernel.  I'm pretty sure
there are various pieces that would make a lot of sense to merge either
way, and getting towards a single target with a broad maintainer and
user base seem like a way better approach that adding weird hooks.  But
I don't think that's even on the table for LSF this year, maybe next
year if everyone involved works hard on it.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Bart Van Assche]

On 03/05/15 14:23, Christoph Hellwig wrote:
> This is exactly why I didn't want to put this topic onto the LSF agenda.
> 
> There are tons of very useful fixes and cleanus in Barts series, and I'd
> really like to get them in ASAP.
> 
> As Nic pointed out (in a slightly unhelpful tone) we're not interested
> in adding any hooks for out of tree code.  But if SCST needs hooks it
> should eithe switch to the in-kerel way of doing it, or if has a clearly
> better way of doing it merge that into the kernel.  I'm pretty sure
> there are various pieces that would make a lot of sense to merge either
> way, and getting towards a single target with a broad maintainer and
> user base seem like a way better approach that adding weird hooks.  But
> I don't think that's even on the table for LSF this year, maybe next
> year if everyone involved works hard on it.

Hello Christoph,

If we would do what Nic proposed - modify SCST such that it uses
configfs instead of sysfs - then that would result in the removal of at
least one SCST feature that is important to its users, namely automatic
population of the configuration filesystem with hardware target port
information. Apparently Nic does not want to convert LIO from configfs
to sysfs. The reason I proposed to add empty transport_register_wwn()
and transport_unregister_wwn() functions in the LIO core is because this
allows LIO to keep using configfs and does not require to remove
features from SCST. BTW, a message that was posted four years ago on the
linux-scsi mailing list contains an excellent explanation of why sysfs
has been chosen for the SCST user space API. From
http://thread.gmane.org/gmane.linux.scsi/65615/focus=65618: <quote>
I think the overall philosophical point here, and it's a good one
because I've heard it from several sources, is that it's not possible to
separate configuration from status completely.  The classic example is
where the kernel has to validate and adjust config information, but the
storage specific one is where events alter the topology.  In either
case, the configfs tree gets out of sync with reality if the kernel does
the adjustment..  Just saying we have to use a user space tool to fix it
up is a bit of a cop out because the kernel has already adjusted its own
configuration, so getting userspace to work out what the kernel's done
and adjust configfs is a bit sub optimal.</quote>

Bart.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Andy Grover]

On 03/05/2015 08:06 AM, Bart Van Assche wrote:

> If we would do what Nic proposed - modify SCST such that it uses
> configfs instead of sysfs - then that would result in the removal of at
> least one SCST feature that is important to its users, namely automatic
> population of the configuration filesystem with hardware target port
> information. Apparently Nic does not want to convert LIO from configfs
> to sysfs.

Clearly configfs is here to stay, but is there anything that says LIO 
couldn't have both? Besides the issue at hand, there are some other bits 
of useful info in LIO that aren't available because of configfs' 
limitations, but that would be possible via sysfs.

-- Andy

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Bart Van Assche]

On 03/05/15 19:39, Andy Grover wrote:
> On 03/05/2015 08:06 AM, Bart Van Assche wrote:
>> If we would do what Nic proposed - modify SCST such that it uses
>> configfs instead of sysfs - then that would result in the removal of at
>> least one SCST feature that is important to its users, namely automatic
>> population of the configuration filesystem with hardware target port
>> information. Apparently Nic does not want to convert LIO from configfs
>> to sysfs.
> 
> Clearly configfs is here to stay, but is there anything that says LIO 
> couldn't have both? Besides the issue at hand, there are some other bits 
> of useful info in LIO that aren't available because of configfs' 
> limitations, but that would be possible via sysfs.

(resending my reply since apparently my previous e-mail didn't make it
to the lists)

Hello Andy,

Let's have a look at FC HBAs. Most FC HBAs have a PCIe connector and
support NPIV. If an NPIV WWN is configured then that WWN is passed from
user space to the kernel. The PCIe specification supports hot-plugging.
Although I'm not sure there exist FC HBAs that support hot-plugging,
upon hot removal of a PCIe HBA the kernel is notified of this and must
remove all NPIV WWNs from the filesystem that is used to configure the
SCSI target subsystem. Since it is not allowed to remove configfs
directories from inside the kernel I think NPIV is an example of a use
case that cannot be handled properly when using configfs for configuring
a SCSI target subsystem, even when combined in some way with sysfs.

Bart.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Bart Van Assche]

On 02/20/15 11:49, Bart Van Assche wrote:
> On 01/19/15 10:36, Bart Van Assche wrote:
>> On 01/19/15 10:22, Christoph Hellwig wrote:
>>> On Thu, Jan 15, 2015 at 05:13:00PM +0100, Bart Van Assche wrote:
>>>> My goal is to realize this proposal without adding hooks for out-of-tree
>>>> code in the upstream kernel. What I had in mind is to raise the
>>>> abstraction level of the API between LIO core and target drivers a
>>>> little bit (e.g. by using accessor functions where necessary instead of
>>>> accessing structure members directly)
>>>
>>> That's very much a hook, althiugh a week one.
>>>
>>> Either way I don't think bringing up a very much political topic
>>> without even any code to discuss isn't a very valueable use of our time
>>> slots.
>>
>> A possible approach is that I start implementing a unified SRP target
>> driver and post that driver together with the necessary LIO and SCST
>> core changes before the LSF/MM starts. That could be a helpful starting
>> point for further discussions.
> 
> (replying to my own e-mail)
> 
> Hello Christoph,
> 
> What I proposed myself consists of three steps:
> 1. Updating the LIO SRP target driver to a more recent version.
> 2. Target driver and LIO core refactoring such that the LIO and
>    SCST APIs can be unified.
> 3. Adding support in SCST for the unified target driver API.
> 
> This work is taking a little more time than I had expected - I still
> have to start with step (3). But I can already show in which direction I
> would like to go for steps (1) and (2). The 43 patches I came up with so
> far for steps (1) and (2) are available in the lio branch of the
> https://github.com/bvanassche/linux git repository. As you can see in
> that repository 42 of these 43 patches make sense even without knowing
> that something like SCST exists.

(again replying to my own e-mail)

Hello everyone,

As promised a prototype of the implementation of the unified target
drive API for SCST has been made available publicly (see also
https://github.com/bvanassche/scst-prototype-for-lsf-mm/tree/unified-target-driver-api).
Although this is still a prototype I think this code illustrates that it
is possible to unify the target driver APIs.

The patches in the unified-target-driver-api branch on top of the
SourceForge SCST trunk are:
0001-Add-.gitignore.patch
0002-scst-Introduce-task_mgmt_fn_pre_exec-in-the-target-t.patch
0003-scst-Add-unified-target-driver-API.patch
0004-srpt-unified-Import-source-files-from-lio-kernel-bra.patch
0005-srpt-unified-Add-out-of-tree-build-infrastructure.patch
0006-srpt-unified-Make-this-driver-build-against-the-unif.patch

Patches 2 and 3 add an implementation of the unified target driver API
in SCST. Header files are added under scst/include with the same names
as the LIO header files. The implementation of that API translates e.g.
between ranges of enumeration constants and also between the target
driver registration templates of the two projects.

Patch 4 imports the SRP target driver source code from the git
repository mentioned in my previous e-mail.

Patch 5 adds a makefile that allows to build that driver as an
out-of-tree kernel driver. The compiler include path is set up such that
the compiler finds the unified header files under scst/include first.

Patch 6 makes the unified SRP target driver build against the SCST
version of the unified API. The following conclusions can be drawn from
that patch:
- A few user space ABI differences are addressed via #ifdefs (GUID
  format "%04x:%04x:%04x:%04x:%04x:%04x:%04x:%04x" for SCST versus
  "0x%04x%04x%04x%04x%04x%04x%04x%04x" for LIO and target driver entry
  point ib_srpt for SCST versus srpt for LIO).
- That patch contains further unification of the target driver API, e.g.
  introduction of functions like transport_get_data_out_buf(),
  target_cmd_done(), transport_get_resid(), transport_sense_valid() etc.
- Since the SCST core manages a list of SCSI target ports, the
  port_list data structure isn't needed in the SCST version of this
  target driver.
- In some areas further unification is needed, e.g. the enable_target()
  / is_target_enabled() callback functions and also for the API for
  session unregistration (see also the transport_deregister_session()
  call). As one can see the persistent reservation TransportID callback
  functions have not yet been unified. However, I think these should be
  unified and also that this is possible without too much effort.
- Another open issue is that creation of target driver configfs/sysfs
  attributes is not yet supported in this prototype of the SCST version
  of the unified target driver API.

Bart.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Andy Grover]

On 03/05/2015 11:25 PM, Bart Van Assche wrote:
> On 03/05/15 19:39, Andy Grover wrote:
>> On 03/05/2015 08:06 AM, Bart Van Assche wrote:
>>> If we would do what Nic proposed - modify SCST such that it uses
>>> configfs instead of sysfs - then that would result in the removal of at
>>> least one SCST feature that is important to its users, namely automatic
>>> population of the configuration filesystem with hardware target port
>>> information. Apparently Nic does not want to convert LIO from configfs
>>> to sysfs.
>>
>> Clearly configfs is here to stay, but is there anything that says LIO
>> couldn't have both? Besides the issue at hand, there are some other bits
>> of useful info in LIO that aren't available because of configfs'
>> limitations, but that would be possible via sysfs.
>
> (resending my reply since apparently my previous e-mail didn't make it
> to the lists)

> Let's have a look at FC HBAs. Most FC HBAs have a PCIe connector and
> support NPIV. If an NPIV WWN is configured then that WWN is passed from
> user space to the kernel. The PCIe specification supports hot-plugging.
> Although I'm not sure there exist FC HBAs that support hot-plugging,
> upon hot removal of a PCIe HBA the kernel is notified of this and must
> remove all NPIV WWNs from the filesystem that is used to configure the
> SCSI target subsystem. Since it is not allowed to remove configfs
> directories from inside the kernel I think NPIV is an example of a use
> case that cannot be handled properly when using configfs for configuring
> a SCSI target subsystem, even when combined in some way with sysfs.

This isn't a problem, because having a configuration for a device's WWNs 
in configfs does not mean the device must be present. When the device 
shows up then the target config will be used for it, and if the device 
is removed the configuration no longer does anything, but still remains.

What's in configfs reflects only the config, which is why I'm wondering 
if adding sysfs support might also be a good thing. sysfs would let LIO 
create entries to better reflect actual running state, and be less 
limited in reporting status on only "things that have config".

For example, LIO requires an ACL for all initiators but has a setting to 
autogenerate these if per-ACL lun-mapping & auth is not desired. There's 
no way for userspace to get a list of generated ACLs -- maybe userspace 
wants to make it easy to convert generated ACLs to actual ACLs. LIO 
can't create configfs nodes in-kernel and each configfs file is limited 
to PAGE_SIZE so we can't just have a configfs entry that lists all of 
them. That's what I was getting at above: configfs works great for 
exactly what we're using it for, but there are other things sysfs does 
better so we may want to use both.

-- Andy

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Nicholas A. Bellinger]

On Fri, 2015-03-06 at 08:25 +0100, Bart Van Assche wrote:
> On 03/05/15 19:39, Andy Grover wrote:
> > On 03/05/2015 08:06 AM, Bart Van Assche wrote:
> >> If we would do what Nic proposed - modify SCST such that it uses
> >> configfs instead of sysfs - then that would result in the removal of at
> >> least one SCST feature that is important to its users, namely automatic
> >> population of the configuration filesystem with hardware target port
> >> information. Apparently Nic does not want to convert LIO from configfs
> >> to sysfs.
> > 
> > Clearly configfs is here to stay, but is there anything that says LIO 
> > couldn't have both? Besides the issue at hand, there are some other bits 
> > of useful info in LIO that aren't available because of configfs' 
> > limitations, but that would be possible via sysfs.
> 
> (resending my reply since apparently my previous e-mail didn't make it
> to the lists)
> 
> Hello Andy,
> 
> Let's have a look at FC HBAs. Most FC HBAs have a PCIe connector and
> support NPIV. If an NPIV WWN is configured then that WWN is passed from
> user space to the kernel. The PCIe specification supports hot-plugging.
> Although I'm not sure there exist FC HBAs that support hot-plugging,
> upon hot removal of a PCIe HBA the kernel is notified of this and must
> remove all NPIV WWNs from the filesystem that is used to configure the
> SCSI target subsystem. Since it is not allowed to remove configfs
> directories from inside the kernel I think NPIV is an example of a use
> case that cannot be handled properly when using configfs for configuring
> a SCSI target subsystem, even when combined in some way with sysfs.

Not exactly.  configfs in this context works similar to hotplug remove
of a FC HBA in initiator mode (or a USB drive even) with a mounted
filesystem on one of the LUNs.

That is, a kobject reference is obtained on Scsi_Host->shost_gendev at
/sys/kernel/config/target/qla2xxx_npiv/$NPIV_WWPN/ creation time,
preventing the final device_release() from being called until the last
reference is dropped via rmdir ../target/qla2xxx_npiv/$NPIV_WWPN/.

The practical method is to use a udev script called at hotplug remove
time (similar to USB) to drop the NPIV configfs groups associated with
device(s) being removed, allowing the underlying kernel data structures
to be released.

--nab

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Sagi Grimberg]

On 3/6/2015 9:15 PM, Andy Grover wrote:
> On 03/05/2015 11:25 PM, Bart Van Assche wrote:
>> On 03/05/15 19:39, Andy Grover wrote:
>>> On 03/05/2015 08:06 AM, Bart Van Assche wrote:
>>>> If we would do what Nic proposed - modify SCST such that it uses
>>>> configfs instead of sysfs - then that would result in the removal of at
>>>> least one SCST feature that is important to its users, namely automatic
>>>> population of the configuration filesystem with hardware target port
>>>> information. Apparently Nic does not want to convert LIO from configfs
>>>> to sysfs.
>>>
>>> Clearly configfs is here to stay, but is there anything that says LIO
>>> couldn't have both? Besides the issue at hand, there are some other bits
>>> of useful info in LIO that aren't available because of configfs'
>>> limitations, but that would be possible via sysfs.
>>
>> (resending my reply since apparently my previous e-mail didn't make it
>> to the lists)
>
>> Let's have a look at FC HBAs. Most FC HBAs have a PCIe connector and
>> support NPIV. If an NPIV WWN is configured then that WWN is passed from
>> user space to the kernel. The PCIe specification supports hot-plugging.
>> Although I'm not sure there exist FC HBAs that support hot-plugging,
>> upon hot removal of a PCIe HBA the kernel is notified of this and must
>> remove all NPIV WWNs from the filesystem that is used to configure the
>> SCSI target subsystem. Since it is not allowed to remove configfs
>> directories from inside the kernel I think NPIV is an example of a use
>> case that cannot be handled properly when using configfs for configuring
>> a SCSI target subsystem, even when combined in some way with sysfs.
>
> This isn't a problem, because having a configuration for a device's WWNs
> in configfs does not mean the device must be present. When the device
> shows up then the target config will be used for it, and if the device
> is removed the configuration no longer does anything, but still remains.
>
> What's in configfs reflects only the config, which is why I'm wondering
> if adding sysfs support might also be a good thing. sysfs would let LIO
> create entries to better reflect actual running state, and be less
> limited in reporting status on only "things that have config".
>
> For example, LIO requires an ACL for all initiators but has a setting to
> autogenerate these if per-ACL lun-mapping & auth is not desired. There's
> no way for userspace to get a list of generated ACLs -- maybe userspace
> wants to make it easy to convert generated ACLs to actual ACLs. LIO
> can't create configfs nodes in-kernel and each configfs file is limited
> to PAGE_SIZE so we can't just have a configfs entry that lists all of
> them.  That's what I was getting at above: configfs works great for
> exactly what we're using it for, but there are other things sysfs does
> better so we may want to use both.

That always annoyed me too. But seeing generated ACLs in sysfs while
user configured ACLs lives in configfs is weird isn't it? I'm wander if
having both sysfs/configfs interfaces might be confusing and/or messy.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Nicholas A. Bellinger]

On Fri, 2015-03-06 at 11:15 -0800, Andy Grover wrote:

<SNIP>

> This isn't a problem, because having a configuration for a device's WWNs 
> in configfs does not mean the device must be present. When the device 
> shows up then the target config will be used for it, and if the device 
> is removed the configuration no longer does anything, but still remains.
> 
> What's in configfs reflects only the config, which is why I'm wondering 
> if adding sysfs support might also be a good thing. sysfs would let LIO 
> create entries to better reflect actual running state, and be less 
> limited in reporting status on only "things that have config".
> 
> For example, LIO requires an ACL for all initiators but has a setting to 
> autogenerate these if per-ACL lun-mapping & auth is not desired. There's 
> no way for userspace to get a list of generated ACLs --

Thanks for the reminder.

Just posted a small series for allowing fabric drivers to expose this
using a new 'dynamic_session' TPG attribute, so that user-space can
optionally perform dynamic -> explicit se_node_acl conversion ...

> maybe userspace 
> wants to make it easy to convert generated ACLs to actual ACLs. LIO 
> can't create configfs nodes in-kernel and each configfs file is limited 
> to PAGE_SIZE so we can't just have a configfs entry that lists all of 
> them.

... which yes, like all sysfs + configfs attrs is currently limited to a
PAGE_SIZE of output.

However, given iSCSI IQNs are on average ~64 bytes (max 224 bytes), that
is still enough for listing 64 (or 18 worst case) sessions created by
dynamically generated se_node_acls .

For FC, this is 18 bytes per FC initiator WWPN string using physical
port addressing.

So for most folk's purposes on a per target endpoint basis, they won't
have enough active sessions to run out of buffer here.

>  That's what I was getting at above: configfs works great for 
> exactly what we're using it for, but there are other things sysfs does 
> better so we may want to use both.
> 

Not sure what you had in mind, but I did attempt using both in the early
days..  Eg: using /sys/block/$BDEV/ as sysfs symlink source for passing
struct block_device into /sys/kernel/config/target/core/$IBLOCK/$DEV/.

IIRC, Greg-KH's response back then was along the lines of 'thanks, but
please stop now and walk away slowly'.

--nab

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Christoph Hellwig]

On Thu, Mar 05, 2015 at 05:06:47PM +0100, Bart Van Assche wrote:
> If we would do what Nic proposed - modify SCST such that it uses
> configfs instead of sysfs - then that would result in the removal of at
> least one SCST feature that is important to its users, namely automatic
> population of the configuration filesystem with hardware target port
> information. Apparently Nic does not want to convert LIO from configfs
> to sysfs. The reason I proposed to add empty transport_register_wwn()
> and transport_unregister_wwn() functions in the LIO core is because this
> allows LIO to keep using configfs and does not require to remove
> features from SCST.

I see your point, but I don't think it matters at this time.  If we had
all the core and driver code unified and the only thing preventing us
from going from two widely ued kernel targets to one was adding the
sysfs interface that would be an easy puill to swallow.  But there is a
lot of work to make that happen.  Just adding a stub that isn't used in
tree to support out of tree modules really isn't something we do, so
I have no problem with a plain outright rejection of it.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Andy Grover]

On 03/06/2015 10:25 PM, Nicholas A. Bellinger wrote:

>>   That's what I was getting at above: configfs works great for
>> exactly what we're using it for, but there are other things sysfs does
>> better so we may want to use both.

> Just posted a small series for allowing fabric drivers to expose
> this using a new 'dynamic_session' TPG attribute, so that user-space
> can optionally perform dynamic -> explicit se_node_acl conversion.
<snip>
> However, given iSCSI IQNs are on average ~64 bytes (max 224 bytes),
> that is still enough for listing 64 (or 18 worst case) sessions
> created by dynamically generated se_node_acls .

Nick thanks for implementing the support for listing dynamic ACLs via 
configfs. This will come in handy, ACK. I'll follow up separately on the 
broader use-both-sysfs-and-configfs question I had.

Regards -- Andy

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Dr. Greg Wettstein]

On Mar 1, 10:59pm, "Nicholas A. Bellinger" wrote:
} Subject: Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target driv

Hi, I hope the week is progressing well for everyone.

> On Sat, 2015-02-28 at 12:59 +0100, Bart Van Assche wrote:
> > On 02/27/15 22:58, Nicholas A. Bellinger wrote:
> > > Looking at how your attempting to drive creation + removal of struct
> > > config_group from within kernel code here:
> > > 
> > > target: Add target port registration API
> > > https://github.com/bvanassche/linux/commit/dbb8bf32db3428ede6ecc688ede1e5e01fc59d88
> > > 
> > > is the exactly the wrong approach to take.
> > > 
> > > The creation and deletion of struct config_group must be driven by
> > > user-space, and by user-space only.  No exceptions will be made.
> > 
> > There exists an approach that preserves the ABI of both SCST and LIO,
> > namely:
> > * Add empty transport_register_wwn() and transport_unregister_wwn()
> >   functions in the LIO core.
> > * Add calls to these functions at the appropriate place in the FC
> >   and SRP target drivers.
> > * In the SCST implementation of the unified target driver API, route
> >   calls to transport_register_wwn() and transport_unregister_wwn() to
> >   scst_register_target() and scst_unregister_target() respectively.
> > 

> NAK.
>
> I'll not consider any hooks in upstream target code, and certainly
> not driver hooks for a control-plane approach unanimously rejected
> by both configfs and sysfs maintainers in 2008 and 2010.
>
> The whole point of target_core_fabric_configfs.c is to transparently
> reference count fabric driver data structures populated underneath
> /sys/kernel/config/target/$FABRIC/, who's design also has the added
> bonus of providing module reference counting to drivers 'for free'.
>
> So until you're prepared to evolve on this issue and stop pretending
> as if the future didn't already happen, this type of discussion at
> LSF is a waste of time.

So, in this spirit of collaborative and collegial exchange, which I
see that Christoph has called for, I have a question for those who
posess familiarity with the LIO control plane design.

In this parousia of total user-space configuration, not necessarily a
bad thing, what would be the target fabric presentation of a storage
target server which has one Qlogic 8242 card and one Qlogic 8362 card
in it?

> --nab

Best wishes for a pleasant remainder of the week to everyone.

Greg

}-- End of excerpt from "Nicholas A. Bellinger"

As always,
Dr. G.W. Wettstein, Ph.D.   Enjellic Systems Development, LLC.
4206 N. 19th Ave.           Specializing in information infra-structure
Fargo, ND  58102            development.
PH: 701-281-1686
FAX: 701-281-3949           EMAIL: greg@enjellic.com
------------------------------------------------------------------------------
"Man, despite his artistic pretensions, his sophistication and many
 accomplishments, owes the fact of his existence to a six-inch layer of
 topsoil and the fact that it rains."
                                -- Anonymous writer on perspective.
                                   GAUSSIAN quote.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Dr. Greg Wettstein]

On Feb 9,  2:16pm, Bart Van Assche wrote:
} Subject: Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target driv

> On 02/03/15 11:06, Dr. Greg Wettstein wrote:
> > It takes a six line patch to the in-kernel Qlogic target driver for
> > SCST to leverage and contribute positively to the state of the
> > in-kernel driver.  A six line patch, which if we were engaging in
> > grounded engineering discussions, implements an interface which we
> > haven't found anyone who suggests is unfounded.

> Hi Greg,

Hi Bart, I hope your week is going well.

> Thanks for your feedback. Although I certainly value your work,
> please note that what I proposed goes further than what you have
> done. If I understood your e-mails correctly what you have done is
> to unify the API between the QLogic initiator and target drivers for
> SCST and LIO. My proposal comprises not only a unification of the
> initiator drivers but also of the target drivers.

Just to clarify issues for discussion our work on the SQATGT driver
centered exclusively on the target component of the Qlogic driver.

Our work focused on providing an SCST equivalant to the tcm_qla2xxx.c
driver which the in-kernel implementation of the Qlogic target driver
uses to interface with the TCM target core.  The {scst,tcm}_qla2xxx
drivers register a function dispatch structure with the Qlogic
hardware target driver, qla_target.c.  These drivers take the actions
requested by the hardware driver and translate them into the necessary
target core processing functions.

As I have indicated the scst_qla2xxx driver could be distributed
separately from the kernel since it only relies on symbols which are
exported from the Qlogic driver.  I'm not familiar with IB and such
but it seems to be a reasonable architecture for both camps of
interest to collaborate on.

> Last week I finally found the time to start working on an sample
> implementation of such a unified driver. I hope that I will be able to
> make that code available for review somewhere next week.

Let us know when that becomes available since we would certainly like
to be assisting and supporting whatever the community agrees to as a
solution.

> Bart.

Have a good week.

}-- End of excerpt from Bart Van Assche

As always,
Dr. G.W. Wettstein, Ph.D.   Enjellic Systems Development, LLC.
4206 N. 19th Ave.           Specializing in information infra-structure
Fargo, ND  58102            development.
PH: 701-281-1686
FAX: 701-281-3949           EMAIL: greg@enjellic.com
------------------------------------------------------------------------------
"... Doc (interviews) is closest in look&feel to a Windows word-processor.
 It's even slow.  Very slow.  Hard to set up fonts and printing in the
 version I have...."
                                -- David Johnson

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Bart Van Assche]

On 02/03/15 11:06, Dr. Greg Wettstein wrote:
> On Jan 19,  1:21am, Christoph Hellwig wrote:
>> On Thu, Jan 15, 2015 at 05:13:00PM +0100, Bart Van Assche wrote:
>>> My goal is to realize this proposal without adding hooks for out-of-tree
>>> code in the upstream kernel. What I had in mind is to raise the
>>> abstraction level of the API between LIO core and target drivers a
>>> little bit (e.g. by using accessor functions where necessary instead of
>>> accessing structure members directly)
> 
>> That's very much a hook, althiugh a week one.
>>
>> Either way I don't think bringing up a very much political topic
>> without even any code to discuss isn't a very valueable use of our time
>> slots.
> 
> There is code, no one is bothering to look at it or understand the
> issues involved.
> 
> It takes a six line patch to the in-kernel Qlogic target driver for
> SCST to leverage and contribute positively to the state of the
> in-kernel driver.  A six line patch, which if we were engaging in
> grounded engineering discussions, implements an interface which we
> haven't found anyone who suggests is unfounded.

Hi Greg,

Thanks for your feedback. Although I certainly value your work, please
note that what I proposed goes further than what you have done. If I
understood your e-mails correctly what you have done is to unify the API
between the QLogic initiator and target drivers for SCST and LIO. My
proposal comprises not only a unification of the initiator drivers but
also of the target drivers.

Last week I finally found the time to start working on an sample
implementation of such a unified driver. I hope that I will be able to
make that code available for review somewhere next week.

Bart.

Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target drivers

[Dr. Greg Wettstein]

On Jan 19,  1:21am, Christoph Hellwig wrote:
} Subject: Re: [Lsf-pc] [LSF/MM TOPIC] Unifying the LIO and SCST target driv

Hi, I hope the week is going well for everyone.

I'm a bit behind on e-mail and getting ready for travel but wanted to
reply back to this thread because of our involvement and interest in
this whole situation.

> On Thu, Jan 15, 2015 at 05:13:00PM +0100, Bart Van Assche wrote:
> > My goal is to realize this proposal without adding hooks for out-of-tree
> > code in the upstream kernel. What I had in mind is to raise the
> > abstraction level of the API between LIO core and target drivers a
> > little bit (e.g. by using accessor functions where necessary instead of
> > accessing structure members directly)

> That's very much a hook, althiugh a week one.
> 
> Either way I don't think bringing up a very much political topic
> without even any code to discuss isn't a very valueable use of our time
> slots.

There is code, no one is bothering to look at it or understand the
issues involved.

It takes a six line patch to the in-kernel Qlogic target driver for
SCST to leverage and contribute positively to the state of the
in-kernel driver.  A six line patch, which if we were engaging in
grounded engineering discussions, implements an interface which we
haven't found anyone who suggests is unfounded.

Our bona-fides in all this is that we developed the SCST target
interface driver which allows SCST to sit on top of the in-kernel
Qlogic target driver.  That driver is in use in a number of locations
driving hundreds and hundreds and hundreds of terrabytes of storage.

See the scst-devel and linux-scsi lists and the following:

ftp://ftp.enjellic.com/pub/scst/sqa_driver-1.1.tar.gz

Based on these experiences and building and debugging very complex
converged networks we can state pretty unequivocably that the current
target driver situation is working to no ones advantage.

The in-kernel target driver hasn't seen significant improvement or
work in over a year.  We carry a number of patches and bug fixes to
the driver which we have identified and fixed locally.  We are so
frustrated with the whole situation we don't even bother to push fixes
upstream.  So the notion that holding the in-kernel target driver as a
sacred bastion of purity will lead to a better codebase is ill
founded.

We cherry pick patches back and forther between the in-kernel and
out-of-kernel drivers and can speak pretty authoritatively to the fact
the in-kernel driver is now slipping behind the out-of-tree code.

There is a significant regression, which we believe can lead to
catastrophic data corruption under the right conditions, in the Qlogic
target driver, both in-tree and out-of-tree.  We believe the problem
is in session handling and have provided logs to Qlogic and were told,
'there is a issue and we are looking into it', but haven't heard
anything back.

The simple fact of the matter is that in modern CNA's/HBA's all of
this behavior is ultimately governed by the firmware in the cards.  No
one has access to that but the vendors.  We can make their job in
supporting the whole storage community easier if we are all looking at
and working on the same code, to the extent that is possible.

I can't speak to Infiniband but in the realm of fibre-channel/FCOE the
only problem is political not technical.

This is not an issue of a vendor electing to keep code out of tree.
The decision was made to take LIO into the kernel and there are not
going to be two in-kernel target stacks, so SCST is going to live out
of tree.  Those of us with significant engineering investments are not
going to dump SCST and move to the in tree code, good, bad or
indifferent.

We are willing and have expended the engineering time to make this
situation better.  An engineering based discussion comes down to what
types of legitimate API's the kernel needs to expose.  We do that all
over the kernel, I'm not sure why storage, other then politics, needs
to be any different.

Best wishes for a productive week.

Greg

}-- End of excerpt from Christoph Hellwig

As always,
Dr. G.W. Wettstein, Ph.D.   Enjellic Systems Development, LLC.
4206 N. 19th Ave.           Specializing in information infra-structure
Fargo, ND  58102            development.
PH: 701-281-1686
FAX: 701-281-3949           EMAIL: greg@enjellic.com
------------------------------------------------------------------------------
"For a successful technology, reality must take precedence over public
 relations, for nature cannot be fooled."
                                -- Richard Feynmann