Re: [Scst-devel] Fwd: Re: linuxcon 2010...

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Chetan Loke]

Hello James and others,

--- On Tue, 8/17/10, James Bottomley <James.Bottomley@suse.de> wrote:

> From: James Bottomley <James.Bottomley@suse.de>
> Subject: Re: [Scst-devel] Fwd: Re:  linuxcon 2010...
> To: "Vladislav Bolkhovitin" <vst@vlnb.net>
> Cc: "scst-devel" <scst-devel@lists.sourceforge.net>, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org
> Date: Tuesday, August 17, 2010, 8:30 PM
> On Mon, 2010-08-16 at 20:20 +0400,
> Vladislav Bolkhovitin wrote:
> > Hello James,
> > 
> > Could you comment rumors that decision about future
> Linux SCSI target 
> > subsystem is done as well as other related rumors:
> 
> If this is related to LSF, the notes on the I/O track are
> here:
> 
> http://lwn.net/Articles/400491/


During the open panel, my question was really specific - 

Q) What is the future of a SCSI-target subsystem in linux. Which 
   target engine/subsystem can we expect?

Your answer) There is place for only 1 target-subsystem in the Linux scsi stack and in the LSF summit the decision was taken to merge LIO. Has that
decision changed since the summit?

As a scst-user what I would like to understand is, what was that decision based on? Because the LSF summit was 'small by invitation' only summit. The notes don't give us an insight on the selection criteria/merits etc.


> 
> > 3. I have heard you said "Vlad wasn't comfortable in
> handing up the 
> > control to the maintainers ... (this is how kernel.org
> works)." I have 
> > no idea what you meant. I have never been asked about
> anything like 
> > that, so I couldn't say anyhow that I'm not
> comfortable with anything. 
> > Could you clarify that?
> > 

3) above is something that I emailed Vlad and the scst community based on our offline conversation after the open panel. If SCST really has licensing issues then I will personally stop using SCST. Since Vlad hasn't
expressed any concerns on the above and neither have you commented on it, is it safe to assume that the licensing requirement is a non-issue?


Chetan Loke



      

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Chetan Loke]

Hello James and others,

--- On Tue, 8/17/10, James Bottomley <James.Bottomley@suse.de> wrote:

> From: James Bottomley <James.Bottomley@suse.de>
> Subject: Re: [Scst-devel] Fwd: Re:  linuxcon 2010...
> To: "Vladislav Bolkhovitin" <vst@vlnb.net>
> Cc: "scst-devel" <scst-devel@lists.sourceforge.net>, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org
> Date: Tuesday, August 17, 2010, 8:30 PM
> On Mon, 2010-08-16 at 20:20 +0400,
> Vladislav Bolkhovitin wrote:
> > Hello James,
> > 
> > Could you comment rumors that decision about future
> Linux SCSI target 
> > subsystem is done as well as other related rumors:
> 
> If this is related to LSF, the notes on the I/O track are
> here:
> 
> http://lwn.net/Articles/400491/


During the open panel, my question was really specific - 

Q) What is the future of a SCSI-target subsystem in linux. Which 
   target engine/subsystem can we expect?

Your answer) There is place for only 1 target-subsystem in the Linux scsi stack and in the LSF summit the decision was taken to merge LIO. Has that
decision changed since the summit?

As a scst-user what I would like to understand is, what was that decision based on? Because the LSF summit was 'small by invitation' only summit. The notes don't give us an insight on the selection criteria/merits etc.


> 
> > 3. I have heard you said "Vlad wasn't comfortable in
> handing up the 
> > control to the maintainers ... (this is how kernel.org
> works)." I have 
> > no idea what you meant. I have never been asked about
> anything like 
> > that, so I couldn't say anyhow that I'm not
> comfortable with anything. 
> > Could you clarify that?
> > 

3) above is something that I emailed Vlad and the scst community based on our offline conversation after the open panel. If SCST really has licensing issues then I will personally stop using SCST. Since Vlad hasn't
expressed any concerns on the above and neither have you commented on it, is it safe to assume that the licensing requirement is a non-issue?


Chetan Loke



      

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[James Bottomley]

On Wed, 2010-08-18 at 07:58 -0700, Chetan Loke wrote:
> Hello James and others,
> 
> --- On Tue, 8/17/10, James Bottomley <James.Bottomley@suse.de> wrote:
> 
> > From: James Bottomley <James.Bottomley@suse.de>
> > Subject: Re: [Scst-devel] Fwd: Re:  linuxcon 2010...
> > To: "Vladislav Bolkhovitin" <vst@vlnb.net>
> > Cc: "scst-devel" <scst-devel@lists.sourceforge.net>, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org
> > Date: Tuesday, August 17, 2010, 8:30 PM
> > On Mon, 2010-08-16 at 20:20 +0400,
> > Vladislav Bolkhovitin wrote:
> > > Hello James,
> > > 
> > > Could you comment rumors that decision about future
> > Linux SCSI target 
> > > subsystem is done as well as other related rumors:
> > 
> > If this is related to LSF, the notes on the I/O track are
> > here:
> > 
> > http://lwn.net/Articles/400491/
> 
> 
> During the open panel, my question was really specific - 
> 
> Q) What is the future of a SCSI-target subsystem in linux. Which 
>    target engine/subsystem can we expect?
> 
> Your answer) There is place for only 1 target-subsystem in the Linux
> scsi stack and in the LSF summit the decision was taken to merge LIO.
> Has that
> decision changed since the summit?

The decision hasn't been taken to merge LIO, but based on what happened
at the summit, I think it's the most viable candidate and will likely be
merged by 2.6.37

> As a scst-user what I would like to understand is, what was that
> decision based on? Because the LSF summit was 'small by invitation'
> only summit. The notes don't give us an insight on the selection
> criteria/merits etc.

The notes list 3, what's unclear about it?

> 
> > 
> > > 3. I have heard you said "Vlad wasn't comfortable in
> > handing up the 
> > > control to the maintainers ... (this is how kernel.org
> > works)." I have 
> > > no idea what you meant. I have never been asked about
> > anything like 
> > > that, so I couldn't say anyhow that I'm not
> > comfortable with anything. 
> > > Could you clarify that?
> > > 
> 
> 3) above is something that I emailed Vlad and the scst community based
> on our offline conversation after the open panel. If SCST really has
> licensing issues then I will personally stop using SCST. Since Vlad
> hasn't
> expressed any concerns on the above and neither have you commented on
> it, is it safe to assume that the licensing requirement is a
> non-issue?

No.

James

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Chetan Loke]

Ok, pls ignore my first question:

I was trying to access https://lwn.net/Articles/399148/

But I just read:
http://lwn.net/Articles/400589/

Chetan

--- On Wed, 8/18/10, Chetan Loke <generationgnu@yahoo.com> wrote:

> From: Chetan Loke <generationgnu@yahoo.com>
> Subject: Re: [Scst-devel] Fwd: Re:  linuxcon 2010...
> To: "Vladislav Bolkhovitin" <vst@vlnb.net>, "James Bottomley" <James.Bottomley@suse.de>
> Cc: "scst-devel" <scst-devel@lists.sourceforge.net>, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org
> Date: Wednesday, August 18, 2010, 2:58 PM
> Hello James and others,
> 
> --- On Tue, 8/17/10, James Bottomley <James.Bottomley@suse.de>
> wrote:
> 
> > From: James Bottomley <James.Bottomley@suse.de>
> > Subject: Re: [Scst-devel] Fwd: Re:  linuxcon 2010...
> > To: "Vladislav Bolkhovitin" <vst@vlnb.net>
> > Cc: "scst-devel" <scst-devel@lists.sourceforge.net>,
> linux-kernel@vger.kernel.org,
> linux-scsi@vger.kernel.org
> > Date: Tuesday, August 17, 2010, 8:30 PM
> > On Mon, 2010-08-16 at 20:20 +0400,
> > Vladislav Bolkhovitin wrote:
> > > Hello James,
> > > 
> > > Could you comment rumors that decision about
> future
> > Linux SCSI target 
> > > subsystem is done as well as other related
> rumors:
> > 
> > If this is related to LSF, the notes on the I/O track
> are
> > here:
> > 
> > http://lwn.net/Articles/400491/
> 
> 
> During the open panel, my question was really specific - 
> 
> Q) What is the future of a SCSI-target subsystem in linux.
> Which 
>    target engine/subsystem can we expect?
> 
> Your answer) There is place for only 1 target-subsystem in
> the Linux scsi stack and in the LSF summit the decision was
> taken to merge LIO. Has that
> decision changed since the summit?
> 
> As a scst-user what I would like to understand is, what was
> that decision based on? Because the LSF summit was 'small by
> invitation' only summit. The notes don't give us an insight
> on the selection criteria/merits etc.
> 
> 
> > 
> > > 3. I have heard you said "Vlad wasn't comfortable
> in
> > handing up the 
> > > control to the maintainers ... (this is how
> kernel.org
> > works)." I have 
> > > no idea what you meant. I have never been asked
> about
> > anything like 
> > > that, so I couldn't say anyhow that I'm not
> > comfortable with anything. 
> > > Could you clarify that?
> > > 
> 
> 3) above is something that I emailed Vlad and the scst
> community based on our offline conversation after the open
> panel. If SCST really has licensing issues then I will
> personally stop using SCST. Since Vlad hasn't
> expressed any concerns on the above and neither have you
> commented on it, is it safe to assume that the licensing
> requirement is a non-issue?
> 
> 
> Chetan Loke
> 
> 
> 
>       
> 
> 
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by 
> 
> Make an app they can't live without
> Enter the BlackBerry Developer Challenge
> http://p.sf.net/sfu/RIM-dev2dev 
> _______________________________________________
> Scst-devel mailing list
> Scst-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scst-devel
> 


      

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Bart Van Assche]

On Wed, Aug 18, 2010 at 5:11 PM, James Bottomley
<James.Bottomley@suse.de> wrote:
>
> On Wed, 2010-08-18 at 07:58 -0700, Chetan Loke wrote:
> > [ ... ]
> >
> > During the open panel, my question was really specific -
> >
> > Q) What is the future of a SCSI-target subsystem in linux. Which
> >    target engine/subsystem can we expect?
> >
> > Your answer) There is place for only 1 target-subsystem in the Linux
> > scsi stack and in the LSF summit the decision was taken to merge LIO.
> > Has that
> > decision changed since the summit?
>
> The decision hasn't been taken to merge LIO, but based on what happened
> at the summit, I think it's the most viable candidate and will likely be
> merged by 2.6.37

(resending as plain text)

No matter which kernel-based storage target subsystem is merged, a
migration path should be provided for SCST users such that these can
continue using their SCST target drivers without too much trouble.
Otherwise you'll upset a large number of SCST users.

Note: an SCST target driver is the code that implements a storage
protocol. See also http://scst.sourceforge.net/scst_pg.html for a
overview diagram.

Bart.

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Bart Van Assche]

On Wed, Aug 18, 2010 at 5:11 PM, James Bottomley
<James.Bottomley@suse.de> wrote:
>
> On Wed, 2010-08-18 at 07:58 -0700, Chetan Loke wrote:
> > [ ... ]
> >
> > During the open panel, my question was really specific -
> >
> > Q) What is the future of a SCSI-target subsystem in linux. Which
> >    target engine/subsystem can we expect?
> >
> > Your answer) There is place for only 1 target-subsystem in the Linux
> > scsi stack and in the LSF summit the decision was taken to merge LIO.
> > Has that
> > decision changed since the summit?
>
> The decision hasn't been taken to merge LIO, but based on what happened
> at the summit, I think it's the most viable candidate and will likely be
> merged by 2.6.37

(resending as plain text)

No matter which kernel-based storage target subsystem is merged, a
migration path should be provided for SCST users such that these can
continue using their SCST target drivers without too much trouble.
Otherwise you'll upset a large number of SCST users.

Note: an SCST target driver is the code that implements a storage
protocol. See also http://scst.sourceforge.net/scst_pg.html for a
overview diagram.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Chetan Loke]

On Wed, Aug 18, 2010 at 11:11 AM, James Bottomley
<James.Bottomley@suse.de> wrote:
> The decision hasn't been taken to merge LIO, but based on what happened
> at the summit, I think it's the most viable candidate and will likely be
> merged by 2.6.37
>

During the open panel, facebook guys and others were tooting that
start-ups thrive because they can hack linux. Well there are quite a
few start-ups that use scst too for creating target appliances.
Has anyone even bothered to glance the scst mailing list to see if
that community is dead or alive?

I for one use scst to create synthetic work-loads and test 200+ VM
nodes in an ESX cluster. Anyone who has worked on a SAN OS will
appreciate the simplicity of SCST. And if folks still can't understand
the SCST code(after reading the README) then they are still welcome to
send an email on SCST. Would you like to make your FC stack go faster,
well please drop us an email on SCST and we will try our best to
further optimize the FC driver.

I know folks who don't understand simple DMA bus traces, FC wire
traces and yet they have the power to influence decisions.

James you are an expert but not everyone is. This is not a venting
session but even folks who are new to target architecture find it easy
to hack SCST.


>> As a scst-user what I would like to understand is, what was that
>> decision based on? Because the LSF summit was 'small by invitation'
>> only summit. The notes don't give us an insight on the selection
>> criteria/merits etc.
>
> The notes list 3, what's unclear about it?

Sorry, my bad. I sent an email earlier. I was trying to access a different link.


>> > > 3. I have heard you said "Vlad wasn't comfortable in
>> > handing up the
>> > > control to the maintainers ... (this is how kernel.org
>> > works)." I have
>> > > no idea what you meant. I have never been asked about
>> > anything like
>> > > that, so I couldn't say anyhow that I'm not
>> > comfortable with anything.
>> > > Could you clarify that?
>> > >
>>
>> 3) above is something that I emailed Vlad and the scst community based
>> on our offline conversation after the open panel. If SCST really has
>> licensing issues then I will personally stop using SCST. Since Vlad
>> hasn't
>> expressed any concerns on the above and neither have you commented on
>> it, is it safe to assume that the licensing requirement is a
>> non-issue?
>
> No.
>

huh? It's dual licensed. GPL and BSD(if I'm not wrong).


> James

--cloke

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[James Bottomley]

On Wed, 2010-08-18 at 12:04 -0400, Chetan Loke wrote:
> On Wed, Aug 18, 2010 at 11:11 AM, James Bottomley
> <James.Bottomley@suse.de> wrote:
> > The decision hasn't been taken to merge LIO, but based on what happened
> > at the summit, I think it's the most viable candidate and will likely be
> > merged by 2.6.37
> >
> 
> During the open panel, facebook guys and others were tooting that
> start-ups thrive because they can hack linux. Well there are quite a
> few start-ups that use scst too for creating target appliances.
> Has anyone even bothered to glance the scst mailing list to see if
> that community is dead or alive?
> 
> I for one use scst to create synthetic work-loads and test 200+ VM
> nodes in an ESX cluster. Anyone who has worked on a SAN OS will
> appreciate the simplicity of SCST. And if folks still can't understand
> the SCST code(after reading the README) then they are still welcome to
> send an email on SCST. Would you like to make your FC stack go faster,
> well please drop us an email on SCST and we will try our best to
> further optimize the FC driver.
> 
> I know folks who don't understand simple DMA bus traces, FC wire
> traces and yet they have the power to influence decisions.
> 
> James you are an expert but not everyone is. This is not a venting
> session but even folks who are new to target architecture find it easy
> to hack SCST.

But that's not really relevant, is it?  I would expect that whatever I
do, even keeping both out of tree, the communities around the solutions
would still exist and be vibrant, since they're out of tree now, nothing
will have changed.

James

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Bart Van Assche]

On Wed, Aug 18, 2010 at 5:11 PM, James Bottomley
<James.Bottomley@suse.de> wrote:
> On Wed, 2010-08-18 at 07:58 -0700, Chetan Loke wrote:
>> Hello James and others,
>>
>> --- On Tue, 8/17/10, James Bottomley <James.Bottomley@suse.de> wrote:
>>
>> > From: James Bottomley <James.Bottomley@suse.de>
>> > Subject: Re: [Scst-devel] Fwd: Re:  linuxcon 2010...
>> > To: "Vladislav Bolkhovitin" <vst@vlnb.net>
>> > Cc: "scst-devel" <scst-devel@lists.sourceforge.net>, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org
>> > Date: Tuesday, August 17, 2010, 8:30 PM
>> > On Mon, 2010-08-16 at 20:20 +0400,
>> > Vladislav Bolkhovitin wrote:
>> > > Hello James,
>> > >
>> > > Could you comment rumors that decision about future
>> > Linux SCSI target
>> > > subsystem is done as well as other related rumors:
>> >
>> > If this is related to LSF, the notes on the I/O track are
>> > here:
>> >
>> > http://lwn.net/Articles/400491/
>>
>>
>> During the open panel, my question was really specific -
>>
>> Q) What is the future of a SCSI-target subsystem in linux. Which
>>    target engine/subsystem can we expect?
>>
>> Your answer) There is place for only 1 target-subsystem in the Linux
>> scsi stack and in the LSF summit the decision was taken to merge LIO.
>> Has that
>> decision changed since the summit?
>
> The decision hasn't been taken to merge LIO, but based on what happened
> at the summit, I think it's the most viable candidate and will likely be
> merged by 2.6.37
>
>> As a scst-user what I would like to understand is, what was that
>> decision based on? Because the LSF summit was 'small by invitation'
>> only summit. The notes don't give us an insight on the selection
>> criteria/merits etc.
>
> The notes list 3, what's unclear about it?

Hello James,

Thanks for taking notes during the storage track and sharing these
notes (http://lwn.net/Articles/400589/). These notes are interesting
but do not reveal why LIO is preferred.

Also, the list with the three acceptance criteria is incomplete. A
very important criterion before any kernel code can be merged upstream
is whether or not there is a maintainer for that code. Someone who has
proven prior kernel coding experience and someone who understands the
new code thoroughly.

Bart.

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Bart Van Assche]

On Wed, Aug 18, 2010 at 5:11 PM, James Bottomley
<James.Bottomley@suse.de> wrote:
> On Wed, 2010-08-18 at 07:58 -0700, Chetan Loke wrote:
>> Hello James and others,
>>
>> --- On Tue, 8/17/10, James Bottomley <James.Bottomley@suse.de> wrote:
>>
>> > From: James Bottomley <James.Bottomley@suse.de>
>> > Subject: Re: [Scst-devel] Fwd: Re:  linuxcon 2010...
>> > To: "Vladislav Bolkhovitin" <vst@vlnb.net>
>> > Cc: "scst-devel" <scst-devel@lists.sourceforge.net>, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org
>> > Date: Tuesday, August 17, 2010, 8:30 PM
>> > On Mon, 2010-08-16 at 20:20 +0400,
>> > Vladislav Bolkhovitin wrote:
>> > > Hello James,
>> > >
>> > > Could you comment rumors that decision about future
>> > Linux SCSI target
>> > > subsystem is done as well as other related rumors:
>> >
>> > If this is related to LSF, the notes on the I/O track are
>> > here:
>> >
>> > http://lwn.net/Articles/400491/
>>
>>
>> During the open panel, my question was really specific -
>>
>> Q) What is the future of a SCSI-target subsystem in linux. Which
>>    target engine/subsystem can we expect?
>>
>> Your answer) There is place for only 1 target-subsystem in the Linux
>> scsi stack and in the LSF summit the decision was taken to merge LIO.
>> Has that
>> decision changed since the summit?
>
> The decision hasn't been taken to merge LIO, but based on what happened
> at the summit, I think it's the most viable candidate and will likely be
> merged by 2.6.37
>
>> As a scst-user what I would like to understand is, what was that
>> decision based on? Because the LSF summit was 'small by invitation'
>> only summit. The notes don't give us an insight on the selection
>> criteria/merits etc.
>
> The notes list 3, what's unclear about it?

Hello James,

Thanks for taking notes during the storage track and sharing these
notes (http://lwn.net/Articles/400589/). These notes are interesting
but do not reveal why LIO is preferred.

Also, the list with the three acceptance criteria is incomplete. A
very important criterion before any kernel code can be merged upstream
is whether or not there is a maintainer for that code. Someone who has
proven prior kernel coding experience and someone who understands the
new code thoroughly.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Joe Landman]

James Bottomley wrote:

>> During the open panel, my question was really specific - 
>>
>> Q) What is the future of a SCSI-target subsystem in linux. Which 
>>    target engine/subsystem can we expect?
>>
>> Your answer) There is place for only 1 target-subsystem in the Linux
>> scsi stack and in the LSF summit the decision was taken to merge LIO.
>> Has that
>> decision changed since the summit?
> 
> The decision hasn't been taken to merge LIO, but based on what happened
> at the summit, I think it's the most viable candidate and will likely be
> merged by 2.6.37

Quick question ... will LIO support SRP?  I should probably run over to 
their lists and ask, but a quick inspection of their site this morning 
shows that they are mostly iSCSI and FC focused.  (LIO folks, please 
feel free to step up and comment)

I'd certainly like to see a single framework, but not at the cost of 
losing important (to us) functionality.  We'd like to continue to use 
SRP, and iSCSI.  We'd like to use iSER (which is in the tgt stack) which 
LIO would give us when merged with tgt.  We are currently using SCST's 
iSCSI and SRP stack within our products.

I believe this also affects the OFED folks, as SRP is one of their 
services (based upon SCST).

Any guidance (in a general sense on the target side, not necessarily 
specific to LIO) on this would be appreciated.  SCST has been a good 
system for us to work with.  I'd hate to lose its functionality, and 
have us be forced to re-engineer some of our backend logic to work 
around the missing bits.  Thanks!

Regards,

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman@scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/jackrabbit
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

James Bottomley, on 08/18/2010 08:18 PM wrote:
>> James you are an expert but not everyone is. This is not a venting
>> session but even folks who are new to target architecture find it easy
>> to hack SCST.
>
> But that's not really relevant, is it?  I would expect that whatever I
> do, even keeping both out of tree, the communities around the solutions
> would still exist and be vibrant, since they're out of tree now, nothing
> will have changed.

The problem that people do believe that the merged code is the best 
code, so the being merged is an immediate HUGE advertisement. So, you 
can't say if a code is inside the mainline tree or not isn't relevant.

This is the source of the questions from the SCST community. SCST is at 
the moment the best code. It's several years ahead any competitor. SCST 
has more functionality, SCST has more users, SCST has bigger community, 
SCST has more testing and, hence, stability, SCST has more support from 
storage vendors, etc. But for some reasons all those suddenly ignored 
and a raw, pursuing SCST code preferred instead.

So, we want to know those reasons. SCST wasn't even really considered. 
Isn't Linux anymore a place where the best code wins? Frankly, it looks 
like the decision was done under closed doors based on rather political 
reasons, like personal connections...

Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Chetan Loke]

On Wed, Aug 18, 2010 at 12:18 PM, James Bottomley
<James.Bottomley@suse.de> wrote:
> On Wed, 2010-08-18 at 12:04 -0400, Chetan Loke wrote:
>> On Wed, Aug 18, 2010 at 11:11 AM, James Bottomley
>> <James.Bottomley@suse.de> wrote:
>> > The decision hasn't been taken to merge LIO, but based on what happened
>> > at the summit, I think it's the most viable candidate and will likely be
>> > merged by 2.6.37
>> >
>>
>> During the open panel, facebook guys and others were tooting that
>> start-ups thrive because they can hack linux. Well there are quite a
>> few start-ups that use scst too for creating target appliances.
>> Has anyone even bothered to glance the scst mailing list to see if
>> that community is dead or alive?
>>
>> I for one use scst to create synthetic work-loads and test 200+ VM
>> nodes in an ESX cluster. Anyone who has worked on a SAN OS will
>> appreciate the simplicity of SCST. And if folks still can't understand
>> the SCST code(after reading the README) then they are still welcome to
>> send an email on SCST. Would you like to make your FC stack go faster,
>> well please drop us an email on SCST and we will try our best to
>> further optimize the FC driver.
>>
>> I know folks who don't understand simple DMA bus traces, FC wire
>> traces and yet they have the power to influence decisions.
>>
>> James you are an expert but not everyone is. This is not a venting
>> session but even folks who are new to target architecture find it easy
>> to hack SCST.
>


> But that's not really relevant, is it?  I would expect that whatever I
> do, even keeping both out of tree, the communities around the solutions
> would still exist and be vibrant, since they're out of tree now, nothing
> will have changed.
>

Of course it's relevant. Not all engineers know about everything when
they venture in a new area. SCST overcomes that. It's not
intimidating. Infact, 3 months back, I asked a storport/miniport(aka
Windows) guy to take a look at scst and after studying the README he
was amazed to see how easy it was. Plus,everyone knows that if a
solution is inbox then it's just easier to maintain. Also no need to
patch/re-spin my iso. And you are missing the point - find me one good
technical conversation thread on LIO! What does that tell you? Why not
give a chance(or atleast consider?) to someone who has a wide/active
user base?

LIO never struggled to push their code upstream and it still is a
viable candidate? On the other hand scst users/developers are ready to
bend over to accommodate all the changes! What does that tell you?

James, how can a community remain vibrant when one solution is
favoured over another w/o any clear explanation and justification?
It's like the old saying - the lips are moving but all I hear is blah
blah blah. We look up to you but why should we accept the outcome of a
closed door LSF session ;)?


> James
-- cloke

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

Chetan Loke, on 08/18/2010 06:58 PM wrote:
> 3) above is something that I emailed Vlad and the scst community
> based on our offline conversation after the open panel. If SCST
> really has licensing issues then I will personally stop using SCST.
> Since Vlad hasn't expressed any concerns on the above and neither
> have you commented on it, is it safe to assume that the licensing
> requirement is a non-issue?

I believe there are no licensing issues with SCST. All the patches I 
have submitted and going to submit licensed under GPLv2.

Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

Joe Landman, on 08/18/2010 08:28 PM wrote:
> Quick question ... will LIO support SRP?  I should probably run over to
> their lists and ask, but a quick inspection of their site this morning
> shows that they are mostly iSCSI and FC focused.  (LIO folks, please
> feel free to step up and comment)
>
> I'd certainly like to see a single framework, but not at the cost of
> losing important (to us) functionality.  We'd like to continue to use
> SRP, and iSCSI.  We'd like to use iSER (which is in the tgt stack) which
> LIO would give us when merged with tgt.  We are currently using SCST's
> iSCSI and SRP stack within our products.

Joe,

For iSER LIO would not give you anything SCST can't give you. With LIO 
you would use the iSER target via STGT pass-through sg/bsg. You can the 
same way use it with SCST scst_local module.

Vlad

RE: [Scst-devel] Fwd: Re: linuxcon 2010...

[jack wang]

James Bottomley, on 08/18/2010 08:18 PM wrote:
>> James you are an expert but not everyone is. This is not a venting
>> session but even folks who are new to target architecture find it easy
>> to hack SCST.
>
> But that's not really relevant, is it?  I would expect that whatever I
> do, even keeping both out of tree, the communities around the solutions
> would still exist and be vibrant, since they're out of tree now, nothing
> will have changed.

The problem that people do believe that the merged code is the best 
code, so the being merged is an immediate HUGE advertisement. So, you 
can't say if a code is inside the mainline tree or not isn't relevant.

This is the source of the questions from the SCST community. SCST is at 
the moment the best code. It's several years ahead any competitor. SCST 
has more functionality, SCST has more users, SCST has bigger community, 
SCST has more testing and, hence, stability, SCST has more support from 
storage vendors, etc. But for some reasons all those suddenly ignored 
and a raw, pursuing SCST code preferred instead.

So, we want to know those reasons. SCST wasn't even really considered. 
Isn't Linux anymore a place where the best code wins? Frankly, it looks 
like the decision was done under closed doors based on rather political 
reasons, like personal connections...

Vlad

[Jack]Vote for SCST as a user and target driver developer based on SCST.
SCST really do good job. 
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [Scst-devel] Fwd: Re: linuxcon 2010...

[jack wang]

James Bottomley, on 08/18/2010 08:18 PM wrote:
>> James you are an expert but not everyone is. This is not a venting
>> session but even folks who are new to target architecture find it easy
>> to hack SCST.
>
> But that's not really relevant, is it?  I would expect that whatever I
> do, even keeping both out of tree, the communities around the solutions
> would still exist and be vibrant, since they're out of tree now, nothing
> will have changed.

The problem that people do believe that the merged code is the best 
code, so the being merged is an immediate HUGE advertisement. So, you 
can't say if a code is inside the mainline tree or not isn't relevant.

This is the source of the questions from the SCST community. SCST is at 
the moment the best code. It's several years ahead any competitor. SCST 
has more functionality, SCST has more users, SCST has bigger community, 
SCST has more testing and, hence, stability, SCST has more support from 
storage vendors, etc. But for some reasons all those suddenly ignored 
and a raw, pursuing SCST code preferred instead.

So, we want to know those reasons. SCST wasn't even really considered. 
Isn't Linux anymore a place where the best code wins? Frankly, it looks 
like the decision was done under closed doors based on rather political 
reasons, like personal connections...

Vlad

[Jack]Vote for SCST as a user and target driver developer based on SCST.
SCST really do good job. 
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Dirk Meister]

Am 19.08.2010 um 03:18 schrieb jack wang:

>
> James Bottomley, on 08/18/2010 08:18 PM wrote:
>>> James you are an expert but not everyone is. This is not a venting
>>> session but even folks who are new to target architecture find it  
>>> easy
>>> to hack SCST.
>>
>> But that's not really relevant, is it?  I would expect that  
>> whatever I
>> do, even keeping both out of tree, the communities around the  
>> solutions
>> would still exist and be vibrant, since they're out of tree now,  
>> nothing
>> will have changed.
>
> The problem that people do believe that the merged code is the best
> code, so the being merged is an immediate HUGE advertisement. So, you
> can't say if a code is inside the mainline tree or not isn't relevant.
>
> This is the source of the questions from the SCST community. SCST is  
> at
> the moment the best code. It's several years ahead any competitor.  
> SCST
> has more functionality, SCST has more users, SCST has bigger  
> community,
> SCST has more testing and, hence, stability, SCST has more support  
> from
> storage vendors, etc. But for some reasons all those suddenly ignored
> and a raw, pursuing SCST code preferred instead.
>
> So, we want to know those reasons. SCST wasn't even really considered.
> Isn't Linux anymore a place where the best code wins? Frankly, it  
> looks
> like the decision was done under closed doors based on rather  
> political
> reasons, like personal connections...
>
> Vlad
>
> [Jack]Vote for SCST as a user and target driver developer based on  
> SCST.
> SCST really do good job.
[dmeister] Vote for SCST. I am working with SCST to develop an user- 
space target backend using the scst_user feature.
The functionality, the stability of SCST, and the wide range of target  
drivers for SCST are really nice.

> --
> To unsubscribe from this list: send the line "unsubscribe linux- 
> scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux- 
> scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Nicholas A. Bellinger]

On Thu, 2010-08-19 at 23:20 +0200, Dirk Meister wrote:
> Am 19.08.2010 um 03:18 schrieb jack wang:
> > [Jack]Vote for SCST as a user and target driver developer based on  
> > SCST.
> > SCST really do good job.
> [dmeister] Vote for SCST. I am working with SCST to develop an user- 
> space target backend using the scst_user feature.
> The functionality, the stability of SCST, and the wide range of target  
> drivers for SCST are really nice.
> 

Just FYI, as myself and other folks do have a need for a proper target
mode kernel fabric -> userspace LUN passthrough, this will be addressed
by building upon Tomo-san and mnc's already existing mainline code in
drivers/scsi/scsi_tgt_*.c

Pretty much everything already exists there for doing a userspace
target, which is then registered as a seperate TCM/STGT subsystem plugin
that can be configfs symlinked into fabric module SCSI ports at will
with full SPC-3 PR and ALUA support for all supported TCM fabric
modules.  The complete discussion for that is mentioned in this thread:
http://lkml.org/lkml/2010/5/27/672  Here is the 'meat' of the
discussion, which is mentioned in the context of the '2nd phase of STGT
<-> LIO compatibility':


"We will be extending the scsi_tgt_[lib,if].c mapped ring interface to
allow TCM to access userspace backstores transparently with existing
kernel level TCM fabric modules, and using the generic configfs fabric
module infrastructure in target_core_fabric_configfs.c for the port and
I_T nexus control plane just as you would with any TCM backstore
subsystem today.

Again, in open source you have to build upon what already exists and
move forward.  The original STGT kernel <-> userspace ring abstraction
and logic in drivers/scsi/scsi_tgt_lib.c:scsi_tgt_queue_command() ->
scsi_tgt_uspace_send_cmd() is already going to do the vast majority of
what is required for handling fabric I/O processing and I_T Nexus and
Port management in kernel space with a userspace backstore.  It is
really just a matter of allowing the STGT ring request to optionally be
sent out to userspace as a standalone LUN instead of as a target port."


Note this code was left out of the last postings on linux-scsi, and will
most likely be left out initially for v2.6.37.  I do plan to have
something up and running on lio-core-2.6.git for one of my QEMU-KVM
projects by then however.  Here is how the code currently looks:

http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=blob;f=drivers/target/target_core_stgt.c;hb=refs/heads/lio-4.0

Best,

--nab

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Ruben Laban]

On Thursday 19 August 2010 at 03:18 (CET), jack wang wrote:
> [Jack]Vote for SCST as a user and target driver developer based on SCST.
> SCST really do good job. 

+1 from a happy SCST end-user.

Regards,

Ruben Laban
Senior Network and Systems Administrator
ISM eCompany

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

Nicholas A. Bellinger, on 08/20/2010 02:29 AM wrote:
> "We will be extending the scsi_tgt_[lib,if].c mapped ring interface to
> allow TCM to access userspace backstores transparently with existing
> kernel level TCM fabric modules, and using the generic configfs fabric
> module infrastructure in target_core_fabric_configfs.c for the port and
> I_T nexus control plane just as you would with any TCM backstore
> subsystem today.
>
> Again, in open source you have to build upon what already exists and
> move forward.  The original STGT kernel<->  userspace ring abstraction
> and logic in drivers/scsi/scsi_tgt_lib.c:scsi_tgt_queue_command() ->
> scsi_tgt_uspace_send_cmd() is already going to do the vast majority of
> what is required for handling fabric I/O processing and I_T Nexus and
> Port management in kernel space with a userspace backstore.  It is
> really just a matter of allowing the STGT ring request to optionally be
> sent out to userspace as a standalone LUN instead of as a target port."

You forgot to mention one small thing. You are going to (re)use current 
STGT interface for user space backend, which in 5 years of being 
mainline has not gained any noticeable interest, because it is 
fundamentally slow. It needs a complete redesign to become fast. In 
contrast, interface provided by scst_user has just 3% overhead comparing 
with fully in-kernel backend and allows to build storage capable of 
handling 1,500,000 IOPSes (Kaminario).

Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Nicholas A. Bellinger]

On Sat, 2010-08-21 at 22:42 +0400, Vladislav Bolkhovitin wrote:
> Nicholas A. Bellinger, on 08/20/2010 02:29 AM wrote:
> > "We will be extending the scsi_tgt_[lib,if].c mapped ring interface to
> > allow TCM to access userspace backstores transparently with existing
> > kernel level TCM fabric modules, and using the generic configfs fabric
> > module infrastructure in target_core_fabric_configfs.c for the port and
> > I_T nexus control plane just as you would with any TCM backstore
> > subsystem today.
> >
> > Again, in open source you have to build upon what already exists and
> > move forward.  The original STGT kernel<->  userspace ring abstraction
> > and logic in drivers/scsi/scsi_tgt_lib.c:scsi_tgt_queue_command() ->
> > scsi_tgt_uspace_send_cmd() is already going to do the vast majority of
> > what is required for handling fabric I/O processing and I_T Nexus and
> > Port management in kernel space with a userspace backstore.  It is
> > really just a matter of allowing the STGT ring request to optionally be
> > sent out to userspace as a standalone LUN instead of as a target port."
> 
> You forgot to mention one small thing. You are going to (re)use current 
> STGT interface for user space backend, which in 5 years of being 
> mainline has not gained any noticeable interest, because it is 
> fundamentally slow.

First, 'build upon' and blind '(re)use' are different approaches.
Building on is an important requirement for working with any existing
mainline code regardless of how much much attention the code itself may
require.  Initially re-using pieces of the mainline code is acceptable
to get a prototype up and running, and since we don't have many users
for this piece of STGT, it will easier to make larger changes (eg: move
beyond simple re-use) without breaking a large legacy base.

This is what I actually prefer moving forward, as it gives us more
flexibility for the future.

>  It needs a complete redesign to become fast.

Secondly, not being the original author of drivers/scsi/scsi_tgt_*, I am
happy to do the work and submit the necessary patches to achieve the
desired goal.  Also, considering now we have the TCM v4 HBA/DEV
abstraction with a fabric independent control path in
target_core_fabric_configfs.c, there suddenly new interest to build upon
tomo's and mnc'c original STGT work in drivers/scsi/scsi_tgt_*.c

Using struct scsi_cmnd allocation from a specially allocated struct
request_queue still my preferred method for doing this, unless tomo and
mnc state otherwise.   Also if we need to change the request_queue from
being per struct Scsi_Host to struct scsi_device that is one thing that
will be fine.  If we need to make more drastic changes to
drivers/scsi/scsi_tgt_if.c that is also fine.

However saying that it needs a 'complete redesign', especially without
ever consulting or offering to creat patches with STGT guys is not how
mainline development functions.

>  In 
> contrast, interface provided by scst_user has just 3% overhead comparing 
> with fully in-kernel backend and allows to build storage capable of 
> handling 1,500,000 IOPSes (Kaminario).
> 

Great!  Please understand that you are more than welcome to create your
own TCM subsystem plugin for your SCST kernel <-> user passthrough
interface so it can take advantage of all the drivers/target/ code has
to offer.   Also, now that target_core_iblock.c, target_core_pscsi.c,
target_core_file.c, and target_core_stgt.c are seperate standalone LKMs
loaded on-demand by TCM/ConfigFS, creating a new TCM struct
se_subsystem_api module will be even easier for you now.

I would even be happy to send a functioning struct se_subsystem_api
skeleton for your efforts once the tcm_mod_builder.py script I am
currently working on for producing unlimited ready-to-run configfs
skeletons for new TCM fabric modules is ready to be pushed.

Best,

--nab

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[James Bottomley]

On Sat, 2010-08-21 at 22:42 +0400, Vladislav Bolkhovitin wrote:
> Nicholas A. Bellinger, on 08/20/2010 02:29 AM wrote:
> > "We will be extending the scsi_tgt_[lib,if].c mapped ring interface to
> > allow TCM to access userspace backstores transparently with existing
> > kernel level TCM fabric modules, and using the generic configfs fabric
> > module infrastructure in target_core_fabric_configfs.c for the port and
> > I_T nexus control plane just as you would with any TCM backstore
> > subsystem today.
> >
> > Again, in open source you have to build upon what already exists and
> > move forward.  The original STGT kernel<->  userspace ring abstraction
> > and logic in drivers/scsi/scsi_tgt_lib.c:scsi_tgt_queue_command() ->
> > scsi_tgt_uspace_send_cmd() is already going to do the vast majority of
> > what is required for handling fabric I/O processing and I_T Nexus and
> > Port management in kernel space with a userspace backstore.  It is
> > really just a matter of allowing the STGT ring request to optionally be
> > sent out to userspace as a standalone LUN instead of as a target port."
> 
> You forgot to mention one small thing. You are going to (re)use current 
> STGT interface for user space backend, which in 5 years of being 
> mainline has not gained any noticeable interest, because it is 
> fundamentally slow.

That's not exactly what the results of a speed comparison one of your
people did said, now is it?  The results were actually not much
difference on line speeds up to GigE.

Interface re-use (or at least ABI compatibility) is the whole point,
it's what makes the solution a drop in replacement.

>  It needs a complete redesign to become fast. In 
> contrast, interface provided by scst_user has just 3% overhead comparing 
> with fully in-kernel backend and allows to build storage capable of 
> handling 1,500,000 IOPSes (Kaminario).

For a replacement, first we get the current userspace code working as
is, then you can propose modifications to make it go faster.

James

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Bart Van Assche]

On Sat, Aug 21, 2010 at 10:43 PM, James Bottomley
<James.Bottomley@suse.de> wrote:
> On Sat, 2010-08-21 at 22:42 +0400, Vladislav Bolkhovitin wrote:
>> [ ... ]
>>
>> You forgot to mention one small thing. You are going to (re)use current
>> STGT interface for user space backend, which in 5 years of being
>> mainline has not gained any noticeable interest, because it is
>> fundamentally slow.
>
> That's not exactly what the results of a speed comparison one of your
> people did said, now is it?  The results were actually not much
> difference on line speeds up to GigE.
>
> [ ... ]

I assume that you are referring to this message:
http://lkml.org/lkml/2008/1/29/387 ? You might have missed the reply
that was posted by Roland Dreier, a highly regarded kernel maintainer
and InfiniBand expert (http://lkml.org/lkml/2008/1/29/402):

"Maybe I'm all wet, but I think iSER vs. SRP should be roughly
comparable.  The exact formatting of various messages etc. is
different but the data path using RDMA is pretty much identical.  So
the big difference between STGT iSER and SCST SRP hints at some big
difference in the efficiency of the two implementations."

Furthermore, I would like to clarify that Vlad hasn't asked me to
start working on the SCST project but that I selected the SCST project
myself after an extensive stability and performance comparison of four
existing open source storage target projects.

Bart.

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[James Bottomley]

On Sun, 2010-08-22 at 09:39 +0200, Bart Van Assche wrote:
> On Sat, Aug 21, 2010 at 10:43 PM, James Bottomley
> <James.Bottomley@suse.de> wrote:
> > On Sat, 2010-08-21 at 22:42 +0400, Vladislav Bolkhovitin wrote:
> >> [ ... ]
> >>
> >> You forgot to mention one small thing. You are going to (re)use current
> >> STGT interface for user space backend, which in 5 years of being
> >> mainline has not gained any noticeable interest, because it is
> >> fundamentally slow.
> >
> > That's not exactly what the results of a speed comparison one of your
> > people did said, now is it?  The results were actually not much
> > difference on line speeds up to GigE.
> >
> > [ ... ]
> 
> I assume that you are referring to this message:
> http://lkml.org/lkml/2008/1/29/387 ? You might have missed the reply
> that was posted by Roland Dreier, a highly regarded kernel maintainer
> and InfiniBand expert (http://lkml.org/lkml/2008/1/29/402):
> 
> "Maybe I'm all wet, but I think iSER vs. SRP should be roughly
> comparable.  The exact formatting of various messages etc. is
> different but the data path using RDMA is pretty much identical.  So
> the big difference between STGT iSER and SCST SRP hints at some big
> difference in the efficiency of the two implementations."

So the phrase "up to GigE" was deliberately in the above to exclude the
disputed infiniband results.  I'm not really interested in re-opening
the arguments over how to interpret those results.  The fact that SCST
and STGT were on par up to 1GbE is enough to refute the contention that
STGT is "fundamentally slow".

James 

> Furthermore, I would like to clarify that Vlad hasn't asked me to
> start working on the SCST project but that I selected the SCST project
> myself after an extensive stability and performance comparison of four
> existing open source storage target projects.
> 
> Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Joe Landman]

James Bottomley wrote:

> So the phrase "up to GigE" was deliberately in the above to exclude the
> disputed infiniband results.  I'm not really interested in re-opening
> the arguments over how to interpret those results.  The fact that SCST
> and STGT were on par up to 1GbE is enough to refute the contention that
> STGT is "fundamentally slow".

We haven't tested this in at least 6 months, but we did test iSCSI over 
10GbE using SCST and STGT.  This was one of the reasons we wound up 
going with SCST.  For the past several years, our performance on SCST 
was much higher than on STGT.

If it helps, we can redo these tests with a modern kernel (2.6.35.x) and 
same backend/frontend.  We've been switching most of our IO performance 
testing to Jens Axboe's excellent fio tool.  I'd be happy to share our 
testing decks with anyone (and collaborate on generating a good useful 
set for general use)

This said, I have to add in my support to SCST and its developers. 
Vlad, Bart, and the rest of the community have been very helpful to us. 
  We have been a consumer rather than a developer to date, so we have 
less of a dog in this fight than others.  We have nothing against LIO or 
STGT.  We will try them and see if they meet our needs.  SRP is a hard 
requirement, and it exists and works great in SCST.  So for us, a 
solution which gives us the best of all worlds would be one where we 
don't have to give up what works well for us, and gets us new 
capabilities/features.  This is more of a wish-list ... we have to keep 
using what works well for us.

Our interest in STGT is that we would like a working iSER.  Vlad has 
suggested a method that we will explore a little later on (next
month).  The LIO folks do look like they have some interesting 
capabilities beyond this, so we will look.  But without SRP, and a fast 
iSCSI, their really isn't much choice for us in the near term.

Regards,

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman@scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/jackrabbit
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Bart Van Assche]

On Mon, Aug 23, 2010 at 3:47 PM, Joe Landman
<landman@scalableinformatics.com> wrote:
>
> James Bottomley wrote:
>
>> So the phrase "up to GigE" was deliberately in the above to exclude the
>> disputed infiniband results.  I'm not really interested in re-opening
>> the arguments over how to interpret those results.  The fact that SCST
>> and STGT were on par up to 1GbE is enough to refute the contention that
>> STGT is "fundamentally slow".
>
> We haven't tested this in at least 6 months, but we did test iSCSI over 10GbE using SCST and STGT.  This was one of the reasons we wound up going with SCST.  For the past several years, our performance on SCST was much higher than on STGT.
>
> If it helps, we can redo these tests with a modern kernel (2.6.35.x) and same backend/frontend.  We've been switching most of our IO performance testing to Jens Axboe's excellent fio tool.  I'd be happy to share our testing decks with anyone (and collaborate on generating a good useful set for general use)
>
> This said, I have to add in my support to SCST and its developers. Vlad, Bart, and the rest of the community have been very helpful to us.  We have been a consumer rather than a developer to date, so we have less of a dog in this fight than others.  We have nothing against LIO or STGT.  We will try them and see if they meet our needs.  SRP is a hard requirement, and it exists and works great in SCST.  So for us, a solution which gives us the best of all worlds would be one where we don't have to give up what works well for us, and gets us new capabilities/features.  This is more of a wish-list ... we have to keep using what works well for us.
>
> Our interest in STGT is that we would like a working iSER.  Vlad has suggested a method that we will explore a little later on (next
> month).  The LIO folks do look like they have some interesting capabilities beyond this, so we will look.  But without SRP, and a fast iSCSI, their really isn't much choice for us in the near term.

(resending as plain text)

There is an important design difference between SCST and LIO: SCST by
defaults creates multiple threads to process the I/O operations for a
storage target, while LIO only creates a single thread per storage
target. This makes SCST perform measurably faster.

Bart.

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Bart Van Assche]

On Mon, Aug 23, 2010 at 3:47 PM, Joe Landman
<landman@scalableinformatics.com> wrote:
>
> James Bottomley wrote:
>
>> So the phrase "up to GigE" was deliberately in the above to exclude the
>> disputed infiniband results.  I'm not really interested in re-opening
>> the arguments over how to interpret those results.  The fact that SCST
>> and STGT were on par up to 1GbE is enough to refute the contention that
>> STGT is "fundamentally slow".
>
> We haven't tested this in at least 6 months, but we did test iSCSI over 10GbE using SCST and STGT.  This was one of the reasons we wound up going with SCST.  For the past several years, our performance on SCST was much higher than on STGT.
>
> If it helps, we can redo these tests with a modern kernel (2.6.35.x) and same backend/frontend.  We've been switching most of our IO performance testing to Jens Axboe's excellent fio tool.  I'd be happy to share our testing decks with anyone (and collaborate on generating a good useful set for general use)
>
> This said, I have to add in my support to SCST and its developers. Vlad, Bart, and the rest of the community have been very helpful to us.  We have been a consumer rather than a developer to date, so we have less of a dog in this fight than others.  We have nothing against LIO or STGT.  We will try them and see if they meet our needs.  SRP is a hard requirement, and it exists and works great in SCST.  So for us, a solution which gives us the best of all worlds would be one where we don't have to give up what works well for us, and gets us new capabilities/features.  This is more of a wish-list ... we have to keep using what works well for us.
>
> Our interest in STGT is that we would like a working iSER.  Vlad has suggested a method that we will explore a little later on (next
> month).  The LIO folks do look like they have some interesting capabilities beyond this, so we will look.  But without SRP, and a fast iSCSI, their really isn't much choice for us in the near term.

(resending as plain text)

There is an important design difference between SCST and LIO: SCST by
defaults creates multiple threads to process the I/O operations for a
storage target, while LIO only creates a single thread per storage
target. This makes SCST perform measurably faster.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Chetan Loke]

On Mon, Aug 23, 2010 at 11:11 AM, Bart Van Assche <bvanassche@acm.org> wrote:

>
> There is an important design difference between SCST and LIO: SCST by
> defaults creates multiple threads to process the I/O operations for a
> storage target, while LIO only creates a single thread per storage target.
> This makes SCST perform measurably faster.
>

Forget that. You could have discussed this if there were code reviews
or other mainline inclusion emails from James B. From what I have
heard, the decision was taken around 8-9 months back.
Would anyone like to either comment/validate/refute this please?  If
not then I would kindly request these guys to stop taking us for a
test drive. And also I'm not sure when was the last time James B.
bench-marked our scsi-stack. Even if I ACK in the xmit-path then I
can't push more than 100K IOPs. But other folks have re-engineered our
linux-scsi stack and from what I've heard they can push > 300K+ IOPs.
So I would just ignore performance discussion because I don't think
folks have done even simple lame experiments in the last 1 year. Or
may be I'm completely wrong and so please enlighten me so that I can
re-run the tests.


> Bart.
>
Chetan Loke

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Chetan Loke]

I actually received 3+ off-post emails asking whether I was talking
about initiator or target in the 100K IOPS case below and what did I
mean by the ACKs.
I was referring to the 'Initiator' side.
ACKs == When scsi-ML down-calls the LLD via the queue-command, process
the sgl's(if you like) and then trigger the scsi_done up-call path.

Chetan Loke

On Mon, Aug 23, 2010 at 12:07 PM, Chetan Loke <chetanloke@gmail.com> wrote:
> On Mon, Aug 23, 2010 at 11:11 AM, Bart Van Assche <bvanassche@acm.org> wrote:
>
>>
>> There is an important design difference between SCST and LIO: SCST by
>> defaults creates multiple threads to process the I/O operations for a
>> storage target, while LIO only creates a single thread per storage target.
>> This makes SCST perform measurably faster.
>>
>
> Forget that. You could have discussed this if there were code reviews
> or other mainline inclusion emails from James B. From what I have
> heard, the decision was taken around 8-9 months back.
> Would anyone like to either comment/validate/refute this please?  If
> not then I would kindly request these guys to stop taking us for a
> test drive. And also I'm not sure when was the last time James B.
> bench-marked our scsi-stack. Even if I ACK in the xmit-path then I
> can't push more than 100K IOPs. But other folks have re-engineered our
> linux-scsi stack and from what I've heard they can push > 300K+ IOPs.
> So I would just ignore performance discussion because I don't think
> folks have done even simple lame experiments in the last 1 year. Or
> may be I'm completely wrong and so please enlighten me so that I can
> re-run the tests.
>
>
>> Bart.
>>
> Chetan Loke
>

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Chetan Loke]

I actually received 3+ off-post emails asking whether I was talking
about initiator or target in the 100K IOPS case below and what did I
mean by the ACKs.
I was referring to the 'Initiator' side.
ACKs == When scsi-ML down-calls the LLD via the queue-command, process
the sgl's(if you like) and then trigger the scsi_done up-call path.

Chetan Loke

On Mon, Aug 23, 2010 at 12:07 PM, Chetan Loke <chetanloke@gmail.com> wrote:
> On Mon, Aug 23, 2010 at 11:11 AM, Bart Van Assche <bvanassche@acm.org> wrote:
>
>>
>> There is an important design difference between SCST and LIO: SCST by
>> defaults creates multiple threads to process the I/O operations for a
>> storage target, while LIO only creates a single thread per storage target.
>> This makes SCST perform measurably faster.
>>
>
> Forget that. You could have discussed this if there were code reviews
> or other mainline inclusion emails from James B. From what I have
> heard, the decision was taken around 8-9 months back.
> Would anyone like to either comment/validate/refute this please?  If
> not then I would kindly request these guys to stop taking us for a
> test drive. And also I'm not sure when was the last time James B.
> bench-marked our scsi-stack. Even if I ACK in the xmit-path then I
> can't push more than 100K IOPs. But other folks have re-engineered our
> linux-scsi stack and from what I've heard they can push > 300K+ IOPs.
> So I would just ignore performance discussion because I don't think
> folks have done even simple lame experiments in the last 1 year. Or
> may be I'm completely wrong and so please enlighten me so that I can
> re-run the tests.
>
>
>> Bart.
>>
> Chetan Loke
>
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

James Bottomley, on 08/23/2010 12:29 AM wrote:
> So the phrase "up to GigE" was deliberately in the above to exclude the
> disputed infiniband results.  I'm not really interested in re-opening
> the arguments over how to interpret those results.  The fact that SCST
> and STGT were on par up to 1GbE is enough to refute the contention that
> STGT is "fundamentally slow".

Well, James, why not 100MbE? If you want a comparison of target 
implementations you need a fast hardware with minimal latency. 
Otherwise, the difference between the implementations can drown in the 
overhead of the accompanying processing. 1GbE is a nearly 10 years ago 
interface. Or are we going to stay ten years behind progress?

Thanks,
Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Pasi Kärkkäinen]

On Mon, Aug 23, 2010 at 02:03:26PM -0400, Chetan Loke wrote:
> I actually received 3+ off-post emails asking whether I was talking
> about initiator or target in the 100K IOPS case below and what did I
> mean by the ACKs.
> I was referring to the 'Initiator' side.
> ACKs == When scsi-ML down-calls the LLD via the queue-command, process
> the sgl's(if you like) and then trigger the scsi_done up-call path.
> 

Uhm, Intel and Microsoft demonstrated over 1 million IOPS
using software iSCSI and a single 10 Gbit Ethernet NIC (Intel 82599).

How come there is such a huge difference? What are we lacking in Linux? 

-- Pasi

> Chetan Loke
> 
> On Mon, Aug 23, 2010 at 12:07 PM, Chetan Loke <chetanloke@gmail.com> wrote:
> > On Mon, Aug 23, 2010 at 11:11 AM, Bart Van Assche <bvanassche@acm.org> wrote:
> >
> >>
> >> There is an important design difference between SCST and LIO: SCST by
> >> defaults creates multiple threads to process the I/O operations for a
> >> storage target, while LIO only creates a single thread per storage target.
> >> This makes SCST perform measurably faster.
> >>
> >
> > Forget that. You could have discussed this if there were code reviews
> > or other mainline inclusion emails from James B. From what I have
> > heard, the decision was taken around 8-9 months back.
> > Would anyone like to either comment/validate/refute this please?  If
> > not then I would kindly request these guys to stop taking us for a
> > test drive. And also I'm not sure when was the last time James B.
> > bench-marked our scsi-stack. Even if I ACK in the xmit-path then I
> > can't push more than 100K IOPs. But other folks have re-engineered our
> > linux-scsi stack and from what I've heard they can push > 300K+ IOPs.
> > So I would just ignore performance discussion because I don't think
> > folks have done even simple lame experiments in the last 1 year. Or
> > may be I'm completely wrong and so please enlighten me so that I can
> > re-run the tests.
> >
> >
> >> Bart.
> >>
> > Chetan Loke
> >
> 
> ------------------------------------------------------------------------------
> Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
> Be part of this innovative community and reach millions of netbook users 
> worldwide. Take advantage of special opportunities to increase revenue and 
> speed time-to-market. Join now, and jumpstart your future.
> http://p.sf.net/sfu/intel-atom-d2d
> _______________________________________________
> Scst-devel mailing list
> Scst-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scst-devel

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Pasi Kärkkäinen]

On Mon, Aug 23, 2010 at 02:03:26PM -0400, Chetan Loke wrote:
> I actually received 3+ off-post emails asking whether I was talking
> about initiator or target in the 100K IOPS case below and what did I
> mean by the ACKs.
> I was referring to the 'Initiator' side.
> ACKs == When scsi-ML down-calls the LLD via the queue-command, process
> the sgl's(if you like) and then trigger the scsi_done up-call path.
> 

Uhm, Intel and Microsoft demonstrated over 1 million IOPS
using software iSCSI and a single 10 Gbit Ethernet NIC (Intel 82599).

How come there is such a huge difference? What are we lacking in Linux? 

-- Pasi

> Chetan Loke
> 
> On Mon, Aug 23, 2010 at 12:07 PM, Chetan Loke <chetanloke@gmail.com> wrote:
> > On Mon, Aug 23, 2010 at 11:11 AM, Bart Van Assche <bvanassche@acm.org> wrote:
> >
> >>
> >> There is an important design difference between SCST and LIO: SCST by
> >> defaults creates multiple threads to process the I/O operations for a
> >> storage target, while LIO only creates a single thread per storage target.
> >> This makes SCST perform measurably faster.
> >>
> >
> > Forget that. You could have discussed this if there were code reviews
> > or other mainline inclusion emails from James B. From what I have
> > heard, the decision was taken around 8-9 months back.
> > Would anyone like to either comment/validate/refute this please?  If
> > not then I would kindly request these guys to stop taking us for a
> > test drive. And also I'm not sure when was the last time James B.
> > bench-marked our scsi-stack. Even if I ACK in the xmit-path then I
> > can't push more than 100K IOPs. But other folks have re-engineered our
> > linux-scsi stack and from what I've heard they can push > 300K+ IOPs.
> > So I would just ignore performance discussion because I don't think
> > folks have done even simple lame experiments in the last 1 year. Or
> > may be I'm completely wrong and so please enlighten me so that I can
> > re-run the tests.
> >
> >
> >> Bart.
> >>
> > Chetan Loke
> >
> 
> ------------------------------------------------------------------------------
> Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
> Be part of this innovative community and reach millions of netbook users 
> worldwide. Take advantage of special opportunities to increase revenue and 
> speed time-to-market. Join now, and jumpstart your future.
> http://p.sf.net/sfu/intel-atom-d2d
> _______________________________________________
> Scst-devel mailing list
> Scst-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scst-devel
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

James Bottomley, on 08/22/2010 12:43 AM wrote:
> Interface re-use (or at least ABI compatibility) is the whole point,
> it's what makes the solution a drop in replacement.

I see now. You want ABI compatibility to keep the "contract" that no 
kernel changes can break applications binary compatibility for unlimited 
time.

OK, we will write the compatibility module. It shouldn't take much time.

But before we start, I'd like to clear 2 related questions:

1. How far the ABI compatibility "contract" goes? Are there cases, where 
it isn't so strong? I'm asking, because I can recall that open-iscsi at 
least once has broken ABI compatibility with user space tools. Was it an 
accidental (but not corrected) mistake or was it deliberate? If the 
latter, then, I guess, there must be some exceptions defining when ABI 
compatibility can be not followed.

2. Currently STGT in the kernel is just 2 files, scsi_tgt_if.c and
scsi_tgt_lib.c (with headers), + ibmvstgt driver. The C files define the 
STGT interface in question. So, if we keep ABI compatibility with the 
new target engine, we would have to keep those 2 files included in the 
kernel, which would effectively mean that STGT would stay in the kernel. 
This would lead to the situation you are trying to avoid: 2 SCSI target 
infrastructures in the kernel. Would it be OK?

Thanks for (eventually!) defining clear rules and removing confusions!

Vlad

Re: Linux I/O subsystem performance (was: linuxcon 2010...)

[Vladislav Bolkhovitin]

Pasi Kärkkäinen, on 08/24/2010 11:25 AM wrote:
> On Mon, Aug 23, 2010 at 02:03:26PM -0400, Chetan Loke wrote:
>> I actually received 3+ off-post emails asking whether I was talking
>> about initiator or target in the 100K IOPS case below and what did I
>> mean by the ACKs.
>> I was referring to the 'Initiator' side.
>> ACKs == When scsi-ML down-calls the LLD via the queue-command, process
>> the sgl's(if you like) and then trigger the scsi_done up-call path.
>>
>
> Uhm, Intel and Microsoft demonstrated over 1 million IOPS
> using software iSCSI and a single 10 Gbit Ethernet NIC (Intel 82599).
>
> How come there is such a huge difference? What are we lacking in Linux?

I also have an impression that Linux I/O subsystem has some performance 
problems. For instance, in one recent SCST performance test only 8 Linux 
initiators with fio as a load generator were able to saturate a single 
SCST target with dual IB cards (SRP) on 4K AIO direct accesses over an 
SSD backend. This rawly means that any initiator took several times (8?) 
more processing time than the target. Hardware used for that target and 
initiators was the same. I can't see on this load why the initiators 
would need to do something more than the target. Well, I know we in SCST 
did an excellent work to maximize performance, but such a difference 
looks too much ;)

Also it looks very suspicious why nobody even tried to match that 
Microsoft/Intel record, even Intel itself who closely works with Linux 
community in the storage area and could do it using the same hardware.

Vlad

Re: Linux I/O subsystem performance (was: linuxcon 2010...)

[Vladislav Bolkhovitin]

Pasi Kärkkäinen, on 08/24/2010 11:25 AM wrote:
> On Mon, Aug 23, 2010 at 02:03:26PM -0400, Chetan Loke wrote:
>> I actually received 3+ off-post emails asking whether I was talking
>> about initiator or target in the 100K IOPS case below and what did I
>> mean by the ACKs.
>> I was referring to the 'Initiator' side.
>> ACKs == When scsi-ML down-calls the LLD via the queue-command, process
>> the sgl's(if you like) and then trigger the scsi_done up-call path.
>>
>
> Uhm, Intel and Microsoft demonstrated over 1 million IOPS
> using software iSCSI and a single 10 Gbit Ethernet NIC (Intel 82599).
>
> How come there is such a huge difference? What are we lacking in Linux?

I also have an impression that Linux I/O subsystem has some performance 
problems. For instance, in one recent SCST performance test only 8 Linux 
initiators with fio as a load generator were able to saturate a single 
SCST target with dual IB cards (SRP) on 4K AIO direct accesses over an 
SSD backend. This rawly means that any initiator took several times (8?) 
more processing time than the target. Hardware used for that target and 
initiators was the same. I can't see on this load why the initiators 
would need to do something more than the target. Well, I know we in SCST 
did an excellent work to maximize performance, but such a difference 
looks too much ;)

Also it looks very suspicious why nobody even tried to match that 
Microsoft/Intel record, even Intel itself who closely works with Linux 
community in the storage area and could do it using the same hardware.

Vlad
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Chris Weiss]

On Tue, Aug 24, 2010 at 9:41 AM, Vladislav Bolkhovitin <vst@vlnb.net> wrote:
> James Bottomley, on 08/22/2010 12:43 AM wrote:
>> Interface re-use (or at least ABI compatibility) is the whole point,
>> it's what makes the solution a drop in replacement.
>
> I see now. You want ABI compatibility to keep the "contract" that no
> kernel changes can break applications binary compatibility for unlimited
> time.

ok now I'm confused, or maybe I'm not understanding ABI correctly, or
maybe you guys are using it in a way that is inconsistent with popular
convention.   As a VMware user, I have experienced fully that the
kernel ABI changes in various places with every release.  VMwares
drivers have to be constantly updated to match changes in kernel
function parameters and even what functions are available.

I've also experienced it with scsi cards, dsl modems, and other 3rd
party drivers.  It's the one big downside to developing for the Linux
kernel, the ABI is /always/ changing.

Re: Linux I/O subsystem performance (was: linuxcon 2010...)

[Matthew Wilcox]

On Tue, Aug 24, 2010 at 06:43:29PM +0400, Vladislav Bolkhovitin wrote:
> Also it looks very suspicious why nobody even tried to match that  
> Microsoft/Intel record, even Intel itself who closely works with Linux  
> community in the storage area and could do it using the same hardware.

You seem to be under the impression that "Intel" is some monolithic
entity.  Despite working with six different storage & performance
groups within Intel, I have no idea what record you're referring to,
nor what hardware it was accomplished with.  Even if I did, I wouldn't
know which group within Intel to contact to see if they still have
the setup.  Then I'd have to convince them that it's in their interest
to try to replicate this on Linux.  And I'd have to be prepared to sink
a considerable quantity of my time into it ... which I don't have.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Chetan Loke]

On Tue, Aug 24, 2010 at 3:25 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> On Mon, Aug 23, 2010 at 02:03:26PM -0400, Chetan Loke wrote:
>> I actually received 3+ off-post emails asking whether I was talking
>> about initiator or target in the 100K IOPS case below and what did I
>> mean by the ACKs.
>> I was referring to the 'Initiator' side.
>> ACKs == When scsi-ML down-calls the LLD via the queue-command, process
>> the sgl's(if you like) and then trigger the scsi_done up-call path.
>>
>
> Uhm, Intel and Microsoft demonstrated over 1 million IOPS
> using software iSCSI and a single 10 Gbit Ethernet NIC (Intel 82599).

Uhm, that's MS(and it's closed tcp-chimney protocols and other
offloads?). And I think we discussed in bits and pieces about this on
scst already. Also, just because the driver is open sourced in linux
may not necessarily mean that we know all the ASIC registers that we
can bit-bang and squeeze every clock cycle out of the ASIC(just a
thought).

> How come there is such a huge difference? What are we lacking in Linux?
I'm not a iscsi-guy. So I can't comment on how the data is moved from
n/w buffers to scsi-buffers etc etc.


>
> -- Pasi
Chetan Loke

>>
>> On Mon, Aug 23, 2010 at 12:07 PM, Chetan Loke <chetanloke@gmail.com> wrote:
>> > On Mon, Aug 23, 2010 at 11:11 AM, Bart Van Assche <bvanassche@acm.org> wrote:
>> >
>> >>
>> >> There is an important design difference between SCST and LIO: SCST by
>> >> defaults creates multiple threads to process the I/O operations for a
>> >> storage target, while LIO only creates a single thread per storage target.
>> >> This makes SCST perform measurably faster.
>> >>
>> >
>> > Forget that. You could have discussed this if there were code reviews
>> > or other mainline inclusion emails from James B. From what I have
>> > heard, the decision was taken around 8-9 months back.
>> > Would anyone like to either comment/validate/refute this please?  If
>> > not then I would kindly request these guys to stop taking us for a
>> > test drive. And also I'm not sure when was the last time James B.
>> > bench-marked our scsi-stack. Even if I ACK in the xmit-path then I
>> > can't push more than 100K IOPs. But other folks have re-engineered our
>> > linux-scsi stack and from what I've heard they can push > 300K+ IOPs.
>> > So I would just ignore performance discussion because I don't think
>> > folks have done even simple lame experiments in the last 1 year. Or
>> > may be I'm completely wrong and so please enlighten me so that I can
>> > re-run the tests.
>> >
>> >
>> >> Bart.
>> >>
>> > Chetan Loke
>> >
>>
>> ------------------------------------------------------------------------------
>> Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
>> Be part of this innovative community and reach millions of netbook users
>> worldwide. Take advantage of special opportunities to increase revenue and
>> speed time-to-market. Join now, and jumpstart your future.
>> http://p.sf.net/sfu/intel-atom-d2d
>> _______________________________________________
>> Scst-devel mailing list
>> Scst-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/scst-devel
>

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Matthew Wilcox]

On Tue, Aug 24, 2010 at 09:51:04AM -0500, Chris Weiss wrote:
> ok now I'm confused, or maybe I'm not understanding ABI correctly, or
> maybe you guys are using it in a way that is inconsistent with popular
> convention.   As a VMware user, I have experienced fully that the
> kernel ABI changes in various places with every release.  VMwares
> drivers have to be constantly updated to match changes in kernel
> function parameters and even what functions are available.

You're talking about in-kernel ABI, which is constantly in flux.  James
is talking about user <-> kernel ABI, which is not supposed to change in
an incompatible way.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[James Bottomley]

On Tue, 2010-08-24 at 18:41 +0400, Vladislav Bolkhovitin wrote:
> James Bottomley, on 08/22/2010 12:43 AM wrote:
> > Interface re-use (or at least ABI compatibility) is the whole point,
> > it's what makes the solution a drop in replacement.
> 
> I see now. You want ABI compatibility to keep the "contract" that no 
> kernel changes can break applications binary compatibility for unlimited 
> time.
> 
> OK, we will write the compatibility module. It shouldn't take much time.
> 
> But before we start, I'd like to clear 2 related questions:
> 
> 1. How far the ABI compatibility "contract" goes? Are there cases, where 
> it isn't so strong? I'm asking, because I can recall that open-iscsi at 
> least once has broken ABI compatibility with user space tools. Was it an 
> accidental (but not corrected) mistake or was it deliberate? If the 
> latter, then, I guess, there must be some exceptions defining when ABI 
> compatibility can be not followed.

I don't think it has to be complete.  As long as the STGT people think
it's good enough, that's fine by me.

> 2. Currently STGT in the kernel is just 2 files, scsi_tgt_if.c and
> scsi_tgt_lib.c (with headers), + ibmvstgt driver. The C files define the 
> STGT interface in question. So, if we keep ABI compatibility with the 
> new target engine, we would have to keep those 2 files included in the 
> kernel,

This isn't really correct.  The ABI is defined by the headers not the
implementation.

>  which would effectively mean that STGT would stay in the kernel. 
> This would lead to the situation you are trying to avoid: 2 SCSI target 
> infrastructures in the kernel. Would it be OK?

If you mean is the marketing solution of wedging two products into the
kernel and calling it a single one going to fly, the answer is no.

James

Re: Linux I/O subsystem performance

[Vladislav Bolkhovitin]

Matthew Wilcox, on 08/24/2010 06:55 PM wrote:
> On Tue, Aug 24, 2010 at 06:43:29PM +0400, Vladislav Bolkhovitin wrote:
>> Also it looks very suspicious why nobody even tried to match that
>> Microsoft/Intel record, even Intel itself who closely works with Linux
>> community in the storage area and could do it using the same hardware.
>
> You seem to be under the impression that "Intel" is some monolithic
> entity.  Despite working with six different storage&  performance
> groups within Intel, I have no idea what record you're referring to,
> nor what hardware it was accomplished with.

It is 
http://communities.intel.com/community/wired/blog/2010/04/22/1-million-iops-how-about-125-million

> Even if I did, I wouldn't
> know which group within Intel to contact to see if they still have
> the setup.  Then I'd have to convince them that it's in their interest
> to try to replicate this on Linux.  And I'd have to be prepared to sink
> a considerable quantity of my time into it ... which I don't have.

Sorry if it looked like I was blaming you. I just was wondering why 
Intel developed Linux drivers for those network adapters and isn't 
interested to similarly demonstrate their performance on Linux.

Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

Nicholas A. Bellinger, on 08/22/2010 12:25 AM wrote:
>> It needs a complete redesign to become fast.
>
> Secondly, not being the original author of drivers/scsi/scsi_tgt_*, I am
> happy to do the work and submit the necessary patches to achieve the
> desired goal.  Also, considering now we have the TCM v4 HBA/DEV
> abstraction with a fabric independent control path in
> target_core_fabric_configfs.c, there suddenly new interest to build upon
> tomo's and mnc'c original STGT work in drivers/scsi/scsi_tgt_*.c
>
> Using struct scsi_cmnd allocation from a specially allocated struct
> request_queue still my preferred method for doing this, unless tomo and
> mnc state otherwise.   Also if we need to change the request_queue from
> being per struct Scsi_Host to struct scsi_device that is one thing that
> will be fine.  If we need to make more drastic changes to
> drivers/scsi/scsi_tgt_if.c that is also fine.

That would be a bad design, because it would couple together 
fundamentally separate things: target and initiator subsystems (server 
and client, eg apache and firefox, sendmail and mutt, etc.). It would 
make the code harder to maintain and more fragile for modifications. On 
the user level it would lead to the need to have target mode core module 
always loaded. It is already so with STGT if you use FC or SRP. With 
them the only way to not have scsi_tgt loaded is to remove STGT on the 
compilation time.

> However saying that it needs a 'complete redesign', especially without
> ever consulting or offering to creat patches with STGT guys is not how
> mainline development functions.

Well, it isn't my habit to bother people asking something about what I 
can find an answer myself. I have spent sufficient amount of time 
hacking Linux kernel (>10 years, from which 8 in the target mode), so I 
can read others C code as easy as a book.

I've already described which flaws I see in the STGT design and nobody 
objected me (one of the objections I repeated above). You can find my 
messages in the list archive. Isn't answering on exact critics a way how 
a cooperative development is supposed to be done? If somebody comes with 
a better solution for an existing problem is he supposed be ignored? Is 
it the way how the mainline development functions?

>> In
>> contrast, interface provided by scst_user has just 3% overhead comparing
>> with fully in-kernel backend and allows to build storage capable of
>> handling 1,500,000 IOPSes (Kaminario).
>
> Please understand that you are more than welcome to create your
> own TCM subsystem plugin for your SCST kernel<->  user passthrough
> interface so it can take advantage of all the drivers/target/ code has
> to offer.

Well, please understand that you are more than welcome to stop 
reinventing a wheel and instead just have the functionality and the 
advantages you referring already long ago implemented in SCST and being 
used to build (using your expression) records breaking storage products. 
You are also more than welcome to add the only extra functionality LIO 
has over SCST, ALUA, into SCST. Your patches are always welcome.

Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

James Bottomley, on 08/24/2010 06:57 PM wrote:
> On Tue, 2010-08-24 at 18:41 +0400, Vladislav Bolkhovitin wrote:
>> James Bottomley, on 08/22/2010 12:43 AM wrote:
>>> Interface re-use (or at least ABI compatibility) is the whole point,
>>> it's what makes the solution a drop in replacement.
>>
>> I see now. You want ABI compatibility to keep the "contract" that no
>> kernel changes can break applications binary compatibility for unlimited
>> time.
>>
>> OK, we will write the compatibility module. It shouldn't take much time.
>>
>> But before we start, I'd like to clear 2 related questions:
>>
>> 1. How far the ABI compatibility "contract" goes? Are there cases, where
>> it isn't so strong? I'm asking, because I can recall that open-iscsi at
>> least once has broken ABI compatibility with user space tools. Was it an
>> accidental (but not corrected) mistake or was it deliberate? If the
>> latter, then, I guess, there must be some exceptions defining when ABI
>> compatibility can be not followed.
> 
> I don't think it has to be complete.  As long as the STGT people think
> it's good enough, that's fine by me.

Tomonori, Mike, could you comment on that, please?
 
>> 2. Currently STGT in the kernel is just 2 files, scsi_tgt_if.c and
>> scsi_tgt_lib.c (with headers), + ibmvstgt driver. The C files define the
>> STGT interface in question. So, if we keep ABI compatibility with the
>> new target engine, we would have to keep those 2 files included in the
>> kernel,
> 
> This isn't really correct.  The ABI is defined by the headers not the
> implementation.

Yes, but we on the target side would not be able to implement the ABI compatible interface without using library functions provided by those C files. Or, at least, it would be much harder.

So, would it be OK for you to keep those files?
 
>> which would effectively mean that STGT would stay in the kernel.
>> This would lead to the situation you are trying to avoid: 2 SCSI target
>> infrastructures in the kernel. Would it be OK?
> 
> If you mean is the marketing solution of wedging two products into the
> kernel and calling it a single one going to fly, the answer is no.

I mean that if we keep those 2 files to ease our ABI compatibility effort, it would effectively mean that we would leave STGT merged. It isn't something we would create, it just would be so itself as a matter of fact. Ultimately, STGT is an user space engine. What it has in the kernel is the interface helper functions to interact with the in-kernel drivers. The simplest way to achieve the ABI compatibility is to make a backend module acting as an STGT in-target driver.

(Actually, I may not ask it, because this is the way how LIO seems[1] implemented that, which was approved on the LSF summit. I only want to make all pros and cons clear from the very beginning.)

Thanks,
Vlad

1. I wrote "seems", because currently LIO has the following code for STGT commands execution: 

int stgt_do_task(se_task_t *task)
{
	stgt_plugin_task_t *st = (stgt_plugin_task_t *) task->transport_req;
	struct Scsi_Host *sh = task->se_dev->se_hba->hba_ptr;
	struct scsi_cmnd *sc;
	int tag = MSG_SIMPLE_TAG;

	sc = scsi_host_get_command(sh, st->stgt_direction, GFP_KERNEL);
	if (!sc) {
		printk(KERN_ERR "Unable to allocate memory for struct"
			" scsi_cmnd\n");
		return PYX_TRANSPORT_LU_COMM_FAILURE;
	}

	memcpy(sc->cmnd, st->stgt_cdb, MAX_COMMAND_SIZE);
	sc->sdb.length = task->task_size;
	sc->sdb.table.sgl = task->task_sg;
	sc->tag = tag;

	BUG();
#warning FIXME: Get struct scsi_lun for scsi_tgt_queue_command()
#if 0
	err = scsi_tgt_queue_command(sc, itn_id, (struct scsi_lun *)&cmd->lun,
			cmd->tag);
	if (err) {
		printk(KERN_INFO "scsi_tgt_queue_command() failed for sc:"
			" %p\n", sc);
		scsi_host_put_command(sh, sc);
	}
#endif
	return PYX_TRANSPORT_SENT_TO_TRANSPORT;
}

which means that this pluging completely not functioning.

Re: Linux I/O subsystem performance (was: linuxcon 2010...)

[Chris Worley]

On Tue, Aug 24, 2010 at 11:43 AM, Vladislav Bolkhovitin <vst@vlnb.net> wrote:
> Pasi Kärkkäinen, on 08/24/2010 11:25 AM wrote:
>>
>> On Mon, Aug 23, 2010 at 02:03:26PM -0400, Chetan Loke wrote:
>>>
>>> I actually received 3+ off-post emails asking whether I was talking
>>> about initiator or target in the 100K IOPS case below and what did I
>>> mean by the ACKs.
>>> I was referring to the 'Initiator' side.
>>> ACKs == When scsi-ML down-calls the LLD via the queue-command, process
>>> the sgl's(if you like) and then trigger the scsi_done up-call path.
>>>
>>
>> Uhm, Intel and Microsoft demonstrated over 1 million IOPS
>> using software iSCSI and a single 10 Gbit Ethernet NIC (Intel 82599).
>>
>> How come there is such a huge difference? What are we lacking in Linux?
>
> I also have an impression that Linux I/O subsystem has some performance
> problems. For instance, in one recent SCST performance test only 8 Linux
> initiators with fio as a load generator were able to saturate a single SCST
> target with dual IB cards (SRP) on 4K AIO direct accesses over an SSD
> backend. This rawly means that any initiator took several times (8?) more
> processing time than the target.

While I can't tell you where the bottlenecks are, I can share some
performance numbers...

4 initiators can get >600K random 4KB IOPS off a single target...
which is ~150% of what the Emulex/Intel/Microsoft results show using 8
targets at 4KB (their 1M IOPS was at 512 byte blocks, which is not a
realistic test point) here:

http://itbrandpulse.com/Documents/Test2010001%20-%20The%20Sun%20Rises%20on%20CNAs%20Test%20Report.pdf

The blog referenced earlier used 10 targets... and I'm not sure how
many 10G ports per target.

In general, my target seems capable of 65% the local small-block
random write performance over IB,  and 85% the local small-block
random read performance.  For large block performance, ~95% efficiency
is easily achievable, read or write (i.e. 5.6GB/s over fabric, where
6GB/s is achievable on the drives locally at 1MB random blocks).
These small-block efficiencies are achievable only when tested with
multiple initiators.

The single initiator is only capable of <150K 4KB IOPS... but gets
full bandwidth w/ larger blocks.

If I were to chose my problem, target or initiator bottleneck, I'd
certainly rather have an initiator bottleneck rather than Microsoft's
target bottleneck.

> Hardware used for that target and
> initiators was the same. I can't see on this load why the initiators would
> need to do something more than the target. Well, I know we in SCST did an
> excellent work to maximize performance, but such a difference looks too much
> ;)
>
> Also it looks very suspicious why nobody even tried to match that
> Microsoft/Intel record, even Intel itself who closely works with Linux
> community in the storage area and could do it using the same hardware.

The numbers are suspicious for other reasons.  "Random" is often used
loosely (and the blog referenced earlier doesn't even claim "random").
 If there is any merging/coalescing going on, then the "IOPS" are
going to look vastly better.  If I allow coalescing, I can easily get
4M 4KB IOPS, but can't honestly call those 4KB IOPS (even if the
benchmark thinks it's doing 4KB I/O).  They need to show that their
advertised block size is maintained end-to-end.

Chris

Re: Linux I/O subsystem performance (was: linuxcon 2010...)

[Chris Worley]

On Tue, Aug 24, 2010 at 11:43 AM, Vladislav Bolkhovitin <vst@vlnb.net> wrote:
> Pasi Kärkkäinen, on 08/24/2010 11:25 AM wrote:
>>
>> On Mon, Aug 23, 2010 at 02:03:26PM -0400, Chetan Loke wrote:
>>>
>>> I actually received 3+ off-post emails asking whether I was talking
>>> about initiator or target in the 100K IOPS case below and what did I
>>> mean by the ACKs.
>>> I was referring to the 'Initiator' side.
>>> ACKs == When scsi-ML down-calls the LLD via the queue-command, process
>>> the sgl's(if you like) and then trigger the scsi_done up-call path.
>>>
>>
>> Uhm, Intel and Microsoft demonstrated over 1 million IOPS
>> using software iSCSI and a single 10 Gbit Ethernet NIC (Intel 82599).
>>
>> How come there is such a huge difference? What are we lacking in Linux?
>
> I also have an impression that Linux I/O subsystem has some performance
> problems. For instance, in one recent SCST performance test only 8 Linux
> initiators with fio as a load generator were able to saturate a single SCST
> target with dual IB cards (SRP) on 4K AIO direct accesses over an SSD
> backend. This rawly means that any initiator took several times (8?) more
> processing time than the target.

While I can't tell you where the bottlenecks are, I can share some
performance numbers...

4 initiators can get >600K random 4KB IOPS off a single target...
which is ~150% of what the Emulex/Intel/Microsoft results show using 8
targets at 4KB (their 1M IOPS was at 512 byte blocks, which is not a
realistic test point) here:

http://itbrandpulse.com/Documents/Test2010001%20-%20The%20Sun%20Rises%20on%20CNAs%20Test%20Report.pdf

The blog referenced earlier used 10 targets... and I'm not sure how
many 10G ports per target.

In general, my target seems capable of 65% the local small-block
random write performance over IB,  and 85% the local small-block
random read performance.  For large block performance, ~95% efficiency
is easily achievable, read or write (i.e. 5.6GB/s over fabric, where
6GB/s is achievable on the drives locally at 1MB random blocks).
These small-block efficiencies are achievable only when tested with
multiple initiators.

The single initiator is only capable of <150K 4KB IOPS... but gets
full bandwidth w/ larger blocks.

If I were to chose my problem, target or initiator bottleneck, I'd
certainly rather have an initiator bottleneck rather than Microsoft's
target bottleneck.

> Hardware used for that target and
> initiators was the same. I can't see on this load why the initiators would
> need to do something more than the target. Well, I know we in SCST did an
> excellent work to maximize performance, but such a difference looks too much
> ;)
>
> Also it looks very suspicious why nobody even tried to match that
> Microsoft/Intel record, even Intel itself who closely works with Linux
> community in the storage area and could do it using the same hardware.

The numbers are suspicious for other reasons.  "Random" is often used
loosely (and the blog referenced earlier doesn't even claim "random").
 If there is any merging/coalescing going on, then the "IOPS" are
going to look vastly better.  If I allow coalescing, I can easily get
4M 4KB IOPS, but can't honestly call those 4KB IOPS (even if the
benchmark thinks it's doing 4KB I/O).  They need to show that their
advertised block size is maintained end-to-end.

Chris
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux I/O subsystem performance

[Matthew Wilcox]

On Tue, Aug 24, 2010 at 09:51:38PM +0400, Vladislav Bolkhovitin wrote:
> Matthew Wilcox, on 08/24/2010 06:55 PM wrote:
>> On Tue, Aug 24, 2010 at 06:43:29PM +0400, Vladislav Bolkhovitin wrote:
>>> Also it looks very suspicious why nobody even tried to match that
>>> Microsoft/Intel record, even Intel itself who closely works with Linux
>>> community in the storage area and could do it using the same hardware.
>>
>> You seem to be under the impression that "Intel" is some monolithic
>> entity.  Despite working with six different storage&  performance
>> groups within Intel, I have no idea what record you're referring to,
>> nor what hardware it was accomplished with.
>
> It is  
> http://communities.intel.com/community/wired/blog/2010/04/22/1-million-iops-how-about-125-million

Ah, iSCSI.  I don't work with that group.  I'm a bit too busy with other projects to pursue a relationship with them right now :-)

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Nicholas A. Bellinger]

On Tue, 2010-08-24 at 23:48 +0400, Vladislav Bolkhovitin wrote:
> James Bottomley, on 08/24/2010 06:57 PM wrote:
> > On Tue, 2010-08-24 at 18:41 +0400, Vladislav Bolkhovitin wrote:
> >> James Bottomley, on 08/22/2010 12:43 AM wrote:
> >>> Interface re-use (or at least ABI compatibility) is the whole point,
> >>> it's what makes the solution a drop in replacement.
> >>
> >> I see now. You want ABI compatibility to keep the "contract" that no
> >> kernel changes can break applications binary compatibility for unlimited
> >> time.
> >>
> >> OK, we will write the compatibility module. It shouldn't take much time.
> >>
> >> But before we start, I'd like to clear 2 related questions:
> >>
> >> 1. How far the ABI compatibility "contract" goes? Are there cases, where
> >> it isn't so strong? I'm asking, because I can recall that open-iscsi at
> >> least once has broken ABI compatibility with user space tools. Was it an
> >> accidental (but not corrected) mistake or was it deliberate? If the
> >> latter, then, I guess, there must be some exceptions defining when ABI
> >> compatibility can be not followed.
> > 
> > I don't think it has to be complete.  As long as the STGT people think
> > it's good enough, that's fine by me.
> 
> Tomonori, Mike, could you comment on that, please?
>  
> >> 2. Currently STGT in the kernel is just 2 files, scsi_tgt_if.c and
> >> scsi_tgt_lib.c (with headers), + ibmvstgt driver. The C files define the
> >> STGT interface in question. So, if we keep ABI compatibility with the
> >> new target engine, we would have to keep those 2 files included in the
> >> kernel,
> > 
> > This isn't really correct.  The ABI is defined by the headers not the
> > implementation.
> 
> Yes, but we on the target side would not be able to implement the ABI compatible interface without using library functions provided by those C files. Or, at least, it would be much harder.
> 
> So, would it be OK for you to keep those files?
>  
> >> which would effectively mean that STGT would stay in the kernel.
> >> This would lead to the situation you are trying to avoid: 2 SCSI target
> >> infrastructures in the kernel. Would it be OK?
> > 
> > If you mean is the marketing solution of wedging two products into the
> > kernel and calling it a single one going to fly, the answer is no.
> 
> I mean that if we keep those 2 files to ease our ABI compatibility effort, it would effectively mean that we would leave STGT merged. It isn't something we would create, it just would be so itself as a matter of fact. Ultimately, STGT is an user space engine. What it has in the kernel is the interface helper functions to interact with the in-kernel drivers. The simplest way to achieve the ABI compatibility is to make a backend module acting as an STGT in-target driver.
> 
> (Actually, I may not ask it, because this is the way how LIO seems[1] implemented that, which was approved on the LSF summit. I only want to make all pros and cons clear from the very beginning.)
> 
> Thanks,
> Vlad
> 
> 1. I wrote "seems", because currently LIO has the following code for STGT commands execution: 
> 
> int stgt_do_task(se_task_t *task)
> {
> 	stgt_plugin_task_t *st = (stgt_plugin_task_t *) task->transport_req;
> 	struct Scsi_Host *sh = task->se_dev->se_hba->hba_ptr;
> 	struct scsi_cmnd *sc;
> 	int tag = MSG_SIMPLE_TAG;
> 
> 	sc = scsi_host_get_command(sh, st->stgt_direction, GFP_KERNEL);
> 	if (!sc) {
> 		printk(KERN_ERR "Unable to allocate memory for struct"
> 			" scsi_cmnd\n");
> 		return PYX_TRANSPORT_LU_COMM_FAILURE;
> 	}
> 
> 	memcpy(sc->cmnd, st->stgt_cdb, MAX_COMMAND_SIZE);
> 	sc->sdb.length = task->task_size;
> 	sc->sdb.table.sgl = task->task_sg;
> 	sc->tag = tag;
> 
> 	BUG();
> #warning FIXME: Get struct scsi_lun for scsi_tgt_queue_command()
> #if 0
> 	err = scsi_tgt_queue_command(sc, itn_id, (struct scsi_lun *)&cmd->lun,
> 			cmd->tag);
> 	if (err) {
> 		printk(KERN_INFO "scsi_tgt_queue_command() failed for sc:"
> 			" %p\n", sc);
> 		scsi_host_put_command(sh, sc);
> 	}
> #endif
> 	return PYX_TRANSPORT_SENT_TO_TRANSPORT;
> }

Vlad,

As mentioned explictly earlier in this thread, my WIP code for the
kernel level subsystem backstore using STGT kernel <-> user CDB
passthrough logic in drivers/target/target_core_stgt.c is a item on my
TODO list, but I will repeat, is NOT being tagged as a mainline .37
item.

Tomo-san, mnc, and other storage folks have been briefed on the
remaining issues that would be involved to get a prototype functioning
with drivers/target/target_core_stgt.c, and rough idea of what it would
take in existing mainline drivers/scsi/scsi_tgt_*.c code.  With the
current WIP code this will allow the userspace CDB -> LUN passthrough to
function transparently with all TCM SPC-4 compliant logic as a
standalone struct se_subsystem_api tcm_core_stgt.ko backstore.

Best,

--nab

Re: Linux I/O subsystem performance

[Vladislav Bolkhovitin]

Chris Worley, on 08/25/2010 12:31 AM wrote:
>> I also have an impression that Linux I/O subsystem has some performance
>> problems. For instance, in one recent SCST performance test only 8 Linux
>> initiators with fio as a load generator were able to saturate a single SCST
>> target with dual IB cards (SRP) on 4K AIO direct accesses over an SSD
>> backend. This rawly means that any initiator took several times (8?) more
>> processing time than the target.
>
> While I can't tell you where the bottlenecks are, I can share some
> performance numbers...
>
> 4 initiators can get>600K random 4KB IOPS off a single target...

Hmm, on the data you sent me only 8 initiators were capable to do so... 
I'm glad to see an improvement here ;).

> which is ~150% of what the Emulex/Intel/Microsoft results show using 8
> targets at 4KB (their 1M IOPS was at 512 byte blocks, which is not a
> realistic test point

 From my, a storage developer's, POV it isn't about if this test is 
realistic or not. 512 bytes tests are good if you want to test how 
processing effective your I/O stack, because they produce the max 
possible CPU/memory/hardware interaction load. Since processing power 
isn't unlimited, in case if it is a bottleneck, N IOPS on 512b < N IOPS 
on 4K * 8 and system with more effective processing will have better 
numbers.

Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Konrad Rzeszutek Wilk]

On Tuesday 24 August 2010 10:51:04 Chris Weiss wrote:
> On Tue, Aug 24, 2010 at 9:41 AM, Vladislav Bolkhovitin <vst@vlnb.net> wrote:
> > James Bottomley, on 08/22/2010 12:43 AM wrote:
> >> Interface re-use (or at least ABI compatibility) is the whole point,
> >> it's what makes the solution a drop in replacement.
> >
> > I see now. You want ABI compatibility to keep the "contract" that no
> > kernel changes can break applications binary compatibility for unlimited
> > time.
>
> ok now I'm confused, or maybe I'm not understanding ABI correctly, or
> maybe you guys are using it in a way that is inconsistent with popular

You are thinking of the KABI. That changes per each release except if you buy 
a vendor product. Red Hat for example keeps an KABI symbol list where they 
guarantee that those parameters, structures ,etc will never change. John 
Masters wrote a nice paper about how they solved this:
http://dup.et.redhat.com/presentations/DriverUpdateProgramTechnical.pdf

I don't have experience with other vendors.

In terms of ABI, think ioctl calls and its a parameters. They are suppose to 
stay the same for long long durations.

> convention.   As a VMware user, I have experienced fully that the
> kernel ABI changes in various places with every release.  VMwares
> drivers have to be constantly updated to match changes in kernel
> function parameters and even what functions are available.
>
> I've also experienced it with scsi cards, dsl modems, and other 3rd
> party drivers.  It's the one big downside to developing for the Linux
> kernel, the ABI is /always/ changing.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Ted Ts'o]

On Wed, Aug 25, 2010 at 06:20:26PM -0400, Konrad Rzeszutek Wilk wrote:
> On Tuesday 24 August 2010 10:51:04 Chris Weiss wrote:
> > On Tue, Aug 24, 2010 at 9:41 AM, Vladislav Bolkhovitin <vst@vlnb.net> wrote:
> > > James Bottomley, on 08/22/2010 12:43 AM wrote:
> > >> Interface re-use (or at least ABI compatibility) is the whole point,
> > >> it's what makes the solution a drop in replacement.
> > >
> > > I see now. You want ABI compatibility to keep the "contract" that no
> > > kernel changes can break applications binary compatibility for unlimited
> > > time.
> >
> > ok now I'm confused, or maybe I'm not understanding ABI correctly, or
> > maybe you guys are using it in a way that is inconsistent with popular
> 
> You are thinking of the KABI. That changes per each release except
> if you buy a vendor product. Red Hat for example keeps an KABI
> symbol list where they guarantee that those parameters, structures
> ,etc will never change. John Masters wrote a nice paper about how
> they solved this:
> http://dup.et.redhat.com/presentations/DriverUpdateProgramTechnical.pdf

Just to make sure people aren't getting more confused.  What Red Hat
calls the KABI (and SLES and Ubuntu do something similar) is the
Kernel ABI which is presented to kernel modules.  This is important
for companies shipping out-of-tree and proprietary kernel modules.
(And we'll ignore the questions about whether such proprietary kernel
modules violate the GPL or not; contact your favorite lawyer for an
opinion on that subject.  It depends on the facts of the case and your
legal jourisdiction, almost certainly.)

> In terms of ABI, think ioctl calls and its a parameters. They are suppose to 
> stay the same for long long durations.

When we talk about the ABI must be constant, this is the kernel
interface presented to userspace programs, including statically linked
userspace progams.   So system calls, ioctl's, etc.

This is what allows you to download or purchase a userspace program
(including silly things like DB2, or Oracle Database, or Adobe Flash),
and it will work even if you upgrade your kernel.

						- Ted

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

Nicholas A. Bellinger, on 08/25/2010 01:23 AM wrote:
> As mentioned explictly earlier in this thread, my WIP code for the
> kernel level subsystem backstore using STGT kernel<->  user CDB
> passthrough logic in drivers/target/target_core_stgt.c is a item on
> my TODO list, but I will repeat, is NOT being tagged as a mainline
> .37 item.

Hmm, I can't understand, if target_core_stgt.c is "NOT being tagged as a 
mainline .37 item", then the STGT ABI compatibility for being a drop in 
replacement for STGT isn't a requirement? Or am I missing something?

> Tomo-san, mnc, and other storage folks have been briefed on the
> remaining issues that would be involved to get a prototype
> functioning with drivers/target/target_core_stgt.c, and rough idea of
> what it would take in existing mainline drivers/scsi/scsi_tgt_*.c
> code.  With the current WIP code this will allow the userspace CDB ->
> LUN passthrough to function transparently with all TCM SPC-4
> compliant logic as a standalone struct se_subsystem_api
> tcm_core_stgt.ko backstore.

This is exactly what we are discussing.

Thanks,
Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Nicholas A. Bellinger]

On Fri, 2010-08-27 at 00:11 +0400, Vladislav Bolkhovitin wrote:
> Nicholas A. Bellinger, on 08/25/2010 01:23 AM wrote:
> > As mentioned explictly earlier in this thread, my WIP code for the
> > kernel level subsystem backstore using STGT kernel<->  user CDB
> > passthrough logic in drivers/target/target_core_stgt.c is a item on
> > my TODO list, but I will repeat, is NOT being tagged as a mainline
> > .37 item.
> 
> Hmm, I can't understand, if target_core_stgt.c is "NOT being tagged as a 
> mainline .37 item", then the STGT ABI compatibility for being a drop in 
> replacement for STGT isn't a requirement? Or am I missing something?
> 

Sorry, but I have no idea what you are trying to conjour up here.

To spell out (again) the TCM/LIO<->STGT compatibility stages that have
been persued pubically over the last months:

I) Create proper userspace tgt.git SG_IO and BSG passthrough into   
   TCM_Loop v4 using high-level multi-fabric WWPN emulation so that TCM 
   Core SPC-4 kernel emulation is exposed to STGT user fabrics, eg: 
   userspace fabric module -> kernel backstore passthrough.  (DONE 
   for .37, and STGT passthrough + BSG backstore support merged into 
   tgt.git by Tomo-san)

II) Complete target_core_stgt.c TCM subsystem plugin for kernel -> user 
    CDB -> LUN passthrough building upon existing 
    drivers/scsi/scsi_tgt_*.c code.  (WIP for .38, made available 
    initially as a seperate standalone .ko module in lio-core-2.6.git)

> > Tomo-san, mnc, and other storage folks have been briefed on the
> > remaining issues that would be involved to get a prototype
> > functioning with drivers/target/target_core_stgt.c, and rough idea of
> > what it would take in existing mainline drivers/scsi/scsi_tgt_*.c
> > code.  With the current WIP code this will allow the userspace CDB ->
> > LUN passthrough to function transparently with all TCM SPC-4
> > compliant logic as a standalone struct se_subsystem_api
> > tcm_core_stgt.ko backstore.
> 
> This is exactly what we are discussing.

Then I suggest you start working on a patch for drivers/scsi/scsi_tgt_*
in order to address what you believe are the preceived shortcomings of
the original design.

Until you can do that, and actually take an impartial look at the
existing drivers/scsi/scsi_tgt_*.c,  it would be a bit difficult to
beleive you genuinely want to steer our current level of interaction
away from continued hand-waving noise into a real rational technical
discourse between yourself and the LIO/STGT development community.

Best,

--nab

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

Nicholas A. Bellinger, on 08/27/2010 01:23 AM wrote:
>> Nicholas A. Bellinger, on 08/25/2010 01:23 AM wrote:
>>> As mentioned explictly earlier in this thread, my WIP code for the
>>> kernel level subsystem backstore using STGT kernel<->   user CDB
>>> passthrough logic in drivers/target/target_core_stgt.c is a item on
>>> my TODO list, but I will repeat, is NOT being tagged as a mainline
>>> .37 item.
>>
>> Hmm, I can't understand, if target_core_stgt.c is "NOT being tagged as a
>> mainline .37 item", then the STGT ABI compatibility for being a drop in
>> replacement for STGT isn't a requirement? Or am I missing something?
>
> Sorry, but I have no idea what you are trying to conjour up here.
>
> To spell out (again) the TCM/LIO<->STGT compatibility stages that have
> been persued pubically over the last months:
>
> I) Create proper userspace tgt.git SG_IO and BSG passthrough into
>     TCM_Loop v4 using high-level multi-fabric WWPN emulation so that TCM
>     Core SPC-4 kernel emulation is exposed to STGT user fabrics, eg:
>     userspace fabric module ->  kernel backstore passthrough.  (DONE
>     for .37, and STGT passthrough + BSG backstore support merged into
>     tgt.git by Tomo-san)
>
> II) Complete target_core_stgt.c TCM subsystem plugin for kernel ->  user
>      CDB ->  LUN passthrough building upon existing
>      drivers/scsi/scsi_tgt_*.c code.  (WIP for .38, made available
>      initially as a seperate standalone .ko module in lio-core-2.6.git)

That would mean that if LIO merged in .37:

1. It would be merged _without_ STGT ABI compatibility, i.e. one of the 
requirements for a new SCSI target infrastructure merge is violated.

2. It would _not_ be a drop in replacement for STGT, because STGT's 
drivers/scsi/scsi_tgt_*.c code would stay in the kernel. Those would 
effectively mean that another requirement for a new SCSI target 
infrastructure merge would be violated: there would be 2 SCSI target 
infrastructures in the kernel and any STGT in-kernel driver (for 
instance, built as an outside module) would continue working. My 
understanding of "drop in replacement" is "one out, another in at once".

Please tell me where I'm wrong? I would appreciate if you be precise and 
answer the above 2 my concerns. There is no point to again repeat what 
you have already written without adding any new information.

>>> Tomo-san, mnc, and other storage folks have been briefed on the
>>> remaining issues that would be involved to get a prototype
>>> functioning with drivers/target/target_core_stgt.c, and rough idea of
>>> what it would take in existing mainline drivers/scsi/scsi_tgt_*.c
>>> code.  With the current WIP code this will allow the userspace CDB ->
>>> LUN passthrough to function transparently with all TCM SPC-4
>>> compliant logic as a standalone struct se_subsystem_api
>>> tcm_core_stgt.ko backstore.
>>
>> This is exactly what we are discussing.
>
> Then I suggest you start working on a patch for drivers/scsi/scsi_tgt_*
> in order to address what you believe are the preceived shortcomings of
> the original design.
>
> Until you can do that, and actually take an impartial look at the
> existing drivers/scsi/scsi_tgt_*.c,  it would be a bit difficult to
> beleive you genuinely want to steer our current level of interaction
> away from continued hand-waving noise into a real rational technical
> discourse between yourself and the LIO/STGT development community.

My look is completely impartial. With all my respect, I'm just quite 
ahead of you and can see what you are unable (or don't want?) to see. I 
did what you are currently doing almost 4 years ago. I have already 
written all the necessary code and addressed all _rose on practice_ 
concerns. But all my attempts to explain my view are just blindly 
ignored without any considerations, so I have no idea how more I can 
explain it.

Thanks,
Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Nicholas A. Bellinger]

On Sat, 2010-08-28 at 21:32 +0400, Vladislav Bolkhovitin wrote:
> Nicholas A. Bellinger, on 08/27/2010 01:23 AM wrote:
> >> Nicholas A. Bellinger, on 08/25/2010 01:23 AM wrote:
> >>> As mentioned explictly earlier in this thread, my WIP code for the
> >>> kernel level subsystem backstore using STGT kernel<->   user CDB
> >>> passthrough logic in drivers/target/target_core_stgt.c is a item on
> >>> my TODO list, but I will repeat, is NOT being tagged as a mainline
> >>> .37 item.
> >>
> >> Hmm, I can't understand, if target_core_stgt.c is "NOT being tagged as a
> >> mainline .37 item", then the STGT ABI compatibility for being a drop in
> >> replacement for STGT isn't a requirement? Or am I missing something?
> >
> > Sorry, but I have no idea what you are trying to conjour up here.
> >
> > To spell out (again) the TCM/LIO<->STGT compatibility stages that have
> > been persued pubically over the last months:
> >
> > I) Create proper userspace tgt.git SG_IO and BSG passthrough into
> >     TCM_Loop v4 using high-level multi-fabric WWPN emulation so that TCM
> >     Core SPC-4 kernel emulation is exposed to STGT user fabrics, eg:
> >     userspace fabric module ->  kernel backstore passthrough.  (DONE
> >     for .37, and STGT passthrough + BSG backstore support merged into
> >     tgt.git by Tomo-san)
> >
> > II) Complete target_core_stgt.c TCM subsystem plugin for kernel ->  user
> >      CDB ->  LUN passthrough building upon existing
> >      drivers/scsi/scsi_tgt_*.c code.  (WIP for .38, made available
> >      initially as a seperate standalone .ko module in lio-core-2.6.git)
> 
> That would mean that if LIO merged in .37:
> 
> 1. It would be merged _without_ STGT ABI compatibility, i.e. one of the 
> requirements for a new SCSI target infrastructure merge is violated.
> 

Sorry, but you are completely wrong here.  This has nothing to do with a
question of STGT kernel 'ABI compatibility' (because there is only one
mainline user!), but has everything to do with being able to expose TCM
kernel level SPC-4 emulation, and make this logic available to existing
userspace fabrics in tgt.git.  Again, we are talking about userspace
STGT fabric module <-> TCM kernel backstore support for .37, which has
already been merged by into tgt.git, and being used by other STGT users
for SG_IO and BSG passthrough.

> 2. It would _not_ be a drop in replacement for STGT, because STGT's 
> drivers/scsi/scsi_tgt_*.c code would stay in the kernel. Those would 
> effectively mean that another requirement for a new SCSI target 
> infrastructure merge would be violated: there would be 2 SCSI target 
> infrastructures in the kernel and any STGT in-kernel driver (for 
> instance, built as an outside module) would continue working. My 
> understanding of "drop in replacement" is "one out, another in at once".
> 

Sorry, but this is just more generic handwaving on your part.  What
constitutes a 'drop in replacement' for STGT is a decision that was to
be made by the STGT developers, not by you.  target_core_stgt.c is an
extraordinarly transparent method of bringing drivers/scsi/scsi_tgt_*
logic into the TCM Core HBA/DEV design, and allows us to build upon and
improve the existing mainline kernel code.

> Please tell me where I'm wrong? I would appreciate if you be precise and 
> answer the above 2 my concerns. There is no point to again repeat what 
> you have already written without adding any new information.
> 
> >>> Tomo-san, mnc, and other storage folks have been briefed on the
> >>> remaining issues that would be involved to get a prototype
> >>> functioning with drivers/target/target_core_stgt.c, and rough idea of
> >>> what it would take in existing mainline drivers/scsi/scsi_tgt_*.c
> >>> code.  With the current WIP code this will allow the userspace CDB ->
> >>> LUN passthrough to function transparently with all TCM SPC-4
> >>> compliant logic as a standalone struct se_subsystem_api
> >>> tcm_core_stgt.ko backstore.
> >>
> >> This is exactly what we are discussing.
> >
> > Then I suggest you start working on a patch for drivers/scsi/scsi_tgt_*
> > in order to address what you believe are the preceived shortcomings of
> > the original design.
> >
> > Until you can do that, and actually take an impartial look at the
> > existing drivers/scsi/scsi_tgt_*.c,  it would be a bit difficult to
> > beleive you genuinely want to steer our current level of interaction
> > away from continued hand-waving noise into a real rational technical
> > discourse between yourself and the LIO/STGT development community.
> 
> My look is completely impartial. With all my respect, I'm just quite 
> ahead of you and can see what you are unable (or don't want?) to see. I 
> did what you are currently doing almost 4 years ago. I have already 
> written all the necessary code and addressed all _rose on practice_ 
> concerns.

It is obvious to even an casual observer from watching the TCM/LIO patch
series that have been flying across the linux-scsi wire the last 24
months that the major features (including PR and ALUA, and new fabric
module drivers) have been developed individual feature bit by feature
bit using a distributed git workflow in a bisectable manner.  Each
series was produced in such a manner that each patch could be reviewed
individually by those interested parties.

This is the big difference between our respective development processes.
This is the case not only for SCSI feature bits that TCM/LIO and SCST
share, but for features in TCM/LIO v4 that are unique and not available
in any other target implemention, anywhere.  And yes, I am most
certainly talking about code beyond just the SCSI level fabric emulation
bits you are mentioning above.

> But all my attempts to explain my view are just blindly 
> ignored without any considerations, so I have no idea how more I can 
> explain it.
> 

Then I suggest you learn a better way of communicating your ideas if you
really want to work with members of the LIO/STGT development
communities.

First, I suggest you start explaining your ideas with actual kernel code
that is 1) human readable and 2) presented in such a manner that makes
it accessable for others with skills possibly different (and greater)
than your own to review and give feedback.

Best,

--nab

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

Nicholas A. Bellinger, on 08/29/2010 12:47 AM wrote:
>>>>> As mentioned explictly earlier in this thread, my WIP code for the
>>>>> kernel level subsystem backstore using STGT kernel<->    user CDB
>>>>> passthrough logic in drivers/target/target_core_stgt.c is a item on
>>>>> my TODO list, but I will repeat, is NOT being tagged as a mainline
>>>>> .37 item.
>>>>
>>>> Hmm, I can't understand, if target_core_stgt.c is "NOT being tagged as a
>>>> mainline .37 item", then the STGT ABI compatibility for being a drop in
>>>> replacement for STGT isn't a requirement? Or am I missing something?
>>>
>>> Sorry, but I have no idea what you are trying to conjour up here.
>>>
>>> To spell out (again) the TCM/LIO<->STGT compatibility stages that have
>>> been persued pubically over the last months:
>>>
>>> I) Create proper userspace tgt.git SG_IO and BSG passthrough into
>>>      TCM_Loop v4 using high-level multi-fabric WWPN emulation so that TCM
>>>      Core SPC-4 kernel emulation is exposed to STGT user fabrics, eg:
>>>      userspace fabric module ->   kernel backstore passthrough.  (DONE
>>>      for .37, and STGT passthrough + BSG backstore support merged into
>>>      tgt.git by Tomo-san)
>>>
>>> II) Complete target_core_stgt.c TCM subsystem plugin for kernel ->   user
>>>       CDB ->   LUN passthrough building upon existing
>>>       drivers/scsi/scsi_tgt_*.c code.  (WIP for .38, made available
>>>       initially as a seperate standalone .ko module in lio-core-2.6.git)
>>
>> That would mean that if LIO merged in .37:
>>
>> 1. It would be merged _without_ STGT ABI compatibility, i.e. one of the
>> requirements for a new SCSI target infrastructure merge is violated.
>>
>
> Sorry, but you are completely wrong here.  This has nothing to do with a
> question of STGT kernel 'ABI compatibility' (because there is only one
> mainline user!), but has everything to do with being able to expose TCM
> kernel level SPC-4 emulation, and make this logic available to existing
> userspace fabrics in tgt.git.  Again, we are talking about userspace
> STGT fabric module<->  TCM kernel backstore support for .37, which has
> already been merged by into tgt.git, and being used by other STGT users
> for SG_IO and BSG passthrough.

You are proving that I'm actually right ;). In the beginning of this 
thread I offered a transition path from STGT to SCST and James rejected 
it, because it didn't confirm a requirement that a new SCSI target 
infrastructure must be STGT ABI compatible. But latter James left this 
decision up to the STGT developers and now you are confirming that they 
don't see any ABI compatibility issues, so my transition path fully 
confirms all the requirements.

>> 2. It would _not_ be a drop in replacement for STGT, because STGT's
>> drivers/scsi/scsi_tgt_*.c code would stay in the kernel. Those would
>> effectively mean that another requirement for a new SCSI target
>> infrastructure merge would be violated: there would be 2 SCSI target
>> infrastructures in the kernel and any STGT in-kernel driver (for
>> instance, built as an outside module) would continue working. My
>> understanding of "drop in replacement" is "one out, another in at once".
>
> Sorry, but this is just more generic handwaving on your part.  What
> constitutes a 'drop in replacement' for STGT is a decision that was to
> be made by the STGT developers, not by you.  target_core_stgt.c is an
> extraordinarly transparent method of bringing drivers/scsi/scsi_tgt_*
> logic into the TCM Core HBA/DEV design, and allows us to build upon and
> improve the existing mainline kernel code.

I'm not deciding anything, I'm only analyzing and seeing a contradiction:

1. James wants only one SCSI target infrastructure in the kernel.

2. If drivers/scsi/scsi_tgt_*.c left after the merge of the new SCSI 
target infrastructure, it would mean that STGT left as well => 2 SCSI 
target infrastructure in the kernel.

For me it doesn't matter. I just want clear rules, hence asking for 
clarification.

> It is obvious to even an casual observer from watching the TCM/LIO patch
> series that have been flying across the linux-scsi wire the last 24
> months that the major features (including PR and ALUA, and new fabric
> module drivers) have been developed individual feature bit by feature
> bit using a distributed git workflow in a bisectable manner.  Each
> series was produced in such a manner that each patch could be reviewed
> individually by those interested parties.

That's a nice, but quite meaningless LIO advertisement. SCST is using 
the same bisectable, distributed and reviewable workflow. If we had a 
kernel.org git account, we would use it as well, but so far we are happy 
with SF.net hosting. BTW, how was you able to get the git account 
without a single patch merged in the kernel? You must have good 
connections in the kernel community.

> This is the big difference between our respective development processes.
> This is the case not only for SCSI feature bits that TCM/LIO and SCST
> share, but for features in TCM/LIO v4 that are unique and not available
> in any other target implemention, anywhere.  And yes, I am most
> certainly talking about code beyond just the SCSI level fabric emulation
> bits you are mentioning above.

Can you list us benefits for an average user of that work?

>> But all my attempts to explain my view are just blindly
>> ignored without any considerations, so I have no idea how more I can
>> explain it.
>>
>
> Then I suggest you learn a better way of communicating your ideas if you
> really want to work with members of the LIO/STGT development
> communities.
>
> First, I suggest you start explaining your ideas with actual kernel code
> that is 1) human readable and 2) presented in such a manner that makes
> it accessable for others with skills possibly different (and greater)
> than your own to review and give feedback.

I have sent patches twice, the second time few months ago. Weren't they 
human readable and accessible for kernel developers who are experts in 
dealing with sent by e-mail patches?

Thanks,
Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Nicholas A. Bellinger]

On Tue, 2010-08-31 at 00:47 +0400, Vladislav Bolkhovitin wrote:
> Nicholas A. Bellinger, on 08/29/2010 12:47 AM wrote:
> >>>>> As mentioned explictly earlier in this thread, my WIP code for the
> >>>>> kernel level subsystem backstore using STGT kernel<->    user CDB
> >>>>> passthrough logic in drivers/target/target_core_stgt.c is a item on
> >>>>> my TODO list, but I will repeat, is NOT being tagged as a mainline
> >>>>> .37 item.
> >>>>
> >>>> Hmm, I can't understand, if target_core_stgt.c is "NOT being tagged as a
> >>>> mainline .37 item", then the STGT ABI compatibility for being a drop in
> >>>> replacement for STGT isn't a requirement? Or am I missing something?
> >>>
> >>> Sorry, but I have no idea what you are trying to conjour up here.
> >>>
> >>> To spell out (again) the TCM/LIO<->STGT compatibility stages that have
> >>> been persued pubically over the last months:
> >>>
> >>> I) Create proper userspace tgt.git SG_IO and BSG passthrough into
> >>>      TCM_Loop v4 using high-level multi-fabric WWPN emulation so that TCM
> >>>      Core SPC-4 kernel emulation is exposed to STGT user fabrics, eg:
> >>>      userspace fabric module ->   kernel backstore passthrough.  (DONE
> >>>      for .37, and STGT passthrough + BSG backstore support merged into
> >>>      tgt.git by Tomo-san)
> >>>
> >>> II) Complete target_core_stgt.c TCM subsystem plugin for kernel ->   user
> >>>       CDB ->   LUN passthrough building upon existing
> >>>       drivers/scsi/scsi_tgt_*.c code.  (WIP for .38, made available
> >>>       initially as a seperate standalone .ko module in lio-core-2.6.git)
> >>
> >> That would mean that if LIO merged in .37:
> >>
> >> 1. It would be merged _without_ STGT ABI compatibility, i.e. one of the
> >> requirements for a new SCSI target infrastructure merge is violated.
> >>
> >
> > Sorry, but you are completely wrong here.  This has nothing to do with a
> > question of STGT kernel 'ABI compatibility' (because there is only one
> > mainline user!), but has everything to do with being able to expose TCM
> > kernel level SPC-4 emulation, and make this logic available to existing
> > userspace fabrics in tgt.git.  Again, we are talking about userspace
> > STGT fabric module<->  TCM kernel backstore support for .37, which has
> > already been merged by into tgt.git, and being used by other STGT users
> > for SG_IO and BSG passthrough.
> 
> You are proving that I'm actually right ;). In the beginning of this 
> thread I offered a transition path from STGT to SCST and James rejected 
> it, because it didn't confirm a requirement that a new SCSI target 
> infrastructure must be STGT ABI compatible. But latter James left this 
> decision up to the STGT developers and now you are confirming that they 
> don't see any ABI compatibility issues, so my transition path fully 
> confirms all the requirements.
> 

Again, I still have no idea what you are trying to conjour up.  The
passthrough for STGT compatibility with TCM_Loop via SG_IO and BSG has
already been merged by Tomo-san into tgt.git.  Futhermore, we are using
the same TCM_Loop high level multi-fabric WWPN emulation together with
new (and old) QEMU HBA emulation into KVM guest using SG_IO and BSG as
well.

> >> 2. It would _not_ be a drop in replacement for STGT, because STGT's
> >> drivers/scsi/scsi_tgt_*.c code would stay in the kernel. Those would
> >> effectively mean that another requirement for a new SCSI target
> >> infrastructure merge would be violated: there would be 2 SCSI target
> >> infrastructures in the kernel and any STGT in-kernel driver (for
> >> instance, built as an outside module) would continue working. My
> >> understanding of "drop in replacement" is "one out, another in at once".
> >
> > Sorry, but this is just more generic handwaving on your part.  What
> > constitutes a 'drop in replacement' for STGT is a decision that was to
> > be made by the STGT developers, not by you.  target_core_stgt.c is an
> > extraordinarly transparent method of bringing drivers/scsi/scsi_tgt_*
> > logic into the TCM Core HBA/DEV design, and allows us to build upon and
> > improve the existing mainline kernel code.
> 
> I'm not deciding anything, I'm only analyzing and seeing a contradiction:
> 
> 1. James wants only one SCSI target infrastructure in the kernel.
> 
> 2. If drivers/scsi/scsi_tgt_*.c left after the merge of the new SCSI 
> target infrastructure, it would mean that STGT left as well => 2 SCSI 
> target infrastructure in the kernel.
> 
> For me it doesn't matter. I just want clear rules, hence asking for 
> clarification.

Seriously, arguing over the use of people's language from weeks ago once
the decision has already been made to move forward is really of little
value at this point.

> 
> > It is obvious to even an casual observer from watching the TCM/LIO patch
> > series that have been flying across the linux-scsi wire the last 24
> > months that the major features (including PR and ALUA, and new fabric
> > module drivers) have been developed individual feature bit by feature
> > bit using a distributed git workflow in a bisectable manner.  Each
> > series was produced in such a manner that each patch could be reviewed
> > individually by those interested parties.
> 
> That's a nice, but quite meaningless LIO advertisement. SCST is using 
> the same bisectable, distributed and reviewable workflow.

Actually, that is incorrect.  Your project uses a centralized
development model, which has it's obvious limitiations in terms of
speed, flexability, and community scale.  But really, don't take my word
for it, you can hear it for yourself directly from the horse's mouth
here:

http://www.youtube.com/watch?v=4XpnKHJAok8

I also very strongly suggest you find a transcript of this talk so you
can really understand what Linus means here wrt to a distributed
development workflow.

>  If we had a 
> kernel.org git account, we would use it as well, but so far we are happy 
> with SF.net hosting. BTW, how was you able to get the git account 
> without a single patch merged in the kernel? You must have good 
> connections in the kernel community.

Please don't turn this into a kernel.org vs. SF.net thing..  There are
plenty of places that offer free git hosting (see github)

> 
> > This is the big difference between our respective development processes.
> > This is the case not only for SCSI feature bits that TCM/LIO and SCST
> > share, but for features in TCM/LIO v4 that are unique and not available
> > in any other target implemention, anywhere.  And yes, I am most
> > certainly talking about code beyond just the SCSI level fabric emulation
> > bits you are mentioning above.
> 
> Can you list us benefits for an average user of that work?

Well, amougst the biggest advantages is actually having a sane ConfigFS
interface that can represent the parent / child relationship between
data structures across multiple LKMs.  Not only does this give us a
proper representation of TCM and fabric data structures that can be
accesses directly by an advanced user in /sys/kernel/config/target/, it
also provides an ideal foundation to build 1) a userspace library in
intrepted languages and 2) create high level applications using said
userspace libraries that allow compatiblity to be contained in the lib
below.

But wrt to actual kernel code (because this is a kernel list), the
shortlog for the RFC posting below nicely sums up what TCM v4 is
bringing to the table:

http://lkml.org/lkml/2010/8/30/88


> >> But all my attempts to explain my view are just blindly
> >> ignored without any considerations, so I have no idea how more I can
> >> explain it.
> >>
> >
> > Then I suggest you learn a better way of communicating your ideas if you
> > really want to work with members of the LIO/STGT development
> > communities.
> >
> > First, I suggest you start explaining your ideas with actual kernel code
> > that is 1) human readable and 2) presented in such a manner that makes
> > it accessable for others with skills possibly different (and greater)
> > than your own to review and give feedback.
> 
> I have sent patches twice, the second time few months ago. Weren't they 
> human readable and accessible for kernel developers who are experts in 
> dealing with sent by e-mail patches?

I think a proper kernel subsystem maintainer workflow has to do alot
more to do these days than just sending patches over email, than it did
say 10 years ago.  Like it or not, git is the language of the mainline
kernel development process, and trying to imagine a *new* kernel
subsystem maintainer (besides say, someone like akpm) not using git is
quite a stretch of the imagination at this point.

Best,

--nab

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

Nicholas A. Bellinger, on 08/31/2010 01:46 AM wrote:
>>> Sorry, but you are completely wrong here.  This has nothing to do with a
>>> question of STGT kernel 'ABI compatibility' (because there is only one
>>> mainline user!), but has everything to do with being able to expose TCM
>>> kernel level SPC-4 emulation, and make this logic available to existing
>>> userspace fabrics in tgt.git.  Again, we are talking about userspace
>>> STGT fabric module<->   TCM kernel backstore support for .37, which has
>>> already been merged by into tgt.git, and being used by other STGT users
>>> for SG_IO and BSG passthrough.
>>
>> You are proving that I'm actually right ;). In the beginning of this
>> thread I offered a transition path from STGT to SCST and James rejected
>> it, because it didn't confirm a requirement that a new SCSI target
>> infrastructure must be STGT ABI compatible. But latter James left this
>> decision up to the STGT developers and now you are confirming that they
>> don't see any ABI compatibility issues, so my transition path fully
>> confirms all the requirements.
>
> Again, I still have no idea what you are trying to conjour up.  The
> passthrough for STGT compatibility with TCM_Loop via SG_IO and BSG has
> already been merged by Tomo-san into tgt.git.  Futhermore, we are using
> the same TCM_Loop high level multi-fabric WWPN emulation together with
> new (and old) QEMU HBA emulation into KVM guest using SG_IO and BSG as
> well.

The patches were good, but don't mislead people about that. For sake of 
clearness, what you called "the passthrough for STGT compatibility with 
TCM_Loop via SG_IO and BSG" has nothing with LIO/TCM_Loop. What merged 
fixed problems STGT had in the pass-through implementation. It at the 
same time fixed problems with SCST's scst_local, so similarly can be 
called "the passthrough for STGT compatibility with scst_local via SG_IO 
and BSG".

The same is for the second half of the above. It's for scst_local in the 
same degree as for TCM_Loop.

>> I'm not deciding anything, I'm only analyzing and seeing a contradiction:
>>
>> 1. James wants only one SCSI target infrastructure in the kernel.
>>
>> 2. If drivers/scsi/scsi_tgt_*.c left after the merge of the new SCSI
>> target infrastructure, it would mean that STGT left as well =>  2 SCSI
>> target infrastructure in the kernel.
>>
>> For me it doesn't matter. I just want clear rules, hence asking for
>> clarification.
>
> Seriously, arguing over the use of people's language from weeks ago once
> the decision has already been made to move forward is really of little
> value at this point.

Does it look I'm arguing? Again, I'm analyzing and asking for 
clarification in the contradictions I see. I don't like when such 
important decisions are done behind closed doors and kept in a secret.

>>> It is obvious to even an casual observer from watching the TCM/LIO patch
>>> series that have been flying across the linux-scsi wire the last 24
>>> months that the major features (including PR and ALUA, and new fabric
>>> module drivers) have been developed individual feature bit by feature
>>> bit using a distributed git workflow in a bisectable manner.  Each
>>> series was produced in such a manner that each patch could be reviewed
>>> individually by those interested parties.
>>
>> That's a nice, but quite meaningless LIO advertisement. SCST is using
>> the same bisectable, distributed and reviewable workflow.
>
> Actually, that is incorrect.  Your project uses a centralized
> development model, which has it's obvious limitiations in terms of
> speed, flexability, and community scale.  But really, don't take my word
> for it, you can hear it for yourself directly from the horse's mouth
> here:
>
> http://www.youtube.com/watch?v=4XpnKHJAok8
>
> I also very strongly suggest you find a transcript of this talk so you
> can really understand what Linus means here wrt to a distributed
> development workflow.

Well, Nicholas, are you really understanding what you are writing? We in 
our projects have fully the same distributed (or centralized, if you 
like it) development model. Have you noticed how many developers SCST 
project has? We have our responsibility areas (target drivers, 
scstadmin, etc.) and commit in them. The way how we get updates for the 
rest of the kernel doesn't matter. Git is better for such huge projects 
as the kernel, but for our relatively small and centralized by nature 
due to small size (sub)projects it doesn't matter and won't bring any value.

Actually, to have all in one place without the rest of the kernel is 
more convenient, because grep over few MBs is a lot faster than over the 
few hundreds of MBs of the full kernel sources.

>>   If we had a
>> kernel.org git account, we would use it as well, but so far we are happy
>> with SF.net hosting. BTW, how was you able to get the git account
>> without a single patch merged in the kernel? You must have good
>> connections in the kernel community.
>
> Please don't turn this into a kernel.org vs. SF.net thing..  There are
> plenty of places that offer free git hosting (see github)

SF.net for a long time has been offering git hosting, but we don't see 
reasons to move to it. SVN has all we need. We are not (yet) so big to 
need git.

But, we can move to git at any moment as soon as it is needed, for 
instance, because of SCST merged in the mainline kernel.

>>> This is the big difference between our respective development processes.
>>> This is the case not only for SCSI feature bits that TCM/LIO and SCST
>>> share, but for features in TCM/LIO v4 that are unique and not available
>>> in any other target implemention, anywhere.  And yes, I am most
>>> certainly talking about code beyond just the SCSI level fabric emulation
>>> bits you are mentioning above.
>>
>> Can you list us benefits for an average user of that work?
>
> Well, amougst the biggest advantages is actually having a sane ConfigFS
> interface that can represent the parent / child relationship between
> data structures across multiple LKMs.

Does STGT represent it in an insane manner?

> Not only does this give us a
> proper representation of TCM and fabric data structures that can be
> accesses directly by an advanced user in /sys/kernel/config/target/, it
> also provides an ideal foundation to build 1) a userspace library in
> intrepted languages and 2) create high level applications using said
> userspace libraries that allow compatiblity to be contained in the lib
> below.
>
> But wrt to actual kernel code (because this is a kernel list), the
> shortlog for the RFC posting below nicely sums up what TCM v4 is
> bringing to the table:
>
> http://lkml.org/lkml/2010/8/30/88

I have no doubts in benefits to TCM. But I have asked about benefits 
_for an average user_. STGT already can do what you listed above.

Also what advantages and additional value it has over what SCST 
*already* provides since 2007?

>>>> But all my attempts to explain my view are just blindly
>>>> ignored without any considerations, so I have no idea how more I can
>>>> explain it.
>>>>
>>>
>>> Then I suggest you learn a better way of communicating your ideas if you
>>> really want to work with members of the LIO/STGT development
>>> communities.
>>>
>>> First, I suggest you start explaining your ideas with actual kernel code
>>> that is 1) human readable and 2) presented in such a manner that makes
>>> it accessable for others with skills possibly different (and greater)
>>> than your own to review and give feedback.
>>
>> I have sent patches twice, the second time few months ago. Weren't they
>> human readable and accessible for kernel developers who are experts in
>> dealing with sent by e-mail patches?
>
> I think a proper kernel subsystem maintainer workflow has to do alot
> more to do these days than just sending patches over email, than it did
> say 10 years ago.  Like it or not, git is the language of the mainline
> kernel development process, and trying to imagine a *new* kernel
> subsystem maintainer (besides say, someone like akpm) not using git is
> quite a stretch of the imagination at this point.

So, do you believe that as soon as we migrate to git all the obstacles 
for the SCST merge would be immediately removed?

Thanks,
Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

Nicholas A. Bellinger, on 08/31/2010 01:46 AM wrote:
>>> Sorry, but you are completely wrong here.  This has nothing to do with a
>>> question of STGT kernel 'ABI compatibility' (because there is only one
>>> mainline user!), but has everything to do with being able to expose TCM
>>> kernel level SPC-4 emulation, and make this logic available to existing
>>> userspace fabrics in tgt.git.  Again, we are talking about userspace
>>> STGT fabric module<->   TCM kernel backstore support for .37, which has
>>> already been merged by into tgt.git, and being used by other STGT users
>>> for SG_IO and BSG passthrough.
>>
>> You are proving that I'm actually right ;). In the beginning of this
>> thread I offered a transition path from STGT to SCST and James rejected
>> it, because it didn't confirm a requirement that a new SCSI target
>> infrastructure must be STGT ABI compatible. But latter James left this
>> decision up to the STGT developers and now you are confirming that they
>> don't see any ABI compatibility issues, so my transition path fully
>> confirms all the requirements.
>
> Again, I still have no idea what you are trying to conjour up.  The
> passthrough for STGT compatibility with TCM_Loop via SG_IO and BSG has
> already been merged by Tomo-san into tgt.git.  Futhermore, we are using
> the same TCM_Loop high level multi-fabric WWPN emulation together with
> new (and old) QEMU HBA emulation into KVM guest using SG_IO and BSG as
> well.

The patches were good, but don't mislead people about that. For sake of 
clearness, what you called "the passthrough for STGT compatibility with 
TCM_Loop via SG_IO and BSG" has nothing with LIO/TCM_Loop. What merged 
fixed problems STGT had in the pass-through implementation. It at the 
same time fixed problems with SCST's scst_local, so similarly can be 
called "the passthrough for STGT compatibility with scst_local via SG_IO 
and BSG".

The same is for the second half of the above. It's for scst_local in the 
same degree as for TCM_Loop.

>> I'm not deciding anything, I'm only analyzing and seeing a contradiction:
>>
>> 1. James wants only one SCSI target infrastructure in the kernel.
>>
>> 2. If drivers/scsi/scsi_tgt_*.c left after the merge of the new SCSI
>> target infrastructure, it would mean that STGT left as well =>  2 SCSI
>> target infrastructure in the kernel.
>>
>> For me it doesn't matter. I just want clear rules, hence asking for
>> clarification.
>
> Seriously, arguing over the use of people's language from weeks ago once
> the decision has already been made to move forward is really of little
> value at this point.

Does it look I'm arguing? Again, I'm analyzing and asking for 
clarification in the contradictions I see. I don't like when such 
important decisions are done behind closed doors and kept in a secret.

>>> It is obvious to even an casual observer from watching the TCM/LIO patch
>>> series that have been flying across the linux-scsi wire the last 24
>>> months that the major features (including PR and ALUA, and new fabric
>>> module drivers) have been developed individual feature bit by feature
>>> bit using a distributed git workflow in a bisectable manner.  Each
>>> series was produced in such a manner that each patch could be reviewed
>>> individually by those interested parties.
>>
>> That's a nice, but quite meaningless LIO advertisement. SCST is using
>> the same bisectable, distributed and reviewable workflow.
>
> Actually, that is incorrect.  Your project uses a centralized
> development model, which has it's obvious limitiations in terms of
> speed, flexability, and community scale.  But really, don't take my word
> for it, you can hear it for yourself directly from the horse's mouth
> here:
>
> http://www.youtube.com/watch?v=4XpnKHJAok8
>
> I also very strongly suggest you find a transcript of this talk so you
> can really understand what Linus means here wrt to a distributed
> development workflow.

Well, Nicholas, are you really understanding what you are writing? We in 
our projects have fully the same distributed (or centralized, if you 
like it) development model. Have you noticed how many developers SCST 
project has? We have our responsibility areas (target drivers, 
scstadmin, etc.) and commit in them. The way how we get updates for the 
rest of the kernel doesn't matter. Git is better for such huge projects 
as the kernel, but for our relatively small and centralized by nature 
due to small size (sub)projects it doesn't matter and won't bring any value.

Actually, to have all in one place without the rest of the kernel is 
more convenient, because grep over few MBs is a lot faster than over the 
few hundreds of MBs of the full kernel sources.

>>   If we had a
>> kernel.org git account, we would use it as well, but so far we are happy
>> with SF.net hosting. BTW, how was you able to get the git account
>> without a single patch merged in the kernel? You must have good
>> connections in the kernel community.
>
> Please don't turn this into a kernel.org vs. SF.net thing..  There are
> plenty of places that offer free git hosting (see github)

SF.net for a long time has been offering git hosting, but we don't see 
reasons to move to it. SVN has all we need. We are not (yet) so big to 
need git.

But, we can move to git at any moment as soon as it is needed, for 
instance, because of SCST merged in the mainline kernel.

>>> This is the big difference between our respective development processes.
>>> This is the case not only for SCSI feature bits that TCM/LIO and SCST
>>> share, but for features in TCM/LIO v4 that are unique and not available
>>> in any other target implemention, anywhere.  And yes, I am most
>>> certainly talking about code beyond just the SCSI level fabric emulation
>>> bits you are mentioning above.
>>
>> Can you list us benefits for an average user of that work?
>
> Well, amougst the biggest advantages is actually having a sane ConfigFS
> interface that can represent the parent / child relationship between
> data structures across multiple LKMs.

Does STGT represent it in an insane manner?

> Not only does this give us a
> proper representation of TCM and fabric data structures that can be
> accesses directly by an advanced user in /sys/kernel/config/target/, it
> also provides an ideal foundation to build 1) a userspace library in
> intrepted languages and 2) create high level applications using said
> userspace libraries that allow compatiblity to be contained in the lib
> below.
>
> But wrt to actual kernel code (because this is a kernel list), the
> shortlog for the RFC posting below nicely sums up what TCM v4 is
> bringing to the table:
>
> http://lkml.org/lkml/2010/8/30/88

I have no doubts in benefits to TCM. But I have asked about benefits 
_for an average user_. STGT already can do what you listed above.

Also what advantages and additional value it has over what SCST 
*already* provides since 2007?

>>>> But all my attempts to explain my view are just blindly
>>>> ignored without any considerations, so I have no idea how more I can
>>>> explain it.
>>>>
>>>
>>> Then I suggest you learn a better way of communicating your ideas if you
>>> really want to work with members of the LIO/STGT development
>>> communities.
>>>
>>> First, I suggest you start explaining your ideas with actual kernel code
>>> that is 1) human readable and 2) presented in such a manner that makes
>>> it accessable for others with skills possibly different (and greater)
>>> than your own to review and give feedback.
>>
>> I have sent patches twice, the second time few months ago. Weren't they
>> human readable and accessible for kernel developers who are experts in
>> dealing with sent by e-mail patches?
>
> I think a proper kernel subsystem maintainer workflow has to do alot
> more to do these days than just sending patches over email, than it did
> say 10 years ago.  Like it or not, git is the language of the mainline
> kernel development process, and trying to imagine a *new* kernel
> subsystem maintainer (besides say, someone like akpm) not using git is
> quite a stretch of the imagination at this point.

So, do you believe that as soon as we migrate to git all the obstacles 
for the SCST merge would be immediately removed?

Thanks,
Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Nicholas A. Bellinger]

On Thu, 2010-09-02 at 23:38 +0400, Vladislav Bolkhovitin wrote:
> Nicholas A. Bellinger, on 08/31/2010 01:46 AM wrote:
> > Again, I still have no idea what you are trying to conjour up.  The
> > passthrough for STGT compatibility with TCM_Loop via SG_IO and BSG has
> > already been merged by Tomo-san into tgt.git.  Futhermore, we are using
> > the same TCM_Loop high level multi-fabric WWPN emulation together with
> > new (and old) QEMU HBA emulation into KVM guest using SG_IO and BSG as
> > well.
> 
> The patches were good, but don't mislead people about that. For sake of 
> clearness, what you called "the passthrough for STGT compatibility with 
> TCM_Loop via SG_IO and BSG" has nothing with LIO/TCM_Loop. What merged 
> fixed problems STGT had in the pass-through implementation. It at the 
> same time fixed problems with SCST's scst_local, so similarly can be 
> called "the passthrough for STGT compatibility with scst_local via SG_IO 
> and BSG".
> 

Uh, I don't think TCM_Loop and scst_local are exactly comparable at this
point.  TCM_Loop supports high level multi-fabric WWPN emulation that
allows it to work transparently with inter-nexus SPC-4 PR operation.
Therefore, using TCM_Loop, STGT userspace fabrics now support both PR
and ALUA emulation from TCM_Loop LUNs.  This is done at struct
config_group_ops->make_group() time to report $PROTO_IDENT and the
necessary WWPN info to TCM Core logic.  TCM_Loop also uses the fabric
indepent configfs handlers for pretty much all of it's control plane
code.

> The same is for the second half of the above. It's for scst_local in the 
> same degree as for TCM_Loop.
> 
> >> I'm not deciding anything, I'm only analyzing and seeing a contradiction:
> >>
> >> 1. James wants only one SCSI target infrastructure in the kernel.
> >>
> >> 2. If drivers/scsi/scsi_tgt_*.c left after the merge of the new SCSI
> >> target infrastructure, it would mean that STGT left as well =>  2 SCSI
> >> target infrastructure in the kernel.
> >>
> >> For me it doesn't matter. I just want clear rules, hence asking for
> >> clarification.
> >
> > Seriously, arguing over the use of people's language from weeks ago once
> > the decision has already been made to move forward is really of little
> > value at this point.
> 
> Does it look I'm arguing? Again, I'm analyzing and asking for 
> clarification in the contradictions I see. I don't like when such 
> important decisions are done behind closed doors and kept in a secret.
> 

I really have no idea what you are talking about 'behind closed doors'..

Have you not been watching the amount of TCM/LIO patches on the
linux-scsi list for the last 18 months..?  

> >>> It is obvious to even an casual observer from watching the TCM/LIO patch
> >>> series that have been flying across the linux-scsi wire the last 24
> >>> months that the major features (including PR and ALUA, and new fabric
> >>> module drivers) have been developed individual feature bit by feature
> >>> bit using a distributed git workflow in a bisectable manner.  Each
> >>> series was produced in such a manner that each patch could be reviewed
> >>> individually by those interested parties.
> >>
> >> That's a nice, but quite meaningless LIO advertisement. SCST is using
> >> the same bisectable, distributed and reviewable workflow.
> >
> > Actually, that is incorrect.  Your project uses a centralized
> > development model, which has it's obvious limitiations in terms of
> > speed, flexability, and community scale.  But really, don't take my word
> > for it, you can hear it for yourself directly from the horse's mouth
> > here:
> >
> > http://www.youtube.com/watch?v=4XpnKHJAok8
> >
> > I also very strongly suggest you find a transcript of this talk so you
> > can really understand what Linus means here wrt to a distributed
> > development workflow.
> 
> Well, Nicholas, are you really understanding what you are writing? We in 
> our projects have fully the same distributed (or centralized, if you 
> like it) development model. Have you noticed how many developers SCST 
> project has? We have our responsibility areas (target drivers, 
> scstadmin, etc.) and commit in them. The way how we get updates for the 
> rest of the kernel doesn't matter. Git is better for such huge projects 
> as the kernel, but for our relatively small and centralized by nature 
> due to small size (sub)projects it doesn't matter and won't bring any value.
> 

I find it hard to beleive you are actually going to agrue against a git
workflow for a target mode subsystem maintainer, well considering that
git was made by a linux kernel maintainer (Linus) for distributed linux
kernel development (everybody else)..?

Again, I suggest you watch the Russian translation of that talk on
youtube and really understand what he means, aside from the insults to
subversion users.

> >>> This is the big difference between our respective development processes.
> >>> This is the case not only for SCSI feature bits that TCM/LIO and SCST
> >>> share, but for features in TCM/LIO v4 that are unique and not available
> >>> in any other target implemention, anywhere.  And yes, I am most
> >>> certainly talking about code beyond just the SCSI level fabric emulation
> >>> bits you are mentioning above.
> >>
> >> Can you list us benefits for an average user of that work?
> >
> > Well, amougst the biggest advantages is actually having a sane ConfigFS
> > interface that can represent the parent / child relationship between
> > data structures across multiple LKMs.
> 
> Does STGT represent it in an insane manner?
> 
> > Not only does this give us a
> > proper representation of TCM and fabric data structures that can be
> > accesses directly by an advanced user in /sys/kernel/config/target/, it
> > also provides an ideal foundation to build 1) a userspace library in
> > intrepted languages and 2) create high level applications using said
> > userspace libraries that allow compatiblity to be contained in the lib
> > below.
> >
> > But wrt to actual kernel code (because this is a kernel list), the
> > shortlog for the RFC posting below nicely sums up what TCM v4 is
> > bringing to the table:
> >
> > http://lkml.org/lkml/2010/8/30/88
> 
> I have no doubts in benefits to TCM. But I have asked about benefits 
> _for an average user_. STGT already can do what you listed above.
> 

Again, from above how it benefits users:

1) a userspace library in intrepted languages and 2) create high level
applications using said userspace libraries that allow compatiblity to
be contained in the lib below.

2) create high level applications using said userspace libraries that
allow compatiblity to be contained in the lib.

> Also what advantages and additional value it has over what SCST 
> *already* provides since 2007?
> 
> >>>> But all my attempts to explain my view are just blindly
> >>>> ignored without any considerations, so I have no idea how more I can
> >>>> explain it.
> >>>>
> >>>
> >>> Then I suggest you learn a better way of communicating your ideas if you
> >>> really want to work with members of the LIO/STGT development
> >>> communities.
> >>>
> >>> First, I suggest you start explaining your ideas with actual kernel code
> >>> that is 1) human readable and 2) presented in such a manner that makes
> >>> it accessable for others with skills possibly different (and greater)
> >>> than your own to review and give feedback.
> >>
> >> I have sent patches twice, the second time few months ago. Weren't they
> >> human readable and accessible for kernel developers who are experts in
> >> dealing with sent by e-mail patches?
> >
> > I think a proper kernel subsystem maintainer workflow has to do alot
> > more to do these days than just sending patches over email, than it did
> > say 10 years ago.  Like it or not, git is the language of the mainline
> > kernel development process, and trying to imagine a *new* kernel
> > subsystem maintainer (besides say, someone like akpm) not using git is
> > quite a stretch of the imagination at this point.
> 
> So, do you believe that as soon as we migrate to git all the obstacles 
> for the SCST merge would be immediately removed?
> 

Uhhh, I don't think so.  Aside from your obvious project workflow
issues, the fact that SCST completly lacks a proper user-space driven
representation of parent / child relationships between kernel level data
structures in a fabric independent manner, while still allow for fabric
dependent groups and attributes is a most definately show-stopper for
me.

Best,

--nab

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Dmitry Torokhov]

On Thu, Sep 02, 2010 at 01:25:58PM -0700, Nicholas A. Bellinger wrote:
> On Thu, 2010-09-02 at 23:38 +0400, Vladislav Bolkhovitin wrote:
> > Nicholas A. Bellinger, on 08/31/2010 01:46 AM wrote:
> > >>> It is obvious to even an casual observer from watching the TCM/LIO patch
> > >>> series that have been flying across the linux-scsi wire the last 24
> > >>> months that the major features (including PR and ALUA, and new fabric
> > >>> module drivers) have been developed individual feature bit by feature
> > >>> bit using a distributed git workflow in a bisectable manner.  Each
> > >>> series was produced in such a manner that each patch could be reviewed
> > >>> individually by those interested parties.
> > >>
> > >> That's a nice, but quite meaningless LIO advertisement. SCST is using
> > >> the same bisectable, distributed and reviewable workflow.
> > >
> > > Actually, that is incorrect.  Your project uses a centralized
> > > development model, which has it's obvious limitiations in terms of
> > > speed, flexability, and community scale.  But really, don't take my word
> > > for it, you can hear it for yourself directly from the horse's mouth
> > > here:
> > >
> > > http://www.youtube.com/watch?v=4XpnKHJAok8
> > >
> > > I also very strongly suggest you find a transcript of this talk so you
> > > can really understand what Linus means here wrt to a distributed
> > > development workflow.
> > 
> > Well, Nicholas, are you really understanding what you are writing? We in 
> > our projects have fully the same distributed (or centralized, if you 
> > like it) development model. Have you noticed how many developers SCST 
> > project has? We have our responsibility areas (target drivers, 
> > scstadmin, etc.) and commit in them. The way how we get updates for the 
> > rest of the kernel doesn't matter. Git is better for such huge projects 
> > as the kernel, but for our relatively small and centralized by nature 
> > due to small size (sub)projects it doesn't matter and won't bring any value.
> > 
> 
> I find it hard to beleive you are actually going to agrue against a git
> workflow for a target mode subsystem maintainer, well considering that
> git was made by a linux kernel maintainer (Linus) for distributed linux
> kernel development (everybody else)..?
>

This is complete BS. Are we going to judge value of a project based on
what kind SCM they chose to use? I guess we should ban Greg KH from
kernel development and rip out USB and driver core layers from the
kernel because he has the audacity to use quilt for his trees.

-- 
Dmitry

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Nicholas A. Bellinger]

On Sun, 2010-09-05 at 13:18 -0700, Dmitry Torokhov wrote:
> On Thu, Sep 02, 2010 at 01:25:58PM -0700, Nicholas A. Bellinger wrote:
> > On Thu, 2010-09-02 at 23:38 +0400, Vladislav Bolkhovitin wrote:
> > > Nicholas A. Bellinger, on 08/31/2010 01:46 AM wrote:
> > > >>> It is obvious to even an casual observer from watching the TCM/LIO patch
> > > >>> series that have been flying across the linux-scsi wire the last 24
> > > >>> months that the major features (including PR and ALUA, and new fabric
> > > >>> module drivers) have been developed individual feature bit by feature
> > > >>> bit using a distributed git workflow in a bisectable manner.  Each
> > > >>> series was produced in such a manner that each patch could be reviewed
> > > >>> individually by those interested parties.
> > > >>
> > > >> That's a nice, but quite meaningless LIO advertisement. SCST is using
> > > >> the same bisectable, distributed and reviewable workflow.
> > > >
> > > > Actually, that is incorrect.  Your project uses a centralized
> > > > development model, which has it's obvious limitiations in terms of
> > > > speed, flexability, and community scale.  But really, don't take my word
> > > > for it, you can hear it for yourself directly from the horse's mouth
> > > > here:
> > > >
> > > > http://www.youtube.com/watch?v=4XpnKHJAok8
> > > >
> > > > I also very strongly suggest you find a transcript of this talk so you
> > > > can really understand what Linus means here wrt to a distributed
> > > > development workflow.
> > > 
> > > Well, Nicholas, are you really understanding what you are writing? We in 
> > > our projects have fully the same distributed (or centralized, if you 
> > > like it) development model. Have you noticed how many developers SCST 
> > > project has? We have our responsibility areas (target drivers, 
> > > scstadmin, etc.) and commit in them. The way how we get updates for the 
> > > rest of the kernel doesn't matter. Git is better for such huge projects 
> > > as the kernel, but for our relatively small and centralized by nature 
> > > due to small size (sub)projects it doesn't matter and won't bring any value.
> > > 
> > 
> > I find it hard to beleive you are actually going to agrue against a git
> > workflow for a target mode subsystem maintainer, well considering that
> > git was made by a linux kernel maintainer (Linus) for distributed linux
> > kernel development (everybody else)..?
> >
> 
> This is complete BS. Are we going to judge value of a project based on
> what kind SCM they chose to use? I guess we should ban Greg KH from
> kernel development and rip out USB and driver core layers from the
> kernel because he has the audacity to use quilt for his trees.
> 

I think the difference between what Linus says and what he actually
means in the above video could be easily misinterpreted, well especially
for some non-english speaking folks.  But I am certainly glad to see
that a russian translation is also available for this google git talk
for those who really want try to understand his reasons (and technical
requirements) for what he is saying.

So as to the specifics about why git is really the only right SCM choice
for mainline target mode maintainership, it really all comes down to
workflow.  Does your SCM allow other people to easily and without-pain
track your upstream subsystem tree changes and 'rebase' as necessary for
their patch series you make improvements..?   If we are talking about
say, a single standalone driver being developed against mainline, then
sure using a SCM like CVS or SVN is perfectly acceptable when you push
to someone upstream like gregkh or akpm via email patch attachments.

However, if we are talking about a subsystem maintainer workflow that
requires many different people to track your subsystem tree for their
own drivers and new feature bits, not being able to keep local branches
for these level developers makes their life excruciatingly painful and
unpleasent as they attempt to pull new upstream changes, especially when
the speed of new upstream development is moving quickly.  This rule
applys even more when said subsystem has a greater scope within the
source tree in question.

Anyways, if we are going to compare SCM distributed vs. centralized
workflow in terms of kernel projects, lets please at least compare
apples to apples here.

Best,

--nab

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Mark Deneen]

On Sun, Sep 5, 2010 at 5:50 PM, Nicholas A. Bellinger
<nab@linux-iscsi.org> wrote:
>
> I think the difference between what Linus says and what he actually
> means in the above video could be easily misinterpreted, well especially
> for some non-english speaking folks.  But I am certainly glad to see
> that a russian translation is also available for this google git talk
> for those who really want try to understand his reasons (and technical
> requirements) for what he is saying.
>
> So as to the specifics about why git is really the only right SCM choice
> for mainline target mode maintainership, it really all comes down to
> workflow.  Does your SCM allow other people to easily and without-pain
> track your upstream subsystem tree changes and 'rebase' as necessary for
> their patch series you make improvements..?   If we are talking about
> say, a single standalone driver being developed against mainline, then
> sure using a SCM like CVS or SVN is perfectly acceptable when you push
> to someone upstream like gregkh or akpm via email patch attachments.
>
> However, if we are talking about a subsystem maintainer workflow that
> requires many different people to track your subsystem tree for their
> own drivers and new feature bits, not being able to keep local branches
> for these level developers makes their life excruciatingly painful and
> unpleasent as they attempt to pull new upstream changes, especially when
> the speed of new upstream development is moving quickly.  This rule
> applys even more when said subsystem has a greater scope within the
> source tree in question.
>
> Anyways, if we are going to compare SCM distributed vs. centralized
> workflow in terms of kernel projects, lets please at least compare
> apples to apples here.
>
> Best,
>
> --nab

I've been trying to keep my 2 cents out of this, as I am merely an
SCST user.  Both of you have valid criticisms; the choice of SCM is
not one of them.  It is nit-picking at best, especially when SCST
could switch to git easily if they so desired.

Seven years ago, would you be pushing BitKeeper?

Kind Regards,
Mark Deneen

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Dmitry Torokhov]

On Sun, Sep 05, 2010 at 02:50:47PM -0700, Nicholas A. Bellinger wrote:
> On Sun, 2010-09-05 at 13:18 -0700, Dmitry Torokhov wrote:
> > On Thu, Sep 02, 2010 at 01:25:58PM -0700, Nicholas A. Bellinger wrote:
> > > On Thu, 2010-09-02 at 23:38 +0400, Vladislav Bolkhovitin wrote:
> > > > Nicholas A. Bellinger, on 08/31/2010 01:46 AM wrote:
> > > > >>> It is obvious to even an casual observer from watching the TCM/LIO patch
> > > > >>> series that have been flying across the linux-scsi wire the last 24
> > > > >>> months that the major features (including PR and ALUA, and new fabric
> > > > >>> module drivers) have been developed individual feature bit by feature
> > > > >>> bit using a distributed git workflow in a bisectable manner.  Each
> > > > >>> series was produced in such a manner that each patch could be reviewed
> > > > >>> individually by those interested parties.
> > > > >>
> > > > >> That's a nice, but quite meaningless LIO advertisement. SCST is using
> > > > >> the same bisectable, distributed and reviewable workflow.
> > > > >
> > > > > Actually, that is incorrect.  Your project uses a centralized
> > > > > development model, which has it's obvious limitiations in terms of
> > > > > speed, flexability, and community scale.  But really, don't take my word
> > > > > for it, you can hear it for yourself directly from the horse's mouth
> > > > > here:
> > > > >
> > > > > http://www.youtube.com/watch?v=4XpnKHJAok8
> > > > >
> > > > > I also very strongly suggest you find a transcript of this talk so you
> > > > > can really understand what Linus means here wrt to a distributed
> > > > > development workflow.
> > > > 
> > > > Well, Nicholas, are you really understanding what you are writing? We in 
> > > > our projects have fully the same distributed (or centralized, if you 
> > > > like it) development model. Have you noticed how many developers SCST 
> > > > project has? We have our responsibility areas (target drivers, 
> > > > scstadmin, etc.) and commit in them. The way how we get updates for the 
> > > > rest of the kernel doesn't matter. Git is better for such huge projects 
> > > > as the kernel, but for our relatively small and centralized by nature 
> > > > due to small size (sub)projects it doesn't matter and won't bring any value.
> > > > 
> > > 
> > > I find it hard to beleive you are actually going to agrue against a git
> > > workflow for a target mode subsystem maintainer, well considering that
> > > git was made by a linux kernel maintainer (Linus) for distributed linux
> > > kernel development (everybody else)..?
> > >
> > 
> > This is complete BS. Are we going to judge value of a project based on
> > what kind SCM they chose to use? I guess we should ban Greg KH from
> > kernel development and rip out USB and driver core layers from the
> > kernel because he has the audacity to use quilt for his trees.
> > 
> 
> I think the difference between what Linus says and what he actually
> means in the above video could be easily misinterpreted, well especially
> for some non-english speaking folks.  But I am certainly glad to see
> that a russian translation is also available for this google git talk
> for those who really want try to understand his reasons (and technical
> requirements) for what he is saying.

Will you please piss off with your insinuations that we do not
Understand English? While it may not be our first language it does not
mean we are having trouble with it.

> 
> So as to the specifics about why git is really the only right SCM choice
> for mainline target mode maintainership, it really all comes down to
> workflow.  Does your SCM allow other people to easily and without-pain
> track your upstream subsystem tree changes and 'rebase' as necessary for
> their patch series you make improvements..?   If we are talking about
> say, a single standalone driver being developed against mainline, then
> sure using a SCM like CVS or SVN is perfectly acceptable when you push
> to someone upstream like gregkh or akpm via email patch attachments.
> 
> However, if we are talking about a subsystem maintainer workflow that
> requires many different people to track your subsystem tree for their
> own drivers and new feature bits, not being able to keep local branches
> for these level developers makes their life excruciatingly painful and
> unpleasent as they attempt to pull new upstream changes, especially when
> the speed of new upstream development is moving quickly.

Haven't you noticed yet that not every maintainer uses git? USB, driver
core, tty, staging and i2c subsystems are based on quilt queues.
Media/DVB main tree is in mercurial. AOE is quilt. And I am pretty sure
there are others, I just don't care about them. And if you prefer to use
git - there are SVN and other importers - please use them.

>  This rule
> applys even more when said subsystem has a greater scope within the
> source tree in question.
> 

I do not think that LIO is any bigger in scope than USB, I2C and others.


> Anyways, if we are going to compare SCM distributed vs. centralized
> workflow in terms of kernel projects, lets please at least compare
> apples to apples here.
> 

No, we should not be comparing SCMs at all here but rather 2 competing
implementations based on quality of the code. You tried to bring SMC
angle in and I am saying that it is BS.

-- 
Dmitry

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Nicholas A. Bellinger]

On Sun, 2010-09-05 at 16:41 -0700, Dmitry Torokhov wrote:
> On Sun, Sep 05, 2010 at 02:50:47PM -0700, Nicholas A. Bellinger wrote:
> > Anyways, if we are going to compare SCM distributed vs. centralized
> > workflow in terms of kernel projects, lets please at least compare
> > apples to apples here.
> > 
> 
> No, we should not be comparing SCMs at all here but rather 2 competing
> implementations based on quality of the code. You tried to bring SMC
> angle in and I am saying that it is BS.
> 

Again, without getting into another pointless flamewar,  I think the
main point here is that a open source project using a distributed
workflow (like git) has major advantages when it comes to working with a
larger group of developers than a centralized model (like SVN) does.

Because being a subsystem maintainer typically involves this type of
complex workflow involving lots of different parties, git is a tool that
was created (originally) expressely for a kernel workflow, and for those
types of people it works really, really well.

So, please understand that code and project workflow is only one of the
reasons why TCM/LIO v4 was selected over SCST.   I invite you to take a
closer look at the RFC Code that has been posted last week if you want
to get into the nitty-gritty techinical details, which this thread has
thus far been avoiding.

Best,

--nab

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Nicholas A. Bellinger]

On Sun, 2010-09-05 at 19:13 -0400, Mark Deneen wrote:
> On Sun, Sep 5, 2010 at 5:50 PM, Nicholas A. Bellinger
> <nab@linux-iscsi.org> wrote:
> >
> > I think the difference between what Linus says and what he actually
> > means in the above video could be easily misinterpreted, well especially
> > for some non-english speaking folks.  But I am certainly glad to see
> > that a russian translation is also available for this google git talk
> > for those who really want try to understand his reasons (and technical
> > requirements) for what he is saying.
> >
> > So as to the specifics about why git is really the only right SCM choice
> > for mainline target mode maintainership, it really all comes down to
> > workflow.  Does your SCM allow other people to easily and without-pain
> > track your upstream subsystem tree changes and 'rebase' as necessary for
> > their patch series you make improvements..?   If we are talking about
> > say, a single standalone driver being developed against mainline, then
> > sure using a SCM like CVS or SVN is perfectly acceptable when you push
> > to someone upstream like gregkh or akpm via email patch attachments.
> >
> > However, if we are talking about a subsystem maintainer workflow that
> > requires many different people to track your subsystem tree for their
> > own drivers and new feature bits, not being able to keep local branches
> > for these level developers makes their life excruciatingly painful and
> > unpleasent as they attempt to pull new upstream changes, especially when
> > the speed of new upstream development is moving quickly.  This rule
> > applys even more when said subsystem has a greater scope within the
> > source tree in question.
> >
> > Anyways, if we are going to compare SCM distributed vs. centralized
> > workflow in terms of kernel projects, lets please at least compare
> > apples to apples here.
> >
> > Best,
> >
> > --nab
> 
> I've been trying to keep my 2 cents out of this, as I am merely an
> SCST user.  Both of you have valid criticisms; the choice of SCM is
> not one of them.  It is nit-picking at best, especially when SCST
> could switch to git easily if they so desired.
> 
> Seven years ago, would you be pushing BitKeeper?
> 

Hi Mark,

I will always be advocating using the best tool for the job in any given
situation.  So absoulutely, I would have picked bitkeeper over tarballs
any day of the week 7 years ago, or over SVN if it had existed back
then.

But again, I think it's an important point that git is a tool that was
made explictly for the linux kernel workflow.  Why would a new subsystem
maintainer is participates in the kernel workflow ever use anything
besides git at this point..?

And sorry, but considering the obvious advantages in terms of workflow
speed and flexibility that git brings to the table for a subsystem
maintainer, calling the choise of SCM a nit-pick item demonstrates a
level certain level of inexperience wrt to mainline kernel workflow.
Which is perfectly OK, but if you really want to understand the issues
at hand in a distributed vs. centrailized SCM model, I strongly suggest
you watch Linus's talk as well.

Best,

--nab

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Mark Deneen]

> Hi Mark,
>
> I will always be advocating using the best tool for the job in any given
> situation.  So absoulutely, I would have picked bitkeeper over tarballs
> any day of the week 7 years ago, or over SVN if it had existed back
> then.

I can't say that I agree with this.  SVN existed, along with many
other open source choices -- the choice of BitKeeper was a mistake.

> But again, I think it's an important point that git is a tool that was
> made explictly for the linux kernel workflow.  Why would a new subsystem
> maintainer is participates in the kernel workflow ever use anything
> besides git at this point..?

Look, I'm not saying that I dislike git.  I use it as my SCM here.
However, git was in its infancy (or not even around) when SCST was
started.  It's not like they had a proprietary vendor go cold turkey
on them, forcing everyone to another solution.

> And sorry, but considering the obvious advantages in terms of workflow
> speed and flexibility that git brings to the table for a subsystem
> maintainer, calling the choise of SCM a nit-pick item demonstrates a
> level certain level of inexperience wrt to mainline kernel workflow.
> Which is perfectly OK, but if you really want to understand the issues
> at hand in a distributed vs. centrailized SCM model, I strongly suggest
> you watch Linus's talk as well.
>
> Best,
>
> --nab

I'm still calling it a nit-pick.  Vlad could switch to git in a short
amount of time if he felt so compelled.  This is like saying that the
quality of a car is based on the style of garage it is parked in.

Kind Regards,
Mark Deneen

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Nicholas A. Bellinger]

On Sun, 2010-09-05 at 20:58 -0400, Mark Deneen wrote:
> > Hi Mark,
> >
> > I will always be advocating using the best tool for the job in any given
> > situation.  So absoulutely, I would have picked bitkeeper over tarballs
> > any day of the week 7 years ago, or over SVN if it had existed back
> > then.
> 
> I can't say that I agree with this.  SVN existed, along with many
> other open source choices -- the choice of BitKeeper was a mistake.
> 

Bitkeeper taught Linus by his own admission that there was actually a
reason to using a SCM for the kernel to begin with, and helped drive
some early git design princables which he also briefly mentioned in the
google git talk.

So I hardly consider this a mistake looking at it from a historical
perspective.

> > But again, I think it's an important point that git is a tool that was
> > made explictly for the linux kernel workflow.  Why would a new subsystem
> > maintainer is participates in the kernel workflow ever use anything
> > besides git at this point..?
> 
> Look, I'm not saying that I dislike git.  I use it as my SCM here.
> However, git was in its infancy (or not even around) when SCST was
> started.  It's not like they had a proprietary vendor go cold turkey
> on them, forcing everyone to another solution.

I am really sorry to hear about SCST's bad timing wrt to the evolution
of git, but I hardly see this as an acceptable excuse for poor mainline
workflow.

> 
> > And sorry, but considering the obvious advantages in terms of workflow
> > speed and flexibility that git brings to the table for a subsystem
> > maintainer, calling the choise of SCM a nit-pick item demonstrates a
> > level certain level of inexperience wrt to mainline kernel workflow.
> > Which is perfectly OK, but if you really want to understand the issues
> > at hand in a distributed vs. centrailized SCM model, I strongly suggest
> > you watch Linus's talk as well.
> >
> > Best,
> >
> > --nab
> 
> I'm still calling it a nit-pick.  Vlad could switch to git in a short
> amount of time if he felt so compelled.  This is like saying that the
> quality of a car is based on the style of garage it is parked in.
> 

Well, if we are going to start talking about car analogies, then I have
one for you.. 8-)

Using a centralized SCM for kernel subsystem workflow in the year 2010
in akin to trying to make a modification to a 18,000 RPM capable engine
in a Ferrari F1 (eg: Linux Kernel), tuned to run at the *highest* levels
of international competition (eg: LKML).  But instead of using the tools
(git) that where explictely designed the F1 engine by it's creator (eg:
Linus aka Enzo Ferrari), you end trying to adjust your F1 engine's
killowatt per litre displacement output using a broken FM tuner knob and
rusty spare tire jack from a 79' Ford Pinto.

Best,

--nab

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Dmitry Torokhov]

On Sun, Sep 05, 2010 at 04:59:54PM -0700, Nicholas A. Bellinger wrote:
> On Sun, 2010-09-05 at 16:41 -0700, Dmitry Torokhov wrote:
> > On Sun, Sep 05, 2010 at 02:50:47PM -0700, Nicholas A. Bellinger wrote:
> > > Anyways, if we are going to compare SCM distributed vs. centralized
> > > workflow in terms of kernel projects, lets please at least compare
> > > apples to apples here.
> > > 
> > 
> > No, we should not be comparing SCMs at all here but rather 2 competing
> > implementations based on quality of the code. You tried to bring SMC
> > angle in and I am saying that it is BS.
> > 
> 
> Again, without getting into another pointless flamewar,  I think the
> main point here is that a open source project using a distributed
> workflow (like git) has major advantages when it comes to working with a
> larger group of developers than a centralized model (like SVN) does.
> 
> Because being a subsystem maintainer typically involves this type of
> complex workflow involving lots of different parties, git is a tool that
> was created (originally) expressely for a kernel workflow, and for those
> types of people it works really, really well.
> 

You may be surprised but I am aware of subsystem maintainer workflow. I
also am aware that there is not a single workflow and that it varies by
maintainer. For some git is indeed the best tool but others (and I
named a few) might prefer something different.

Once again: should we declare all subsystems that do not use git as
primary SCM be inferior and drop them from the kernel?

> So, please understand that code and project workflow is only one of the
> reasons why TCM/LIO v4 was selected over SCST.   I invite you to take a
> closer look at the RFC Code that has been posted last week if you want
> to get into the nitty-gritty techinical details, which this thread has
> thus far been avoiding.

Yes, I agree that it would be much more productive if you concentrated
on technical details when answering Vlad's questions rather than
branching into immaterial argument over SCM choice.

-- 
Dmitry

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Dmitry Torokhov]

On Sun, Sep 05, 2010 at 05:12:19PM -0700, Nicholas A. Bellinger wrote:
> On Sun, 2010-09-05 at 19:13 -0400, Mark Deneen wrote:
> > On Sun, Sep 5, 2010 at 5:50 PM, Nicholas A. Bellinger
> > <nab@linux-iscsi.org> wrote:
> > >
> > > I think the difference between what Linus says and what he actually
> > > means in the above video could be easily misinterpreted, well especially
> > > for some non-english speaking folks.  But I am certainly glad to see
> > > that a russian translation is also available for this google git talk
> > > for those who really want try to understand his reasons (and technical
> > > requirements) for what he is saying.
> > >
> > > So as to the specifics about why git is really the only right SCM choice
> > > for mainline target mode maintainership, it really all comes down to
> > > workflow.  Does your SCM allow other people to easily and without-pain
> > > track your upstream subsystem tree changes and 'rebase' as necessary for
> > > their patch series you make improvements..?   If we are talking about
> > > say, a single standalone driver being developed against mainline, then
> > > sure using a SCM like CVS or SVN is perfectly acceptable when you push
> > > to someone upstream like gregkh or akpm via email patch attachments.
> > >
> > > However, if we are talking about a subsystem maintainer workflow that
> > > requires many different people to track your subsystem tree for their
> > > own drivers and new feature bits, not being able to keep local branches
> > > for these level developers makes their life excruciatingly painful and
> > > unpleasent as they attempt to pull new upstream changes, especially when
> > > the speed of new upstream development is moving quickly.  This rule
> > > applys even more when said subsystem has a greater scope within the
> > > source tree in question.
> > >
> > > Anyways, if we are going to compare SCM distributed vs. centralized
> > > workflow in terms of kernel projects, lets please at least compare
> > > apples to apples here.
> > >
> > > Best,
> > >
> > > --nab
> > 
> > I've been trying to keep my 2 cents out of this, as I am merely an
> > SCST user.  Both of you have valid criticisms; the choice of SCM is
> > not one of them.  It is nit-picking at best, especially when SCST
> > could switch to git easily if they so desired.
> > 
> > Seven years ago, would you be pushing BitKeeper?
> > 
> 
> Hi Mark,
> 
> I will always be advocating using the best tool for the job in any given
> situation.  So absoulutely, I would have picked bitkeeper over tarballs
> any day of the week 7 years ago, or over SVN if it had existed back
> then.
> 
> But again, I think it's an important point that git is a tool that was
> made explictly for the linux kernel workflow.  Why would a new subsystem
> maintainer is participates in the kernel workflow ever use anything
> besides git at this point..?
> 
> And sorry, but considering the obvious advantages in terms of workflow
> speed and flexibility that git brings to the table for a subsystem
> maintainer, calling the choise of SCM a nit-pick item demonstrates a
> level certain level of inexperience wrt to mainline kernel workflow.

Will you please stop being condescending? Yes, it is nit-picking. Many
subsystems and features are being developed using patch series stored
not in git but somewhere else. Have you noticed that akpm uses his own
patch utils because git is not the best tool _for him_. Git is perfect
for Linus's and DaveM workflows (as they in turn do pulls), whereas
maintainers that prefer patch-based submissions may or may not use git,
depending on their preferences.

The choice between 2 implementations should be based purely on design and
code quality (with maintainer being reasonable and flexible enough for
additional points).

-- 
Dmitry

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[James Bottomley]

On Sun, 2010-09-05 at 16:59 -0700, Nicholas A. Bellinger wrote:
> On Sun, 2010-09-05 at 16:41 -0700, Dmitry Torokhov wrote:
> > On Sun, Sep 05, 2010 at 02:50:47PM -0700, Nicholas A. Bellinger wrote:
> > > Anyways, if we are going to compare SCM distributed vs. centralized
> > > workflow in terms of kernel projects, lets please at least compare
> > > apples to apples here.
> > > 
> > 
> > No, we should not be comparing SCMs at all here but rather 2 competing
> > implementations based on quality of the code. You tried to bring SMC
> > angle in and I am saying that it is BS.
> > 
> 
> Again, without getting into another pointless flamewar,  I think the
> main point here is that a open source project using a distributed
> workflow (like git) has major advantages when it comes to working with a
> larger group of developers than a centralized model (like SVN) does.
> 
> Because being a subsystem maintainer typically involves this type of
> complex workflow involving lots of different parties, git is a tool that
> was created (originally) expressely for a kernel workflow, and for those
> types of people it works really, really well.

Oh, for god's sake children.  Why does every LIO vs SCST discussion turn
into a pointless flameware over stuff no-one really cares about?  If
none of you has anything substantive to say: don't say it.

Since patches into SCSI go over the mailing list for review and
integration (and running checkpatch.pl on ... this would be a hint), I
don't really give a toss how they're generated.

> So, please understand that code and project workflow is only one of the
> reasons why TCM/LIO v4 was selected over SCST.

It isn't yet ... your code still has to be reviewed properly.  My
preferred reviewer is currently honing his skills on a diet of raw beef
in Argentina, but hopefully he'll get around to it shortly.

James

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Bart Van Assche]

On Mon, Sep 6, 2010 at 12:39 PM, James Bottomley
<James.Bottomley@suse.de> wrote:
>
> [ ... ]
>
> > So, please understand that code and project workflow is only one of the
> > reasons why TCM/LIO v4 was selected over SCST.
>
> It isn't yet ... your code still has to be reviewed properly.  My
> preferred reviewer is currently honing his skills on a diet of raw beef
> in Argentina, but hopefully he'll get around to it shortly.

The SCST developers and the SCST users will definitely appreciate it
if review efforts will be spread evenly over both projects.

Bart.

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Bart Van Assche]

On Mon, Sep 6, 2010 at 12:39 PM, James Bottomley
<James.Bottomley@suse.de> wrote:
>
> [ ... ]
>
> > So, please understand that code and project workflow is only one of the
> > reasons why TCM/LIO v4 was selected over SCST.
>
> It isn't yet ... your code still has to be reviewed properly.  My
> preferred reviewer is currently honing his skills on a diet of raw beef
> in Argentina, but hopefully he'll get around to it shortly.

The SCST developers and the SCST users will definitely appreciate it
if review efforts will be spread evenly over both projects.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[James Bottomley]

On Mon, 2010-09-06 at 13:02 +0200, Bart Van Assche wrote:
> On Mon, Sep 6, 2010 at 12:39 PM, James Bottomley
> <James.Bottomley@suse.de> wrote:
> >
> > [ ... ]
> >
> > > So, please understand that code and project workflow is only one of the
> > > reasons why TCM/LIO v4 was selected over SCST.
> >
> > It isn't yet ... your code still has to be reviewed properly.  My
> > preferred reviewer is currently honing his skills on a diet of raw beef
> > in Argentina, but hopefully he'll get around to it shortly.
> 
> The SCST developers and the SCST users will definitely appreciate it
> if review efforts will be spread evenly over both projects.

SCST isn't at the starting gate yet, so there's nothing currently to
review.

James

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

James Bottomley, on 09/06/2010 03:27 PM wrote:
> On Mon, 2010-09-06 at 13:02 +0200, Bart Van Assche wrote:
>> On Mon, Sep 6, 2010 at 12:39 PM, James Bottomley
>> <James.Bottomley@suse.de>  wrote:
>>>
>>> [ ... ]
>>>
>>>> So, please understand that code and project workflow is only one of the
>>>> reasons why TCM/LIO v4 was selected over SCST.
>>>
>>> It isn't yet ... your code still has to be reviewed properly.  My
>>> preferred reviewer is currently honing his skills on a diet of raw beef
>>> in Argentina, but hopefully he'll get around to it shortly.
>>
>> The SCST developers and the SCST users will definitely appreciate it
>> if review efforts will be spread evenly over both projects.
>
> SCST isn't at the starting gate yet, so there's nothing currently to
> review.

Hmm, there are patches for review for several months 
(http://lkml.org/lkml/2010/4/13/146). Nothing major changed since that time.

But we are going to send patches for the latest code this week anyway.

Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Chetan Loke]

On Thu, Sep 2, 2010 at 4:25 PM, Nicholas A. Bellinger
<nab@linux-iscsi.org> wrote:
> I really have no idea what you are talking about 'behind closed doors'..
>
> Have you not been watching the amount of TCM/LIO patches on the
> linux-scsi list for the last 18 months..?
>

Please don't make me go public. I'm trying to keep my mouth shut
purely out of NDA. Inspite of that I've mentioned it publicly in my
last emails that the decision was made ~9 months ago? Someone told me
'LIO will be merged eventually and NOT scst'. This ain't open-source
development as we know it.

Also there are exactly 'ZERO' users/developers yelling 'NOT' to
include scst. So please stop your PR about LIO. Since no target
subsystems were ever chosen, I would think that LIO's list should have
some meat, no?And by now, the whole community knows that you have near
to zero meaningful/technical conversations on LIO's mailing list.  So
I'm not sure when you talk about your git workflow and your patches,
what you are actually talking about.


>> >>> It is obvious to even an casual observer from watching the TCM/LIO patch
>> >>> series that have been flying across the linux-scsi wire the last 24
>> >>> months that the major features (including PR and ALUA, and new fabric
>> >>> module drivers) have been developed individual feature bit by feature
>> >>> bit using a distributed git workflow in a bisectable manner.  Each
>> >>> series was produced in such a manner that each patch could be reviewed
>> >>> individually by those interested parties.

Nothing needs to bisected. Just go to scst's svn repo and see the
checkin-flow over there if you want to satisfy your thirst. There are
no rules defined whatsoever that says that the patches need to follow
git-workflow. Since you are not an official kernel subsystem
maintainer no need to send you patches to get a buy-in.
Patches will be sent to James B and the scsi/kernel mailing list.


> Again, I suggest you watch the Russian translation of that talk on
> youtube and really understand what he means, aside from the insults to
> subversion users.
>

svn is more than capable for what it is supposed to do. So no need to
tutor us on the source-control.


> 1) a userspace library in intrepted languages and 2) create high level
> applications using said userspace libraries that allow compatiblity to
> be contained in the lib below.
>

An average user doesn't implement storage blobs. So I don't want to
worry about what they would think.This is your useless LIO PR and not
a missing feature in scst.


> Uhhh, I don't think so.  Aside from your obvious project workflow
> issues, the fact that SCST completly lacks a proper user-space driven
> representation of parent / child relationships between kernel level data
> structures in a fabric independent manner, while still allow for fabric
> dependent groups and attributes is a most definately show-stopper for
> me.
>

Show stopper for you? Sorry but you seem to be self-proclaimed
judge/maintainer pushing/speaking your vendor's/customer's agenda.

Ever noticed how screwed up the fc representations were? Take NPIV for
instance. No one laid down rules at that time? And just because your
so called vendor's/customer's may now be used to your LIO-way of doing
things, doesn't mean the whole community should be forced to do that.


> --nab

Chetan Loke
<English ain't my first language. I don't need any translation because
I speak one language - It's, Linux! The language and choice of the GNU
generation>

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Chetan Loke]

On Thu, Sep 2, 2010 at 4:25 PM, Nicholas A. Bellinger
<nab@linux-iscsi.org> wrote:
> I really have no idea what you are talking about 'behind closed doors'..
>
> Have you not been watching the amount of TCM/LIO patches on the
> linux-scsi list for the last 18 months..?
>

Please don't make me go public. I'm trying to keep my mouth shut
purely out of NDA. Inspite of that I've mentioned it publicly in my
last emails that the decision was made ~9 months ago? Someone told me
'LIO will be merged eventually and NOT scst'. This ain't open-source
development as we know it.

Also there are exactly 'ZERO' users/developers yelling 'NOT' to
include scst. So please stop your PR about LIO. Since no target
subsystems were ever chosen, I would think that LIO's list should have
some meat, no?And by now, the whole community knows that you have near
to zero meaningful/technical conversations on LIO's mailing list.  So
I'm not sure when you talk about your git workflow and your patches,
what you are actually talking about.


>> >>> It is obvious to even an casual observer from watching the TCM/LIO patch
>> >>> series that have been flying across the linux-scsi wire the last 24
>> >>> months that the major features (including PR and ALUA, and new fabric
>> >>> module drivers) have been developed individual feature bit by feature
>> >>> bit using a distributed git workflow in a bisectable manner.  Each
>> >>> series was produced in such a manner that each patch could be reviewed
>> >>> individually by those interested parties.

Nothing needs to bisected. Just go to scst's svn repo and see the
checkin-flow over there if you want to satisfy your thirst. There are
no rules defined whatsoever that says that the patches need to follow
git-workflow. Since you are not an official kernel subsystem
maintainer no need to send you patches to get a buy-in.
Patches will be sent to James B and the scsi/kernel mailing list.


> Again, I suggest you watch the Russian translation of that talk on
> youtube and really understand what he means, aside from the insults to
> subversion users.
>

svn is more than capable for what it is supposed to do. So no need to
tutor us on the source-control.


> 1) a userspace library in intrepted languages and 2) create high level
> applications using said userspace libraries that allow compatiblity to
> be contained in the lib below.
>

An average user doesn't implement storage blobs. So I don't want to
worry about what they would think.This is your useless LIO PR and not
a missing feature in scst.


> Uhhh, I don't think so.  Aside from your obvious project workflow
> issues, the fact that SCST completly lacks a proper user-space driven
> representation of parent / child relationships between kernel level data
> structures in a fabric independent manner, while still allow for fabric
> dependent groups and attributes is a most definately show-stopper for
> me.
>

Show stopper for you? Sorry but you seem to be self-proclaimed
judge/maintainer pushing/speaking your vendor's/customer's agenda.

Ever noticed how screwed up the fc representations were? Take NPIV for
instance. No one laid down rules at that time? And just because your
so called vendor's/customer's may now be used to your LIO-way of doing
things, doesn't mean the whole community should be forced to do that.


> --nab

Chetan Loke
<English ain't my first language. I don't need any translation because
I speak one language - It's, Linux! The language and choice of the GNU
generation>
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Greg KH]

On Sun, Sep 05, 2010 at 02:50:47PM -0700, Nicholas A. Bellinger wrote:
> So as to the specifics about why git is really the only right SCM choice
> for mainline target mode maintainership, it really all comes down to
> workflow.  Does your SCM allow other people to easily and without-pain
> track your upstream subsystem tree changes and 'rebase' as necessary for
> their patch series you make improvements..?   If we are talking about
> say, a single standalone driver being developed against mainline, then
> sure using a SCM like CVS or SVN is perfectly acceptable when you push
> to someone upstream like gregkh or akpm via email patch attachments.

I actually use git to manage my quilt patch tree so that others can work
with me.  Don't confuse a scm with a patch management tool, they are two
vastly different things and have no relation to each other here.

And this sub-topic has NOTHING to do with the main issue here, so I
don't know why you are arguing over it.  Please take it elsewhere.

thanks,

greg k-h

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

James Bottomley, on 09/06/2010 02:39 PM wrote:
>>>> Anyways, if we are going to compare SCM distributed vs. centralized
>>>> workflow in terms of kernel projects, lets please at least compare
>>>> apples to apples here.
>>>
>>> No, we should not be comparing SCMs at all here but rather 2 competing
>>> implementations based on quality of the code. You tried to bring SMC
>>> angle in and I am saying that it is BS.
>>
>> Again, without getting into another pointless flamewar,  I think the
>> main point here is that a open source project using a distributed
>> workflow (like git) has major advantages when it comes to working with a
>> larger group of developers than a centralized model (like SVN) does.
>>
>> Because being a subsystem maintainer typically involves this type of
>> complex workflow involving lots of different parties, git is a tool that
>> was created (originally) expressely for a kernel workflow, and for those
>> types of people it works really, really well.
>
> Oh, for god's sake children.  Why does every LIO vs SCST discussion turn
> into a pointless flameware over stuff no-one really cares about?  If
> none of you has anything substantive to say: don't say it.

James, sorry, but you can't blame us. I keep asking for clear rules and 
don't receive much. So, there are speculations and pseudo-rules, which 
sometimes go to the absurd, as in this SVN vs Git case. No surprise 
then, that people have risen against this absurd (thanks a lot to them 
for support!).

Frankly, in all the situation around Linux SCSI targets I for quite a 
long time feeling myself as a hero of a Kafka novel. Supposed to be 
goals are to have the best code doing its job the best, but on practice 
nobody cares about technical arguments and figuring out which subsystem 
is the best. Instead, everything lives it own incomprehensible life, 
where doesn't matter what you are doing, all already long ago decided 
behind your back and you will never be told why. All accurate statements 
not heard or, at best, called "handwaving", but dirty public opinion 
manipulations based on half- and less-than-half- truth have very warn 
welcome. This atmosphere is unhealthy, really.

> Since patches into SCSI go over the mailing list for review and
> integration (and running checkpatch.pl on ... this would be a hint), I
> don't really give a toss how they're generated.

Great to hear that! Thanks!

Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

Nicholas A. Bellinger, on 09/03/2010 12:25 AM wrote:
>>> Again, I still have no idea what you are trying to conjour up.  The
>>> passthrough for STGT compatibility with TCM_Loop via SG_IO and BSG has
>>> already been merged by Tomo-san into tgt.git.  Futhermore, we are using
>>> the same TCM_Loop high level multi-fabric WWPN emulation together with
>>> new (and old) QEMU HBA emulation into KVM guest using SG_IO and BSG as
>>> well.
>>
>> The patches were good, but don't mislead people about that. For sake of
>> clearness, what you called "the passthrough for STGT compatibility with
>> TCM_Loop via SG_IO and BSG" has nothing with LIO/TCM_Loop. What merged
>> fixed problems STGT had in the pass-through implementation. It at the
>> same time fixed problems with SCST's scst_local, so similarly can be
>> called "the passthrough for STGT compatibility with scst_local via SG_IO
>> and BSG".
>>
>
> Uh, I don't think TCM_Loop and scst_local are exactly comparable at this
> point.  TCM_Loop supports high level multi-fabric WWPN emulation that
> allows it to work transparently with inter-nexus SPC-4 PR operation.
> Therefore, using TCM_Loop, STGT userspace fabrics now support both PR
> and ALUA emulation from TCM_Loop LUNs.  This is done at struct
> config_group_ops->make_group() time to report $PROTO_IDENT and the
> necessary WWPN info to TCM Core logic.  TCM_Loop also uses the fabric
> indepent configfs handlers for pretty much all of it's control plane
> code.

TCM_Loop and scst_local are exactly comparable and do absolutely the 
same thing. If you see any difference, please be precise. Internal 
implementation doesn't matter as soon as it doesn't affect the 
functionality STGT uses.

>> I have no doubts in benefits to TCM. But I have asked about benefits
>> _for an average user_. STGT already can do what you listed above.
>>
>
> Again, from above how it benefits users:
>
> 1) a userspace library in intrepted languages and 2) create high level
> applications using said userspace libraries that allow compatiblity to
> be contained in the lib below.
>
> 2) create high level applications using said userspace libraries that
> allow compatiblity to be contained in the lib.

All those can be done for STGT. LIO isn't needed for that. So, value for 
STGT users still not questionable to me.

Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

Nicholas A. Bellinger, on 09/03/2010 12:25 AM wrote:
>>> Again, I still have no idea what you are trying to conjour up.  The
>>> passthrough for STGT compatibility with TCM_Loop via SG_IO and BSG has
>>> already been merged by Tomo-san into tgt.git.  Futhermore, we are using
>>> the same TCM_Loop high level multi-fabric WWPN emulation together with
>>> new (and old) QEMU HBA emulation into KVM guest using SG_IO and BSG as
>>> well.
>>
>> The patches were good, but don't mislead people about that. For sake of
>> clearness, what you called "the passthrough for STGT compatibility with
>> TCM_Loop via SG_IO and BSG" has nothing with LIO/TCM_Loop. What merged
>> fixed problems STGT had in the pass-through implementation. It at the
>> same time fixed problems with SCST's scst_local, so similarly can be
>> called "the passthrough for STGT compatibility with scst_local via SG_IO
>> and BSG".
>>
>
> Uh, I don't think TCM_Loop and scst_local are exactly comparable at this
> point.  TCM_Loop supports high level multi-fabric WWPN emulation that
> allows it to work transparently with inter-nexus SPC-4 PR operation.
> Therefore, using TCM_Loop, STGT userspace fabrics now support both PR
> and ALUA emulation from TCM_Loop LUNs.  This is done at struct
> config_group_ops->make_group() time to report $PROTO_IDENT and the
> necessary WWPN info to TCM Core logic.  TCM_Loop also uses the fabric
> indepent configfs handlers for pretty much all of it's control plane
> code.

TCM_Loop and scst_local are exactly comparable and do absolutely the 
same thing. If you see any difference, please be precise. Internal 
implementation doesn't matter as soon as it doesn't affect the 
functionality STGT uses.

>> I have no doubts in benefits to TCM. But I have asked about benefits
>> _for an average user_. STGT already can do what you listed above.
>>
>
> Again, from above how it benefits users:
>
> 1) a userspace library in intrepted languages and 2) create high level
> applications using said userspace libraries that allow compatiblity to
> be contained in the lib below.
>
> 2) create high level applications using said userspace libraries that
> allow compatiblity to be contained in the lib.

All those can be done for STGT. LIO isn't needed for that. So, value for 
STGT users still not questionable to me.

Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Nicholas A. Bellinger]

On Tue, 2010-09-07 at 01:47 +0400, Vladislav Bolkhovitin wrote:
> James Bottomley, on 09/06/2010 02:39 PM wrote:
> >>>> Anyways, if we are going to compare SCM distributed vs. centralized
> >>>> workflow in terms of kernel projects, lets please at least compare
> >>>> apples to apples here.
> >>>
> >>> No, we should not be comparing SCMs at all here but rather 2 competing
> >>> implementations based on quality of the code. You tried to bring SMC
> >>> angle in and I am saying that it is BS.
> >>
> >> Again, without getting into another pointless flamewar,  I think the
> >> main point here is that a open source project using a distributed
> >> workflow (like git) has major advantages when it comes to working with a
> >> larger group of developers than a centralized model (like SVN) does.
> >>
> >> Because being a subsystem maintainer typically involves this type of
> >> complex workflow involving lots of different parties, git is a tool that
> >> was created (originally) expressely for a kernel workflow, and for those
> >> types of people it works really, really well.
> >
> > Oh, for god's sake children.  Why does every LIO vs SCST discussion turn
> > into a pointless flameware over stuff no-one really cares about?  If
> > none of you has anything substantive to say: don't say it.
> 
> James, sorry, but you can't blame us. I keep asking for clear rules and 
> don't receive much. So, there are speculations and pseudo-rules, which 
> sometimes go to the absurd, as in this SVN vs Git case. No surprise 
> then, that people have risen against this absurd (thanks a lot to them 
> for support!).
> 
> Frankly, in all the situation around Linux SCSI targets I for quite a 
> long time feeling myself as a hero of a Kafka novel. Supposed to be 
> goals are to have the best code doing its job the best, but on practice 
> nobody cares about technical arguments and figuring out which subsystem 
> is the best. Instead, everything lives it own incomprehensible life, 
> where doesn't matter what you are doing, all already long ago decided 
> behind your back and you will never be told why. All accurate statements 
> not heard or, at best, called "handwaving", but dirty public opinion 
> manipulations based on half- and less-than-half- truth have very warn 
> welcome. This atmosphere is unhealthy, really.

Sorry Vlad, but this is simply not the truth.  You have had ample time
to comment on the hundreds of TCM/LIO patches posted to linux-scsi and
lkml over the last years, but you have chosen never to comment on a
*single* one then, or even on a single one now of the dozens that have
been posted in the last 3 weeks while this thread has been lumbering
forward..

So at this point, I will once again to refrain from any non technical
interaction with yourself.  If you have geninue concerns about any of
the TCM/LIO v4 code, then I suggest that you and your devels make them
known from within threads containing [PATCH] and [RFC] tags, because I
will not be bothering with anything that does not contain comments on
creating new or improving existing design and code.

Best,

--nab

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[david]

On Mon, 6 Sep 2010, Nicholas A. Bellinger wrote:

> So at this point, I will once again to refrain from any non technical
> interaction with yourself.  If you have geninue concerns about any of
> the TCM/LIO v4 code, then I suggest that you and your devels make them
> known from within threads containing [PATCH] and [RFC] tags, because I
> will not be bothering with anything that does not contain comments on
> creating new or improving existing design and code.

I haven't seen any techinical discussion from you either.

all I see is how your favorite has been blessed as being the next thing to 
be included, even though it still needs work and Vlad's can't even be 
looked at.

that's hardly a technical discussion.

Vlad seems to be trying to document features of the different options, 
naturally he knows his option better than anyone else's. He offered to 
correct his listing with any information provided by anyone else, and 
instead of corrections to the feature lists, we just got accusations of 
handwaving and a reiteration that your favorite was selected in some 
meeting to be merged.

I would like to point out that over the years there have been many things 
that were expected to be merged that didn't get merged. Don't take any 
statement in any meeting to be a final decision. It doesn't matter how 
promising yourcode looked at that point in time, until it's merged you may 
find that it doesn't get merged. And even after it gets merged, if someone 
else has better code, their code may replace yours if the other code is 
better.

As a user, I would sure like to see more information about the two major 
choices and a lot less "this is what was picked at an invitation-only 
meeting where one option wasn't represented"

David Lang

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Dmitry Torokhov]

On Mon, Sep 06, 2010 at 02:55:35PM -0700, Nicholas A. Bellinger wrote:
> On Tue, 2010-09-07 at 01:47 +0400, Vladislav Bolkhovitin wrote:
> > James Bottomley, on 09/06/2010 02:39 PM wrote:
> > >>>> Anyways, if we are going to compare SCM distributed vs. centralized
> > >>>> workflow in terms of kernel projects, lets please at least compare
> > >>>> apples to apples here.
> > >>>
> > >>> No, we should not be comparing SCMs at all here but rather 2 competing
> > >>> implementations based on quality of the code. You tried to bring SMC
> > >>> angle in and I am saying that it is BS.
> > >>
> > >> Again, without getting into another pointless flamewar,  I think the
> > >> main point here is that a open source project using a distributed
> > >> workflow (like git) has major advantages when it comes to working with a
> > >> larger group of developers than a centralized model (like SVN) does.
> > >>
> > >> Because being a subsystem maintainer typically involves this type of
> > >> complex workflow involving lots of different parties, git is a tool that
> > >> was created (originally) expressely for a kernel workflow, and for those
> > >> types of people it works really, really well.
> > >
> > > Oh, for god's sake children.  Why does every LIO vs SCST discussion turn
> > > into a pointless flameware over stuff no-one really cares about?  If
> > > none of you has anything substantive to say: don't say it.
> > 
> > James, sorry, but you can't blame us. I keep asking for clear rules and 
> > don't receive much. So, there are speculations and pseudo-rules, which 
> > sometimes go to the absurd, as in this SVN vs Git case. No surprise 
> > then, that people have risen against this absurd (thanks a lot to them 
> > for support!).
> > 
> > Frankly, in all the situation around Linux SCSI targets I for quite a 
> > long time feeling myself as a hero of a Kafka novel. Supposed to be 
> > goals are to have the best code doing its job the best, but on practice 
> > nobody cares about technical arguments and figuring out which subsystem 
> > is the best. Instead, everything lives it own incomprehensible life, 
> > where doesn't matter what you are doing, all already long ago decided 
> > behind your back and you will never be told why. All accurate statements 
> > not heard or, at best, called "handwaving", but dirty public opinion 
> > manipulations based on half- and less-than-half- truth have very warn 
> > welcome. This atmosphere is unhealthy, really.
> 
> Sorry Vlad, but this is simply not the truth.  You have had ample time
> to comment on the hundreds of TCM/LIO patches posted to linux-scsi and
> lkml over the last years, but you have chosen never to comment on a
> *single* one then, or even on a single one now of the dozens that have
> been posted in the last 3 weeks while this thread has been lumbering
> forward..
> 
> So at this point, I will once again to refrain from any non technical
> interaction with yourself.  If you have geninue concerns about any of
> the TCM/LIO v4 code, then I suggest that you and your devels make them
> known from within threads containing [PATCH] and [RFC] tags, because I
> will not be bothering with anything that does not contain comments on
> creating new or improving existing design and code.
> 

I think this is somewhat backwards...

Vlad appears to be asserting that SCST is more feature-complete that LIO
at this point. It also seems that LIO is somewhat younger than SCST. So
at this point it might be interesting to see:

1. What are the shortcomings of SCST design compared to LIO and why LIO
developers chose to come with their own solution instead of
collaborating with SCST folks?

2. What features are missing from SCST that are currently available in
LIO?

Once this is sorted out and [most] everyone agrees that LIO is indeed
technically superior (even if maybe not as mature yet) solution, then it
would make sense to request SCST developers to go to file/line depth of
the review.

Thanks.

-- 
Dmitry

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Chetan Loke]

On Mon, Sep 6, 2010 at 8:44 PM, Dmitry Torokhov
<dmitry.torokhov@gmail.com> wrote:
> I think this is somewhat backwards...
>
> Vlad appears to be asserting that SCST is more feature-complete that LIO
> at this point. It also seems that LIO is somewhat younger than SCST. So
> at this point it might be interesting to see:
>
> 1. What are the shortcomings of SCST design compared to LIO and why LIO
> developers chose to come with their own solution instead of
> collaborating with SCST folks?
>
> 2. What features are missing from SCST that are currently available in
> LIO?
>
> Once this is sorted out and [most] everyone agrees that LIO is indeed
> technically superior (even if maybe not as mature yet) solution, then it
> would make sense to request SCST developers to go to file/line depth of
> the review.
>

I would also appreciate an overview or a block diagram of how things
work in LIO. I hope that's not too much to ask for? That way we can
compare/contrast how things work from 10,000 feet level.
I for one don't want to look at a single patch and comment -
i) Oh, change this variable here because it doesn't follow linux coding style.
ii) You dropped a lock in queue-cmd? Good. qla's driver has been doing
it for sometime, no? So you could have just looked at that LLDD.

Sorry, not trying to criticize anything but would like to offer my 2 cents too.

> --
> Dmitry

Chetan Loke

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Bart Van Assche]

On Tue, Sep 7, 2010 at 2:44 AM, Dmitry Torokhov
<dmitry.torokhov@gmail.com> wrote:
>
> [ ... ]
>
> Vlad appears to be asserting that SCST is more feature-complete that LIO
> at this point. It also seems that LIO is somewhat younger than SCST. So
> at this point it might be interesting to see:
>
> 1. What are the shortcomings of SCST design compared to LIO and why LIO
> developers chose to come with their own solution instead of
> collaborating with SCST folks?
>
> 2. What features are missing from SCST that are currently available in
> LIO?
>
> Once this is sorted out and [most] everyone agrees that LIO is indeed
> technically superior (even if maybe not as mature yet) solution, then it
> would make sense to request SCST developers to go to file/line depth of
> the review.

You seem to have missed the start of this thread. The design of SCST
is significantly more advanced than that of LIO, and it has already
been explained in this thread why
(http://www.spinics.net/lists/linux-scsi/msg45856.html).

Bart.

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Bart Van Assche]

On Tue, Sep 7, 2010 at 5:45 AM, Chetan Loke <chetanloke@gmail.com> wrote:
> [ ... ]
>
> I would also appreciate an overview or a block diagram of how things
> work in LIO. I hope that's not too much to ask for? That way we can
> compare/contrast how things work from 10,000 feet level.
> I for one don't want to look at a single patch and comment -
> i) Oh, change this variable here because it doesn't follow linux coding style.
> ii) You dropped a lock in queue-cmd? Good. qla's driver has been doing
> it for sometime, no? So you could have just looked at that LLDD.
>
> Sorry, not trying to criticize anything but would like to offer my 2 cents too.

I agree with you. Not only the code of the two projects should be
reviewed but their respective designs should be reviewed too.

Bart.

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Dmitry Torokhov]

On Tue, Sep 07, 2010 at 08:08:37AM +0200, Bart Van Assche wrote:
> On Tue, Sep 7, 2010 at 2:44 AM, Dmitry Torokhov
> <dmitry.torokhov@gmail.com> wrote:
> >
> > [ ... ]
> >
> > Vlad appears to be asserting that SCST is more feature-complete that LIO
> > at this point. It also seems that LIO is somewhat younger than SCST. So
> > at this point it might be interesting to see:
> >
> > 1. What are the shortcomings of SCST design compared to LIO and why LIO
> > developers chose to come with their own solution instead of
> > collaborating with SCST folks?
> >
> > 2. What features are missing from SCST that are currently available in
> > LIO?
> >
> > Once this is sorted out and [most] everyone agrees that LIO is indeed
> > technically superior (even if maybe not as mature yet) solution, then it
> > would make sense to request SCST developers to go to file/line depth of
> > the review.
> 
> You seem to have missed the start of this thread. The design of SCST
> is significantly more advanced than that of LIO, and it has already
> been explained in this thread why
> (http://www.spinics.net/lists/linux-scsi/msg45856.html).
> 

The question was directed to LIO folks as they appear to disagree with
this statement.

-- 
Dmitry

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Hannes Reinecke]

Bart Van Assche wrote:
> On Tue, Sep 7, 2010 at 2:44 AM, Dmitry Torokhov
> <dmitry.torokhov@gmail.com> wrote:
>> [ ... ]
>>
>> Vlad appears to be asserting that SCST is more feature-complete that LIO
>> at this point. It also seems that LIO is somewhat younger than SCST. So
>> at this point it might be interesting to see:
>>
>> 1. What are the shortcomings of SCST design compared to LIO and why LIO
>> developers chose to come with their own solution instead of
>> collaborating with SCST folks?
>>
>> 2. What features are missing from SCST that are currently available in
>> LIO?
>>
>> Once this is sorted out and [most] everyone agrees that LIO is indeed
>> technically superior (even if maybe not as mature yet) solution, then it
>> would make sense to request SCST developers to go to file/line depth of
>> the review.
> 
> You seem to have missed the start of this thread. The design of SCST
> is significantly more advanced than that of LIO, and it has already
> been explained in this thread why
> (http://www.spinics.net/lists/linux-scsi/msg45856.html).
> 
Hmm. That link seems to be misrouted. I was hoping for a design
overview of SCST there; however it's just a complain to
James Bottomley etc.

Care to send the correct link?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Bart Van Assche]

On Tue, Sep 7, 2010 at 8:29 AM, Hannes Reinecke <hare@suse.de> wrote:
>
> Bart Van Assche wrote:
> [ ... ]
> > You seem to have missed the start of this thread. The design of SCST
> > is significantly more advanced than that of LIO, and it has already
> > been explained in this thread why
> > (http://www.spinics.net/lists/linux-scsi/msg45856.html).
> >
> Hmm. That link seems to be misrouted. I was hoping for a design
> overview of SCST there; however it's just a complain to
> James Bottomley etc.
>
> Care to send the correct link?

An SCST design overview diagram together with SCST API documentation
can be found here: http://scst.sourceforge.net/scst_pg.html (sorry for
the language imperfections on this page, which should not affect
readability though).

Some distinctive features of SCST that give it its high bandwidth and
low latency but that are not mentioned on that page are:
* Multithreaded command processing, even for a single LUN.
* Much care has been taken to minimize command processing latency.
This is why a dedicated SGV cache has been implemented in SCST, which
is, as far as I know, a unique feature.

It is also important to know that the SCST project has a large user
base, and that the SCST users appreciate the proven stability and
reliability of the SCST software and also its excellent SCSI
conformance.

Bart.

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

Bart Van Assche, on 09/07/2010 10:45 AM wrote:
> On Tue, Sep 7, 2010 at 8:29 AM, Hannes Reinecke<hare@suse.de>  wrote:
>>
>> Bart Van Assche wrote:
>> [ ... ]
>>> You seem to have missed the start of this thread. The design of SCST
>>> is significantly more advanced than that of LIO, and it has already
>>> been explained in this thread why
>>> (http://www.spinics.net/lists/linux-scsi/msg45856.html).
>>>
>> Hmm. That link seems to be misrouted. I was hoping for a design
>> overview of SCST there; however it's just a complain to
>> James Bottomley etc.
>>
>> Care to send the correct link?
>
> An SCST design overview diagram together with SCST API documentation
> can be found here: http://scst.sourceforge.net/scst_pg.html (sorry for
> the language imperfections on this page, which should not affect
> readability though).

For iSCSI-like transports, which support immediate/unsolicited data, 
additional description of the processing flow is here.

In short, in this flow target drivers called by the SCST core 
(preprocessing_done() callback) after preprocessing of this command 
finished and the command has allocated buffer, so now the driver can 
continue processing of the command and transfer in its buffer 
immediate/unsolicited data, if there are any.

In details it is as the following:

After iSCSI-SCST received new command in scsi_cmnd_start() it calls 
scst_rx_cmd() to create scst_cmd then after it initialized necessary 
fields in it (CDB, task attribute, expected data direction, etc.) it 
calls scst_cmd_init_stage1_done() to notify SCST that scst_cmd 
initialization finished and its processing can be started. Then SCST 
parses SCSI CDB to determines data transfer length and direction (this 
is necessary to prevent various DoS'es from remote initiators like 
sending a command with wrong data direction; try this in the 
pass-through case and you will see what happens) and allocates necessary 
data buffer. From this point the following 2 scenarios are possible:

1. Regular path, when SCST does all the processing directly as simple 
function calls. SCST in the end of processing calls 
iscsi_preprocessing_done() callback, which sets scst_state and returns. 
Following the call stack the execution flow will return to 
scsi_cmnd_start() and continue there.

2. During processing SCST has to change context (switch to another 
thread). It can be necessary, e.g., to handle some management activity. 
Here everything is the same as in (1) above, but after 
scst_cmd_init_stage1_done() returned scsi_cmnd_start() returns 1 and 
further processing of this connection is suspended. Then 
iscsi_preprocessing_done() will resume it. Then process_read_io() will 
call cmnd_rx_continue().

Then iSCSI-SCST will do necessary checks and if all correct:

   - For READ commands and commands without data transfers, calls in 
iscsi_restart_cmnd() scst_restart_cmd() to tell SCST to start command's 
execution.

   - Fro WRITE commands it receives immediate data, sends the necessary 
R2T PDUs (send_r2t()), receives data for them and then calls 
scst_restart_cmd().

Then, after the command is finished, SCST returns to iSCSI-SCST status 
of the command's execution in iscsi_xmit_response() callback, which 
using send_data_rsp() makes the necessary Data-In PDUs and then either 
directly sends them using iscsi_try_local_processing()->iscsi_send() (to 
minimize latency) or passes to the write thread.

Other internal SCST processing is as shown on the "The commands 
processing flow" in http://scst.sourceforge.net/scst_pg.html.

Thanks,
Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

Nicholas A. Bellinger, on 09/07/2010 01:55 AM wrote:
>> Frankly, in all the situation around Linux SCSI targets I for quite a
>> long time feeling myself as a hero of a Kafka novel. Supposed to be
>> goals are to have the best code doing its job the best, but on practice
>> nobody cares about technical arguments and figuring out which subsystem
>> is the best. Instead, everything lives it own incomprehensible life,
>> where doesn't matter what you are doing, all already long ago decided
>> behind your back and you will never be told why. All accurate statements
>> not heard or, at best, called "handwaving", but dirty public opinion
>> manipulations based on half- and less-than-half- truth have very warn
>> welcome. This atmosphere is unhealthy, really.
>
> Sorry Vlad, but this is simply not the truth.  You have had ample time
> to comment on the hundreds of TCM/LIO patches posted to linux-scsi and
> lkml over the last years, but you have chosen never to comment on a
> *single* one then, or even on a single one now of the dozens that have
> been posted in the last 3 weeks while this thread has been lumbering
> forward..

Actually, http://scst.sourceforge.net/comparison.html is the result of a 
very careful and comprehensive review of your code, most likely, the 
best it has ever had. Until high level questions are answered (Dmitry 
perfectly summarized them), there's no point to go into low level 
details reviewing particular patches.

Vlad

Re: [Scst-devel] Fwd: Re: linuxcon 2010...

[Vladislav Bolkhovitin]

Dmitry Torokhov, on 09/07/2010 04:44 AM wrote:
>> So at this point, I will once again to refrain from any non technical
>> interaction with yourself.  If you have geninue concerns about any of
>> the TCM/LIO v4 code, then I suggest that you and your devels make them
>> known from within threads containing [PATCH] and [RFC] tags, because I
>> will not be bothering with anything that does not contain comments on
>> creating new or improving existing design and code.
>>
>
> I think this is somewhat backwards...
>
> Vlad appears to be asserting that SCST is more feature-complete that LIO
> at this point. It also seems that LIO is somewhat younger than SCST. So
> at this point it might be interesting to see:
>
> 1. What are the shortcomings of SCST design compared to LIO and why LIO
> developers chose to come with their own solution instead of
> collaborating with SCST folks?
>
> 2. What features are missing from SCST that are currently available in
> LIO?
>
> Once this is sorted out and [most] everyone agrees that LIO is indeed
> technically superior (even if maybe not as mature yet) solution, then it
> would make sense to request SCST developers to go to file/line depth of
> the review.

Those are exactly the questions trying to hear answers on which I'm 
hitting the wall in past time.

Thanks!
Vlad

Re: Linux I/O subsystem performance (was: linuxcon 2010...)

[Chris Worley]

On Tue, Aug 24, 2010 at 2:31 PM, Chris Worley <worleys@gmail.com> wrote:
> On Tue, Aug 24, 2010 at 11:43 AM, Vladislav Bolkhovitin <vst@vlnb.net> wrote:
>> Pasi Kärkkäinen, on 08/24/2010 11:25 AM wrote:
>>>
>>> On Mon, Aug 23, 2010 at 02:03:26PM -0400, Chetan Loke wrote:
>>>>
>>>> I actually received 3+ off-post emails asking whether I was talking
>>>> about initiator or target in the 100K IOPS case below and what did I
>>>> mean by the ACKs.
>>>> I was referring to the 'Initiator' side.
>>>> ACKs == When scsi-ML down-calls the LLD via the queue-command, process
>>>> the sgl's(if you like) and then trigger the scsi_done up-call path.
>>>>
>>>
>>> Uhm, Intel and Microsoft demonstrated over 1 million IOPS
>>> using software iSCSI and a single 10 Gbit Ethernet NIC (Intel 82599).
>>>
>>> How come there is such a huge difference? What are we lacking in Linux?
>>
>> I also have an impression that Linux I/O subsystem has some performance
>> problems. For instance, in one recent SCST performance test only 8 Linux
>> initiators with fio as a load generator were able to saturate a single SCST
>> target with dual IB cards (SRP) on 4K AIO direct accesses over an SSD
>> backend. This rawly means that any initiator took several times (8?) more
>> processing time than the target.
>
> While I can't tell you where the bottlenecks are, I can share some
> performance numbers...

I've been asked to share more details of the single SRP initiator
case, comparing Windows to Linux...

The configurations tested are represented by four digits separated by dashes:

- The number of initiators used in the test (always one in this case).
- The number of target ports used.
- The number of initiator ports used.
- the number of drives used.

SRP Upstream Initiator

                            1-1-1-1 1-1-1-2  1-2-2-2   1-1-1-4
1-2-2-4   1-1-1-8  1-2-2-8
Random Write    122880  141568  206592  144384  163840  141824  165376
30/70 R/W mix     72113   123136 144640  143616  163072  145920  163584
70/30 R/W mix     55938     91392 114176  135680  156160  145920  162304
Random Read     50688     78336 107008  121600  149760  143872  161536

SRP Windows Initiator

                           1-?-1-1 1-?-1-2   1-?-2-2  1-?-1-4  1-?-2-4
 1-?-1-8    1-?-2-8
Random Write     57774  116738  114464  146972  202891                  221819
30/70 R/W mix    49719    95697    97831  154328  181221                  227786
70/30 R/W mix    45242    90694    89559  167341  176178                  244661
Random Read     48016    94867   92984  178227  183631                  257449

Note that the question marks are where I'm not sure how Windows is
using the second target port... in Linux, you select the target port
from the initiator, but there's no such option in Windows, so the
target port could be used in those cases.  The 1-1-1-8 case is where I
tried to force it to use just one target port (by disabling the target
port), and Windows wouldn't do any I/O at all.

Chris
>
> 4 initiators can get >600K random 4KB IOPS off a single target...
> which is ~150% of what the Emulex/Intel/Microsoft results show using 8
> targets at 4KB (their 1M IOPS was at 512 byte blocks, which is not a
> realistic test point) here:
>
> http://itbrandpulse.com/Documents/Test2010001%20-%20The%20Sun%20Rises%20on%20CNAs%20Test%20Report.pdf
>
> The blog referenced earlier used 10 targets... and I'm not sure how
> many 10G ports per target.
>
> In general, my target seems capable of 65% the local small-block
> random write performance over IB,  and 85% the local small-block
> random read performance.  For large block performance, ~95% efficiency
> is easily achievable, read or write (i.e. 5.6GB/s over fabric, where
> 6GB/s is achievable on the drives locally at 1MB random blocks).
> These small-block efficiencies are achievable only when tested with
> multiple initiators.
>
> The single initiator is only capable of <150K 4KB IOPS... but gets
> full bandwidth w/ larger blocks.
>
> If I were to chose my problem, target or initiator bottleneck, I'd
> certainly rather have an initiator bottleneck rather than Microsoft's
> target bottleneck.
>
>> Hardware used for that target and
>> initiators was the same. I can't see on this load why the initiators would
>> need to do something more than the target. Well, I know we in SCST did an
>> excellent work to maximize performance, but such a difference looks too much
>> ;)
>>
>> Also it looks very suspicious why nobody even tried to match that
>> Microsoft/Intel record, even Intel itself who closely works with Linux
>> community in the storage area and could do it using the same hardware.
>
> The numbers are suspicious for other reasons.  "Random" is often used
> loosely (and the blog referenced earlier doesn't even claim "random").
>  If there is any merging/coalescing going on, then the "IOPS" are
> going to look vastly better.  If I allow coalescing, I can easily get
> 4M 4KB IOPS, but can't honestly call those 4KB IOPS (even if the
> benchmark thinks it's doing 4KB I/O).  They need to show that their
> advertised block size is maintained end-to-end.
>
> Chris
>

Re: Linux I/O subsystem performance (was: linuxcon 2010...)

[Chris Worley]

On Tue, Aug 24, 2010 at 2:31 PM, Chris Worley <worleys@gmail.com> wrote:
> On Tue, Aug 24, 2010 at 11:43 AM, Vladislav Bolkhovitin <vst@vlnb.net> wrote:
>> Pasi Kärkkäinen, on 08/24/2010 11:25 AM wrote:
>>>
>>> On Mon, Aug 23, 2010 at 02:03:26PM -0400, Chetan Loke wrote:
>>>>
>>>> I actually received 3+ off-post emails asking whether I was talking
>>>> about initiator or target in the 100K IOPS case below and what did I
>>>> mean by the ACKs.
>>>> I was referring to the 'Initiator' side.
>>>> ACKs == When scsi-ML down-calls the LLD via the queue-command, process
>>>> the sgl's(if you like) and then trigger the scsi_done up-call path.
>>>>
>>>
>>> Uhm, Intel and Microsoft demonstrated over 1 million IOPS
>>> using software iSCSI and a single 10 Gbit Ethernet NIC (Intel 82599).
>>>
>>> How come there is such a huge difference? What are we lacking in Linux?
>>
>> I also have an impression that Linux I/O subsystem has some performance
>> problems. For instance, in one recent SCST performance test only 8 Linux
>> initiators with fio as a load generator were able to saturate a single SCST
>> target with dual IB cards (SRP) on 4K AIO direct accesses over an SSD
>> backend. This rawly means that any initiator took several times (8?) more
>> processing time than the target.
>
> While I can't tell you where the bottlenecks are, I can share some
> performance numbers...

I've been asked to share more details of the single SRP initiator
case, comparing Windows to Linux...

The configurations tested are represented by four digits separated by dashes:

- The number of initiators used in the test (always one in this case).
- The number of target ports used.
- The number of initiator ports used.
- the number of drives used.

SRP Upstream Initiator

                            1-1-1-1 1-1-1-2  1-2-2-2   1-1-1-4
1-2-2-4   1-1-1-8  1-2-2-8
Random Write    122880  141568  206592  144384  163840  141824  165376
30/70 R/W mix     72113   123136 144640  143616  163072  145920  163584
70/30 R/W mix     55938     91392 114176  135680  156160  145920  162304
Random Read     50688     78336 107008  121600  149760  143872  161536

SRP Windows Initiator

                           1-?-1-1 1-?-1-2   1-?-2-2  1-?-1-4  1-?-2-4
 1-?-1-8    1-?-2-8
Random Write     57774  116738  114464  146972  202891                  221819
30/70 R/W mix    49719    95697    97831  154328  181221                  227786
70/30 R/W mix    45242    90694    89559  167341  176178                  244661
Random Read     48016    94867   92984  178227  183631                  257449

Note that the question marks are where I'm not sure how Windows is
using the second target port... in Linux, you select the target port
from the initiator, but there's no such option in Windows, so the
target port could be used in those cases.  The 1-1-1-8 case is where I
tried to force it to use just one target port (by disabling the target
port), and Windows wouldn't do any I/O at all.

Chris
>
> 4 initiators can get >600K random 4KB IOPS off a single target...
> which is ~150% of what the Emulex/Intel/Microsoft results show using 8
> targets at 4KB (their 1M IOPS was at 512 byte blocks, which is not a
> realistic test point) here:
>
> http://itbrandpulse.com/Documents/Test2010001%20-%20The%20Sun%20Rises%20on%20CNAs%20Test%20Report.pdf
>
> The blog referenced earlier used 10 targets... and I'm not sure how
> many 10G ports per target.
>
> In general, my target seems capable of 65% the local small-block
> random write performance over IB,  and 85% the local small-block
> random read performance.  For large block performance, ~95% efficiency
> is easily achievable, read or write (i.e. 5.6GB/s over fabric, where
> 6GB/s is achievable on the drives locally at 1MB random blocks).
> These small-block efficiencies are achievable only when tested with
> multiple initiators.
>
> The single initiator is only capable of <150K 4KB IOPS... but gets
> full bandwidth w/ larger blocks.
>
> If I were to chose my problem, target or initiator bottleneck, I'd
> certainly rather have an initiator bottleneck rather than Microsoft's
> target bottleneck.
>
>> Hardware used for that target and
>> initiators was the same. I can't see on this load why the initiators would
>> need to do something more than the target. Well, I know we in SCST did an
>> excellent work to maximize performance, but such a difference looks too much
>> ;)
>>
>> Also it looks very suspicious why nobody even tried to match that
>> Microsoft/Intel record, even Intel itself who closely works with Linux
>> community in the storage area and could do it using the same hardware.
>
> The numbers are suspicious for other reasons.  "Random" is often used
> loosely (and the blog referenced earlier doesn't even claim "random").
>  If there is any merging/coalescing going on, then the "IOPS" are
> going to look vastly better.  If I allow coalescing, I can easily get
> 4M 4KB IOPS, but can't honestly call those 4KB IOPS (even if the
> benchmark thinks it's doing 4KB I/O).  They need to show that their
> advertised block size is maintained end-to-end.
>
> Chris
>
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html