SCSI Performance regression [was Re: [PATCH 0/6] tcm_vhost/virtio-scsi WIP code for-3.6]

[James Bottomley <James.Bottomley@HansenPartnership.com>]

On Thu, 2012-07-05 at 20:01 -0700, Nicholas A. Bellinger wrote:

> So I'm pretty sure this discrepancy is attributed to the small block
> random I/O bottleneck currently present for all Linux/SCSI core LLDs
> regardless of physical or virtual storage fabric.
> 
> The SCSI wide host-lock less conversion that happened in .38 code back
> in 2010, and subsequently having LLDs like virtio-scsi convert to run in
> host-lock-less mode have helped to some extent..  But it's still not
> enough..
> 
> Another example where we've been able to prove this bottleneck recently
> is with the following target setup:
> 
> *) Intel Romley production machines with 128 GB of DDR-3 memory
> *) 4x FusionIO ioDrive 2 (1.5 TB @ PCI-e Gen2 x2)
> *) Mellanox PCI-exress Gen3 HCA running at 56 gb/sec 
> *) Infiniband SRP Target backported to RHEL 6.2 + latest OFED
> 
> In this setup using ib_srpt + IBLOCK w/ emulate_write_cache=1 +
> iomemory_vsl export we end up avoiding SCSI core bottleneck on the
> target machine, just as with the tcm_vhost example here for host kernel
> side processing with vhost.
> 
> Using Linux IB SRP initiator + Windows Server 2008 R2 SCSI-miniport SRP
> (OFED) Initiator connected to four ib_srpt LUNs, we've observed that
> MSFT SCSI is currently outperforming RHEL 6.2 on the order of ~285K vs.
> ~215K with heavy random 4k WRITE iometer / fio tests.  Note this with an
> optimized queue_depth ib_srp client w/ noop I/O schedulering, but is
> still lacking the host_lock-less patches on RHEL 6.2 OFED..
> 
> This bottleneck has been mentioned by various people (including myself)
> on linux-scsi the last 18 months, and I've proposed that that it be
> discussed at KS-2012 so we can start making some forward progress:

Well, no, it hasn't.  You randomly drop things like this into unrelated
email (I suppose that is a mention in strict English construction) but
it's not really enough to get anyone to pay attention since they mostly
stopped reading at the top, if they got that far: most people just go by
subject when wading through threads initially.

But even if anyone noticed, a statement that RHEL6.2 (on a 2.6.32
kernel, which is now nearly three years old) is 25% slower than W2k8R2
on infiniband isn't really going to get anyone excited either
(particularly when you mention OFED, which usually means a stack
replacement on Linux anyway).

What people might pay attention to is evidence that there's a problem in
3.5-rc6 (without any OFED crap).  If you're not going to bother
investigating, it has to be in an environment they can reproduce (so
ordinary hardware, not infiniband) otherwise it gets ignored as an
esoteric hardware issue.

James