MSI interrupt support with vioscsi.c miniport driver

MSI interrupt support with vioscsi.c miniport driver

[Nicholas A. Bellinger]

Hi Yan,

So recently I've been doing some KVM guest performance comparisons
between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
vhost-scsi using PCIe flash backend devices.

I've noticed that small block random performance for the MSFT guest is
at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
being well below what the Linux guest with scsi-mq + virtio-scsi is
capable of (~500K).

After searching through the various vioscsi registry settings, it
appears that MSIEnabled is being explicitly disabled (0x00000000), that
is different from what vioscsi.inx is currently defining:

[pnpsafe_pci_addreg_msix]
HKR, "Interrupt Management",, 0x00000010
HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4

Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
here, regardless of MSI_SUPPORTED:

  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340

Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
appears to be the default setting for the driver included in the offical
virtio-win iso builds, right..?

Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
build of my own, but before going down the WDK development rabbit whole,
I'd like to better understand why you've explicitly disabled this logic
within vioscsi.c code to start..?

Is there anything that needs to be addressed / carried over from
virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
miniport code..?

TIA!

--nab

[Qemu-devel] MSI interrupt support with vioscsi.c miniport driver

[Nicholas A. Bellinger]

Hi Yan,

So recently I've been doing some KVM guest performance comparisons
between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
vhost-scsi using PCIe flash backend devices.

I've noticed that small block random performance for the MSFT guest is
at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
being well below what the Linux guest with scsi-mq + virtio-scsi is
capable of (~500K).

After searching through the various vioscsi registry settings, it
appears that MSIEnabled is being explicitly disabled (0x00000000), that
is different from what vioscsi.inx is currently defining:

[pnpsafe_pci_addreg_msix]
HKR, "Interrupt Management",, 0x00000010
HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4

Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
here, regardless of MSI_SUPPORTED:

  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340

Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
appears to be the default setting for the driver included in the offical
virtio-win iso builds, right..?

Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
build of my own, but before going down the WDK development rabbit whole,
I'd like to better understand why you've explicitly disabled this logic
within vioscsi.c code to start..?

Is there anything that needs to be addressed / carried over from
virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
miniport code..?

TIA!

--nab

Re: MSI interrupt support with vioscsi.c miniport driver

[Yan Vugenfirer]

Hi Nicholas,

Adding Vadim Rozenfeld who wrote the virtio-scsi driver.

Best regards,
Yan.

On Feb 7, 2014, at 10:14 PM, Nicholas A. Bellinger <nab@linux-iscsi.org> wrote:

> Hi Yan,
> 
> So recently I've been doing some KVM guest performance comparisons
> between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> vhost-scsi using PCIe flash backend devices.
> 
> I've noticed that small block random performance for the MSFT guest is
> at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> being well below what the Linux guest with scsi-mq + virtio-scsi is
> capable of (~500K).
> 
> After searching through the various vioscsi registry settings, it
> appears that MSIEnabled is being explicitly disabled (0x00000000), that
> is different from what vioscsi.inx is currently defining:
> 
> [pnpsafe_pci_addreg_msix]
> HKR, "Interrupt Management",, 0x00000010
> HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> 
> Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> here, regardless of MSI_SUPPORTED:
> 
>  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> 
> Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> appears to be the default setting for the driver included in the offical
> virtio-win iso builds, right..?
> 
> Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> build of my own, but before going down the WDK development rabbit whole,
> I'd like to better understand why you've explicitly disabled this logic
> within vioscsi.c code to start..?
> 
> Is there anything that needs to be addressed / carried over from
> virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> miniport code..?
> 
> TIA!
> 
> --nab
> 
> 

Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver

[Yan Vugenfirer]

Hi Nicholas,

Adding Vadim Rozenfeld who wrote the virtio-scsi driver.

Best regards,
Yan.

On Feb 7, 2014, at 10:14 PM, Nicholas A. Bellinger <nab@linux-iscsi.org> wrote:

> Hi Yan,
> 
> So recently I've been doing some KVM guest performance comparisons
> between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> vhost-scsi using PCIe flash backend devices.
> 
> I've noticed that small block random performance for the MSFT guest is
> at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> being well below what the Linux guest with scsi-mq + virtio-scsi is
> capable of (~500K).
> 
> After searching through the various vioscsi registry settings, it
> appears that MSIEnabled is being explicitly disabled (0x00000000), that
> is different from what vioscsi.inx is currently defining:
> 
> [pnpsafe_pci_addreg_msix]
> HKR, "Interrupt Management",, 0x00000010
> HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> 
> Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> here, regardless of MSI_SUPPORTED:
> 
>  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> 
> Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> appears to be the default setting for the driver included in the offical
> virtio-win iso builds, right..?
> 
> Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> build of my own, but before going down the WDK development rabbit whole,
> I'd like to better understand why you've explicitly disabled this logic
> within vioscsi.c code to start..?
> 
> Is there anything that needs to be addressed / carried over from
> virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> miniport code..?
> 
> TIA!
> 
> --nab
> 
> 

Re: MSI interrupt support with vioscsi.c miniport driver

[Vadim Rozenfeld]

On Sun, 2014-02-09 at 11:24 +0200, Yan Vugenfirer wrote:
> Hi Nicholas,
> 
> Adding Vadim Rozenfeld who wrote the virtio-scsi driver.
> 
> Best regards,
> Yan.
> 
> On Feb 7, 2014, at 10:14 PM, Nicholas A. Bellinger <nab@linux-iscsi.org> wrote:
> 
> > Hi Yan,
> > 
> > So recently I've been doing some KVM guest performance comparisons
> > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > vhost-scsi using PCIe flash backend devices.
> > 
> > I've noticed that small block random performance for the MSFT guest is
> > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > capable of (~500K).
> > 
> > After searching through the various vioscsi registry settings, it
> > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > is different from what vioscsi.inx is currently defining:
> > 
> > [pnpsafe_pci_addreg_msix]
> > HKR, "Interrupt Management",, 0x00000010
> > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > 
> > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > here, regardless of MSI_SUPPORTED:
> > 
> >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > 
> > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > appears to be the default setting for the driver included in the offical
> > virtio-win iso builds, right..?
> > 
> > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > build of my own, but before going down the WDK development rabbit whole,
> > I'd like to better understand why you've explicitly disabled this logic
> > within vioscsi.c code to start..?
> > 
> > Is there anything that needs to be addressed / carried over from
> > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > miniport code..?

Hi Nicholas,

I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
reasons decided to keep it disabled until adding mq support.


You definitely should be able to turn on MSI_SUPPORTED, rebuild the
driver, and switch MSISupported to 1 to make vioscsi driver working in
MSI mode.
   
Cheers,
Vadim.

> > 
> > TIA!
> > 
> > --nab
> > 
> > 
> 

Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver

[Vadim Rozenfeld]

On Sun, 2014-02-09 at 11:24 +0200, Yan Vugenfirer wrote:
> Hi Nicholas,
> 
> Adding Vadim Rozenfeld who wrote the virtio-scsi driver.
> 
> Best regards,
> Yan.
> 
> On Feb 7, 2014, at 10:14 PM, Nicholas A. Bellinger <nab@linux-iscsi.org> wrote:
> 
> > Hi Yan,
> > 
> > So recently I've been doing some KVM guest performance comparisons
> > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > vhost-scsi using PCIe flash backend devices.
> > 
> > I've noticed that small block random performance for the MSFT guest is
> > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > capable of (~500K).
> > 
> > After searching through the various vioscsi registry settings, it
> > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > is different from what vioscsi.inx is currently defining:
> > 
> > [pnpsafe_pci_addreg_msix]
> > HKR, "Interrupt Management",, 0x00000010
> > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > 
> > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > here, regardless of MSI_SUPPORTED:
> > 
> >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > 
> > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > appears to be the default setting for the driver included in the offical
> > virtio-win iso builds, right..?
> > 
> > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > build of my own, but before going down the WDK development rabbit whole,
> > I'd like to better understand why you've explicitly disabled this logic
> > within vioscsi.c code to start..?
> > 
> > Is there anything that needs to be addressed / carried over from
> > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > miniport code..?

Hi Nicholas,

I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
reasons decided to keep it disabled until adding mq support.


You definitely should be able to turn on MSI_SUPPORTED, rebuild the
driver, and switch MSISupported to 1 to make vioscsi driver working in
MSI mode.
   
Cheers,
Vadim.

> > 
> > TIA!
> > 
> > --nab
> > 
> > 
> 

Re: MSI interrupt support with vioscsi.c miniport driver

[Nicholas A. Bellinger]

Hi Vadim & Yan,

On Sun, 2014-02-09 at 22:35 +1100, Vadim Rozenfeld wrote:
> On Sun, 2014-02-09 at 11:24 +0200, Yan Vugenfirer wrote:
> > Hi Nicholas,
> > 
> > Adding Vadim Rozenfeld who wrote the virtio-scsi driver.
> > 
> > Best regards,
> > Yan.
> > 
> > On Feb 7, 2014, at 10:14 PM, Nicholas A. Bellinger <nab@linux-iscsi.org> wrote:
> > 
> > > Hi Yan,
> > > 
> > > So recently I've been doing some KVM guest performance comparisons
> > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > > vhost-scsi using PCIe flash backend devices.
> > > 
> > > I've noticed that small block random performance for the MSFT guest is
> > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > > capable of (~500K).
> > > 
> > > After searching through the various vioscsi registry settings, it
> > > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > > is different from what vioscsi.inx is currently defining:
> > > 
> > > [pnpsafe_pci_addreg_msix]
> > > HKR, "Interrupt Management",, 0x00000010
> > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > > 
> > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > > here, regardless of MSI_SUPPORTED:
> > > 
> > >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > > 
> > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > > appears to be the default setting for the driver included in the offical
> > > virtio-win iso builds, right..?
> > > 
> > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > > build of my own, but before going down the WDK development rabbit whole,
> > > I'd like to better understand why you've explicitly disabled this logic
> > > within vioscsi.c code to start..?
> > > 
> > > Is there anything that needs to be addressed / carried over from
> > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > > miniport code..?
> 
> Hi Nicholas,
> 
> I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
> reasons decided to keep it disabled until adding mq support.
> 
> 
> You definitely should be able to turn on MSI_SUPPORTED, rebuild the
> driver, and switch MSISupported to 1 to make vioscsi driver working in
> MSI mode.
>    

Thanks for the quick response.  We'll give MSI_SUPPORTED=1 a shot over
the next days with a test build on Server 2012 / Server 2008 R2 and see
how things go..

Thanks again!

--nab

Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver

[Nicholas A. Bellinger]

Hi Vadim & Yan,

On Sun, 2014-02-09 at 22:35 +1100, Vadim Rozenfeld wrote:
> On Sun, 2014-02-09 at 11:24 +0200, Yan Vugenfirer wrote:
> > Hi Nicholas,
> > 
> > Adding Vadim Rozenfeld who wrote the virtio-scsi driver.
> > 
> > Best regards,
> > Yan.
> > 
> > On Feb 7, 2014, at 10:14 PM, Nicholas A. Bellinger <nab@linux-iscsi.org> wrote:
> > 
> > > Hi Yan,
> > > 
> > > So recently I've been doing some KVM guest performance comparisons
> > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > > vhost-scsi using PCIe flash backend devices.
> > > 
> > > I've noticed that small block random performance for the MSFT guest is
> > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > > capable of (~500K).
> > > 
> > > After searching through the various vioscsi registry settings, it
> > > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > > is different from what vioscsi.inx is currently defining:
> > > 
> > > [pnpsafe_pci_addreg_msix]
> > > HKR, "Interrupt Management",, 0x00000010
> > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > > 
> > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > > here, regardless of MSI_SUPPORTED:
> > > 
> > >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > > 
> > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > > appears to be the default setting for the driver included in the offical
> > > virtio-win iso builds, right..?
> > > 
> > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > > build of my own, but before going down the WDK development rabbit whole,
> > > I'd like to better understand why you've explicitly disabled this logic
> > > within vioscsi.c code to start..?
> > > 
> > > Is there anything that needs to be addressed / carried over from
> > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > > miniport code..?
> 
> Hi Nicholas,
> 
> I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
> reasons decided to keep it disabled until adding mq support.
> 
> 
> You definitely should be able to turn on MSI_SUPPORTED, rebuild the
> driver, and switch MSISupported to 1 to make vioscsi driver working in
> MSI mode.
>    

Thanks for the quick response.  We'll give MSI_SUPPORTED=1 a shot over
the next days with a test build on Server 2012 / Server 2008 R2 and see
how things go..

Thanks again!

--nab

Re: MSI interrupt support with vioscsi.c miniport driver

[Nicholas A. Bellinger]

On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote:

<SNIP>

> > > > Hi Yan,
> > > > 
> > > > So recently I've been doing some KVM guest performance comparisons
> > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > > > vhost-scsi using PCIe flash backend devices.
> > > > 
> > > > I've noticed that small block random performance for the MSFT guest is
> > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > > > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > > > capable of (~500K).
> > > > 
> > > > After searching through the various vioscsi registry settings, it
> > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > > > is different from what vioscsi.inx is currently defining:
> > > > 
> > > > [pnpsafe_pci_addreg_msix]
> > > > HKR, "Interrupt Management",, 0x00000010
> > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > > > 
> > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > > > here, regardless of MSI_SUPPORTED:
> > > > 
> > > >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > > > 
> > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > > > appears to be the default setting for the driver included in the offical
> > > > virtio-win iso builds, right..?
> > > > 
> > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > > > build of my own, but before going down the WDK development rabbit whole,
> > > > I'd like to better understand why you've explicitly disabled this logic
> > > > within vioscsi.c code to start..?
> > > > 
> > > > Is there anything that needs to be addressed / carried over from
> > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > > > miniport code..?
> > 
> > Hi Nicholas,
> > 
> > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
> > reasons decided to keep it disabled until adding mq support.
> > 
> > 
> > You definitely should be able to turn on MSI_SUPPORTED, rebuild the
> > driver, and switch MSISupported to 1 to make vioscsi driver working in
> > MSI mode.
> >    
> 
> Thanks for the quick response.  We'll give MSI_SUPPORTED=1 a shot over
> the next days with a test build on Server 2012 / Server 2008 R2 and see
> how things go..
> 

Just a quick update on progress.

I've been able to successfully build + load a unsigned vioscsi.sys
driver on Server 2012 with WDK 8.0.

Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant
performance and efficiency gain, on the order of 100K to 225K IOPs for
4K block random I/O workload, depending on read/write mix.

Below is a simple patch to enable MSI operation by default.  Any chance
to apply this separate from future mq efforts..?

Thanks,

--nab

>From 89adb6d5800386d44b36737d1587e0ffc09c4902 Mon Sep 17 00:00:00 2001
From: Nicholas Bellinger <nab@linux-iscsi.org>
Date: Fri, 14 Feb 2014 10:26:04 -0800
Subject: [PATCH] vioscsi: Set MSI_SUPPORTED=1 by default

Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
---
 vioscsi/SOURCES         | 2 +-
 vioscsi/vioscsi.c       | 2 --
 vioscsi/vioscsi.inx     | 2 +-
 vioscsi/vioscsi.vcxproj | 6 +++---
 4 files changed, 5 insertions(+), 7 deletions(-)

diff --git a/vioscsi/SOURCES b/vioscsi/SOURCES
index f2083de..f631bd2 100644
--- a/vioscsi/SOURCES
+++ b/vioscsi/SOURCES
@@ -6,7 +6,7 @@ C_DEFINES = -D_MINORVERSION_=$(_BUILD_MINOR_VERSION_) $(C_DEFINES)
 C_DEFINES = -D_NT_TARGET_MAJ=$(_NT_TARGET_MAJ) $(C_DEFINES)
 C_DEFINES = -D_NT_TARGET_MIN=$(_RHEL_RELEASE_VERSION_) $(C_DEFINES)
 
-C_DEFINES = -DMSI_SUPPORTED=0 $(C_DEFINES)
+C_DEFINES = -DMSI_SUPPORTED=1 $(C_DEFINES)
 C_DEFINES = -DINDIRECT_SUPPORTED=1 $(C_DEFINES)
 TARGETLIBS=$(SDK_LIB_PATH)\storport.lib ..\VirtIO\$(O)\virtiolib.lib
 
diff --git a/vioscsi/vioscsi.c b/vioscsi/vioscsi.c
index 77c0e46..70b9bb4 100644
--- a/vioscsi/vioscsi.c
+++ b/vioscsi/vioscsi.c
@@ -337,8 +337,6 @@ ENTER_FN();
         adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1;
     }
 
-    adaptExt->msix_enabled = FALSE;
-
     RhelDbgPrint(TRACE_LEVEL_ERROR, ("breaks_number = %x  queue_depth = %x\n",
                 ConfigInfo->NumberOfPhysicalBreaks,
                 adaptExt->queue_depth));
diff --git a/vioscsi/vioscsi.inx b/vioscsi/vioscsi.inx
index cc94b7c..ec717c6 100644
--- a/vioscsi/vioscsi.inx
+++ b/vioscsi/vioscsi.inx
@@ -79,7 +79,7 @@ HKR, "Parameters", "BusType", %REG_DWORD%, 0x00000001
 [pnpsafe_pci_addreg_msix]
 HKR, "Interrupt Management",, 0x00000010
 HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
-HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
+HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 1
 HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
 HKR, "Interrupt Management\Affinity Policy",, 0x00000010
 HKR, "Interrupt Management\Affinity Policy", DevicePolicy, 0x00010001, 5
diff --git a/vioscsi/vioscsi.vcxproj b/vioscsi/vioscsi.vcxproj
index 889e513..68d4e85 100644
--- a/vioscsi/vioscsi.vcxproj
+++ b/vioscsi/vioscsi.vcxproj
@@ -66,7 +66,7 @@
   <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|Win32'">
     <ClCompile>
       <WarningLevel>Level3</WarningLevel>
-      <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions>
     </ClCompile>
     <Link>
       <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies>
@@ -95,7 +95,7 @@
   <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|x64'">
     <ClCompile>
       <WarningLevel>Level3</WarningLevel>
-      <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions>
     </ClCompile>
     <Link>
       <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies>
@@ -140,4 +140,4 @@
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
   <ImportGroup Label="ExtensionTargets">
   </ImportGroup>
-</Project>
\ No newline at end of file
+</Project>
-- 
1.8.5.2.msysgit.0

Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver

[Nicholas A. Bellinger]

On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote:

<SNIP>

> > > > Hi Yan,
> > > > 
> > > > So recently I've been doing some KVM guest performance comparisons
> > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > > > vhost-scsi using PCIe flash backend devices.
> > > > 
> > > > I've noticed that small block random performance for the MSFT guest is
> > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > > > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > > > capable of (~500K).
> > > > 
> > > > After searching through the various vioscsi registry settings, it
> > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > > > is different from what vioscsi.inx is currently defining:
> > > > 
> > > > [pnpsafe_pci_addreg_msix]
> > > > HKR, "Interrupt Management",, 0x00000010
> > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > > > 
> > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > > > here, regardless of MSI_SUPPORTED:
> > > > 
> > > >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > > > 
> > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > > > appears to be the default setting for the driver included in the offical
> > > > virtio-win iso builds, right..?
> > > > 
> > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > > > build of my own, but before going down the WDK development rabbit whole,
> > > > I'd like to better understand why you've explicitly disabled this logic
> > > > within vioscsi.c code to start..?
> > > > 
> > > > Is there anything that needs to be addressed / carried over from
> > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > > > miniport code..?
> > 
> > Hi Nicholas,
> > 
> > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
> > reasons decided to keep it disabled until adding mq support.
> > 
> > 
> > You definitely should be able to turn on MSI_SUPPORTED, rebuild the
> > driver, and switch MSISupported to 1 to make vioscsi driver working in
> > MSI mode.
> >    
> 
> Thanks for the quick response.  We'll give MSI_SUPPORTED=1 a shot over
> the next days with a test build on Server 2012 / Server 2008 R2 and see
> how things go..
> 

Just a quick update on progress.

I've been able to successfully build + load a unsigned vioscsi.sys
driver on Server 2012 with WDK 8.0.

Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant
performance and efficiency gain, on the order of 100K to 225K IOPs for
4K block random I/O workload, depending on read/write mix.

Below is a simple patch to enable MSI operation by default.  Any chance
to apply this separate from future mq efforts..?

Thanks,

--nab

>From 89adb6d5800386d44b36737d1587e0ffc09c4902 Mon Sep 17 00:00:00 2001
From: Nicholas Bellinger <nab@linux-iscsi.org>
Date: Fri, 14 Feb 2014 10:26:04 -0800
Subject: [PATCH] vioscsi: Set MSI_SUPPORTED=1 by default

Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
---
 vioscsi/SOURCES         | 2 +-
 vioscsi/vioscsi.c       | 2 --
 vioscsi/vioscsi.inx     | 2 +-
 vioscsi/vioscsi.vcxproj | 6 +++---
 4 files changed, 5 insertions(+), 7 deletions(-)

diff --git a/vioscsi/SOURCES b/vioscsi/SOURCES
index f2083de..f631bd2 100644
--- a/vioscsi/SOURCES
+++ b/vioscsi/SOURCES
@@ -6,7 +6,7 @@ C_DEFINES = -D_MINORVERSION_=$(_BUILD_MINOR_VERSION_) $(C_DEFINES)
 C_DEFINES = -D_NT_TARGET_MAJ=$(_NT_TARGET_MAJ) $(C_DEFINES)
 C_DEFINES = -D_NT_TARGET_MIN=$(_RHEL_RELEASE_VERSION_) $(C_DEFINES)
 
-C_DEFINES = -DMSI_SUPPORTED=0 $(C_DEFINES)
+C_DEFINES = -DMSI_SUPPORTED=1 $(C_DEFINES)
 C_DEFINES = -DINDIRECT_SUPPORTED=1 $(C_DEFINES)
 TARGETLIBS=$(SDK_LIB_PATH)\storport.lib ..\VirtIO\$(O)\virtiolib.lib
 
diff --git a/vioscsi/vioscsi.c b/vioscsi/vioscsi.c
index 77c0e46..70b9bb4 100644
--- a/vioscsi/vioscsi.c
+++ b/vioscsi/vioscsi.c
@@ -337,8 +337,6 @@ ENTER_FN();
         adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1;
     }
 
-    adaptExt->msix_enabled = FALSE;
-
     RhelDbgPrint(TRACE_LEVEL_ERROR, ("breaks_number = %x  queue_depth = %x\n",
                 ConfigInfo->NumberOfPhysicalBreaks,
                 adaptExt->queue_depth));
diff --git a/vioscsi/vioscsi.inx b/vioscsi/vioscsi.inx
index cc94b7c..ec717c6 100644
--- a/vioscsi/vioscsi.inx
+++ b/vioscsi/vioscsi.inx
@@ -79,7 +79,7 @@ HKR, "Parameters", "BusType", %REG_DWORD%, 0x00000001
 [pnpsafe_pci_addreg_msix]
 HKR, "Interrupt Management",, 0x00000010
 HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
-HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
+HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 1
 HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
 HKR, "Interrupt Management\Affinity Policy",, 0x00000010
 HKR, "Interrupt Management\Affinity Policy", DevicePolicy, 0x00010001, 5
diff --git a/vioscsi/vioscsi.vcxproj b/vioscsi/vioscsi.vcxproj
index 889e513..68d4e85 100644
--- a/vioscsi/vioscsi.vcxproj
+++ b/vioscsi/vioscsi.vcxproj
@@ -66,7 +66,7 @@
   <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|Win32'">
     <ClCompile>
       <WarningLevel>Level3</WarningLevel>
-      <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions>
     </ClCompile>
     <Link>
       <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies>
@@ -95,7 +95,7 @@
   <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|x64'">
     <ClCompile>
       <WarningLevel>Level3</WarningLevel>
-      <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions>
     </ClCompile>
     <Link>
       <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies>
@@ -140,4 +140,4 @@
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
   <ImportGroup Label="ExtensionTargets">
   </ImportGroup>
-</Project>
\ No newline at end of file
+</Project>
-- 
1.8.5.2.msysgit.0

Re: MSI interrupt support with vioscsi.c miniport driver

[Nicholas A. Bellinger]

On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote:
> On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote:
> 
> <SNIP>
> 
> > > > > Hi Yan,
> > > > > 
> > > > > So recently I've been doing some KVM guest performance comparisons
> > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > > > > vhost-scsi using PCIe flash backend devices.
> > > > > 
> > > > > I've noticed that small block random performance for the MSFT guest is
> > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > > > > capable of (~500K).
> > > > > 
> > > > > After searching through the various vioscsi registry settings, it
> > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > > > > is different from what vioscsi.inx is currently defining:
> > > > > 
> > > > > [pnpsafe_pci_addreg_msix]
> > > > > HKR, "Interrupt Management",, 0x00000010
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > > > > 
> > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > > > > here, regardless of MSI_SUPPORTED:
> > > > > 
> > > > >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > > > > 
> > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > > > > appears to be the default setting for the driver included in the offical
> > > > > virtio-win iso builds, right..?
> > > > > 
> > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > > > > build of my own, but before going down the WDK development rabbit whole,
> > > > > I'd like to better understand why you've explicitly disabled this logic
> > > > > within vioscsi.c code to start..?
> > > > > 
> > > > > Is there anything that needs to be addressed / carried over from
> > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > > > > miniport code..?
> > > 
> > > Hi Nicholas,
> > > 
> > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
> > > reasons decided to keep it disabled until adding mq support.
> > > 
> > > 
> > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the
> > > driver, and switch MSISupported to 1 to make vioscsi driver working in
> > > MSI mode.
> > >    
> > 
> > Thanks for the quick response.  We'll give MSI_SUPPORTED=1 a shot over
> > the next days with a test build on Server 2012 / Server 2008 R2 and see
> > how things go..
> > 
> 
> Just a quick update on progress.
> 
> I've been able to successfully build + load a unsigned vioscsi.sys
> driver on Server 2012 with WDK 8.0.
> 
> Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant
> performance and efficiency gain, on the order of 100K to 225K IOPs for
> 4K block random I/O workload, depending on read/write mix.
> 

One other performance related question..

In vioscsi.c:VioScsiFindAdapter() code, the default setting for
adaptExt->queue_depth ends up getting set to 32 (pageNum / 4) when
indirect mode is enabled in the following bits:

    if(adaptExt->indirect) {
        adaptExt->queue_depth = max(2, (pageNum / 4));
    } else {
        adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1;
    }

Looking at viostor/virtio_stor.c:VirtIoFindAdapter() code, the default
setting for ->queue_depth appears to be 128 (pageNum):

#if (INDIRECT_SUPPORTED)
    if(!adaptExt->dump_mode) {
        adaptExt->indirect = CHECKBIT(adaptExt->features, VIRTIO_RING_F_INDIRECT_DESC);
    }
    if(adaptExt->indirect) {
        adaptExt->queue_depth = pageNum;
    }
#else
    adaptExt->indirect = 0;
#endif

Is there a reason for the lower queue_depth for vioscsi vs. viostor..?

How about using min(adaptExt->scsi_config.cmd_per_lun, pageNum) instead..?

Thanks!

-nab

Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver

[Nicholas A. Bellinger]

On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote:
> On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote:
> 
> <SNIP>
> 
> > > > > Hi Yan,
> > > > > 
> > > > > So recently I've been doing some KVM guest performance comparisons
> > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > > > > vhost-scsi using PCIe flash backend devices.
> > > > > 
> > > > > I've noticed that small block random performance for the MSFT guest is
> > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > > > > capable of (~500K).
> > > > > 
> > > > > After searching through the various vioscsi registry settings, it
> > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > > > > is different from what vioscsi.inx is currently defining:
> > > > > 
> > > > > [pnpsafe_pci_addreg_msix]
> > > > > HKR, "Interrupt Management",, 0x00000010
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > > > > 
> > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > > > > here, regardless of MSI_SUPPORTED:
> > > > > 
> > > > >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > > > > 
> > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > > > > appears to be the default setting for the driver included in the offical
> > > > > virtio-win iso builds, right..?
> > > > > 
> > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > > > > build of my own, but before going down the WDK development rabbit whole,
> > > > > I'd like to better understand why you've explicitly disabled this logic
> > > > > within vioscsi.c code to start..?
> > > > > 
> > > > > Is there anything that needs to be addressed / carried over from
> > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > > > > miniport code..?
> > > 
> > > Hi Nicholas,
> > > 
> > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
> > > reasons decided to keep it disabled until adding mq support.
> > > 
> > > 
> > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the
> > > driver, and switch MSISupported to 1 to make vioscsi driver working in
> > > MSI mode.
> > >    
> > 
> > Thanks for the quick response.  We'll give MSI_SUPPORTED=1 a shot over
> > the next days with a test build on Server 2012 / Server 2008 R2 and see
> > how things go..
> > 
> 
> Just a quick update on progress.
> 
> I've been able to successfully build + load a unsigned vioscsi.sys
> driver on Server 2012 with WDK 8.0.
> 
> Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant
> performance and efficiency gain, on the order of 100K to 225K IOPs for
> 4K block random I/O workload, depending on read/write mix.
> 

One other performance related question..

In vioscsi.c:VioScsiFindAdapter() code, the default setting for
adaptExt->queue_depth ends up getting set to 32 (pageNum / 4) when
indirect mode is enabled in the following bits:

    if(adaptExt->indirect) {
        adaptExt->queue_depth = max(2, (pageNum / 4));
    } else {
        adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1;
    }

Looking at viostor/virtio_stor.c:VirtIoFindAdapter() code, the default
setting for ->queue_depth appears to be 128 (pageNum):

#if (INDIRECT_SUPPORTED)
    if(!adaptExt->dump_mode) {
        adaptExt->indirect = CHECKBIT(adaptExt->features, VIRTIO_RING_F_INDIRECT_DESC);
    }
    if(adaptExt->indirect) {
        adaptExt->queue_depth = pageNum;
    }
#else
    adaptExt->indirect = 0;
#endif

Is there a reason for the lower queue_depth for vioscsi vs. viostor..?

How about using min(adaptExt->scsi_config.cmd_per_lun, pageNum) instead..?

Thanks!

-nab

Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver

[Vadim Rozenfeld]

On Tue, 2014-02-18 at 13:11 -0800, Nicholas A. Bellinger wrote:
> On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote:
> > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote:
> > 
> > <SNIP>
> > 
> > > > > > Hi Yan,
> > > > > > 
> > > > > > So recently I've been doing some KVM guest performance comparisons
> > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > > > > > vhost-scsi using PCIe flash backend devices.
> > > > > > 
> > > > > > I've noticed that small block random performance for the MSFT guest is
> > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > > > > > capable of (~500K).
> > > > > > 
> > > > > > After searching through the various vioscsi registry settings, it
> > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > > > > > is different from what vioscsi.inx is currently defining:
> > > > > > 
> > > > > > [pnpsafe_pci_addreg_msix]
> > > > > > HKR, "Interrupt Management",, 0x00000010
> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > > > > > 
> > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > > > > > here, regardless of MSI_SUPPORTED:
> > > > > > 
> > > > > >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > > > > > 
> > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > > > > > appears to be the default setting for the driver included in the offical
> > > > > > virtio-win iso builds, right..?
> > > > > > 
> > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > > > > > build of my own, but before going down the WDK development rabbit whole,
> > > > > > I'd like to better understand why you've explicitly disabled this logic
> > > > > > within vioscsi.c code to start..?
> > > > > > 
> > > > > > Is there anything that needs to be addressed / carried over from
> > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > > > > > miniport code..?
> > > > 
> > > > Hi Nicholas,
> > > > 
> > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
> > > > reasons decided to keep it disabled until adding mq support.
> > > > 
> > > > 
> > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the
> > > > driver, and switch MSISupported to 1 to make vioscsi driver working in
> > > > MSI mode.
> > > >    
> > > 
> > > Thanks for the quick response.  We'll give MSI_SUPPORTED=1 a shot over
> > > the next days with a test build on Server 2012 / Server 2008 R2 and see
> > > how things go..
> > > 
> > 
> > Just a quick update on progress.
> > 
> > I've been able to successfully build + load a unsigned vioscsi.sys
> > driver on Server 2012 with WDK 8.0.
> > 
> > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant
> > performance and efficiency gain, on the order of 100K to 225K IOPs for
> > 4K block random I/O workload, depending on read/write mix.
> > 
> 
> One other performance related question..
> 
> In vioscsi.c:VioScsiFindAdapter() code, the default setting for
> adaptExt->queue_depth ends up getting set to 32 (pageNum / 4) when
> indirect mode is enabled in the following bits:
> 
>     if(adaptExt->indirect) {
>         adaptExt->queue_depth = max(2, (pageNum / 4));
>     } else {
>         adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1;
>     }
> 
> Looking at viostor/virtio_stor.c:VirtIoFindAdapter() code, the default
> setting for ->queue_depth appears to be 128 (pageNum):
> 
> #if (INDIRECT_SUPPORTED)
>     if(!adaptExt->dump_mode) {
>         adaptExt->indirect = CHECKBIT(adaptExt->features, VIRTIO_RING_F_INDIRECT_DESC);
>     }
>     if(adaptExt->indirect) {
>         adaptExt->queue_depth = pageNum;
>     }
> #else
>     adaptExt->indirect = 0;
> #endif
> 
> Is there a reason for the lower queue_depth for vioscsi vs. viostor..?

It's a horrible work around for the following bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1013443

I'm going to remove it as soon as found better solution for it.

Best regards,
Vadim.


> 
> How about using min(adaptExt->scsi_config.cmd_per_lun, pageNum) instead..?
> 
> Thanks!
> 
> -nab
> 
> 

Re: MSI interrupt support with vioscsi.c miniport driver

[Vadim Rozenfeld]

On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote:
> On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote:
> 
> <SNIP>
> 
> > > > > Hi Yan,
> > > > > 
> > > > > So recently I've been doing some KVM guest performance comparisons
> > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > > > > vhost-scsi using PCIe flash backend devices.
> > > > > 
> > > > > I've noticed that small block random performance for the MSFT guest is
> > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > > > > capable of (~500K).
> > > > > 
> > > > > After searching through the various vioscsi registry settings, it
> > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > > > > is different from what vioscsi.inx is currently defining:
> > > > > 
> > > > > [pnpsafe_pci_addreg_msix]
> > > > > HKR, "Interrupt Management",, 0x00000010
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > > > > 
> > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > > > > here, regardless of MSI_SUPPORTED:
> > > > > 
> > > > >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > > > > 
> > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > > > > appears to be the default setting for the driver included in the offical
> > > > > virtio-win iso builds, right..?
> > > > > 
> > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > > > > build of my own, but before going down the WDK development rabbit whole,
> > > > > I'd like to better understand why you've explicitly disabled this logic
> > > > > within vioscsi.c code to start..?
> > > > > 
> > > > > Is there anything that needs to be addressed / carried over from
> > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > > > > miniport code..?
> > > 
> > > Hi Nicholas,
> > > 
> > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
> > > reasons decided to keep it disabled until adding mq support.
> > > 
> > > 
> > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the
> > > driver, and switch MSISupported to 1 to make vioscsi driver working in
> > > MSI mode.
> > >    
> > 
> > Thanks for the quick response.  We'll give MSI_SUPPORTED=1 a shot over
> > the next days with a test build on Server 2012 / Server 2008 R2 and see
> > how things go..
> > 
> 
> Just a quick update on progress.
> 
> I've been able to successfully build + load a unsigned vioscsi.sys
> driver on Server 2012 with WDK 8.0.
> 
> Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant
> performance and efficiency gain, on the order of 100K to 225K IOPs for
> 4K block random I/O workload, depending on read/write mix.
> 
> Below is a simple patch to enable MSI operation by default.  Any chance
> to apply this separate from future mq efforts..?

Yes, we differently can enable MSI and rebuild vioscsi.
But then we need to re-spin WHQL testing for this particular
driver. This process requires a lot of resources, and I doubt that
it will be initiated soon, unless we have some significant amount of
bug-fixes.

Best regards,
Vadim.     

> 
> Thanks,
> 
> --nab
> 
> From 89adb6d5800386d44b36737d1587e0ffc09c4902 Mon Sep 17 00:00:00 2001
> From: Nicholas Bellinger <nab@linux-iscsi.org>
> Date: Fri, 14 Feb 2014 10:26:04 -0800
> Subject: [PATCH] vioscsi: Set MSI_SUPPORTED=1 by default
> 
> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
> ---
>  vioscsi/SOURCES         | 2 +-
>  vioscsi/vioscsi.c       | 2 --
>  vioscsi/vioscsi.inx     | 2 +-
>  vioscsi/vioscsi.vcxproj | 6 +++---
>  4 files changed, 5 insertions(+), 7 deletions(-)
> 
> diff --git a/vioscsi/SOURCES b/vioscsi/SOURCES
> index f2083de..f631bd2 100644
> --- a/vioscsi/SOURCES
> +++ b/vioscsi/SOURCES
> @@ -6,7 +6,7 @@ C_DEFINES = -D_MINORVERSION_=$(_BUILD_MINOR_VERSION_) $(C_DEFINES)
>  C_DEFINES = -D_NT_TARGET_MAJ=$(_NT_TARGET_MAJ) $(C_DEFINES)
>  C_DEFINES = -D_NT_TARGET_MIN=$(_RHEL_RELEASE_VERSION_) $(C_DEFINES)
>  
> -C_DEFINES = -DMSI_SUPPORTED=0 $(C_DEFINES)
> +C_DEFINES = -DMSI_SUPPORTED=1 $(C_DEFINES)
>  C_DEFINES = -DINDIRECT_SUPPORTED=1 $(C_DEFINES)
>  TARGETLIBS=$(SDK_LIB_PATH)\storport.lib ..\VirtIO\$(O)\virtiolib.lib
>  
> diff --git a/vioscsi/vioscsi.c b/vioscsi/vioscsi.c
> index 77c0e46..70b9bb4 100644
> --- a/vioscsi/vioscsi.c
> +++ b/vioscsi/vioscsi.c
> @@ -337,8 +337,6 @@ ENTER_FN();
>          adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1;
>      }
>  
> -    adaptExt->msix_enabled = FALSE;
> -
>      RhelDbgPrint(TRACE_LEVEL_ERROR, ("breaks_number = %x  queue_depth = %x\n",
>                  ConfigInfo->NumberOfPhysicalBreaks,
>                  adaptExt->queue_depth));
> diff --git a/vioscsi/vioscsi.inx b/vioscsi/vioscsi.inx
> index cc94b7c..ec717c6 100644
> --- a/vioscsi/vioscsi.inx
> +++ b/vioscsi/vioscsi.inx
> @@ -79,7 +79,7 @@ HKR, "Parameters", "BusType", %REG_DWORD%, 0x00000001
>  [pnpsafe_pci_addreg_msix]
>  HKR, "Interrupt Management",, 0x00000010
>  HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> -HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> +HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 1
>  HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
>  HKR, "Interrupt Management\Affinity Policy",, 0x00000010
>  HKR, "Interrupt Management\Affinity Policy", DevicePolicy, 0x00010001, 5
> diff --git a/vioscsi/vioscsi.vcxproj b/vioscsi/vioscsi.vcxproj
> index 889e513..68d4e85 100644
> --- a/vioscsi/vioscsi.vcxproj
> +++ b/vioscsi/vioscsi.vcxproj
> @@ -66,7 +66,7 @@
>    <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|Win32'">
>      <ClCompile>
>        <WarningLevel>Level3</WarningLevel>
> -      <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
> +      <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions>
>      </ClCompile>
>      <Link>
>        <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies>
> @@ -95,7 +95,7 @@
>    <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|x64'">
>      <ClCompile>
>        <WarningLevel>Level3</WarningLevel>
> -      <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
> +      <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions>
>      </ClCompile>
>      <Link>
>        <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies>
> @@ -140,4 +140,4 @@
>    <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
>    <ImportGroup Label="ExtensionTargets">
>    </ImportGroup>
> -</Project>
> \ No newline at end of file
> +</Project>

Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver

[Vadim Rozenfeld]

On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote:
> On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote:
> 
> <SNIP>
> 
> > > > > Hi Yan,
> > > > > 
> > > > > So recently I've been doing some KVM guest performance comparisons
> > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > > > > vhost-scsi using PCIe flash backend devices.
> > > > > 
> > > > > I've noticed that small block random performance for the MSFT guest is
> > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > > > > capable of (~500K).
> > > > > 
> > > > > After searching through the various vioscsi registry settings, it
> > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > > > > is different from what vioscsi.inx is currently defining:
> > > > > 
> > > > > [pnpsafe_pci_addreg_msix]
> > > > > HKR, "Interrupt Management",, 0x00000010
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > > > > 
> > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > > > > here, regardless of MSI_SUPPORTED:
> > > > > 
> > > > >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > > > > 
> > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > > > > appears to be the default setting for the driver included in the offical
> > > > > virtio-win iso builds, right..?
> > > > > 
> > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > > > > build of my own, but before going down the WDK development rabbit whole,
> > > > > I'd like to better understand why you've explicitly disabled this logic
> > > > > within vioscsi.c code to start..?
> > > > > 
> > > > > Is there anything that needs to be addressed / carried over from
> > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > > > > miniport code..?
> > > 
> > > Hi Nicholas,
> > > 
> > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
> > > reasons decided to keep it disabled until adding mq support.
> > > 
> > > 
> > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the
> > > driver, and switch MSISupported to 1 to make vioscsi driver working in
> > > MSI mode.
> > >    
> > 
> > Thanks for the quick response.  We'll give MSI_SUPPORTED=1 a shot over
> > the next days with a test build on Server 2012 / Server 2008 R2 and see
> > how things go..
> > 
> 
> Just a quick update on progress.
> 
> I've been able to successfully build + load a unsigned vioscsi.sys
> driver on Server 2012 with WDK 8.0.
> 
> Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant
> performance and efficiency gain, on the order of 100K to 225K IOPs for
> 4K block random I/O workload, depending on read/write mix.
> 
> Below is a simple patch to enable MSI operation by default.  Any chance
> to apply this separate from future mq efforts..?

Yes, we differently can enable MSI and rebuild vioscsi.
But then we need to re-spin WHQL testing for this particular
driver. This process requires a lot of resources, and I doubt that
it will be initiated soon, unless we have some significant amount of
bug-fixes.

Best regards,
Vadim.     

> 
> Thanks,
> 
> --nab
> 
> From 89adb6d5800386d44b36737d1587e0ffc09c4902 Mon Sep 17 00:00:00 2001
> From: Nicholas Bellinger <nab@linux-iscsi.org>
> Date: Fri, 14 Feb 2014 10:26:04 -0800
> Subject: [PATCH] vioscsi: Set MSI_SUPPORTED=1 by default
> 
> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
> ---
>  vioscsi/SOURCES         | 2 +-
>  vioscsi/vioscsi.c       | 2 --
>  vioscsi/vioscsi.inx     | 2 +-
>  vioscsi/vioscsi.vcxproj | 6 +++---
>  4 files changed, 5 insertions(+), 7 deletions(-)
> 
> diff --git a/vioscsi/SOURCES b/vioscsi/SOURCES
> index f2083de..f631bd2 100644
> --- a/vioscsi/SOURCES
> +++ b/vioscsi/SOURCES
> @@ -6,7 +6,7 @@ C_DEFINES = -D_MINORVERSION_=$(_BUILD_MINOR_VERSION_) $(C_DEFINES)
>  C_DEFINES = -D_NT_TARGET_MAJ=$(_NT_TARGET_MAJ) $(C_DEFINES)
>  C_DEFINES = -D_NT_TARGET_MIN=$(_RHEL_RELEASE_VERSION_) $(C_DEFINES)
>  
> -C_DEFINES = -DMSI_SUPPORTED=0 $(C_DEFINES)
> +C_DEFINES = -DMSI_SUPPORTED=1 $(C_DEFINES)
>  C_DEFINES = -DINDIRECT_SUPPORTED=1 $(C_DEFINES)
>  TARGETLIBS=$(SDK_LIB_PATH)\storport.lib ..\VirtIO\$(O)\virtiolib.lib
>  
> diff --git a/vioscsi/vioscsi.c b/vioscsi/vioscsi.c
> index 77c0e46..70b9bb4 100644
> --- a/vioscsi/vioscsi.c
> +++ b/vioscsi/vioscsi.c
> @@ -337,8 +337,6 @@ ENTER_FN();
>          adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1;
>      }
>  
> -    adaptExt->msix_enabled = FALSE;
> -
>      RhelDbgPrint(TRACE_LEVEL_ERROR, ("breaks_number = %x  queue_depth = %x\n",
>                  ConfigInfo->NumberOfPhysicalBreaks,
>                  adaptExt->queue_depth));
> diff --git a/vioscsi/vioscsi.inx b/vioscsi/vioscsi.inx
> index cc94b7c..ec717c6 100644
> --- a/vioscsi/vioscsi.inx
> +++ b/vioscsi/vioscsi.inx
> @@ -79,7 +79,7 @@ HKR, "Parameters", "BusType", %REG_DWORD%, 0x00000001
>  [pnpsafe_pci_addreg_msix]
>  HKR, "Interrupt Management",, 0x00000010
>  HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> -HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> +HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 1
>  HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
>  HKR, "Interrupt Management\Affinity Policy",, 0x00000010
>  HKR, "Interrupt Management\Affinity Policy", DevicePolicy, 0x00010001, 5
> diff --git a/vioscsi/vioscsi.vcxproj b/vioscsi/vioscsi.vcxproj
> index 889e513..68d4e85 100644
> --- a/vioscsi/vioscsi.vcxproj
> +++ b/vioscsi/vioscsi.vcxproj
> @@ -66,7 +66,7 @@
>    <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|Win32'">
>      <ClCompile>
>        <WarningLevel>Level3</WarningLevel>
> -      <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
> +      <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions>
>      </ClCompile>
>      <Link>
>        <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies>
> @@ -95,7 +95,7 @@
>    <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|x64'">
>      <ClCompile>
>        <WarningLevel>Level3</WarningLevel>
> -      <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
> +      <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions>
>      </ClCompile>
>      <Link>
>        <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies>
> @@ -140,4 +140,4 @@
>    <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
>    <ImportGroup Label="ExtensionTargets">
>    </ImportGroup>
> -</Project>
> \ No newline at end of file
> +</Project>

Re: MSI interrupt support with vioscsi.c miniport driver

[Nicholas A. Bellinger]

On Wed, 2014-02-19 at 19:03 +1100, Vadim Rozenfeld wrote:
> On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote:
> > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote:
> > 
> > <SNIP>
> > 
> > > > > > Hi Yan,
> > > > > > 
> > > > > > So recently I've been doing some KVM guest performance comparisons
> > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > > > > > vhost-scsi using PCIe flash backend devices.
> > > > > > 
> > > > > > I've noticed that small block random performance for the MSFT guest is
> > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > > > > > capable of (~500K).
> > > > > > 
> > > > > > After searching through the various vioscsi registry settings, it
> > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > > > > > is different from what vioscsi.inx is currently defining:
> > > > > > 
> > > > > > [pnpsafe_pci_addreg_msix]
> > > > > > HKR, "Interrupt Management",, 0x00000010
> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > > > > > 
> > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > > > > > here, regardless of MSI_SUPPORTED:
> > > > > > 
> > > > > >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > > > > > 
> > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > > > > > appears to be the default setting for the driver included in the offical
> > > > > > virtio-win iso builds, right..?
> > > > > > 
> > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > > > > > build of my own, but before going down the WDK development rabbit whole,
> > > > > > I'd like to better understand why you've explicitly disabled this logic
> > > > > > within vioscsi.c code to start..?
> > > > > > 
> > > > > > Is there anything that needs to be addressed / carried over from
> > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > > > > > miniport code..?
> > > > 
> > > > Hi Nicholas,
> > > > 
> > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
> > > > reasons decided to keep it disabled until adding mq support.
> > > > 
> > > > 
> > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the
> > > > driver, and switch MSISupported to 1 to make vioscsi driver working in
> > > > MSI mode.
> > > >    
> > > 
> > > Thanks for the quick response.  We'll give MSI_SUPPORTED=1 a shot over
> > > the next days with a test build on Server 2012 / Server 2008 R2 and see
> > > how things go..
> > > 
> > 
> > Just a quick update on progress.
> > 
> > I've been able to successfully build + load a unsigned vioscsi.sys
> > driver on Server 2012 with WDK 8.0.
> > 
> > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant
> > performance and efficiency gain, on the order of 100K to 225K IOPs for
> > 4K block random I/O workload, depending on read/write mix.
> > 
> > Below is a simple patch to enable MSI operation by default.  Any chance
> > to apply this separate from future mq efforts..?
> 
> Yes, we differently can enable MSI and rebuild vioscsi.
> But then we need to re-spin WHQL testing for this particular
> driver. This process requires a lot of resources, and I doubt that
> it will be initiated soon, unless we have some significant amount of
> bug-fixes.
> 

Any idea on a rough time frame to expect an official WHQL build with MSI
enabled..?

Or, would it be possible to generate some -BETA builds that are at least
signed and don't require extra hoops to jump through for testing..?

Thanks again,

--nab

Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver

[Nicholas A. Bellinger]

On Wed, 2014-02-19 at 19:03 +1100, Vadim Rozenfeld wrote:
> On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote:
> > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote:
> > 
> > <SNIP>
> > 
> > > > > > Hi Yan,
> > > > > > 
> > > > > > So recently I've been doing some KVM guest performance comparisons
> > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > > > > > vhost-scsi using PCIe flash backend devices.
> > > > > > 
> > > > > > I've noticed that small block random performance for the MSFT guest is
> > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > > > > > capable of (~500K).
> > > > > > 
> > > > > > After searching through the various vioscsi registry settings, it
> > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > > > > > is different from what vioscsi.inx is currently defining:
> > > > > > 
> > > > > > [pnpsafe_pci_addreg_msix]
> > > > > > HKR, "Interrupt Management",, 0x00000010
> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > > > > > 
> > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > > > > > here, regardless of MSI_SUPPORTED:
> > > > > > 
> > > > > >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > > > > > 
> > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > > > > > appears to be the default setting for the driver included in the offical
> > > > > > virtio-win iso builds, right..?
> > > > > > 
> > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > > > > > build of my own, but before going down the WDK development rabbit whole,
> > > > > > I'd like to better understand why you've explicitly disabled this logic
> > > > > > within vioscsi.c code to start..?
> > > > > > 
> > > > > > Is there anything that needs to be addressed / carried over from
> > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > > > > > miniport code..?
> > > > 
> > > > Hi Nicholas,
> > > > 
> > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
> > > > reasons decided to keep it disabled until adding mq support.
> > > > 
> > > > 
> > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the
> > > > driver, and switch MSISupported to 1 to make vioscsi driver working in
> > > > MSI mode.
> > > >    
> > > 
> > > Thanks for the quick response.  We'll give MSI_SUPPORTED=1 a shot over
> > > the next days with a test build on Server 2012 / Server 2008 R2 and see
> > > how things go..
> > > 
> > 
> > Just a quick update on progress.
> > 
> > I've been able to successfully build + load a unsigned vioscsi.sys
> > driver on Server 2012 with WDK 8.0.
> > 
> > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant
> > performance and efficiency gain, on the order of 100K to 225K IOPs for
> > 4K block random I/O workload, depending on read/write mix.
> > 
> > Below is a simple patch to enable MSI operation by default.  Any chance
> > to apply this separate from future mq efforts..?
> 
> Yes, we differently can enable MSI and rebuild vioscsi.
> But then we need to re-spin WHQL testing for this particular
> driver. This process requires a lot of resources, and I doubt that
> it will be initiated soon, unless we have some significant amount of
> bug-fixes.
> 

Any idea on a rough time frame to expect an official WHQL build with MSI
enabled..?

Or, would it be possible to generate some -BETA builds that are at least
signed and don't require extra hoops to jump through for testing..?

Thanks again,

--nab

Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver

[Vadim Rozenfeld]

On Wed, 2014-02-19 at 15:25 -0800, Nicholas A. Bellinger wrote:
> On Wed, 2014-02-19 at 19:03 +1100, Vadim Rozenfeld wrote:
> > On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote:
> > > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote:
> > > 
> > > <SNIP>
> > > 
> > > > > > > Hi Yan,
> > > > > > > 
> > > > > > > So recently I've been doing some KVM guest performance comparisons
> > > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > > > > > > vhost-scsi using PCIe flash backend devices.
> > > > > > > 
> > > > > > > I've noticed that small block random performance for the MSFT guest is
> > > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > > > > > > capable of (~500K).
> > > > > > > 
> > > > > > > After searching through the various vioscsi registry settings, it
> > > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > > > > > > is different from what vioscsi.inx is currently defining:
> > > > > > > 
> > > > > > > [pnpsafe_pci_addreg_msix]
> > > > > > > HKR, "Interrupt Management",, 0x00000010
> > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > > > > > > 
> > > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > > > > > > here, regardless of MSI_SUPPORTED:
> > > > > > > 
> > > > > > >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > > > > > > 
> > > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > > > > > > appears to be the default setting for the driver included in the offical
> > > > > > > virtio-win iso builds, right..?
> > > > > > > 
> > > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > > > > > > build of my own, but before going down the WDK development rabbit whole,
> > > > > > > I'd like to better understand why you've explicitly disabled this logic
> > > > > > > within vioscsi.c code to start..?
> > > > > > > 
> > > > > > > Is there anything that needs to be addressed / carried over from
> > > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > > > > > > miniport code..?
> > > > > 
> > > > > Hi Nicholas,
> > > > > 
> > > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
> > > > > reasons decided to keep it disabled until adding mq support.
> > > > > 
> > > > > 
> > > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the
> > > > > driver, and switch MSISupported to 1 to make vioscsi driver working in
> > > > > MSI mode.
> > > > >    
> > > > 
> > > > Thanks for the quick response.  We'll give MSI_SUPPORTED=1 a shot over
> > > > the next days with a test build on Server 2012 / Server 2008 R2 and see
> > > > how things go..
> > > > 
> > > 
> > > Just a quick update on progress.
> > > 
> > > I've been able to successfully build + load a unsigned vioscsi.sys
> > > driver on Server 2012 with WDK 8.0.
> > > 
> > > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant
> > > performance and efficiency gain, on the order of 100K to 225K IOPs for
> > > 4K block random I/O workload, depending on read/write mix.
> > > 
> > > Below is a simple patch to enable MSI operation by default.  Any chance
> > > to apply this separate from future mq efforts..?
> > 
> > Yes, we differently can enable MSI and rebuild vioscsi.
> > But then we need to re-spin WHQL testing for this particular
> > driver. This process requires a lot of resources, and I doubt that
> > it will be initiated soon, unless we have some significant amount of
> > bug-fixes.
> > 
> 
> Any idea on a rough time frame to expect an official WHQL build with MSI
> enabled..?
In June for sure :)
> Or, would it be possible to generate some -BETA builds that are at least
> signed and don't require extra hoops to jump through for testing..?

It's doable. 
I hope we can make a new build next week.
Best regards,
Vadim.

> 
> Thanks again,
> 
> --nab
> 
>