[SPDK] Code robustness test

[SPDK] Code robustness test

[E.W.Z.]

[-- Attachment #1: Type: text/plain, Size: 271 bytes --]

Hi,
Is there anything that can wreck havoc on NVMe to check how well the code copes with it? Something like a script or utility that runs in background and resets PCIe controller, detaches disks and anything else that could theoretically happen to NVMe.
Thanks,
E.


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 1573 bytes --]

Re: [SPDK] Code robustness test

[Ernest Zaslavsky]

[-- Attachment #1: Type: text/plain, Size: 4214 bytes --]

Sounds like just what the doctor ordered. Will check it out. Thanks!


Sent from Mail for Windows 10

From: Szwed, Maciej
Sent: Thursday, February 8, 2018 9:12 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] Code robustness test

Hi,
For reset you can use our error injection framework. You can find it in this location: spdk\lib\bdev\error . You can configure it using RPC commands. What it basically does is to add another layer on top of existing one (like NVMe) and simulate errors. It can simulate resets. Let me know if this is something that you were looking for.

Regards,
Maciek

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zaslavsky
Sent: Thursday, February 8, 2018 6:57 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Code robustness test

Hi Paul,
Thanks for your reply. I wasn’t clear enough with my question. Sure you have tests in SPDK, but I was asking for something that may assist me in testing my code. I’m writing something that uses NVMe driver. I’m trying to handle errors where applicable. Now I want to simulate real world problems which may occur (like controller reset) and see if my code handles such a situation.
Thanks,
E.

From: Luse, Paul E
Sent: Thursday, February 8, 2018 1:37 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] Code robustness test

Hi,

SPDK is tested in “tiers” and nvme is no different other than it may have more robust tests than some of the other modules due to its maturity:

- unit tests:  Look in /spdk/test/unit/lib/nvme and you’ll find “per file” unit tests that use C-unit.  These are, of course, SW only tests focused around exercising the public API in each file geared towards catching regressions.  Every new public function added is required to have a new unit test. Coverage reports are available on the CI page at spdk.io and there are clearly opportunities there for anyone that wants to beef it up.
- component level tests attempt to isolate just a pustular module (sometimes a few other are required to be involved) and use either real hardware or run on a VM with emulated NVMe devices.  They are in spdk/test/lib/nvme mainly via nvme.sh however you’ll find a few standalone focused tests that operate solely on the component, like sgl, in cases where we really want to throw strange things at complicated code (like sgl and prp creation). You’ll find a few other focused tets in here like hotplug.sh for hotplug testing and aer for Async Event Requests and overhead for latency measurements
- system: there are bunch of other tests that use NVMe as a back end but are focused on system level applications that are part of SPDK like the NVMe-oF target tests, the vHost tests. iSCSI tests, etc.
- performance: these are pretty manual right now so there’s not much to point at but as the name implies they’re really about performance stress testing.

Basically there’s a TON of tests in the repo. There’s a lot of activity right now ongoing to beef up the automated testing, to move our CI over to Jenkins and to rework the dozens of bash scripts that control the various tests.  The goal there is to make it easier for anyone to come in and find what they are interested in to either understand testing, help beef it up or reproduce a failure locally that they encountered on a patch in CI.

After you dig around a little feel free to ask more questions and someone will jump in and provide more detail.  As the effort I just mentioned makes progress there will be new docs added to http://spdk.io that explain the test strategy and hierarchy in a clear way - we hope ☺

-Paul



From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of E.W.Z.
Sent: Wednesday, February 7, 2018 9:14 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Code robustness test

Hi,
Is there anything that can wreck havoc on NVMe to check how well the code copes with it? Something like a script or utility that runs in background and resets PCIe controller, detaches disks and anything else that could theoretically happen to NVMe.
Thanks,
E.




[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 8955 bytes --]

Re: [SPDK] Code robustness test

[Szwed, Maciej]

[-- Attachment #1: Type: text/plain, Size: 4039 bytes --]

Hi,
For reset you can use our error injection framework. You can find it in this location: spdk\lib\bdev\error . You can configure it using RPC commands. What it basically does is to add another layer on top of existing one (like NVMe) and simulate errors. It can simulate resets. Let me know if this is something that you were looking for.

Regards,
Maciek

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zaslavsky
Sent: Thursday, February 8, 2018 6:57 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Code robustness test

Hi Paul,
Thanks for your reply. I wasn’t clear enough with my question. Sure you have tests in SPDK, but I was asking for something that may assist me in testing my code. I’m writing something that uses NVMe driver. I’m trying to handle errors where applicable. Now I want to simulate real world problems which may occur (like controller reset) and see if my code handles such a situation.
Thanks,
E.

From: Luse, Paul E<mailto:paul.e.luse(a)intel.com>
Sent: Thursday, February 8, 2018 1:37 AM
To: Storage Performance Development Kit<mailto:spdk(a)lists.01.org>
Subject: Re: [SPDK] Code robustness test

Hi,

SPDK is tested in “tiers” and nvme is no different other than it may have more robust tests than some of the other modules due to its maturity:

- unit tests:  Look in /spdk/test/unit/lib/nvme and you’ll find “per file” unit tests that use C-unit.  These are, of course, SW only tests focused around exercising the public API in each file geared towards catching regressions.  Every new public function added is required to have a new unit test. Coverage reports are available on the CI page at spdk.io and there are clearly opportunities there for anyone that wants to beef it up.
- component level tests attempt to isolate just a pustular module (sometimes a few other are required to be involved) and use either real hardware or run on a VM with emulated NVMe devices.  They are in spdk/test/lib/nvme mainly via nvme.sh however you’ll find a few standalone focused tests that operate solely on the component, like sgl, in cases where we really want to throw strange things at complicated code (like sgl and prp creation). You’ll find a few other focused tets in here like hotplug.sh for hotplug testing and aer for Async Event Requests and overhead for latency measurements
- system: there are bunch of other tests that use NVMe as a back end but are focused on system level applications that are part of SPDK like the NVMe-oF target tests, the vHost tests. iSCSI tests, etc.
- performance: these are pretty manual right now so there’s not much to point at but as the name implies they’re really about performance stress testing.

Basically there’s a TON of tests in the repo. There’s a lot of activity right now ongoing to beef up the automated testing, to move our CI over to Jenkins and to rework the dozens of bash scripts that control the various tests.  The goal there is to make it easier for anyone to come in and find what they are interested in to either understand testing, help beef it up or reproduce a failure locally that they encountered on a patch in CI.

After you dig around a little feel free to ask more questions and someone will jump in and provide more detail.  As the effort I just mentioned makes progress there will be new docs added to http://spdk.io that explain the test strategy and hierarchy in a clear way - we hope ☺

-Paul



From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of E.W.Z.
Sent: Wednesday, February 7, 2018 9:14 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] Code robustness test

Hi,
Is there anything that can wreck havoc on NVMe to check how well the code copes with it? Something like a script or utility that runs in background and resets PCIe controller, detaches disks and anything else that could theoretically happen to NVMe.
Thanks,
E.



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 9745 bytes --]

Re: [SPDK] Code robustness test

[Ernest Zaslavsky]

[-- Attachment #1: Type: text/plain, Size: 3364 bytes --]

Hi Paul,
Thanks for your reply. I wasn’t clear enough with my question. Sure you have tests in SPDK, but I was asking for something that may assist me in testing my code. I’m writing something that uses NVMe driver. I’m trying to handle errors where applicable. Now I want to simulate real world problems which may occur (like controller reset) and see if my code handles such a situation.
Thanks,
E.

From: Luse, Paul E
Sent: Thursday, February 8, 2018 1:37 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] Code robustness test

Hi,

SPDK is tested in “tiers” and nvme is no different other than it may have more robust tests than some of the other modules due to its maturity:

- unit tests:  Look in /spdk/test/unit/lib/nvme and you’ll find “per file” unit tests that use C-unit.  These are, of course, SW only tests focused around exercising the public API in each file geared towards catching regressions.  Every new public function added is required to have a new unit test. Coverage reports are available on the CI page at spdk.io and there are clearly opportunities there for anyone that wants to beef it up.
- component level tests attempt to isolate just a pustular module (sometimes a few other are required to be involved) and use either real hardware or run on a VM with emulated NVMe devices.  They are in spdk/test/lib/nvme mainly via nvme.sh however you’ll find a few standalone focused tests that operate solely on the component, like sgl, in cases where we really want to throw strange things at complicated code (like sgl and prp creation). You’ll find a few other focused tets in here like hotplug.sh for hotplug testing and aer for Async Event Requests and overhead for latency measurements
- system: there are bunch of other tests that use NVMe as a back end but are focused on system level applications that are part of SPDK like the NVMe-oF target tests, the vHost tests. iSCSI tests, etc.
- performance: these are pretty manual right now so there’s not much to point at but as the name implies they’re really about performance stress testing.

Basically there’s a TON of tests in the repo. There’s a lot of activity right now ongoing to beef up the automated testing, to move our CI over to Jenkins and to rework the dozens of bash scripts that control the various tests.  The goal there is to make it easier for anyone to come in and find what they are interested in to either understand testing, help beef it up or reproduce a failure locally that they encountered on a patch in CI.

After you dig around a little feel free to ask more questions and someone will jump in and provide more detail.  As the effort I just mentioned makes progress there will be new docs added to http://spdk.io that explain the test strategy and hierarchy in a clear way - we hope ☺

-Paul



From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of E.W.Z.
Sent: Wednesday, February 7, 2018 9:14 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Code robustness test

Hi,
Is there anything that can wreck havoc on NVMe to check how well the code copes with it? Something like a script or utility that runs in background and resets PCIe controller, detaches disks and anything else that could theoretically happen to NVMe.
Thanks,
E.



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 6825 bytes --]

Re: [SPDK] Code robustness test

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 2794 bytes --]

Hi,

SPDK is tested in “tiers” and nvme is no different other than it may have more robust tests than some of the other modules due to its maturity:

- unit tests:  Look in /spdk/test/unit/lib/nvme and you’ll find “per file” unit tests that use C-unit.  These are, of course, SW only tests focused around exercising the public API in each file geared towards catching regressions.  Every new public function added is required to have a new unit test. Coverage reports are available on the CI page at spdk.io and there are clearly opportunities there for anyone that wants to beef it up.
- component level tests attempt to isolate just a pustular module (sometimes a few other are required to be involved) and use either real hardware or run on a VM with emulated NVMe devices.  They are in spdk/test/lib/nvme mainly via nvme.sh however you’ll find a few standalone focused tests that operate solely on the component, like sgl, in cases where we really want to throw strange things at complicated code (like sgl and prp creation). You’ll find a few other focused tets in here like hotplug.sh for hotplug testing and aer for Async Event Requests and overhead for latency measurements
- system: there are bunch of other tests that use NVMe as a back end but are focused on system level applications that are part of SPDK like the NVMe-oF target tests, the vHost tests. iSCSI tests, etc.
- performance: these are pretty manual right now so there’s not much to point at but as the name implies they’re really about performance stress testing.

Basically there’s a TON of tests in the repo. There’s a lot of activity right now ongoing to beef up the automated testing, to move our CI over to Jenkins and to rework the dozens of bash scripts that control the various tests.  The goal there is to make it easier for anyone to come in and find what they are interested in to either understand testing, help beef it up or reproduce a failure locally that they encountered on a patch in CI.

After you dig around a little feel free to ask more questions and someone will jump in and provide more detail.  As the effort I just mentioned makes progress there will be new docs added to http://spdk.io that explain the test strategy and hierarchy in a clear way - we hope ☺

-Paul


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of E.W.Z.
Sent: Wednesday, February 7, 2018 9:14 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Code robustness test

Hi,
Is there anything that can wreck havoc on NVMe to check how well the code copes with it? Something like a script or utility that runs in background and resets PCIe controller, detaches disks and anything else that could theoretically happen to NVMe.
Thanks,
E.


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 6390 bytes --]