Re: [SPDK] CI latent failures

* Re: [SPDK] CI latent failures
@ 2018-07-20 13:37 Luse, Paul E
  0 siblings, 0 replies; 5+ messages in thread
From: Luse, Paul E @ 2018-07-20 13:37 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 5282 bytes --]

Hi Maciej,

Thanks for doing this! I've added the SPDK dist list to the email as this is the kind of info that everyone can benefit from. Not only do we need more community members stepping up and doing this kind of analysis, but if others are experiencing any of these things we might be able to get some traction in debug.

Are you by any chance able to cross reference these with github issues (not a major deep comparison, at least a scan) to try and identify any that are already reported and if so put the CI failure link in github?  For those you can find maybe enter one issue per item but wait probably 24 hours to see if anyone out there jumps up ans says "I've been looking at that one" (there's always hope)

Thanks again
Paul

Hi,
As we are close to release we wanted to look closer at CI latent failures in case there is something that we should address before release. I went through all reported failures from this quarter. I was able to divide those failures to 11 cases. Some of the failures where caused by failing test environment, some were caused by other software, some were probably caused by some bugs.... Below is more detailed report for each of the 11 cases. I didn't want to go too deep into debugging, but rather conclude based on my experience, so if you see that something is actually very different than I assumed please speak up.

Case 1.
Issue: Segmentation fault on NVMf shutdown
Number of failures: 8
Link to latest failure: http://spdk.intel.com/public/spdk/builds/review/97a5813b0bc8fa6b00aabf8b31f8a0b0776e8458.1531262768/fedora-06/build.log
Description: This one occurs from time to time starting from around mid-June and is one of the latest ones. I think this one should examined closer as a potential bug.

Case 2.
Issue: sock.c:249:11: runtime error: member access within null pointer of type 'struct spdk_sock' (aka spdk_iscsi_conn_destruct)
Number of failures: 5
Link to latest failure: http://spdk.intel.com/public/spdk/builds/review/0585c2426071f9506dbe520234e2c3ee2f5aee7d.1531468630/fedora-06/build.log
Description: This one occurs from time to time and is one of the latest ones. I think this one should examined closer as a potential bug.

Case 3.
Issue: nbd.c: 879:spdk_nbd_start: *ERROR*: ioctl(NBD_SET_SOCK) failed: Device or resource busy
Number of failures: 2
Link to latest failure: http://spdk.intel.com/public/spdk/builds/review/c3ba7cb2ce0bd01ec61f93e852cd37d110fa9d9d.1531431333/ubuntu17.10/build.log
Description: Looks like issue with NBD access (environment clean up after failure issue?).

Case 4.
Issue: thread.c: 343:spdk_io_device_register: *ERROR*: io_device 0x1dd6040 already registered
Number of failures: 4
Link to latest failure: http://spdk.intel.com/public/spdk/builds/review/3aa13c878c94c8925bbf3647ca67b666dfda6c75.1530826421/fedora-02/build.log
Description: These series of 4 failures occurred one after another around 2 weeks ago and never happened again. It looks like there was some bug that was fixed. We should monitor if it happens again.

Case 5.
Issue: rmmod: ERROR: Module nvme_rdma is in use
Number of failures: 2
Link to latest failure: http://spdk.intel.com/public/spdk/builds/review/1939cfb85ee7c2d7707ade0e49c0e8bd01b04961.1529624900/fedora-06/build.log
Description: Looks like issue with nvme_rdma module (environment clean up after failure issue?).

Case 6.
Issue: initiator.sh: line 43:  9367 Bus error               (core dumped) $rootdir/test/bdev/bdevperf/bdevperf -c $testdir/bdev.conf -q 128 -s 4096 -w verify -t 5 -d 512
Number of failures: 1
Link to latest failure: http://spdk.intel.com/public/spdk/builds/review/5b32f26f10d709f939443f777956574b77955f09.1528385513/fedora-06/build.log
Description: Happened only once at the beginning of June. We should monitor if it happens again.

Case 7.
Issue: VM shutdown issue
Number of failures: 2
Link to latest failure: <sorry, no link here>
Description: VM failed to shut down - test script timed out.

Case 8.
Issue: NVMf segmentation fault on disconnect
Number of failures: 6
Link to latest failure: http://spdk.intel.com/public/spdk/builds/review/9031d3bf2a94d15c2119c2e582a2ce5362cfa461.1526309227/fedora-06/build.log
Description: Happened one after another in May. I guess it was related to some bug that has been fixed.

Case 9.
Issue: Assan scan issue
Number of failures: 3
Link to latest failure: <sorry, no link here>
Description: Happened in May in short period of time. Probably fixed right now.

Case 10.
Issue: conn.c: 741:spdk_iscsi_conn_read_data: *ERROR*: spdk_sock_recv() failed, errno 104: Connection reset by peer
Number of failures: 3
Link to latest failure: http://spdk.intel.com/public/spdk/builds/review/f77339505066367ec6c4279dd230fe1a64626032.1525681774/fedora-06/build.log
Description: Happened one after another in May. Looks like environment failure.

Case 11.
Issue: nvme_rdma.c:1394:nvme_rdma_ctrlr_construct: *ERROR*: failed to create admin qpair
Number of failures: 1
Link to latest failure: http://spdk.intel.com/public/spdk/builds/review/e7713a2746bc8e72b4ec2b17622e102aebdf8c79.1525462134/fedora-03/build.log
Description: Happened only once at the beginning of May. We should monitor if it happens again.

Maciek

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 11625 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread