Re: [SPDK] Is there an issue with the builder?

Re: [SPDK] Is there an issue with the builder?

[Stojaczyk, Dariusz]

[-- Attachment #1: Type: text/plain, Size: 2021 bytes --]

The RBD failure looks like a bug in iSCSI - maybe it's the same issue as in https://github.com/spdk/spdk/issues/725 ?

+CC Shuhei who's working on #725.

> -----Original Message-----
> From: Shahar Salzman [mailto:shahar.salzman(a)kaminario.com]
> Sent: Wednesday, April 3, 2019 4:11 PM
> To: Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>; Storage Performance
> Development Kit <spdk(a)lists.01.org>
> Subject: Re: Is there an issue with the builder?
> 
> RBD_autotest:
> https://dqtibwqq6s6ux.cloudfront.net/results/autotest-per-
> patch/builds/27755/archive/RBD_autotest/index.html
> 
> 
> blockdev_autotest
> 
> (No logs generated)
> 
> nvme_autotest:
> https://dqtibwqq6s6ux.cloudfront.net/results/autotest-per-
> patch/builds/27755/archive/nvme_autotest/index.html
> 
> ________________________________
> 
> From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Stojaczyk, Dariusz
> <dariusz.stojaczyk(a)intel.com>
> Sent: Wednesday, April 3, 2019 4:35 PM
> To: Storage Performance Development Kit
> Subject: Re: [SPDK] Is there an issue with the builder?
> 
> That's probably an intermittent failure. Can you point me to the
> exact test that's failing?
> 
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar
> Salzman
> > Sent: Wednesday, April 3, 2019 3:12 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] Is there an issue with the builder?
> >
> > Hi,
> >
> > I am getting this strange message on a few of the tests:
> >
> > Build timed out (after 15 minutes). Marking the build as failed.
> > Build was aborted
> >
> > Is there an issue with the builder?
> >
> > Shahar
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Is there an issue with the builder?

[Howell, Seth]

[-- Attachment #1: Type: text/plain, Size: 4428 bytes --]

Hi guys,

I don't think raising the timeout is the right approach. We have it there for the purpose of killing a machine when it is hung which I believe is what is happening here. If we raise the timeout in response to timeouts, then I think we are only going to decrease the throughput of the build pool without finding the root cause.
For example, on a passing test, the rbd autotest takes about 8 minutes, and blockdev and nvme take about 5 minutes each. I think that the current timeout of 15 minutes is probably sufficient, although I am open to auditing each job to make sure there aren't any approaching too close to the proposed timeout (Maybe increase anything that normally finishes within 1 minute of the build timeout).

If there isn't a specific test or piece of code that seems to be causing a test to hang, i.e. if we see consistent timeouts on the same machine running a specific test, but those timeouts always occur at different places in the test, I think it's safe to say that there is something wrong with the way that machine is configured and it needs to be fixed up.

Thanks,

Seth

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dariusz
Sent: Wednesday, April 3, 2019 9:57 AM
To: Luse, Paul E <paul.e.luse(a)intel.com>; Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Is there an issue with the builder?

I just saw it on my patch from today: https://dqtibwqq6s6ux.cloudfront.net/results/autotest-per-patch/builds/27728/archive/unittest_autotests/build.log

It looks suspicious, a completely different test suite (unittests) timed out on building almost the same unit tests.

> -----Original Message-----
> From: Luse, Paul E
> Sent: Wednesday, April 3, 2019 5:47 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>; 
> Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>
> Subject: RE: Is there an issue with the builder?
> 
> Hi Guys-
> 
> So this happened to one of my patches last week also. Maybe it's worth 
> bumping up the timeout from 15 to maybe 20 min and keeping an eye on 
> things for a while? I looked through the 2ng log below and didn't see 
> any one item that seemed to be the culprit.
> 
> -Paul
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar 
> Salzman
> Sent: Wednesday, April 3, 2019 7:11 AM
> To: Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>; Storage 
> Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Is there an issue with the builder?
> 
> RBD_autotest:
> https://dqtibwqq6s6ux.cloudfront.net/results/autotest-per-
> patch/builds/27755/archive/RBD_autotest/index.html
> 
> blockdev_autotest
> (No logs generated)
> 
> nvme_autotest:
> https://dqtibwqq6s6ux.cloudfront.net/results/autotest-per-
> patch/builds/27755/archive/nvme_autotest/index.html
> ________________________________
> From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Stojaczyk, Dariusz 
> <dariusz.stojaczyk(a)intel.com>
> Sent: Wednesday, April 3, 2019 4:35 PM
> To: Storage Performance Development Kit
> Subject: Re: [SPDK] Is there an issue with the builder?
> 
> That's probably an intermittent failure. Can you point me to the exact 
> test that's failing?
> 
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar 
> > Salzman
> > Sent: Wednesday, April 3, 2019 3:12 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] Is there an issue with the builder?
> >
> > Hi,
> >
> > I am getting this strange message on a few of the tests:
> >
> > Build timed out (after 15 minutes). Marking the build as failed.
> > Build was aborted
> >
> > Is there an issue with the builder?
> >
> > Shahar
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Is there an issue with the builder?

[Stojaczyk, Dariusz]

[-- Attachment #1: Type: text/plain, Size: 2854 bytes --]

I just saw it on my patch from today: https://dqtibwqq6s6ux.cloudfront.net/results/autotest-per-patch/builds/27728/archive/unittest_autotests/build.log

It looks suspicious, a completely different test suite (unittests) timed out on building almost the same unit tests.

> -----Original Message-----
> From: Luse, Paul E
> Sent: Wednesday, April 3, 2019 5:47 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>; Stojaczyk,
> Dariusz <dariusz.stojaczyk(a)intel.com>
> Subject: RE: Is there an issue with the builder?
> 
> Hi Guys-
> 
> So this happened to one of my patches last week also. Maybe it's worth
> bumping up the timeout from 15 to maybe 20 min and keeping an eye on
> things for a while? I looked through the 2ng log below and didn't see any one
> item that seemed to be the culprit.
> 
> -Paul
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar Salzman
> Sent: Wednesday, April 3, 2019 7:11 AM
> To: Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>; Storage Performance
> Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Is there an issue with the builder?
> 
> RBD_autotest:
> https://dqtibwqq6s6ux.cloudfront.net/results/autotest-per-
> patch/builds/27755/archive/RBD_autotest/index.html
> 
> blockdev_autotest
> (No logs generated)
> 
> nvme_autotest:
> https://dqtibwqq6s6ux.cloudfront.net/results/autotest-per-
> patch/builds/27755/archive/nvme_autotest/index.html
> ________________________________
> From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Stojaczyk, Dariusz
> <dariusz.stojaczyk(a)intel.com>
> Sent: Wednesday, April 3, 2019 4:35 PM
> To: Storage Performance Development Kit
> Subject: Re: [SPDK] Is there an issue with the builder?
> 
> That's probably an intermittent failure. Can you point me to the exact test
> that's failing?
> 
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar
> > Salzman
> > Sent: Wednesday, April 3, 2019 3:12 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] Is there an issue with the builder?
> >
> > Hi,
> >
> > I am getting this strange message on a few of the tests:
> >
> > Build timed out (after 15 minutes). Marking the build as failed.
> > Build was aborted
> >
> > Is there an issue with the builder?
> >
> > Shahar
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Is there an issue with the builder?

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 2170 bytes --]

Hi Guys-

So this happened to one of my patches last week also. Maybe it's worth bumping up the timeout from 15 to maybe 20 min and keeping an eye on things for a while? I looked through the 2ng log below and didn't see any one item that seemed to be the culprit.

-Paul

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar Salzman
Sent: Wednesday, April 3, 2019 7:11 AM
To: Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>; Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Is there an issue with the builder?

RBD_autotest:
https://dqtibwqq6s6ux.cloudfront.net/results/autotest-per-patch/builds/27755/archive/RBD_autotest/index.html

blockdev_autotest
(No logs generated)

nvme_autotest:
https://dqtibwqq6s6ux.cloudfront.net/results/autotest-per-patch/builds/27755/archive/nvme_autotest/index.html
________________________________
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>
Sent: Wednesday, April 3, 2019 4:35 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] Is there an issue with the builder?

That's probably an intermittent failure. Can you point me to the exact test that's failing?

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar 
> Salzman
> Sent: Wednesday, April 3, 2019 3:12 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Is there an issue with the builder?
>
> Hi,
>
> I am getting this strange message on a few of the tests:
>
> Build timed out (after 15 minutes). Marking the build as failed.
> Build was aborted
>
> Is there an issue with the builder?
>
> Shahar
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Is there an issue with the builder?

[Shahar Salzman]

[-- Attachment #1: Type: text/plain, Size: 1437 bytes --]

RBD_autotest:
https://dqtibwqq6s6ux.cloudfront.net/results/autotest-per-patch/builds/27755/archive/RBD_autotest/index.html

blockdev_autotest
(No logs generated)

nvme_autotest:
https://dqtibwqq6s6ux.cloudfront.net/results/autotest-per-patch/builds/27755/archive/nvme_autotest/index.html
________________________________
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>
Sent: Wednesday, April 3, 2019 4:35 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] Is there an issue with the builder?

That's probably an intermittent failure. Can you point me to the
exact test that's failing?

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar Salzman
> Sent: Wednesday, April 3, 2019 3:12 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Is there an issue with the builder?
>
> Hi,
>
> I am getting this strange message on a few of the tests:
>
> Build timed out (after 15 minutes). Marking the build as failed.
> Build was aborted
>
> Is there an issue with the builder?
>
> Shahar
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Is there an issue with the builder?

[Stojaczyk, Dariusz]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

That's probably an intermittent failure. Can you point me to the
exact test that's failing?

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar Salzman
> Sent: Wednesday, April 3, 2019 3:12 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Is there an issue with the builder?
> 
> Hi,
> 
> I am getting this strange message on a few of the tests:
> 
> Build timed out (after 15 minutes). Marking the build as failed.
> Build was aborted
> 
> Is there an issue with the builder?
> 
> Shahar
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

[SPDK] Is there an issue with the builder?

[Shahar Salzman]

[-- Attachment #1: Type: text/plain, Size: 201 bytes --]

Hi,

I am getting this strange message on a few of the tests:

Build timed out (after 15 minutes). Marking the build as failed.
Build was aborted

Is there an issue with the builder?

Shahar