From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by yocto-www.yoctoproject.org (Postfix, from userid 118) id 50A25E00C99; Sat, 5 Jan 2019 13:45:42 -0800 (PST) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on yocto-www.yoctoproject.org X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HTML_MESSAGE, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-HAM-Report: * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no * trust * [209.85.221.174 listed in list.dnswl.org] * 0.0 HTML_MESSAGE BODY: HTML included in message * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's * domain * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid Received: from mail-vk1-f174.google.com (mail-vk1-f174.google.com [209.85.221.174]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id 34CF4E00901 for ; Sat, 5 Jan 2019 13:45:40 -0800 (PST) Received: by mail-vk1-f174.google.com with SMTP id y14so8670007vkd.1 for ; Sat, 05 Jan 2019 13:45:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=a9ID8l6Gezbe8yfC+Z6Vk5MhOwN49QnFmlZ+eZZyBQc=; b=E2QDbaRBSqtl1bKU9kJhwBa0mHz3whwWR6uvX54W12uxT1iotZuhFjPNYI8Hxz31aO 6hqFRv57cJ/nIjM32s+R1HcOr7n65iNoLYx6Ns+278ZNhnYCkkbilkgVebXYATBXgsb7 UhbkNW/56NLt6K2Ca4iV+VPs3zhB4y+V4NtkY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=a9ID8l6Gezbe8yfC+Z6Vk5MhOwN49QnFmlZ+eZZyBQc=; b=j68V300Q1ymkHER3EMyuV7Oy8b+YQ1LQLk5yUepfeUnz7MzqFTeWtbCbtPUzG+dVvD JSOWHO7R9MX+0KKRgpb22U/OotH71Eq5uzjEJtpz7dbkUPNfbPNcqp5pGy0qQq6huTEa 4fIlrO9HD8F52rgtir8t4RcgiDfH9wr1mo9ztO94iOY+hfvQTXumZA9v/tqDEx1kk4YM eVH1sV0r1DyMJ31GNdoIjxfUidz3Yna6IJarihCCuu+U+JjHYTKe3qhNlRAGoUucWgF1 LDHNR9B6MSDGAAYA6NF9oyPIwbWQkwbrWWlYR3BinPciXPZtEhEl1SbLU+sTkgDxGVcd RjdQ== X-Gm-Message-State: AJcUukfrtoGsaW3pi2IGNvfRKgVMFzMzVJwMb/I4GEdTZSM6tNQTQZqF HRNV9g5TuDwJkRiKfFGYWGt6AxxuxLG1wccwm9sgdA== X-Google-Smtp-Source: ALg8bN776mewfgIhWwibbkxbm363tzooNrX9pNbk+0wuJ7ktFYwXcvlcLrBcUsiLA9LCrT2G5l+twTSNmtCTiNla3gE= X-Received: by 2002:a1f:4d47:: with SMTP id a68mr20110077vkb.34.1546724740327; Sat, 05 Jan 2019 13:45:40 -0800 (PST) MIME-Version: 1.0 References: <20190103180218.6968-1-anibal.limon@linaro.org> <6fc3263d9097d1af78c96df089bfe1ca3184efdc.camel@linuxfoundation.org> In-Reply-To: From: Anibal Limon Date: Sat, 5 Jan 2019 15:45:29 -0600 Message-ID: To: Richard Purdie Cc: yocto@yoctoproject.org Subject: Re: [PATCH][ptest-runner] ptest-runner: Add support timeout failure type to XML result file X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Jan 2019 21:45:42 -0000 Content-Type: multipart/alternative; boundary="000000000000b690c2057ebcead3" --000000000000b690c2057ebcead3 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, 5 Jan 2019 at 12:52, wrote: > On Sat, 2019-01-05 at 11:30 -0600, Anibal Limon wrote: > > > > > > On Thu, 3 Jan 2019 at 15:45, > > wrote: > > > On Thu, 2019-01-03 at 12:02 -0600, An=C3=ADbal Lim=C3=B3n wrote: > > > > The ptest-runner support logging results to stdout and to a XML > > > file > > > > in stdout the ptest is mark as: > > > > > > > > ... > > > > ERROR: Exit status is 1 > > > > TIMEOUT: ptest-directory > > > > ... > > > > > > > > Add the same support in XML file for example, > > > > > > > > ... > > > > > > > > > > > 1'> > > > > > > > > > > > > ... > > > > > > > > [YOCTO #13088] > > > > > > > > Signed-off-by: An=C3=ADbal Lim=C3=B3n > > > > > > Sounds good, thanks! > > > > > > I have a feeling this will show up a few bugs but we should find > > > and > > > fix the ones which are timing out... > > > > > > I'd still be interested in the time each directory takes too if we > > > can > > > get that as it would help us decide which tests to run. > > > > The timeout is set to 5 minutes (300 secs) by default and currently > > there is no way to set a timeout per ptest, > > I will create other patch to add the timeout value to stdout and XML > > file. > > That is handy but what I mean is the execution time each one takes. If > we have test which take 5s its easier to run them regularly than tests > which take 500s. Having an idea of their relative execution time would > therefore be helpful. > Got it, I can add a duration for example:. In stdout will be (after end), ... BEGIN: TIME END: ptest-directory DURATION: Ns END: TIME ... In XML: ... ... or ... Ns ... What is the better? or any other? > I assume the timeout takes effect after no output for that period? > Right, the timeout is based on stdout data. Anibal > > Cheers, > > Richard > > --000000000000b690c2057ebcead3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Sat, 5 Jan 2019 at 12:52, <richard.purdie@linuxfoundati= on.org> wrote:
On Sat, 2019-01-05 at 11:30 -0600, Anibal Limon wrote:
>
>
> On Thu, 3 Jan 2019 at 15:45, <richard.purdie@linuxfoundation.org&g= t;
> wrote:
> > On Thu, 2019-01-03 at 12:02 -0600, An=C3=ADbal Lim=C3=B3n wrote:<= br> > > > The ptest-runner support logging results to stdout and to a = XML
> > file
> > > in stdout the ptest is mark as:
> > >
> > > ...
> > > ERROR: Exit status is 1
> > > TIMEOUT: ptest-directory
> > > ...
> > >
> > > Add the same support in XML file for example,
> > >
> > > ...
> > > <testcase classname=3D'ptest-directory' name=3D&#= 39;run-ptest'>
> > >=C2=A0 =C2=A0<failure type=3D'exit_code' message= =3D'run-ptest exited with code:
> > > 1'></failure>
> > >=C2=A0 =C2=A0<failure type=3D'timeout'/>
> > > </testcase>
> > > ...
> > >
> > > [YOCTO #13088]
> > >
> > > Signed-off-by: An=C3=ADbal Lim=C3=B3n <anibal.limon@linaro.org><= br> > >
> > Sounds good, thanks!
> >
> > I have a feeling this will show up a few bugs but we should find<= br> > > and
> > fix the ones which are timing out...
> >
> > I'd still be interested in the time each directory takes too = if we
> > can
> > get that as it would help us decide which tests to run.
>
> The timeout is set to 5 minutes (300 secs) by default and currently > there is no way to set a timeout per ptest,
> I will create other patch to add the timeout value to stdout and XML > file.

That is handy but what I mean is the execution time each one takes. If
we have test which take 5s its easier to run them regularly than tests
which take 500s. Having an idea of their relative execution time would
therefore be helpful.

Got it, I can add= a duration for example:.

In stdout will be (after= end),

...
BEGIN: TIME
END: pt= est-directory
DURATION: Ns
END: TIME
...<= /div>

In XML:

...
<= ;testcase classname=3D'%s' name=3D'run-ptest' duration=3D&#= 39;Ns'>
</testcase>
...

=
or=C2=A0

...
<testcas= e classname=3D'%s' name=3D'run-ptest' >
&l= t;duration>Ns</duration>
</testcase>
...=

What is the better? or any other?=C2=A0


I assume the timeout takes effect after no output for that period?

Right, the timeout is based on stdout data.

Anibal=C2=A0

Cheers,

Richard

--000000000000b690c2057ebcead3--