From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=yrtm8LBv/3RnKmtIyK4t0oV7+aB6eP5/cq1ITIJRbYE=; b=sgNJpj4WBN1LWYJ+tfp3YFveLS0rNtWJBshQkgmcw+89vq8zwywx0RaSUZagPEhm5b BhLmm+ORabRPKdqfgsTBzVzLlUsIqoW+inkxUBI2AcC7qP8lRsa60pfJJB1jC7oSkiIz 03d7cHq16d86nwVAogNKhbqu7a/9QZgjQZfH7Y2SvdcxQ4BaSHuQB6fCSHfhcS0NmCQE j0ymWs2oVEs35Cx+EwF9huChybtIB55tyZstQ8pB2VcgHhxaxRix4wuynJhQEd5gDFYx gQE9ybnMhiP3Pu9rvyF90dvWigEJ/7VgDzlffLxJ7u3h2Kx13XgMMEcHolSuNikvafep fVQQ== MIME-Version: 1.0 In-Reply-To: References: From: dhinakar k Date: Fri, 31 Mar 2017 09:21:21 +0530 Message-ID: Content-Type: multipart/alternative; boundary=f4030435448c68e410054bfebba3 Subject: Re: [Fuego] UART/USB Connectivity, History Database, Test result format List-Id: Mailing list for the Fuego test framework List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Bird, Timothy" Cc: "fuego@lists.linuxfoundation.org" --f4030435448c68e410054bfebba3 Content-Type: text/plain; charset=UTF-8 Hi Tim, That was quite a detailed reply. Thanks a lot. Fuego looks promising, I am looking forward to the future releases of Fuego. Regards, Dhinakar On Thu, Mar 30, 2017 at 11:21 PM, Bird, Timothy wrote: > > > > -----Original Message----- > > From: dhinakar k on Tuesday, March 28, 2017 7:34 PM > > > > Hi Tim, > > > > That was a very detailed reply. > > Thank you very much. > > > > Just one follow up question. > > How does Fuego compare with Avocado? > > If you had checked it out already please let me know. > > I did take a quick look at Avacado a while back, and put > some notes at: > http://bird.org/fuego/Other_test_systems#Avacado > > Avacado has some very nice features. > It seems like it excels at handling large test matrices. It has yaml > files to define the different test variations. > > They also interface well with Jenkins - a single command-line from a > Jenkins > job can start an avocado test, and they product output that is easily > parsed > by existing Jenkins post-processors. > e.g. 'avacado run /bin/true sleeptest passtest --xunit > "$WORKSPACE/results.xml"...' > > They have an 'avacado server' concept, where each host can execute jobs > from > another machine (I believe - I haven't tried it out). > > So far, it looks like Avacado is used for virtualization testing. I don't > see a > lot of different tests that run on this framework. > > Here are some of the major difference, that I can see (this is with limited > study of Avacado, so I apologize for any omissions of features on their > part): > 1) Fuego is both a test framework, and a distribution of tests themselves. > * We have about 60 tests now, with about 90 in our 'next' branch > * the goal is to have hundreds of tests available, for a wide variety > of system features and issues > 2) Fuego provides a consistent, containerized back end (docker) so that the > builds of the test software can be uniform between different sites. > * this will help when comparing results from different sites > * multi-site results comparisons are not done yet, but on our > roadmap > 3) Fuego has a multi-node test API (host/target), that is specially geared > for > embedded linux testing > * the footprint on the target is intentionally very small, using > existing POSIX features > * IOW - there is no specialized target agent on the target board > (except for a transport agent, like sshd or adb) > * Fuego is geared towards testing of final production images of the > software stacks for products > 4) Fuego is behind Avacado in terms of: > a) unified test ouput (we're working on that now) > b) test results sharing > * (also in-progress - 'runs' can be packaged and shared now to a > central server, in our 'next' branch) > * however, there's only rudimentary test display and analysis on > the Fuego server at the moment > * Avacado seems to be using Jenkins for this - and we should > look at them more closely to see what they're doing > > There are more, but hopefully this is helpful to see some of the > differences. > -- Tim > > --f4030435448c68e410054bfebba3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Tim,

That was quite a det= ailed reply.
Thanks a lot.
Fuego looks promising, I a= m looking forward to the future releases of Fuego.

Regards,
Dhinakar=C2=A0
On Thu, Mar 30, 2017 at 11:21 PM, Bird, Timoth= y <Tim.Bird@sony.com> wrote:


> -----Original Message-----
> From: dhinakar k on Tuesday, March 28, 2017 7:34 PM
>
> Hi Tim,
>
> That was a very detailed reply.
> Thank you very much.
>
> Just one follow up question.
> How does Fuego compare with Avocado?
> If you had checked it out already please let me know.

I did take a quick look at Avacado a while back, and put
some notes at:
http://bird.org/fuego/Other_test_systems#Avacad= o

Avacado has some very nice features.
It seems like it excels at handling large test matrices.=C2=A0 It has yaml<= br> files to define the different test variations.

They also interface well with Jenkins - a single command-line from a Jenkin= s
job can start an avocado test, and they product output that is easily parse= d
by existing Jenkins post-processors.
e.g. 'avacado run /bin/true sleeptest passtest --xunit "$WORKSPACE= /results.xml"...'

They have an 'avacado server' concept, where each host can execute = jobs from
another machine (I believe - I haven't tried it out).

So far, it looks like Avacado is used for virtualization testing.=C2=A0 I d= on't see a
lot of different tests that run on this framework.

Here are some of the major difference, that I can see (this is with limited=
study of Avacado, so I apologize for any omissions of features on their par= t):
1) Fuego is both a test framework, and a distribution of tests themselves.<= br> =C2=A0 * We have about 60 tests now, with about 90 in our 'next' br= anch
=C2=A0 =C2=A0 =C2=A0* the goal is to have hundreds of tests available, for = a wide variety of system features and issues
2) Fuego provides a consistent, containerized back end (docker) so that the=
=C2=A0builds of the test software can be uniform between different sites. =C2=A0 =C2=A0* this will help when comparing results from different sites =C2=A0 =C2=A0 =C2=A0 =C2=A0* multi-site results comparisons are not done ye= t, but on our roadmap
3) Fuego has a multi-node test API (host/target), that is specially geared = for
embedded linux testing
=C2=A0 * the footprint on the target is intentionally very small, using exi= sting POSIX features
=C2=A0 =C2=A0 =C2=A0* IOW - there is no specialized target agent on the tar= get board (except for a transport agent, like sshd or adb)
=C2=A0 * Fuego is geared towards testing of final production images of the = software stacks for products
4) Fuego is behind Avacado in terms of:
=C2=A0 =C2=A0a) unified test ouput (we're working on that now)
=C2=A0 =C2=A0b) test results sharing
=C2=A0 =C2=A0 =C2=A0 =C2=A0* (also in-progress - 'runs' can be pack= aged and shared now to a central server, in our 'next' branch)
=C2=A0 =C2=A0 =C2=A0 =C2=A0* however, there's only rudimentary test dis= play and analysis on the Fuego server at the moment
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* Avacado seems to be using Jenkin= s for this - and we should look at them more closely to see what they'r= e doing

There are more, but hopefully this is helpful to see some of the difference= s.
=C2=A0-- Tim


--f4030435448c68e410054bfebba3--