[review-request]Fixed old UI tests

[review-request]Fixed old UI tests

[Mihail, StanciuX]

[-- Attachment #1: Type: text/plain, Size: 273 bytes --]

Hello,

We've fixed the older UI tests to work with the latest changes to IDs.
You can find the code here:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/toaster_tests_fixed

Please let us know if anything needs updating.

Regards,
Mihai


[-- Attachment #2: Type: text/html, Size: 2339 bytes --]

Re: [review-request]Fixed old UI tests

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 2204 bytes --]

I have one major comment on these tests: the use of time.sleep(X) isn't a
great way of waiting for page loads. It would be much better if you used a
wait_until() style API, as provided by Selenium. That way you just wait for
an element to become visible on the page, or present in its source (or
whatever), before continuing with the next step in the test.

If you are interested in how to approach this, I've repurposed some code
for Toaster which was originally written by an Intel colleague (Damien
Lespiau) working on automated tests for patchwork, which you can see here:
https://github.com/dlespiau/patchwork/blob/master/patchwork/tests/browser.py

My variation on this code is in the Toaster code base at:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/tree/bitbake/lib/toaster/tests/browser/selenium_helpers.py

(The main differences are that I removed the screenshot method, only
initialise the Selenium driver once per test class rather than once per
test, and have added a couple more convenience methods like
get_page_source().)

You can see an example of how to use this in a Selenium test at:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/tree/bitbake/lib/toaster/tests/browser/test_all_builds_page.py
(see the test_show_tasks_with_suffix() test method, which uses
wait_until_present())

I would recommend using this approach as it is much less fragile than an
arbitrary sleep. If you need any assistance in figuring out how to convert
to that API, please let me know.

Thanks.
Elliot

On 8 April 2016 at 15:15, Mihail, StanciuX <stanciux.mihail@intel.com>
wrote:

> Hello,
>
>
>
> We’ve fixed the older UI tests to work with the latest changes to IDs.
>
> You can find the code here:
>
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/toaster_tests_fixed
>
>
>
> Please let us know if anything needs updating.
>
>
>
> Regards,
>
> Mihai
>
>
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>
>


-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 3891 bytes --]

Re: [review-request]Fixed old UI tests

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 1202 bytes --]

Does this branch supersede vhangan/automated_toaster_ui_tests
<http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/automated_toaster_ui_tests>
 ?

I tried to cherry pick the commits off vhangan/automated_toaster_ui_tests
<http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/automated_toaster_ui_tests>
then
the commits off vhangan/toaster_tests_fixed
<http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/toaster_tests_fixed>,
but they conflict with each other.

Thanks.
Elliot

On 8 April 2016 at 15:15, Mihail, StanciuX <stanciux.mihail@intel.com>
wrote:

> Hello,
>
>
>
> We’ve fixed the older UI tests to work with the latest changes to IDs.
>
> You can find the code here:
>
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/toaster_tests_fixed
>
>
>
> Please let us know if anything needs updating.
>
>
>
> Regards,
>
> Mihai
>
>
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>
>


-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 2553 bytes --]

Re: [review-request]Fixed old UI tests

[Mihail, StanciuX]

[-- Attachment #1: Type: text/plain, Size: 1700 bytes --]

Hi Elliot,

There shouldn’t have been conflicts between them, aside from removing the deprecated tests.
I’ll investigate and get back to you. Would it be ok if I rebase them on top of each other and give you back only one mammoth-sized commit?

Mihai

From: Smith, Elliot [mailto:elliot.smith@intel.com]
Sent: Monday, April 11, 2016 7:43 PM
To: Mihail, StanciuX
Cc: toaster@yoctoproject.org; Hangan, ValentinX; Roman, Alexandru CostinX
Subject: Re: [Toaster] [review-request]Fixed old UI tests

Does this branch supersede vhangan/automated_toaster_ui_tests<http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/automated_toaster_ui_tests> ?

I tried to cherry pick the commits off vhangan/automated_toaster_ui_tests<http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/automated_toaster_ui_tests> then the commits off vhangan/toaster_tests_fixed<http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/toaster_tests_fixed>, but they conflict with each other.

Thanks.
Elliot

On 8 April 2016 at 15:15, Mihail, StanciuX <stanciux.mihail@intel.com<mailto:stanciux.mihail@intel.com>> wrote:
Hello,

We’ve fixed the older UI tests to work with the latest changes to IDs.
You can find the code here:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/toaster_tests_fixed

Please let us know if anything needs updating.

Regards,
Mihai


--
_______________________________________________
toaster mailing list
toaster@yoctoproject.org<mailto:toaster@yoctoproject.org>
https://lists.yoctoproject.org/listinfo/toaster



--
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 6900 bytes --]

Re: [review-request]Fixed old UI tests

[Mihail, StanciuX]

[-- Attachment #1: Type: text/plain, Size: 1696 bytes --]

Hello,

I’ve resolved the conflicts here:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mstanciu/merged_ui_tests

They should rebase over master with no issues.

Please let us know if more changes are needed.

Regards,
Mihai

From: Smith, Elliot [mailto:elliot.smith@intel.com]
Sent: Monday, April 11, 2016 7:43 PM
To: Mihail, StanciuX
Cc: toaster@yoctoproject.org; Hangan, ValentinX; Roman, Alexandru CostinX
Subject: Re: [Toaster] [review-request]Fixed old UI tests

Does this branch supersede vhangan/automated_toaster_ui_tests<http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/automated_toaster_ui_tests> ?

I tried to cherry pick the commits off vhangan/automated_toaster_ui_tests<http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/automated_toaster_ui_tests> then the commits off vhangan/toaster_tests_fixed<http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/toaster_tests_fixed>, but they conflict with each other.

Thanks.
Elliot

On 8 April 2016 at 15:15, Mihail, StanciuX <stanciux.mihail@intel.com<mailto:stanciux.mihail@intel.com>> wrote:
Hello,

We’ve fixed the older UI tests to work with the latest changes to IDs.
You can find the code here:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/toaster_tests_fixed

Please let us know if anything needs updating.

Regards,
Mihai


--
_______________________________________________
toaster mailing list
toaster@yoctoproject.org<mailto:toaster@yoctoproject.org>
https://lists.yoctoproject.org/listinfo/toaster



--
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 7727 bytes --]

Re: [review-request]Fixed old UI tests

[Brian Avery]

Hi,

A couple of comments:
1) I strongly agree with Elliot that the time.sleep should go away.
Since we are migrating alot of tests to different machines/places,
assuming that we know the machine characteristics is certainly going
to go poorly. The wait_until API makes more sense.

2) Selenium tests in a QA framework for tests that require actual
builds make sense.  For anything else that can be done with fixture
data, it would be much much better to extend elliot's Django tests in
bitbake/lib/toaster/tests/browser.  These are fast to run since they
don't need actual bitbake interactions.  This means that the devs
can/will run them and will also be able to keep them up to date.  This
way, the tests can perform a CI use for the devs and, since QA can
also wrap these tests, it would let QA use them too. If we use the
tests as is, it seems very very likely that they will become stale
very soon, especially considering the theme changes that are ongoing
at the moment.  I've added the bitbake/lib/toaster/tests/browser tests
to the Contribute to Toaster page so that people remember to run/keep
them up to date!

3) For merging into the bitbake/lib/toaster/tests/browser tests, these
need some work for readability/maintainability.


-bavery
an Intel employee

On Tue, Apr 12, 2016 at 8:15 AM, Mihail, StanciuX
<stanciux.mihail@intel.com> wrote:
> Hello,
>
>
>
> I’ve resolved the conflicts here:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mstanciu/merged_ui_tests
>
>
>
> They should rebase over master with no issues.
>
>
>
> Please let us know if more changes are needed.
>
>
>
> Regards,
>
> Mihai
>
>
>
> From: Smith, Elliot [mailto:elliot.smith@intel.com]
> Sent: Monday, April 11, 2016 7:43 PM
> To: Mihail, StanciuX
> Cc: toaster@yoctoproject.org; Hangan, ValentinX; Roman, Alexandru CostinX
> Subject: Re: [Toaster] [review-request]Fixed old UI tests
>
>
>
> Does this branch supersede vhangan/automated_toaster_ui_tests ?
>
>
>
> I tried to cherry pick the commits off vhangan/automated_toaster_ui_tests
> then the commits off vhangan/toaster_tests_fixed, but they conflict with
> each other.
>
>
>
> Thanks.
>
> Elliot
>
>
>
> On 8 April 2016 at 15:15, Mihail, StanciuX <stanciux.mihail@intel.com>
> wrote:
>
> Hello,
>
>
>
> We’ve fixed the older UI tests to work with the latest changes to IDs.
>
> You can find the code here:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/toaster_tests_fixed
>
>
>
> Please let us know if anything needs updating.
>
>
>
> Regards,
>
> Mihai
>
>
>
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>
>
>
>
>
> --
>
> Elliot Smith
> Software Engineer
> Intel Open Source Technology Centre
>
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>

Re: [review-request]Fixed old UI tests

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 4402 bytes --]

On 12 April 2016 at 20:26, Brian Avery <avery.brian@gmail.com> wrote:

> A couple of comments:
> 1) I strongly agree with Elliot that the time.sleep should go away.
> Since we are migrating alot of tests to different machines/places,
> assuming that we know the machine characteristics is certainly going
> to go poorly. The wait_until API makes more sense.
>

I am available to assist with translating any QA tests to Toaster's
Selenium test suite. I can also write a wiki page or something to provide
guidance in how to do this.

As Brian said, it's much quicker to run UI tests using test data, as you
don't need to run entire builds. The difficulty is in setting up the test
data, but most of the time I can figure out what needs to go where.

The rest of it is fairly standard Selenium, though the helpers already in
the code base can make your code smaller and easier to follow. They also
provide the wait_unti() functionality I mentioned previously (to replace
time.sleep()). Splitting the code over multiple files makes it easier to
find/fix tests, too.

As Brian also said, if the tests are brought into our main browser test
suite, we will also help to maintain them.

Let me know what I can do to help.

Thanks.
Elliot


>
> 2) Selenium tests in a QA framework for tests that require actual
> builds make sense.  For anything else that can be done with fixture
> data, it would be much much better to extend elliot's Django tests in
> bitbake/lib/toaster/tests/browser.  These are fast to run since they
> don't need actual bitbake interactions.  This means that the devs
> can/will run them and will also be able to keep them up to date.  This
> way, the tests can perform a CI use for the devs and, since QA can
> also wrap these tests, it would let QA use them too. If we use the
> tests as is, it seems very very likely that they will become stale
> very soon, especially considering the theme changes that are ongoing
> at the moment.  I've added the bitbake/lib/toaster/tests/browser tests
> to the Contribute to Toaster page so that people remember to run/keep
> them up to date!
>
> 3) For merging into the bitbake/lib/toaster/tests/browser tests, these
> need some work for readability/maintainability.
>
>
> -bavery
> an Intel employee
>
> On Tue, Apr 12, 2016 at 8:15 AM, Mihail, StanciuX
> <stanciux.mihail@intel.com> wrote:
> > Hello,
> >
> >
> >
> > I’ve resolved the conflicts here:
> >
> >
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mstanciu/merged_ui_tests
> >
> >
> >
> > They should rebase over master with no issues.
> >
> >
> >
> > Please let us know if more changes are needed.
> >
> >
> >
> > Regards,
> >
> > Mihai
> >
> >
> >
> > From: Smith, Elliot [mailto:elliot.smith@intel.com]
> > Sent: Monday, April 11, 2016 7:43 PM
> > To: Mihail, StanciuX
> > Cc: toaster@yoctoproject.org; Hangan, ValentinX; Roman, Alexandru
> CostinX
> > Subject: Re: [Toaster] [review-request]Fixed old UI tests
> >
> >
> >
> > Does this branch supersede vhangan/automated_toaster_ui_tests ?
> >
> >
> >
> > I tried to cherry pick the commits off vhangan/automated_toaster_ui_tests
> > then the commits off vhangan/toaster_tests_fixed, but they conflict with
> > each other.
> >
> >
> >
> > Thanks.
> >
> > Elliot
> >
> >
> >
> > On 8 April 2016 at 15:15, Mihail, StanciuX <stanciux.mihail@intel.com>
> > wrote:
> >
> > Hello,
> >
> >
> >
> > We’ve fixed the older UI tests to work with the latest changes to IDs.
> >
> > You can find the code here:
> >
> >
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=vhangan/toaster_tests_fixed
> >
> >
> >
> > Please let us know if anything needs updating.
> >
> >
> >
> > Regards,
> >
> > Mihai
> >
> >
> >
> >
> > --
> > _______________________________________________
> > toaster mailing list
> > toaster@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/toaster
> >
> >
> >
> >
> >
> > --
> >
> > Elliot Smith
> > Software Engineer
> > Intel Open Source Technology Centre
> >
> >
> > --
> > _______________________________________________
> > toaster mailing list
> > toaster@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/toaster
> >
>



-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 6867 bytes --]