[Test Automation] [Update Sevice] Failed: Test Firmware Tftp Upload Image.Install Same Image Two Times

[Test Automation] [Update Sevice] Failed: Test Firmware Tftp Upload Image.Install Same Image Two Times

[CS20 CHLi30]

[-- Attachment #1: Type: text/plain, Size: 2211 bytes --]

Hi All,

Recently, we met 1 test item fail "Test Firmware Tftp Upload Image.Install Same Image Two Times" in Update Service of OpenBMC Test Automation.
After checking the log, we found that due to phosphor-software-manager merged this commit
https://github.com/openbmc/phosphor-bmc-code-mgmt/commit/13e0991160ac73915b7dc1bd031fbf7424093cb0

Quote:
image_manager: Delete image dir if same version exists
If the uploaded version already exists, no version D-Bus object is created, but the directory in /tmp/ with the image files remain even though there is no use for it,
also there is no way for the user to delete it except rebooting the BMC. This takes up space and may cause subsequent uploads to fail with out-of-space.
Delete it if the version already exists.

Currently, we just skip this test item due to that against the latest phosphor-software-manager design.
Did you have any idea about this issue symptom? Is it need to remove this test item?

Hi Adriana,

According your commit, I understand this is a good fix to clear no use image files in /tmp folder.
But, it seems against test item of update service in openbmc test automation.
The test script will check if the existing firmware is functional according image_id in /tmp folder, if existing then it's not allow to update same image two times.
However, after your commit then test script will fail due to image_id dir be deleted. Did you have any suggestion for this symptom?

Sincerely,
Tim
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Nuvoton is strictly prohibited; and any information in this email irrelevant to the official business of Nuvoton shall be deemed as neither given nor endorsed by Nuvoton.

[-- Attachment #2: Type: text/html, Size: 5748 bytes --]

Re: [Test Automation] [Update Sevice] Failed: Test Firmware Tftp Upload Image.Install Same Image Two Times

[Adriana Kobylak]

On 2020-11-18 19:51, CS20 CHLi30 wrote:
> 
> Recently, we met 1 test item fail "Test Firmware Tftp Upload
> Image.Install Same Image Two Times" in Update Service of OpenBMC Test
> Automation.

> According your commit, I understand this is a good fix to clear no use
> image files in /tmp folder.
> 
> But, it seems against test item of update service in openbmc test
> automation.
> 
> The test script will check if the existing firmware is functional
> according image_id in /tmp folder, if existing then it's not allow to
> update same image two times.
> 
> However, after your commit then test script will fail due to image_id
> dir be deleted. Did you have any suggestion for this symptom?

Hi Tim,

Could you provide a bit more details about this test case, you hint that 
it may exist in the openbmc-test-automation repo?

The openbmc-test-automation handles this scenario by extracting the 
MANIFEST file out of the code update tarball before the upload and 
reading the VERSION string, then comparing that with the VERSION in the 
/etc/os-release file in the BMC to determine if the version is already 
running in the BMC.

I wouldn't recommend trying to generate the id from within the test case 
because the algorithm may change. So hopefully checking the version 
string is an alternative that'd work.

RE: [Test Automation] [Update Sevice] Failed: Test Firmware Tftp Upload Image.Install Same Image Two Times

[CS20 CHLi30]

Hi Adriana,

Thanks your prompt response. You can check more detail from the robot script:
https://github.com/openbmc/openbmc-test-automation/blob/master/redfish/update_service/test_firmware_tftp_upload_image.robot

Test failed at Get Latest Image ID return false, due to /tmp/images/${image_id} didn't exist.
Below are failed log and test script for you refer it. If there is any question, just let me know. Thank you.

[Failed log]
Keyword 'Get Latest Image ID' failed after retrying...
The last error was: **ERROR** Invalid variable value: image_id: <blank> <str>

[Test script]
Same Firmware Install Two Times
    [Documentation]  Download same image twice to BMC via ImageURI. Second attempt would fail.
    [Arguments]  ${apply_time}  ${tftp_server}=${TFTP_SERVER}  ${image_file_name}=${IMAGE_FILE_NAME}

    # Description of argument(s):
    # apply_time       ApplyTime allowed values (e.g. "OnReset", "Immediate").
    # tftp_server      Server IP.
    # image_file_name  Image file name.

    ImageURI Download Install  ${apply_time}

    # Download image from TFTP server via ImageURI to BMC.
    Redfish.Post  /redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdate
    ...  body={"ImageURI": "tftp://${tftp_server}/${image_file_name}"}

    ${image_version}=  Get Image Version From TFTP Server  ${tftp_server}  ${image_file_name}
    ${software_inventory_record}=  Get Software Inventory State By Version
    ...  ${image_version}
    Rprint Vars  software_inventory_record

    ${image_id}=  Wait Until Keyword Succeeds  180 sec  10 sec  Get Latest Image ID
    Rprint Vars  image_id

    Check Image Update Progress State  match_state='Enabled'  image_id=${image_id}
    # Check if the existing firmware is functional.
    Pass Execution If  ${software_inventory_record['functional']}
    ...  The existing ${image_version} firmware is already functional.

Get Latest Image ID
    [Documentation]  Return the ID of the most recently extracted image.
    # Note: This keyword will fail if there is no such file.

    # Example: # ls /tmp/images/
    #            1b714fb7
    ${image_id}=  Get Latest File  /tmp/images/
    Valid Value  image_id

    # Though an image sub-directory was found, it really isn't valid unless
    # the MANIFEST file is present.
    BMC Execute Command  ls -l /tmp/images/${image_id}/MANIFEST

    [Return]  ${image_id}

Best regards,
Tim
-----Original Message-----
From: Adriana Kobylak [mailto:anoo@linux.ibm.com]
Sent: Thursday, November 19, 2020 11:40 AM
To: CS20 CHLi30 <CHLI30@nuvoton.com>
Cc: anoo@us.ibm.com; openbmc@lists.ozlabs.org
Subject: Re: [Test Automation] [Update Sevice] Failed: Test Firmware Tftp Upload Image.Install Same Image Two Times

On 2020-11-18 19:51, CS20 CHLi30 wrote:
>
> Recently, we met 1 test item fail "Test Firmware Tftp Upload
> Image.Install Same Image Two Times" in Update Service of OpenBMC Test
> Automation.

> According your commit, I understand this is a good fix to clear no use
> image files in /tmp folder.
>
> But, it seems against test item of update service in openbmc test
> automation.
>
> The test script will check if the existing firmware is functional
> according image_id in /tmp folder, if existing then it's not allow to
> update same image two times.
>
> However, after your commit then test script will fail due to image_id
> dir be deleted. Did you have any suggestion for this symptom?

Hi Tim,

Could you provide a bit more details about this test case, you hint that it may exist in the openbmc-test-automation repo?

The openbmc-test-automation handles this scenario by extracting the MANIFEST file out of the code update tarball before the upload and reading the VERSION string, then comparing that with the VERSION in the /etc/os-release file in the BMC to determine if the version is already running in the BMC.

I wouldn't recommend trying to generate the id from within the test case because the algorithm may change. So hopefully checking the version string is an alternative that'd work.
________________________________
________________________________
 The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Nuvoton is strictly prohibited; and any information in this email irrelevant to the official business of Nuvoton shall be deemed as neither given nor endorsed by Nuvoton.

RE: [Test Automation] [Update Service] Failed: Test Firmware Tftp Upload Image.Install Same Image Two Times

[George Keishing]

[-- Attachment #1.1: Type: text/plain, Size: 5967 bytes --]


Hi Tim,

    ${image_id}=  Wait Until Keyword Succeeds  180 sec  10 sec  Get Latest
Image ID
    Rprint Vars  image_id


           It looks like the image didn't get downloaded / transferred from
the TFTP server to BMC  /tmp/images/  within 3 minutes 10 seconds.  If the
TFTP server is slower or take longer time, it could lead to this failure.

           It worked when I tried, but hey its been sometime since, we will
try again and see if it working or not on the latest upstream master image.

Thanks and Regards,
   George Keishing




From:	CS20 CHLi30 <CHLI30@nuvoton.com>
To:	Adriana Kobylak <anoo@linux.ibm.com>
Cc:	"openbmc@lists.ozlabs.org" <openbmc@lists.ozlabs.org>,
            "anoo@us.ibm.com" <anoo@us.ibm.com>, CS20 CHLi30
            <CHLI30@nuvoton.com>
Date:	19-11-2020 12:38 PM
Subject:	[EXTERNAL] RE: [Test Automation] [Update Sevice] Failed: Test
            Firmware Tftp Upload Image.Install Same Image Two Times
Sent by:	"openbmc" <openbmc-bounces
            +gkeishin=in.ibm.com@lists.ozlabs.org>



Hi Adriana,

Thanks your prompt response. You can check more detail from the robot
script:
https://github.com/openbmc/openbmc-test-automation/blob/master/redfish/update_service/test_firmware_tftp_upload_image.robot


Test failed at Get Latest Image ID return false, due to /tmp/images/$
{image_id} didn't exist.
Below are failed log and test script for you refer it. If there is any
question, just let me know. Thank you.

[Failed log]
Keyword 'Get Latest Image ID' failed after retrying...
The last error was: **ERROR** Invalid variable value: image_id: <blank>
<str>

[Test script]
Same Firmware Install Two Times
    [Documentation]  Download same image twice to BMC via ImageURI. Second
attempt would fail.
    [Arguments]  ${apply_time}  ${tftp_server}=${TFTP_SERVER}  $
{image_file_name}=${IMAGE_FILE_NAME}

    # Description of argument(s):
    # apply_time       ApplyTime allowed values (e.g. "OnReset",
"Immediate").
    # tftp_server      Server IP.
    # image_file_name  Image file name.

    ImageURI Download Install  ${apply_time}

    # Download image from TFTP server via ImageURI to BMC.

Redfish.Post  /redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdate
    ...  body={"ImageURI": "tftp://${tftp_server}/${image_file_name}"}

    ${image_version}=  Get Image Version From TFTP Server  ${tftp_server}
${image_file_name}
    ${software_inventory_record}=  Get Software Inventory State By Version
    ...  ${image_version}
    Rprint Vars  software_inventory_record

    ${image_id}=  Wait Until Keyword Succeeds  180 sec  10 sec  Get Latest
Image ID
    Rprint Vars  image_id

    Check Image Update Progress State  match_state='Enabled'  image_id=$
{image_id}
    # Check if the existing firmware is functional.
    Pass Execution If  ${software_inventory_record['functional']}
    ...  The existing ${image_version} firmware is already functional.

Get Latest Image ID
    [Documentation]  Return the ID of the most recently extracted image.
    # Note: This keyword will fail if there is no such file.

    # Example: # ls /tmp/images/
    #            1b714fb7
    ${image_id}=  Get Latest File  /tmp/images/
    Valid Value  image_id

    # Though an image sub-directory was found, it really isn't valid unless
    # the MANIFEST file is present.
    BMC Execute Command  ls -l /tmp/images/${image_id}/MANIFEST

    [Return]  ${image_id}

Best regards,
Tim
-----Original Message-----
From: Adriana Kobylak [mailto:anoo@linux.ibm.com]
Sent: Thursday, November 19, 2020 11:40 AM
To: CS20 CHLi30 <CHLI30@nuvoton.com>
Cc: anoo@us.ibm.com; openbmc@lists.ozlabs.org
Subject: Re: [Test Automation] [Update Sevice] Failed: Test Firmware Tftp
Upload Image.Install Same Image Two Times

On 2020-11-18 19:51, CS20 CHLi30 wrote:
>
> Recently, we met 1 test item fail "Test Firmware Tftp Upload
> Image.Install Same Image Two Times" in Update Service of OpenBMC Test
> Automation.

> According your commit, I understand this is a good fix to clear no use
> image files in /tmp folder.
>
> But, it seems against test item of update service in openbmc test
> automation.
>
> The test script will check if the existing firmware is functional
> according image_id in /tmp folder, if existing then it's not allow to
> update same image two times.
>
> However, after your commit then test script will fail due to image_id
> dir be deleted. Did you have any suggestion for this symptom?

Hi Tim,

Could you provide a bit more details about this test case, you hint that it
may exist in the openbmc-test-automation repo?

The openbmc-test-automation handles this scenario by extracting the
MANIFEST file out of the code update tarball before the upload and reading
the VERSION string, then comparing that with the VERSION in
the /etc/os-release file in the BMC to determine if the version is already
running in the BMC.

I wouldn't recommend trying to generate the id from within the test case
because the algorithm may change. So hopefully checking the version string
is an alternative that'd work.
________________________________
________________________________
 The privileged confidential information contained in this email is
intended for use only by the addressees as indicated by the original sender
of this email. If you are not the addressee indicated in this email or are
not responsible for delivery of the email to such a person, please kindly
reply to the sender indicating this fact and delete all copies of it from
your computer and network server immediately. Your cooperation is highly
appreciated. It is advised that any unauthorized use of confidential
information of Nuvoton is strictly prohibited; and any information in this
email irrelevant to the official business of Nuvoton shall be deemed as
neither given nor endorsed by Nuvoton.




[-- Attachment #1.2: Type: text/html, Size: 8383 bytes --]

[-- Attachment #2: graycol.gif --]
[-- Type: image/gif, Size: 105 bytes --]