Native curl and SSL CA certificates

Native curl and SSL CA certificates

[Iván Castell]

[-- Attachment #1: Type: text/plain, Size: 2433 bytes --]

Hello forum.

Using the "rocko" branch. The SSL CA certificates "ca-certificates.crt"
don't appear inside this directory:

build/tmp/work/x86_64-linux/curl-native/7.54.1-r0/recipe-sysroot-native/etc/ssl/certs/

That generates issues when trying to build images that use a https
repository of rpm packages:

This is the repository configuration included:

$ yocto-adv-rpm.repo
[yocto-adv-rpm]
name=Rocko Yocto Repo
baseurl=https://storage.googleapis.com/my_repo/
gpgkey=https://storage.googleapis.com/my_repo/PACKAGEFEED-GPG-KEY-rocko
enabled=1
gpgcheck=1

This repository configuration doesn't work fine. When I try to build
"myimage" recipe, I always get this error:

ERROR: myimage-1.0-r0 do_rootfs: [log_check] myimage: found 1 error message
in the logfile:
[log_check] Failed to synchronize cache for repo 'yocto-adv-rpm', disabling.
ERROR: myimage-1.0-r0 do_rootfs: Function failed: do_rootfs
ERROR: Logfile of failure stored in:
/home/yocto/yocto/build/tmp/work/machine-poky-linux/myimage/1.0-r0/temp/log.do_rootfs.731
ERROR: Task
(/home/yocto/yocto/sources/meta-mylayer/recipes-images/myimage.bb:do_root

I found some relevant information inside this file:

yocto/build/tmp/work/machine-poky-linux/myimage/1.0-r0/temp/dnf.librepo.log

The relevant part:

15:56:41 lr_download: Downloading started
15:56:41 check_transfer_statuses: Transfer finished: repodata/repomd.xml
(Effective url: https://storage.googleapis.com/my_repo/repodata/repomd.xml)
15:56:41 check_finished_transfer_status: Fatal error - Curl code (77):
Problem with the SSL CA cert (path? access rights?) for
https://storage.googleapis.com/my_repo/repodata/repomd.xml [error setting
certificate verify locations:
  CAfile:
/home/yocto/yocto/build/tmp/work/x86_64-linux/curl-native/7.54.1-r0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt
  CApath: none]
15:56:41 lr_yum_download_repomd: repomd.xml download was unsuccessful

This can be fixed copying "ca-certificates.crt" inside the provided
directory (manually):

$ cp /etc/ssl/certs/ca-certificates.crt
/home/yocto/yocto/build/tmp/work/x86_64-linux/curl-native/7.54.1-r0/recipe-sysroot-native/etc/ssl/certs/

I assume "ca-certificates.crt" file should be already included in that
path.

Is this a bug related with curl or ca-certificates recipe? What should be
the right way to fix it?

Thank you in advance.

Kind regards,
  -- Ivan

[-- Attachment #2: Type: text/html, Size: 3424 bytes --]

Re: Native curl and SSL CA certificates

[Alexander Kanavin]

On 05/08/2018 05:55 PM, Iván Castell wrote:
> Is this a bug related with curl or ca-certificates recipe? What should 
> be the right way to fix it?

Fetching and installing packages over the network during image creation 
is not supported or tested in YP. You need to build them locally, with 
recipes.


Alex

Re: Native curl and SSL CA certificates

[Iván Castell]

[-- Attachment #1: Type: text/plain, Size: 1221 bytes --]

But I am not fetching nor installing packages over the network during image
creation. I just build an image using local recipes (standard procedure).
One of those local recipes sets up a remote repository for rpm packages
(adding /etc/yum.repos.d/yocto-adv-rpm.repo to the final image). The
purpose of that remote repository is using it to update rpm packages on
target devices when they are running in production.

In fact, I don't understand why yocto needs to synchronize that cache for
'yocto-adv-rpm' repo during build time. It doesn't have any sense for me.
But the fact is that when the ca-certificates.crt is properly installed,
the build process ends fine. If that file is not properly installed, the
build process fails with the error reported in my previous message.


2018-05-08 21:15 GMT+02:00 Alexander Kanavin <
alexander.kanavin@linux.intel.com>:

> On 05/08/2018 05:55 PM, Iván Castell wrote:
>
>> Is this a bug related with curl or ca-certificates recipe? What should be
>> the right way to fix it?
>>
>
> Fetching and installing packages over the network during image creation is
> not supported or tested in YP. You need to build them locally, with recipes.
>
>
> Alex
>

[-- Attachment #2: Type: text/html, Size: 1757 bytes --]

Re: Native curl and SSL CA certificates

[Iván Castell]

[-- Attachment #1: Type: text/plain, Size: 3675 bytes --]

Just to provide all the details, this is the bbappend I add to my custom
layer:

$ cat recipes-devtools/dnf/dnf_%.bbappend

FILESEXTRAPATHS_prepend := "${THISDIR}/files:"

SRC_URI += " \
    file://yocto-adv-rpm.repo \
"

do_install_append () {
    install -d ${D}/etc/yum.repos.d
    install -m 0600 ${WORKDIR}/yocto-adv-rpm.repo
${D}/etc/yum.repos.d/yocto-adv-rpm.repo
}

FILES_${PN} += "/etc/yum.repos.d"

The contets of yocto-adv-rpm.repo are in the previous message.


2018-05-09 8:29 GMT+02:00 Iván Castell <icastell@nayarsystems.com>:

> But I am not fetching nor installing packages over the network during
> image creation. I just build an image using local recipes (standard
> procedure). One of those local recipes sets up a remote repository for rpm
> packages (adding /etc/yum.repos.d/yocto-adv-rpm.repo to the final image).
> The purpose of that remote repository is using it to update rpm packages on
> target devices when they are running in production.
>
> In fact, I don't understand why yocto needs to synchronize that cache for
> 'yocto-adv-rpm' repo during build time. It doesn't have any sense for me.
> But the fact is that when the ca-certificates.crt is properly installed,
> the build process ends fine. If that file is not properly installed, the
> build process fails with the error reported in my previous message.
>
>
> 2018-05-08 21:15 GMT+02:00 Alexander Kanavin <alexander.kanavin@linux.
> intel.com>:
>
>> On 05/08/2018 05:55 PM, Iván Castell wrote:
>>
>>> Is this a bug related with curl or ca-certificates recipe? What should
>>> be the right way to fix it?
>>>
>>
>> Fetching and installing packages over the network during image creation
>> is not supported or tested in YP. You need to build them locally, with
>> recipes.
>>
>>
>> Alex
>>
>
>
>
>


-- 




*NOTA LEGAL*
Este correo electrónico y, en su caso, cualquier fichero anexo al mismo,
contiene información de carácter confidencial exclusivamente dirigida a su
destinatario y se encuentra protegido por Ley. Cualquier persona distinta
de su destinataria tiene prohibida su reproducción, uso, divulgación, copia
o impresión total o parcial. Si ha recibido este correo electrónico por
error, se ruega lo notifique de inmediato al remitente borrando el mensaje
original juntamente con sus ficheros anexos. Gracias.

De conformidad con lo establecido en la LOPD, NAYAR SYSTEMS SL garantiza la
adopción de las medidas necesarias para asegurar el tratamiento
confidencial de los datos de carácter personal. Así mismo le informamos de
la inclusión de sus datos en un fichero bajo la responsabilidad de NAYAR
SYSTEMS SL, con la finalidad de poder atender los compromisos derivados de
la relación que mantenemos con usted. Si lo desea, puede ejercer sus
derechos de acceso, rectificación, cancelación y oposición mediante un
escrito a la siguiente dirección: info@nayarsystems.com

*LEGAL NOTE*
This email and any attachments to it contains is confidential information
exclusively intended for the recipients. Any divulgation, copy or
distribution to third parties is prohibited without written permission of
NAYAR SYSTEMS SL. If you have received this e-mail in error, please notify
the sender immediately. In accordance with Law 15/1999 of 13 December on
the Protection of Personal Data, the NAYAR SYSTEMS SL guarantees that it
has adopted the necessary measures to ensure the confidential treatment of
personal information. We also inform you that you can exercise your access,
rectification, cancellation and opposition rights by send us a mail to:
info@nayarsystems.com

[-- Attachment #2: Type: text/html, Size: 5404 bytes --]

Re: Native curl and SSL CA certificates

[Alexander Kanavin]

On 05/09/2018 09:29 AM, Iván Castell wrote:
> But I am not fetching nor installing packages over the network during 
> image creation. I just build an image using local recipes (standard 
> procedure). One of those local recipes sets up a remote repository for 
> rpm packages (adding /etc/yum.repos.d/yocto-adv-rpm.repo to the final 
> image). The purpose of that remote repository is using it to update rpm 
> packages on target devices when they are running in production.
> 
> In fact, I don't understand why yocto needs to synchronize that cache 
> for 'yocto-adv-rpm' repo during build time. It doesn't have any sense 
> for me. But the fact is that when the ca-certificates.crt is properly 
> installed, the build process ends fine. If that file is not properly 
> installed, the build process fails with the error reported in my 
> previous message.

During image creation dnf is run several times, and it picks up its own 
configuration from the target rootfs. It is definitely not recommended 
to change that configuration behind dnf's back via installed recipes.

The supported way to configure remote repositories is via PACKAGE_FEED_URIS:
https://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html#using-runtime-package-management

Alex

Re: Native curl and SSL CA certificates

[Iván Castell]

[-- Attachment #1: Type: text/plain, Size: 1953 bytes --]

Thank you very much for your explanation Mr. Alexander, it was really
helpfull to understand my issue.

I fixed it removing completely my dnf bbappend recipe from my custom layer
and adding this variable to my distro.conf file:

    PACKAGE_FEED_URIS = "https://storage.googleapis.com/my_repo/"

After that, at the end of the build process the image contains a valid
/etc/yum.d/oe-remote-repo file and all the necesary stuff to manage it.
There is no need to copy "ca-certificates.crt" manually at all.

Now its working as expected! :-)


2018-05-09 8:56 GMT+02:00 Alexander Kanavin <
alexander.kanavin@linux.intel.com>:

> On 05/09/2018 09:29 AM, Iván Castell wrote:
>
>> But I am not fetching nor installing packages over the network during
>> image creation. I just build an image using local recipes (standard
>> procedure). One of those local recipes sets up a remote repository for rpm
>> packages (adding /etc/yum.repos.d/yocto-adv-rpm.repo to the final
>> image). The purpose of that remote repository is using it to update rpm
>> packages on target devices when they are running in production.
>>
>> In fact, I don't understand why yocto needs to synchronize that cache for
>> 'yocto-adv-rpm' repo during build time. It doesn't have any sense for me.
>> But the fact is that when the ca-certificates.crt is properly installed,
>> the build process ends fine. If that file is not properly installed, the
>> build process fails with the error reported in my previous message.
>>
>
> During image creation dnf is run several times, and it picks up its own
> configuration from the target rootfs. It is definitely not recommended to
> change that configuration behind dnf's back via installed recipes.
>
> The supported way to configure remote repositories is via
> PACKAGE_FEED_URIS:
> https://www.yoctoproject.org/docs/latest/dev-manual/dev-manu
> al.html#using-runtime-package-management
>
> Alex
>

[-- Attachment #2: Type: text/html, Size: 4784 bytes --]

Re: Native curl and SSL CA certificates

[Iván Castell]

[-- Attachment #1: Type: text/plain, Size: 5042 bytes --]

Once the image was build and cloned into the target device, this error
appears when trying to update dnf database from target device:

    $ dnf clean all
    $ dnf update
    DNF version: 2.7.5
    cachedir: /var/cache/dnf
    repo oe-remote-repo: 0x6235C65BD7C1F0D8 already imported
    Cannot download 'https://storage.googleapis.com/my_repo': GPG
verification is enabled, but GPG signature repomd.xml.asc is not available:
Status code: 403 for https://storage.googleapis.com/my_repo/repodata/
    Failed to synchronize cache for repo 'oe-remote-repo', disabling.

The "repodata" directory does not appear inside the package feed directory.
I managed to generate that "repodata" directory executing this commands in
my host machine:

    $ bitbake createrepo-c-native -caddto_recipe_sysroot
    $ oe-run-native createrepo-c-native createrepo_c
/home/yocto/yocto/build/tmp/deploy/rpm

After that, "repodata" directory appears in the package feed and
"repomd.xml" file is inside available, but "repomd.xml.asc" is still not
found.

Do I am missing something?

Thank you in advance!


2018-05-09 13:44 GMT+02:00 Iván Castell <icastell@nayarsystems.com>:

> Thank you very much for your explanation Mr. Alexander, it was really
> helpfull to understand my issue.
>
> I fixed it removing completely my dnf bbappend recipe from my custom layer
> and adding this variable to my distro.conf file:
>
>     PACKAGE_FEED_URIS = "https://storage.googleapis.com/my_repo/"
>
> After that, at the end of the build process the image contains a valid
> /etc/yum.d/oe-remote-repo file and all the necesary stuff to manage it.
> There is no need to copy "ca-certificates.crt" manually at all.
>
> Now its working as expected! :-)
>
>
> 2018-05-09 8:56 GMT+02:00 Alexander Kanavin <alexander.kanavin@linux.
> intel.com>:
>
>> On 05/09/2018 09:29 AM, Iván Castell wrote:
>>
>>> But I am not fetching nor installing packages over the network during
>>> image creation. I just build an image using local recipes (standard
>>> procedure). One of those local recipes sets up a remote repository for rpm
>>> packages (adding /etc/yum.repos.d/yocto-adv-rpm.repo to the final
>>> image). The purpose of that remote repository is using it to update rpm
>>> packages on target devices when they are running in production.
>>>
>>> In fact, I don't understand why yocto needs to synchronize that cache
>>> for 'yocto-adv-rpm' repo during build time. It doesn't have any sense for
>>> me. But the fact is that when the ca-certificates.crt is properly
>>> installed, the build process ends fine. If that file is not properly
>>> installed, the build process fails with the error reported in my previous
>>> message.
>>>
>>
>> During image creation dnf is run several times, and it picks up its own
>> configuration from the target rootfs. It is definitely not recommended to
>> change that configuration behind dnf's back via installed recipes.
>>
>> The supported way to configure remote repositories is via
>> PACKAGE_FEED_URIS:
>> https://www.yoctoproject.org/docs/latest/dev-manual/dev-manu
>> al.html#using-runtime-package-management
>>
>> Alex
>>
>
>
>
>


-- 




*NOTA LEGAL*
Este correo electrónico y, en su caso, cualquier fichero anexo al mismo,
contiene información de carácter confidencial exclusivamente dirigida a su
destinatario y se encuentra protegido por Ley. Cualquier persona distinta
de su destinataria tiene prohibida su reproducción, uso, divulgación, copia
o impresión total o parcial. Si ha recibido este correo electrónico por
error, se ruega lo notifique de inmediato al remitente borrando el mensaje
original juntamente con sus ficheros anexos. Gracias.

De conformidad con lo establecido en la LOPD, NAYAR SYSTEMS SL garantiza la
adopción de las medidas necesarias para asegurar el tratamiento
confidencial de los datos de carácter personal. Así mismo le informamos de
la inclusión de sus datos en un fichero bajo la responsabilidad de NAYAR
SYSTEMS SL, con la finalidad de poder atender los compromisos derivados de
la relación que mantenemos con usted. Si lo desea, puede ejercer sus
derechos de acceso, rectificación, cancelación y oposición mediante un
escrito a la siguiente dirección: info@nayarsystems.com

*LEGAL NOTE*
This email and any attachments to it contains is confidential information
exclusively intended for the recipients. Any divulgation, copy or
distribution to third parties is prohibited without written permission of
NAYAR SYSTEMS SL. If you have received this e-mail in error, please notify
the sender immediately. In accordance with Law 15/1999 of 13 December on
the Protection of Personal Data, the NAYAR SYSTEMS SL guarantees that it
has adopted the necessary measures to ensure the confidential treatment of
personal information. We also inform you that you can exercise your access,
rectification, cancellation and opposition rights by send us a mail to:
info@nayarsystems.com

[-- Attachment #2: Type: text/html, Size: 9211 bytes --]

Re: Native curl and SSL CA certificates

[Iván Castell]

[-- Attachment #1: Type: text/plain, Size: 7254 bytes --]

Ok that's already fixed. I could manage it executing this in host machine
after building the image:

    $ bitbake package-index


2018-05-10 16:01 GMT+02:00 Iván Castell <icastell@nayarsystems.com>:

> Once the image was build and cloned into the target device, this error
> appears when trying to update dnf database from target device:
>
>     $ dnf clean all
>     $ dnf update
>     DNF version: 2.7.5
>     cachedir: /var/cache/dnf
>     repo oe-remote-repo: 0x6235C65BD7C1F0D8 already imported
>     Cannot download 'https://storage.googleapis.com/my_repo': GPG
> verification is enabled, but GPG signature repomd.xml.asc is not available:
> Status code: 403 for https://storage.googleapis.com/my_repo/repodata/
>     Failed to synchronize cache for repo 'oe-remote-repo', disabling.
>
> The "repodata" directory does not appear inside the package feed
> directory. I managed to generate that "repodata" directory executing this
> commands in my host machine:
>
>     $ bitbake createrepo-c-native -caddto_recipe_sysroot
>     $ oe-run-native createrepo-c-native createrepo_c
> /home/yocto/yocto/build/tmp/deploy/rpm
>
> After that, "repodata" directory appears in the package feed and
> "repomd.xml" file is inside available, but "repomd.xml.asc" is still not
> found.
>
> Do I am missing something?
>
> Thank you in advance!
>
>
> 2018-05-09 13:44 GMT+02:00 Iván Castell <icastell@nayarsystems.com>:
>
>> Thank you very much for your explanation Mr. Alexander, it was really
>> helpfull to understand my issue.
>>
>> I fixed it removing completely my dnf bbappend recipe from my custom
>> layer and adding this variable to my distro.conf file:
>>
>>     PACKAGE_FEED_URIS = "https://storage.googleapis.com/my_repo/"
>>
>> After that, at the end of the build process the image contains a valid
>> /etc/yum.d/oe-remote-repo file and all the necesary stuff to manage it.
>> There is no need to copy "ca-certificates.crt" manually at all.
>>
>> Now its working as expected! :-)
>>
>>
>> 2018-05-09 8:56 GMT+02:00 Alexander Kanavin <
>> alexander.kanavin@linux.intel.com>:
>>
>>> On 05/09/2018 09:29 AM, Iván Castell wrote:
>>>
>>>> But I am not fetching nor installing packages over the network during
>>>> image creation. I just build an image using local recipes (standard
>>>> procedure). One of those local recipes sets up a remote repository for rpm
>>>> packages (adding /etc/yum.repos.d/yocto-adv-rpm.repo to the final
>>>> image). The purpose of that remote repository is using it to update rpm
>>>> packages on target devices when they are running in production.
>>>>
>>>> In fact, I don't understand why yocto needs to synchronize that cache
>>>> for 'yocto-adv-rpm' repo during build time. It doesn't have any sense for
>>>> me. But the fact is that when the ca-certificates.crt is properly
>>>> installed, the build process ends fine. If that file is not properly
>>>> installed, the build process fails with the error reported in my previous
>>>> message.
>>>>
>>>
>>> During image creation dnf is run several times, and it picks up its own
>>> configuration from the target rootfs. It is definitely not recommended to
>>> change that configuration behind dnf's back via installed recipes.
>>>
>>> The supported way to configure remote repositories is via
>>> PACKAGE_FEED_URIS:
>>> https://www.yoctoproject.org/docs/latest/dev-manual/dev-manu
>>> al.html#using-runtime-package-management
>>>
>>> Alex
>>>
>>
>>
>>
>>
>
>
> --
>
>
>
>
> *NOTA LEGAL*
> Este correo electrónico y, en su caso, cualquier fichero anexo al mismo,
> contiene información de carácter confidencial exclusivamente dirigida a su
> destinatario y se encuentra protegido por Ley. Cualquier persona distinta
> de su destinataria tiene prohibida su reproducción, uso, divulgación, copia
> o impresión total o parcial. Si ha recibido este correo electrónico por
> error, se ruega lo notifique de inmediato al remitente borrando el mensaje
> original juntamente con sus ficheros anexos. Gracias.
>
> De conformidad con lo establecido en la LOPD, NAYAR SYSTEMS SL garantiza
> la adopción de las medidas necesarias para asegurar el tratamiento
> confidencial de los datos de carácter personal. Así mismo le informamos de
> la inclusión de sus datos en un fichero bajo la responsabilidad de NAYAR
> SYSTEMS SL, con la finalidad de poder atender los compromisos derivados de
> la relación que mantenemos con usted. Si lo desea, puede ejercer sus
> derechos de acceso, rectificación, cancelación y oposición mediante un
> escrito a la siguiente dirección: info@nayarsystems.com
>
> *LEGAL NOTE*
> This email and any attachments to it contains is confidential information
> exclusively intended for the recipients. Any divulgation, copy or
> distribution to third parties is prohibited without written permission of
> NAYAR SYSTEMS SL. If you have received this e-mail in error, please notify
> the sender immediately. In accordance with Law 15/1999 of 13 December on
> the Protection of Personal Data, the NAYAR SYSTEMS SL guarantees that it
> has adopted the necessary measures to ensure the confidential treatment of
> personal information. We also inform you that you can exercise your access,
> rectification, cancellation and opposition rights by send us a mail to:
> info@nayarsystems.com
>
>


-- 




*NOTA LEGAL*
Este correo electrónico y, en su caso, cualquier fichero anexo al mismo,
contiene información de carácter confidencial exclusivamente dirigida a su
destinatario y se encuentra protegido por Ley. Cualquier persona distinta
de su destinataria tiene prohibida su reproducción, uso, divulgación, copia
o impresión total o parcial. Si ha recibido este correo electrónico por
error, se ruega lo notifique de inmediato al remitente borrando el mensaje
original juntamente con sus ficheros anexos. Gracias.

De conformidad con lo establecido en la LOPD, NAYAR SYSTEMS SL garantiza la
adopción de las medidas necesarias para asegurar el tratamiento
confidencial de los datos de carácter personal. Así mismo le informamos de
la inclusión de sus datos en un fichero bajo la responsabilidad de NAYAR
SYSTEMS SL, con la finalidad de poder atender los compromisos derivados de
la relación que mantenemos con usted. Si lo desea, puede ejercer sus
derechos de acceso, rectificación, cancelación y oposición mediante un
escrito a la siguiente dirección: info@nayarsystems.com

*LEGAL NOTE*
This email and any attachments to it contains is confidential information
exclusively intended for the recipients. Any divulgation, copy or
distribution to third parties is prohibited without written permission of
NAYAR SYSTEMS SL. If you have received this e-mail in error, please notify
the sender immediately. In accordance with Law 15/1999 of 13 December on
the Protection of Personal Data, the NAYAR SYSTEMS SL guarantees that it
has adopted the necessary measures to ensure the confidential treatment of
personal information. We also inform you that you can exercise your access,
rectification, cancellation and opposition rights by send us a mail to:
info@nayarsystems.com

[-- Attachment #2: Type: text/html, Size: 12431 bytes --]