CentOS 8 cloud images not working under VM tests

CentOS 8 cloud images not working under VM tests

[John Snow]

This test doesn't appear to work for me:

def build_image(self, img):
    cimg = self._download_with_cache("https://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.3.2011-20201204.2.x86_64.qcow2")
    img_tmp = img + ".tmp"
    subprocess.check_call(["ln", "-f", cimg, img_tmp])
    self.exec_qemu_img("resize", img_tmp, "50G")
    self.boot(img_tmp, extra_args = ["-cdrom", self.gen_cloud_init_iso()])
    self.wait_ssh()
    ^^^^^^^^^^^^^^^

It appears to be expecting to be able to use passwordless entry, but
that doesn't appear to actually work in this case.

It looks like the cloud iso generate step is supposed to handle
setting up keys -- and everything appears as if it's working -- but I
get SSH timeouts at this step.

From what I can see:

DEBUG:root:ssh_cmd: ssh -t -o StrictHostKeyChecking=no -o
UserKnownHostsFile=/dev/null -o ConnectTimeout=1 -p 41729 -i
/home/jsnow/src/qemu/bin/git/vm-test-35u779h4.tmp/id_rsa -o
SendEnv=https_proxy -o SendEnv=http_proxy -o SendEnv=ftp_proxy -o
SendEnv=no_proxy qemu@127.0.0.1 exit 0
Warning: Permanently added '[127.0.0.1]:41729' (ED25519) to the list
of known hosts.
qemu@127.0.0.1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

...and the /home/jsnow/src/qemu/bin/git/vm-test-35u779h4.tmp/id_rsa
file looks identical to the qemu.git/tests/keys/id_rsa file, save for
a missing trailing newline.

From a subsequent run, turning SSH debug on, I see this:

debug1: Offering public key:
/home/jsnow/src/qemu/bin/git/vm-test-o_x2vdwo.tmp/id_rsa RSA
SHA256:6TUK9PSgWR+CbTEKA6E9IyizVjt2ZW5ble/Mg4wUiao explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51

... Which looks like the usual kind of bog-standard "Unrecognized key"
kind of answer, IIUC.

Is this working for anyone else, or can anyone offer some debugging
tips on what's gone wrong here? I'm assuming it worked as of
fb1fa97c69ca2299158229c78fea68c4036e6c1b ... Daniel, any clues?

--js

Re: CentOS 8 cloud images not working under VM tests

[Thomas Huth]

On 11/06/2022 01.28, John Snow wrote:
> This test doesn't appear to work for me:
> 
> def build_image(self, img):
>      cimg = self._download_with_cache("https://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.3.2011-20201204.2.x86_64.qcow2")
>      img_tmp = img + ".tmp"
>      subprocess.check_call(["ln", "-f", cimg, img_tmp])
>      self.exec_qemu_img("resize", img_tmp, "50G")
>      self.boot(img_tmp, extra_args = ["-cdrom", self.gen_cloud_init_iso()])
>      self.wait_ssh()
>      ^^^^^^^^^^^^^^^
> 
> It appears to be expecting to be able to use passwordless entry, but
> that doesn't appear to actually work in this case.
> 
> It looks like the cloud iso generate step is supposed to handle
> setting up keys -- and everything appears as if it's working -- but I
> get SSH timeouts at this step.
> 
>  From what I can see:
> 
> DEBUG:root:ssh_cmd: ssh -t -o StrictHostKeyChecking=no -o
> UserKnownHostsFile=/dev/null -o ConnectTimeout=1 -p 41729 -i
> /home/jsnow/src/qemu/bin/git/vm-test-35u779h4.tmp/id_rsa -o
> SendEnv=https_proxy -o SendEnv=http_proxy -o SendEnv=ftp_proxy -o
> SendEnv=no_proxy qemu@127.0.0.1 exit 0
> Warning: Permanently added '[127.0.0.1]:41729' (ED25519) to the list
> of known hosts.
> qemu@127.0.0.1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
> 
> ...and the /home/jsnow/src/qemu/bin/git/vm-test-35u779h4.tmp/id_rsa
> file looks identical to the qemu.git/tests/keys/id_rsa file, save for
> a missing trailing newline.
> 
>  From a subsequent run, turning SSH debug on, I see this:
> 
> debug1: Offering public key:
> /home/jsnow/src/qemu/bin/git/vm-test-o_x2vdwo.tmp/id_rsa RSA
> SHA256:6TUK9PSgWR+CbTEKA6E9IyizVjt2ZW5ble/Mg4wUiao explicit
> debug3: send packet: type 50
> debug2: we sent a publickey packet, wait for reply
> debug3: receive packet: type 51
> 
> ... Which looks like the usual kind of bog-standard "Unrecognized key"
> kind of answer, IIUC.
> 
> Is this working for anyone else, or can anyone offer some debugging
> tips on what's gone wrong here?
It seems to work for me - maybe it's some issue with a newer version of ssh 
on your host? (I'm still using RHEL 8 here).

Anyway, the VM dies shortly afterwards since it tries to install some 
additional packages, and non-Stream CentOS 8 has been disabled at the end of 
last year. So this test is certainly broken since half a year already and 
nobody noticed until now. I think you can either remove it, or it should get 
updated to CentosStream instead.

  Thomas

Re: CentOS 8 cloud images not working under VM tests

[John Snow]

On Mon, Jun 13, 2022 at 2:05 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 11/06/2022 01.28, John Snow wrote:
> > This test doesn't appear to work for me:
> >
> > def build_image(self, img):
> >      cimg = self._download_with_cache("https://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.3.2011-20201204.2.x86_64.qcow2")
> >      img_tmp = img + ".tmp"
> >      subprocess.check_call(["ln", "-f", cimg, img_tmp])
> >      self.exec_qemu_img("resize", img_tmp, "50G")
> >      self.boot(img_tmp, extra_args = ["-cdrom", self.gen_cloud_init_iso()])
> >      self.wait_ssh()
> >      ^^^^^^^^^^^^^^^
> >
> > It appears to be expecting to be able to use passwordless entry, but
> > that doesn't appear to actually work in this case.
> >
> > It looks like the cloud iso generate step is supposed to handle
> > setting up keys -- and everything appears as if it's working -- but I
> > get SSH timeouts at this step.
> >
> >  From what I can see:
> >
> > DEBUG:root:ssh_cmd: ssh -t -o StrictHostKeyChecking=no -o
> > UserKnownHostsFile=/dev/null -o ConnectTimeout=1 -p 41729 -i
> > /home/jsnow/src/qemu/bin/git/vm-test-35u779h4.tmp/id_rsa -o
> > SendEnv=https_proxy -o SendEnv=http_proxy -o SendEnv=ftp_proxy -o
> > SendEnv=no_proxy qemu@127.0.0.1 exit 0
> > Warning: Permanently added '[127.0.0.1]:41729' (ED25519) to the list
> > of known hosts.
> > qemu@127.0.0.1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
> >
> > ...and the /home/jsnow/src/qemu/bin/git/vm-test-35u779h4.tmp/id_rsa
> > file looks identical to the qemu.git/tests/keys/id_rsa file, save for
> > a missing trailing newline.
> >
> >  From a subsequent run, turning SSH debug on, I see this:
> >
> > debug1: Offering public key:
> > /home/jsnow/src/qemu/bin/git/vm-test-o_x2vdwo.tmp/id_rsa RSA
> > SHA256:6TUK9PSgWR+CbTEKA6E9IyizVjt2ZW5ble/Mg4wUiao explicit
> > debug3: send packet: type 50
> > debug2: we sent a publickey packet, wait for reply
> > debug3: receive packet: type 51
> >
> > ... Which looks like the usual kind of bog-standard "Unrecognized key"
> > kind of answer, IIUC.
> >
> > Is this working for anyone else, or can anyone offer some debugging
> > tips on what's gone wrong here?
> It seems to work for me - maybe it's some issue with a newer version of ssh
> on your host? (I'm still using RHEL 8 here).
>
> Anyway, the VM dies shortly afterwards since it tries to install some
> additional packages, and non-Stream CentOS 8 has been disabled at the end of
> last year. So this test is certainly broken since half a year already and
> nobody noticed until now. I think you can either remove it, or it should get
> updated to CentosStream instead.
>
>   Thomas
>

Don't really have the interest to upgrade it myself, so if it's been
broken for half a year, out it goes.

--js

Re: CentOS 8 cloud images not working under VM tests

[John Snow]

On Mon, Jun 13, 2022 at 12:36 PM John Snow <jsnow@redhat.com> wrote:
>
> On Mon, Jun 13, 2022 at 2:05 AM Thomas Huth <thuth@redhat.com> wrote:
> >
> > On 11/06/2022 01.28, John Snow wrote:
> > > This test doesn't appear to work for me:
> > >
> > > def build_image(self, img):
> > >      cimg = self._download_with_cache("https://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.3.2011-20201204.2.x86_64.qcow2")
> > >      img_tmp = img + ".tmp"
> > >      subprocess.check_call(["ln", "-f", cimg, img_tmp])
> > >      self.exec_qemu_img("resize", img_tmp, "50G")
> > >      self.boot(img_tmp, extra_args = ["-cdrom", self.gen_cloud_init_iso()])
> > >      self.wait_ssh()
> > >      ^^^^^^^^^^^^^^^
> > >
> > > It appears to be expecting to be able to use passwordless entry, but
> > > that doesn't appear to actually work in this case.
> > >
> > > It looks like the cloud iso generate step is supposed to handle
> > > setting up keys -- and everything appears as if it's working -- but I
> > > get SSH timeouts at this step.
> > >
> > >  From what I can see:
> > >
> > > DEBUG:root:ssh_cmd: ssh -t -o StrictHostKeyChecking=no -o
> > > UserKnownHostsFile=/dev/null -o ConnectTimeout=1 -p 41729 -i
> > > /home/jsnow/src/qemu/bin/git/vm-test-35u779h4.tmp/id_rsa -o
> > > SendEnv=https_proxy -o SendEnv=http_proxy -o SendEnv=ftp_proxy -o
> > > SendEnv=no_proxy qemu@127.0.0.1 exit 0
> > > Warning: Permanently added '[127.0.0.1]:41729' (ED25519) to the list
> > > of known hosts.
> > > qemu@127.0.0.1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
> > >
> > > ...and the /home/jsnow/src/qemu/bin/git/vm-test-35u779h4.tmp/id_rsa
> > > file looks identical to the qemu.git/tests/keys/id_rsa file, save for
> > > a missing trailing newline.
> > >
> > >  From a subsequent run, turning SSH debug on, I see this:
> > >
> > > debug1: Offering public key:
> > > /home/jsnow/src/qemu/bin/git/vm-test-o_x2vdwo.tmp/id_rsa RSA
> > > SHA256:6TUK9PSgWR+CbTEKA6E9IyizVjt2ZW5ble/Mg4wUiao explicit
> > > debug3: send packet: type 50
> > > debug2: we sent a publickey packet, wait for reply
> > > debug3: receive packet: type 51
> > >
> > > ... Which looks like the usual kind of bog-standard "Unrecognized key"
> > > kind of answer, IIUC.
> > >
> > > Is this working for anyone else, or can anyone offer some debugging
> > > tips on what's gone wrong here?
> > It seems to work for me - maybe it's some issue with a newer version of ssh
> > on your host? (I'm still using RHEL 8 here).
> >
> > Anyway, the VM dies shortly afterwards since it tries to install some
> > additional packages, and non-Stream CentOS 8 has been disabled at the end of
> > last year. So this test is certainly broken since half a year already and
> > nobody noticed until now. I think you can either remove it, or it should get
> > updated to CentosStream instead.
> >
> >   Thomas
> >
>
> Don't really have the interest to upgrade it myself, so if it's been
> broken for half a year, out it goes.
>

It turns out it's because the setup was failing, but we use "ln" to
make a temporary image instead of cp -- so if the setup screws up,
we've permanently damaged the cached image as well. You have to delete
the original downloaded image to restore original behavior. Then, as
you say, the old image doesn't work anymore anyway. Replacing it with
a new CentOS 8 stream image works fine, if we replace the LN with CP
like the fedora recipe does. However, some of the iotests fail due to
permissions issues with docker and FUSE.

Still trying to figure out the FUSE stuff.

I tried to update aarch64, but that test seems to need even more work.
I think I will probably just delete it, it doesn't seem like anyone is
running it or looking after it.

--js