Emulate Rpi with QEMU fails

Emulate Rpi with QEMU fails

[Thomas]

Hi,

I'm trying to emulate Rpi with QEMU.
I found
[url=https://www.raspberry-pi-geek.de/ausgaben/rpg/2014/04/raspberry-pi-emulieren/]this[/url]
arcticle in Raspberry Pi Geek documenting the steps including persistent
storage on host.

However when starting the emulation with command
qemu-system-arm -M versatilepb -cpu arm1176 -m 256 -serial stdio -hda
2020-08-20-raspios-buster-armhf-lite.img -net
"user,hostfwd=tcp::5022-:22" -dtb versatile-pb-buster.dtb -kernel
kernel-qemu-5.4.51-buster -append "root=/dev/sda2 rootfstype=ext4 rw
panic=1" -no-reboot
I get this error:
VFS: Cannot open root device "sda2" or unknown-block(0,0): error -6
Please append a correct "root=" boot option; here are the available
partitions:
0100            4096 ram0
 (driver?)
0101            4096 ram1
 (driver?)
0102            4096 ram2
 (driver?)
0103            4096 ram3
 (driver?)
0104            4096 ram4
 (driver?)
0105            4096 ram5
 (driver?)
0106            4096 ram6
 (driver?)
0107            4096 ram7
 (driver?)
0108            4096 ram8
 (driver?)
0109            4096 ram9
 (driver?)
010a            4096 ram10
 (driver?)
010b            4096 ram11
 (driver?)
010c            4096 ram12
 (driver?)
010d            4096 ram13
 (driver?)
010e            4096 ram14
 (driver?)
010f            4096 ram15
 (driver?)
1f00           65536 mtdblock0
 (driver?)
Kernel panic - not syncing: VFS: Unable to mount root fs on
unknown-block(0,0)

I assume this is related to the content in fstab of RaspiOS:
proc            /proc           proc    defaults          0       0
PARTUUID=907af7d0-01  /boot           vfat    defaults          0       2
PARTUUID=907af7d0-02  /               ext4    defaults,noatime  0       1

Can you please advise how to fix this error?

THX

Re: Emulate Rpi with QEMU fails

[Alex Bennée]

Thomas <74cmonty@gmail.com> writes:

> Hi,
>
> I'm trying to emulate Rpi with QEMU.
> I found
> [url=1]this[/url]
> arcticle in Raspberry Pi Geek documenting the steps including persistent
> storage on host.
>
> However when starting the emulation with command
> qemu-system-arm -M versatilepb -cpu arm1176 -m 256 -serial stdio -hda
> 2020-08-20-raspios-buster-armhf-lite.img -net
> "user,hostfwd=tcp::5022-:22" -dtb versatile-pb-buster.dtb -kernel
> kernel-qemu-5.4.51-buster -append "root=/dev/sda2 rootfstype=ext4 rw
> panic=1" -no-reboot

Let's start with the fact you are using a versatilepb machine type with
a versatilepb dtb and not the rasppi model. The manual goes into more
details about why you can't generally share kernels built for one ARM
system on another:

  https://qemu.readthedocs.io/en/latest/system/target-arm.html

Since that article was written we have added models for the -M raspi2
and -M raspi3 which might work better with a kernel written for it
although the emulation isn't complete.

> I get this error:
> VFS: Cannot open root device "sda2" or unknown-block(0,0): error -6
> Please append a correct "root=" boot option; here are the available
> partitions:
> 0100            4096 ram0
>  (driver?)
> 0101            4096 ram1
>  (driver?)
> 0102            4096 ram2
>  (driver?)
> 0103            4096 ram3
>  (driver?)
> 0104            4096 ram4
>  (driver?)
> 0105            4096 ram5
>  (driver?)
> 0106            4096 ram6
>  (driver?)
> 0107            4096 ram7
>  (driver?)
> 0108            4096 ram8
>  (driver?)
> 0109            4096 ram9
>  (driver?)
> 010a            4096 ram10
>  (driver?)
> 010b            4096 ram11
>  (driver?)
> 010c            4096 ram12
>  (driver?)
> 010d            4096 ram13
>  (driver?)
> 010e            4096 ram14
>  (driver?)
> 010f            4096 ram15
>  (driver?)
> 1f00           65536 mtdblock0
>  (driver?)
> Kernel panic - not syncing: VFS: Unable to mount root fs on
> unknown-block(0,0)
>
> I assume this is related to the content in fstab of RaspiOS:
> proc            /proc           proc    defaults          0       0
> PARTUUID=907af7d0-01  /boot           vfat    defaults          0       2
> PARTUUID=907af7d0-02  /               ext4    defaults,noatime  0       1
>
> Can you please advise how to fix this error?
>
> THX


-- 
Alex Bennée

Re: Emulate Rpi with QEMU fails

[Peter Maydell]

On Sun, 4 Oct 2020 at 18:44, Alex Bennée <alex.bennee@linaro.org> wrote:
> Thomas <74cmonty@gmail.com> writes:
> > I'm trying to emulate Rpi with QEMU.
> > I found
> > [url=1]this[/url]
> > arcticle in Raspberry Pi Geek documenting the steps including persistent
> > storage on host.
> >
> > However when starting the emulation with command
> > qemu-system-arm -M versatilepb -cpu arm1176 -m 256 -serial stdio -hda
> > 2020-08-20-raspios-buster-armhf-lite.img -net
> > "user,hostfwd=tcp::5022-:22" -dtb versatile-pb-buster.dtb -kernel
> > kernel-qemu-5.4.51-buster -append "root=/dev/sda2 rootfstype=ext4 rw
> > panic=1" -no-reboot
>
> Let's start with the fact you are using a versatilepb machine type with
> a versatilepb dtb and not the rasppi model.

Given the name of the kernel image, this probably actually *is*
built for versatilepb, or it wouldn't have got as far as failing
to mount the root partition. There seems to be a lot of confusion
in the raspberry pi community about the difference between
running the raspi userspace plus a for-versatilepb kernel
versus running a full raspi setup.

Anyway, failing to mount the rootfs and not listing any
sda devices is not a problem with the fstab, because the system
hasn't got as far as being able to mount the filesystem with a
fstab on it yet. One possibility is that the kernel is
missing the device drivers for either PCI or for the SCSI
controller that gets plugged in to versatilepb by default.

My guess at the cause is that you're trying to boot a Linux 5.something
kernel and you've run into the issue described in this thread:
https://lists.gnu.org/archive/html/qemu-discuss/2020-09/msg00023.html
where the Linux 5.x sym53c8xx scsi driver is not compatible with QEMU's
emulation of that device. If that's the case then you should see
earlier in the kernel boot log error messages similar to the ones
that Roger reported. The fix would be either to use an older
kernel, or to change the QEMU commandline to use a different
SCSI controller (or to use a virtio block device).

thanks
-- PMM

Re: Emulate Rpi with QEMU fails

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Sun, 4 Oct 2020 at 18:44, Alex Bennée <alex.bennee@linaro.org> wrote:
>> Thomas <74cmonty@gmail.com> writes:
>> > I'm trying to emulate Rpi with QEMU.
>> > I found
>> > [url=1]this[/url]
>> > arcticle in Raspberry Pi Geek documenting the steps including persistent
>> > storage on host.
>> >
>> > However when starting the emulation with command
>> > qemu-system-arm -M versatilepb -cpu arm1176 -m 256 -serial stdio -hda
>> > 2020-08-20-raspios-buster-armhf-lite.img -net
>> > "user,hostfwd=tcp::5022-:22" -dtb versatile-pb-buster.dtb -kernel
>> > kernel-qemu-5.4.51-buster -append "root=/dev/sda2 rootfstype=ext4 rw
>> > panic=1" -no-reboot
>>
>> Let's start with the fact you are using a versatilepb machine type with
>> a versatilepb dtb and not the rasppi model.
>
> Given the name of the kernel image, this probably actually *is*
> built for versatilepb, or it wouldn't have got as far as failing
> to mount the root partition. There seems to be a lot of confusion
> in the raspberry pi community about the difference between
> running the raspi userspace plus a for-versatilepb kernel
> versus running a full raspi setup.

Ahh your German is considerably better than mine ;-) Looking more
closely at the blog it seems to be predicated on extracting a Raspbian
kernel which at least stands a fighting chance of being a multi_config
kernel - like the buster above.

I can see why these sorts of shenanigans used to be pulled when there
where no RaspPi models although if all you want to do is run an ARM user
space what's wrong with using linux-user for this sort of thing?

> Anyway, failing to mount the rootfs and not listing any
> sda devices is not a problem with the fstab, because the system
> hasn't got as far as being able to mount the filesystem with a
> fstab on it yet. One possibility is that the kernel is
> missing the device drivers for either PCI or for the SCSI
> controller that gets plugged in to versatilepb by default.
>
> My guess at the cause is that you're trying to boot a Linux 5.something
> kernel and you've run into the issue described in this thread:
> https://lists.gnu.org/archive/html/qemu-discuss/2020-09/msg00023.html
> where the Linux 5.x sym53c8xx scsi driver is not compatible with QEMU's
> emulation of that device. If that's the case then you should see
> earlier in the kernel boot log error messages similar to the ones
> that Roger reported. The fix would be either to use an older
> kernel, or to change the QEMU commandline to use a different
> SCSI controller (or to use a virtio block device).

Do we have any documentation for the RaspPi models? The acceptance tests
look like they support the inbuilt MMC/SD controller device:

        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
                               serial_kernel_cmdline[uart_id] +
                               ' root=/dev/mmcblk0p2 rootwait ' +
                               'dwc_otg.fiq_fsm_enable=0')

It would be useful to fill the hole in the documentation so gently steer
people away from these hybrid franken-machine approaches.

>
> thanks
> -- PMM


-- 
Alex Bennée

Re: Emulate Rpi with QEMU fails

[Thomas Schneider]

Hello,

thanks for your replies.

I must admit that I don't fully understand your analysis.
However you made some conclusions that are correct.

In fact I have found a Github repo 
<https://github.com/dhruvvyas90/qemu-rpi-kernel> where a specific kernel 
and versatile-pb are provided + instructions for lauching the emulation 
with the original RPi image file:
$ qemu-system-arm \
   -M versatilepb \
   -cpu arm1176 \
   -m 256 \
   -drive 
"file=/.../2020-05-27-raspios-buster-lite-armhf.img,if=none,index=0,media=disk,format=raw,id=disk0" 
\
   -device 
"virtio-blk-pci,drive=disk0,disable-modern=on,disable-legacy=off" \
   -net "user,hostfwd=tcp::5022-:22" \
   -dtb /.../versatile-pb-buster-5.4.51.dtb \
   -kernel /.../kernel-qemu-5.4.51-buster \
   -append 'root=/dev/vda2 panic=1' \
   -no-reboot

This means it is more recent than the Raspberry Pi Geek article, and the 
emulation works.
But I'm not sure if this usable considering the added models -M raspi2 
and -M raspi3.

Can you please advise how to proceed?

In addition I would like to know if there's a memory limitation using 
models -M raspi2 and -M raspi3?
To my understanding there's a limitation to 256MB using -M versatilepb.
If yes, I consider to another raw image located on host's temporary 
filesystem and use this a swap in the client.

And how can I make use of a client network device that is based on 
host's tap device connected to a network bridge?

THX


Am 05.10.2020 um 11:40 schrieb Alex Bennée:
> Peter Maydell <peter.maydell@linaro.org> writes:
>
>> On Sun, 4 Oct 2020 at 18:44, Alex Bennée <alex.bennee@linaro.org> wrote:
>>> Thomas <74cmonty@gmail.com> writes:
>>>> I'm trying to emulate Rpi with QEMU.
>>>> I found
>>>> [url=1]this[/url]
>>>> arcticle in Raspberry Pi Geek documenting the steps including persistent
>>>> storage on host.
>>>>
>>>> However when starting the emulation with command
>>>> qemu-system-arm -M versatilepb -cpu arm1176 -m 256 -serial stdio -hda
>>>> 2020-08-20-raspios-buster-armhf-lite.img -net
>>>> "user,hostfwd=tcp::5022-:22" -dtb versatile-pb-buster.dtb -kernel
>>>> kernel-qemu-5.4.51-buster -append "root=/dev/sda2 rootfstype=ext4 rw
>>>> panic=1" -no-reboot
>>> Let's start with the fact you are using a versatilepb machine type with
>>> a versatilepb dtb and not the rasppi model.
>> Given the name of the kernel image, this probably actually *is*
>> built for versatilepb, or it wouldn't have got as far as failing
>> to mount the root partition. There seems to be a lot of confusion
>> in the raspberry pi community about the difference between
>> running the raspi userspace plus a for-versatilepb kernel
>> versus running a full raspi setup.
> Ahh your German is considerably better than mine ;-) Looking more
> closely at the blog it seems to be predicated on extracting a Raspbian
> kernel which at least stands a fighting chance of being a multi_config
> kernel - like the buster above.
>
> I can see why these sorts of shenanigans used to be pulled when there
> where no RaspPi models although if all you want to do is run an ARM user
> space what's wrong with using linux-user for this sort of thing?
>
>> Anyway, failing to mount the rootfs and not listing any
>> sda devices is not a problem with the fstab, because the system
>> hasn't got as far as being able to mount the filesystem with a
>> fstab on it yet. One possibility is that the kernel is
>> missing the device drivers for either PCI or for the SCSI
>> controller that gets plugged in to versatilepb by default.
>>
>> My guess at the cause is that you're trying to boot a Linux 5.something
>> kernel and you've run into the issue described in this thread:
>> https://lists.gnu.org/archive/html/qemu-discuss/2020-09/msg00023.html
>> where the Linux 5.x sym53c8xx scsi driver is not compatible with QEMU's
>> emulation of that device. If that's the case then you should see
>> earlier in the kernel boot log error messages similar to the ones
>> that Roger reported. The fix would be either to use an older
>> kernel, or to change the QEMU commandline to use a different
>> SCSI controller (or to use a virtio block device).
> Do we have any documentation for the RaspPi models? The acceptance tests
> look like they support the inbuilt MMC/SD controller device:
>
>          kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
>                                 serial_kernel_cmdline[uart_id] +
>                                 ' root=/dev/mmcblk0p2 rootwait ' +
>                                 'dwc_otg.fiq_fsm_enable=0')
>
> It would be useful to fill the hole in the documentation so gently steer
> people away from these hybrid franken-machine approaches.
>
>> thanks
>> -- PMM
>

Re: Emulate Rpi with QEMU fails

[Paul Zimmerman]

If you can upgrade to Qemu 5.1 or newer, you should be able to use the
builtin raspi2 or raspi3 emulation directly. Version 5.1 adds support for the
USB controller on the Pi.

There is no limitation on the amount of memory you can use with this
method, and the networking should work.

Here is the command line I use to run the Raspbian image
2019-09-26-raspbian-buster. I extracted bcm2709-rpi-2-b and
kernel7.img from the FAT partition inside the image file.

qemu-system-arm -M raspi2 -drive
file=bootpi/2019-09-26-raspbian-buster.img,format=raw,if=sd -dtb
bcm2709-rpi-2-b.dtb -kernel kernel7.img -append 'rw
earlycon=pl011,0x3f201000 console=ttyAMA0 loglevel=8
root=/dev/mmcblk0p2 fsck.repair=yes net.ifnames=0 rootwait memtest=1
dwc_otg.fiq_fsm_enable=0' -serial stdio -no-reboot -netdev
user,id=net0 -usb -device usb-kbd -device usb-tablet -device
usb-net,netdev=net0

Hope this helps.

On Mon, Oct 5, 2020 at 3:51 AM Thomas Schneider <74cmonty@gmail.com> wrote:
>
> Hello,
>
> thanks for your replies.
>
> I must admit that I don't fully understand your analysis.
> However you made some conclusions that are correct.
>
> In fact I have found a Github repo
> <https://github.com/dhruvvyas90/qemu-rpi-kernel> where a specific kernel
> and versatile-pb are provided + instructions for lauching the emulation
> with the original RPi image file:
> $ qemu-system-arm \
>    -M versatilepb \
>    -cpu arm1176 \
>    -m 256 \
>    -drive
> "file=/.../2020-05-27-raspios-buster-lite-armhf.img,if=none,index=0,media=disk,format=raw,id=disk0"
> \
>    -device
> "virtio-blk-pci,drive=disk0,disable-modern=on,disable-legacy=off" \
>    -net "user,hostfwd=tcp::5022-:22" \
>    -dtb /.../versatile-pb-buster-5.4.51.dtb \
>    -kernel /.../kernel-qemu-5.4.51-buster \
>    -append 'root=/dev/vda2 panic=1' \
>    -no-reboot
>
> This means it is more recent than the Raspberry Pi Geek article, and the
> emulation works.
> But I'm not sure if this usable considering the added models -M raspi2
> and -M raspi3.
>
> Can you please advise how to proceed?
>
> In addition I would like to know if there's a memory limitation using
> models -M raspi2 and -M raspi3?
> To my understanding there's a limitation to 256MB using -M versatilepb.
> If yes, I consider to another raw image located on host's temporary
> filesystem and use this a swap in the client.
>
> And how can I make use of a client network device that is based on
> host's tap device connected to a network bridge?
>
> THX
>
>
> Am 05.10.2020 um 11:40 schrieb Alex Bennée:
> > Peter Maydell <peter.maydell@linaro.org> writes:
> >
> >> On Sun, 4 Oct 2020 at 18:44, Alex Bennée <alex.bennee@linaro.org> wrote:
> >>> Thomas <74cmonty@gmail.com> writes:
> >>>> I'm trying to emulate Rpi with QEMU.
> >>>> I found
> >>>> [url=1]this[/url]
> >>>> arcticle in Raspberry Pi Geek documenting the steps including persistent
> >>>> storage on host.
> >>>>
> >>>> However when starting the emulation with command
> >>>> qemu-system-arm -M versatilepb -cpu arm1176 -m 256 -serial stdio -hda
> >>>> 2020-08-20-raspios-buster-armhf-lite.img -net
> >>>> "user,hostfwd=tcp::5022-:22" -dtb versatile-pb-buster.dtb -kernel
> >>>> kernel-qemu-5.4.51-buster -append "root=/dev/sda2 rootfstype=ext4 rw
> >>>> panic=1" -no-reboot
> >>> Let's start with the fact you are using a versatilepb machine type with
> >>> a versatilepb dtb and not the rasppi model.
> >> Given the name of the kernel image, this probably actually *is*
> >> built for versatilepb, or it wouldn't have got as far as failing
> >> to mount the root partition. There seems to be a lot of confusion
> >> in the raspberry pi community about the difference between
> >> running the raspi userspace plus a for-versatilepb kernel
> >> versus running a full raspi setup.
> > Ahh your German is considerably better than mine ;-) Looking more
> > closely at the blog it seems to be predicated on extracting a Raspbian
> > kernel which at least stands a fighting chance of being a multi_config
> > kernel - like the buster above.
> >
> > I can see why these sorts of shenanigans used to be pulled when there
> > where no RaspPi models although if all you want to do is run an ARM user
> > space what's wrong with using linux-user for this sort of thing?
> >
> >> Anyway, failing to mount the rootfs and not listing any
> >> sda devices is not a problem with the fstab, because the system
> >> hasn't got as far as being able to mount the filesystem with a
> >> fstab on it yet. One possibility is that the kernel is
> >> missing the device drivers for either PCI or for the SCSI
> >> controller that gets plugged in to versatilepb by default.
> >>
> >> My guess at the cause is that you're trying to boot a Linux 5.something
> >> kernel and you've run into the issue described in this thread:
> >> https://lists.gnu.org/archive/html/qemu-discuss/2020-09/msg00023.html
> >> where the Linux 5.x sym53c8xx scsi driver is not compatible with QEMU's
> >> emulation of that device. If that's the case then you should see
> >> earlier in the kernel boot log error messages similar to the ones
> >> that Roger reported. The fix would be either to use an older
> >> kernel, or to change the QEMU commandline to use a different
> >> SCSI controller (or to use a virtio block device).
> > Do we have any documentation for the RaspPi models? The acceptance tests
> > look like they support the inbuilt MMC/SD controller device:
> >
> >          kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
> >                                 serial_kernel_cmdline[uart_id] +
> >                                 ' root=/dev/mmcblk0p2 rootwait ' +
> >                                 'dwc_otg.fiq_fsm_enable=0')
> >
> > It would be useful to fill the hole in the documentation so gently steer
> > people away from these hybrid franken-machine approaches.
> >
> >> thanks
> >> -- PMM
> >
>
>

Re: Emulate Rpi with QEMU fails

[Thomas Schneider]

Hello Paul,

many thanks for sharing this info.

Can you confirm that the emulated RPi with your command will use 
"internal QEMU" network, means the client cannot be accessed from any 
other device in LAN?
If yes, what is required to setup a TAP connected to host's network bridge?

Regards
Thomas

Am 06.10.2020 um 00:08 schrieb Paul Zimmerman:
> If you can upgrade to Qemu 5.1 or newer, you should be able to use the
> builtin raspi2 or raspi3 emulation directly. Version 5.1 adds support for the
> USB controller on the Pi.
>
> There is no limitation on the amount of memory you can use with this
> method, and the networking should work.
>
> Here is the command line I use to run the Raspbian image
> 2019-09-26-raspbian-buster. I extracted bcm2709-rpi-2-b and
> kernel7.img from the FAT partition inside the image file.
>
> qemu-system-arm -M raspi2 -drive
> file=bootpi/2019-09-26-raspbian-buster.img,format=raw,if=sd -dtb
> bcm2709-rpi-2-b.dtb -kernel kernel7.img -append 'rw
> earlycon=pl011,0x3f201000 console=ttyAMA0 loglevel=8
> root=/dev/mmcblk0p2 fsck.repair=yes net.ifnames=0 rootwait memtest=1
> dwc_otg.fiq_fsm_enable=0' -serial stdio -no-reboot -netdev
> user,id=net0 -usb -device usb-kbd -device usb-tablet -device
> usb-net,netdev=net0
>
> Hope this helps.
>
> On Mon, Oct 5, 2020 at 3:51 AM Thomas Schneider <74cmonty@gmail.com> wrote:
>> Hello,
>>
>> thanks for your replies.
>>
>> I must admit that I don't fully understand your analysis.
>> However you made some conclusions that are correct.
>>
>> In fact I have found a Github repo
>> <https://github.com/dhruvvyas90/qemu-rpi-kernel> where a specific kernel
>> and versatile-pb are provided + instructions for lauching the emulation
>> with the original RPi image file:
>> $ qemu-system-arm \
>>     -M versatilepb \
>>     -cpu arm1176 \
>>     -m 256 \
>>     -drive
>> "file=/.../2020-05-27-raspios-buster-lite-armhf.img,if=none,index=0,media=disk,format=raw,id=disk0"
>> \
>>     -device
>> "virtio-blk-pci,drive=disk0,disable-modern=on,disable-legacy=off" \
>>     -net "user,hostfwd=tcp::5022-:22" \
>>     -dtb /.../versatile-pb-buster-5.4.51.dtb \
>>     -kernel /.../kernel-qemu-5.4.51-buster \
>>     -append 'root=/dev/vda2 panic=1' \
>>     -no-reboot
>>
>> This means it is more recent than the Raspberry Pi Geek article, and the
>> emulation works.
>> But I'm not sure if this usable considering the added models -M raspi2
>> and -M raspi3.
>>
>> Can you please advise how to proceed?
>>
>> In addition I would like to know if there's a memory limitation using
>> models -M raspi2 and -M raspi3?
>> To my understanding there's a limitation to 256MB using -M versatilepb.
>> If yes, I consider to another raw image located on host's temporary
>> filesystem and use this a swap in the client.
>>
>> And how can I make use of a client network device that is based on
>> host's tap device connected to a network bridge?
>>
>> THX
>>
>>
>> Am 05.10.2020 um 11:40 schrieb Alex Bennée:
>>> Peter Maydell <peter.maydell@linaro.org> writes:
>>>
>>>> On Sun, 4 Oct 2020 at 18:44, Alex Bennée <alex.bennee@linaro.org> wrote:
>>>>> Thomas <74cmonty@gmail.com> writes:
>>>>>> I'm trying to emulate Rpi with QEMU.
>>>>>> I found
>>>>>> [url=1]this[/url]
>>>>>> arcticle in Raspberry Pi Geek documenting the steps including persistent
>>>>>> storage on host.
>>>>>>
>>>>>> However when starting the emulation with command
>>>>>> qemu-system-arm -M versatilepb -cpu arm1176 -m 256 -serial stdio -hda
>>>>>> 2020-08-20-raspios-buster-armhf-lite.img -net
>>>>>> "user,hostfwd=tcp::5022-:22" -dtb versatile-pb-buster.dtb -kernel
>>>>>> kernel-qemu-5.4.51-buster -append "root=/dev/sda2 rootfstype=ext4 rw
>>>>>> panic=1" -no-reboot
>>>>> Let's start with the fact you are using a versatilepb machine type with
>>>>> a versatilepb dtb and not the rasppi model.
>>>> Given the name of the kernel image, this probably actually *is*
>>>> built for versatilepb, or it wouldn't have got as far as failing
>>>> to mount the root partition. There seems to be a lot of confusion
>>>> in the raspberry pi community about the difference between
>>>> running the raspi userspace plus a for-versatilepb kernel
>>>> versus running a full raspi setup.
>>> Ahh your German is considerably better than mine ;-) Looking more
>>> closely at the blog it seems to be predicated on extracting a Raspbian
>>> kernel which at least stands a fighting chance of being a multi_config
>>> kernel - like the buster above.
>>>
>>> I can see why these sorts of shenanigans used to be pulled when there
>>> where no RaspPi models although if all you want to do is run an ARM user
>>> space what's wrong with using linux-user for this sort of thing?
>>>
>>>> Anyway, failing to mount the rootfs and not listing any
>>>> sda devices is not a problem with the fstab, because the system
>>>> hasn't got as far as being able to mount the filesystem with a
>>>> fstab on it yet. One possibility is that the kernel is
>>>> missing the device drivers for either PCI or for the SCSI
>>>> controller that gets plugged in to versatilepb by default.
>>>>
>>>> My guess at the cause is that you're trying to boot a Linux 5.something
>>>> kernel and you've run into the issue described in this thread:
>>>> https://lists.gnu.org/archive/html/qemu-discuss/2020-09/msg00023.html
>>>> where the Linux 5.x sym53c8xx scsi driver is not compatible with QEMU's
>>>> emulation of that device. If that's the case then you should see
>>>> earlier in the kernel boot log error messages similar to the ones
>>>> that Roger reported. The fix would be either to use an older
>>>> kernel, or to change the QEMU commandline to use a different
>>>> SCSI controller (or to use a virtio block device).
>>> Do we have any documentation for the RaspPi models? The acceptance tests
>>> look like they support the inbuilt MMC/SD controller device:
>>>
>>>           kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
>>>                                  serial_kernel_cmdline[uart_id] +
>>>                                  ' root=/dev/mmcblk0p2 rootwait ' +
>>>                                  'dwc_otg.fiq_fsm_enable=0')
>>>
>>> It would be useful to fill the hole in the documentation so gently steer
>>> people away from these hybrid franken-machine approaches.
>>>
>>>> thanks
>>>> -- PMM
>>

Re: Emulate Rpi with QEMU fails

[Paul Zimmerman]

On Mon, Oct 5, 2020 at 11:58 PM Thomas Schneider <74cmonty@gmail.com> wrote:
>
> Hello Paul,
>
> many thanks for sharing this info.
>
> Can you confirm that the emulated RPi with your command will use
> "internal QEMU" network, means the client cannot be accessed from any
> other device in LAN?
> If yes, what is required to setup a TAP connected to host's network bridge?
>
> Regards
> Thomas

That sets up User networking, so yes, generally you cannot access the
client from the outside network.

If you set up a bridge device on the host using TAP, then you can change
the "-netdev user,id=net0" in the command line to
"-netdev tap,helper=/usr/local/libexec/qemu-bridge-helper,id=net0"
(assuming your qemu was installed in /usr/local) and you should get a fully
functional network connection. I found that the bridge device must be named
"br0" or this will not work.

I found that setting up a bridge device is pretty finicky, though. You can
google for instructions on how to do it. I *think* these are the steps I
followed to make it work for me. This is on Ubuntu, other Linux
distributions work differently I believe. My ethernet device is 'eth0'
and my host ip address is '192.168.0.54', so you should change those
as needed for your environment.

sudo ip link add br0 type bridge
sudo ip tuntap add dev tap0 mode tap
sudo ip link set dev tap0 master br0
sudo ip link set dev eth0 master br0
sudo ip link set dev br0 up
sudo ip address delete 192.168.0.54/24 dev eth0
sudo ip address add 192.168.0.54/24 dev br0
sudo ip route add default via 192.168.0.1 dev br0

- Paul

>
> Am 06.10.2020 um 00:08 schrieb Paul Zimmerman:
> > If you can upgrade to Qemu 5.1 or newer, you should be able to use the
> > builtin raspi2 or raspi3 emulation directly. Version 5.1 adds support for the
> > USB controller on the Pi.
> >
> > There is no limitation on the amount of memory you can use with this
> > method, and the networking should work.
> >
> > Here is the command line I use to run the Raspbian image
> > 2019-09-26-raspbian-buster. I extracted bcm2709-rpi-2-b and
> > kernel7.img from the FAT partition inside the image file.
> >
> > qemu-system-arm -M raspi2 -drive
> > file=bootpi/2019-09-26-raspbian-buster.img,format=raw,if=sd -dtb
> > bcm2709-rpi-2-b.dtb -kernel kernel7.img -append 'rw
> > earlycon=pl011,0x3f201000 console=ttyAMA0 loglevel=8
> > root=/dev/mmcblk0p2 fsck.repair=yes net.ifnames=0 rootwait memtest=1
> > dwc_otg.fiq_fsm_enable=0' -serial stdio -no-reboot -netdev
> > user,id=net0 -usb -device usb-kbd -device usb-tablet -device
> > usb-net,netdev=net0
> >
> > Hope this helps.
> >
> > On Mon, Oct 5, 2020 at 3:51 AM Thomas Schneider <74cmonty@gmail.com> wrote:
> >> Hello,
> >>
> >> thanks for your replies.
> >>
> >> I must admit that I don't fully understand your analysis.
> >> However you made some conclusions that are correct.
> >>
> >> In fact I have found a Github repo
> >> <https://github.com/dhruvvyas90/qemu-rpi-kernel> where a specific kernel
> >> and versatile-pb are provided + instructions for lauching the emulation
> >> with the original RPi image file:
> >> $ qemu-system-arm \
> >>     -M versatilepb \
> >>     -cpu arm1176 \
> >>     -m 256 \
> >>     -drive
> >> "file=/.../2020-05-27-raspios-buster-lite-armhf.img,if=none,index=0,media=disk,format=raw,id=disk0"
> >> \
> >>     -device
> >> "virtio-blk-pci,drive=disk0,disable-modern=on,disable-legacy=off" \
> >>     -net "user,hostfwd=tcp::5022-:22" \
> >>     -dtb /.../versatile-pb-buster-5.4.51.dtb \
> >>     -kernel /.../kernel-qemu-5.4.51-buster \
> >>     -append 'root=/dev/vda2 panic=1' \
> >>     -no-reboot
> >>
> >> This means it is more recent than the Raspberry Pi Geek article, and the
> >> emulation works.
> >> But I'm not sure if this usable considering the added models -M raspi2
> >> and -M raspi3.
> >>
> >> Can you please advise how to proceed?
> >>
> >> In addition I would like to know if there's a memory limitation using
> >> models -M raspi2 and -M raspi3?
> >> To my understanding there's a limitation to 256MB using -M versatilepb.
> >> If yes, I consider to another raw image located on host's temporary
> >> filesystem and use this a swap in the client.
> >>
> >> And how can I make use of a client network device that is based on
> >> host's tap device connected to a network bridge?
> >>
> >> THX
> >>
> >>
> >> Am 05.10.2020 um 11:40 schrieb Alex Bennée:
> >>> Peter Maydell <peter.maydell@linaro.org> writes:
> >>>
> >>>> On Sun, 4 Oct 2020 at 18:44, Alex Bennée <alex.bennee@linaro.org> wrote:
> >>>>> Thomas <74cmonty@gmail.com> writes:
> >>>>>> I'm trying to emulate Rpi with QEMU.
> >>>>>> I found
> >>>>>> [url=1]this[/url]
> >>>>>> arcticle in Raspberry Pi Geek documenting the steps including persistent
> >>>>>> storage on host.
> >>>>>>
> >>>>>> However when starting the emulation with command
> >>>>>> qemu-system-arm -M versatilepb -cpu arm1176 -m 256 -serial stdio -hda
> >>>>>> 2020-08-20-raspios-buster-armhf-lite.img -net
> >>>>>> "user,hostfwd=tcp::5022-:22" -dtb versatile-pb-buster.dtb -kernel
> >>>>>> kernel-qemu-5.4.51-buster -append "root=/dev/sda2 rootfstype=ext4 rw
> >>>>>> panic=1" -no-reboot
> >>>>> Let's start with the fact you are using a versatilepb machine type with
> >>>>> a versatilepb dtb and not the rasppi model.
> >>>> Given the name of the kernel image, this probably actually *is*
> >>>> built for versatilepb, or it wouldn't have got as far as failing
> >>>> to mount the root partition. There seems to be a lot of confusion
> >>>> in the raspberry pi community about the difference between
> >>>> running the raspi userspace plus a for-versatilepb kernel
> >>>> versus running a full raspi setup.
> >>> Ahh your German is considerably better than mine ;-) Looking more
> >>> closely at the blog it seems to be predicated on extracting a Raspbian
> >>> kernel which at least stands a fighting chance of being a multi_config
> >>> kernel - like the buster above.
> >>>
> >>> I can see why these sorts of shenanigans used to be pulled when there
> >>> where no RaspPi models although if all you want to do is run an ARM user
> >>> space what's wrong with using linux-user for this sort of thing?
> >>>
> >>>> Anyway, failing to mount the rootfs and not listing any
> >>>> sda devices is not a problem with the fstab, because the system
> >>>> hasn't got as far as being able to mount the filesystem with a
> >>>> fstab on it yet. One possibility is that the kernel is
> >>>> missing the device drivers for either PCI or for the SCSI
> >>>> controller that gets plugged in to versatilepb by default.
> >>>>
> >>>> My guess at the cause is that you're trying to boot a Linux 5.something
> >>>> kernel and you've run into the issue described in this thread:
> >>>> https://lists.gnu.org/archive/html/qemu-discuss/2020-09/msg00023.html
> >>>> where the Linux 5.x sym53c8xx scsi driver is not compatible with QEMU's
> >>>> emulation of that device. If that's the case then you should see
> >>>> earlier in the kernel boot log error messages similar to the ones
> >>>> that Roger reported. The fix would be either to use an older
> >>>> kernel, or to change the QEMU commandline to use a different
> >>>> SCSI controller (or to use a virtio block device).
> >>> Do we have any documentation for the RaspPi models? The acceptance tests
> >>> look like they support the inbuilt MMC/SD controller device:
> >>>
> >>>           kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
> >>>                                  serial_kernel_cmdline[uart_id] +
> >>>                                  ' root=/dev/mmcblk0p2 rootwait ' +
> >>>                                  'dwc_otg.fiq_fsm_enable=0')
> >>>
> >>> It would be useful to fill the hole in the documentation so gently steer
> >>> people away from these hybrid franken-machine approaches.
> >>>
> >>>> thanks
> >>>> -- PMM
> >>
>

Re: Emulate Rpi with QEMU fails

[Alex Bennée]

Thomas Schneider <74cmonty@gmail.com> writes:

> Hello Paul,
>
> many thanks for sharing this info.
>
> Can you confirm that the emulated RPi with your command will use 
> "internal QEMU" network, means the client cannot be accessed from any 
> other device in LAN?

The support for user-mode and TAP networking is orthogonal to the
emulated device. However if you only want a few ports it's quite easy to
use port forwarding, e.g:

  -netdev user,id=unet,hostfwd=tcp::2222-:22

which forwards 2222 to port 22 on the device. I have an alias in
.ssh/config for accessing my QEMU devices.

> If yes, what is required to setup a TAP connected to host's network
> bridge?

I'll defer to others for this but generally when I want proper bridged
networking for a VM I use virt-manager/libvirt to configure it because
it can be quite fiddly to do by hand.

-- 
Alex Bennée

Re: Emulate Rpi with QEMU fails

[Thomas]

Hello!

Many thanks for your support.

I managed to get emulated RPi starting.

However there's one question I want to ask:
How can I accelerate the startup sequence?
I mean booting the emulated RPi takes more than 3 minutes.

Regards
Thomas

Am 06.10.20 um 11:58 schrieb Alex Bennée:
> Thomas Schneider <74cmonty@gmail.com> writes:
>
>> Hello Paul,
>>
>> many thanks for sharing this info.
>>
>> Can you confirm that the emulated RPi with your command will use 
>> "internal QEMU" network, means the client cannot be accessed from any 
>> other device in LAN?
> The support for user-mode and TAP networking is orthogonal to the
> emulated device. However if you only want a few ports it's quite easy to
> use port forwarding, e.g:
>
>   -netdev user,id=unet,hostfwd=tcp::2222-:22
>
> which forwards 2222 to port 22 on the device. I have an alias in
> .ssh/config for accessing my QEMU devices.
>
>> If yes, what is required to setup a TAP connected to host's network
>> bridge?
> I'll defer to others for this but generally when I want proper bridged
> networking for a VM I use virt-manager/libvirt to configure it because
> it can be quite fiddly to do by hand.
>

Re: Emulate Rpi with QEMU fails

[Paul Zimmerman]

On Tue, Oct 6, 2020 at 11:28 PM Thomas <74cmonty@gmail.com> wrote:
>
> Hello!
>
> Many thanks for your support.
>
> I managed to get emulated RPi starting.
>
> However there's one question I want to ask:
> How can I accelerate the startup sequence?
> I mean booting the emulated RPi takes more than 3 minutes.
>
> Regards
> Thomas

Get a faster computer? ;)

On my Intel i7 desktop it takes about 40 seconds to boot to the login:
prompt on the serial console, and about 1 min 8 seconds before the
GUI is up. On my 5 year old laptop it's probably twice that. I don't know
of any way to make it go faster.

- Paul

>
> Am 06.10.20 um 11:58 schrieb Alex Bennée:
> > Thomas Schneider <74cmonty@gmail.com> writes:
> >
> >> Hello Paul,
> >>
> >> many thanks for sharing this info.
> >>
> >> Can you confirm that the emulated RPi with your command will use
> >> "internal QEMU" network, means the client cannot be accessed from any
> >> other device in LAN?
> > The support for user-mode and TAP networking is orthogonal to the
> > emulated device. However if you only want a few ports it's quite easy to
> > use port forwarding, e.g:
> >
> >   -netdev user,id=unet,hostfwd=tcp::2222-:22
> >
> > which forwards 2222 to port 22 on the device. I have an alias in
> > .ssh/config for accessing my QEMU devices.
> >
> >> If yes, what is required to setup a TAP connected to host's network
> >> bridge?
> > I'll defer to others for this but generally when I want proper bridged
> > networking for a VM I use virt-manager/libvirt to configure it because
> > it can be quite fiddly to do by hand.
> >
>

Re: Emulate Rpi with QEMU fails

[Thomas Schneider]

Hi,

I already considered the host CPU power.
However I have this 
<https://ark.intel.com/content/www/us/en/ark/products/33924/intel-core-2-quad-processor-q9550-12m-cache-2-83-ghz-1333-mhz-fsb.html> 
CPU
Intel Core 2 Quad Q9550 2,83 GHz
and assumed this should be powerful enough for RPi emulation.

But maybe my assumption was too optimistic.


Am 07.10.2020 um 08:50 schrieb Paul Zimmerman:
> On Tue, Oct 6, 2020 at 11:28 PM Thomas <74cmonty@gmail.com> wrote:
>> Hello!
>>
>> Many thanks for your support.
>>
>> I managed to get emulated RPi starting.
>>
>> However there's one question I want to ask:
>> How can I accelerate the startup sequence?
>> I mean booting the emulated RPi takes more than 3 minutes.
>>
>> Regards
>> Thomas
> Get a faster computer? ;)
>
> On my Intel i7 desktop it takes about 40 seconds to boot to the login:
> prompt on the serial console, and about 1 min 8 seconds before the
> GUI is up. On my 5 year old laptop it's probably twice that. I don't know
> of any way to make it go faster.
>
> - Paul
>
>> Am 06.10.20 um 11:58 schrieb Alex Bennée:
>>> Thomas Schneider <74cmonty@gmail.com> writes:
>>>
>>>> Hello Paul,
>>>>
>>>> many thanks for sharing this info.
>>>>
>>>> Can you confirm that the emulated RPi with your command will use
>>>> "internal QEMU" network, means the client cannot be accessed from any
>>>> other device in LAN?
>>> The support for user-mode and TAP networking is orthogonal to the
>>> emulated device. However if you only want a few ports it's quite easy to
>>> use port forwarding, e.g:
>>>
>>>    -netdev user,id=unet,hostfwd=tcp::2222-:22
>>>
>>> which forwards 2222 to port 22 on the device. I have an alias in
>>> .ssh/config for accessing my QEMU devices.
>>>
>>>> If yes, what is required to setup a TAP connected to host's network
>>>> bridge?
>>> I'll defer to others for this but generally when I want proper bridged
>>> networking for a VM I use virt-manager/libvirt to configure it because
>>> it can be quite fiddly to do by hand.
>>>

Re: Emulate Rpi with QEMU fails

[Alex Bennée]

Thomas Schneider <74cmonty@gmail.com> writes:

> Hi,
>
> I already considered the host CPU power.
> However I have this 
> <https://ark.intel.com/content/www/us/en/ark/products/33924/intel-core-2-quad-processor-q9550-12m-cache-2-83-ghz-1333-mhz-fsb.html> 
> CPU
> Intel Core 2 Quad Q9550 2,83 GHz
> and assumed this should be powerful enough for RPi emulation.

For each emulated instruction you can be running between 6-10 host
instructions on average. We have certainly improved the performance of
the emulation over time and take advantage of multiple threads but in
the end system emulation will always be fairly expensive.

> But maybe my assumption was too optimistic.

You can use perf to record your boot and analyse where QEMU is spending
it's time. Unless there is a major outlier though it's unlikely to be
easy to optimise.

>
>
> Am 07.10.2020 um 08:50 schrieb Paul Zimmerman:
>> On Tue, Oct 6, 2020 at 11:28 PM Thomas <74cmonty@gmail.com> wrote:
>>> Hello!
>>>
>>> Many thanks for your support.
>>>
>>> I managed to get emulated RPi starting.
>>>
>>> However there's one question I want to ask:
>>> How can I accelerate the startup sequence?
>>> I mean booting the emulated RPi takes more than 3 minutes.
>>>
>>> Regards
>>> Thomas
>> Get a faster computer? ;)
>>
>> On my Intel i7 desktop it takes about 40 seconds to boot to the login:
>> prompt on the serial console, and about 1 min 8 seconds before the
>> GUI is up. On my 5 year old laptop it's probably twice that. I don't know
>> of any way to make it go faster.
>>
>> - Paul
>>
>>> Am 06.10.20 um 11:58 schrieb Alex Bennée:
>>>> Thomas Schneider <74cmonty@gmail.com> writes:
>>>>
>>>>> Hello Paul,
>>>>>
>>>>> many thanks for sharing this info.
>>>>>
>>>>> Can you confirm that the emulated RPi with your command will use
>>>>> "internal QEMU" network, means the client cannot be accessed from any
>>>>> other device in LAN?
>>>> The support for user-mode and TAP networking is orthogonal to the
>>>> emulated device. However if you only want a few ports it's quite easy to
>>>> use port forwarding, e.g:
>>>>
>>>>    -netdev user,id=unet,hostfwd=tcp::2222-:22
>>>>
>>>> which forwards 2222 to port 22 on the device. I have an alias in
>>>> .ssh/config for accessing my QEMU devices.
>>>>
>>>>> If yes, what is required to setup a TAP connected to host's network
>>>>> bridge?
>>>> I'll defer to others for this but generally when I want proper bridged
>>>> networking for a VM I use virt-manager/libvirt to configure it because
>>>> it can be quite fiddly to do by hand.
>>>>


-- 
Alex Bennée

Re: Emulate Rpi with QEMU fails

[Thomas Schneider]

Are you referring to this tool?
https://github.com/stefano-garzarella/qemu-boot-time 
<https://github.com/stefano-garzarella/qemu-boot-time>


Am 07.10.2020 um 13:00 schrieb Alex Bennée:
> perf to record your boot

Re: Emulate Rpi with QEMU fails

[Alex Bennée]

Thomas Schneider <74cmonty@gmail.com> writes:

> Are you referring to this tool?
> https://github.com/stefano-garzarella/qemu-boot-time 
> <https://github.com/stefano-garzarella/qemu-boot-time>

No - just plain perf:

  perf record $QEMU $ARGS

Then a "perf report" which will show you the hotspots, for example:

   8.92%  qemu-system-aar  qemu-system-aarch64      [.] helper_lookup_tb_ptr
   4.76%  qemu-system-aar  qemu-system-aarch64      [.] liveness_pass_1
   3.69%  qemu-system-aar  qemu-system-aarch64      [.] tcg_gen_code
   2.95%  qemu-system-aar  qemu-system-aarch64      [.] qht_lookup_custom
   2.93%  qemu-system-aar  qemu-system-aarch64      [.] tcg_optimize
   1.28%  qemu-system-aar  qemu-system-aarch64      [.] tcg_out_opc.isra.15
   1.09%  qemu-system-aar  qemu-system-aarch64      [.] get_phys_addr_lpae
   1.09%  qemu-system-aar  [kernel.kallsyms]        [k] isolate_freepages_block
   1.05%  qemu-system-aar  qemu-system-aarch64      [.] cpu_get_tb_cpu_state
   0.98%  qemu-system-aar  [kernel.kallsyms]        [k] do_syscall_64
   0.94%  qemu-system-aar  qemu-system-aarch64      [.] tb_lookup_cmp
   0.78%  qemu-system-aar  qemu-system-aarch64      [.] init_ts_info
   0.73%  qemu-system-aar  qemu-system-aarch64      [.] tb_htable_lookup
   0.73%  qemu-system-aar  qemu-system-aarch64      [.] tb_gen_code
   0.73%  qemu-system-aar  qemu-system-aarch64      [.] tlb_set_page_with_attrs

>
>
> Am 07.10.2020 um 13:00 schrieb Alex Bennée:
>> perf to record your boot


-- 
Alex Bennée

Re: Emulate Rpi with QEMU fails

[Thomas]

Interesting enough is: my top figure reported by perf is like yours:

Samples: 6M of event 'cycles:u', Event count (approx.): 1936571734942
Overhead  Command          Shared Object                  Symbol
   7,95%  qemu-system-arm  qemu-system-arm                [.]
helper_lookup_tb_ptr                                                        
◆
   4,16%  qemu-system-arm  qemu-system-arm                [.]
cpu_get_tb_cpu_state                                                        
▒
   2,52%  qemu-system-arm  libpthread-2.32.so             [.]
__pthread_mutex_lock                                                        
▒
   2,06%  qemu-system-arm  qemu-system-arm                [.]
qht_lookup_custom                                                           
▒
   1,66%  qemu-system-arm  qemu-system-arm                [.]
tlb_set_page_with_attrs                                                     
▒
   1,61%  qemu-system-arm  libpthread-2.32.so             [.]
__pthread_mutex_unlock_usercnt                                              
▒
   1,59%  qemu-system-arm  qemu-system-arm                [.]
get_phys_addr                                                               
▒
   1,27%  qemu-system-arm  qemu-system-arm                [.]
cpu_exec                                                                    
▒
   1,23%  qemu-system-arm  qemu-system-arm                [.]
object_class_dynamic_cast_assert                                            
▒
   0,98%  qemu-system-arm  libc-2.32.so                   [.]
_int_malloc                                                                 
▒
   0,95%  qemu-system-arm  qemu-system-arm                [.]
object_dynamic_cast_assert                                                  
▒
   0,95%  qemu-system-arm  qemu-system-arm                [.]
tb_htable_lookup                                                            
▒
   0,92%  qemu-system-arm  qemu-system-arm                [.]
tcg_gen_code                                                                
▒
   0,91%  qemu-system-arm  qemu-system-arm                [.]
qemu_mutex_lock_impl                                                        
▒
   0,75%  qemu-system-arm  qemu-system-arm                [.]
get_page_addr_code_hostp                                                    
▒
   0,73%  qemu-system-arm  qemu-system-arm                [.]
tcg_optimize                                                                
▒
   0,71%  qemu-system-arm  qemu-system-arm                [.]
qemu_mutex_unlock_impl                                                      
▒
   0,69%  qemu-system-arm  libc-2.32.so                   [.]
_int_free                                                                   
▒
   0,64%  qemu-system-arm  qemu-system-arm                [.]
tb_flush_jmp_cache                                                          
▒
   0,63%  qemu-system-arm  qemu-system-arm                [.]
address_space_ldl_le                                                        
▒
   0,62%  qemu-system-arm  libpthread-2.32.so             [.]
__lll_lock_wait                                                             
▒
   0,58%  qemu-system-arm  libpthread-2.32.so             [.]
pthread_cond_wait@@GLIBC_2.3.2                                              
▒
   0,52%  qemu-system-arm  qemu-system-arm                [.]
cpu_reset_interrupt                                                         
▒
   0,52%  qemu-system-arm  libc-2.32.so                   [.]
cfree@GLIBC_2.2.5                                                           
▒
   0,52%  qemu-system-arm  qemu-system-arm                [.]
qemu_set_irq                                                                
▒

However the absolute time is 3-4 minutes.
And I don't know where I should start optimization now.



Am 07.10.20 um 14:02 schrieb Alex Bennée:
> Thomas Schneider <74cmonty@gmail.com> writes:
>
>> Are you referring to this tool?
>> https://github.com/stefano-garzarella/qemu-boot-time 
>> <https://github.com/stefano-garzarella/qemu-boot-time>
> No - just plain perf:
>
>   perf record $QEMU $ARGS
>
> Then a "perf report" which will show you the hotspots, for example:
>
>    8.92%  qemu-system-aar  qemu-system-aarch64      [.] helper_lookup_tb_ptr
>    4.76%  qemu-system-aar  qemu-system-aarch64      [.] liveness_pass_1
>    3.69%  qemu-system-aar  qemu-system-aarch64      [.] tcg_gen_code
>    2.95%  qemu-system-aar  qemu-system-aarch64      [.] qht_lookup_custom
>    2.93%  qemu-system-aar  qemu-system-aarch64      [.] tcg_optimize
>    1.28%  qemu-system-aar  qemu-system-aarch64      [.] tcg_out_opc.isra.15
>    1.09%  qemu-system-aar  qemu-system-aarch64      [.] get_phys_addr_lpae
>    1.09%  qemu-system-aar  [kernel.kallsyms]        [k] isolate_freepages_block
>    1.05%  qemu-system-aar  qemu-system-aarch64      [.] cpu_get_tb_cpu_state
>    0.98%  qemu-system-aar  [kernel.kallsyms]        [k] do_syscall_64
>    0.94%  qemu-system-aar  qemu-system-aarch64      [.] tb_lookup_cmp
>    0.78%  qemu-system-aar  qemu-system-aarch64      [.] init_ts_info
>    0.73%  qemu-system-aar  qemu-system-aarch64      [.] tb_htable_lookup
>    0.73%  qemu-system-aar  qemu-system-aarch64      [.] tb_gen_code
>    0.73%  qemu-system-aar  qemu-system-aarch64      [.] tlb_set_page_with_attrs
>
>>
>> Am 07.10.2020 um 13:00 schrieb Alex Bennée:
>>> perf to record your boot
>

Re: Emulate Rpi with QEMU fails

[Paul Zimmerman]

Hi Thomas,

What does 'top' say while the emulation is running? I have an 8 cpu-thread
system, yet 'top' never shows more than about 300% cpu. I would have
thought it would get closer to 800% cpu. And running qemu as root with
nice -20 doesn't seem to make much difference.

- Paul

On Thu, Oct 8, 2020 at 12:00 AM Thomas <74cmonty@gmail.com> wrote:
>
> Interesting enough is: my top figure reported by perf is like yours:
>
> Samples: 6M of event 'cycles:u', Event count (approx.): 1936571734942
> Overhead  Command          Shared Object                  Symbol
>    7,95%  qemu-system-arm  qemu-system-arm                [.]
> helper_lookup_tb_ptr
> ◆
>    4,16%  qemu-system-arm  qemu-system-arm                [.]
> cpu_get_tb_cpu_state
> ▒
>    2,52%  qemu-system-arm  libpthread-2.32.so             [.]
> __pthread_mutex_lock
> ▒
>    2,06%  qemu-system-arm  qemu-system-arm                [.]
> qht_lookup_custom
> ▒
>    1,66%  qemu-system-arm  qemu-system-arm                [.]
> tlb_set_page_with_attrs
> ▒
>    1,61%  qemu-system-arm  libpthread-2.32.so             [.]
> __pthread_mutex_unlock_usercnt
> ▒
>    1,59%  qemu-system-arm  qemu-system-arm                [.]
> get_phys_addr
> ▒
>    1,27%  qemu-system-arm  qemu-system-arm                [.]
> cpu_exec
> ▒
>    1,23%  qemu-system-arm  qemu-system-arm                [.]
> object_class_dynamic_cast_assert
> ▒
>    0,98%  qemu-system-arm  libc-2.32.so                   [.]
> _int_malloc
> ▒
>    0,95%  qemu-system-arm  qemu-system-arm                [.]
> object_dynamic_cast_assert
> ▒
>    0,95%  qemu-system-arm  qemu-system-arm                [.]
> tb_htable_lookup
> ▒
>    0,92%  qemu-system-arm  qemu-system-arm                [.]
> tcg_gen_code
> ▒
>    0,91%  qemu-system-arm  qemu-system-arm                [.]
> qemu_mutex_lock_impl
> ▒
>    0,75%  qemu-system-arm  qemu-system-arm                [.]
> get_page_addr_code_hostp
> ▒
>    0,73%  qemu-system-arm  qemu-system-arm                [.]
> tcg_optimize
> ▒
>    0,71%  qemu-system-arm  qemu-system-arm                [.]
> qemu_mutex_unlock_impl
> ▒
>    0,69%  qemu-system-arm  libc-2.32.so                   [.]
> _int_free
> ▒
>    0,64%  qemu-system-arm  qemu-system-arm                [.]
> tb_flush_jmp_cache
> ▒
>    0,63%  qemu-system-arm  qemu-system-arm                [.]
> address_space_ldl_le
> ▒
>    0,62%  qemu-system-arm  libpthread-2.32.so             [.]
> __lll_lock_wait
> ▒
>    0,58%  qemu-system-arm  libpthread-2.32.so             [.]
> pthread_cond_wait@@GLIBC_2.3.2
> ▒
>    0,52%  qemu-system-arm  qemu-system-arm                [.]
> cpu_reset_interrupt
> ▒
>    0,52%  qemu-system-arm  libc-2.32.so                   [.]
> cfree@GLIBC_2.2.5
> ▒
>    0,52%  qemu-system-arm  qemu-system-arm                [.]
> qemu_set_irq
> ▒
>
> However the absolute time is 3-4 minutes.
> And I don't know where I should start optimization now.
>
>
>
> Am 07.10.20 um 14:02 schrieb Alex Bennée:
> > Thomas Schneider <74cmonty@gmail.com> writes:
> >
> >> Are you referring to this tool?
> >> https://github.com/stefano-garzarella/qemu-boot-time
> >> <https://github.com/stefano-garzarella/qemu-boot-time>
> > No - just plain perf:
> >
> >   perf record $QEMU $ARGS
> >
> > Then a "perf report" which will show you the hotspots, for example:
> >
> >    8.92%  qemu-system-aar  qemu-system-aarch64      [.] helper_lookup_tb_ptr
> >    4.76%  qemu-system-aar  qemu-system-aarch64      [.] liveness_pass_1
> >    3.69%  qemu-system-aar  qemu-system-aarch64      [.] tcg_gen_code
> >    2.95%  qemu-system-aar  qemu-system-aarch64      [.] qht_lookup_custom
> >    2.93%  qemu-system-aar  qemu-system-aarch64      [.] tcg_optimize
> >    1.28%  qemu-system-aar  qemu-system-aarch64      [.] tcg_out_opc.isra.15
> >    1.09%  qemu-system-aar  qemu-system-aarch64      [.] get_phys_addr_lpae
> >    1.09%  qemu-system-aar  [kernel.kallsyms]        [k] isolate_freepages_block
> >    1.05%  qemu-system-aar  qemu-system-aarch64      [.] cpu_get_tb_cpu_state
> >    0.98%  qemu-system-aar  [kernel.kallsyms]        [k] do_syscall_64
> >    0.94%  qemu-system-aar  qemu-system-aarch64      [.] tb_lookup_cmp
> >    0.78%  qemu-system-aar  qemu-system-aarch64      [.] init_ts_info
> >    0.73%  qemu-system-aar  qemu-system-aarch64      [.] tb_htable_lookup
> >    0.73%  qemu-system-aar  qemu-system-aarch64      [.] tb_gen_code
> >    0.73%  qemu-system-aar  qemu-system-aarch64      [.] tlb_set_page_with_attrs
> >
> >>
> >> Am 07.10.2020 um 13:00 schrieb Alex Bennée:
> >>> perf to record your boot
> >
>

Re: Emulate Rpi with QEMU fails

[Paul Zimmerman]

Running 'top -H' (to show individual threads) I see qemu is using 6 to 7
threads while running, and each thread is taking from 15% to 70% or so
of cpu time. So probably qemu is not able to spread the work among the
threads evenly enough to use all the cpu time available.

Might be an interesting area of investigation for someone motivated
enough :)

- Paul

On Thu, Oct 8, 2020 at 2:07 PM Paul Zimmerman <pauldzim@gmail.com> wrote:
>
> Hi Thomas,
>
> What does 'top' say while the emulation is running? I have an 8 cpu-thread
> system, yet 'top' never shows more than about 300% cpu. I would have
> thought it would get closer to 800% cpu. And running qemu as root with
> nice -20 doesn't seem to make much difference.
>
> - Paul
>
> On Thu, Oct 8, 2020 at 12:00 AM Thomas <74cmonty@gmail.com> wrote:
> >
> > Interesting enough is: my top figure reported by perf is like yours:
> >
> > Samples: 6M of event 'cycles:u', Event count (approx.): 1936571734942
> > Overhead  Command          Shared Object                  Symbol
> >    7,95%  qemu-system-arm  qemu-system-arm                [.]
> > helper_lookup_tb_ptr
> > ◆
> >    4,16%  qemu-system-arm  qemu-system-arm                [.]
> > cpu_get_tb_cpu_state
> > ▒
> >    2,52%  qemu-system-arm  libpthread-2.32.so             [.]
> > __pthread_mutex_lock
> > ▒
> >    2,06%  qemu-system-arm  qemu-system-arm                [.]
> > qht_lookup_custom
> > ▒
> >    1,66%  qemu-system-arm  qemu-system-arm                [.]
> > tlb_set_page_with_attrs
> > ▒
> >    1,61%  qemu-system-arm  libpthread-2.32.so             [.]
> > __pthread_mutex_unlock_usercnt
> > ▒
> >    1,59%  qemu-system-arm  qemu-system-arm                [.]
> > get_phys_addr
> > ▒
> >    1,27%  qemu-system-arm  qemu-system-arm                [.]
> > cpu_exec
> > ▒
> >    1,23%  qemu-system-arm  qemu-system-arm                [.]
> > object_class_dynamic_cast_assert
> > ▒
> >    0,98%  qemu-system-arm  libc-2.32.so                   [.]
> > _int_malloc
> > ▒
> >    0,95%  qemu-system-arm  qemu-system-arm                [.]
> > object_dynamic_cast_assert
> > ▒
> >    0,95%  qemu-system-arm  qemu-system-arm                [.]
> > tb_htable_lookup
> > ▒
> >    0,92%  qemu-system-arm  qemu-system-arm                [.]
> > tcg_gen_code
> > ▒
> >    0,91%  qemu-system-arm  qemu-system-arm                [.]
> > qemu_mutex_lock_impl
> > ▒
> >    0,75%  qemu-system-arm  qemu-system-arm                [.]
> > get_page_addr_code_hostp
> > ▒
> >    0,73%  qemu-system-arm  qemu-system-arm                [.]
> > tcg_optimize
> > ▒
> >    0,71%  qemu-system-arm  qemu-system-arm                [.]
> > qemu_mutex_unlock_impl
> > ▒
> >    0,69%  qemu-system-arm  libc-2.32.so                   [.]
> > _int_free
> > ▒
> >    0,64%  qemu-system-arm  qemu-system-arm                [.]
> > tb_flush_jmp_cache
> > ▒
> >    0,63%  qemu-system-arm  qemu-system-arm                [.]
> > address_space_ldl_le
> > ▒
> >    0,62%  qemu-system-arm  libpthread-2.32.so             [.]
> > __lll_lock_wait
> > ▒
> >    0,58%  qemu-system-arm  libpthread-2.32.so             [.]
> > pthread_cond_wait@@GLIBC_2.3.2
> > ▒
> >    0,52%  qemu-system-arm  qemu-system-arm                [.]
> > cpu_reset_interrupt
> > ▒
> >    0,52%  qemu-system-arm  libc-2.32.so                   [.]
> > cfree@GLIBC_2.2.5
> > ▒
> >    0,52%  qemu-system-arm  qemu-system-arm                [.]
> > qemu_set_irq
> > ▒
> >
> > However the absolute time is 3-4 minutes.
> > And I don't know where I should start optimization now.
> >
> >
> >
> > Am 07.10.20 um 14:02 schrieb Alex Bennée:
> > > Thomas Schneider <74cmonty@gmail.com> writes:
> > >
> > >> Are you referring to this tool?
> > >> https://github.com/stefano-garzarella/qemu-boot-time
> > >> <https://github.com/stefano-garzarella/qemu-boot-time>
> > > No - just plain perf:
> > >
> > >   perf record $QEMU $ARGS
> > >
> > > Then a "perf report" which will show you the hotspots, for example:
> > >
> > >    8.92%  qemu-system-aar  qemu-system-aarch64      [.] helper_lookup_tb_ptr
> > >    4.76%  qemu-system-aar  qemu-system-aarch64      [.] liveness_pass_1
> > >    3.69%  qemu-system-aar  qemu-system-aarch64      [.] tcg_gen_code
> > >    2.95%  qemu-system-aar  qemu-system-aarch64      [.] qht_lookup_custom
> > >    2.93%  qemu-system-aar  qemu-system-aarch64      [.] tcg_optimize
> > >    1.28%  qemu-system-aar  qemu-system-aarch64      [.] tcg_out_opc.isra.15
> > >    1.09%  qemu-system-aar  qemu-system-aarch64      [.] get_phys_addr_lpae
> > >    1.09%  qemu-system-aar  [kernel.kallsyms]        [k] isolate_freepages_block
> > >    1.05%  qemu-system-aar  qemu-system-aarch64      [.] cpu_get_tb_cpu_state
> > >    0.98%  qemu-system-aar  [kernel.kallsyms]        [k] do_syscall_64
> > >    0.94%  qemu-system-aar  qemu-system-aarch64      [.] tb_lookup_cmp
> > >    0.78%  qemu-system-aar  qemu-system-aarch64      [.] init_ts_info
> > >    0.73%  qemu-system-aar  qemu-system-aarch64      [.] tb_htable_lookup
> > >    0.73%  qemu-system-aar  qemu-system-aarch64      [.] tb_gen_code
> > >    0.73%  qemu-system-aar  qemu-system-aarch64      [.] tlb_set_page_with_attrs
> > >
> > >>
> > >> Am 07.10.2020 um 13:00 schrieb Alex Bennée:
> > >>> perf to record your boot
> > >
> >

Re: Emulate Rpi with QEMU fails

[Alex Bennée]

Thomas <74cmonty@gmail.com> writes:

> Interesting enough is: my top figure reported by perf is like yours:
>
> Samples: 6M of event 'cycles:u', Event count (approx.): 1936571734942
> Overhead  Command          Shared Object                  Symbol
>    7,95%  qemu-system-arm  qemu-system-arm                [.]
> helper_lookup_tb_ptr                                                        
> ◆
>    4,16%  qemu-system-arm  qemu-system-arm                [.]
> cpu_get_tb_cpu_state                                                        
> ▒

Not quite - cpu_get_tb_state should be a fair bit cheaper now since the
optimisations to hflags where introduced.

>    2,52%  qemu-system-arm  libpthread-2.32.so             [.]
> __pthread_mutex_lock                                                        
> ▒

This indicated there is some lock contention. The QSP profiler can
help here.

Are you running the latest state of master?

-- 
Alex Bennée