[Qemu-devel] [Bug 1829779] [NEW] qemu-system-arm and qemu-system-aarch64 QMP hangs after kernel boots

[Qemu-devel] [Bug 1829779] [NEW] qemu-system-arm and qemu-system-aarch64 QMP hangs after kernel boots

[Cleber Rosa]

Public bug reported:

After booting a Linux kernel on both arm and aarch64, the QMP sockets gets unresponsive. Initially, this was thought to be limited to "quit" commands, but it reproduced with others (such as in this
reproducer).  This is a partial log output:
    
   >>> {'execute': 'qmp_capabilities'}
   <<< {'return': {}}
   Booting Linux on physical CPU 0x0000000000 [0x410fd034]
   Linux version 4.18.16-300.fc29.aarch64 (mockbuild@buildvm-aarch64-02.arm.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Sat Oct 20 23:12:22 UTC 2018
   ...
   Policy zone: DMA32
   Kernel command line: printk.time=0 console=ttyAMA0
   >>> {'execute': 'stop'}
   <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470173}, 'event': 'STOP'}
   <<< {'return': {}}
   >>> {'execute': 'cont'}
   <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470849}, 'event': 'RESUME'}
   <<< {'return': {}}
   >>> {'execute': 'stop'}
    
Sometimes it takes just the first "stop" command.  Overall, I was able to reproduce 100% of times when applied on top of 6d8e75d41c58892ccc5d4ad61c4da476684c1c83.

The reproducer test can be seen/fetched at:
 - https://github.com/clebergnu/qemu/commit/c778e28c24030c4a36548b714293b319f4bf18df

And test results from Travis CI can be seen at:
 - https://travis-ci.org/clebergnu/qemu/jobs/534915669

For convenience purposes, here's qemu-system-aarch64 launching and hanging on the first "stop":
 - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3615
 - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3645

And here's qemu-system-arm hanging the very same way:
 - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3780
 - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3800

** Affects: qemu
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1829779

Title:
  qemu-system-arm and qemu-system-aarch64 QMP hangs after kernel boots

Status in QEMU:
  New

Bug description:
  After booting a Linux kernel on both arm and aarch64, the QMP sockets gets unresponsive. Initially, this was thought to be limited to "quit" commands, but it reproduced with others (such as in this
  reproducer).  This is a partial log output:
      
     >>> {'execute': 'qmp_capabilities'}
     <<< {'return': {}}
     Booting Linux on physical CPU 0x0000000000 [0x410fd034]
     Linux version 4.18.16-300.fc29.aarch64 (mockbuild@buildvm-aarch64-02.arm.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Sat Oct 20 23:12:22 UTC 2018
     ...
     Policy zone: DMA32
     Kernel command line: printk.time=0 console=ttyAMA0
     >>> {'execute': 'stop'}
     <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470173}, 'event': 'STOP'}
     <<< {'return': {}}
     >>> {'execute': 'cont'}
     <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470849}, 'event': 'RESUME'}
     <<< {'return': {}}
     >>> {'execute': 'stop'}
      
  Sometimes it takes just the first "stop" command.  Overall, I was able to reproduce 100% of times when applied on top of 6d8e75d41c58892ccc5d4ad61c4da476684c1c83.

  The reproducer test can be seen/fetched at:
   - https://github.com/clebergnu/qemu/commit/c778e28c24030c4a36548b714293b319f4bf18df

  And test results from Travis CI can be seen at:
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669

  For convenience purposes, here's qemu-system-aarch64 launching and hanging on the first "stop":
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3615
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3645

  And here's qemu-system-arm hanging the very same way:
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3780
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3800

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1829779/+subscriptions

[Qemu-devel] [Bug 1829779] Re: qemu-system-arm and qemu-system-aarch64 QMP hangs after kernel boots

[Cleber Rosa]

** Description changed:

  After booting a Linux kernel on both arm and aarch64, the QMP sockets gets unresponsive. Initially, this was thought to be limited to "quit" commands, but it reproduced with others (such as in this
  reproducer).  This is a partial log output:
-     
-    >>> {'execute': 'qmp_capabilities'}
-    <<< {'return': {}}
-    Booting Linux on physical CPU 0x0000000000 [0x410fd034]
-    Linux version 4.18.16-300.fc29.aarch64 (mockbuild@buildvm-aarch64-02.arm.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Sat Oct 20 23:12:22 UTC 2018
-    ...
-    Policy zone: DMA32
-    Kernel command line: printk.time=0 console=ttyAMA0
-    >>> {'execute': 'stop'}
-    <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470173}, 'event': 'STOP'}
-    <<< {'return': {}}
-    >>> {'execute': 'cont'}
-    <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470849}, 'event': 'RESUME'}
-    <<< {'return': {}}
-    >>> {'execute': 'stop'}
-     
- Sometimes it takes just the first "stop" command.  Overall, I was able to reproduce 100% of times when applied on top of 6d8e75d41c58892ccc5d4ad61c4da476684c1c83.
+ 
+    >>> {'execute': 'qmp_capabilities'}
+    <<< {'return': {}}
+    Booting Linux on physical CPU 0x0000000000 [0x410fd034]
+    Linux version 4.18.16-300.fc29.aarch64 (mockbuild@buildvm-aarch64-02.arm.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Sat Oct 20 23:12:22 UTC 2018
+    ...
+    Policy zone: DMA32
+    Kernel command line: printk.time=0 console=ttyAMA0
+    >>> {'execute': 'stop'}
+    <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470173}, 'event': 'STOP'}
+    <<< {'return': {}}
+    >>> {'execute': 'cont'}
+    <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470849}, 'event': 'RESUME'}
+    <<< {'return': {}}
+    >>> {'execute': 'stop'}
+ 
+ Sometimes it takes just the first "stop" command.  Overall, I was able
+ to reproduce 100% of times when applied on top of
+ 6d8e75d41c58892ccc5d4ad61c4da476684c1c83.
  
  The reproducer test can be seen/fetched at:
-  - https://github.com/clebergnu/qemu/commit/c778e28c24030c4a36548b714293b319f4bf18df
+  - https://github.com/clebergnu/qemu/commit/c778e28c24030c4a36548b714293b319f4bf18df
  
  And test results from Travis CI can be seen at:
-  - https://travis-ci.org/clebergnu/qemu/jobs/534915669
+  - https://travis-ci.org/clebergnu/qemu/jobs/534915669
  
  For convenience purposes, here's qemu-system-aarch64 launching and hanging on the first "stop":
-  - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3615
-  - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3645
+  - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3634
+  - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3664
  
  And here's qemu-system-arm hanging the very same way:
-  - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3780
-  - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3800
+  - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3799
+  - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3819

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1829779

Title:
  qemu-system-arm and qemu-system-aarch64 QMP hangs after kernel boots

Status in QEMU:
  New

Bug description:
  After booting a Linux kernel on both arm and aarch64, the QMP sockets gets unresponsive. Initially, this was thought to be limited to "quit" commands, but it reproduced with others (such as in this
  reproducer).  This is a partial log output:

     >>> {'execute': 'qmp_capabilities'}
     <<< {'return': {}}
     Booting Linux on physical CPU 0x0000000000 [0x410fd034]
     Linux version 4.18.16-300.fc29.aarch64 (mockbuild@buildvm-aarch64-02.arm.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Sat Oct 20 23:12:22 UTC 2018
     ...
     Policy zone: DMA32
     Kernel command line: printk.time=0 console=ttyAMA0
     >>> {'execute': 'stop'}
     <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470173}, 'event': 'STOP'}
     <<< {'return': {}}
     >>> {'execute': 'cont'}
     <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470849}, 'event': 'RESUME'}
     <<< {'return': {}}
     >>> {'execute': 'stop'}

  Sometimes it takes just the first "stop" command.  Overall, I was able
  to reproduce 100% of times when applied on top of
  6d8e75d41c58892ccc5d4ad61c4da476684c1c83.

  The reproducer test can be seen/fetched at:
   - https://github.com/clebergnu/qemu/commit/c778e28c24030c4a36548b714293b319f4bf18df

  And test results from Travis CI can be seen at:
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669

  For convenience purposes, here's qemu-system-aarch64 launching and hanging on the first "stop":
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3634
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3664

  And here's qemu-system-arm hanging the very same way:
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3799
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3819

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1829779/+subscriptions

[Qemu-devel] [Bug 1829779] Re: qemu-system-arm and qemu-system-aarch64 QMP hangs after kernel boots

[Cleber Rosa]

I have an update on this.  Eric and myself attempted to zero in the
exact cause.  A few things we discovered:

 1) It has nothing to do with having a kernel running
 2) It has to do with having a chardev that is a server socket.  This
    test produces command line arguments such as:

   -chardev socket,id=console,path=<path>.sock,server,nowait \
   -serial chardev:console

 3) It doesn't seem to have a connection to the test infrastructure code
    (python/qemu/qmp/*), as a I made a number of experiments which
    yielded no differences in behavior.

So, the reproducer given at:

https://github.com/clebergnu/qemu/commit/c778e28c24030c4a36548b714293b319f4bf18df

Continues to be be valid (and continues to be limited to arm and aarch64).
Now, after a number of experiments, the following was found to be a 100%
reproducible *workaround*:

https://github.com/clebergnu/qemu/commit/e1713f3b91972ad57c089f276c54db3f3fa63423

That basically shutdowns the *console* socket before proceeding with further QMP
interaction.  The effectiveness of the workaround can be seen here:

 aarch64 command line:
  - https://travis-ci.org/clebergnu/qemu/jobs/535459499#L3633
 aarch64 QMP interaction:
  - https://travis-ci.org/clebergnu/qemu/jobs/535459499#L3663

 arm command line:
  - https://travis-ci.org/clebergnu/qemu/jobs/535459499#L3747
 arm QMP interaction:
  - https://travis-ci.org/clebergnu/qemu/jobs/535459499#L3767

I hope this provides a few more hints into the real issue.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1829779

Title:
  qemu-system-arm and qemu-system-aarch64 QMP hangs after kernel boots

Status in QEMU:
  New

Bug description:
  After booting a Linux kernel on both arm and aarch64, the QMP sockets gets unresponsive. Initially, this was thought to be limited to "quit" commands, but it reproduced with others (such as in this
  reproducer).  This is a partial log output:

     >>> {'execute': 'qmp_capabilities'}
     <<< {'return': {}}
     Booting Linux on physical CPU 0x0000000000 [0x410fd034]
     Linux version 4.18.16-300.fc29.aarch64 (mockbuild@buildvm-aarch64-02.arm.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Sat Oct 20 23:12:22 UTC 2018
     ...
     Policy zone: DMA32
     Kernel command line: printk.time=0 console=ttyAMA0
     >>> {'execute': 'stop'}
     <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470173}, 'event': 'STOP'}
     <<< {'return': {}}
     >>> {'execute': 'cont'}
     <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470849}, 'event': 'RESUME'}
     <<< {'return': {}}
     >>> {'execute': 'stop'}

  Sometimes it takes just the first "stop" command.  Overall, I was able
  to reproduce 100% of times when applied on top of
  6d8e75d41c58892ccc5d4ad61c4da476684c1c83.

  The reproducer test can be seen/fetched at:
   - https://github.com/clebergnu/qemu/commit/c778e28c24030c4a36548b714293b319f4bf18df

  And test results from Travis CI can be seen at:
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669

  For convenience purposes, here's qemu-system-aarch64 launching and hanging on the first "stop":
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3634
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3664

  And here's qemu-system-arm hanging the very same way:
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3799
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3819

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1829779/+subscriptions

[Qemu-devel] [Bug 1829779] Re: qemu-system-arm and qemu-system-aarch64 QMP hangs after kernel boots

[Philippe Mathieu-Daudé]

** Changed in: qemu
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1829779

Title:
  qemu-system-arm and qemu-system-aarch64 QMP hangs after kernel boots

Status in QEMU:
  Confirmed

Bug description:
  After booting a Linux kernel on both arm and aarch64, the QMP sockets gets unresponsive. Initially, this was thought to be limited to "quit" commands, but it reproduced with others (such as in this
  reproducer).  This is a partial log output:

     >>> {'execute': 'qmp_capabilities'}
     <<< {'return': {}}
     Booting Linux on physical CPU 0x0000000000 [0x410fd034]
     Linux version 4.18.16-300.fc29.aarch64 (mockbuild@buildvm-aarch64-02.arm.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Sat Oct 20 23:12:22 UTC 2018
     ...
     Policy zone: DMA32
     Kernel command line: printk.time=0 console=ttyAMA0
     >>> {'execute': 'stop'}
     <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470173}, 'event': 'STOP'}
     <<< {'return': {}}
     >>> {'execute': 'cont'}
     <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470849}, 'event': 'RESUME'}
     <<< {'return': {}}
     >>> {'execute': 'stop'}

  Sometimes it takes just the first "stop" command.  Overall, I was able
  to reproduce 100% of times when applied on top of
  6d8e75d41c58892ccc5d4ad61c4da476684c1c83.

  The reproducer test can be seen/fetched at:
   - https://github.com/clebergnu/qemu/commit/c778e28c24030c4a36548b714293b319f4bf18df

  And test results from Travis CI can be seen at:
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669

  For convenience purposes, here's qemu-system-aarch64 launching and hanging on the first "stop":
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3634
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3664

  And here's qemu-system-arm hanging the very same way:
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3799
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3819

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1829779/+subscriptions

[Bug 1829779] Re: qemu-system-arm and qemu-system-aarch64 QMP hangs after kernel boots

[Thomas Huth]

A patch for this bug has been merged here:
https://git.qemu.org/?p=qemu.git;a=commitdiff;h=085809670201c6d3a33e3
... can we close this ticket now?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1829779

Title:
  qemu-system-arm and qemu-system-aarch64 QMP hangs after kernel boots

Status in QEMU:
  Confirmed

Bug description:
  After booting a Linux kernel on both arm and aarch64, the QMP sockets gets unresponsive. Initially, this was thought to be limited to "quit" commands, but it reproduced with others (such as in this
  reproducer).  This is a partial log output:

     >>> {'execute': 'qmp_capabilities'}
     <<< {'return': {}}
     Booting Linux on physical CPU 0x0000000000 [0x410fd034]
     Linux version 4.18.16-300.fc29.aarch64 (mockbuild@buildvm-aarch64-02.arm.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Sat Oct 20 23:12:22 UTC 2018
     ...
     Policy zone: DMA32
     Kernel command line: printk.time=0 console=ttyAMA0
     >>> {'execute': 'stop'}
     <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470173}, 'event': 'STOP'}
     <<< {'return': {}}
     >>> {'execute': 'cont'}
     <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470849}, 'event': 'RESUME'}
     <<< {'return': {}}
     >>> {'execute': 'stop'}

  Sometimes it takes just the first "stop" command.  Overall, I was able
  to reproduce 100% of times when applied on top of
  6d8e75d41c58892ccc5d4ad61c4da476684c1c83.

  The reproducer test can be seen/fetched at:
   - https://github.com/clebergnu/qemu/commit/c778e28c24030c4a36548b714293b319f4bf18df

  And test results from Travis CI can be seen at:
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669

  For convenience purposes, here's qemu-system-aarch64 launching and hanging on the first "stop":
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3634
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3664

  And here's qemu-system-arm hanging the very same way:
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3799
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3819

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1829779/+subscriptions

[Bug 1829779] Re: qemu-system-arm and qemu-system-aarch64 QMP hangs after kernel boots

[Laurent Vivier]

** Changed in: qemu
       Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1829779

Title:
  qemu-system-arm and qemu-system-aarch64 QMP hangs after kernel boots

Status in QEMU:
  Fix Released

Bug description:
  After booting a Linux kernel on both arm and aarch64, the QMP sockets gets unresponsive. Initially, this was thought to be limited to "quit" commands, but it reproduced with others (such as in this
  reproducer).  This is a partial log output:

     >>> {'execute': 'qmp_capabilities'}
     <<< {'return': {}}
     Booting Linux on physical CPU 0x0000000000 [0x410fd034]
     Linux version 4.18.16-300.fc29.aarch64 (mockbuild@buildvm-aarch64-02.arm.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Sat Oct 20 23:12:22 UTC 2018
     ...
     Policy zone: DMA32
     Kernel command line: printk.time=0 console=ttyAMA0
     >>> {'execute': 'stop'}
     <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470173}, 'event': 'STOP'}
     <<< {'return': {}}
     >>> {'execute': 'cont'}
     <<< {'timestamp': {'seconds': 1558370331, 'microseconds': 470849}, 'event': 'RESUME'}
     <<< {'return': {}}
     >>> {'execute': 'stop'}

  Sometimes it takes just the first "stop" command.  Overall, I was able
  to reproduce 100% of times when applied on top of
  6d8e75d41c58892ccc5d4ad61c4da476684c1c83.

  The reproducer test can be seen/fetched at:
   - https://github.com/clebergnu/qemu/commit/c778e28c24030c4a36548b714293b319f4bf18df

  And test results from Travis CI can be seen at:
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669

  For convenience purposes, here's qemu-system-aarch64 launching and hanging on the first "stop":
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3634
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3664

  And here's qemu-system-arm hanging the very same way:
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3799
   - https://travis-ci.org/clebergnu/qemu/jobs/534915669#L3819

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1829779/+subscriptions