[PATCH 0/3] Migration mechanism with FD

[PATCH 0/3] Migration mechanism with FD

[Oksana Vohchana]

To test migration through the file descriptor we should build and provide
a path to socket_scm_helper file. This way is inconvenient for acceptance
testing.
This series provides new functions to receive and send messages over a UNIX
socket. And adds a new migration test.

Oksana Vohchana (3):
  Adding functions _send_fds and _recv_fds
  Updates send_fd_scm function
  Acceptance test: FD migration

 python/qemu/machine.py        | 74 ++++++++++++++++++++++++-----------
 tests/acceptance/migration.py | 20 ++++++++++
 2 files changed, 72 insertions(+), 22 deletions(-)

-- 
2.21.1

[PATCH 1/3] Adding functions _send_fds and _recv_fds

[Oksana Vohchana]

It provides new possibilities to send or receive data through the Unix domain
socket file descriptor.
This is useful for obtaining a socket that belongs to a different network
namespace.

Signed-off-by: Oksana Vohchana <ovoshcha@redhat.com>
---
 python/qemu/machine.py | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/python/qemu/machine.py b/python/qemu/machine.py
index 183d8f3d38..8c5bd64795 100644
--- a/python/qemu/machine.py
+++ b/python/qemu/machine.py
@@ -24,6 +24,7 @@ import subprocess
 import shutil
 import socket
 import tempfile
+import array
 
 from . import qmp
 
@@ -155,6 +156,23 @@ class QEMUMachine(object):
         self._args.append(','.join(options))
         return self
 
+    def _recv_fds(self, sock, msglen=8192, maxfds=4096):
+        """
+        Function from https://docs.python.org/3/library/socket.html#socket.socket.recvmsg
+        """
+        fds = array.array("i")
+        msg, ancdata, flags, addr = sock.recvmsg(msglen, socket.CMSG_LEN(maxfds * fds.itemsize))
+        for cmsg_level, cmsg_type, cmsg_data in ancdata:
+            if cmsg_level == socket.SOL_SOCKET and cmsg_type == socket.SCM_RIGHTS:
+                fds.frombytes(cmsg_data[:len(cmsg_data) - (len(cmsg_data) % fds.itemsize)])
+        return msg, list(fds)
+
+    def _send_fds(self, sock, msg, fds):
+        """
+        Function from https://docs.python.org/3/library/socket.html#socket.socket.sendmsg
+        """
+        return sock.sendmsg([msg], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, array.array("i", fds))])
+
     def send_fd_scm(self, fd=None, file_path=None):
         """
         Send an fd or file_path to socket_scm_helper.
-- 
2.21.1

[PATCH 2/3] Updates send_fd_scm function

[Oksana Vohchana]

A qemu-iotest uses for FD-migration test a helper program "socket_scm_helper".
And it makes some problems if you didn't build it with a QEMU. And now we can
use new methods for the socket that allow us to send a file/socket descriptor
(with access and permissions) from one process to another.

Signed-off-by: Oksana Vohchana <ovoshcha@redhat.com>
---
 python/qemu/machine.py | 56 +++++++++++++++++++++++++-----------------
 1 file changed, 34 insertions(+), 22 deletions(-)

diff --git a/python/qemu/machine.py b/python/qemu/machine.py
index 8c5bd64795..0936b71856 100644
--- a/python/qemu/machine.py
+++ b/python/qemu/machine.py
@@ -173,19 +173,24 @@ class QEMUMachine(object):
         """
         return sock.sendmsg([msg], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, array.array("i", fds))])
 
-    def send_fd_scm(self, fd=None, file_path=None):
+    def send_fd_scm(self, fd=None, file_path=None, data=None):
         """
-        Send an fd or file_path to socket_scm_helper.
+        Can be used in two different cases.
+        Send an fd or file_path to socket_scm_helper or
+        provide data and fd to send it to the socket.
 
-        Exactly one of fd and file_path must be given.
+        Exactly one of fd and file_path must be given to the case of socket_scm_helper
         If it is file_path, the helper will open that file and pass its own fd.
+
+        To second case need adds data that include a QMP request and fd
         """
         # In iotest.py, the qmp should always use unix socket.
         assert self._qmp.is_scm_available()
-        if self._socket_scm_helper is None:
-            raise QEMUMachineError("No path to socket_scm_helper set")
-        if not os.path.exists(self._socket_scm_helper):
-            raise QEMUMachineError("%s does not exist" %
+        if data is None:
+            if self._socket_scm_helper is None:
+                raise QEMUMachineError("No path to socket_scm_helper set or data not provided")
+            if not os.path.exists(self._socket_scm_helper):
+                raise QEMUMachineError("%s does not exist" %
                                    self._socket_scm_helper)
 
         # This did not exist before 3.4, but since then it is
@@ -195,24 +200,31 @@ class QEMUMachine(object):
             if fd is not None:
                 os.set_inheritable(fd, True)
 
-        fd_param = ["%s" % self._socket_scm_helper,
-                    "%d" % self._qmp.get_sock_fd()]
+        if data is None:
+            fd_param = ["%s" % self._socket_scm_helper,
+                        "%d" % self._qmp.get_sock_fd()]
+            if file_path is not None:
+                assert fd is None
+                fd_param.append(file_path)
+            else:
+                assert fd is not None
+                fd_param.append(str(fd))
 
-        if file_path is not None:
-            assert fd is None
-            fd_param.append(file_path)
-        else:
-            assert fd is not None
-            fd_param.append(str(fd))
+            devnull = open(os.path.devnull, 'rb')
+            proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE,
+                                    stderr=subprocess.STDOUT, close_fds=False)
+            output = proc.communicate()[0]
+            if output:
+                LOG.debug(output)
 
-        devnull = open(os.path.devnull, 'rb')
-        proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE,
-                                stderr=subprocess.STDOUT, close_fds=False)
-        output = proc.communicate()[0]
-        if output:
-            LOG.debug(output)
+            return proc.returncode
 
-        return proc.returncode
+        else:
+            sock_fd = socket.fromfd(self._qmp.get_sock_fd(), socket.AF_UNIX, socket.SOCK_STREAM)
+            fds_param = [fd, self._qmp.get_sock_fd()]
+            self._send_fds(sock_fd, data, fds_param)
+            self._recv_fds(sock_fd)
+            return self
 
     @staticmethod
     def _remove_if_exists(path):
-- 
2.21.1

[PATCH 3/3] Acceptance test: FD migration

[Oksana Vohchana]

Adds a new migration test through the file descriptor.

Signed-off-by: Oksana Vohchana <ovoshcha@redhat.com>
---
 tests/acceptance/migration.py | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/tests/acceptance/migration.py b/tests/acceptance/migration.py
index a8367ca023..b96a897f3b 100644
--- a/tests/acceptance/migration.py
+++ b/tests/acceptance/migration.py
@@ -10,7 +10,10 @@
 # later.  See the COPYING file in the top-level directory.
 
 
+import os
 import tempfile
+from socket import socketpair, AF_UNIX, SOCK_STREAM
+
 from avocado_qemu import Test
 from avocado import skipUnless
 
@@ -75,3 +78,20 @@ class Migration(Test):
         """
         free_port = self._get_free_port()
         dest_uri = 'exec:nc -l localhost %u' % free_port
+
+    def test_migration_with_fd(self):
+        opaque = 'fd-migration'
+        data_to_send = b"{\"execute\": \"getfd\",  \"arguments\": {\"fdname\": \"fd-migration\"}}"
+        send_socket, recv_socket = socketpair(AF_UNIX, SOCK_STREAM)
+        fd1 = send_socket.fileno()
+        fd2 = recv_socket.fileno()
+        os.set_inheritable(fd2, True)
+
+        source_vm = self.get_vm()
+        source_vm.launch()
+        source_vm.send_fd_scm(fd=fd1, data=data_to_send)
+
+        dest_vm = self.get_vm('-incoming', 'fd:%s' % fd2)
+        dest_vm.launch()
+        source_vm.qmp('migrate', uri='fd:%s' % opaque)
+        self.assert_migration(source_vm, dest_vm)
-- 
2.21.1

Re: [PATCH 0/3] Migration mechanism with FD

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200219163344.27651-1-ovoshcha@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [PATCH 0/3] Migration mechanism with FD
Message-id: 20200219163344.27651-1-ovoshcha@redhat.com
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
5429c0a Acceptance test: FD migration
f415c4c Updates send_fd_scm function
810c13a Adding functions _send_fds and _recv_fds

=== OUTPUT BEGIN ===
1/3 Checking commit 810c13a52356 (Adding functions _send_fds and _recv_fds)
WARNING: line over 80 characters
#33: FILE: python/qemu/machine.py:161:
+        Function from https://docs.python.org/3/library/socket.html#socket.socket.recvmsg

ERROR: line over 90 characters
#36: FILE: python/qemu/machine.py:164:
+        msg, ancdata, flags, addr = sock.recvmsg(msglen, socket.CMSG_LEN(maxfds * fds.itemsize))

WARNING: line over 80 characters
#38: FILE: python/qemu/machine.py:166:
+            if cmsg_level == socket.SOL_SOCKET and cmsg_type == socket.SCM_RIGHTS:

ERROR: line over 90 characters
#39: FILE: python/qemu/machine.py:167:
+                fds.frombytes(cmsg_data[:len(cmsg_data) - (len(cmsg_data) % fds.itemsize)])

WARNING: line over 80 characters
#44: FILE: python/qemu/machine.py:172:
+        Function from https://docs.python.org/3/library/socket.html#socket.socket.sendmsg

ERROR: line over 90 characters
#46: FILE: python/qemu/machine.py:174:
+        return sock.sendmsg([msg], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, array.array("i", fds))])

total: 3 errors, 3 warnings, 30 lines checked

Patch 1/3 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

2/3 Checking commit f415c4c2bd6a (Updates send_fd_scm function)
WARNING: line over 80 characters
#32: FILE: python/qemu/machine.py:182:
+        Exactly one of fd and file_path must be given to the case of socket_scm_helper

ERROR: line over 90 characters
#45: FILE: python/qemu/machine.py:191:
+                raise QEMUMachineError("No path to socket_scm_helper set or data not provided")

WARNING: line over 80 characters
#68: FILE: python/qemu/machine.py:214:
+            proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE,

ERROR: line over 90 characters
#91: FILE: python/qemu/machine.py:223:
+            sock_fd = socket.fromfd(self._qmp.get_sock_fd(), socket.AF_UNIX, socket.SOCK_STREAM)

total: 2 errors, 2 warnings, 78 lines checked

Patch 2/3 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

3/3 Checking commit 5429c0afdc21 (Acceptance test: FD migration)
ERROR: line over 90 characters
#34: FILE: tests/acceptance/migration.py:84:
+        data_to_send = b"{\"execute\": \"getfd\",  \"arguments\": {\"fdname\": \"fd-migration\"}}"

total: 1 errors, 0 warnings, 30 lines checked

Patch 3/3 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20200219163344.27651-1-ovoshcha@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com