From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F7CAC433DB for ; Fri, 15 Jan 2021 22:29:56 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E294A2082D for ; Fri, 15 Jan 2021 22:29:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E294A2082D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.68656.122936 (Exim 4.92) (envelope-from ) id 1l0XbF-0005zO-VJ; Fri, 15 Jan 2021 22:29:45 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 68656.122936; Fri, 15 Jan 2021 22:29:45 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0XbF-0005zH-S9; Fri, 15 Jan 2021 22:29:45 +0000 Received: by outflank-mailman (input) for mailman id 68656; Fri, 15 Jan 2021 22:29:44 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0XbE-0005yt-Cc for xen-devel@lists.xenproject.org; Fri, 15 Jan 2021 22:29:44 +0000 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id bd079286-63f5-49b5-988f-056593f7e08d; Fri, 15 Jan 2021 22:29:42 +0000 (UTC) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: bd079286-63f5-49b5-988f-056593f7e08d DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1610749782; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=5pQM3lIU9VscMHgCPRTFeVHeDqTEdY/MaJ0UMyKuskM=; b=ez28wBQ3YQuDtDPCoo6E51KEmRj/bXqrpYofD/G53X8PydxKLfGEpapr 5qzK6O69lpjnZnStfp/n8K8KgPvL3MVLaTHYfOxDvMy8M4rPGQbN3G5Kc YgrJBDJaRAvTlReG9tfZRhMvH0y1d9fFY7M94tH0FZeVEVNYQa8fPEoTm w=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: v3T9mzOxFsM/w1SoMYxm4UeL1Y6fS8wq1xtP8i0zMbVU0QloCamS2eELQ60D1mjE81iKFaG/gj qhyp+tvb6IsgoHNRZ5hZseQ7AlBjyA7aFdccmIB6I6ZSklq68src79VmbHaB6yaDPkGxW+aJ8k +Xd1WPmo08SA3WQGM29bHfJl3inu/MAIoYOraOnt9bMj7dzPGDeEma/URNB4/fWRDGxhn3Fvep IvgiHC8xsX9wKbQxFullRFPW+EVLbVqDdAGXKDXhsbtBEWJ98ACpcNU3zlkURNRBlSaHwnIFni D+Y= X-SBRS: 5.1 X-MesageID: 35262758 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.79,350,1602561600"; d="scan'208";a="35262758" From: =?UTF-8?q?Edwin=20T=C3=B6r=C3=B6k?= To: CC: =?UTF-8?q?Edwin=20T=C3=B6r=C3=B6k?= , "Doug Goldstein" , Andrew Cooper , George Dunlap , Ian Jackson , Jan Beulich , Julien Grall , "Stefano Stabellini" , Wei Liu , Christian Lindig , David Scott Subject: [PATCH v2 0/2] oxenstored build enhancements Date: Fri, 15 Jan 2021 22:28:39 +0000 Message-ID: X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit The patches were posted previously, this is a repost after the XSA series. For convenience here is a tree with all patch series applied: https://github.com/edwintorok/xen/pull/1 Edwin Török (2): automation/: add Ubuntu:focal container Makefile: add build-tools-oxenstored Makefile | 6 +++ automation/build/ubuntu/focal.dockerfile | 48 ++++++++++++++++++++++++ automation/scripts/containerize | 1 + tools/ocaml/Makefile | 8 ++++ 4 files changed, 63 insertions(+) create mode 100644 automation/build/ubuntu/focal.dockerfile -- 2.29.2 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47E5FC433E0 for ; Fri, 15 Jan 2021 22:30:16 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 10035221ED for ; Fri, 15 Jan 2021 22:30:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 10035221ED Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.68665.122996 (Exim 4.92) (envelope-from ) id 1l0XbZ-0006im-3n; Fri, 15 Jan 2021 22:30:05 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 68665.122996; Fri, 15 Jan 2021 22:30:05 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0XbY-0006iC-Uv; Fri, 15 Jan 2021 22:30:04 +0000 Received: by outflank-mailman (input) for mailman id 68665; Fri, 15 Jan 2021 22:30:04 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0XbY-00062b-8M for xen-devel@lists.xenproject.org; Fri, 15 Jan 2021 22:30:04 +0000 Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 7f41701a-fcbb-470c-add1-26a4471c89b8; Fri, 15 Jan 2021 22:29:50 +0000 (UTC) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 7f41701a-fcbb-470c-add1-26a4471c89b8 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1610749790; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=n1WuQbqtmDQimGVawL4xRfbF3r1NhtY+3oJY6Ga9Pxk=; b=hoPhu0u0PWQSFWTPJfAVXUnZGCf056eBBcFndzMg0CrtKa9N/rpmrUeM rmJpl/6snIQ9hMSIc5Dwr4nz32n165HidOcfLA369Ayi0TqSlBVklgnxs OOzeALxwaCK9B1xQNn2b9aiz6v6Yaedm3IGv8ICMG8hYXHwQTj7FvyjLk Q=; Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: mNUn/0Dbj8QNwPFA7G1QXiAi3eGuR7FWJYXDkgMj/IprpMNZNRvtqp8WHmbPaZPNL1q4gVJGhq r3Y3FEUdabxTZpSC9LGY+x+R5ZG85KufBpbnLTMAq2KJzZ3/zGnpxEt3VW+I4oMNbAkVhbyTVn Lyd4u9g1qBkyRiSY6GBjScu58rVjaGFEjRvobRvNwRKZTK3DiE2lSwyriAgRH9Bza8kij4l1y2 qzZcNOHwx5qC9FNmaEoE5aju9ZfA/rN9rLQBNjc1Bo3Ub5QeqeWDbQo9O0JV/JbWrr14KdKXbh 3xQ= X-SBRS: 5.1 X-MesageID: 35434451 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.79,350,1602561600"; d="scan'208";a="35434451" From: =?UTF-8?q?Edwin=20T=C3=B6r=C3=B6k?= To: CC: =?UTF-8?q?Edwin=20T=C3=B6r=C3=B6k?= , "Andrew Cooper" , George Dunlap , Ian Jackson , Jan Beulich , "Julien Grall" , Stefano Stabellini , Wei Liu , Juergen Gross , Christian Lindig , David Scott Subject: [PATCH v2 0/8] tools/ocaml/xenstored: implement live update Date: Fri, 15 Jan 2021 22:28:42 +0000 Message-ID: X-Mailer: git-send-email 2.27.0 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Message-ID: <20210115222842.IVOQHMC-UdZO91aQpSafh91STmMLTLMMkWpMe7hgiko@z> This is the oxenstored live update code that was shipped with the XSAs. It reuses the existing text-based dump format to ease backporting to old versions of Xen. There is a separate series that implements the C xenstored migration binary format on top of this. Feedback welcome either on this PR or on the mailing list: https://github.com/edwintorok/xen/pull/1 Edvin Torok (1): tools/ocaml/xenstored: Implement live update for socket connections Edwin Török (6): Add workaround for xenstore-control flood issues docs/designs/xenstore-migration.md: clarify that deletes are recursive tools/ocaml/xenstored: only quit on SIGTERM when a reload is possible tools/ocaml/xenstored: Automatically resume when possible tools/ocaml/xenstored: add cooperative live-update command tools/ocaml/xenstored: start live update process Juergen Gross (1): tools/xenstore: add live update command to xenstore-control docs/designs/xenstore-migration.md | 3 +- docs/misc/xenstore.txt | 21 ++ tools/ocaml/xenstored/connection.ml | 56 ++++- tools/ocaml/xenstored/connections.ml | 8 + tools/ocaml/xenstored/logging.ml | 3 + tools/ocaml/xenstored/parse_arg.ml | 4 + tools/ocaml/xenstored/process.ml | 139 +++++++++++ tools/ocaml/xenstored/stdext.ml | 6 + tools/ocaml/xenstored/store.ml | 2 +- tools/ocaml/xenstored/utils.ml | 12 + tools/ocaml/xenstored/xenstored.ml | 126 +++++++--- tools/xenstore/Makefile | 3 +- tools/xenstore/xenstore_control.c | 345 +++++++++++++++++++++++++-- tools/xenstore/xenstored_core.c | 7 +- 14 files changed, 681 insertions(+), 54 deletions(-) -- 2.29.2 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67A75C433E0 for ; Fri, 15 Jan 2021 22:52:38 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1EF7A23128 for ; Fri, 15 Jan 2021 22:52:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1EF7A23128 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.68723.123092 (Exim 4.92) (envelope-from ) id 1l0XxG-0001YZ-4c; Fri, 15 Jan 2021 22:52:30 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 68723.123092; Fri, 15 Jan 2021 22:52:30 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0XxG-0001YR-1J; Fri, 15 Jan 2021 22:52:30 +0000 Received: by outflank-mailman (input) for mailman id 68723; Fri, 15 Jan 2021 22:52:29 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0XxF-0001Xg-8O for xen-devel@lists.xenproject.org; Fri, 15 Jan 2021 22:52:29 +0000 Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id d40a164c-905e-4ebd-8f6d-6c4a62ce12d2; Fri, 15 Jan 2021 22:52:28 +0000 (UTC) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: d40a164c-905e-4ebd-8f6d-6c4a62ce12d2 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1610751148; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=j1flMhmnhr2W72ToV4rX7ve/S0UGMy7SacdgcJmBrno=; b=VX0x0RLyek8KAKKK0svvbQShQUS/R8atGepPJKeFqUnZ5ON8Qh6jp5OA YibA2gla3pnAu8xtiH8q4IxVfQVBwZTYpBAUEl7rfeoMyqAH0e/bu+4yi jr055CG452v2YgE+K5aA7L/mOGdJbwtuIc1Hh+JgdEp8xApu0UN8aHh9L U=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: +r4GU7APRWnb5MPQdc5H2XV7LHS4DjMkO1n0izfx/JQaPAQ5Y0bh+K5YHtvgq2ItqquuuiWox9 8CuC9BDxWwKdB2uBUeATEF01KJgl5YU/Nc98xvlhV6oKm1IAcYll8wWL7A8t1rEb29iC/KgSmx opS63dGw0meT34EEG3ID4t4M9Qxj1F17Kw8cGoaib1ie8bnrMILJKNqN74w9DRIHRRvAQEEsYE Xg3epnE0Sm7ESHGJi7biMnDfnd6h9Q4vo9Je8QrKKnOK67OfU0AaWMGbpxSFqvu764uFms7qAi i5o= X-SBRS: 5.1 X-MesageID: 36512244 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.79,350,1602561600"; d="scan'208";a="36512244" From: =?UTF-8?q?Edwin=20T=C3=B6r=C3=B6k?= To: CC: =?UTF-8?q?Edwin=20T=C3=B6r=C3=B6k?= , "Christian Lindig" , David Scott , "Ian Jackson" , Wei Liu Subject: [PATCH v4 0/4] tools/ocaml/xenstored: optimizations Date: Fri, 15 Jan 2021 22:28:54 +0000 Message-ID: X-Mailer: git-send-email 2.27.0 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Message-ID: <20210115222854.3D96NbUhvxe-e97WyK1r4wfKxXrsrAk1WTXBwxAyCI0@z> Various speed optimizations that have already been posted, but committing them was delayed to avoid conflicts with XSAs. The XSAs are out, so these are ready to go now. The switch to Maps may expose bugs in certain xenstored clients, which previously relied on iteration order of the DIRECTORY response. In our testing we found one such case, which turned out to be a bug in a testsuite (it always dropped the 1st xenstore key). For convenience here is a tree with all patch series applied: https://github.com/edwintorok/xen/pull/1 Edwin Török (4): tools/ocaml/xenstored: replace hand rolled GC with weak GC references tools/ocaml/xenstored: backport find_opt/update from 4.06 tools/ocaml/xenstored: use more efficient node trees tools/ocaml/xenstored: use more efficient tries tools/ocaml/xenstored/connection.ml | 3 -- tools/ocaml/xenstored/connections.ml | 2 +- tools/ocaml/xenstored/history.ml | 14 ------ tools/ocaml/xenstored/stdext.ml | 19 ++++++++ tools/ocaml/xenstored/store.ml | 51 +++++++++----------- tools/ocaml/xenstored/symbol.ml | 70 +++++++--------------------- tools/ocaml/xenstored/symbol.mli | 22 +++------ tools/ocaml/xenstored/trie.ml | 61 +++++++++++------------- tools/ocaml/xenstored/trie.mli | 26 +++++------ tools/ocaml/xenstored/xenstored.ml | 16 +------ 10 files changed, 109 insertions(+), 175 deletions(-) -- 2.29.2 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D2F7C433DB for ; Fri, 15 Jan 2021 22:52:40 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1979E23128 for ; Fri, 15 Jan 2021 22:52:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1979E23128 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.68722.123080 (Exim 4.92) (envelope-from ) id 1l0XxE-0001X0-Sj; Fri, 15 Jan 2021 22:52:28 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 68722.123080; Fri, 15 Jan 2021 22:52:28 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0XxE-0001Wt-Pb; Fri, 15 Jan 2021 22:52:28 +0000 Received: by outflank-mailman (input) for mailman id 68722; Fri, 15 Jan 2021 22:52:28 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0XxD-0001Wj-Vz for xen-devel@lists.xenproject.org; Fri, 15 Jan 2021 22:52:28 +0000 Received: from esa3.hc3370-68.iphmx.com (unknown [216.71.145.155]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 81aee72b-5873-48f9-8ec4-18b5b71456e3; Fri, 15 Jan 2021 22:52:26 +0000 (UTC) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 81aee72b-5873-48f9-8ec4-18b5b71456e3 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1610751146; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pMliw4Us0bv6+eVRKUsauMTVdrInK1//NFHS94/G89U=; b=FapoXOkjuq9SV0AXOj/IdRJPWcF4fVI96BVDd5BZaCheAtR7LGJApDyg kdsojWteU7ljrtUmj4Q24uKytXLLr0ONTIfKrk9fjt2rEJTXGuss35AQk TYNqDHluvvrcoWZYTFzdVltqaIgYfpI+y2DQ/5v2X2+3MEwe0CyfUIonx s=; Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: H/MapdRtKDCnDRZ3UDnJSF4bj7XAnHLQcRkO/WIILBVKiec75/qf0qX0ShC8EM16qZd+Ej1xG3 y4LK9IhYvNc9ak1aBD9yMZttMBXpC+nJGpk0iVvJG8P2Rf5kkqmEXk3q4Gq+VZbkaGJDwvhFff 59qMFrUjJ2/LgyPsYOo0jzS1qn630v/qwak4MPgD4+ophOtQU2g0TmvbeNpkkfpFiYnYAvFwEj U0tZl6dr5W8yJaN4g9HZSZCZvjY7TrvrR68tUkYi1pZ2AdeLsnoaBgJPAOdNL1ceIl+SsW7kg0 1xk= X-SBRS: 5.1 X-MesageID: 35217494 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.79,350,1602561600"; d="scan'208";a="35217494" From: =?UTF-8?q?Edwin=20T=C3=B6r=C3=B6k?= To: CC: =?UTF-8?q?Edwin=20T=C3=B6r=C3=B6k?= , "Christian Lindig" , David Scott , "Ian Jackson" , Wei Liu Subject: [PATCH v2 0/2] tools/ocaml/libs/xc: domid control Date: Fri, 15 Jan 2021 22:28:51 +0000 Message-ID: X-Mailer: git-send-email 2.27.0 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Message-ID: <20210115222851.IWMTB4j8I3m4gpv-kYKTlPt2s2uiShvbgtGpCMbqUdg@z> For debugging/testing purposes we want to be able to control the domid from the XAPI toolstack too. Xen supports this since a long time. For convenience here is a tree with all patch series applied: https://github.com/edwintorok/xen/pull/1 Edwin Török (2): tools/ocaml/xenstored: trim txhistory on xenbus reconnect tools/ocaml/libs/xc: backward compatible domid control at domain creation time tools/ocaml/libs/xc/xenctrl.ml | 5 ++++- tools/ocaml/libs/xc/xenctrl.mli | 4 ++-- tools/ocaml/libs/xc/xenctrl_stubs.c | 6 +++--- tools/ocaml/xenstored/connection.ml | 2 +- tools/ocaml/xenstored/history.ml | 4 ++++ tools/ocaml/xenstored/process.ml | 4 ++-- 6 files changed, 16 insertions(+), 9 deletions(-) -- 2.29.2 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1210CC433E9 for ; Fri, 15 Jan 2021 22:59:17 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C113C23A5A for ; Fri, 15 Jan 2021 22:59:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C113C23A5A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.68791.123314 (Exim 4.92) (envelope-from ) id 1l0Y3e-0003Y8-Go; Fri, 15 Jan 2021 22:59:06 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 68791.123314; Fri, 15 Jan 2021 22:59:06 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0Y3d-0003Wy-FY; Fri, 15 Jan 2021 22:59:05 +0000 Received: by outflank-mailman (input) for mailman id 68791; Fri, 15 Jan 2021 22:59:02 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0Xz5-0001Wj-3M for xen-devel@lists.xenproject.org; Fri, 15 Jan 2021 22:54:23 +0000 Received: from esa3.hc3370-68.iphmx.com (unknown [216.71.145.155]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id a7959dc6-aee4-42b9-bae7-9e5954f77553; Fri, 15 Jan 2021 22:52:50 +0000 (UTC) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: a7959dc6-aee4-42b9-bae7-9e5954f77553 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1610751170; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=G+KPhlYkGQeeYPZr1P0VSdiQ7/luOgaO/89bcWFrKkM=; b=HbmCVM1llOu3T6XjgD/FhQX+W3SI0Q3zlpU03T21yaLKH/PmS8MPAy2b dGo6LiQmjTgY477mlruag7snZqgHikFTxt0H3hBWgl40/YliSTgFfZaBH 55LI1nkiRST12R5inQ1XcQeOLTmiEnD9/ZLVKdA0YcSFo+jvfQyr0qJmg M=; Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: v5wvKcfyw8IqcUlyRTHxevnQyBk0uySCiCW0KZQOetnFh0BUEPg2Biedn7TXeH2V2wpa4KJrb+ XPHIvYIrQXlSgCCsf3izHyOMNbmvgH7ZDSAjIdONMqEb/6IaUdPzcxvHJhVuixLng3cra7q3vv sfsQMovSIqLDZ4AfX3uctx2DbVYPX5nE1k+GN4iYFSpYPHBHBk1Mb/7DAsB55cgmzJbC6LQe5O 0eWCHoCY3xDJNUo/ICGmbQ+ECfmue1CQ4SnT5c8Mr0r3fquwGkgF4D3nl1A/o9h4FLsZuPfmpW dL8= X-SBRS: 5.1 X-MesageID: 35217516 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.79,350,1602561600"; d="scan'208";a="35217516" From: =?UTF-8?q?Edwin=20T=C3=B6r=C3=B6k?= To: CC: =?UTF-8?q?Edwin=20T=C3=B6r=C3=B6k?= , "Christian Lindig" , David Scott , "Ian Jackson" , Wei Liu , Andrew Cooper , George Dunlap , "Jan Beulich" , Julien Grall , "Stefano Stabellini" Subject: [PATCH v1 0/4] tools/ocaml/xenstored: bugfixes Date: Fri, 15 Jan 2021 22:29:05 +0000 Message-ID: X-Mailer: git-send-email 2.27.0 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Message-ID: <20210115222905.4dJEObOPDve5QsdRs3nNt9jH1V0xiUTL4vd1RFS9Fo4@z> Fix bugs discovered by structured fuzz testing. See 'tools/ocaml/xenstored: structured fuzz testing' series for the tests. These have been discovered while testing the XSA fixes, but they are not security related. For convenience here is a tree with all patch series applied: https://github.com/edwintorok/xen/pull/1 Edwin Török (4): tools/ocaml/libs/xb: do not crash after xenbus is unmapped tools/ocaml/xenstored: fix quota calculation for mkdir EEXIST tools/ocaml/xenstored: reject invalid watch paths early tools/ocaml/xenstored: mkdir conflicts were sometimes missed tools/ocaml/libs/xb/xs_ring_stubs.c | 3 +++ tools/ocaml/xenstored/connection.ml | 5 ++--- tools/ocaml/xenstored/connections.ml | 4 +++- tools/ocaml/xenstored/store.ml | 1 + tools/ocaml/xenstored/transaction.ml | 2 +- 5 files changed, 10 insertions(+), 5 deletions(-) -- 2.29.2 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0C91C433E6 for ; Fri, 15 Jan 2021 22:59:16 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B47B123128 for ; Fri, 15 Jan 2021 22:59:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B47B123128 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.68788.123294 (Exim 4.92) (envelope-from ) id 1l0Y3c-0003T0-4W; Fri, 15 Jan 2021 22:59:04 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 68788.123294; Fri, 15 Jan 2021 22:59:03 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0Y3b-0003SC-JH; Fri, 15 Jan 2021 22:59:03 +0000 Received: by outflank-mailman (input) for mailman id 68788; Fri, 15 Jan 2021 22:59:02 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0Xyq-0001Wj-2i for xen-devel@lists.xenproject.org; Fri, 15 Jan 2021 22:54:08 +0000 Received: from esa3.hc3370-68.iphmx.com (unknown [216.71.145.155]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id d7b4281f-478a-451d-b52d-3c7806537371; Fri, 15 Jan 2021 22:52:48 +0000 (UTC) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: d7b4281f-478a-451d-b52d-3c7806537371 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1610751168; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=t/wN5r497w7vQLhqSzDZyRk5eZ015ahFg35m5CK+F7I=; b=E/V9QABiSgXb+ALEGugU5/sD7m1EgBF+MIMtHIA2pdGEc2Jtpgyt2yJA Lx4S7MquQubmDpA5OgTFvBJl7cksAaftqM1bPO4OgWFLYuLo1vJcnin1l wurVAgCdx2Fa8U460RtgarTnFUdeKzxue0c3j8vviDULbsh0Z1upSpQVR o=; Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: dQZ84jtoEGpafJSkbBrECJ1pJGTpbORqbXZ1iJci+lMm3KuEXEM/6GZwM7TDwReKTe6lpmcwg3 T1EHeLNOPtzaHWIYjw/FVWEiO6gMt+tpaegSHjx4gc0bRqhE3Te1/bbviQ6qvtfU5Ia4ztu/2B 3nr0Q3/h+cHzn+Jlps0NteeAr58oxlXtAcGrJqqYZSNQMRg/L4oWLzDbWfwKWUxkd97HwX4ae4 GLtfrLQ8HnA2RslieaB3MDhUOuX56wgJBKiy2bRyTvkTcKpk7CmDKoxnD8HkAvJhgWpuH5fH7C DFs= X-SBRS: 5.1 X-MesageID: 35217504 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.79,350,1602561600"; d="scan'208";a="35217504" From: =?UTF-8?q?Edwin=20T=C3=B6r=C3=B6k?= To: CC: =?UTF-8?q?Edwin=20T=C3=B6r=C3=B6k?= , "Christian Lindig" , David Scott , "Ian Jackson" , Wei Liu Subject: [PATCH v1 0/5] tools/ocaml/xenstored: structured fuzz testing Date: Fri, 15 Jan 2021 22:28:59 +0000 Message-ID: X-Mailer: git-send-email 2.27.0 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Message-ID: <20210115222859.axohYRzPLByZXoiYYM05sJwWxYTYNBzyBK68jLhIcME@z> Test that live update functionality works correctly by writing a Quickcheck-style property test that asserts that oxenstored state is the same whether the live-update command was run or not. Crowbar is used for supplying random numbers to quickcheck, which optionally works with AFL as a driver, but can also be used without external dependencies. This is also called structured fuzz testing, because we don't fuzz the xenstore commands or state directly, but generate some valid looking trees and commands based on fuzzing the parameters of these commands. Inspired by the 'qcstm' package and presentation at ICFP 2020, but using 'crowbar' instead, due to its integration with AFL. This is a work in progress, transaction live updates do not work yet, and should be split out from this series. For convenience here is a series with all patches applied, review welcome either there or on the mailing list: https://github.com/edwintorok/xen/pull/1 This series hasn't been reviewed yet, but has been in development during the XSA series. It (and myself) kept finding new XSAs and bugs to fix, which left the test itself in an unfinished state, it is possible to improve it to find more bugs. Edwin Török (5): tools/ocaml: add unit test skeleton with Dune build system tools/ocaml/xenstored: implement the live migration binary format tools/ocaml/xenstored: add binary dump format support tools/ocaml/xenstored: add support for binary format Add structured fuzzing unit test tools/ocaml/.gitignore | 2 + tools/ocaml/Makefile | 52 ++ tools/ocaml/dune-project | 5 + tools/ocaml/libs/eventchn/dune | 8 + tools/ocaml/libs/mmap/dune | 8 + tools/ocaml/libs/xb/dune | 7 + tools/ocaml/libs/xc/dune | 9 + tools/ocaml/libs/xs/dune | 4 + tools/ocaml/xen.opam | 0 tools/ocaml/xenstore.opam | 0 tools/ocaml/xenstored.opam | 18 + tools/ocaml/xenstored/Makefile | 3 +- tools/ocaml/xenstored/connection.ml | 63 +- tools/ocaml/xenstored/disk.ml | 318 ++++++++ tools/ocaml/xenstored/dune | 19 + tools/ocaml/xenstored/parse_arg.ml | 2 +- tools/ocaml/xenstored/perms.ml | 2 + tools/ocaml/xenstored/poll.ml | 3 +- tools/ocaml/xenstored/process.ml | 12 +- tools/ocaml/xenstored/store.ml | 1 + tools/ocaml/xenstored/test/dune | 11 + tools/ocaml/xenstored/test/generator.ml | 189 +++++ tools/ocaml/xenstored/test/model.ml | 253 ++++++ tools/ocaml/xenstored/test/old/arbitrary.ml | 261 +++++++ tools/ocaml/xenstored/test/old/gen_paths.ml | 66 ++ .../xenstored/test/old/xenstored_test.ml | 527 +++++++++++++ tools/ocaml/xenstored/test/pathtree.ml | 40 + tools/ocaml/xenstored/test/testable.ml | 364 +++++++++ tools/ocaml/xenstored/test/types.ml | 427 ++++++++++ tools/ocaml/xenstored/test/xenctrl.ml | 48 ++ tools/ocaml/xenstored/test/xeneventchn.ml | 50 ++ tools/ocaml/xenstored/test/xenstored_test.ml | 147 ++++ tools/ocaml/xenstored/test/xs_protocol.ml | 733 ++++++++++++++++++ tools/ocaml/xenstored/transaction.ml | 119 ++- tools/ocaml/xenstored/xenstored.ml | 205 ++++- 35 files changed, 3918 insertions(+), 58 deletions(-) create mode 100644 tools/ocaml/.gitignore create mode 100644 tools/ocaml/dune-project create mode 100644 tools/ocaml/libs/eventchn/dune create mode 100644 tools/ocaml/libs/mmap/dune create mode 100644 tools/ocaml/libs/xb/dune create mode 100644 tools/ocaml/libs/xc/dune create mode 100644 tools/ocaml/libs/xs/dune create mode 100644 tools/ocaml/xen.opam create mode 100644 tools/ocaml/xenstore.opam create mode 100644 tools/ocaml/xenstored.opam create mode 100644 tools/ocaml/xenstored/dune create mode 100644 tools/ocaml/xenstored/test/dune create mode 100644 tools/ocaml/xenstored/test/generator.ml create mode 100644 tools/ocaml/xenstored/test/model.ml create mode 100644 tools/ocaml/xenstored/test/old/arbitrary.ml create mode 100644 tools/ocaml/xenstored/test/old/gen_paths.ml create mode 100644 tools/ocaml/xenstored/test/old/xenstored_test.ml create mode 100644 tools/ocaml/xenstored/test/pathtree.ml create mode 100644 tools/ocaml/xenstored/test/testable.ml create mode 100644 tools/ocaml/xenstored/test/types.ml create mode 100644 tools/ocaml/xenstored/test/xenctrl.ml create mode 100644 tools/ocaml/xenstored/test/xeneventchn.ml create mode 100644 tools/ocaml/xenstored/test/xenstored_test.ml create mode 100644 tools/ocaml/xenstored/test/xs_protocol.ml -- 2.29.2