From: "Edwin Török" <edvin.torok@citrix.com>
To: <xen-devel@lists.xenproject.org>
Cc: "Edwin Török" <edvin.torok@citrix.com>,
"Doug Goldstein" <cardoe@cardoe.com>,
"Andrew Cooper" <andrew.cooper3@citrix.com>,
"George Dunlap" <george.dunlap@citrix.com>,
"Ian Jackson" <iwj@xenproject.org>,
"Jan Beulich" <jbeulich@suse.com>,
"Julien Grall" <julien@xen.org>,
"Stefano Stabellini" <sstabellini@kernel.org>,
"Wei Liu" <wl@xen.org>,
"Christian Lindig" <christian.lindig@citrix.com>,
"David Scott" <dave@recoil.org>
Subject: [PATCH v2 0/2] oxenstored build enhancements
Date: Fri, 15 Jan 2021 22:28:39 +0000 [thread overview]
Message-ID: <cover.1610748224.git.edvin.torok@citrix.com> (raw)
The patches were posted previously, this is a repost after the XSA series.
For convenience here is a tree with all patch series applied:
https://github.com/edwintorok/xen/pull/1
Edwin Török (2):
automation/: add Ubuntu:focal container
Makefile: add build-tools-oxenstored
Makefile | 6 +++
automation/build/ubuntu/focal.dockerfile | 48 ++++++++++++++++++++++++
automation/scripts/containerize | 1 +
tools/ocaml/Makefile | 8 ++++
4 files changed, 63 insertions(+)
create mode 100644 automation/build/ubuntu/focal.dockerfile
--
2.29.2
WARNING: multiple messages have this Message-ID (diff)
From: "Edwin Török" <edvin.torok@citrix.com>
To: <xen-devel@lists.xenproject.org>
Cc: "Edwin Török" <edvin.torok@citrix.com>,
"Andrew Cooper" <andrew.cooper3@citrix.com>,
"George Dunlap" <george.dunlap@citrix.com>,
"Ian Jackson" <iwj@xenproject.org>,
"Jan Beulich" <jbeulich@suse.com>,
"Julien Grall" <julien@xen.org>,
"Stefano Stabellini" <sstabellini@kernel.org>,
"Wei Liu" <wl@xen.org>, "Juergen Gross" <jgross@suse.com>,
"Christian Lindig" <christian.lindig@citrix.com>,
"David Scott" <dave@recoil.org>
Subject: [PATCH v2 0/8] tools/ocaml/xenstored: implement live update
Date: Fri, 15 Jan 2021 22:28:42 +0000 [thread overview]
Message-ID: <cover.1610748224.git.edvin.torok@citrix.com> (raw)
Message-ID: <20210115222842.IVOQHMC-UdZO91aQpSafh91STmMLTLMMkWpMe7hgiko@z> (raw)
In-Reply-To: <cover.1610748224.git.edvin.torok@citrix.com>
This is the oxenstored live update code that was shipped with the XSAs.
It reuses the existing text-based dump format to ease backporting to
old versions of Xen.
There is a separate series that implements the C xenstored migration binary
format on top of this.
Feedback welcome either on this PR or on the mailing list:
https://github.com/edwintorok/xen/pull/1
Edvin Torok (1):
tools/ocaml/xenstored: Implement live update for socket connections
Edwin Török (6):
Add workaround for xenstore-control flood issues
docs/designs/xenstore-migration.md: clarify that deletes are recursive
tools/ocaml/xenstored: only quit on SIGTERM when a reload is possible
tools/ocaml/xenstored: Automatically resume when possible
tools/ocaml/xenstored: add cooperative live-update command
tools/ocaml/xenstored: start live update process
Juergen Gross (1):
tools/xenstore: add live update command to xenstore-control
docs/designs/xenstore-migration.md | 3 +-
docs/misc/xenstore.txt | 21 ++
tools/ocaml/xenstored/connection.ml | 56 ++++-
tools/ocaml/xenstored/connections.ml | 8 +
tools/ocaml/xenstored/logging.ml | 3 +
tools/ocaml/xenstored/parse_arg.ml | 4 +
tools/ocaml/xenstored/process.ml | 139 +++++++++++
tools/ocaml/xenstored/stdext.ml | 6 +
tools/ocaml/xenstored/store.ml | 2 +-
tools/ocaml/xenstored/utils.ml | 12 +
tools/ocaml/xenstored/xenstored.ml | 126 +++++++---
tools/xenstore/Makefile | 3 +-
tools/xenstore/xenstore_control.c | 345 +++++++++++++++++++++++++--
tools/xenstore/xenstored_core.c | 7 +-
14 files changed, 681 insertions(+), 54 deletions(-)
--
2.29.2
WARNING: multiple messages have this Message-ID (diff)
From: "Edwin Török" <edvin.torok@citrix.com>
To: <xen-devel@lists.xenproject.org>
Cc: "Edwin Török" <edvin.torok@citrix.com>,
"Christian Lindig" <christian.lindig@citrix.com>,
"David Scott" <dave@recoil.org>,
"Ian Jackson" <iwj@xenproject.org>, "Wei Liu" <wl@xen.org>
Subject: [PATCH v4 0/4] tools/ocaml/xenstored: optimizations
Date: Fri, 15 Jan 2021 22:28:54 +0000 [thread overview]
Message-ID: <cover.1610748224.git.edvin.torok@citrix.com> (raw)
Message-ID: <20210115222854.3D96NbUhvxe-e97WyK1r4wfKxXrsrAk1WTXBwxAyCI0@z> (raw)
In-Reply-To: <cover.1610748224.git.edvin.torok@citrix.com>
Various speed optimizations that have already been posted,
but committing them was delayed to avoid conflicts with XSAs.
The XSAs are out, so these are ready to go now.
The switch to Maps may expose bugs in certain xenstored clients,
which previously relied on iteration order of the DIRECTORY response.
In our testing we found one such case, which turned out to be a bug
in a testsuite (it always dropped the 1st xenstore key).
For convenience here is a tree with all patch series applied:
https://github.com/edwintorok/xen/pull/1
Edwin Török (4):
tools/ocaml/xenstored: replace hand rolled GC with weak GC references
tools/ocaml/xenstored: backport find_opt/update from 4.06
tools/ocaml/xenstored: use more efficient node trees
tools/ocaml/xenstored: use more efficient tries
tools/ocaml/xenstored/connection.ml | 3 --
tools/ocaml/xenstored/connections.ml | 2 +-
tools/ocaml/xenstored/history.ml | 14 ------
tools/ocaml/xenstored/stdext.ml | 19 ++++++++
tools/ocaml/xenstored/store.ml | 51 +++++++++-----------
tools/ocaml/xenstored/symbol.ml | 70 +++++++---------------------
tools/ocaml/xenstored/symbol.mli | 22 +++------
tools/ocaml/xenstored/trie.ml | 61 +++++++++++-------------
tools/ocaml/xenstored/trie.mli | 26 +++++------
tools/ocaml/xenstored/xenstored.ml | 16 +------
10 files changed, 109 insertions(+), 175 deletions(-)
--
2.29.2
WARNING: multiple messages have this Message-ID (diff)
From: "Edwin Török" <edvin.torok@citrix.com>
To: <xen-devel@lists.xenproject.org>
Cc: "Edwin Török" <edvin.torok@citrix.com>,
"Christian Lindig" <christian.lindig@citrix.com>,
"David Scott" <dave@recoil.org>,
"Ian Jackson" <iwj@xenproject.org>, "Wei Liu" <wl@xen.org>
Subject: [PATCH v2 0/2] tools/ocaml/libs/xc: domid control
Date: Fri, 15 Jan 2021 22:28:51 +0000 [thread overview]
Message-ID: <cover.1610748224.git.edvin.torok@citrix.com> (raw)
Message-ID: <20210115222851.IWMTB4j8I3m4gpv-kYKTlPt2s2uiShvbgtGpCMbqUdg@z> (raw)
In-Reply-To: <cover.1610748224.git.edvin.torok@citrix.com>
For debugging/testing purposes we want to be able to control the domid
from the XAPI toolstack too. Xen supports this since a long time.
For convenience here is a tree with all patch series applied:
https://github.com/edwintorok/xen/pull/1
Edwin Török (2):
tools/ocaml/xenstored: trim txhistory on xenbus reconnect
tools/ocaml/libs/xc: backward compatible domid control at domain
creation time
tools/ocaml/libs/xc/xenctrl.ml | 5 ++++-
tools/ocaml/libs/xc/xenctrl.mli | 4 ++--
tools/ocaml/libs/xc/xenctrl_stubs.c | 6 +++---
tools/ocaml/xenstored/connection.ml | 2 +-
tools/ocaml/xenstored/history.ml | 4 ++++
tools/ocaml/xenstored/process.ml | 4 ++--
6 files changed, 16 insertions(+), 9 deletions(-)
--
2.29.2
WARNING: multiple messages have this Message-ID (diff)
From: "Edwin Török" <edvin.torok@citrix.com>
To: <xen-devel@lists.xenproject.org>
Cc: "Edwin Török" <edvin.torok@citrix.com>,
"Christian Lindig" <christian.lindig@citrix.com>,
"David Scott" <dave@recoil.org>,
"Ian Jackson" <iwj@xenproject.org>, "Wei Liu" <wl@xen.org>,
"Andrew Cooper" <andrew.cooper3@citrix.com>,
"George Dunlap" <george.dunlap@citrix.com>,
"Jan Beulich" <jbeulich@suse.com>,
"Julien Grall" <julien@xen.org>,
"Stefano Stabellini" <sstabellini@kernel.org>
Subject: [PATCH v1 0/4] tools/ocaml/xenstored: bugfixes
Date: Fri, 15 Jan 2021 22:29:05 +0000 [thread overview]
Message-ID: <cover.1610748224.git.edvin.torok@citrix.com> (raw)
Message-ID: <20210115222905.4dJEObOPDve5QsdRs3nNt9jH1V0xiUTL4vd1RFS9Fo4@z> (raw)
In-Reply-To: <cover.1610748224.git.edvin.torok@citrix.com>
Fix bugs discovered by structured fuzz testing.
See 'tools/ocaml/xenstored: structured fuzz testing' series for the tests.
These have been discovered while testing the XSA fixes, but they are not security related.
For convenience here is a tree with all patch series applied:
https://github.com/edwintorok/xen/pull/1
Edwin Török (4):
tools/ocaml/libs/xb: do not crash after xenbus is unmapped
tools/ocaml/xenstored: fix quota calculation for mkdir EEXIST
tools/ocaml/xenstored: reject invalid watch paths early
tools/ocaml/xenstored: mkdir conflicts were sometimes missed
tools/ocaml/libs/xb/xs_ring_stubs.c | 3 +++
tools/ocaml/xenstored/connection.ml | 5 ++---
tools/ocaml/xenstored/connections.ml | 4 +++-
tools/ocaml/xenstored/store.ml | 1 +
tools/ocaml/xenstored/transaction.ml | 2 +-
5 files changed, 10 insertions(+), 5 deletions(-)
--
2.29.2
WARNING: multiple messages have this Message-ID (diff)
From: "Edwin Török" <edvin.torok@citrix.com>
To: <xen-devel@lists.xenproject.org>
Cc: "Edwin Török" <edvin.torok@citrix.com>,
"Christian Lindig" <christian.lindig@citrix.com>,
"David Scott" <dave@recoil.org>,
"Ian Jackson" <iwj@xenproject.org>, "Wei Liu" <wl@xen.org>
Subject: [PATCH v1 0/5] tools/ocaml/xenstored: structured fuzz testing
Date: Fri, 15 Jan 2021 22:28:59 +0000 [thread overview]
Message-ID: <cover.1610748224.git.edvin.torok@citrix.com> (raw)
Message-ID: <20210115222859.axohYRzPLByZXoiYYM05sJwWxYTYNBzyBK68jLhIcME@z> (raw)
In-Reply-To: <cover.1610748224.git.edvin.torok@citrix.com>
Test that live update functionality works correctly by writing a
Quickcheck-style property test that asserts that oxenstored state
is the same whether the live-update command was run or not.
Crowbar is used for supplying random numbers to quickcheck,
which optionally works with AFL as a driver, but can also be used
without external dependencies.
This is also called structured fuzz testing, because we don't fuzz
the xenstore commands or state directly, but generate some valid looking
trees and commands based on fuzzing the parameters of these commands.
Inspired by the 'qcstm' package and presentation at ICFP 2020,
but using 'crowbar' instead, due to its integration with AFL.
This is a work in progress, transaction live updates do not work yet,
and should be split out from this series.
For convenience here is a series with all patches applied,
review welcome either there or on the mailing list:
https://github.com/edwintorok/xen/pull/1
This series hasn't been reviewed yet, but has been in development during
the XSA series.
It (and myself) kept finding new XSAs and bugs to fix, which left the test itself in
an unfinished state, it is possible to improve it to find more bugs.
Edwin Török (5):
tools/ocaml: add unit test skeleton with Dune build system
tools/ocaml/xenstored: implement the live migration binary format
tools/ocaml/xenstored: add binary dump format support
tools/ocaml/xenstored: add support for binary format
Add structured fuzzing unit test
tools/ocaml/.gitignore | 2 +
tools/ocaml/Makefile | 52 ++
tools/ocaml/dune-project | 5 +
tools/ocaml/libs/eventchn/dune | 8 +
tools/ocaml/libs/mmap/dune | 8 +
tools/ocaml/libs/xb/dune | 7 +
tools/ocaml/libs/xc/dune | 9 +
tools/ocaml/libs/xs/dune | 4 +
tools/ocaml/xen.opam | 0
tools/ocaml/xenstore.opam | 0
tools/ocaml/xenstored.opam | 18 +
tools/ocaml/xenstored/Makefile | 3 +-
tools/ocaml/xenstored/connection.ml | 63 +-
tools/ocaml/xenstored/disk.ml | 318 ++++++++
tools/ocaml/xenstored/dune | 19 +
tools/ocaml/xenstored/parse_arg.ml | 2 +-
tools/ocaml/xenstored/perms.ml | 2 +
tools/ocaml/xenstored/poll.ml | 3 +-
tools/ocaml/xenstored/process.ml | 12 +-
tools/ocaml/xenstored/store.ml | 1 +
tools/ocaml/xenstored/test/dune | 11 +
tools/ocaml/xenstored/test/generator.ml | 189 +++++
tools/ocaml/xenstored/test/model.ml | 253 ++++++
tools/ocaml/xenstored/test/old/arbitrary.ml | 261 +++++++
tools/ocaml/xenstored/test/old/gen_paths.ml | 66 ++
.../xenstored/test/old/xenstored_test.ml | 527 +++++++++++++
tools/ocaml/xenstored/test/pathtree.ml | 40 +
tools/ocaml/xenstored/test/testable.ml | 364 +++++++++
tools/ocaml/xenstored/test/types.ml | 427 ++++++++++
tools/ocaml/xenstored/test/xenctrl.ml | 48 ++
tools/ocaml/xenstored/test/xeneventchn.ml | 50 ++
tools/ocaml/xenstored/test/xenstored_test.ml | 147 ++++
tools/ocaml/xenstored/test/xs_protocol.ml | 733 ++++++++++++++++++
tools/ocaml/xenstored/transaction.ml | 119 ++-
tools/ocaml/xenstored/xenstored.ml | 205 ++++-
35 files changed, 3918 insertions(+), 58 deletions(-)
create mode 100644 tools/ocaml/.gitignore
create mode 100644 tools/ocaml/dune-project
create mode 100644 tools/ocaml/libs/eventchn/dune
create mode 100644 tools/ocaml/libs/mmap/dune
create mode 100644 tools/ocaml/libs/xb/dune
create mode 100644 tools/ocaml/libs/xc/dune
create mode 100644 tools/ocaml/libs/xs/dune
create mode 100644 tools/ocaml/xen.opam
create mode 100644 tools/ocaml/xenstore.opam
create mode 100644 tools/ocaml/xenstored.opam
create mode 100644 tools/ocaml/xenstored/dune
create mode 100644 tools/ocaml/xenstored/test/dune
create mode 100644 tools/ocaml/xenstored/test/generator.ml
create mode 100644 tools/ocaml/xenstored/test/model.ml
create mode 100644 tools/ocaml/xenstored/test/old/arbitrary.ml
create mode 100644 tools/ocaml/xenstored/test/old/gen_paths.ml
create mode 100644 tools/ocaml/xenstored/test/old/xenstored_test.ml
create mode 100644 tools/ocaml/xenstored/test/pathtree.ml
create mode 100644 tools/ocaml/xenstored/test/testable.ml
create mode 100644 tools/ocaml/xenstored/test/types.ml
create mode 100644 tools/ocaml/xenstored/test/xenctrl.ml
create mode 100644 tools/ocaml/xenstored/test/xeneventchn.ml
create mode 100644 tools/ocaml/xenstored/test/xenstored_test.ml
create mode 100644 tools/ocaml/xenstored/test/xs_protocol.ml
--
2.29.2
next reply other threads:[~2021-01-15 22:29 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-15 22:28 Edwin Török [this message]
2021-01-15 22:29 ` [PATCH v1 0/4] tools/ocaml/xenstored: bugfixes Edwin Török
2021-01-15 22:28 ` [PATCH v1 0/5] tools/ocaml/xenstored: structured fuzz testing Edwin Török
2021-01-15 22:28 ` [PATCH v4 0/4] tools/ocaml/xenstored: optimizations Edwin Török
2021-01-15 22:28 ` [PATCH v2 0/2] tools/ocaml/libs/xc: domid control Edwin Török
2021-01-15 22:28 ` [PATCH v2 0/8] tools/ocaml/xenstored: implement live update Edwin Török
2021-01-15 22:28 ` [PATCH v2 1/2] automation/: add Ubuntu:focal container Edwin Török
2021-01-15 22:28 ` [PATCH v2 2/2] Makefile: add build-tools-oxenstored Edwin Török
2021-01-15 22:28 ` [PATCH v2 1/8] tools/xenstore: add live update command to xenstore-control Edwin Török
2021-01-18 7:50 ` Jürgen Groß
2021-01-18 9:40 ` Edwin Torok
2021-01-15 22:28 ` [PATCH v2 2/8] Add workaround for xenstore-control flood issues Edwin Török
2021-01-15 22:28 ` [PATCH v2 3/8] docs/designs/xenstore-migration.md: clarify that deletes are recursive Edwin Török
2021-01-22 13:04 ` Jürgen Groß
2021-01-22 14:44 ` Edwin Torok
2021-01-15 22:28 ` [PATCH v2 4/8] tools/ocaml/xenstored: only quit on SIGTERM when a reload is possible Edwin Török
2021-01-18 7:51 ` Jürgen Groß
2021-01-18 9:28 ` Edwin Torok
2021-01-15 22:28 ` [PATCH v2 5/8] tools/ocaml/xenstored: Automatically resume when possible Edwin Török
2021-01-15 22:28 ` [PATCH v2 6/8] tools/ocaml/xenstored: add cooperative live-update command Edwin Török
2021-01-15 22:28 ` [PATCH v2 7/8] tools/ocaml/xenstored: start live update process Edwin Török
2021-01-15 22:28 ` [PATCH v2 8/8] tools/ocaml/xenstored: Implement live update for socket connections Edwin Török
2021-01-15 22:28 ` [PATCH v2 1/2] tools/ocaml/xenstored: trim txhistory on xenbus reconnect Edwin Török
2021-01-15 22:28 ` [PATCH v2 2/2] tools/ocaml/libs/xc: backward compatible domid control at domain creation time Edwin Török
2021-01-15 22:28 ` [PATCH v4 1/4] tools/ocaml/xenstored: replace hand rolled GC with weak GC references Edwin Török
2021-01-15 22:28 ` [PATCH v4 2/4] tools/ocaml/xenstored: backport find_opt/update from 4.06 Edwin Török
2021-01-15 22:28 ` [PATCH v4 3/4] tools/ocaml/xenstored: use more efficient node trees Edwin Török
2021-01-15 22:28 ` [PATCH v4 4/4] tools/ocaml/xenstored: use more efficient tries Edwin Török
2021-01-15 22:29 ` [PATCH v1 1/5] tools/ocaml: add unit test skeleton with Dune build system Edwin Török
2021-01-15 22:29 ` [PATCH v1 2/5] tools/ocaml/xenstored: implement the live migration binary format Edwin Török
2021-01-15 22:29 ` [PATCH v1 3/5] tools/ocaml/xenstored: add binary dump format support Edwin Török
2021-01-15 22:29 ` [PATCH v1 4/5] tools/ocaml/xenstored: add support for binary format Edwin Török
2021-01-15 22:29 ` [PATCH v1 5/5] Add structured fuzzing unit test Edwin Török
2021-01-15 22:29 ` [PATCH v1 1/4] tools/ocaml/libs/xb: do not crash after xenbus is unmapped Edwin Török
2021-01-15 22:29 ` [PATCH v1 2/4] tools/ocaml/xenstored: fix quota calculation for mkdir EEXIST Edwin Török
2021-01-15 22:29 ` [PATCH v1 3/4] tools/ocaml/xenstored: reject invalid watch paths early Edwin Török
2021-01-15 22:29 ` [PATCH v1 4/4] tools/ocaml/xenstored: mkdir conflicts were sometimes missed Edwin Török
2021-01-21 11:15 ` [PATCH v4 0/4] tools/ocaml/xenstored: optimizations Christian Lindig
2021-01-21 11:16 ` [PATCH v2 0/2] tools/ocaml/libs/xc: domid control Christian Lindig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1610748224.git.edvin.torok@citrix.com \
--to=edvin.torok@citrix.com \
--cc=andrew.cooper3@citrix.com \
--cc=cardoe@cardoe.com \
--cc=christian.lindig@citrix.com \
--cc=dave@recoil.org \
--cc=george.dunlap@citrix.com \
--cc=iwj@xenproject.org \
--cc=jbeulich@suse.com \
--cc=julien@xen.org \
--cc=sstabellini@kernel.org \
--cc=wl@xen.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.