Xen-Devel Archive on lore.kernel.org
 help / color / Atom feed
From: Pawel Wieczorkiewicz <wipawel@amazon.de>
To: <xen-devel@lists.xen.org>, <xen-devel@lists.xenproject.org>
Cc: Pawel Wieczorkiewicz <wipawel@amazon.de>,
	wipawel@amazon.com, Ross Lagerwall <ross.lagerwall@citrix.com>,
	mpohlack@amazon.com,
	Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Subject: [Xen-devel] [PATCH 05/14] livepatch: Implement pre-|post- apply|revert hooks
Date: Wed, 21 Aug 2019 08:19:22 +0000
Message-ID: <20190821081931.90887-6-wipawel@amazon.de> (raw)
In-Reply-To: <20190821081931.90887-1-wipawel@amazon.de>

This is an implementation of 4 new livepatch module vetoing hooks,
that can be optionally supplied along with modules.
Hooks that currently exists in the livepatch mechanism aren't agile
enough and have various limitations:
* run only from within a quiescing zone
* cannot conditionally prevent applying or reverting
* do not have access to the module context
To address these limitations the following has been implemented:
1) pre-apply hook
  runs before the apply action is scheduled for execution. Its main
  purpose is to prevent from applying a hotpatch when certain
  expected conditions aren't met or when mutating actions implemented
  in the hook fail or cannot be executed.

2) post-apply hook
  runs after the apply action has been executed and quiescing zone
  exited. Its main purpose is to provide an ability to follow-up on
  actions performed by the pre- hook, when module application was
  successful or undo certain preparation steps of the pre- hook in
  case of a failure. The success/failure error code is proviVded to
  the post- hooks via the rc field of the payload structure.

3) pre-revert hook
  runs before the revert action is scheduled for execution. Its main
  purpose is to prevent from reverting a hotpatch when certain
  expected conditions aren't met or when mutating actions implemented
  in the hook fail or cannot be executed.

4) post-revert hook
  runs after the revert action has been executed and quiescing zone
  exited. Its main purpose is to perform cleanup of all previously
  executed mutating actions in order to restore the original system
  state from before the current module application.
  The success/failure error code is provided to the post- hooks via
  the rc field of the payload structure.

The replace action performs atomically the following actions:
- revert all applied modules
- apply a single replacement module.
With the vetoing hooks in place various inter-hook dependencies may
arise. Also, during the revert part of the operation certain vetoing
hooks may detect failing conditions that previously were satisfied.
That could in turn lead to situation when the revert part must be
rolled back with all the pre- and post- hooks re-applied, which again
can't be guaranteed to always succeed.
The simplest response to this complication is to disallow the replace
action completely on modules with vetoing hooks.

Signed-off-by: Pawel Wieczorkiewicz <wipawel@amazon.de>
Reviewed-by: Andra-Irina Paraschiv <andraprs@amazon.com>
Reviewed-by: Petre Eftime <epetre@amazon.com>
Reviewed-by: Martin Pohlack <mpohlack@amazon.de>
Reviewed-by: Norbert Manthey <nmanthey@amazon.de>
---
 xen/common/livepatch.c              | 179 ++++++++++++++++++++++++++++++++----
 xen/include/xen/livepatch_payload.h |  27 ++++++
 2 files changed, 189 insertions(+), 17 deletions(-)

diff --git a/xen/common/livepatch.c b/xen/common/livepatch.c
index ed5756a032..464c07ad28 100644
--- a/xen/common/livepatch.c
+++ b/xen/common/livepatch.c
@@ -28,6 +28,8 @@
 #include <asm/alternative.h>
 #include <asm/event.h>
 
+#define is_hook_enabled(hook) ({ (hook) && *(hook); })
+
 /*
  * Protects against payload_list operations and also allows only one
  * caller in schedule_work.
@@ -501,6 +503,35 @@ static int check_special_sections(const struct livepatch_elf *elf)
     return 0;
 }
 
+/*
+ * Lookup specified section and when exists assign its address to a specified hook.
+ * Perform section pointer and size validation: single hook sections must contain a
+ * single pointer only.
+ */
+#define LIVEPATCH_ASSIGN_SINGLE_HOOK(elf, hook, section_name) do {                        \
+    const struct livepatch_elf_sec *__sec = livepatch_elf_sec_by_name(elf, section_name); \
+    if ( !__sec )                                                                         \
+        break;                                                                            \
+    if ( !section_ok(elf, __sec, sizeof(*hook)) || __sec->sec->sh_size != sizeof(*hook) ) \
+        return -EINVAL;                                                                   \
+    hook = __sec->load_addr;                                                              \
+} while (0)
+
+/*
+ * Lookup specified section and when exists assign its address to a specified hook.
+ * Perform section pointer and size validation: multi hook sections must contain an
+ * array whose size must be a multiple of the array's items size.
+ */
+#define LIVEPATCH_ASSIGN_MULTI_HOOK(elf, hook, nhooks, section_name) do {                 \
+    const struct livepatch_elf_sec *__sec = livepatch_elf_sec_by_name(elf, section_name); \
+    if ( !__sec )                                                                         \
+        break;                                                                            \
+    if ( !section_ok(elf, __sec, sizeof(*hook)) )                                         \
+        return -EINVAL;                                                                   \
+    hook = __sec->load_addr;                                                              \
+    nhooks = __sec->sec->sh_size / sizeof(*hook);                                         \
+} while (0)
+
 static int prepare_payload(struct payload *payload,
                            struct livepatch_elf *elf)
 {
@@ -552,25 +583,14 @@ static int prepare_payload(struct payload *payload,
             return rc;
     }
 
-    sec = livepatch_elf_sec_by_name(elf, ".livepatch.hooks.load");
-    if ( sec )
-    {
-        if ( !section_ok(elf, sec, sizeof(*payload->load_funcs)) )
-            return -EINVAL;
-
-        payload->load_funcs = sec->load_addr;
-        payload->n_load_funcs = sec->sec->sh_size / sizeof(*payload->load_funcs);
-    }
+    LIVEPATCH_ASSIGN_MULTI_HOOK(elf, payload->load_funcs, payload->n_load_funcs, ".livepatch.hooks.load");
+    LIVEPATCH_ASSIGN_MULTI_HOOK(elf, payload->unload_funcs, payload->n_unload_funcs, ".livepatch.hooks.unload");
 
-    sec = livepatch_elf_sec_by_name(elf, ".livepatch.hooks.unload");
-    if ( sec )
-    {
-        if ( !section_ok(elf, sec, sizeof(*payload->unload_funcs)) )
-            return -EINVAL;
+    LIVEPATCH_ASSIGN_SINGLE_HOOK(elf, payload->hooks.apply.pre, ".livepatch.hooks.preapply");
+    LIVEPATCH_ASSIGN_SINGLE_HOOK(elf, payload->hooks.apply.post, ".livepatch.hooks.postapply");
+    LIVEPATCH_ASSIGN_SINGLE_HOOK(elf, payload->hooks.revert.pre, ".livepatch.hooks.prerevert");
+    LIVEPATCH_ASSIGN_SINGLE_HOOK(elf, payload->hooks.revert.post, ".livepatch.hooks.postrevert");
 
-        payload->unload_funcs = sec->load_addr;
-        payload->n_unload_funcs = sec->sec->sh_size / sizeof(*payload->unload_funcs);
-    }
     sec = livepatch_elf_sec_by_name(elf, ELF_BUILD_ID_NOTE);
     if ( sec )
     {
@@ -1217,6 +1237,39 @@ static bool_t is_work_scheduled(const struct payload *data)
     return livepatch_work.do_work && livepatch_work.data == data;
 }
 
+/*
+ * Check if payload has any of the vetoing, non-atomic hooks assigned.
+ * A vetoing, non-atmic hook may perform an operation that changes the
+ * hypervisor state and may not be guaranteed to succeed. Result of
+ * such operation may be returned and may change the livepatch workflow.
+ * Such hooks may require additional cleanup actions performed by other
+ * hooks. Thus they are not suitable for replace action.
+ */
+static inline bool_t has_payload_any_vetoing_hooks(const struct payload *payload)
+{
+    return is_hook_enabled(payload->hooks.apply.pre) ||
+           is_hook_enabled(payload->hooks.apply.post) ||
+           is_hook_enabled(payload->hooks.revert.pre) ||
+           is_hook_enabled(payload->hooks.revert.post);
+}
+
+/*
+ * Checks if any of the already applied hotpatches has any vetoing,
+ * non-atomic hooks assigned.
+ */
+static inline bool_t livepatch_applied_have_vetoing_hooks(void)
+{
+    struct payload *p;
+
+    list_for_each_entry ( p, &applied_list, applied_list )
+    {
+        if ( has_payload_any_vetoing_hooks(p) )
+            return true;
+    }
+
+    return false;
+}
+
 static int schedule_work(struct payload *data, uint32_t cmd, uint32_t timeout)
 {
     ASSERT(spin_is_locked(&payload_lock));
@@ -1317,6 +1370,7 @@ void check_for_livepatch_work(void)
     {
         struct payload *p;
         unsigned int cpus;
+        bool_t action_done = false;
 
         p = livepatch_work.data;
         if ( !get_cpu_maps() )
@@ -1369,6 +1423,7 @@ void check_for_livepatch_work(void)
             livepatch_do_action();
             /* Serialize and flush out the CPU via CPUID instruction (on x86). */
             arch_livepatch_post_action();
+            action_done = true;
             local_irq_restore(flags);
         }
 
@@ -1381,6 +1436,43 @@ void check_for_livepatch_work(void)
         /* put_cpu_maps has an barrier(). */
         put_cpu_maps();
 
+        if ( action_done )
+        {
+            switch ( livepatch_work.cmd )
+            {
+            case LIVEPATCH_ACTION_REVERT:
+                if ( is_hook_enabled(p->hooks.revert.post) )
+                {
+                    printk(XENLOG_INFO LIVEPATCH "%s: Calling post-revert hook function with rc=%d\n",
+                           p->name, p->rc);
+
+                    (*p->hooks.revert.post)(p);
+                }
+                break;
+
+            case LIVEPATCH_ACTION_APPLY:
+                if ( is_hook_enabled(p->hooks.apply.post) )
+                {
+                    printk(XENLOG_INFO LIVEPATCH "%s: Calling post-apply hook function with rc=%d\n",
+                           p->name, p->rc);
+
+                    (*p->hooks.apply.post)(p);
+                }
+                break;
+
+            case LIVEPATCH_ACTION_REPLACE:
+                if ( has_payload_any_vetoing_hooks(p) )
+                {
+                    /* It should be impossible to get here since livepatch_action() guards against that. */
+                    panic(LIVEPATCH "%s: REPLACE action is not supported on hotpatches with vetoing hooks!\n",
+                            p->name);
+                    ASSERT_UNREACHABLE();
+                }
+            default:
+                break;
+            }
+        }
+
         printk(XENLOG_INFO LIVEPATCH "%s finished %s with rc=%d\n",
                p->name, names[livepatch_work.cmd], p->rc);
     }
@@ -1516,6 +1608,21 @@ static int livepatch_action(struct xen_sysctl_livepatch_action *action)
                 rc = -EBUSY;
                 break;
             }
+
+            if ( is_hook_enabled(data->hooks.revert.pre) )
+            {
+                printk(XENLOG_INFO LIVEPATCH "%s: Calling pre-revert hook function\n", data->name);
+
+                rc = (*data->hooks.revert.pre)(data);
+                if ( rc )
+                {
+                    printk(XENLOG_ERR LIVEPATCH "%s: pre-revert hook failed (rc=%d), aborting!\n",
+                           data->name, rc);
+                    data->rc = rc;
+                    break;
+                }
+            }
+
             data->rc = -EAGAIN;
             rc = schedule_work(data, action->cmd, action->timeout);
         }
@@ -1549,6 +1656,20 @@ static int livepatch_action(struct xen_sysctl_livepatch_action *action)
                     break;
             }
 
+            if ( is_hook_enabled(data->hooks.apply.pre) )
+            {
+                printk(XENLOG_INFO LIVEPATCH "%s: Calling pre-apply hook function\n", data->name);
+
+                rc = (*data->hooks.apply.pre)(data);
+                if ( rc )
+                {
+                    printk(XENLOG_ERR LIVEPATCH "%s: pre-apply hook failed (rc=%d), aborting!\n",
+                           data->name, rc);
+                    data->rc = rc;
+                    break;
+                }
+            }
+
             data->rc = -EAGAIN;
             rc = schedule_work(data, action->cmd, action->timeout);
         }
@@ -1560,6 +1681,30 @@ static int livepatch_action(struct xen_sysctl_livepatch_action *action)
             rc = build_id_dep(data, 1 /* against hypervisor. */);
             if ( rc )
                 break;
+
+            /*
+             * REPLACE action is not supported on hotpatches with vetoing hooks.
+             * Vetoing hooks usually perform mutating actions on the system and
+             * typically exist in pairs (pre- hook doing an action and post- hook
+             * undoing the action). Coalescing all hooks from all applied modules
+             * cannot be performed without inspecting potential dependencies between
+             * the mutating hooks and hence cannot be performed automatically by
+             * the replace action. Also, the replace action cannot safely assume a
+             * successful revert of all the module with vetoing hooks. When one
+             * of the hooks fails due to not meeting certain conditions the whole
+             * replace operation must have been reverted with all previous pre- and
+             * post- hooks re-executed (which cannot be guaranteed to succeed).
+             * The simplest response to this complication is disallow replace
+             * action on modules with vetoing hooks.
+             */
+            if ( has_payload_any_vetoing_hooks(data) || livepatch_applied_have_vetoing_hooks() )
+            {
+                printk(XENLOG_ERR LIVEPATCH "%s: REPLACE action is not supported on hotpatches with vetoing hooks!\n",
+                       data->name);
+                rc = -EOPNOTSUPP;
+                break;
+            }
+
             data->rc = -EAGAIN;
             rc = schedule_work(data, action->cmd, action->timeout);
         }
diff --git a/xen/include/xen/livepatch_payload.h b/xen/include/xen/livepatch_payload.h
index 99613af2db..cd20944cc4 100644
--- a/xen/include/xen/livepatch_payload.h
+++ b/xen/include/xen/livepatch_payload.h
@@ -21,6 +21,16 @@ typedef struct payload livepatch_payload_t;
 typedef void livepatch_loadcall_t(void);
 typedef void livepatch_unloadcall_t(void);
 
+typedef int livepatch_precall_t(livepatch_payload_t *arg);
+typedef void livepatch_postcall_t(livepatch_payload_t *arg);
+
+struct livepatch_hooks {
+    struct {
+        livepatch_precall_t *const *pre;
+        livepatch_postcall_t *const *post;
+    } apply, revert;
+};
+
 struct payload {
     uint32_t state;                      /* One of the LIVEPATCH_STATE_*. */
     int32_t rc;                          /* 0 or -XEN_EXX. */
@@ -47,6 +57,7 @@ struct payload {
     struct livepatch_build_id xen_dep;   /* ELFNOTE_DESC(.livepatch.xen_depends). */
     livepatch_loadcall_t *const *load_funcs;   /* The array of funcs to call after */
     livepatch_unloadcall_t *const *unload_funcs;/* load and unload of the payload. */
+    struct livepatch_hooks hooks;        /* Pre and post hooks for apply and revert */
     unsigned int n_load_funcs;           /* Nr of the funcs to load and execute. */
     unsigned int n_unload_funcs;         /* Nr of funcs to call durung unload. */
     char name[XEN_LIVEPATCH_NAME_SIZE];  /* Name of it. */
@@ -76,6 +87,22 @@ struct payload {
      livepatch_unloadcall_t *__weak \
         const livepatch_unload_data_##_fn __section(".livepatch.hooks.unload") = _fn;
 
+#define LIVEPATCH_PREAPPLY_HOOK(_fn) \
+    livepatch_precall_t *__attribute__((weak, used)) \
+        const livepatch_preapply_data_##_fn __section(".livepatch.hooks.preapply") = _fn;
+
+#define LIVEPATCH_POSTAPPLY_HOOK(_fn) \
+    livepatch_postcall_t *__attribute__((weak, used)) \
+        const livepatch_postapply_data_##_fn __section(".livepatch.hooks.postapply") = _fn;
+
+#define LIVEPATCH_PREREVERT_HOOK(_fn) \
+    livepatch_precall_t *__attribute__((weak, used)) \
+        const livepatch_prerevert_data_##_fn __section(".livepatch.hooks.prerevert") = _fn;
+
+#define LIVEPATCH_POSTREVERT_HOOK(_fn) \
+    livepatch_postcall_t *__attribute__((weak, used)) \
+        const livepatch_postrevert_data_##_fn __section(".livepatch.hooks.postrevert") = _fn;
+
 #endif /* __XEN_LIVEPATCH_PAYLOAD_H__ */
 
 /*
-- 
2.16.5




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Ralf Herbrich
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

  parent reply index

Thread overview: 66+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-08-21  8:19 [Xen-devel] [PATCH 00/14] livepatch: new features and fixes Pawel Wieczorkiewicz
2019-08-21  8:19 ` [Xen-devel] [PATCH 01/14] livepatch: Always check hypervisor build ID upon hotpatch upload Pawel Wieczorkiewicz
2019-08-21 18:16   ` Konrad Rzeszutek Wilk
2019-08-21  8:19 ` [Xen-devel] [PATCH 02/14] livepatch: Allow to override inter-modules buildid dependency Pawel Wieczorkiewicz
2019-08-21  8:19 ` [Xen-devel] [PATCH 03/14] python: Add XC binding for Xen build ID Pawel Wieczorkiewicz
2019-08-21  8:19 ` [Xen-devel] [PATCH 04/14] livepatch: Export payload structure via livepatch_payload.h Pawel Wieczorkiewicz
2019-08-21  8:19 ` Pawel Wieczorkiewicz [this message]
2019-08-21  8:19 ` [Xen-devel] [PATCH 06/14] livepatch: Add support for apply|revert action replacement hooks Pawel Wieczorkiewicz
2019-08-21 18:31   ` Konrad Rzeszutek Wilk
2019-08-21 19:06     ` Wieczorkiewicz, Pawel
2019-08-26 14:30       ` Konrad Rzeszutek Wilk
2019-08-21  8:19 ` [Xen-devel] [PATCH 07/14] livepatch: Do not enforce ELF_LIVEPATCH_FUNC section presence Pawel Wieczorkiewicz
2019-08-21  8:19 ` [Xen-devel] [PATCH 08/14] livepatch: always print XENLOG_ERR information Pawel Wieczorkiewicz
2019-08-21  8:19 ` [Xen-devel] [PATCH 09/14] livepatch: Add per-function applied/reverted state tracking marker Pawel Wieczorkiewicz
2019-08-21 18:28   ` Konrad Rzeszutek Wilk
2019-08-21 19:00     ` Wieczorkiewicz, Pawel
2019-08-21 21:34   ` Julien Grall
2019-08-22  7:44     ` Wieczorkiewicz, Pawel
2019-08-22 10:07       ` Julien Grall
2019-08-22 10:20         ` Wieczorkiewicz, Pawel
2019-08-22 10:43           ` Julien Grall
2019-08-22 11:15             ` Wieczorkiewicz, Pawel
2019-08-22 15:02               ` Julien Grall
2019-08-22 10:29   ` Julien Grall
2019-08-22 11:02     ` Wieczorkiewicz, Pawel
2019-08-22 15:30       ` Julien Grall
2019-08-22 15:42         ` Wieczorkiewicz, Pawel
2019-08-21  8:19 ` [Xen-devel] [PATCH 10/14] livepatch: Add support for inline asm hotpatching expectations Pawel Wieczorkiewicz
2019-08-21 18:30   ` Konrad Rzeszutek Wilk
2019-08-21 19:02     ` Wieczorkiewicz, Pawel
2019-08-22 10:31   ` Julien Grall
2019-08-22 11:03     ` Wieczorkiewicz, Pawel
2019-08-21  8:19 ` [Xen-devel] [PATCH 11/14] livepatch: Add support for modules .modinfo section metadata Pawel Wieczorkiewicz
2019-08-21  8:19 ` [Xen-devel] [PATCH 12/14] livepatch: Handle arbitrary size names with the list operation Pawel Wieczorkiewicz
2019-08-21  8:19 ` [Xen-devel] [PATCH 13/14] livepatch: Add metadata runtime retrieval mechanism Pawel Wieczorkiewicz
2019-08-21  8:19 ` [Xen-devel] [PATCH 14/14] livepatch: Add python bindings for livepatch operations Pawel Wieczorkiewicz
2019-08-22 21:55   ` Marek Marczykowski-Górecki
2019-08-27  8:46 ` [Xen-devel] [PATCH v2 00/12] livepatch: new features and fixes Pawel Wieczorkiewicz
2019-08-27  8:46   ` [Xen-devel] [PATCH v2 01/12] livepatch: Always check hypervisor build ID upon hotpatch upload Pawel Wieczorkiewicz
2019-08-27  8:46   ` [Xen-devel] [PATCH v2 02/12] livepatch: Allow to override inter-modules buildid dependency Pawel Wieczorkiewicz
2019-08-27  8:46   ` [Xen-devel] [PATCH v2 03/12] livepatch: Export payload structure via livepatch_payload.h Pawel Wieczorkiewicz
2019-08-27  8:46   ` [Xen-devel] [PATCH v2 04/12] livepatch: Implement pre-|post- apply|revert hooks Pawel Wieczorkiewicz
2019-08-27  8:46   ` [Xen-devel] [PATCH v2 05/12] livepatch: Add support for apply|revert action replacement hooks Pawel Wieczorkiewicz
2019-08-27 16:58     ` Konrad Rzeszutek Wilk
2019-08-28  7:37       ` Wieczorkiewicz, Pawel
2019-08-27  8:46   ` [Xen-devel] [PATCH v2 06/12] livepatch: Do not enforce ELF_LIVEPATCH_FUNC section presence Pawel Wieczorkiewicz
2019-08-27  8:46   ` [Xen-devel] [PATCH v2 07/12] livepatch: Add per-function applied/reverted state tracking marker Pawel Wieczorkiewicz
2019-08-27  8:46   ` [Xen-devel] [PATCH v2 08/12] livepatch: Add support for inline asm hotpatching expectations Pawel Wieczorkiewicz
2019-08-29 14:34     ` Konrad Rzeszutek Wilk
2019-08-29 15:29       ` Wieczorkiewicz, Pawel
2019-08-29 15:58     ` Konrad Rzeszutek Wilk
2019-08-29 16:16       ` Wieczorkiewicz, Pawel
2019-08-29 17:49         ` Konrad Rzeszutek Wilk
2019-08-29 19:07           ` Wieczorkiewicz, Pawel
2019-08-29 20:48             ` Konrad Rzeszutek Wilk
2019-09-05 18:05     ` Konrad Rzeszutek Wilk
2019-08-27  8:46   ` [Xen-devel] [PATCH v2 09/12] livepatch: Add support for modules .modinfo section metadata Pawel Wieczorkiewicz
2019-08-27  8:46   ` [Xen-devel] [PATCH v2 10/12] livepatch: Handle arbitrary size names with the list operation Pawel Wieczorkiewicz
2019-08-27  8:46   ` [Xen-devel] [PATCH v2 11/12] livepatch: Add metadata runtime retrieval mechanism Pawel Wieczorkiewicz
2019-08-29 20:48     ` Konrad Rzeszutek Wilk
2019-08-27  8:46   ` [Xen-devel] [PATCH v2 12/12] livepatch: Add python bindings for livepatch operations Pawel Wieczorkiewicz
2019-08-28 13:21     ` Marek Marczykowski-Górecki
2019-08-29 19:23   ` [Xen-devel] [PATCH v2 00/12] livepatch: new features and fixes Konrad Rzeszutek Wilk
2019-09-05 19:13   ` Konrad Rzeszutek Wilk
2019-09-06 22:52     ` Julien Grall
2019-09-06 22:42   ` Julien Grall

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190821081931.90887-6-wipawel@amazon.de \
    --to=wipawel@amazon.de \
    --cc=konrad.wilk@oracle.com \
    --cc=mpohlack@amazon.com \
    --cc=ross.lagerwall@citrix.com \
    --cc=wipawel@amazon.com \
    --cc=xen-devel@lists.xen.org \
    --cc=xen-devel@lists.xenproject.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Xen-Devel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/xen-devel/0 xen-devel/git/0.git
	git clone --mirror https://lore.kernel.org/xen-devel/1 xen-devel/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 xen-devel xen-devel/ https://lore.kernel.org/xen-devel \
		xen-devel@lists.xenproject.org xen-devel@lists.xen.org
	public-inbox-index xen-devel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.xenproject.lists.xen-devel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git