From: Gavin Shan <gshan@redhat.com> To: Marc Zyngier <maz@kernel.org> Cc: kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, peterx@redhat.com, pbonzini@redhat.com, corbet@lwn.net, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, catalin.marinas@arm.com, will@kernel.org, shuah@kernel.org, seanjc@google.com, drjones@redhat.com, dmatlack@google.com, bgardon@google.com, ricarkol@google.com, zhenyzha@redhat.com, shan.gavin@gmail.com Subject: Re: [PATCH v1 1/5] KVM: arm64: Enable ring-based dirty memory tracking Date: Mon, 22 Aug 2022 11:58:20 +1000 [thread overview] Message-ID: <41fb5a1f-29a9-e6bb-9fab-4c83a2a8fce5@redhat.com> (raw) In-Reply-To: <87lerkwtm5.wl-maz@kernel.org> Hi Marc, On 8/19/22 6:00 PM, Marc Zyngier wrote: > On Fri, 19 Aug 2022 01:55:57 +0100, > Gavin Shan <gshan@redhat.com> wrote: >> >> The ring-based dirty memory tracking has been available and enabled >> on x86 for a while. The feature is beneficial when the number of >> dirty pages is small in a checkpointing system or live migration >> scenario. More details can be found from fb04a1eddb1a ("KVM: X86: >> Implement ring-based dirty memory tracking"). >> >> This enables the ring-based dirty memory tracking on ARM64. It's >> notable that no extra reserved ring entries are needed on ARM64 >> because the huge pages are always split into base pages when page >> dirty tracking is enabled. > > Can you please elaborate on this? Adding a per-CPU ring of course > results in extra memory allocation, so there must be a subtle > x86-specific detail that I'm not aware of... > Sure. I guess it's helpful to explain how it works in next revision. Something like below: This enables the ring-based dirty memory tracking on ARM64. The feature is enabled by CONFIG_HAVE_KVM_DIRTY_RING, detected and enabled by CONFIG_HAVE_KVM_DIRTY_RING. A ring buffer is created on every vcpu and each entry is described by 'struct kvm_dirty_gfn'. The ring buffer is pushed by host when page becomes dirty and pulled by userspace. A vcpu exit is forced when the ring buffer becomes full. The ring buffers on all vcpus can be reset by ioctl command KVM_RESET_DIRTY_RINGS. Yes, I think so. Adding a per-CPU ring results in extra memory allocation. However, it's avoiding synchronization among multiple vcpus when dirty pages happen on multiple vcpus. More discussion can be found from [1] [1] https://patchwork.kernel.org/project/kvm/patch/BL2PR08MB4812F929A2760BC40EA757CF0630@BL2PR08MB481.namprd08.prod.outlook.com/ (comment#8 from Radim Krčmář on May 3, 2016, 2:11 p.m. UTC) >> >> Signed-off-by: Gavin Shan <gshan@redhat.com> >> --- >> Documentation/virt/kvm/api.rst | 2 +- >> arch/arm64/include/uapi/asm/kvm.h | 1 + >> arch/arm64/kvm/Kconfig | 1 + >> arch/arm64/kvm/arm.c | 8 ++++++++ >> 4 files changed, 11 insertions(+), 1 deletion(-) >> >> diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst >> index abd7c32126ce..19fa1ac017ed 100644 >> --- a/Documentation/virt/kvm/api.rst >> +++ b/Documentation/virt/kvm/api.rst >> @@ -8022,7 +8022,7 @@ regardless of what has actually been exposed through the CPUID leaf. >> 8.29 KVM_CAP_DIRTY_LOG_RING >> --------------------------- >> >> -:Architectures: x86 >> +:Architectures: x86, arm64 >> :Parameters: args[0] - size of the dirty log ring >> >> KVM is capable of tracking dirty memory using ring buffers that are >> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h >> index 3bb134355874..7e04b0b8d2b2 100644 >> --- a/arch/arm64/include/uapi/asm/kvm.h >> +++ b/arch/arm64/include/uapi/asm/kvm.h >> @@ -43,6 +43,7 @@ >> #define __KVM_HAVE_VCPU_EVENTS >> >> #define KVM_COALESCED_MMIO_PAGE_OFFSET 1 >> +#define KVM_DIRTY_LOG_PAGE_OFFSET 64 > > For context, the documentation says: > > <quote> > - if KVM_CAP_DIRTY_LOG_RING is available, a number of pages at > KVM_DIRTY_LOG_PAGE_OFFSET * PAGE_SIZE. [...] > </quote> > > What is the reason for picking this particular value? > It's inherited from x86. I don't think it has to be this particular value. The value is used to distinguish the region's owners like kvm_run, KVM_PIO_PAGE_OFFSET, KVM_COALESCED_MMIO_PAGE_OFFSET, and KVM_DIRTY_LOG_PAGE_OFFSET. How about to have 2 for KVM_DIRTY_LOG_PAGE_OFFSET in next revision? The virtual area is cheap, I guess it's also nice to use x86's pattern to have 64 for KVM_DIRTY_LOG_PAGE_OFFSET. #define KVM_COALESCED_MMIO_PAGE_OFFSET 1 #define KVM_DIRTY_LOG_PAGE_OFFSET 2 >> >> #define KVM_REG_SIZE(id) \ >> (1U << (((id) & KVM_REG_SIZE_MASK) >> KVM_REG_SIZE_SHIFT)) >> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig >> index 815cc118c675..0309b2d0f2da 100644 >> --- a/arch/arm64/kvm/Kconfig >> +++ b/arch/arm64/kvm/Kconfig >> @@ -32,6 +32,7 @@ menuconfig KVM >> select KVM_VFIO >> select HAVE_KVM_EVENTFD >> select HAVE_KVM_IRQFD >> + select HAVE_KVM_DIRTY_RING >> select HAVE_KVM_MSI >> select HAVE_KVM_IRQCHIP >> select HAVE_KVM_IRQ_ROUTING >> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c >> index 986cee6fbc7f..3de6b9b39db7 100644 >> --- a/arch/arm64/kvm/arm.c >> +++ b/arch/arm64/kvm/arm.c >> @@ -866,6 +866,14 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) >> if (!ret) >> ret = 1; >> >> + /* Force vcpu exit if its dirty ring is soft-full */ >> + if (unlikely(vcpu->kvm->dirty_ring_size && >> + kvm_dirty_ring_soft_full(&vcpu->dirty_ring))) { >> + vcpu->run->exit_reason = KVM_EXIT_DIRTY_RING_FULL; >> + trace_kvm_dirty_ring_exit(vcpu); >> + ret = 0; >> + } >> + > > Why can't this be moved to kvm_vcpu_exit_request() instead? I would > also very much like the check to be made a common helper with x86. > > A seemingly approach would be to make this a request on dirty log > insertion, and avoid the whole "check the log size" on every run, > which adds pointless overhead to unsuspecting users (aka everyone). > I though of having the check in kvm_vcpu_exit_request(). The various exit reasons are prioritized. x86 gives KVM_EXIT_DIRTY_RING_FULL the highest priority and ARM64 is just to follow. I don't think it really matters. I will improve it accordingly in next revision: - Change kvm_dirty_ring_soft_full() to something as below in dirty_ring.c bool kvm_dirty_ring_soft_full(struct kvm_vcpu *vcpu) { struct kvm *kvm = vcpu->vcpu; struct kvm_dirty_ring *ring = &vcpu->dirty_ring; if (unlikely(kvm->dirty_ring_size && kvm_dirty_ring_used(ring) >= ring->soft_limit)) { vcpu->run->exit_reason = KVM_EXIT_DIRTY_RING_FULL; trace_kvm_dirty_ring_exit(vcpu); return true; } return false; } - Use the modified kvm_dirty_ring_soft_full() in kvm_vcpu_exit_request(). Userspace needs KVM_EXIT_DIRTY_RING_FULL to collect the dirty log in time. Otherwise, the dirty log in the ring buffer will be overwritten. I'm not sure if anything else I missed? Thanks, Gavin
WARNING: multiple messages have this Message-ID (diff)
From: Gavin Shan <gshan@redhat.com> To: Marc Zyngier <maz@kernel.org> Cc: kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, peterx@redhat.com, pbonzini@redhat.com, corbet@lwn.net, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, catalin.marinas@arm.com, will@kernel.org, shuah@kernel.org, seanjc@google.com, drjones@redhat.com, dmatlack@google.com, bgardon@google.com, ricarkol@google.com, zhenyzha@redhat.com, shan.gavin@gmail.com Subject: Re: [PATCH v1 1/5] KVM: arm64: Enable ring-based dirty memory tracking Date: Mon, 22 Aug 2022 11:58:20 +1000 [thread overview] Message-ID: <41fb5a1f-29a9-e6bb-9fab-4c83a2a8fce5@redhat.com> (raw) In-Reply-To: <87lerkwtm5.wl-maz@kernel.org> Hi Marc, On 8/19/22 6:00 PM, Marc Zyngier wrote: > On Fri, 19 Aug 2022 01:55:57 +0100, > Gavin Shan <gshan@redhat.com> wrote: >> >> The ring-based dirty memory tracking has been available and enabled >> on x86 for a while. The feature is beneficial when the number of >> dirty pages is small in a checkpointing system or live migration >> scenario. More details can be found from fb04a1eddb1a ("KVM: X86: >> Implement ring-based dirty memory tracking"). >> >> This enables the ring-based dirty memory tracking on ARM64. It's >> notable that no extra reserved ring entries are needed on ARM64 >> because the huge pages are always split into base pages when page >> dirty tracking is enabled. > > Can you please elaborate on this? Adding a per-CPU ring of course > results in extra memory allocation, so there must be a subtle > x86-specific detail that I'm not aware of... > Sure. I guess it's helpful to explain how it works in next revision. Something like below: This enables the ring-based dirty memory tracking on ARM64. The feature is enabled by CONFIG_HAVE_KVM_DIRTY_RING, detected and enabled by CONFIG_HAVE_KVM_DIRTY_RING. A ring buffer is created on every vcpu and each entry is described by 'struct kvm_dirty_gfn'. The ring buffer is pushed by host when page becomes dirty and pulled by userspace. A vcpu exit is forced when the ring buffer becomes full. The ring buffers on all vcpus can be reset by ioctl command KVM_RESET_DIRTY_RINGS. Yes, I think so. Adding a per-CPU ring results in extra memory allocation. However, it's avoiding synchronization among multiple vcpus when dirty pages happen on multiple vcpus. More discussion can be found from [1] [1] https://patchwork.kernel.org/project/kvm/patch/BL2PR08MB4812F929A2760BC40EA757CF0630@BL2PR08MB481.namprd08.prod.outlook.com/ (comment#8 from Radim Krčmář on May 3, 2016, 2:11 p.m. UTC) >> >> Signed-off-by: Gavin Shan <gshan@redhat.com> >> --- >> Documentation/virt/kvm/api.rst | 2 +- >> arch/arm64/include/uapi/asm/kvm.h | 1 + >> arch/arm64/kvm/Kconfig | 1 + >> arch/arm64/kvm/arm.c | 8 ++++++++ >> 4 files changed, 11 insertions(+), 1 deletion(-) >> >> diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst >> index abd7c32126ce..19fa1ac017ed 100644 >> --- a/Documentation/virt/kvm/api.rst >> +++ b/Documentation/virt/kvm/api.rst >> @@ -8022,7 +8022,7 @@ regardless of what has actually been exposed through the CPUID leaf. >> 8.29 KVM_CAP_DIRTY_LOG_RING >> --------------------------- >> >> -:Architectures: x86 >> +:Architectures: x86, arm64 >> :Parameters: args[0] - size of the dirty log ring >> >> KVM is capable of tracking dirty memory using ring buffers that are >> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h >> index 3bb134355874..7e04b0b8d2b2 100644 >> --- a/arch/arm64/include/uapi/asm/kvm.h >> +++ b/arch/arm64/include/uapi/asm/kvm.h >> @@ -43,6 +43,7 @@ >> #define __KVM_HAVE_VCPU_EVENTS >> >> #define KVM_COALESCED_MMIO_PAGE_OFFSET 1 >> +#define KVM_DIRTY_LOG_PAGE_OFFSET 64 > > For context, the documentation says: > > <quote> > - if KVM_CAP_DIRTY_LOG_RING is available, a number of pages at > KVM_DIRTY_LOG_PAGE_OFFSET * PAGE_SIZE. [...] > </quote> > > What is the reason for picking this particular value? > It's inherited from x86. I don't think it has to be this particular value. The value is used to distinguish the region's owners like kvm_run, KVM_PIO_PAGE_OFFSET, KVM_COALESCED_MMIO_PAGE_OFFSET, and KVM_DIRTY_LOG_PAGE_OFFSET. How about to have 2 for KVM_DIRTY_LOG_PAGE_OFFSET in next revision? The virtual area is cheap, I guess it's also nice to use x86's pattern to have 64 for KVM_DIRTY_LOG_PAGE_OFFSET. #define KVM_COALESCED_MMIO_PAGE_OFFSET 1 #define KVM_DIRTY_LOG_PAGE_OFFSET 2 >> >> #define KVM_REG_SIZE(id) \ >> (1U << (((id) & KVM_REG_SIZE_MASK) >> KVM_REG_SIZE_SHIFT)) >> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig >> index 815cc118c675..0309b2d0f2da 100644 >> --- a/arch/arm64/kvm/Kconfig >> +++ b/arch/arm64/kvm/Kconfig >> @@ -32,6 +32,7 @@ menuconfig KVM >> select KVM_VFIO >> select HAVE_KVM_EVENTFD >> select HAVE_KVM_IRQFD >> + select HAVE_KVM_DIRTY_RING >> select HAVE_KVM_MSI >> select HAVE_KVM_IRQCHIP >> select HAVE_KVM_IRQ_ROUTING >> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c >> index 986cee6fbc7f..3de6b9b39db7 100644 >> --- a/arch/arm64/kvm/arm.c >> +++ b/arch/arm64/kvm/arm.c >> @@ -866,6 +866,14 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) >> if (!ret) >> ret = 1; >> >> + /* Force vcpu exit if its dirty ring is soft-full */ >> + if (unlikely(vcpu->kvm->dirty_ring_size && >> + kvm_dirty_ring_soft_full(&vcpu->dirty_ring))) { >> + vcpu->run->exit_reason = KVM_EXIT_DIRTY_RING_FULL; >> + trace_kvm_dirty_ring_exit(vcpu); >> + ret = 0; >> + } >> + > > Why can't this be moved to kvm_vcpu_exit_request() instead? I would > also very much like the check to be made a common helper with x86. > > A seemingly approach would be to make this a request on dirty log > insertion, and avoid the whole "check the log size" on every run, > which adds pointless overhead to unsuspecting users (aka everyone). > I though of having the check in kvm_vcpu_exit_request(). The various exit reasons are prioritized. x86 gives KVM_EXIT_DIRTY_RING_FULL the highest priority and ARM64 is just to follow. I don't think it really matters. I will improve it accordingly in next revision: - Change kvm_dirty_ring_soft_full() to something as below in dirty_ring.c bool kvm_dirty_ring_soft_full(struct kvm_vcpu *vcpu) { struct kvm *kvm = vcpu->vcpu; struct kvm_dirty_ring *ring = &vcpu->dirty_ring; if (unlikely(kvm->dirty_ring_size && kvm_dirty_ring_used(ring) >= ring->soft_limit)) { vcpu->run->exit_reason = KVM_EXIT_DIRTY_RING_FULL; trace_kvm_dirty_ring_exit(vcpu); return true; } return false; } - Use the modified kvm_dirty_ring_soft_full() in kvm_vcpu_exit_request(). Userspace needs KVM_EXIT_DIRTY_RING_FULL to collect the dirty log in time. Otherwise, the dirty log in the ring buffer will be overwritten. I'm not sure if anything else I missed? Thanks, Gavin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2022-08-22 1:58 UTC|newest] Thread overview: 98+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-08-19 0:55 [PATCH v1 0/5] KVM: arm64: Enable ring-based dirty memory tracking Gavin Shan 2022-08-19 0:55 ` Gavin Shan 2022-08-19 0:55 ` Gavin Shan 2022-08-19 0:55 ` [PATCH v1 1/5] " Gavin Shan 2022-08-19 0:55 ` Gavin Shan 2022-08-19 0:55 ` Gavin Shan 2022-08-19 8:00 ` Marc Zyngier 2022-08-19 8:00 ` Marc Zyngier 2022-08-19 8:00 ` Marc Zyngier 2022-08-22 1:58 ` Gavin Shan [this message] 2022-08-22 1:58 ` Gavin Shan 2022-08-22 18:55 ` Peter Xu 2022-08-22 18:55 ` Peter Xu 2022-08-22 18:55 ` Peter Xu 2022-08-23 3:19 ` Gavin Shan 2022-08-23 3:19 ` Gavin Shan 2022-08-23 3:19 ` Gavin Shan 2022-08-22 21:42 ` Marc Zyngier 2022-08-22 21:42 ` Marc Zyngier 2022-08-22 21:42 ` Marc Zyngier 2022-08-23 5:22 ` Gavin Shan 2022-08-23 5:22 ` Gavin Shan 2022-08-23 5:22 ` Gavin Shan 2022-08-23 13:58 ` Peter Xu 2022-08-23 13:58 ` Peter Xu 2022-08-23 13:58 ` Peter Xu 2022-08-23 19:17 ` Marc Zyngier 2022-08-23 19:17 ` Marc Zyngier 2022-08-23 19:17 ` Marc Zyngier 2022-08-23 21:20 ` Peter Xu 2022-08-23 21:20 ` Peter Xu 2022-08-23 21:20 ` Peter Xu 2022-08-23 22:47 ` Marc Zyngier 2022-08-23 22:47 ` Marc Zyngier 2022-08-23 22:47 ` Marc Zyngier 2022-08-23 23:19 ` Peter Xu 2022-08-23 23:19 ` Peter Xu 2022-08-23 23:19 ` Peter Xu 2022-08-24 14:45 ` Marc Zyngier 2022-08-24 14:45 ` Marc Zyngier 2022-08-24 14:45 ` Marc Zyngier 2022-08-24 16:21 ` Peter Xu 2022-08-24 16:21 ` Peter Xu 2022-08-24 16:21 ` Peter Xu 2022-08-24 20:57 ` Marc Zyngier 2022-08-24 20:57 ` Marc Zyngier 2022-08-24 20:57 ` Marc Zyngier 2022-08-26 6:05 ` Gavin Shan 2022-08-26 6:05 ` Gavin Shan 2022-08-26 6:05 ` Gavin Shan 2022-08-26 10:50 ` Paolo Bonzini 2022-08-26 10:50 ` Paolo Bonzini 2022-08-26 10:50 ` Paolo Bonzini 2022-08-26 15:49 ` Marc Zyngier 2022-08-26 15:49 ` Marc Zyngier 2022-08-26 15:49 ` Marc Zyngier 2022-08-27 8:27 ` Paolo Bonzini 2022-08-27 8:27 ` Paolo Bonzini 2022-08-27 8:27 ` Paolo Bonzini 2022-08-29 10:27 ` Paolo Bonzini 2022-08-23 14:44 ` Oliver Upton 2022-08-23 14:44 ` Oliver Upton 2022-08-23 14:44 ` Oliver Upton 2022-08-23 20:35 ` Marc Zyngier 2022-08-23 20:35 ` Marc Zyngier 2022-08-23 20:35 ` Marc Zyngier 2022-08-26 10:58 ` Paolo Bonzini 2022-08-26 10:58 ` Paolo Bonzini 2022-08-26 10:58 ` Paolo Bonzini 2022-08-26 15:28 ` Marc Zyngier 2022-08-26 15:28 ` Marc Zyngier 2022-08-26 15:28 ` Marc Zyngier 2022-08-30 14:42 ` Peter Xu 2022-08-30 14:42 ` Peter Xu 2022-08-30 14:42 ` Peter Xu 2022-09-02 0:19 ` Paolo Bonzini 2022-09-02 0:19 ` Paolo Bonzini 2022-09-02 0:19 ` Paolo Bonzini 2022-08-19 0:55 ` [PATCH v1 2/5] KVM: selftests: Use host page size to map ring buffer in dirty_log_test Gavin Shan 2022-08-19 0:55 ` Gavin Shan 2022-08-19 0:55 ` Gavin Shan 2022-08-19 0:55 ` [PATCH v1 3/5] KVM: selftests: Dirty host pages " Gavin Shan 2022-08-19 0:55 ` Gavin Shan 2022-08-19 0:55 ` Gavin Shan 2022-08-19 5:28 ` Andrew Jones 2022-08-19 5:28 ` Andrew Jones 2022-08-19 5:28 ` Andrew Jones 2022-08-22 6:29 ` Gavin Shan 2022-08-22 6:29 ` Gavin Shan 2022-08-23 3:09 ` Gavin Shan 2022-08-23 3:09 ` Gavin Shan 2022-08-23 3:09 ` Gavin Shan 2022-08-19 0:56 ` [PATCH v1 4/5] KVM: selftests: Clear dirty ring states between two modes " Gavin Shan 2022-08-19 0:56 ` Gavin Shan 2022-08-19 0:56 ` Gavin Shan 2022-08-19 0:56 ` [PATCH v1 5/5] KVM: selftests: Automate choosing dirty ring size " Gavin Shan 2022-08-19 0:56 ` Gavin Shan 2022-08-19 0:56 ` Gavin Shan
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=41fb5a1f-29a9-e6bb-9fab-4c83a2a8fce5@redhat.com \ --to=gshan@redhat.com \ --cc=alexandru.elisei@arm.com \ --cc=bgardon@google.com \ --cc=catalin.marinas@arm.com \ --cc=corbet@lwn.net \ --cc=dmatlack@google.com \ --cc=drjones@redhat.com \ --cc=james.morse@arm.com \ --cc=kvm@vger.kernel.org \ --cc=kvmarm@lists.cs.columbia.edu \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-doc@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-kselftest@vger.kernel.org \ --cc=maz@kernel.org \ --cc=oliver.upton@linux.dev \ --cc=pbonzini@redhat.com \ --cc=peterx@redhat.com \ --cc=ricarkol@google.com \ --cc=seanjc@google.com \ --cc=shan.gavin@gmail.com \ --cc=shuah@kernel.org \ --cc=suzuki.poulose@arm.com \ --cc=will@kernel.org \ --cc=zhenyzha@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.