From: Pavel Fedin <p.fedin@samsung.com> To: 'Andre Przywara' <andre.przywara@arm.com>, marc.zyngier@arm.com, christoffer.dall@linaro.org Cc: eric.auger@linaro.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org Subject: RE: [PATCH v3 08/16] KVM: arm64: handle ITS related GICv3 redistributor registers Date: Thu, 22 Oct 2015 18:46:18 +0300 [thread overview] Message-ID: <015d01d10ce0$cb7009a0$62501ce0$@samsung.com> (raw) In-Reply-To: <1444229726-31559-9-git-send-email-andre.przywara@arm.com> Hello! During my work on live migration i found a big bug in your implementation. > -----Original Message----- > From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On Behalf Of Andre Przywara > Sent: Wednesday, October 07, 2015 5:55 PM > To: marc.zyngier@arm.com; christoffer.dall@linaro.org > Cc: eric.auger@linaro.org; p.fedin@samsung.com; kvmarm@lists.cs.columbia.edu; linux-arm- > kernel@lists.infradead.org; kvm@vger.kernel.org > Subject: [PATCH v3 08/16] KVM: arm64: handle ITS related GICv3 redistributor registers > > In the GICv3 redistributor there are the PENDBASER and PROPBASER > registers which we did not emulate so far, as they only make sense > when having an ITS. In preparation for that emulate those MMIO > accesses by storing the 64-bit data written into it into a variable > which we later read in the ITS emulation. > > Signed-off-by: Andre Przywara <andre.przywara@arm.com> > --- > Changelog v2..v3: > - rename vgic_handle_base_register to vgic_reg64_access() > > include/kvm/arm_vgic.h | 8 ++++++++ > virt/kvm/arm/vgic-v3-emul.c | 44 ++++++++++++++++++++++++++++++++++++++++++++ > virt/kvm/arm/vgic.c | 31 +++++++++++++++++++++++++++++++ > virt/kvm/arm/vgic.h | 2 ++ > 4 files changed, 85 insertions(+) > > diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h > index 067ad09..06c33bc 100644 > --- a/include/kvm/arm_vgic.h > +++ b/include/kvm/arm_vgic.h > @@ -272,6 +272,14 @@ struct vgic_dist { > /* Virtual irq to hwirq mapping */ > spinlock_t irq_phys_map_lock; > struct list_head irq_phys_map_list; > + > + /* Address of LPI configuration table shared by all redistributors */ > + u64 propbaser; > + > + /* Addresses of LPI pending tables per redistributor */ > + u64 *pendbaser; > + > + bool lpis_enabled; > }; > > struct vgic_v2_cpu_if { > diff --git a/virt/kvm/arm/vgic-v3-emul.c b/virt/kvm/arm/vgic-v3-emul.c > index a8cf669..6939f7c 100644 > --- a/virt/kvm/arm/vgic-v3-emul.c > +++ b/virt/kvm/arm/vgic-v3-emul.c > @@ -651,6 +651,38 @@ static bool handle_mmio_cfg_reg_redist(struct kvm_vcpu *vcpu, > return vgic_handle_cfg_reg(reg, mmio, offset); > } > > +/* We don't trigger any actions here, just store the register value */ > +static bool handle_mmio_propbaser_redist(struct kvm_vcpu *vcpu, > + struct kvm_exit_mmio *mmio, > + phys_addr_t offset) > +{ > + struct vgic_dist *dist = &vcpu->kvm->arch.vgic; > + int mode = ACCESS_READ_VALUE; > + > + /* Storing a value with LPIs already enabled is undefined */ > + mode |= dist->lpis_enabled ? ACCESS_WRITE_IGNORED : ACCESS_WRITE_VALUE; > + vgic_reg64_access(mmio, offset, &dist->propbaser, mode); > + > + return false; > +} > + > +/* We don't trigger any actions here, just store the register value */ > +static bool handle_mmio_pendbaser_redist(struct kvm_vcpu *vcpu, > + struct kvm_exit_mmio *mmio, > + phys_addr_t offset) > +{ > + struct kvm_vcpu *rdvcpu = mmio->private; > + struct vgic_dist *dist = &vcpu->kvm->arch.vgic; > + int mode = ACCESS_READ_VALUE; > + > + /* Storing a value with LPIs already enabled is undefined */ > + mode |= dist->lpis_enabled ? ACCESS_WRITE_IGNORED : ACCESS_WRITE_VALUE; Here you store lpis_enabled globally, and this is plain wrong. Linux kernel separately programs PENDBASER and enables LPIs on every CPU. Therefore, after CPU #0 is initialized (this happens much earlier than everything else), dist->lpis_enabled is set to true, and subsequent PROPBASER writes, even for different redistributors, will be ignored. As a result, you'll get dist->pendbaser[n] == NULL forever, where n > 0. And your its_sync_lpi_pending_table() actually reads some garbage from physical address 0 of the guest. Attempts to write data to that region silently corrupts random qemu data during migration, that's how i discovered it. > + vgic_reg64_access(mmio, offset, > + &dist->pendbaser[rdvcpu->vcpu_id], mode); > + > + return false; > +} > + > #define SGI_base(x) ((x) + SZ_64K) > > static const struct vgic_io_range vgic_redist_ranges[] = { > @@ -679,6 +711,18 @@ static const struct vgic_io_range vgic_redist_ranges[] = { > .handle_mmio = handle_mmio_raz_wi, > }, > { > + .base = GICR_PENDBASER, > + .len = 0x08, > + .bits_per_irq = 0, > + .handle_mmio = handle_mmio_pendbaser_redist, > + }, > + { > + .base = GICR_PROPBASER, > + .len = 0x08, > + .bits_per_irq = 0, > + .handle_mmio = handle_mmio_propbaser_redist, > + }, > + { > .base = GICR_IDREGS, > .len = 0x30, > .bits_per_irq = 0, > diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c > index 4219f22..11bf692 100644 > --- a/virt/kvm/arm/vgic.c > +++ b/virt/kvm/arm/vgic.c > @@ -471,6 +471,37 @@ void vgic_reg_access(struct kvm_exit_mmio *mmio, u32 *reg, > } > } > > +/* handle a 64-bit register access */ > +void vgic_reg64_access(struct kvm_exit_mmio *mmio, phys_addr_t offset, > + u64 *basereg, int mode) > +{ > + u32 reg; > + u64 breg; > + > + switch (offset & ~3) { > + case 0x00: > + breg = *basereg; > + reg = lower_32_bits(breg); > + vgic_reg_access(mmio, ®, offset & 3, mode); > + if (mmio->is_write && (mode & ACCESS_WRITE_VALUE)) { > + breg &= GENMASK_ULL(63, 32); > + breg |= reg; > + *basereg = breg; > + } > + break; > + case 0x04: > + breg = *basereg; > + reg = upper_32_bits(breg); > + vgic_reg_access(mmio, ®, offset & 3, mode); > + if (mmio->is_write && (mode & ACCESS_WRITE_VALUE)) { > + breg = lower_32_bits(breg); > + breg |= (u64)reg << 32; > + *basereg = breg; > + } > + break; > + } > +} > + > bool handle_mmio_raz_wi(struct kvm_vcpu *vcpu, struct kvm_exit_mmio *mmio, > phys_addr_t offset) > { > diff --git a/virt/kvm/arm/vgic.h b/virt/kvm/arm/vgic.h > index a093f5c..104f780 100644 > --- a/virt/kvm/arm/vgic.h > +++ b/virt/kvm/arm/vgic.h > @@ -71,6 +71,8 @@ void vgic_reg_access(struct kvm_exit_mmio *mmio, u32 *reg, > phys_addr_t offset, int mode); > bool handle_mmio_raz_wi(struct kvm_vcpu *vcpu, struct kvm_exit_mmio *mmio, > phys_addr_t offset); > +void vgic_reg64_access(struct kvm_exit_mmio *mmio, phys_addr_t offset, > + u64 *basereg, int mode); > > static inline > u32 mmio_data_read(struct kvm_exit_mmio *mmio, u32 mask) > -- > 2.5.1 > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html Kind regards, Pavel Fedin Expert Engineer Samsung Electronics Research center Russia
WARNING: multiple messages have this Message-ID (diff)
From: p.fedin@samsung.com (Pavel Fedin) To: linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 08/16] KVM: arm64: handle ITS related GICv3 redistributor registers Date: Thu, 22 Oct 2015 18:46:18 +0300 [thread overview] Message-ID: <015d01d10ce0$cb7009a0$62501ce0$@samsung.com> (raw) In-Reply-To: <1444229726-31559-9-git-send-email-andre.przywara@arm.com> Hello! During my work on live migration i found a big bug in your implementation. > -----Original Message----- > From: kvm-owner at vger.kernel.org [mailto:kvm-owner at vger.kernel.org] On Behalf Of Andre Przywara > Sent: Wednesday, October 07, 2015 5:55 PM > To: marc.zyngier at arm.com; christoffer.dall at linaro.org > Cc: eric.auger at linaro.org; p.fedin at samsung.com; kvmarm at lists.cs.columbia.edu; linux-arm- > kernel at lists.infradead.org; kvm at vger.kernel.org > Subject: [PATCH v3 08/16] KVM: arm64: handle ITS related GICv3 redistributor registers > > In the GICv3 redistributor there are the PENDBASER and PROPBASER > registers which we did not emulate so far, as they only make sense > when having an ITS. In preparation for that emulate those MMIO > accesses by storing the 64-bit data written into it into a variable > which we later read in the ITS emulation. > > Signed-off-by: Andre Przywara <andre.przywara@arm.com> > --- > Changelog v2..v3: > - rename vgic_handle_base_register to vgic_reg64_access() > > include/kvm/arm_vgic.h | 8 ++++++++ > virt/kvm/arm/vgic-v3-emul.c | 44 ++++++++++++++++++++++++++++++++++++++++++++ > virt/kvm/arm/vgic.c | 31 +++++++++++++++++++++++++++++++ > virt/kvm/arm/vgic.h | 2 ++ > 4 files changed, 85 insertions(+) > > diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h > index 067ad09..06c33bc 100644 > --- a/include/kvm/arm_vgic.h > +++ b/include/kvm/arm_vgic.h > @@ -272,6 +272,14 @@ struct vgic_dist { > /* Virtual irq to hwirq mapping */ > spinlock_t irq_phys_map_lock; > struct list_head irq_phys_map_list; > + > + /* Address of LPI configuration table shared by all redistributors */ > + u64 propbaser; > + > + /* Addresses of LPI pending tables per redistributor */ > + u64 *pendbaser; > + > + bool lpis_enabled; > }; > > struct vgic_v2_cpu_if { > diff --git a/virt/kvm/arm/vgic-v3-emul.c b/virt/kvm/arm/vgic-v3-emul.c > index a8cf669..6939f7c 100644 > --- a/virt/kvm/arm/vgic-v3-emul.c > +++ b/virt/kvm/arm/vgic-v3-emul.c > @@ -651,6 +651,38 @@ static bool handle_mmio_cfg_reg_redist(struct kvm_vcpu *vcpu, > return vgic_handle_cfg_reg(reg, mmio, offset); > } > > +/* We don't trigger any actions here, just store the register value */ > +static bool handle_mmio_propbaser_redist(struct kvm_vcpu *vcpu, > + struct kvm_exit_mmio *mmio, > + phys_addr_t offset) > +{ > + struct vgic_dist *dist = &vcpu->kvm->arch.vgic; > + int mode = ACCESS_READ_VALUE; > + > + /* Storing a value with LPIs already enabled is undefined */ > + mode |= dist->lpis_enabled ? ACCESS_WRITE_IGNORED : ACCESS_WRITE_VALUE; > + vgic_reg64_access(mmio, offset, &dist->propbaser, mode); > + > + return false; > +} > + > +/* We don't trigger any actions here, just store the register value */ > +static bool handle_mmio_pendbaser_redist(struct kvm_vcpu *vcpu, > + struct kvm_exit_mmio *mmio, > + phys_addr_t offset) > +{ > + struct kvm_vcpu *rdvcpu = mmio->private; > + struct vgic_dist *dist = &vcpu->kvm->arch.vgic; > + int mode = ACCESS_READ_VALUE; > + > + /* Storing a value with LPIs already enabled is undefined */ > + mode |= dist->lpis_enabled ? ACCESS_WRITE_IGNORED : ACCESS_WRITE_VALUE; Here you store lpis_enabled globally, and this is plain wrong. Linux kernel separately programs PENDBASER and enables LPIs on every CPU. Therefore, after CPU #0 is initialized (this happens much earlier than everything else), dist->lpis_enabled is set to true, and subsequent PROPBASER writes, even for different redistributors, will be ignored. As a result, you'll get dist->pendbaser[n] == NULL forever, where n > 0. And your its_sync_lpi_pending_table() actually reads some garbage from physical address 0 of the guest. Attempts to write data to that region silently corrupts random qemu data during migration, that's how i discovered it. > + vgic_reg64_access(mmio, offset, > + &dist->pendbaser[rdvcpu->vcpu_id], mode); > + > + return false; > +} > + > #define SGI_base(x) ((x) + SZ_64K) > > static const struct vgic_io_range vgic_redist_ranges[] = { > @@ -679,6 +711,18 @@ static const struct vgic_io_range vgic_redist_ranges[] = { > .handle_mmio = handle_mmio_raz_wi, > }, > { > + .base = GICR_PENDBASER, > + .len = 0x08, > + .bits_per_irq = 0, > + .handle_mmio = handle_mmio_pendbaser_redist, > + }, > + { > + .base = GICR_PROPBASER, > + .len = 0x08, > + .bits_per_irq = 0, > + .handle_mmio = handle_mmio_propbaser_redist, > + }, > + { > .base = GICR_IDREGS, > .len = 0x30, > .bits_per_irq = 0, > diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c > index 4219f22..11bf692 100644 > --- a/virt/kvm/arm/vgic.c > +++ b/virt/kvm/arm/vgic.c > @@ -471,6 +471,37 @@ void vgic_reg_access(struct kvm_exit_mmio *mmio, u32 *reg, > } > } > > +/* handle a 64-bit register access */ > +void vgic_reg64_access(struct kvm_exit_mmio *mmio, phys_addr_t offset, > + u64 *basereg, int mode) > +{ > + u32 reg; > + u64 breg; > + > + switch (offset & ~3) { > + case 0x00: > + breg = *basereg; > + reg = lower_32_bits(breg); > + vgic_reg_access(mmio, ®, offset & 3, mode); > + if (mmio->is_write && (mode & ACCESS_WRITE_VALUE)) { > + breg &= GENMASK_ULL(63, 32); > + breg |= reg; > + *basereg = breg; > + } > + break; > + case 0x04: > + breg = *basereg; > + reg = upper_32_bits(breg); > + vgic_reg_access(mmio, ®, offset & 3, mode); > + if (mmio->is_write && (mode & ACCESS_WRITE_VALUE)) { > + breg = lower_32_bits(breg); > + breg |= (u64)reg << 32; > + *basereg = breg; > + } > + break; > + } > +} > + > bool handle_mmio_raz_wi(struct kvm_vcpu *vcpu, struct kvm_exit_mmio *mmio, > phys_addr_t offset) > { > diff --git a/virt/kvm/arm/vgic.h b/virt/kvm/arm/vgic.h > index a093f5c..104f780 100644 > --- a/virt/kvm/arm/vgic.h > +++ b/virt/kvm/arm/vgic.h > @@ -71,6 +71,8 @@ void vgic_reg_access(struct kvm_exit_mmio *mmio, u32 *reg, > phys_addr_t offset, int mode); > bool handle_mmio_raz_wi(struct kvm_vcpu *vcpu, struct kvm_exit_mmio *mmio, > phys_addr_t offset); > +void vgic_reg64_access(struct kvm_exit_mmio *mmio, phys_addr_t offset, > + u64 *basereg, int mode); > > static inline > u32 mmio_data_read(struct kvm_exit_mmio *mmio, u32 mask) > -- > 2.5.1 > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo at vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html Kind regards, Pavel Fedin Expert Engineer Samsung Electronics Research center Russia
next prev parent reply other threads:[~2015-10-22 15:46 UTC|newest] Thread overview: 101+ messages / expand[flat|nested] mbox.gz Atom feed top 2015-10-07 14:55 [PATCH v3 00/16] KVM: arm64: GICv3 ITS emulation Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-07 14:55 ` [PATCH v3 01/16] KVM: arm/arm64: VGIC: don't track used LRs in the distributor Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-07 14:55 ` [PATCH v3 02/16] KVM: arm/arm64: remove now unused code after stay-in-LR rework Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-07 14:55 ` [PATCH v3 03/16] KVM: extend struct kvm_msi to hold a 32-bit device ID Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-07 14:55 ` [PATCH v3 04/16] KVM: arm/arm64: add emulation model specific destroy function Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-07 14:55 ` [PATCH v3 05/16] KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-07 14:55 ` [PATCH v3 06/16] KVM: arm/arm64: make GIC frame address initialization model specific Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-07 14:55 ` [PATCH v3 07/16] KVM: arm64: Introduce new MMIO region for the ITS base address Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-07 14:55 ` [PATCH v3 08/16] KVM: arm64: handle ITS related GICv3 redistributor registers Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-22 15:46 ` Pavel Fedin [this message] 2015-10-22 15:46 ` Pavel Fedin 2015-10-22 15:55 ` Pavel Fedin 2015-10-22 15:55 ` Pavel Fedin 2015-10-07 14:55 ` [PATCH v3 09/16] KVM: arm64: introduce ITS emulation file with stub functions Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-07 14:55 ` [PATCH v3 10/16] KVM: arm64: implement basic ITS register handlers Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-07 14:55 ` [PATCH v3 11/16] KVM: arm64: add data structures to model ITS interrupt translation Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-07 14:55 ` [PATCH v3 12/16] KVM: arm64: handle pending bit for LPIs in ITS emulation Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-07 15:10 ` Pavel Fedin 2015-10-07 15:10 ` Pavel Fedin 2015-10-07 15:35 ` Marc Zyngier 2015-10-07 15:35 ` Marc Zyngier 2015-10-07 15:46 ` Pavel Fedin 2015-10-07 15:46 ` Pavel Fedin 2015-10-07 15:49 ` Marc Zyngier 2015-10-07 15:49 ` Marc Zyngier 2015-10-12 7:40 ` Pavel Fedin 2015-10-12 7:40 ` Pavel Fedin 2015-10-12 11:39 ` Pavel Fedin 2015-10-12 11:39 ` Pavel Fedin 2015-10-12 14:17 ` Andre Przywara 2015-10-12 14:17 ` Andre Przywara 2015-10-07 14:55 ` [PATCH v3 13/16] KVM: arm64: sync LPI configuration and pending tables Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-21 11:29 ` Pavel Fedin 2015-10-21 11:29 ` Pavel Fedin 2015-10-07 14:55 ` [PATCH v3 14/16] KVM: arm64: implement ITS command queue command handlers Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-14 12:26 ` Pavel Fedin 2015-10-14 12:26 ` Pavel Fedin 2015-10-07 14:55 ` [PATCH v3 15/16] KVM: arm64: implement MSI injection in ITS emulation Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-11-25 13:28 ` Pavel Fedin 2015-11-25 13:28 ` Pavel Fedin 2015-10-07 14:55 ` [PATCH v3 16/16] KVM: arm64: enable ITS emulation as a virtual MSI controller Andre Przywara 2015-10-07 14:55 ` Andre Przywara 2015-10-07 16:05 ` [PATCH v3 00/16] KVM: arm64: GICv3 ITS emulation Pavel Fedin 2015-10-07 16:05 ` Pavel Fedin 2015-10-07 16:22 ` Marc Zyngier 2015-10-07 16:22 ` Marc Zyngier 2015-10-07 18:09 ` Pavel Fedin 2015-10-07 18:09 ` Pavel Fedin 2015-10-07 19:48 ` Marc Zyngier 2015-10-07 19:48 ` Marc Zyngier 2015-10-07 19:48 ` Marc Zyngier 2015-10-08 8:41 ` Pavel Fedin 2015-10-08 8:41 ` Pavel Fedin 2015-10-10 15:37 ` Christoffer Dall 2015-10-10 15:37 ` Christoffer Dall 2015-10-12 14:12 ` Andre Przywara 2015-10-12 14:12 ` Andre Przywara 2015-10-12 15:18 ` Pavel Fedin 2015-10-12 15:18 ` Pavel Fedin 2015-10-14 8:48 ` Eric Auger 2015-10-14 8:48 ` Eric Auger 2015-10-14 8:50 ` Pavel Fedin 2015-10-14 8:50 ` Pavel Fedin 2015-10-13 15:46 ` Pavel Fedin 2015-10-13 15:46 ` Pavel Fedin 2016-03-09 11:35 ` Tomasz Nowicki 2016-03-09 11:35 ` Tomasz Nowicki 2016-03-13 18:16 ` Christoffer Dall 2016-03-13 18:16 ` Christoffer Dall 2016-03-14 11:13 ` Andre Przywara 2016-03-14 11:13 ` Andre Przywara 2016-03-14 17:29 ` Peter Maydell 2016-03-14 17:29 ` Peter Maydell 2016-03-14 17:54 ` Marc Zyngier 2016-03-14 17:54 ` Marc Zyngier 2016-03-14 18:20 ` Andre Przywara 2016-03-14 18:20 ` Andre Przywara 2016-03-14 18:36 ` Marc Zyngier 2016-03-14 18:36 ` Marc Zyngier 2016-03-18 9:40 ` Christoffer Dall 2016-03-18 9:40 ` Christoffer Dall 2016-03-18 17:14 ` Peter Maydell 2016-03-18 17:14 ` Peter Maydell 2016-03-18 9:38 ` Christoffer Dall 2016-03-18 9:38 ` Christoffer Dall
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to='015d01d10ce0$cb7009a0$62501ce0$@samsung.com' \ --to=p.fedin@samsung.com \ --cc=andre.przywara@arm.com \ --cc=christoffer.dall@linaro.org \ --cc=eric.auger@linaro.org \ --cc=kvm@vger.kernel.org \ --cc=kvmarm@lists.cs.columbia.edu \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=marc.zyngier@arm.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.