From: Nathan Chancellor <nathan@kernel.org>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Andrew Cooper <Andrew.Cooper3@citrix.com>,
Thomas Gleixner <tglx@linutronix.de>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"x86@kernel.org" <x86@kernel.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
Tim Chen <tim.c.chen@linux.intel.com>,
Josh Poimboeuf <jpoimboe@kernel.org>,
Pawan Gupta <pawan.kumar.gupta@linux.intel.com>,
Johannes Wikner <kwikner@ethz.ch>,
Alyssa Milburn <alyssa.milburn@linux.intel.com>,
Jann Horn <jannh@google.com>, "H.J. Lu" <hjl.tools@gmail.com>,
Joao Moreira <joao.moreira@intel.com>,
Joseph Nuzman <joseph.nuzman@intel.com>,
Steven Rostedt <rostedt@goodmis.org>,
Juergen Gross <jgross@suse.com>,
Masami Hiramatsu <mhiramat@kernel.org>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
K Prateek Nayak <kprateek.nayak@amd.com>,
Eric Dumazet <edumazet@google.com>,
Sean Christopherson <sean.j.christopherson@intel.com>,
Paolo Bonzini <pbonzini@redhat.com>,
kvm list <kvm@vger.kernel.org>,
Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Subject: Re: KVM vs AMD: Re: [PATCH v3 48/59] x86/retbleed: Add SKL return thunk
Date: Fri, 4 Nov 2022 08:32:57 -0700 [thread overview]
Message-ID: <Y2UwqXd3NFYJrjWG@dev-arch.thelio-3990X> (raw)
In-Reply-To: <Y2UJPrgYTtKHblnh@hirez.programming.kicks-ass.net>
On Fri, Nov 04, 2022 at 01:44:46PM +0100, Peter Zijlstra wrote:
> On Thu, Nov 03, 2022 at 10:53:54PM +0000, Andrew Cooper wrote:
> > On 21/10/2022 16:21, Nathan Chancellor wrote:
> > > On Fri, Oct 21, 2022 at 11:53:09AM +0200, Peter Zijlstra wrote:
> > >> On Thu, Oct 20, 2022 at 04:10:28PM -0700, Nathan Chancellor wrote:
> > >>> This commit is now in -next as commit 5d8213864ade ("x86/retbleed: Add
> > >>> SKL return thunk"). I just bisected an immediate reboot on my AMD test
> > >>> system when starting a virtual machine with QEMU + KVM to it (see the
> > >>> bisect log below). My Intel test systems do not show this.
> > >>> Unfortunately, I do not have much more information, as there are no logs
> > >>> in journalctl, which makes sense as the reboot occurs immediately after
> > >>> I hit the enter key for the QEMU command.
> > >>>
> > >>> If there is any further information I can provide or patches I can test
> > >>> for further debugging, I am more than happy to do so.
> > >> Moo :-(
> > >>
> > >> you happen to have a .config for me?
> > > Sure thing, sorry I did not provide it in the first place! Attached. It
> > > has been run through localmodconfig for the particular machine but I
> > > assume the core pieces should still be present.
> >
> > Following up from some debugging on IRC.
> >
> > The problem is that FILL_RETURN_BUFFER now has a per-cpu variable
> > access, and AMD SVM has a fun optimisation where the VMRUN instruction
> > doesn't swap, amongst other things, %gs.
> >
> > per-cpu variables only become safe following
> > vmload(__sme_page_pa(sd->save_area)); in svm_vcpu_enter_exit().
> >
> > Given that retbleed=force ought to work on non-skylake hardware, the
> > appropriate fix is to move the VMLOAD/VMSAVE's down into asm and put
> > them adjacent to VMRUN.
> >
> > This also addresses an undocumented dependency where its only the memory
> > clobber in vmload() which stops the compiler moving
> > svm_vcpu_enter_exit()'s calculation of sd into an unsafe position.
>
> So, aside from wasting the entire morning on resuscitating my AMD
> Interlagos, I ended up with the below patch which seems to work.
>
> Not being a virt person, I'm sure I've messed up something, please
> advise.
I too am not a virt person but this survives spawning a guest on the
host and in the guest, which is the extent of the testing I do with KVM
on a regular basis.
Tested-by: Nathan Chancellor <nathan@kernel.org>
Thanks again for looking into it and Andrew for the assists along the
way!
> ---
> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index 58f0077d9357..f7ee1eedacfe 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -3929,11 +3929,8 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu)
> * the state doesn't need to be copied between vmcb01 and
> * vmcb02 when switching vmcbs for nested virtualization.
> */
> - vmload(svm->vmcb01.pa);
> - __svm_vcpu_run(vmcb_pa, (unsigned long *)&vcpu->arch.regs);
> - vmsave(svm->vmcb01.pa);
> -
> - vmload(__sme_page_pa(sd->save_area));
> + __svm_vcpu_run(vmcb_pa, (unsigned long *)&vcpu->arch.regs,
> + svm->vmcb01.pa, __sme_page_pa(sd->save_area));
> }
>
> guest_state_exit_irqoff();
> diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
> index 6a7686bf6900..2a038def7ac7 100644
> --- a/arch/x86/kvm/svm/svm.h
> +++ b/arch/x86/kvm/svm/svm.h
> @@ -684,6 +684,7 @@ void sev_es_unmap_ghcb(struct vcpu_svm *svm);
> /* vmenter.S */
>
> void __svm_sev_es_vcpu_run(unsigned long vmcb_pa);
> -void __svm_vcpu_run(unsigned long vmcb_pa, unsigned long *regs);
> +void __svm_vcpu_run(unsigned long vmcb_pa, unsigned long *regs,
> + unsigned long guest_vmcb_pa, unsigned long host_vmcb_pa);
>
> #endif
> diff --git a/arch/x86/kvm/svm/vmenter.S b/arch/x86/kvm/svm/vmenter.S
> index 09eacf19d718..50f200f7b773 100644
> --- a/arch/x86/kvm/svm/vmenter.S
> +++ b/arch/x86/kvm/svm/vmenter.S
> @@ -32,8 +32,10 @@
>
> /**
> * __svm_vcpu_run - Run a vCPU via a transition to SVM guest mode
> - * @vmcb_pa: unsigned long
> - * @regs: unsigned long * (to guest registers)
> + * @vmcb_pa: unsigned long
> + * @regs: unsigned long * (to guest registers)
> + * @guest_vmcb_pa: unsigned long
> + * @host_vmcb_pa: unsigned long
> */
> SYM_FUNC_START(__svm_vcpu_run)
> push %_ASM_BP
> @@ -51,9 +53,18 @@ SYM_FUNC_START(__svm_vcpu_run)
> /* Save @regs. */
> push %_ASM_ARG2
>
> + /* Save host_vmcb_pa */
> + push %_ASM_ARG4
> +
> + /* Save guest_vmcb_pa */
> + push %_ASM_ARG3
> +
> /* Save @vmcb. */
> push %_ASM_ARG1
>
> + /* Save guest_vmcb_pa */
> + push %_ASM_ARG3
> +
> /* Move @regs to RAX. */
> mov %_ASM_ARG2, %_ASM_AX
>
> @@ -75,15 +86,29 @@ SYM_FUNC_START(__svm_vcpu_run)
> mov VCPU_R15(%_ASM_AX), %r15
> #endif
>
> + /* POP and VMLOAD @guest_vmcb01_pa */
> + pop %_ASM_AX
> +1: vmload %_ASM_AX
> +2:
> /* "POP" @vmcb to RAX. */
> pop %_ASM_AX
>
> /* Enter guest mode */
> sti
>
> -1: vmrun %_ASM_AX
> +3: vmrun %_ASM_AX
> +4:
> + cli
>
> -2: cli
> + /* POP and VMSAVE @guest_vmcb01_pa */
> + pop %_ASM_AX
> +5: vmsave %_ASM_AX
> +6:
> + /* POP and VMLOAD @host_vmcb01_pa */
> + pop %_ASM_AX
> +7: vmload %_ASM_AX
> +8:
> + /* Now host %GS is live */
>
> #ifdef CONFIG_RETPOLINE
> /* IMPORTANT: Stuff the RSB immediately after VM-Exit, before RET! */
> @@ -160,11 +185,26 @@ SYM_FUNC_START(__svm_vcpu_run)
> pop %_ASM_BP
> RET
>
> -3: cmpb $0, kvm_rebooting
> +10: cmpb $0, kvm_rebooting
> jne 2b
> ud2
>
> - _ASM_EXTABLE(1b, 3b)
> +30: cmpb $0, kvm_rebooting
> + jne 4b
> + ud2
> +
> +50: cmpb $0, kvm_rebooting
> + jne 6b
> + ud2
> +
> +70: cmpb $0, kvm_rebooting
> + jne 8b
> + ud2
> +
> + _ASM_EXTABLE(1b, 10b)
> + _ASM_EXTABLE(3b, 30b)
> + _ASM_EXTABLE(5b, 50b)
> + _ASM_EXTABLE(7b, 70b)
>
> SYM_FUNC_END(__svm_vcpu_run)
>
next prev parent reply other threads:[~2022-11-04 15:33 UTC|newest]
Thread overview: 138+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-15 11:10 [PATCH v3 00/59] x86/retbleed: Call depth tracking mitigation Peter Zijlstra
2022-09-15 11:10 ` [PATCH v3 01/59] x86/paravirt: Ensure proper alignment Peter Zijlstra
2022-09-21 11:08 ` [tip: x86/paravirt] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 02/59] x86/cpu: Remove segment load from switch_to_new_gdt() Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 03/59] x86/cpu: Get rid of redundant switch_to_new_gdt() invocations Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 04/59] x86/cpu: Re-enable stackprotector Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 05/59] x86/modules: Set VM_FLUSH_RESET_PERMS in module_alloc() Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 06/59] x86/vdso: Ensure all kernel code is seen by objtool Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 07/59] x86: Sanitize linker script Peter Zijlstra
2022-10-07 16:03 ` Borislav Petkov
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 08/59] arch: Introduce CONFIG_FUNCTION_ALIGNMENT Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:10 ` [PATCH v3 09/59] x86/asm: Differentiate between code and function alignment Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 10/59] x86/error_inject: Align function properly Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:10 ` [PATCH v3 11/59] x86/paravirt: Properly align PV functions Peter Zijlstra
2022-09-15 14:34 ` Juergen Gross
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 12/59] x86/entry: Align SYM_CODE_START() variants Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 13/59] crypto: x86/camellia: Remove redundant alignments Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 14/59] crypto: x86/cast5: " Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 15/59] crypto: x86/crct10dif-pcl: " Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 16/59] crypto: x86/serpent: " Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 17/59] crypto: x86/sha1: Remove custom alignments Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 18/59] crypto: x86/sha256: " Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 19/59] crypto: x86/sm[34]: Remove redundant alignments Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:10 ` [PATCH v3 20/59] crypto: twofish: " Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 21/59] crypto: x86/poly1305: Remove custom function alignment Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 22/59] x86: Put hot per CPU variables into a struct Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 23/59] x86/percpu: Move preempt_count next to current_task Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 24/59] x86/percpu: Move cpu_number " Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 25/59] x86/percpu: Move current_top_of_stack " Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 26/59] x86/percpu: Move irq_stack variables " Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 27/59] x86/softirq: Move softirq pending next to current task Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 28/59] objtool: Allow !PC relative relocations Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 29/59] objtool: Track init section Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 30/59] objtool: Add .call_sites section Peter Zijlstra
2022-10-17 14:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 31/59] objtool: Add --hacks=skylake Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 32/59] objtool: Allow STT_NOTYPE -> STT_FUNC+0 tail-calls Peter Zijlstra
2022-09-22 5:27 ` Pawan Gupta
2022-09-22 10:29 ` Peter Zijlstra
2022-09-22 10:47 ` Peter Zijlstra
2022-09-22 13:15 ` Peter Zijlstra
2022-09-23 14:35 ` Peter Zijlstra
2022-09-23 17:36 ` Pawan Gupta
2022-09-15 11:11 ` [PATCH v3 33/59] objtool: Fix find_{symbol,func}_containing() Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 34/59] objtool: Allow symbol range comparisons for IBT/ENDBR Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 35/59] x86/entry: Make sync_regs() invocation a tail call Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 36/59] ftrace: Add HAVE_DYNAMIC_FTRACE_NO_PATCHABLE Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 37/59] x86/putuser: Provide room for padding Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 38/59] x86/Kconfig: Add CONFIG_CALL_THUNKS Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 39/59] x86/Kconfig: Introduce function padding Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 40/59] x86/retbleed: Add X86_FEATURE_CALL_DEPTH Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 41/59] x86/alternatives: Provide text_poke_copy_locked() Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 42/59] x86/entry: Make some entry symbols global Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 43/59] x86/paravirt: Make struct paravirt_call_site unconditionally available Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 44/59] x86/callthunks: Add call patching for call depth tracking Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 45/59] x86/modules: Add call patching Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 46/59] x86/returnthunk: Allow different return thunks Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 47/59] x86/asm: Provide ALTERNATIVE_3 Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 48/59] x86/retbleed: Add SKL return thunk Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-10-20 23:10 ` [PATCH v3 48/59] " Nathan Chancellor
2022-10-21 9:53 ` Peter Zijlstra
2022-10-21 15:21 ` Nathan Chancellor
2022-11-03 22:53 ` KVM vs AMD: " Andrew Cooper
2022-11-04 12:44 ` Peter Zijlstra
2022-11-04 15:29 ` Andrew Cooper
2022-11-04 15:32 ` Nathan Chancellor [this message]
2022-11-07 9:37 ` Paolo Bonzini
2022-09-15 11:11 ` [PATCH v3 49/59] x86/retpoline: Add SKL retthunk retpolines Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 50/59] x86/retbleed: Add SKL call thunk Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 51/59] x86/calldepth: Add ret/call counting for debug Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2022-09-15 11:11 ` [PATCH v3 52/59] static_call: Add call depth tracking support Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 53/59] kallsyms: Take callthunks into account Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 54/59] x86/orc: Make it callthunk aware Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 55/59] x86/bpf: Emit call depth accounting if required Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
2023-01-05 21:49 ` [PATCH v3 55/59] " Joan Bruguera
2022-09-15 11:11 ` [PATCH v3 56/59] x86/ftrace: Remove ftrace_epilogue() Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-10-20 15:17 ` [tip: x86/urgent] " tip-bot2 for Peter Zijlstra
2022-12-09 15:41 ` [PATCH v3 56/59] " Steven Rostedt
2022-09-15 11:11 ` [PATCH v3 57/59] x86/ftrace: Rebalance RSB Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 58/59] x86/ftrace: Make it call depth tracking aware Peter Zijlstra
2022-09-21 10:19 ` [PATCH v3.1 " Peter Zijlstra
2022-09-21 18:45 ` Pawan Gupta
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-09-15 11:11 ` [PATCH v3 59/59] x86/retbleed: Add call depth tracking mitigation Peter Zijlstra
2022-10-17 14:53 ` [tip: x86/core] " tip-bot2 for Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y2UwqXd3NFYJrjWG@dev-arch.thelio-3990X \
--to=nathan@kernel.org \
--cc=Andrew.Cooper3@citrix.com \
--cc=alyssa.milburn@linux.intel.com \
--cc=ast@kernel.org \
--cc=daniel@iogearbox.net \
--cc=edumazet@google.com \
--cc=hjl.tools@gmail.com \
--cc=jannh@google.com \
--cc=jgross@suse.com \
--cc=joao.moreira@intel.com \
--cc=joseph.nuzman@intel.com \
--cc=jpoimboe@kernel.org \
--cc=kprateek.nayak@amd.com \
--cc=kvm@vger.kernel.org \
--cc=kwikner@ethz.ch \
--cc=linux-kernel@vger.kernel.org \
--cc=mhiramat@kernel.org \
--cc=pawan.kumar.gupta@linux.intel.com \
--cc=pbonzini@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=sean.j.christopherson@intel.com \
--cc=suravee.suthikulpanit@amd.com \
--cc=tglx@linutronix.de \
--cc=tim.c.chen@linux.intel.com \
--cc=torvalds@linux-foundation.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).