From: Borislav Petkov <bp@alien8.de>
To: Joerg Roedel <joro@8bytes.org>
Cc: x86@kernel.org, hpa@zytor.com, Andy Lutomirski <luto@kernel.org>,
Dave Hansen <dave.hansen@linux.intel.com>,
Peter Zijlstra <peterz@infradead.org>,
Thomas Hellstrom <thellstrom@vmware.com>,
Jiri Slaby <jslaby@suse.cz>,
Dan Williams <dan.j.williams@intel.com>,
Tom Lendacky <thomas.lendacky@amd.com>,
Juergen Gross <jgross@suse.com>,
Kees Cook <keescook@chromium.org>,
David Rientjes <rientjes@google.com>,
Cfir Cohen <cfir@google.com>, Erdem Aktas <erdemaktas@google.com>,
Masami Hiramatsu <mhiramat@kernel.org>,
Mike Stunes <mstunes@vmware.com>, Joerg Roedel <jroedel@suse.de>,
linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
virtualization@lists.linux-foundation.org
Subject: Re: [PATCH v3 23/75] x86/boot/compressed/64: Setup GHCB Based VC Exception handler
Date: Tue, 12 May 2020 20:11:57 +0200 [thread overview]
Message-ID: <20200512181157.GD6859@zn.tnic> (raw)
In-Reply-To: <20200428151725.31091-24-joro@8bytes.org>
On Tue, Apr 28, 2020 at 05:16:33PM +0200, Joerg Roedel wrote:
> From: Joerg Roedel <jroedel@suse.de>
>
> Install an exception handler for #VC exception that uses a GHCB. Also
> add the infrastructure for handling different exit-codes by decoding
> the instruction that caused the exception and error handling.
>
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
> arch/x86/Kconfig | 1 +
> arch/x86/boot/compressed/Makefile | 3 +
> arch/x86/boot/compressed/idt_64.c | 4 +
> arch/x86/boot/compressed/idt_handlers_64.S | 3 +-
> arch/x86/boot/compressed/misc.c | 7 +
> arch/x86/boot/compressed/misc.h | 7 +
> arch/x86/boot/compressed/sev-es.c | 110 +++++++++++++++
> arch/x86/include/asm/sev-es.h | 39 ++++++
> arch/x86/include/uapi/asm/svm.h | 1 +
> arch/x86/kernel/sev-es-shared.c | 154 +++++++++++++++++++++
> 10 files changed, 328 insertions(+), 1 deletion(-)
>
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 1197b5596d5a..2ba5f74f186d 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1523,6 +1523,7 @@ config AMD_MEM_ENCRYPT
> select DYNAMIC_PHYSICAL_MASK
> select ARCH_USE_MEMREMAP_PROT
> select ARCH_HAS_FORCE_DMA_UNENCRYPTED
> + select INSTRUCTION_DECODER
> ---help---
> Say yes to enable support for the encryption of system memory.
> This requires an AMD processor that supports Secure Memory
> diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
> index a7847a1ef63a..8372b85c9c0e 100644
> --- a/arch/x86/boot/compressed/Makefile
> +++ b/arch/x86/boot/compressed/Makefile
> @@ -41,6 +41,9 @@ KBUILD_CFLAGS += -Wno-pointer-sign
> KBUILD_CFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=)
> KBUILD_CFLAGS += -fno-asynchronous-unwind-tables
>
> +# sev-es.c inludes generated $(objtree)/arch/x86/lib/inat-tables.c
"includes"
> +CFLAGS_sev-es.o += -I$(objtree)/arch/x86/lib/
Does it?
I see
#include "../../lib/inat.c"
#include "../../lib/insn.c"
only and with the above CFLAGS-line removed, it builds still.
Leftover from earlier?
> +
> KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__
> GCOV_PROFILE := n
> UBSAN_SANITIZE :=n
> diff --git a/arch/x86/boot/compressed/idt_64.c b/arch/x86/boot/compressed/idt_64.c
> index f8295d68b3e1..44d20c4f47c9 100644
> --- a/arch/x86/boot/compressed/idt_64.c
> +++ b/arch/x86/boot/compressed/idt_64.c
> @@ -45,5 +45,9 @@ void load_stage2_idt(void)
>
> set_idt_entry(X86_TRAP_PF, boot_page_fault);
>
> +#ifdef CONFIG_AMD_MEM_ENCRYPT
> + set_idt_entry(X86_TRAP_VC, boot_stage2_vc);
> +#endif
if IS_ENABLED()...
...
> +static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt)
> +{
> + char buffer[MAX_INSN_SIZE];
> + enum es_result ret;
> +
> + memcpy(buffer, (unsigned char *)ctxt->regs->ip, MAX_INSN_SIZE);
> +
> + insn_init(&ctxt->insn, buffer, MAX_INSN_SIZE, 1);
> + insn_get_length(&ctxt->insn);
> +
> + ret = ctxt->insn.immediate.got ? ES_OK : ES_DECODE_FAILED;
Why are we checking whether the immediate? insn_get_length() sets
insn->length unconditionally while insn_get_immediate() can error out
and not set ->got... ?
> +
> + return ret;
> +}
...
> +static bool sev_es_setup_ghcb(void)
> +{
> + if (!sev_es_negotiate_protocol())
> + sev_es_terminate(GHCB_SEV_ES_REASON_PROTOCOL_UNSUPPORTED);
> +
> + if (set_page_decrypted((unsigned long)&boot_ghcb_page))
> + return false;
> +
> + /* Page is now mapped decrypted, clear it */
> + memset(&boot_ghcb_page, 0, sizeof(boot_ghcb_page));
> +
> + boot_ghcb = &boot_ghcb_page;
> +
> + /* Initialize lookup tables for the instruction decoder */
> + inat_init_tables();
Yeah, that call doesn't logically belong in this function AFAICT as this
function should setup the GHCB only. You can move it to the caller.
> +
> + return true;
> +}
> +
> +void sev_es_shutdown_ghcb(void)
> +{
> + if (!boot_ghcb)
> + return;
> +
> + /*
> + * GHCB Page must be flushed from the cache and mapped encrypted again.
> + * Otherwise the running kernel will see strange cache effects when
> + * trying to use that page.
> + */
> + if (set_page_encrypted((unsigned long)&boot_ghcb_page))
> + error("Can't map GHCB page encrypted");
Is that error() call enough?
Shouldn't we BUG_ON() here or mark that page Reserved or so, so that
nothing uses it during the system lifetime and thus avoid the strange
cache effects?
...
> +static enum es_result sev_es_ghcb_hv_call(struct ghcb *ghcb,
> + struct es_em_ctxt *ctxt,
> + u64 exit_code, u64 exit_info_1,
> + u64 exit_info_2)
> +{
> + enum es_result ret;
> +
> + /* Fill in protocol and format specifiers */
> + ghcb->protocol_version = GHCB_PROTOCOL_MAX;
> + ghcb->ghcb_usage = GHCB_DEFAULT_USAGE;
> +
> + ghcb_set_sw_exit_code(ghcb, exit_code);
> + ghcb_set_sw_exit_info_1(ghcb, exit_info_1);
> + ghcb_set_sw_exit_info_2(ghcb, exit_info_2);
> +
> + sev_es_wr_ghcb_msr(__pa(ghcb));
> + VMGEXIT();
> +
> + if ((ghcb->save.sw_exit_info_1 & 0xffffffff) == 1) {
^^^^^^^^^^^
(1UL << 32) - 1
I guess.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
next prev parent reply other threads:[~2020-05-12 18:12 UTC|newest]
Thread overview: 160+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-04-28 15:16 [PATCH v3 00/75] x86: SEV-ES Guest Support Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 01/75] KVM: SVM: Add GHCB definitions Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 02/75] KVM: SVM: Add GHCB Accessor functions Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 03/75] KVM: SVM: Use __packed shorthand Joerg Roedel
2020-04-29 10:12 ` Borislav Petkov
2020-04-28 15:16 ` [PATCH v3 04/75] x86/cpufeatures: Add SEV-ES CPU feature Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 05/75] x86/traps: Move some definitions to <asm/trap_defs.h> Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 06/75] x86/insn: Make inat-tables.c suitable for pre-decompression code Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 07/75] x86/umip: Factor out instruction fetch Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 08/75] x86/umip: Factor out instruction decoding Joerg Roedel
2020-04-30 16:31 ` Borislav Petkov
2020-04-28 15:16 ` [PATCH v3 09/75] x86/insn: Add insn_get_modrm_reg_off() Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 10/75] x86/insn: Add insn_rep_prefix() helper Joerg Roedel
2020-05-04 8:46 ` Borislav Petkov
2020-04-28 15:16 ` [PATCH v3 11/75] x86/boot/compressed/64: Disable red-zone usage Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 12/75] x86/boot/compressed/64: Switch to __KERNEL_CS after GDT is loaded Joerg Roedel
2020-05-04 10:41 ` Borislav Petkov
2020-05-04 11:27 ` Joerg Roedel
2020-05-04 18:30 ` [tip: x86/boot] " tip-bot2 for Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 13/75] x86/boot/compressed/64: Add IDT Infrastructure Joerg Roedel
2020-05-04 10:54 ` Borislav Petkov
2020-05-04 11:28 ` Joerg Roedel
2020-06-03 9:06 ` Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 14/75] x86/boot/compressed/64: Rename kaslr_64.c to ident_map_64.c Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 15/75] x86/boot/compressed/64: Add page-fault handler Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 16/75] x86/boot/compressed/64: Always switch to own page-table Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 17/75] x86/boot/compressed/64: Don't pre-map memory in KASLR code Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 18/75] x86/boot/compressed/64: Change add_identity_map() to take start and end Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 19/75] x86/boot/compressed/64: Add stage1 #VC handler Joerg Roedel
2020-05-09 9:05 ` Borislav Petkov
2020-04-28 15:16 ` [PATCH v3 20/75] x86/boot/compressed/64: Call set_sev_encryption_mask earlier Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 21/75] x86/boot/compressed/64: Check return value of kernel_ident_mapping_init() Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 22/75] x86/boot/compressed/64: Add set_page_en/decrypted() helpers Joerg Roedel
2020-05-11 10:02 ` Borislav Petkov
2020-04-28 15:16 ` [PATCH v3 23/75] x86/boot/compressed/64: Setup GHCB Based VC Exception handler Joerg Roedel
2020-05-11 20:07 ` Borislav Petkov
2020-06-03 10:08 ` Joerg Roedel
2020-05-12 18:11 ` Borislav Petkov [this message]
2020-05-12 21:08 ` Joerg Roedel
2020-05-13 8:59 ` Borislav Petkov
2020-04-28 15:16 ` [PATCH v3 24/75] x86/boot/compressed/64: Unmap GHCB page before booting the kernel Joerg Roedel
2020-05-13 11:13 ` Borislav Petkov
2020-05-13 11:30 ` Joerg Roedel
2020-05-13 11:46 ` Borislav Petkov
2020-06-03 10:40 ` Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 25/75] x86/sev-es: Add support for handling IOIO exceptions Joerg Roedel
2020-05-13 17:58 ` Borislav Petkov
2020-05-16 7:57 ` Borislav Petkov
2020-06-03 14:19 ` Joerg Roedel
2020-05-20 6:20 ` Sean Christopherson
2020-06-03 14:23 ` Joerg Roedel
2020-06-03 23:07 ` Sean Christopherson
2020-06-04 10:15 ` Joerg Roedel
2020-06-04 14:59 ` Sean Christopherson
2020-06-11 10:03 ` Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 26/75] x86/fpu: Move xgetbv()/xsetbv() into separate header Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 27/75] x86/sev-es: Add CPUID handling to #VC handler Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 28/75] x86/idt: Move IDT to data segment Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 29/75] x86/idt: Split idt_data setup out of set_intr_gate() Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 30/75] x86/idt: Move two function from k/idt.c to i/a/desc.h Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 31/75] x86/head/64: Install boot GDT Joerg Roedel
2020-05-18 8:23 ` Borislav Petkov
2020-06-04 11:48 ` Joerg Roedel
2020-06-04 14:13 ` Borislav Petkov
2020-04-28 15:16 ` [PATCH v3 32/75] x86/head/64: Reload GDT after switch to virtual addresses Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 33/75] x86/head/64: Load segment registers earlier Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 34/75] x86/head/64: Switch to initial stack earlier Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 35/75] x86/head/64: Build k/head64.c with -fno-stack-protector Joerg Roedel
2020-05-19 9:15 ` Borislav Petkov
2020-06-03 15:21 ` Joerg Roedel
2020-05-19 13:58 ` Brian Gerst
2020-06-03 15:18 ` Joerg Roedel
2020-06-03 17:14 ` Brian Gerst
2020-04-28 15:16 ` [PATCH v3 36/75] x86/head/64: Load IDT earlier Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 37/75] x86/head/64: Move early exception dispatch to C code Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 38/75] x86/sev-es: Add SEV-ES Feature Detection Joerg Roedel
2020-05-20 8:39 ` Borislav Petkov
2020-06-03 15:24 ` Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 39/75] x86/sev-es: Print SEV-ES info into kernel log Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 40/75] x86/sev-es: Compile early handler code into kernel image Joerg Roedel
2020-05-20 9:14 ` Borislav Petkov
2020-06-04 11:54 ` Joerg Roedel
2020-06-04 15:19 ` Borislav Petkov
2020-06-11 10:05 ` Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 41/75] x86/sev-es: Setup early #VC handler Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 42/75] x86/sev-es: Setup GHCB based boot " Joerg Roedel
2020-05-20 19:22 ` Borislav Petkov
2020-06-04 12:07 ` Joerg Roedel
2020-06-04 15:30 ` Borislav Petkov
2020-06-11 10:14 ` Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 43/75] x86/sev-es: Setup per-cpu GHCBs for the runtime handler Joerg Roedel
2020-05-22 8:33 ` Borislav Petkov
2020-04-28 15:16 ` [PATCH v3 44/75] x86/sev-es: Allocate and Map IST stacks for #VC handler Joerg Roedel
2020-05-22 9:49 ` Borislav Petkov
2020-04-28 15:16 ` [PATCH v3 45/75] x86/dumpstack/64: Handle #VC exception stacks Joerg Roedel
2020-05-22 13:06 ` Borislav Petkov
2020-04-28 15:16 ` [PATCH v3 46/75] x86/sev-es: Shift #VC IST Stack in nmi_enter()/nmi_exit() Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 47/75] x86/sev-es: Add Runtime #VC Exception Handler Joerg Roedel
2020-05-23 7:59 ` Borislav Petkov
2020-06-11 11:48 ` Joerg Roedel
2020-06-11 17:38 ` Sean Christopherson
2020-06-11 18:16 ` Joerg Roedel
2020-06-12 13:13 ` Borislav Petkov
2020-06-11 11:53 ` Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 48/75] x86/sev-es: Wire up existing #VC exit-code handlers Joerg Roedel
2020-04-28 15:16 ` [PATCH v3 49/75] x86/sev-es: Handle instruction fetches from user-space Joerg Roedel
2020-05-23 9:23 ` Borislav Petkov
2020-04-28 15:17 ` [PATCH v3 50/75] x86/sev-es: Do not crash on #VC exceptions " Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 51/75] x86/sev-es: Handle MMIO events Joerg Roedel
2020-05-20 6:32 ` Sean Christopherson
2020-06-11 12:40 ` Joerg Roedel
2020-05-25 8:02 ` Borislav Petkov
2020-04-28 15:17 ` [PATCH v3 52/75] x86/sev-es: Handle MMIO String Instructions Joerg Roedel
2020-05-25 9:47 ` Borislav Petkov
2020-04-28 15:17 ` [PATCH v3 53/75] x86/sev-es: Handle MSR events Joerg Roedel
2020-05-25 9:53 ` Borislav Petkov
2020-04-28 15:17 ` [PATCH v3 54/75] x86/sev-es: Handle DR7 read/write events Joerg Roedel
2020-05-25 10:59 ` Borislav Petkov
2020-06-11 13:06 ` Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 55/75] x86/sev-es: Handle WBINVD Events Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 56/75] x86/sev-es: Handle RDTSC(P) Events Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 57/75] x86/sev-es: Handle RDPMC Events Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 58/75] x86/sev-es: Handle INVD Events Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 59/75] x86/sev-es: Handle MONITOR/MONITORX Events Joerg Roedel
2020-05-20 6:38 ` Sean Christopherson
2020-06-11 13:10 ` Joerg Roedel
2020-06-11 17:13 ` Sean Christopherson
2020-06-11 19:33 ` Tom Lendacky
2020-06-12 9:25 ` Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 60/75] x86/sev-es: Handle MWAIT/MWAITX Events Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 61/75] x86/sev-es: Handle VMMCALL Events Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 62/75] x86/sev-es: Handle #AC Events Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 63/75] x86/sev-es: Handle #DB Events Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 64/75] x86/sev-es: Cache CPUID results for improved performance Joerg Roedel
2020-05-06 18:08 ` Mike Stunes
2020-05-06 23:02 ` Tom Lendacky
2020-05-20 5:16 ` Sean Christopherson
2020-05-26 9:19 ` Borislav Petkov
2020-05-27 17:49 ` Tom Lendacky
2020-05-27 15:34 ` Tom Lendacky
2020-06-12 9:12 ` Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 65/75] x86/paravirt: Allow hypervisor specific VMMCALL handling under SEV-ES Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 66/75] x86/kvm: Add KVM " Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 67/75] x86/vmware: Add VMware specific handling for VMMCALL " Joerg Roedel
2020-05-28 12:38 ` Borislav Petkov
2020-04-28 15:17 ` [PATCH v3 68/75] x86/realmode: Add SEV-ES specific trampoline entry point Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 69/75] x86/realmode: Setup AP jump table Joerg Roedel
2020-05-29 9:02 ` Borislav Petkov
2020-05-29 16:21 ` Tom Lendacky
2020-04-28 15:17 ` [PATCH v3 70/75] x86/head/64: Setup TSS early for secondary CPUs Joerg Roedel
2020-05-07 10:51 ` [x86/head/64] e5a6f186af: BUG:kernel_hang_in_boot_stage kernel test robot
2020-06-02 15:46 ` [PATCH v3 70/75] x86/head/64: Setup TSS early for secondary CPUs Borislav Petkov
2020-04-28 15:17 ` [PATCH v3 71/75] x86/head/64: Don't call verify_cpu() on starting APs Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 72/75] x86/head/64: Rename start_cpu0 Joerg Roedel
2020-04-28 15:17 ` [PATCH v3 73/75] x86/sev-es: Support CPU offline/online Joerg Roedel
2020-06-03 9:54 ` Borislav Petkov
2020-04-28 15:17 ` [PATCH v3 74/75] x86/sev-es: Handle NMI State Joerg Roedel
2020-06-03 9:59 ` Borislav Petkov
2020-04-28 15:17 ` [PATCH v3 75/75] x86/efi: Add GHCB mappings when SEV-ES is active Joerg Roedel
2020-06-03 13:52 ` Borislav Petkov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200512181157.GD6859@zn.tnic \
--to=bp@alien8.de \
--cc=cfir@google.com \
--cc=dan.j.williams@intel.com \
--cc=dave.hansen@linux.intel.com \
--cc=erdemaktas@google.com \
--cc=hpa@zytor.com \
--cc=jgross@suse.com \
--cc=joro@8bytes.org \
--cc=jroedel@suse.de \
--cc=jslaby@suse.cz \
--cc=keescook@chromium.org \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=luto@kernel.org \
--cc=mhiramat@kernel.org \
--cc=mstunes@vmware.com \
--cc=peterz@infradead.org \
--cc=rientjes@google.com \
--cc=thellstrom@vmware.com \
--cc=thomas.lendacky@amd.com \
--cc=virtualization@lists.linux-foundation.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).