From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from list by lists.gnu.org with archive (Exim 4.90_1) id 1mFE10-0007Tl-7T for mharc-grub-devel@gnu.org; Sun, 15 Aug 2021 07:09:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39054) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFE0y-0007TT-FP for grub-devel@gnu.org; Sun, 15 Aug 2021 07:09:16 -0400 Received: from out2-smtp.messagingengine.com ([66.111.4.26]:45249) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFE0v-0005CU-Vk for grub-devel@gnu.org; Sun, 15 Aug 2021 07:09:16 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 04E575C0050; Sun, 15 Aug 2021 07:09:12 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Sun, 15 Aug 2021 07:09:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:mime-version:content-type; s=fm1; bh=lrvZ3pScKqMCqnY0K0sc7iO2zqADxHY8XdaFfvcSr3g=; b=MnqolovOYQl+ 1LKCVyLPXJ/E4o8+x2MP2iXGbVLtOnUb3h7pFb6CLKw3wESQBQp4Xfvvd+khYmT8 svy27uGmwkojo5OixQloUTTiZgKo2YjESoGwtTfmE8eSpeuePXRBcvE2cmeo/YCH oSnuAgwnlF9YG+db3CGa0JFZF5uGhWiWn6QkK3v/Q/5iaanmWQ9o43ODkzblmLiB 2cLAdrx82h+Mg5WUPSIBxcF/u7+zTtb6Q+8Zez8VwJqEff2fNoVJ0gVfwA2tX1QF 6A1zi1fbTDH8KUtT+DNgzp1Qi/BPUiQUlvHnknUfegxx8yHQS0Ugcp197baWYvAK 6KonPQXUSA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:message-id :mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm3; bh=lrvZ3pScKqMCqnY0K0sc7iO2zqADx HY8XdaFfvcSr3g=; b=EfiZPlTVFLmW6RM4Rn8dSEoqs/W1S6PknWRAwKpn06SU6 rgAj01qh8JFNxhNHUC3Wl7LM6aOLHMYCaY96wk3W8RkvYpXS5D5hKiyOZlL23Q2p OASxNH2mseehLIvGl9sC9JieMa3i5w48Dd6wH/7xeML2l9/5ncY0MFaP6LLstQRg tV7YX8ehSGarYRRKZXvFVNCfKAXsnTxPdAGPqfsvs+6q/5+o/StNExN4o88uQRmx O3DVwdH2pFN+pg+d8kFQsjtFjcBRexlob79TVyL5JQmpcO5msVBiDb0zfuqf2px5 ds7pnVG/GW913hoFGTIb36ynpJf6vXzov8JyoM5sg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrkeelgdefiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvuffkgggtugesghdtreertddtvdenucfhrhhomheprfgrthhrihgtkhcu ufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvghrnhepje eifedvueelfffgjeduffdvgefhiefgjefgvddvfeduvefffeevfffhgfekieffnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhssehpkhhsrd himh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 15 Aug 2021 07:09:10 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 5b6a5c8d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Sun, 15 Aug 2021 11:09:07 +0000 (UTC) Date: Sun, 15 Aug 2021 13:09:06 +0200 From: Patrick Steinhardt To: grub-devel@gnu.org Cc: Leif Lindholm , Daniel Kiper , Stefan Berger Subject: [PATCH v3 0/6] Runtime allocation of memory regions Message-ID: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="4TwDkkbCBpj+3TRL" Content-Disposition: inline Received-SPF: pass client-ip=66.111.4.26; envelope-from=ps@pks.im; helo=out2-smtp.messagingengine.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: grub-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: The development of GNU GRUB List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Aug 2021 11:09:16 -0000 --4TwDkkbCBpj+3TRL Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, this is the third version of my patch set to implement runtime allocation of additional memory regions. Changes compared to v2: - A new preparatory patch was added to remove unused code which unloaded modules on OOM. - Patch 2/4 has been split up into two patches: one to drop the logic where we request a quarter of available memory and then put bounds to it, and one to split apart request of additional regions and initialization of the EFI MM system. - Flags are now `unsigned int` instead of `unsigned`. - `add_memory_regions ()` now gets all flags instead of only a single flag `consecutive`. - Flags are now defines and not an enum anymore. - The callback function is now called `grub_mm_add_region_func_t` instead of `grub_mm_region_func_t`. Flags and its variable have been renamed accordingly. Patrick Patrick Steinhardt (6): mm: Drop unused unloading of modules on OOM mm: Allow dynamically requesting additional memory regions efi: mm: Always request a fixed number of pages on init efi: mm: Extract function to add memory regions efi: mm: Pass up errors from `add_memory_regions ()` efi: mm: Implement runtime addition of pages grub-core/kern/dl.c | 20 ---------- grub-core/kern/efi/mm.c | 83 +++++++++++++++++++---------------------- grub-core/kern/mm.c | 12 +++--- include/grub/dl.h | 1 - include/grub/mm.h | 16 ++++++++ 5 files changed, 61 insertions(+), 71 deletions(-) Range-diff against v2: 1: cf709a8a1 < -: --------- mm: Allow dynamically requesting additional m= emory regions -: --------- > 1: 7f2738a64 mm: Drop unused unloading of modules on OOM -: --------- > 2: 3f0ec2a76 mm: Allow dynamically requesting additional m= emory regions 2: 2a9ccee9e ! 3: 5da49ddf1 efi: mm: Extract function to add memory regio= ns @@ Metadata Author: Patrick Steinhardt =20 ## Commit message ## - efi: mm: Extract function to add memory regions + efi: mm: Always request a fixed number of pages on init =20 - In preparation of support for runtime-allocating additional memory - region, this patch extracts the function to retrieve the EFI memor= y map - and add a subset of it to GRUB's own memory regions. + When initializing the EFI memory subsytem, we will by default requ= est a + quarter of the available memory, bounded by a minimum/maximum valu= e. + Given that we're about to extend the EFI memory system to dynamica= lly + request additional pages from the firmware as required, this scali= ng of + requested memory based on available memory will not make a lot of = sense + anymore. =20 - Note that this commit also changes how many bytes we request by de= fault. - Previously, we would've tried to allocate a quarter of available s= ystem - memory, bounded by a minimum/maximum value. As we're about to impl= ement - runtime allocation of memory, we now instead always request the mi= nimum - amount of bytes and let the memory allocator call out to our callb= ack. + Remove this logic as a preparatory patch such that we'll instead d= efer + to the runtime memory allocator. Note that ideally, we'd want to c= hange + this after dynamic requesting of pages has been implemented for th= e EFI + platform. But because we'll need to split up initialization of the + memory subsystem and the request of pages from the firmware, we'd = have + to duplicate quite some logic at first only to remove it afterwards + again. This seems quite pointless, so we instead have patches slig= htly + out of order. =20 Signed-off-by: Patrick Steinhardt =20 @@ grub-core/kern/efi/mm.c: filter_memory_map (grub_efi_memory_descript= or_t *memory /* Add memory regions. */ static void add_memory_regions (grub_efi_memory_descriptor_t *memory_map, -@@ grub-core/kern/efi/mm.c: add_memory_regions (grub_efi_memory_descri= ptor_t *memory_map, -=20 - addr =3D grub_efi_allocate_pages_real (start, pages, - GRUB_EFI_ALLOCATE_ADDRESS, -- GRUB_EFI_LOADER_CODE); =20 -+ GRUB_EFI_LOADER_CODE); - if (! addr) - grub_fatal ("cannot allocate conventional memory %p with %u pages", - (void *) ((grub_addr_t) start), -@@ grub-core/kern/efi/mm.c: print_memory_map (grub_efi_memory_descript= or_t *memory_map, - } - #endif -=20 --void --grub_efi_mm_init (void) -+static grub_err_t -+grub_efi_mm_add_regions (grub_efi_uint64_t required_bytes) - { - grub_efi_memory_descriptor_t *memory_map; - grub_efi_memory_descriptor_t *memory_map_end; @@ grub-core/kern/efi/mm.c: grub_efi_mm_init (void) grub_efi_memory_descriptor_t *filtered_memory_map_end; grub_efi_uintn_t map_size; @@ grub-core/kern/efi/mm.c: grub_efi_mm_init (void) int mm_status; =20 /* Prepare a memory region to store two memory maps. */ - memory_map =3D grub_efi_allocate_any_pages (2 * BYTES_TO_PAGES (MEM= ORY_MAP_SIZE)); - if (! memory_map) -- grub_fatal ("cannot allocate memory"); -+ return grub_error (GRUB_ERR_OUT_OF_MEMORY, "cannot allocate memor= y"); -=20 - /* Obtain descriptors for available memory. */ - map_size =3D MEMORY_MAP_SIZE; -@@ grub-core/kern/efi/mm.c: grub_efi_mm_init (void) -=20 - memory_map =3D grub_efi_allocate_any_pages (2 * BYTES_TO_PAGES = (map_size)); - if (! memory_map) -- grub_fatal ("cannot allocate memory"); -+ return grub_error (GRUB_ERR_OUT_OF_MEMORY, "cannot allocate memory"); -=20 - mm_status =3D grub_efi_get_memory_map (&map_size, memory_map, 0, - &desc_size, 0); - } -=20 - if (mm_status < 0) -- grub_fatal ("cannot get memory map"); -+ return grub_error (GRUB_ERR_OUT_OF_MEMORY, "cannot get memory map= "); -=20 - memory_map_end =3D NEXT_MEMORY_DESCRIPTOR (memory_map, map_size); -=20 @@ grub-core/kern/efi/mm.c: grub_efi_mm_init (void) filtered_memory_map_end =3D filter_memory_map (memory_map, filtered= _memory_map, desc_size, memory_map_end); @@ grub-core/kern/efi/mm.c: grub_efi_mm_init (void) /* Allocate memory regions for GRUB's memory management. */ add_memory_regions (filtered_memory_map, desc_size, - filtered_memory_map_end, required_pages); -+ filtered_memory_map_end, -+ BYTES_TO_PAGES (required_bytes)); ++ filtered_memory_map_end, BYTES_TO_PAGES (DEFAULT_HEAP_SIZE)); =20 #if 0 /* For debug. */ -@@ grub-core/kern/efi/mm.c: grub_efi_mm_init (void) - /* Release the memory maps. */ - grub_efi_free_pages ((grub_addr_t) memory_map, - 2 * BYTES_TO_PAGES (MEMORY_MAP_SIZE)); -+ -+ return GRUB_ERR_NONE; -+} -+ -+void -+grub_efi_mm_init (void) -+{ -+ if (grub_efi_mm_add_regions (DEFAULT_HEAP_SIZE) !=3D GRUB_ERR_NONE) -+ grub_fatal (grub_errmsg); - } -=20 - #if defined (__aarch64__) || defined (__arm__) || defined (__riscv) -: --------- > 4: 7b2f5fcb5 efi: mm: Extract function to add memory regio= ns 3: ff7e5be9f ! 5: e90bbcf04 efi: mm: Pass up errors from `add_memory_regi= ons ()` @@ Commit message from the firmware at runtime, where it doesn't make sense anymore = to fail hard. =20 - Refactor the function to return an error to prepare for this. + Refactor the function to return an error to prepare for this. Note= that + this does not change the behaviour when initializing the memory sy= stem + because `grub_efi_mm_init ()` knows to call `grub_fatal ()` in case + `grub_efi_mm_add_regions ()` returns an error. =20 Signed-off-by: Patrick Steinhardt =20 @@ grub-core/kern/efi/mm.c: grub_efi_mm_add_regions (grub_efi_uint64_t = required_byt =20 /* Allocate memory regions for GRUB's memory management. */ - add_memory_regions (filtered_memory_map, desc_size, -- filtered_memory_map_end, -- BYTES_TO_PAGES (required_bytes)); +- filtered_memory_map_end, BYTES_TO_PAGES (required_bytes)); + err =3D add_memory_regions (filtered_memory_map, desc_size, + filtered_memory_map_end, + BYTES_TO_PAGES (required_bytes)); 4: a27f5b047 ! 6: 0a3ef3d2a efi: mm: Implement runtime addition of pages @@ Commit message efi: mm: Implement runtime addition of pages =20 Adjust the interface of `grub_efi_mm_add_regions ()` to take a set= of - `GRUB_MM_REGION_*` flags, which most notably is currently only the + `GRUB_MM_ADD_REGION_*` flags, which most notably is currently only= the `CONSECUTVE` flag. This allows us to set the function up as callba= ck for the memory subsystem and have it call out to us in case there's not enough pages available in the current heap. @@ grub-core/kern/efi/mm.c: static grub_err_t grub_efi_memory_descriptor_t *memory_map_end, - grub_efi_uint64_t required_pages) + grub_efi_uint64_t required_pages, -+ char consecutive) ++ unsigned int flags) { grub_efi_memory_descriptor_t *desc; =20 @@ grub-core/kern/efi/mm.c: add_memory_regions (grub_efi_memory_descrip= tor_t *memor start =3D desc->physical_start; pages =3D desc->num_pages; + -+ if (pages < required_pages && consecutive) ++ if (pages < required_pages && (flags & GRUB_MM_ADD_REGION_CONSE= CUTIVE)) + continue; + if (pages > required_pages) @@ grub-core/kern/efi/mm.c: print_memory_map (grub_efi_memory_descripto= r_t *memory_ =20 static grub_err_t -grub_efi_mm_add_regions (grub_efi_uint64_t required_bytes) -+grub_efi_mm_add_regions (grub_efi_uint64_t required_bytes, unsigned f= lags) ++grub_efi_mm_add_regions (grub_efi_uint64_t required_bytes, unsigned i= nt flags) { grub_efi_memory_descriptor_t *memory_map; grub_efi_memory_descriptor_t *memory_map_end; @@ grub-core/kern/efi/mm.c: grub_efi_mm_add_regions (grub_efi_uint64_t = required_byt filtered_memory_map_end, - BYTES_TO_PAGES (required_bytes)); + BYTES_TO_PAGES (required_bytes), -+ flags & GRUB_MM_REGION_CONSECUTIVE); ++ flags); if (err !=3D GRUB_ERR_NONE) return err; =20 @@ grub-core/kern/efi/mm.c: grub_efi_mm_add_regions (grub_efi_uint64_t = required_byt grub_efi_mm_init (void) { - if (grub_efi_mm_add_regions (DEFAULT_HEAP_SIZE) !=3D GRUB_ERR_NONE) -+ if (grub_efi_mm_add_regions (DEFAULT_HEAP_SIZE, 0) !=3D GRUB_ERR_NO= NE) - grub_fatal (grub_errmsg); -+ grub_mm_region_fn =3D grub_efi_mm_add_regions; ++ if (grub_efi_mm_add_regions (DEFAULT_HEAP_SIZE, GRUB_MM_ADD_REGION_= NONE) !=3D GRUB_ERR_NONE) + grub_fatal ("%s", grub_errmsg); ++ grub_mm_add_region_fn =3D grub_efi_mm_add_regions; } =20 #if defined (__aarch64__) || defined (__arm__) || defined (__riscv) --=20 2.32.0 --4TwDkkbCBpj+3TRL Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEF9hrgiFbCdvenl/rVbJhu7ckPpQFAmEY9dEACgkQVbJhu7ck PpRv3g//bL6TG3ZB4zUTzd/MpqALSZyY4s900i8Ex7wiF4/9JR8Tric8TWJSYwrD 56YFfLZ/6yS2p/3S7ibMVAF/R8iTtZqQ/rzIHeeTfe6FJnOLtiSqePBNWujKYt0o 89mDlA5o9kG1n8g9BQMnljutQXVCPCfWAnpH3TsMZI0vMU93NWhryK4G2HS9YJOs 8PumvTdCLmbvGJdMnFjsUuaYn45+V3jEs+yZ8Vocc1JTe4WkLg1i4tXLnNwszNuI yaumD5P+NbrECRGQyzZeJl3wUzBANoN1unJrL9K9NCghS1I15JZ9coI3erUTg5U1 o2ek8EG07s/SIxXJpe79MfRDJ4EEHpVJX0EKmzrK2AR4eE1qvX8d9mgQ52k/py1+ PFUrMgcK1ezoA26gSTCeEEkZstwkA7RiCFR+ETdIXCLj/0Npq7GBCJZJ4j7pFSdC aDmb8/uwoxM65tnzqMCEMtNW9PeF82SFgizaaeqK/tuSoWzez3yIqmHHqe1eC28H 8+KR+YvzGpLfHYAQKOAPMAOOZqxcuH01acF9fWfTtjAepw7ETQzvfapmvuB8ImA8 Dhf05g6Q06aL2xHjw3tqmJXzNweF+3uYrCVsbq+rnO3xWW8j5XJioJgdOzYuAI/p HXKNLk6raVWdzi6PW0G5CpKNR1Hcz4NgNljEU8OkyEn5O79hrjY= =1+fw -----END PGP SIGNATURE----- --4TwDkkbCBpj+3TRL--