[PATCH v2 0/8] reduce maximum memory usage

[PATCH v2 0/8] reduce maximum memory usage

[Thomas Weißschuh]

The processing of vmlinux.o with objtool is the most memory-intensive step
of a kernel build. By reducing the maximum memory usage here we can reduce
the maximum memory usage of the whole kernel build.
Therefore memory pressure on memory starved machines is relieved during
kernel builds and the build is faster as less swapping has to occur.

To: Josh Poimboeuf <jpoimboe@kernel.org>
To: Peter Zijlstra <peterz@infradead.org>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>

---
Changes in v2:
- Warn on out of range values for reloc->type
- Also reduce size of struct special_alt
- Note: v1 did not make it to the lists, only to individual recipients

---
Thomas Weißschuh (8):
      objtool: make struct entries[] static and const
      objtool: make struct check_options static
      objtool: allocate multiple structures with calloc()
      objtool: introduce function elf_reloc_set_type
      objtool: reduce memory usage of struct reloc
      objtool: optimize layout of struct symbol
      objtool: optimize layout of struct special_alt
      objtool: explicitly cleanup resources on success

 tools/objtool/builtin-check.c           |  4 ++-
 tools/objtool/check.c                   |  6 ++--
 tools/objtool/elf.c                     | 56 +++++++++++++++++++--------------
 tools/objtool/include/objtool/builtin.h |  2 --
 tools/objtool/include/objtool/elf.h     | 13 +++++---
 tools/objtool/include/objtool/special.h |  2 +-
 tools/objtool/special.c                 |  6 ++--
 7 files changed, 51 insertions(+), 38 deletions(-)
---
base-commit: 1b929c02afd37871d5afb9d498426f83432e71c2
change-id: 20221216-objtool-memory-06db3b8bf111

Best regards,
-- 
Thomas Weißschuh <linux@weissschuh.net>

[PATCH v2 1/8] objtool: make struct entries[] static and const

[Thomas Weißschuh]

This data is not modified and not used outside of special.c.

Also adapt its users to the constness.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/objtool/special.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tools/objtool/special.c b/tools/objtool/special.c
index 9c8d827f69af..baa85c31526b 100644
--- a/tools/objtool/special.c
+++ b/tools/objtool/special.c
@@ -26,7 +26,7 @@ struct special_entry {
 	unsigned char key; /* jump_label key */
 };
 
-struct special_entry entries[] = {
+static const struct special_entry entries[] = {
 	{
 		.sec = ".altinstructions",
 		.group = true,
@@ -65,7 +65,7 @@ static void reloc_to_sec_off(struct reloc *reloc, struct section **sec,
 	*off = reloc->sym->offset + reloc->addend;
 }
 
-static int get_alt_entry(struct elf *elf, struct special_entry *entry,
+static int get_alt_entry(struct elf *elf, const struct special_entry *entry,
 			 struct section *sec, int idx,
 			 struct special_alt *alt)
 {
@@ -139,7 +139,7 @@ static int get_alt_entry(struct elf *elf, struct special_entry *entry,
  */
 int special_get_alts(struct elf *elf, struct list_head *alts)
 {
-	struct special_entry *entry;
+	const struct special_entry *entry;
 	struct section *sec;
 	unsigned int nr_entries;
 	struct special_alt *alt;

-- 
2.39.0

[tip: objtool/core] objtool: Make struct entries[] static and const

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     4bf285955dcac1810c5677bf69f5ae7a2e9be1cf
Gitweb:        https://git.kernel.org/tip/4bf285955dcac1810c5677bf69f5ae7a2e9be1cf
Author:        Thomas Weißschuh <linux@weissschuh.net>
AuthorDate:    Tue, 27 Dec 2022 16:00:57 
Committer:     Josh Poimboeuf <jpoimboe@kernel.org>
CommitterDate: Mon, 30 Jan 2023 16:28:18 -08:00

objtool: Make struct entries[] static and const

This data is not modified and not used outside of special.c.

Also adapt its users to the constness.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
Link: https://lore.kernel.org/r/20221216-objtool-memory-v2-1-17968f85a464@weissschuh.net
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 tools/objtool/special.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tools/objtool/special.c b/tools/objtool/special.c
index 9c8d827..baa85c3 100644
--- a/tools/objtool/special.c
+++ b/tools/objtool/special.c
@@ -26,7 +26,7 @@ struct special_entry {
 	unsigned char key; /* jump_label key */
 };
 
-struct special_entry entries[] = {
+static const struct special_entry entries[] = {
 	{
 		.sec = ".altinstructions",
 		.group = true,
@@ -65,7 +65,7 @@ static void reloc_to_sec_off(struct reloc *reloc, struct section **sec,
 	*off = reloc->sym->offset + reloc->addend;
 }
 
-static int get_alt_entry(struct elf *elf, struct special_entry *entry,
+static int get_alt_entry(struct elf *elf, const struct special_entry *entry,
 			 struct section *sec, int idx,
 			 struct special_alt *alt)
 {
@@ -139,7 +139,7 @@ static int get_alt_entry(struct elf *elf, struct special_entry *entry,
  */
 int special_get_alts(struct elf *elf, struct list_head *alts)
 {
-	struct special_entry *entry;
+	const struct special_entry *entry;
 	struct section *sec;
 	unsigned int nr_entries;
 	struct special_alt *alt;

[tip: objtool/core] objtool: Make struct entries[] static and const

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     d93ee0553cf2e83c1696a18423bcf05b94b85e1d
Gitweb:        https://git.kernel.org/tip/d93ee0553cf2e83c1696a18423bcf05b94b85e1d
Author:        Thomas Weißschuh <linux@weissschuh.net>
AuthorDate:    Tue, 27 Dec 2022 16:00:57 
Committer:     Josh Poimboeuf <jpoimboe@kernel.org>
CommitterDate: Wed, 01 Feb 2023 09:15:22 -08:00

objtool: Make struct entries[] static and const

This data is not modified and not used outside of special.c.

Also adapt its users to the constness.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
Link: https://lore.kernel.org/r/20221216-objtool-memory-v2-1-17968f85a464@weissschuh.net
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 tools/objtool/special.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tools/objtool/special.c b/tools/objtool/special.c
index 9c8d827..baa85c3 100644
--- a/tools/objtool/special.c
+++ b/tools/objtool/special.c
@@ -26,7 +26,7 @@ struct special_entry {
 	unsigned char key; /* jump_label key */
 };
 
-struct special_entry entries[] = {
+static const struct special_entry entries[] = {
 	{
 		.sec = ".altinstructions",
 		.group = true,
@@ -65,7 +65,7 @@ static void reloc_to_sec_off(struct reloc *reloc, struct section **sec,
 	*off = reloc->sym->offset + reloc->addend;
 }
 
-static int get_alt_entry(struct elf *elf, struct special_entry *entry,
+static int get_alt_entry(struct elf *elf, const struct special_entry *entry,
 			 struct section *sec, int idx,
 			 struct special_alt *alt)
 {
@@ -139,7 +139,7 @@ static int get_alt_entry(struct elf *elf, struct special_entry *entry,
  */
 int special_get_alts(struct elf *elf, struct list_head *alts)
 {
-	struct special_entry *entry;
+	const struct special_entry *entry;
 	struct section *sec;
 	unsigned int nr_entries;
 	struct special_alt *alt;

[PATCH v2 2/8] objtool: make struct check_options static

[Thomas Weißschuh]

It is not used outside of builtin-check.c.

Also remove the unused declaration from builtin.h .

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/objtool/builtin-check.c           | 2 +-
 tools/objtool/include/objtool/builtin.h | 2 --
 2 files changed, 1 insertion(+), 3 deletions(-)

diff --git a/tools/objtool/builtin-check.c b/tools/objtool/builtin-check.c
index a4f39407bf59..7c175198d09f 100644
--- a/tools/objtool/builtin-check.c
+++ b/tools/objtool/builtin-check.c
@@ -65,7 +65,7 @@ static int parse_hacks(const struct option *opt, const char *str, int unset)
 	return found ? 0 : -1;
 }
 
-const struct option check_options[] = {
+static const struct option check_options[] = {
 	OPT_GROUP("Actions:"),
 	OPT_CALLBACK_OPTARG('h', "hacks", NULL, NULL, "jump_label,noinstr,skylake", "patch toolchain bugs/limitations", parse_hacks),
 	OPT_BOOLEAN('i', "ibt", &opts.ibt, "validate and annotate IBT"),
diff --git a/tools/objtool/include/objtool/builtin.h b/tools/objtool/include/objtool/builtin.h
index fa45044e3863..2a108e648b7a 100644
--- a/tools/objtool/include/objtool/builtin.h
+++ b/tools/objtool/include/objtool/builtin.h
@@ -7,8 +7,6 @@
 
 #include <subcmd/parse-options.h>
 
-extern const struct option check_options[];
-
 struct opts {
 	/* actions: */
 	bool dump_orc;

-- 
2.39.0

[tip: objtool/core] objtool: Make struct check_options static

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     3b066cdf0893993dcbfabe9ac443e6ff84df6218
Gitweb:        https://git.kernel.org/tip/3b066cdf0893993dcbfabe9ac443e6ff84df6218
Author:        Thomas Weißschuh <linux@weissschuh.net>
AuthorDate:    Tue, 27 Dec 2022 16:00:58 
Committer:     Josh Poimboeuf <jpoimboe@kernel.org>
CommitterDate: Mon, 30 Jan 2023 16:28:18 -08:00

objtool: Make struct check_options static

It is not used outside of builtin-check.c.

Also remove the unused declaration from builtin.h .

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
Link: https://lore.kernel.org/r/20221216-objtool-memory-v2-2-17968f85a464@weissschuh.net
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 tools/objtool/builtin-check.c           | 2 +-
 tools/objtool/include/objtool/builtin.h | 2 --
 2 files changed, 1 insertion(+), 3 deletions(-)

diff --git a/tools/objtool/builtin-check.c b/tools/objtool/builtin-check.c
index a4f3940..7c17519 100644
--- a/tools/objtool/builtin-check.c
+++ b/tools/objtool/builtin-check.c
@@ -65,7 +65,7 @@ static int parse_hacks(const struct option *opt, const char *str, int unset)
 	return found ? 0 : -1;
 }
 
-const struct option check_options[] = {
+static const struct option check_options[] = {
 	OPT_GROUP("Actions:"),
 	OPT_CALLBACK_OPTARG('h', "hacks", NULL, NULL, "jump_label,noinstr,skylake", "patch toolchain bugs/limitations", parse_hacks),
 	OPT_BOOLEAN('i', "ibt", &opts.ibt, "validate and annotate IBT"),
diff --git a/tools/objtool/include/objtool/builtin.h b/tools/objtool/include/objtool/builtin.h
index fa45044..2a108e6 100644
--- a/tools/objtool/include/objtool/builtin.h
+++ b/tools/objtool/include/objtool/builtin.h
@@ -7,8 +7,6 @@
 
 #include <subcmd/parse-options.h>
 
-extern const struct option check_options[];
-
 struct opts {
 	/* actions: */
 	bool dump_orc;

[tip: objtool/core] objtool: Make struct check_options static

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     cfd66e81799f4a2fdc6447fa99bdb1871f45ff08
Gitweb:        https://git.kernel.org/tip/cfd66e81799f4a2fdc6447fa99bdb1871f45ff08
Author:        Thomas Weißschuh <linux@weissschuh.net>
AuthorDate:    Tue, 27 Dec 2022 16:00:58 
Committer:     Josh Poimboeuf <jpoimboe@kernel.org>
CommitterDate: Wed, 01 Feb 2023 09:15:23 -08:00

objtool: Make struct check_options static

It is not used outside of builtin-check.c.

Also remove the unused declaration from builtin.h .

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
Link: https://lore.kernel.org/r/20221216-objtool-memory-v2-2-17968f85a464@weissschuh.net
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 tools/objtool/builtin-check.c           | 2 +-
 tools/objtool/include/objtool/builtin.h | 2 --
 2 files changed, 1 insertion(+), 3 deletions(-)

diff --git a/tools/objtool/builtin-check.c b/tools/objtool/builtin-check.c
index a4f3940..7c17519 100644
--- a/tools/objtool/builtin-check.c
+++ b/tools/objtool/builtin-check.c
@@ -65,7 +65,7 @@ static int parse_hacks(const struct option *opt, const char *str, int unset)
 	return found ? 0 : -1;
 }
 
-const struct option check_options[] = {
+static const struct option check_options[] = {
 	OPT_GROUP("Actions:"),
 	OPT_CALLBACK_OPTARG('h', "hacks", NULL, NULL, "jump_label,noinstr,skylake", "patch toolchain bugs/limitations", parse_hacks),
 	OPT_BOOLEAN('i', "ibt", &opts.ibt, "validate and annotate IBT"),
diff --git a/tools/objtool/include/objtool/builtin.h b/tools/objtool/include/objtool/builtin.h
index fa45044..2a108e6 100644
--- a/tools/objtool/include/objtool/builtin.h
+++ b/tools/objtool/include/objtool/builtin.h
@@ -7,8 +7,6 @@
 
 #include <subcmd/parse-options.h>
 
-extern const struct option check_options[];
-
 struct opts {
 	/* actions: */
 	bool dump_orc;

[PATCH v2 3/8] objtool: allocate multiple structures with calloc()

[Thomas Weißschuh]

By using calloc() instead of malloc() in a loop, libc does not have to
keep around bookkeeping information for each single structure.

This reduces maximum memory usage while processing vmlinux.o from
3153325 KB to 3035668 KB (-3.7%) on my notebooks "localmodconfig".

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/objtool/elf.c                 | 42 ++++++++++++++++++-------------------
 tools/objtool/include/objtool/elf.h |  4 ++++
 2 files changed, 25 insertions(+), 21 deletions(-)

diff --git a/tools/objtool/elf.c b/tools/objtool/elf.c
index 7eae95f33a72..9c326efb8cd9 100644
--- a/tools/objtool/elf.c
+++ b/tools/objtool/elf.c
@@ -285,13 +285,13 @@ static int read_sections(struct elf *elf)
 	    !elf_alloc_hash(section_name, sections_nr))
 		return -1;
 
+	elf->section_data = calloc(sections_nr, sizeof(*sec));
+	if (!elf->section_data) {
+		perror("calloc");
+		return -1;
+	}
 	for (i = 0; i < sections_nr; i++) {
-		sec = malloc(sizeof(*sec));
-		if (!sec) {
-			perror("malloc");
-			return -1;
-		}
-		memset(sec, 0, sizeof(*sec));
+		sec = &elf->section_data[i];
 
 		INIT_LIST_HEAD(&sec->symbol_list);
 		INIT_LIST_HEAD(&sec->reloc_list);
@@ -423,13 +423,13 @@ static int read_symbols(struct elf *elf)
 	    !elf_alloc_hash(symbol_name, symbols_nr))
 		return -1;
 
+	elf->symbol_data = calloc(symbols_nr, sizeof(*sym));
+	if (!elf->symbol_data) {
+		perror("calloc");
+		return -1;
+	}
 	for (i = 0; i < symbols_nr; i++) {
-		sym = malloc(sizeof(*sym));
-		if (!sym) {
-			perror("malloc");
-			return -1;
-		}
-		memset(sym, 0, sizeof(*sym));
+		sym = &elf->symbol_data[i];
 
 		sym->idx = i;
 
@@ -919,13 +919,13 @@ static int read_relocs(struct elf *elf)
 		sec->base->reloc = sec;
 
 		nr_reloc = 0;
+		sec->reloc_data = calloc(sec->sh.sh_size / sec->sh.sh_entsize, sizeof(*reloc));
+		if (!sec->reloc_data) {
+			perror("calloc");
+			return -1;
+		}
 		for (i = 0; i < sec->sh.sh_size / sec->sh.sh_entsize; i++) {
-			reloc = malloc(sizeof(*reloc));
-			if (!reloc) {
-				perror("malloc");
-				return -1;
-			}
-			memset(reloc, 0, sizeof(*reloc));
+			reloc = &sec->reloc_data[i];
 			switch (sec->sh.sh_type) {
 			case SHT_REL:
 				if (read_rel_reloc(sec, i, reloc, &symndx))
@@ -1458,16 +1458,16 @@ void elf_close(struct elf *elf)
 		list_for_each_entry_safe(sym, tmpsym, &sec->symbol_list, list) {
 			list_del(&sym->list);
 			hash_del(&sym->hash);
-			free(sym);
 		}
 		list_for_each_entry_safe(reloc, tmpreloc, &sec->reloc_list, list) {
 			list_del(&reloc->list);
 			hash_del(&reloc->hash);
-			free(reloc);
 		}
 		list_del(&sec->list);
-		free(sec);
+		free(sec->reloc_data);
 	}
 
+	free(elf->symbol_data);
+	free(elf->section_data);
 	free(elf);
 }
diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h
index bb60fd42b46f..1c90f0ac0d53 100644
--- a/tools/objtool/include/objtool/elf.h
+++ b/tools/objtool/include/objtool/elf.h
@@ -39,6 +39,7 @@ struct section {
 	char *name;
 	int idx;
 	bool changed, text, rodata, noinstr, init, truncate;
+	struct reloc *reloc_data;
 };
 
 struct symbol {
@@ -104,6 +105,9 @@ struct elf {
 	struct hlist_head *section_hash;
 	struct hlist_head *section_name_hash;
 	struct hlist_head *reloc_hash;
+
+	struct section *section_data;
+	struct symbol *symbol_data;
 };
 
 #define OFFSET_STRIDE_BITS	4

-- 
2.39.0

Re: [PATCH v2 3/8] objtool: allocate multiple structures with calloc()

[Josh Poimboeuf]

On Tue, Dec 27, 2022 at 04:00:59PM +0000, Thomas Weißschuh wrote:
> By using calloc() instead of malloc() in a loop, libc does not have to
> keep around bookkeeping information for each single structure.

If we cared about memory leaks, this wouldn't really work because some
structures are added in later, after the reading of the original
sections and symbols.  Luckily we don't care :-)

-- 
Josh

[tip: objtool/core] objtool: Allocate multiple structures with calloc()

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     389244804db54977b9316f0dc027ab69b3c1e8bf
Gitweb:        https://git.kernel.org/tip/389244804db54977b9316f0dc027ab69b3c1e8bf
Author:        Thomas Weißschuh <linux@weissschuh.net>
AuthorDate:    Tue, 27 Dec 2022 16:00:59 
Committer:     Josh Poimboeuf <jpoimboe@kernel.org>
CommitterDate: Mon, 30 Jan 2023 16:28:19 -08:00

objtool: Allocate multiple structures with calloc()

By using calloc() instead of malloc() in a loop, libc does not have to
keep around bookkeeping information for each single structure.

This reduces maximum memory usage while processing vmlinux.o from
3153325 KB to 3035668 KB (-3.7%) on my notebooks "localmodconfig".

Note this introduces memory leaks, because some additional structs get
added to the lists later after reading the symbols and sections from the
original object.  Luckily we don't really care about memory leaks in
objtool.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
Link: https://lore.kernel.org/r/20221216-objtool-memory-v2-3-17968f85a464@weissschuh.net
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 tools/objtool/elf.c                 | 42 ++++++++++++++--------------
 tools/objtool/include/objtool/elf.h |  4 +++-
 2 files changed, 25 insertions(+), 21 deletions(-)

diff --git a/tools/objtool/elf.c b/tools/objtool/elf.c
index 64443a7..6806ce0 100644
--- a/tools/objtool/elf.c
+++ b/tools/objtool/elf.c
@@ -284,13 +284,13 @@ static int read_sections(struct elf *elf)
 	    !elf_alloc_hash(section_name, sections_nr))
 		return -1;
 
+	elf->section_data = calloc(sections_nr, sizeof(*sec));
+	if (!elf->section_data) {
+		perror("calloc");
+		return -1;
+	}
 	for (i = 0; i < sections_nr; i++) {
-		sec = malloc(sizeof(*sec));
-		if (!sec) {
-			perror("malloc");
-			return -1;
-		}
-		memset(sec, 0, sizeof(*sec));
+		sec = &elf->section_data[i];
 
 		INIT_LIST_HEAD(&sec->symbol_list);
 		INIT_LIST_HEAD(&sec->reloc_list);
@@ -422,13 +422,13 @@ static int read_symbols(struct elf *elf)
 	    !elf_alloc_hash(symbol_name, symbols_nr))
 		return -1;
 
+	elf->symbol_data = calloc(symbols_nr, sizeof(*sym));
+	if (!elf->symbol_data) {
+		perror("calloc");
+		return -1;
+	}
 	for (i = 0; i < symbols_nr; i++) {
-		sym = malloc(sizeof(*sym));
-		if (!sym) {
-			perror("malloc");
-			return -1;
-		}
-		memset(sym, 0, sizeof(*sym));
+		sym = &elf->symbol_data[i];
 
 		sym->idx = i;
 
@@ -918,13 +918,13 @@ static int read_relocs(struct elf *elf)
 		sec->base->reloc = sec;
 
 		nr_reloc = 0;
+		sec->reloc_data = calloc(sec->sh.sh_size / sec->sh.sh_entsize, sizeof(*reloc));
+		if (!sec->reloc_data) {
+			perror("calloc");
+			return -1;
+		}
 		for (i = 0; i < sec->sh.sh_size / sec->sh.sh_entsize; i++) {
-			reloc = malloc(sizeof(*reloc));
-			if (!reloc) {
-				perror("malloc");
-				return -1;
-			}
-			memset(reloc, 0, sizeof(*reloc));
+			reloc = &sec->reloc_data[i];
 			switch (sec->sh.sh_type) {
 			case SHT_REL:
 				if (read_rel_reloc(sec, i, reloc, &symndx))
@@ -1453,16 +1453,16 @@ void elf_close(struct elf *elf)
 		list_for_each_entry_safe(sym, tmpsym, &sec->symbol_list, list) {
 			list_del(&sym->list);
 			hash_del(&sym->hash);
-			free(sym);
 		}
 		list_for_each_entry_safe(reloc, tmpreloc, &sec->reloc_list, list) {
 			list_del(&reloc->list);
 			hash_del(&reloc->hash);
-			free(reloc);
 		}
 		list_del(&sec->list);
-		free(sec);
+		free(sec->reloc_data);
 	}
 
+	free(elf->symbol_data);
+	free(elf->section_data);
 	free(elf);
 }
diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h
index bb60fd4..1c90f0a 100644
--- a/tools/objtool/include/objtool/elf.h
+++ b/tools/objtool/include/objtool/elf.h
@@ -39,6 +39,7 @@ struct section {
 	char *name;
 	int idx;
 	bool changed, text, rodata, noinstr, init, truncate;
+	struct reloc *reloc_data;
 };
 
 struct symbol {
@@ -104,6 +105,9 @@ struct elf {
 	struct hlist_head *section_hash;
 	struct hlist_head *section_name_hash;
 	struct hlist_head *reloc_hash;
+
+	struct section *section_data;
+	struct symbol *symbol_data;
 };
 
 #define OFFSET_STRIDE_BITS	4

[tip: objtool/core] objtool: Allocate multiple structures with calloc()

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     8045b8f0b17edf375849f83c80dd05194850b6ed
Gitweb:        https://git.kernel.org/tip/8045b8f0b17edf375849f83c80dd05194850b6ed
Author:        Thomas Weißschuh <linux@weissschuh.net>
AuthorDate:    Tue, 27 Dec 2022 16:00:59 
Committer:     Josh Poimboeuf <jpoimboe@kernel.org>
CommitterDate: Wed, 01 Feb 2023 09:15:23 -08:00

objtool: Allocate multiple structures with calloc()

By using calloc() instead of malloc() in a loop, libc does not have to
keep around bookkeeping information for each single structure.

This reduces maximum memory usage while processing vmlinux.o from
3153325 KB to 3035668 KB (-3.7%) on my notebooks "localmodconfig".

Note this introduces memory leaks, because some additional structs get
added to the lists later after reading the symbols and sections from the
original object.  Luckily we don't really care about memory leaks in
objtool.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
Link: https://lore.kernel.org/r/20221216-objtool-memory-v2-3-17968f85a464@weissschuh.net
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 tools/objtool/elf.c                 | 42 ++++++++++++++--------------
 tools/objtool/include/objtool/elf.h |  4 +++-
 2 files changed, 25 insertions(+), 21 deletions(-)

diff --git a/tools/objtool/elf.c b/tools/objtool/elf.c
index 64443a7..6806ce0 100644
--- a/tools/objtool/elf.c
+++ b/tools/objtool/elf.c
@@ -284,13 +284,13 @@ static int read_sections(struct elf *elf)
 	    !elf_alloc_hash(section_name, sections_nr))
 		return -1;
 
+	elf->section_data = calloc(sections_nr, sizeof(*sec));
+	if (!elf->section_data) {
+		perror("calloc");
+		return -1;
+	}
 	for (i = 0; i < sections_nr; i++) {
-		sec = malloc(sizeof(*sec));
-		if (!sec) {
-			perror("malloc");
-			return -1;
-		}
-		memset(sec, 0, sizeof(*sec));
+		sec = &elf->section_data[i];
 
 		INIT_LIST_HEAD(&sec->symbol_list);
 		INIT_LIST_HEAD(&sec->reloc_list);
@@ -422,13 +422,13 @@ static int read_symbols(struct elf *elf)
 	    !elf_alloc_hash(symbol_name, symbols_nr))
 		return -1;
 
+	elf->symbol_data = calloc(symbols_nr, sizeof(*sym));
+	if (!elf->symbol_data) {
+		perror("calloc");
+		return -1;
+	}
 	for (i = 0; i < symbols_nr; i++) {
-		sym = malloc(sizeof(*sym));
-		if (!sym) {
-			perror("malloc");
-			return -1;
-		}
-		memset(sym, 0, sizeof(*sym));
+		sym = &elf->symbol_data[i];
 
 		sym->idx = i;
 
@@ -918,13 +918,13 @@ static int read_relocs(struct elf *elf)
 		sec->base->reloc = sec;
 
 		nr_reloc = 0;
+		sec->reloc_data = calloc(sec->sh.sh_size / sec->sh.sh_entsize, sizeof(*reloc));
+		if (!sec->reloc_data) {
+			perror("calloc");
+			return -1;
+		}
 		for (i = 0; i < sec->sh.sh_size / sec->sh.sh_entsize; i++) {
-			reloc = malloc(sizeof(*reloc));
-			if (!reloc) {
-				perror("malloc");
-				return -1;
-			}
-			memset(reloc, 0, sizeof(*reloc));
+			reloc = &sec->reloc_data[i];
 			switch (sec->sh.sh_type) {
 			case SHT_REL:
 				if (read_rel_reloc(sec, i, reloc, &symndx))
@@ -1453,16 +1453,16 @@ void elf_close(struct elf *elf)
 		list_for_each_entry_safe(sym, tmpsym, &sec->symbol_list, list) {
 			list_del(&sym->list);
 			hash_del(&sym->hash);
-			free(sym);
 		}
 		list_for_each_entry_safe(reloc, tmpreloc, &sec->reloc_list, list) {
 			list_del(&reloc->list);
 			hash_del(&reloc->hash);
-			free(reloc);
 		}
 		list_del(&sec->list);
-		free(sec);
+		free(sec->reloc_data);
 	}
 
+	free(elf->symbol_data);
+	free(elf->section_data);
 	free(elf);
 }
diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h
index bb60fd4..1c90f0a 100644
--- a/tools/objtool/include/objtool/elf.h
+++ b/tools/objtool/include/objtool/elf.h
@@ -39,6 +39,7 @@ struct section {
 	char *name;
 	int idx;
 	bool changed, text, rodata, noinstr, init, truncate;
+	struct reloc *reloc_data;
 };
 
 struct symbol {
@@ -104,6 +105,9 @@ struct elf {
 	struct hlist_head *section_hash;
 	struct hlist_head *section_name_hash;
 	struct hlist_head *reloc_hash;
+
+	struct section *section_data;
+	struct symbol *symbol_data;
 };
 
 #define OFFSET_STRIDE_BITS	4

[PATCH v2 4/8] objtool: introduce function elf_reloc_set_type

[Thomas Weißschuh]

An upcoming patch needs to perform validation when setting reloc->type
so introduce a helper to contain this validation.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/objtool/check.c               |  6 +++---
 tools/objtool/elf.c                 | 11 ++++++++---
 tools/objtool/include/objtool/elf.h |  2 ++
 3 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/tools/objtool/check.c b/tools/objtool/check.c
index 4350be739f4f..971ee6826de7 100644
--- a/tools/objtool/check.c
+++ b/tools/objtool/check.c
@@ -1355,7 +1355,7 @@ static void annotate_call_site(struct objtool_file *file,
 	 */
 	if (opts.hack_noinstr && insn->sec->noinstr && sym->profiling_func) {
 		if (reloc) {
-			reloc->type = R_NONE;
+			elf_reloc_set_type(reloc, R_NONE);
 			elf_write_reloc(file->elf, reloc);
 		}
 
@@ -1384,7 +1384,7 @@ static void annotate_call_site(struct objtool_file *file,
 			WARN_FUNC("Tail call to __fentry__ !?!?", insn->sec, insn->offset);
 		if (opts.mnop) {
 			if (reloc) {
-				reloc->type = R_NONE;
+				elf_reloc_set_type(reloc, R_NONE);
 				elf_write_reloc(file->elf, reloc);
 			}
 
@@ -1863,7 +1863,7 @@ static int handle_jump_alt(struct objtool_file *file,
 		struct reloc *reloc = insn_reloc(file, orig_insn);
 
 		if (reloc) {
-			reloc->type = R_NONE;
+			elf_reloc_set_type(reloc, R_NONE);
 			elf_write_reloc(file->elf, reloc);
 		}
 		elf_write_insn(file->elf, orig_insn->sec,
diff --git a/tools/objtool/elf.c b/tools/objtool/elf.c
index 9c326efb8cd9..ee355beb0d82 100644
--- a/tools/objtool/elf.c
+++ b/tools/objtool/elf.c
@@ -555,7 +555,7 @@ int elf_add_reloc(struct elf *elf, struct section *sec, unsigned long offset,
 
 	reloc->sec = sec->reloc;
 	reloc->offset = offset;
-	reloc->type = type;
+	elf_reloc_set_type(reloc, type);
 	reloc->sym = sym;
 	reloc->addend = addend;
 
@@ -872,7 +872,7 @@ static int read_rel_reloc(struct section *sec, int i, struct reloc *reloc, unsig
 		WARN_ELF("gelf_getrel");
 		return -1;
 	}
-	reloc->type = GELF_R_TYPE(reloc->rel.r_info);
+	elf_reloc_set_type(reloc, GELF_R_TYPE(reloc->rel.r_info));
 	reloc->addend = 0;
 	reloc->offset = reloc->rel.r_offset;
 	*symndx = GELF_R_SYM(reloc->rel.r_info);
@@ -885,7 +885,7 @@ static int read_rela_reloc(struct section *sec, int i, struct reloc *reloc, unsi
 		WARN_ELF("gelf_getrela");
 		return -1;
 	}
-	reloc->type = GELF_R_TYPE(reloc->rela.r_info);
+	elf_reloc_set_type(reloc, GELF_R_TYPE(reloc->rela.r_info));
 	reloc->addend = reloc->rela.r_addend;
 	reloc->offset = reloc->rela.r_offset;
 	*symndx = GELF_R_SYM(reloc->rela.r_info);
@@ -1471,3 +1471,8 @@ void elf_close(struct elf *elf)
 	free(elf->section_data);
 	free(elf);
 }
+
+void elf_reloc_set_type(struct reloc *reloc, int type)
+{
+	reloc->type = type;
+}
diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h
index 1c90f0ac0d53..33ec6cf72325 100644
--- a/tools/objtool/include/objtool/elf.h
+++ b/tools/objtool/include/objtool/elf.h
@@ -83,6 +83,8 @@ struct reloc {
 	bool jump_table_start;
 };
 
+void elf_reloc_set_type(struct reloc *reloc, int type);
+
 #define ELF_HASH_BITS	20
 
 struct elf {

-- 
2.39.0

[PATCH v2 5/8] objtool: reduce memory usage of struct reloc

[Thomas Weißschuh]

Use a smaller type for the relocation type and move it to a location in
the structure where it avoids wasted padding bytes.

Technically ELF could use up to four bytes for the type.
But until now only types up to number 43 have been defined.

Reduce the size of struct reloc on x86_64 from 120 to 112 bytes.
This structure is allocated a lot and never freed.

This reduces maximum memory usage while processing vmlinux.o from
3035668 KB to 2919716 KB (-3.8%) on my notebooks "localmodconfig".

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/objtool/elf.c                 | 3 +++
 tools/objtool/include/objtool/elf.h | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/tools/objtool/elf.c b/tools/objtool/elf.c
index ee355beb0d82..182452adaa71 100644
--- a/tools/objtool/elf.c
+++ b/tools/objtool/elf.c
@@ -1474,5 +1474,8 @@ void elf_close(struct elf *elf)
 
 void elf_reloc_set_type(struct reloc *reloc, int type)
 {
+	if (type >= (1U << (8 * sizeof(reloc->type))))
+		WARN("reloc->type out of range: %d", type);
+
 	reloc->type = type;
 }
diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h
index 33ec6cf72325..2b5becad5a0a 100644
--- a/tools/objtool/include/objtool/elf.h
+++ b/tools/objtool/include/objtool/elf.h
@@ -77,10 +77,10 @@ struct reloc {
 	struct symbol *sym;
 	struct list_head sym_reloc_entry;
 	unsigned long offset;
-	unsigned int type;
 	s64 addend;
 	int idx;
 	bool jump_table_start;
+	unsigned char type;
 };
 
 void elf_reloc_set_type(struct reloc *reloc, int type);

-- 
2.39.0

Re: [PATCH v2 5/8] objtool: reduce memory usage of struct reloc

[Rong Tao]

Hi, Thomas!

Is this likely to cause most RELOC failures? As shown in the following
example:

	#include <bfd.h>
	#include <stdio.h>

	int main(void)
	{
		printf("%d\n", BFD_RELOC_S12Z_OPR);
		return 0;
	}

The BFD_RELOC_S12Z_OPR equal to 2366.

Best wishes.
Rong Tao

Re: [PATCH v2 5/8] objtool: reduce memory usage of struct reloc

[Thomas Weißschuh]

Hi Rong,

On Thu, Dec 29, 2022 at 09:33:32AM +0800, Rong Tao wrote:
> Is this likely to cause most RELOC failures? As shown in the following
> example:
> 
> 	#include <bfd.h>
> 	#include <stdio.h>
> 
> 	int main(void)
> 	{
> 		printf("%d\n", BFD_RELOC_S12Z_OPR);
> 		return 0;
> 	}
> 
> The BFD_RELOC_S12Z_OPR equal to 2366.

I'm not familiar with libbfd, so please correct me if I'm wrong.

To me BFD_RELOC_S12Z_OPR looks like a value that is only used by libbfd.
objtool does not use libbfd.

The only values that objtools should see for the relocation types are
the R_* constants from elf.h as they are used by the compiler and linker
in the raw elf binary files.

These never exceed the value 255, one byte. Indeed they seem to have
been explicitly chosen like this.

$ grep 'define R_.*NUM' /usr/include/elf.h
#define R_68K_NUM	43
#define R_386_NUM	   44
#define R_SPARC_NUM		253
#define R_MIPS_NUM		128
#define R_ALPHA_NUM		46
#define R_ARM_NUM		256
#define R_390_NUM		62
#define R_CRIS_NUM		20
#define R_X86_64_NUM		43
#define R_MN10300_NUM		35
#define R_M32R_NUM		256	/* Keep this the last entry. */
#define R_TILEPRO_NUM		130
#define R_TILEGX_NUM		130
#define R_RISCV_NUM		59

These _NUM constants are the highest actually used values, plus one.
So all real values are smaller than 256.

Thomas

Re: [PATCH v2 5/8] objtool: reduce memory usage of struct reloc

[Rong Tao]

Hi, Thomas:

In /usr/include/elf.h has:

#define ELF32_R_TYPE(val)		((val) & 0xff)
#define ELF64_R_TYPE(i)			((i) & 0xffffffff)
                                       ^^^^^^^^^^

So, I still feel that keeping 'unsigned int' is a good option. Can we just
use __attribute__((packed)) for wasted padding bytes?

Reviewed-by: Rong Tao <rongtao@cestc.cn>

Re: [PATCH v2 5/8] objtool: reduce memory usage of struct reloc

[Thomas Weißschuh]

Hi Rong,

On Thu, Dec 29, 2022 at 11:29:07AM +0800, Rong Tao wrote:
> Hi, Thomas:
> 
> In /usr/include/elf.h has:
> 
> #define ELF32_R_TYPE(val)		((val) & 0xff)
> #define ELF64_R_TYPE(i)			((i) & 0xffffffff)
>                                        ^^^^^^^^^^
> 
> So, I still feel that keeping 'unsigned int' is a good option. Can we just
> use __attribute__((packed)) for wasted padding bytes?

As the struct contains addresses, packing it will create a ton of
compiler warnings and will be much more likely to break than reducing
the width of 'type'.

Given that reducing the width of 'type'

* is currently absolutely safe,
* is unlikely to break in the future, as the elf designers seem to be
  actively trying to stay within this range anyways,
* saves 8 bytes from a very heavily allocated struct, having a
  measurable impact on the build process,

I continue to propose this aproach.

Let's let the maintainers decide.

Thomas

Re: [PATCH v2 5/8] objtool: reduce memory usage of struct reloc

[Josh Poimboeuf]

On Tue, Dec 27, 2022 at 04:01:01PM +0000, Thomas Weißschuh wrote:
>  void elf_reloc_set_type(struct reloc *reloc, int type)
>  {
> +	if (type >= (1U << (8 * sizeof(reloc->type))))
> +		WARN("reloc->type out of range: %d", type);
> +
>  	reloc->type = type;
>  }
> diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h
> index 33ec6cf72325..2b5becad5a0a 100644
> --- a/tools/objtool/include/objtool/elf.h
> +++ b/tools/objtool/include/objtool/elf.h
> @@ -77,10 +77,10 @@ struct reloc {
>  	struct symbol *sym;
>  	struct list_head sym_reloc_entry;
>  	unsigned long offset;
> -	unsigned int type;
>  	s64 addend;
>  	int idx;
>  	bool jump_table_start;
> +	unsigned char type;
>  };

I'd rather not because

  a) the helper function isn't very intuitive and we'll probably forget
     to use it

  b) some arches need more than 256 types (see for example aarch64 which
     objtool may need to support one of these days)

That said, feel free to rearrange the struct fields to at least get some
of those bytes back.

-- 
Josh

[PATCH v2 6/8] objtool: optimize layout of struct symbol

[Thomas Weißschuh]

Reduce the size of struct symbol on x86_64 from 208 to 200 bytes.
This structure is allocated a lot and never freed.

This reduces maximum memory usage while processing vmlinux.o from
2919716 KB to 2917988 KB (-0.5%) on my notebooks "localmodconfig".

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/objtool/include/objtool/elf.h | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h
index 2b5becad5a0a..f41573ff1f8b 100644
--- a/tools/objtool/include/objtool/elf.h
+++ b/tools/objtool/include/objtool/elf.h
@@ -50,12 +50,11 @@ struct symbol {
 	GElf_Sym sym;
 	struct section *sec;
 	char *name;
-	unsigned int idx;
-	unsigned char bind, type;
+	unsigned int idx, len;
 	unsigned long offset;
-	unsigned int len;
 	unsigned long __subtree_last;
 	struct symbol *pfunc, *cfunc, *alias;
+	unsigned char bind, type;
 	u8 uaccess_safe      : 1;
 	u8 static_call_tramp : 1;
 	u8 retpoline_thunk   : 1;

-- 
2.39.0

[tip: objtool/core] objtool: Optimize layout of struct symbol

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     55156d43d62124a92ab80c8f31661609465e257b
Gitweb:        https://git.kernel.org/tip/55156d43d62124a92ab80c8f31661609465e257b
Author:        Thomas Weißschuh <linux@weissschuh.net>
AuthorDate:    Tue, 27 Dec 2022 16:01:02 
Committer:     Josh Poimboeuf <jpoimboe@kernel.org>
CommitterDate: Mon, 30 Jan 2023 16:28:20 -08:00

objtool: Optimize layout of struct symbol

Reduce the size of struct symbol on x86_64 from 208 to 200 bytes.
This structure is allocated a lot and never freed.

This reduces maximum memory usage while processing vmlinux.o from
2919716 KB to 2917988 KB (-0.5%) on my notebooks "localmodconfig".

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
Link: https://lore.kernel.org/r/20221216-objtool-memory-v2-6-17968f85a464@weissschuh.net
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 tools/objtool/include/objtool/elf.h | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h
index 1c90f0a..ad0024d 100644
--- a/tools/objtool/include/objtool/elf.h
+++ b/tools/objtool/include/objtool/elf.h
@@ -50,12 +50,11 @@ struct symbol {
 	GElf_Sym sym;
 	struct section *sec;
 	char *name;
-	unsigned int idx;
-	unsigned char bind, type;
+	unsigned int idx, len;
 	unsigned long offset;
-	unsigned int len;
 	unsigned long __subtree_last;
 	struct symbol *pfunc, *cfunc, *alias;
+	unsigned char bind, type;
 	u8 uaccess_safe      : 1;
 	u8 static_call_tramp : 1;
 	u8 retpoline_thunk   : 1;

[tip: objtool/core] objtool: Optimize layout of struct symbol

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     21a899f9fc78be6b289ee4627bccadf560930eb5
Gitweb:        https://git.kernel.org/tip/21a899f9fc78be6b289ee4627bccadf560930eb5
Author:        Thomas Weißschuh <linux@weissschuh.net>
AuthorDate:    Tue, 27 Dec 2022 16:01:02 
Committer:     Josh Poimboeuf <jpoimboe@kernel.org>
CommitterDate: Wed, 01 Feb 2023 09:15:24 -08:00

objtool: Optimize layout of struct symbol

Reduce the size of struct symbol on x86_64 from 208 to 200 bytes.
This structure is allocated a lot and never freed.

This reduces maximum memory usage while processing vmlinux.o from
2919716 KB to 2917988 KB (-0.5%) on my notebooks "localmodconfig".

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
Link: https://lore.kernel.org/r/20221216-objtool-memory-v2-6-17968f85a464@weissschuh.net
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 tools/objtool/include/objtool/elf.h | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h
index 1c90f0a..ad0024d 100644
--- a/tools/objtool/include/objtool/elf.h
+++ b/tools/objtool/include/objtool/elf.h
@@ -50,12 +50,11 @@ struct symbol {
 	GElf_Sym sym;
 	struct section *sec;
 	char *name;
-	unsigned int idx;
-	unsigned char bind, type;
+	unsigned int idx, len;
 	unsigned long offset;
-	unsigned int len;
 	unsigned long __subtree_last;
 	struct symbol *pfunc, *cfunc, *alias;
+	unsigned char bind, type;
 	u8 uaccess_safe      : 1;
 	u8 static_call_tramp : 1;
 	u8 retpoline_thunk   : 1;

[PATCH v2 7/8] objtool: optimize layout of struct special_alt

[Thomas Weißschuh]

Reduce the size of struct special_alt from 72 to 64 bytes.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/objtool/include/objtool/special.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/objtool/include/objtool/special.h b/tools/objtool/include/objtool/special.h
index dc4721e19002..86d4af9c5aa9 100644
--- a/tools/objtool/include/objtool/special.h
+++ b/tools/objtool/include/objtool/special.h
@@ -19,6 +19,7 @@ struct special_alt {
 	bool skip_orig;
 	bool skip_alt;
 	bool jump_or_nop;
+	u8 key_addend;
 
 	struct section *orig_sec;
 	unsigned long orig_off;
@@ -27,7 +28,6 @@ struct special_alt {
 	unsigned long new_off;
 
 	unsigned int orig_len, new_len; /* group only */
-	u8 key_addend;
 };
 
 int special_get_alts(struct elf *elf, struct list_head *alts);

-- 
2.39.0

[tip: objtool/core] objtool: Optimize layout of struct special_alt

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     391e6d2abd652b460d9017b2f8a8ded508c09558
Gitweb:        https://git.kernel.org/tip/391e6d2abd652b460d9017b2f8a8ded508c09558
Author:        Thomas Weißschuh <linux@weissschuh.net>
AuthorDate:    Tue, 27 Dec 2022 16:01:03 
Committer:     Josh Poimboeuf <jpoimboe@kernel.org>
CommitterDate: Mon, 30 Jan 2023 16:28:20 -08:00

objtool: Optimize layout of struct special_alt

Reduce the size of struct special_alt from 72 to 64 bytes.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
Link: https://lore.kernel.org/r/20221216-objtool-memory-v2-7-17968f85a464@weissschuh.net
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 tools/objtool/include/objtool/special.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/objtool/include/objtool/special.h b/tools/objtool/include/objtool/special.h
index dc4721e..86d4af9 100644
--- a/tools/objtool/include/objtool/special.h
+++ b/tools/objtool/include/objtool/special.h
@@ -19,6 +19,7 @@ struct special_alt {
 	bool skip_orig;
 	bool skip_alt;
 	bool jump_or_nop;
+	u8 key_addend;
 
 	struct section *orig_sec;
 	unsigned long orig_off;
@@ -27,7 +28,6 @@ struct special_alt {
 	unsigned long new_off;
 
 	unsigned int orig_len, new_len; /* group only */
-	u8 key_addend;
 };
 
 int special_get_alts(struct elf *elf, struct list_head *alts);

[tip: objtool/core] objtool: Optimize layout of struct special_alt

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     a20717aca33b1ff133f513721050fe6c3d7f97b5
Gitweb:        https://git.kernel.org/tip/a20717aca33b1ff133f513721050fe6c3d7f97b5
Author:        Thomas Weißschuh <linux@weissschuh.net>
AuthorDate:    Tue, 27 Dec 2022 16:01:03 
Committer:     Josh Poimboeuf <jpoimboe@kernel.org>
CommitterDate: Wed, 01 Feb 2023 09:15:24 -08:00

objtool: Optimize layout of struct special_alt

Reduce the size of struct special_alt from 72 to 64 bytes.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
Link: https://lore.kernel.org/r/20221216-objtool-memory-v2-7-17968f85a464@weissschuh.net
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 tools/objtool/include/objtool/special.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/objtool/include/objtool/special.h b/tools/objtool/include/objtool/special.h
index dc4721e..86d4af9 100644
--- a/tools/objtool/include/objtool/special.h
+++ b/tools/objtool/include/objtool/special.h
@@ -19,6 +19,7 @@ struct special_alt {
 	bool skip_orig;
 	bool skip_alt;
 	bool jump_or_nop;
+	u8 key_addend;
 
 	struct section *orig_sec;
 	unsigned long orig_off;
@@ -27,7 +28,6 @@ struct special_alt {
 	unsigned long new_off;
 
 	unsigned int orig_len, new_len; /* group only */
-	u8 key_addend;
 };
 
 int special_get_alts(struct elf *elf, struct list_head *alts);

[PATCH v2 8/8] objtool: explicitly cleanup resources on success

[Thomas Weißschuh]

Previously the file was only closed and resources properly freed on
errors, not on normal exits.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/objtool/builtin-check.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tools/objtool/builtin-check.c b/tools/objtool/builtin-check.c
index 7c175198d09f..e11c766b98ce 100644
--- a/tools/objtool/builtin-check.c
+++ b/tools/objtool/builtin-check.c
@@ -229,5 +229,7 @@ int objtool_run(int argc, const char **argv)
 	if (file->elf->changed)
 		return elf_write(file->elf);
 
+	elf_close(file->elf);
+
 	return 0;
 }

-- 
2.39.0

Re: [PATCH v2 8/8] objtool: explicitly cleanup resources on success

[Josh Poimboeuf]

On Tue, Dec 27, 2022 at 04:01:04PM +0000, Thomas Weißschuh wrote:
> Previously the file was only closed and resources properly freed on
> errors, not on normal exits.
> 
> Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
> ---
>  tools/objtool/builtin-check.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/tools/objtool/builtin-check.c b/tools/objtool/builtin-check.c
> index 7c175198d09f..e11c766b98ce 100644
> --- a/tools/objtool/builtin-check.c
> +++ b/tools/objtool/builtin-check.c
> @@ -229,5 +229,7 @@ int objtool_run(int argc, const char **argv)
>  	if (file->elf->changed)
>  		return elf_write(file->elf);
>  
> +	elf_close(file->elf);
> +
>  	return 0;

I'd rather not unless there's a real benefit to doing this, as it will
just slow down the exit.

-- 
Josh

Re: [PATCH v2 0/8] objtool: reduce maximum memory usage

[Thomas Weißschuh]

On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Weißschuh wrote:
> The processing of vmlinux.o with objtool is the most memory-intensive step
> of a kernel build. By reducing the maximum memory usage here we can reduce
> the maximum memory usage of the whole kernel build.
> Therefore memory pressure on memory starved machines is relieved during
> kernel builds and the build is faster as less swapping has to occur.

Friendly ping.

These patches can also applied one by one, the only dependency is from
patch 5 to patch 4.

Thanks,
Thomas

> To: Josh Poimboeuf <jpoimboe@kernel.org>
> To: Peter Zijlstra <peterz@infradead.org>
> Cc: linux-kernel@vger.kernel.org
> Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
> 
> ---
> Changes in v2:
> - Warn on out of range values for reloc->type
> - Also reduce size of struct special_alt
> - Note: v1 did not make it to the lists, only to individual recipients
> 
> ---
> Thomas Weißschuh (8):
>       objtool: make struct entries[] static and const
>       objtool: make struct check_options static
>       objtool: allocate multiple structures with calloc()
>       objtool: introduce function elf_reloc_set_type
>       objtool: reduce memory usage of struct reloc
>       objtool: optimize layout of struct symbol
>       objtool: optimize layout of struct special_alt
>       objtool: explicitly cleanup resources on success
> 
>  tools/objtool/builtin-check.c           |  4 ++-
>  tools/objtool/check.c                   |  6 ++--
>  tools/objtool/elf.c                     | 56 +++++++++++++++++++--------------
>  tools/objtool/include/objtool/builtin.h |  2 --
>  tools/objtool/include/objtool/elf.h     | 13 +++++---
>  tools/objtool/include/objtool/special.h |  2 +-
>  tools/objtool/special.c                 |  6 ++--
>  7 files changed, 51 insertions(+), 38 deletions(-)
> ---
> base-commit: 1b929c02afd37871d5afb9d498426f83432e71c2
> change-id: 20221216-objtool-memory-06db3b8bf111
> 
> Best regards,
> -- 
> Thomas Weißschuh <linux@weissschuh.net>

Re: [PATCH v2 0/8] objtool: reduce maximum memory usage

[Josh Poimboeuf]

On Sun, Jan 29, 2023 at 09:43:39PM +0000, Thomas Weißschuh wrote:
> On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Weißschuh wrote:
> > The processing of vmlinux.o with objtool is the most memory-intensive step
> > of a kernel build. By reducing the maximum memory usage here we can reduce
> > the maximum memory usage of the whole kernel build.
> > Therefore memory pressure on memory starved machines is relieved during
> > kernel builds and the build is faster as less swapping has to occur.
> 
> Friendly ping.
> 
> These patches can also applied one by one, the only dependency is from
> patch 5 to patch 4.

Thanks, I'll go ahead and take five of them now.

-- 
Josh

Re: [PATCH v2 0/8] objtool: reduce maximum memory usage

[Thomas Weißschuh]

On Mon, Jan 30, 2023 at 04:03:56PM -0800, Josh Poimboeuf wrote:
> On Sun, Jan 29, 2023 at 09:43:39PM +0000, Thomas Weißschuh wrote:
> > On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Weißschuh wrote:
> > > The processing of vmlinux.o with objtool is the most memory-intensive step
> > > of a kernel build. By reducing the maximum memory usage here we can reduce
> > > the maximum memory usage of the whole kernel build.
> > > Therefore memory pressure on memory starved machines is relieved during
> > > kernel builds and the build is faster as less swapping has to occur.
> > 
> > Friendly ping.
> > 
> > These patches can also applied one by one, the only dependency is from
> > patch 5 to patch 4.
> 
> Thanks, I'll go ahead and take five of them now.

Thanks.

I have another half-finished series that replaces the doubly-linked
list_heads used by elf.h with a custom singly-linked list.
This would save a few pointers per struct.

Do you think this is worth it?

Thomas

Re: [PATCH v2 0/8] objtool: reduce maximum memory usage

[Josh Poimboeuf]

On Tue, Jan 31, 2023 at 03:54:42AM +0000, Thomas Weißschuh wrote:
> On Mon, Jan 30, 2023 at 04:03:56PM -0800, Josh Poimboeuf wrote:
> > On Sun, Jan 29, 2023 at 09:43:39PM +0000, Thomas Weißschuh wrote:
> > > On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Weißschuh wrote:
> > > > The processing of vmlinux.o with objtool is the most memory-intensive step
> > > > of a kernel build. By reducing the maximum memory usage here we can reduce
> > > > the maximum memory usage of the whole kernel build.
> > > > Therefore memory pressure on memory starved machines is relieved during
> > > > kernel builds and the build is faster as less swapping has to occur.
> > > 
> > > Friendly ping.
> > > 
> > > These patches can also applied one by one, the only dependency is from
> > > patch 5 to patch 4.
> > 
> > Thanks, I'll go ahead and take five of them now.
> 
> Thanks.
> 
> I have another half-finished series that replaces the doubly-linked
> list_heads used by elf.h with a custom singly-linked list.
> This would save a few pointers per struct.
> 
> Do you think this is worth it?

Maybe, depending on the memory savings.

-- 
Josh

Re: [PATCH v2 0/8] objtool: reduce maximum memory usage

[Josh Poimboeuf]

On Tue, Jan 31, 2023 at 09:27:15AM -0800, Josh Poimboeuf wrote:
> On Tue, Jan 31, 2023 at 03:54:42AM +0000, Thomas Weißschuh wrote:
> > On Mon, Jan 30, 2023 at 04:03:56PM -0800, Josh Poimboeuf wrote:
> > > On Sun, Jan 29, 2023 at 09:43:39PM +0000, Thomas Weißschuh wrote:
> > > > On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Weißschuh wrote:
> > > > > The processing of vmlinux.o with objtool is the most memory-intensive step
> > > > > of a kernel build. By reducing the maximum memory usage here we can reduce
> > > > > the maximum memory usage of the whole kernel build.
> > > > > Therefore memory pressure on memory starved machines is relieved during
> > > > > kernel builds and the build is faster as less swapping has to occur.
> > > > 
> > > > Friendly ping.
> > > > 
> > > > These patches can also applied one by one, the only dependency is from
> > > > patch 5 to patch 4.
> > > 
> > > Thanks, I'll go ahead and take five of them now.
> > 
> > Thanks.
> > 
> > I have another half-finished series that replaces the doubly-linked
> > list_heads used by elf.h with a custom singly-linked list.
> > This would save a few pointers per struct.
> > 
> > Do you think this is worth it?
> 
> Maybe, depending on the memory savings.

FYI, there were more memory usage complaints, so Peter worked up a nice
series to do this and more:

  https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/log/?h=objtool/core

-- 
Josh

RE: [PATCH v2 0/8] objtool: reduce maximum memory usage

[David Laight]

From: Thomas Weißschuh.
> Sent: 31 January 2023 03:55
...
> I have another half-finished series that replaces the doubly-linked
> list_heads used by elf.h with a custom singly-linked list.
> This would save a few pointers per struct.
> 
> Do you think this is worth it?

If you allocate the structures in blocks of (say) 256 you
can use an array of pointers to the blocks and then a
32bit index instead of a 64bit pointer.

For real space-saving you might decide that the index can
never exceed 2^^24 and use a bitfield!

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)