[PATCH v2 1/5] kbuild: add a tool to generate a list of files ignored by git

[PATCH v2 1/5] kbuild: add a tool to generate a list of files ignored by git

[Masahiro Yamada]

In short, the motivation of this commit is to build a source package
without cleaning the source tree.

The deb-pkg and (src)rpm-pkg targets first run 'make clean' before
creating a source tarball. Otherwise build artifacts such as *.o,
*.a, etc. would be included in the tarball. Yet, the tarball ends up
containing several garbage files since 'make clean' does not clean
everything.

Cleaning the tree everytime is annoying since it makes the incremental
build impossible. It is desirable to create a source tarball without
cleaning the tree.

In fact, there are some ways to archive this.

The easiest way is 'git archive'. Actually, 'make perf-tar*-src-pkg'
does this way, but I do not like it because it works only when the source
tree is managed by git, and all files you want in the tarball must be
committed in advance.

I want to make it work without relying on git. We can do this.

Files that are not tracked by git are generated files. We can list them
out by parsing the .gitignore files. Of course, .gitignore does not cover
all the cases, but it works well enough.

tar(1) claims to support it:

  --exclude-vcs-ignores

    Exclude files that match patterns read from VCS-specific ignore files.
    Supported files are: .cvsignore, .gitignore, .bzrignore, and .hgignore.

The best scenario would be to use 'tar --exclude-vcs-ignores', but this
option does not work. --exclude-vcs-ignore does not understand any of
the negation (!), preceding slash, following slash, etc.. So, this option
is just useless.

Hence, I wrote this gitignore parser. The previous version [1], written
in Python, was so slow. This version is implemented in C, so it works
much faster.

This tool tarverses the source tree, parsing the .gitignore files. It
prints the file paths that are not tracked by git. This is suitable to
be passed to tar's --exclude-from= option.

[How to test this tool]

  $ git clean -dfx
  $ make -s -j$(nproc) defconfig all                       # or allmodconifg or whatever
  $ git archive -o ../linux1.tar --prefix=./ HEAD
  $ tar tf ../linux1.tar | LANG=C sort > ../file-list1     # files emitted by 'git archive'
  $ make scripts_exclude
    HOSTCC  scripts/gen-exclude
  $ scripts/gen-exclude --prefix=./ -o ../exclude-list
  $ tar cf ../linux2.tar --exclude-from=../exclude-list .
  $ tar tf ../linux2.tar | LANG=C sort > ../file-list2     # files emitted by 'tar'
  $ diff  ../file-list1 ../file-list2 | grep -E '^(<|>)'
  < ./Documentation/devicetree/bindings/.yamllint
  < ./drivers/clk/.kunitconfig
  < ./drivers/gpu/drm/tests/.kunitconfig
  < ./drivers/gpu/drm/vc4/tests/.kunitconfig
  < ./drivers/hid/.kunitconfig
  < ./fs/ext4/.kunitconfig
  < ./fs/fat/.kunitconfig
  < ./kernel/kcsan/.kunitconfig
  < ./lib/kunit/.kunitconfig
  < ./mm/kfence/.kunitconfig
  < ./net/sunrpc/.kunitconfig
  < ./tools/testing/selftests/arm64/tags/
  < ./tools/testing/selftests/arm64/tags/.gitignore
  < ./tools/testing/selftests/arm64/tags/Makefile
  < ./tools/testing/selftests/arm64/tags/run_tags_test.sh
  < ./tools/testing/selftests/arm64/tags/tags_test.c
  < ./tools/testing/selftests/kvm/.gitignore
  < ./tools/testing/selftests/kvm/Makefile
  < ./tools/testing/selftests/kvm/config
  < ./tools/testing/selftests/kvm/settings

The source tarball contains most of files that are tracked by git. You
see some diffs, but it is just because some .gitignore files are wrong.

  $ git ls-files -i -c --exclude-per-directory=.gitignore
  Documentation/devicetree/bindings/.yamllint
  drivers/clk/.kunitconfig
  drivers/gpu/drm/tests/.kunitconfig
  drivers/hid/.kunitconfig
  fs/ext4/.kunitconfig
  fs/fat/.kunitconfig
  kernel/kcsan/.kunitconfig
  lib/kunit/.kunitconfig
  mm/kfence/.kunitconfig
  tools/testing/selftests/arm64/tags/.gitignore
  tools/testing/selftests/arm64/tags/Makefile
  tools/testing/selftests/arm64/tags/run_tags_test.sh
  tools/testing/selftests/arm64/tags/tags_test.c
  tools/testing/selftests/kvm/.gitignore
  tools/testing/selftests/kvm/Makefile
  tools/testing/selftests/kvm/config
  tools/testing/selftests/kvm/settings

[1]: https://lore.kernel.org/all/20230128173843.765212-1-masahiroy@kernel.org/

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

Changes in v2:
  - Reimplement in C

 Makefile              |   4 +
 scripts/.gitignore    |   1 +
 scripts/Makefile      |   2 +-
 scripts/gen-exclude.c | 639 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 645 insertions(+), 1 deletion(-)
 create mode 100644 scripts/gen-exclude.c

diff --git a/Makefile b/Makefile
index 339121558928..01c6f4ff515f 100644
--- a/Makefile
+++ b/Makefile
@@ -1652,6 +1652,10 @@ distclean: mrproper
 %pkg: include/config/kernel.release FORCE
 	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.package $@
 
+PHONY += scripts_exclude
+scripts_exclude: scripts_basic
+	$(Q)$(MAKE) $(build)=scripts scripts/gen-exclude
+
 # Brief documentation of the typical targets used
 # ---------------------------------------------------------------------------
 
diff --git a/scripts/.gitignore b/scripts/.gitignore
index 6e9ce6720a05..7f433bc1461c 100644
--- a/scripts/.gitignore
+++ b/scripts/.gitignore
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0-only
 /asn1_compiler
+/gen-exclude
 /generate_rust_target
 /insert-sys-cert
 /kallsyms
diff --git a/scripts/Makefile b/scripts/Makefile
index 32b6ba722728..5dcd7f57607f 100644
--- a/scripts/Makefile
+++ b/scripts/Makefile
@@ -38,7 +38,7 @@ HOSTCFLAGS_sorttable.o += -DMCOUNT_SORT_ENABLED
 endif
 
 # The following programs are only built on demand
-hostprogs += unifdef
+hostprogs += gen-exclude unifdef
 
 # The module linker script is preprocessed on demand
 targets += module.lds
diff --git a/scripts/gen-exclude.c b/scripts/gen-exclude.c
new file mode 100644
index 000000000000..3f23b9c934e2
--- /dev/null
+++ b/scripts/gen-exclude.c
@@ -0,0 +1,639 @@
+// SPDX-License-Identifier: GPL-2.0-only
+//
+// Traverse the source tree, parsing all .gitignore files, and print file paths
+// that are not tracked by git.
+// The output is suitable to the --exclude-from option of tar.
+//
+// Copyright (C) 2023 Masahiro Yamada <masahiroy@kernel.org>
+
+#include <dirent.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <fnmatch.h>
+#include <getopt.h>
+#include <stdarg.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/types.h>
+#include <unistd.h>
+#include <sys/stat.h>
+
+// struct pattern - represent an ignore pattern (a line in .gitignroe)
+// @negate:          negate the pattern (prefixing '!')
+// @dir_only:        only matches directories (trailing '/')
+// @path_match:      true if the glob pattern is a path instead of a file name
+// @double_asterisk: true if the glob pattern contains double asterisks ('**')
+// @glob:            glob pattern
+struct pattern {
+	bool negate;
+	bool dir_only;
+	bool path_match;
+	bool double_asterisk;
+	char glob[];
+};
+
+// struct gitignore - represent a .gitignore file
+// @nr_patterns: a number of patterns that exists in this .gitignore
+// @patterns:    array of pointers of patterns
+struct gitignore {
+	unsigned int nr_patterns;
+	struct pattern **patterns;
+};
+
+static struct gitignore *gitignores;
+static int nr_gitignores;
+static unsigned int alloced_gitignores;
+
+static bool debug_on;
+static int depth;
+static FILE *out_fp;
+static char *prefix = "";
+static char *progname;
+
+static void __attribute__ ((noreturn)) perror_exit(const char *s)
+{
+	perror(s);
+	exit(EXIT_FAILURE);
+}
+
+static void __attribute__ ((noreturn)) error_exit(const char *fmt, ...)
+{
+	va_list args;
+
+	fprintf(stderr, "%s: error: ", progname);
+
+	va_start(args, fmt);
+	vfprintf(stderr, fmt, args);
+	va_end(args);
+
+	exit(EXIT_FAILURE);
+}
+
+static void debug(const char *fmt, ...)
+{
+	va_list args;
+	int i;
+
+	if (!debug_on)
+		return;
+
+	fprintf(stderr, "[DEBUG]");
+
+	for (i = 0; i < depth * 2; i++)
+		fputc(' ', stderr);
+
+	va_start(args, fmt);
+	vfprintf(stderr, fmt, args);
+	va_end(args);
+}
+
+static void *xrealloc(void *ptr, size_t size)
+{
+	ptr = realloc(ptr, size);
+	if (!ptr)
+		perror_exit(progname);
+
+	return ptr;
+}
+
+static void *xmalloc(size_t size)
+{
+	return xrealloc(NULL, size);
+}
+
+static bool simple_match(const char *string, const char *pattern)
+{
+	return fnmatch(pattern, string, FNM_PATHNAME) == 0;
+}
+
+// Handle double asterisks ("**") matching.
+// FIXME:
+//  This function does not work if double asterisks apppear multiple times,
+//  like "foo/**/bar/**/baz".
+static bool double_asterisk_match(const char *path, const char *pattern)
+{
+	const char *p;
+	int slash_diff = 0;
+	char *modified_pattern;
+	size_t len;
+	bool result = false;
+	char *q;
+
+	for (p = path; *p; p++)
+		if (*p == '/')
+			slash_diff++;
+
+	for (p = pattern; *p; p++)
+		if (*p == '/')
+			slash_diff--;
+
+	len = strlen(pattern) + 1;
+
+	if (slash_diff > 0)
+		len += slash_diff * 2;
+	modified_pattern = xmalloc(len);
+
+	q = modified_pattern;
+	for (p = pattern; *p; p++) {
+		if (*p == '*' && *(p + 1) == '*' && *(p + 2) == '/') {
+			// "**/" means zero of more sequences of '*/".
+			// "foo**/bar" matches "foobar", "foo*/bar",
+			// "foo*/*/bar", etc.
+			while (slash_diff-- > 0) {
+				*q++ = '*';
+				*q++ = '/';
+			}
+
+			if (slash_diff == 0) {
+				*q++ = '*';
+				*q++ = '/';
+			}
+
+			if (slash_diff < 0)
+				slash_diff++;
+			p += 2;
+		} else if (*p == '/' && *(p + 1) == '*' && *(p + 2) == '*' &&
+			   *(p + 3) == '\0') {
+			// A trailing "/**" matches everything inside.
+			while (slash_diff-- >= 0) {
+				*q++ = '/';
+				*q++ = '*';
+			}
+
+			p += 2;
+		} else {
+			// Copy other patterns as-is.
+			// Other consecutive asterisks are considered regular
+			// asterisks, but fnmatch() handle them like that.
+			*q++ = *p;
+		}
+	}
+
+	*q = '\0';
+
+	result = simple_match(path, modified_pattern);
+
+	free(modified_pattern);
+
+	return result;
+}
+
+// Return true if the given path is ignored by git.
+static bool is_ignored(const char *path, const char *name, bool is_dir)
+{
+	int i, j;
+
+	// ignore more patterns that are not specified by .gitignore
+	if ((is_dir && !strcmp(name, ".git")) ||
+	    simple_match(name, "*.rej") || simple_match(name, "*.mbx"))
+		return true;
+
+	// search the .gitignore in the same directory, and then in any
+	// parent directory up to the top-level.
+	for (i = nr_gitignores - 1; i >= 0; i--) {
+		// search the patterns in the reverse order because the last
+		// matching pattern wins.
+		for (j = gitignores[i].nr_patterns - 1; j >= 0; j--) {
+			struct pattern *pat = gitignores[i].patterns[j];
+
+			if (!is_dir && pat->dir_only)
+				continue;
+
+			if (!pat->path_match) {
+				// If the pattern has no slash at the beginning
+				// or middle, it matches against the basename.
+				// Most cases fall into this.
+				// Double asterisks behave as regular asterisks.
+				if (!simple_match(name, pat->glob))
+					continue;
+			} else if (!pat->double_asterisk) {
+				// Unless the pattern has double asterisks, it
+				// is still simple but matches against the path.
+				if (!simple_match(path, pat->glob))
+					continue;
+			} else {
+				// Double asterisks with a slash.
+				// This is complex, but rare.
+				if (!double_asterisk_match(path, pat->glob))
+					continue;
+			}
+
+			debug("%s: matches %s%s%s\n", path, pat->negate ? "!" : "",
+			      pat->glob, pat->dir_only ? "/" : "");
+
+			return !pat->negate;
+		}
+	}
+
+	debug("%s: no match\n", path);
+
+	return false;
+}
+
+static void *load_gitignore(const char *dirpath)
+{
+	struct stat st;
+	int fd;
+	char *buf;
+	int ret;
+	char path[PATH_MAX];
+
+	ret = snprintf(path, sizeof(path), "%s/.gitignore", dirpath);
+	if (ret >= sizeof(path))
+		error_exit("%s: too long path was truncated\n", path);
+
+	// If .gitignore does not exist in this directory, open() fails.
+	// It is ok, just skip it.
+	fd = open(path, O_RDONLY);
+	if (fd < 0)
+		return NULL;
+
+	if (fstat(fd, &st) < 0)
+		perror_exit(path);
+
+	buf = xmalloc(st.st_size + 1);
+	if (read(fd, buf, st.st_size) != st.st_size)
+		perror_exit(path);
+
+	buf[st.st_size] = '\0';
+	if (close(fd))
+		perror_exit(path);
+
+	return buf;
+}
+
+// Check if the substring @needle is contained in the string that starts at @p
+// and ends at @q. (just like strstr() with unterminated 'haystack')
+static bool contain_str(const char *p, const char *q, const char *needle)
+{
+	size_t len = strlen(needle);
+	int i;
+
+	while (p <= q) {
+		bool match = true;
+
+		for (i = 0; i < len; i++) {
+			if (needle[i] != p[i]) {
+				match = false;
+				break;
+			}
+		}
+
+		if (match)
+			return true;
+		p++;
+	}
+
+	return false;
+}
+
+// Check if the unquoted sequence of the given charcter @c is found at the end
+// of the string that starts at @p and ends at @q. If found, return the
+// pointer to such a sequence, otherwise NULL.
+static const char *find_trailing_chars(const char *p, const char *q, char c)
+{
+	bool quoted = false;
+	const char *found = NULL;
+
+	while (p <= q) {
+		if (!quoted && *p == c) {
+			if (!found)
+				found = p;
+		} else {
+			found = NULL;
+
+			if (!quoted && *p == '\\')
+				quoted = true;
+			else
+				quoted = false;
+		}
+
+		p++;
+	}
+
+	return found;
+}
+
+// Parse '.gitignore' in the given directory.
+static void parse_gitignore(const char *dirpath, struct gitignore *g)
+{
+	int alloced_patterns = 0;
+	struct pattern *pat;
+	void *buf;
+	const char *p;
+
+	g->nr_patterns = 0;
+	g->patterns = NULL;
+
+	buf = load_gitignore(dirpath);
+	if (!buf)
+		return;
+
+	debug("Parse %s/.gitignore\n", dirpath);
+
+	p = buf;
+
+	while (*p) {
+		const char *q, *e, *found;
+		bool negate = false;
+		bool dir_only = false;
+		bool path_match = false;
+		bool double_asterisk = false;
+		int len;
+
+		e = p;
+
+		while (*e && *e != '\n')
+			e++;
+
+		q = e - 1;
+
+		// skip comments
+		if (*p == '#')
+			goto next;
+
+		// trailing spaces are ignored unless they are quoted with
+		// backslash.
+		found = find_trailing_chars(p, q, ' ');
+		if (found)
+			q = found - 1;
+
+		// The prefix '!' negates the pattern
+		if (*p == '!') {
+			p++;
+			negate = true;
+		}
+
+		// If there is slash(es) that is not escaped at the end of the pattern,
+		// it matches only directories.
+		found = find_trailing_chars(p, q, '/');
+		if (found) {
+			dir_only = true;
+			q = found - 1;
+		}
+
+		// skip empty lines
+		if (p > q)
+			goto next;
+
+		// It is tricky to support double asterisks.
+		// Mark it to handle it specially later.
+		if (contain_str(p, q, "**/") || contain_str(p, q, "/**"))
+			double_asterisk = true;
+
+		if (contain_str(p, q, "/")) {
+			if (*p == '/')
+				p++;
+			path_match = true;
+		}
+
+		len = q - p + 1;
+
+		if (path_match)
+			// we need more room to store dirpath and '/'
+			len += strlen(dirpath) + 1;
+
+		pat = xmalloc(sizeof(*pat) + len + 1);
+		pat->negate = negate;
+		pat->dir_only = dir_only;
+		pat->path_match = path_match;
+		pat->double_asterisk = double_asterisk;
+
+		pat->glob[0] = '\0';
+
+		if (path_match) {
+			strcpy(pat->glob, dirpath);
+			strcat(pat->glob, "/");
+		}
+
+		strncat(pat->glob, p, q - p + 1);
+
+		debug("Add pattern: %s%s%s\n", pat->negate ? "!" : "",
+		      pat->glob, pat->dir_only ? "/" : "");
+
+		if (g->nr_patterns >= alloced_patterns) {
+			alloced_patterns += 128;
+			g->patterns = xrealloc(g->patterns,
+					       sizeof(*g->patterns) * alloced_patterns);
+		}
+
+		g->patterns[g->nr_patterns++] = pat;
+
+next:
+		if (!*e)
+			break;
+
+		p = e + 1;
+	}
+
+	free(buf);
+}
+
+// Add a gitignore entry every time we descend a directory level.
+// If .gitignore does not exist in the directory, add an empty entry.
+static void add_gitignore(const char *dirpath)
+{
+	if (nr_gitignores >= alloced_gitignores) {
+		alloced_gitignores += 32;
+		gitignores = xrealloc(gitignores,
+				      sizeof(*gitignores) * alloced_gitignores);
+	}
+
+	parse_gitignore(dirpath, &gitignores[nr_gitignores]);
+
+	nr_gitignores++;
+}
+
+// Pop a gitignore entry every time we ascend back a directory level.
+static void remove_gitignore(void)
+{
+	struct gitignore *g;
+
+	nr_gitignores--;
+	if (nr_gitignores < 0)
+		error_exit("BUG\n");
+
+	g = &gitignores[nr_gitignores];
+
+	while (g->nr_patterns-- > 0)
+		free(g->patterns[g->nr_patterns]);
+
+	free(g->patterns);
+}
+
+// If we find an ignored file path, print it.
+static void print_path(const char *path)
+{
+	if (strlen(path) < 2)
+		error_exit("BUG\n");
+
+	fprintf(out_fp, "%s%s\n", prefix, path + 2);
+}
+
+// Traverse the entire directory tree, parsing .gitignore files.
+// Print file paths that are not tracked by git.
+//
+// Return true if all files under the directory are ignored, false otherwise.
+static bool traverse_directory(const char *dirpath)
+{
+	DIR *dirp;
+	bool all_ignored = true;
+
+	debug("Enter[%d]: %s\n", depth, dirpath);
+	depth++;
+
+	// We do not know whether .gitignore exists in this directory or not.
+	// Anyway, try to open it.
+	add_gitignore(dirpath);
+
+	dirp = opendir(dirpath);
+	if (!dirp)
+		perror_exit(dirpath);
+
+	while (1) {
+		char path[PATH_MAX];
+		struct dirent *d;
+		int ret;
+
+		errno = 0;
+		d = readdir(dirp);
+		if (!d) {
+			// readdir() returns NULL on the end of the directory
+			// steam, and also on an error. To distinguish them,
+			// error should be checked.
+			if (errno)
+				perror_exit(dirpath);
+			break;
+		}
+
+		if (!strcmp(d->d_name, "..") || !strcmp(d->d_name, "."))
+			continue;
+
+		ret = snprintf(path, sizeof(path), "%s/%s", dirpath, d->d_name);
+		if (ret >= sizeof(path))
+			error_exit("%s: too long path was truncated\n", path);
+
+		if (is_ignored(path, d->d_name, d->d_type & DT_DIR)) {
+			debug("Ignore: %s\n", path);
+			print_path(path);
+		} else {
+			if ((d->d_type & DT_DIR) && !(d->d_type & DT_LNK)) {
+				if (!traverse_directory(path))
+					all_ignored = false;
+			} else {
+				all_ignored = false;
+			}
+		}
+	}
+
+	if (closedir(dirp))
+		perror_exit(dirpath);
+
+	// If all the files under this directory, ignore this directory as well.
+	// This is useful to not create an empty directory in the tarball.
+	if (all_ignored)
+		print_path(dirpath);
+
+	remove_gitignore();
+
+	depth--;
+	debug("Leave[%d]: %s\n", depth, dirpath);
+
+	return all_ignored;
+}
+
+static void usage(void)
+{
+	fprintf(stderr,
+		"usage: %s [options]\n"
+		"\n"
+		"Print files that are not ignored by git\n"
+		"\n"
+		"options:\n"
+		"  -d, --debug            print debug messages to stderr\n"
+		"  -h, --help             show this help message and exit\n"
+		"  -o, --output FILE      output to a file (default: '-', i.e. stdout)\n"
+		"  -p, --prefix PREFIX    prefix added to each path\n"
+		"  -r, --rootdir DIR      root of the source tree (default: current working directory):\n",
+		progname);
+}
+
+int main(int argc, char *argv[])
+{
+	const char *rootdir = ".";
+	const char *output = "-";
+
+	progname = strrchr(argv[0], '/');
+	if (progname)
+		progname++;
+	else
+		progname = argv[0];
+
+	while (1) {
+		static struct option long_options[] = {
+			{"debug",   no_argument,       NULL, 'd'},
+			{"help",    no_argument,       NULL, 'h'},
+			{"output",  required_argument, NULL, 'o'},
+			{"prefix",  required_argument, NULL, 'p'},
+			{"rootdir", required_argument, NULL, 'r'},
+			{},
+		};
+
+		int c = getopt_long(argc, argv, "dho:p:r:", long_options, NULL);
+
+		if (c == -1)
+			break;
+
+		switch (c) {
+		case 'd':
+			debug_on = true;
+			break;
+		case 'h':
+			usage();
+			exit(0);
+		case 'o':
+			output = optarg;
+			break;
+		case 'p':
+			prefix = optarg;
+			break;
+		case 'r':
+			rootdir = optarg;
+			break;
+		case '?':
+			usage();
+			/* fallthrough */
+		default:
+			exit(EXIT_FAILURE);
+		}
+	}
+
+	if (chdir(rootdir))
+		perror_exit(rootdir);
+
+	if (strcmp(output, "-")) {
+		out_fp = fopen(output, "w");
+		if (!out_fp)
+			perror_exit(output);
+	} else {
+		out_fp = stdout;
+	}
+
+	traverse_directory(".");
+
+	free(gitignores);
+
+	if (nr_gitignores != 0 || depth != 0)
+		error_exit("BUG\n");
+
+	fflush(out_fp);
+	if (ferror(out_fp))
+		error_exit("not all data was written to the output\n");
+
+	if (fclose(out_fp))
+		perror_exit(output);
+
+	return 0;
+}
-- 
2.34.1

[PATCH v2 2/5] kbuild: deb-pkg: create source package without cleaning

[Masahiro Yamada]

If you run 'make deb-pkg', all objects are lost due to 'make clean',
which makes the incremental builds impossible.

Instead of cleaning, pass the exclude list to tar's --exclude-from
option.

Previously, *.diff.gz contained some check-in files such as
.clang-format, .cocciconfig.

With this commit, *.diff.gz will only contain the .config and debian/.
The other source files will go into the tarball.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

(no changes since v1)

 scripts/Makefile.package | 31 ++++++++++++++++++++++++++-----
 scripts/package/mkdebian | 25 +++++++++++++++++++++++++
 2 files changed, 51 insertions(+), 5 deletions(-)

diff --git a/scripts/Makefile.package b/scripts/Makefile.package
index dfbf40454a99..5ac0a2dec01d 100644
--- a/scripts/Makefile.package
+++ b/scripts/Makefile.package
@@ -50,6 +50,25 @@ fi ; \
 tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \
 	--transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3)
 
+# Source Tarball
+# ---------------------------------------------------------------------------
+
+PHONY += gen-exclude
+gen-exclude:
+	$(Q)$(MAKE) -f $(srctree)/Makefile scripts_exclude
+
+quiet_cmd_exclude_list = GEN     $@
+      cmd_exclude_list = scripts/gen-exclude --prefix=./ --rootdir=$(srctree) > $@; echo "./$@" >> $@
+
+.exclude-list: gen-exclude
+	$(call cmd,exclude_list)
+
+quiet_cmd_tar = TAR     $@
+      cmd_tar = tar -I $(KGZIP) -c -f $@ -C $(srctree) --exclude-from=$< --exclude=./$@ --transform 's:^\.:linux:S' .
+
+%.tar.gz: .exclude-list
+	$(call cmd,tar)
+
 # rpm-pkg
 # ---------------------------------------------------------------------------
 PHONY += rpm-pkg
@@ -81,12 +100,11 @@ binrpm-pkg:
 
 PHONY += deb-pkg
 deb-pkg:
-	$(MAKE) clean
 	$(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian
-	$(call cmd,src_tar,$(KDEB_SOURCENAME))
-	origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
-		mv $(KDEB_SOURCENAME).tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
-	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) --source-option=-sP -i.git -us -uc
+	$(Q)origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
+		$(MAKE) -f $(srctree)/scripts/Makefile.package ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
+	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
+		--build=source,binary --source-option=-sP -nc -us -uc
 
 PHONY += bindeb-pkg
 bindeb-pkg:
@@ -174,4 +192,7 @@ help:
 	@echo '  perf-tarxz-src-pkg  - Build $(perf-tar).tar.xz source tarball'
 	@echo '  perf-tarzst-src-pkg - Build $(perf-tar).tar.zst source tarball'
 
+PHONY += FORCE
+FORCE:
+
 .PHONY: $(PHONY)
diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
index c3bbef7a6754..2f612617cbcf 100755
--- a/scripts/package/mkdebian
+++ b/scripts/package/mkdebian
@@ -84,6 +84,8 @@ set_debarch() {
 	fi
 }
 
+rm -rf debian
+
 # Some variables and settings used throughout the script
 version=$KERNELRELEASE
 if [ -n "$KDEB_PKGVERSION" ]; then
@@ -135,6 +137,29 @@ fi
 mkdir -p debian/source/
 echo "1.0" > debian/source/format
 
+# Ugly: ignore anything except .config or debian/
+# (is there a cleaner way to do this?)
+cat<<'EOF' > debian/source/local-options
+diff-ignore
+
+extend-diff-ignore = ^[^.d]
+
+extend-diff-ignore = ^\.[^c]
+extend-diff-ignore = ^\.c($|[^o])
+extend-diff-ignore = ^\.co($|[^n])
+extend-diff-ignore = ^\.con($|[^f])
+extend-diff-ignore = ^\.conf($|[^i])
+extend-diff-ignore = ^\.confi($|[^g])
+extend-diff-ignore = ^\.config.
+
+extend-diff-ignore = ^d($|[^e])
+extend-diff-ignore = ^de($|[^b])
+extend-diff-ignore = ^deb($|[^i])
+extend-diff-ignore = ^debi($|[^a])
+extend-diff-ignore = ^debia($|[^n])
+extend-diff-ignore = ^debian[^/]
+EOF
+
 echo $debarch > debian/arch
 extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)"
 extra_build_depends="$extra_build_depends, $(if_enabled_echo CONFIG_SYSTEM_TRUSTED_KEYRING libssl-dev:native)"
-- 
2.34.1

[PATCH v2 3/5] kbuild: rpm-pkg: build binary packages from source rpm

[Masahiro Yamada]

The build rules of rpm-pkg and srcrpm-pkg are almost the same.
Remove the code duplication.

Change rpm-pkg to build binary packages from the source package generated
by srcrpm-pkg.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

(no changes since v1)

 scripts/Makefile.package | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/scripts/Makefile.package b/scripts/Makefile.package
index 5ac0a2dec01d..97e146885e53 100644
--- a/scripts/Makefile.package
+++ b/scripts/Makefile.package
@@ -72,11 +72,9 @@ quiet_cmd_tar = TAR     $@
 # rpm-pkg
 # ---------------------------------------------------------------------------
 PHONY += rpm-pkg
-rpm-pkg:
-	$(MAKE) clean
-	$(CONFIG_SHELL) $(MKSPEC) >$(objtree)/kernel.spec
-	$(call cmd,src_tar,$(KERNELPATH),kernel.spec)
-	+rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -ta $(KERNELPATH).tar.gz \
+rpm-pkg: srpm = $(shell rpmspec --srpm --query --queryformat='%{name}-%{VERSION}-%{RELEASE}.src.rpm' kernel.spec)
+rpm-pkg: srcrpm-pkg
+	+rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -rb $(srpm) \
 	--define='_smp_mflags %{nil}'
 
 # srcrpm-pkg
-- 
2.34.1

[PATCH v2 4/5] kbuild: srcrpm-pkg: create source package without cleaning

[Masahiro Yamada]

If you run 'make (src)rpm-pkg', all objects are lost due to 'make clean',
which makes the incremental builds impossible.

Instead of cleaning, pass the exclude list to tar's --exclude-from
option.

Previously, the .config was contained in the source tarball.

With this commit, the source rpm consists of separate linux.tar.gz
and .config.

Remove stale comments. Now, 'make (src)rpm-pkg' works with O= option.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

(no changes since v1)

 scripts/Makefile.package | 50 +++-------------------------------------
 scripts/package/mkspec   |  8 +++----
 2 files changed, 7 insertions(+), 51 deletions(-)

diff --git a/scripts/Makefile.package b/scripts/Makefile.package
index 97e146885e53..0e16ecfddad6 100644
--- a/scripts/Makefile.package
+++ b/scripts/Makefile.package
@@ -3,53 +3,11 @@
 
 include $(srctree)/scripts/Kbuild.include
 
-# RPM target
-# ---------------------------------------------------------------------------
-# The rpm target generates two rpm files:
-# /usr/src/packages/SRPMS/kernel-2.6.7rc2-1.src.rpm
-# /usr/src/packages/RPMS/i386/kernel-2.6.7rc2-1.<arch>.rpm
-# The src.rpm files includes all source for the kernel being built
-# The <arch>.rpm includes kernel configuration, modules etc.
-#
-# Process to create the rpm files
-# a) clean the kernel
-# b) Generate .spec file
-# c) Build a tar ball, using symlink to make kernel version
-#    first entry in the path
-# d) and pack the result to a tar.gz file
-# e) generate the rpm files, based on kernel.spec
-# - Use /. to avoid tar packing just the symlink
-
-# Note that the rpm-pkg target cannot be used with KBUILD_OUTPUT,
-# but the binrpm-pkg target can; for some reason O= gets ignored.
-
-# Remove hyphens since they have special meaning in RPM filenames
-KERNELPATH := kernel-$(subst -,_,$(KERNELRELEASE))
 KDEB_SOURCENAME ?= linux-upstream
 KBUILD_PKG_ROOTCMD ?="fakeroot -u"
 export KDEB_SOURCENAME
-# Include only those top-level files that are needed by make, plus the GPL copy
-TAR_CONTENT := Documentation LICENSES arch block certs crypto drivers fs \
-               include init io_uring ipc kernel lib mm net rust \
-               samples scripts security sound tools usr virt \
-               .config Makefile \
-               Kbuild Kconfig COPYING $(wildcard localversion*)
 MKSPEC     := $(srctree)/scripts/package/mkspec
 
-quiet_cmd_src_tar = TAR     $(2).tar.gz
-      cmd_src_tar = \
-if test "$(objtree)" != "$(srctree)"; then \
-	echo >&2; \
-	echo >&2 "  ERROR:"; \
-	echo >&2 "  Building source tarball is not possible outside the"; \
-	echo >&2 "  kernel source tree. Don't set KBUILD_OUTPUT, or use the"; \
-	echo >&2 "  binrpm-pkg or bindeb-pkg target instead."; \
-	echo >&2; \
-	false; \
-fi ; \
-tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \
-	--transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3)
-
 # Source Tarball
 # ---------------------------------------------------------------------------
 
@@ -80,12 +38,10 @@ rpm-pkg: srcrpm-pkg
 # srcrpm-pkg
 # ---------------------------------------------------------------------------
 PHONY += srcrpm-pkg
-srcrpm-pkg:
-	$(MAKE) clean
+srcrpm-pkg: linux.tar.gz
 	$(CONFIG_SHELL) $(MKSPEC) >$(objtree)/kernel.spec
-	$(call cmd,src_tar,$(KERNELPATH),kernel.spec)
-	+rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -ts $(KERNELPATH).tar.gz \
-	--define='_smp_mflags %{nil}' --define='_srcrpmdir $(srctree)'
+	+rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -bs kernel.spec \
+	--define='_smp_mflags %{nil}' --define='_sourcedir .' --define='_srcrpmdir .'
 
 # binrpm-pkg
 # ---------------------------------------------------------------------------
diff --git a/scripts/package/mkspec b/scripts/package/mkspec
index 108c0cb95436..83a64d9d7372 100755
--- a/scripts/package/mkspec
+++ b/scripts/package/mkspec
@@ -47,7 +47,8 @@ sed -e '/^DEL/d' -e 's/^\t*//' <<EOF
 	Group: System Environment/Kernel
 	Vendor: The Linux Community
 	URL: https://www.kernel.org
-$S	Source: kernel-$__KERNELRELEASE.tar.gz
+$S	Source0: linux.tar.gz
+$S	Source1: .config
 	Provides: $PROVIDES
 $S	BuildRequires: bc binutils bison dwarves
 $S	BuildRequires: (elfutils-libelf-devel or libelf-devel) flex
@@ -83,9 +84,8 @@ $S$M	This package provides kernel headers and makefiles sufficient to build modu
 $S$M	against the $__KERNELRELEASE kernel package.
 $S$M
 $S	%prep
-$S	%setup -q
-$S	rm -f scripts/basic/fixdep scripts/kconfig/conf
-$S	rm -f tools/objtool/{fixdep,objtool}
+$S	%setup -q -n linux
+$S	cp %{SOURCE1} .
 $S
 $S	%build
 $S	$MAKE %{?_smp_mflags} KERNELRELEASE=$KERNELRELEASE KBUILD_BUILD_VERSION=%{release}
-- 
2.34.1

[PATCH v2 5/5] kbuild: deb-pkg: hide KDEB_SOURCENAME from Makefile

[Masahiro Yamada]

scripts/Makefile.package does not need to know the value of
KDEB_SOURCENAME because the source name can be taken from
debian/changelog by using dpkg-parsechangelog.

Move the default of KDEB_SOURCENAME (i.e. linux-upstream) to
scripts/package/mkdebian.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

Changes in v2:
 - New patch

 scripts/Makefile.package | 23 +++++++++++++++--------
 scripts/package/mkdebian |  2 +-
 2 files changed, 16 insertions(+), 9 deletions(-)

diff --git a/scripts/Makefile.package b/scripts/Makefile.package
index 0e16ecfddad6..2ee39e0f24ee 100644
--- a/scripts/Makefile.package
+++ b/scripts/Makefile.package
@@ -3,9 +3,7 @@
 
 include $(srctree)/scripts/Kbuild.include
 
-KDEB_SOURCENAME ?= linux-upstream
 KBUILD_PKG_ROOTCMD ?="fakeroot -u"
-export KDEB_SOURCENAME
 MKSPEC     := $(srctree)/scripts/package/mkspec
 
 # Source Tarball
@@ -52,19 +50,28 @@ binrpm-pkg:
 	+rpmbuild $(RPMOPTS) --define "_builddir $(objtree)" --target \
 		$(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec
 
+# deb-pkg, bindeb-pkg
+# ---------------------------------------------------------------------------
+
 PHONY += deb-pkg
-deb-pkg:
-	$(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian
-	$(Q)origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
-		$(MAKE) -f $(srctree)/scripts/Makefile.package ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
+deb-pkg: source = $(shell dpkg-parsechangelog -S Source)
+deb-pkg: orig-version = $(shell dpkg-parsechangelog -S Version | sed 's/-[^-]*$$//')
+deb-pkg: debian
+	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.package ../$(source)_$(orig-version).orig.tar.gz
 	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
 		--build=source,binary --source-option=-sP -nc -us -uc
 
 PHONY += bindeb-pkg
-bindeb-pkg:
-	$(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian
+bindeb-pkg: debian
 	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -b -nc -uc
 
+quiet_cmd_debianize = GEN     $@
+      cmd_debianize = $(srctree)/scripts/package/mkdebian
+
+PHONY += debian
+debian:
+	$(call cmd,debianize)
+
 PHONY += intdeb-pkg
 intdeb-pkg:
 	+$(CONFIG_SHELL) $(srctree)/scripts/package/builddeb
diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
index 2f612617cbcf..0c1ed6215a02 100755
--- a/scripts/package/mkdebian
+++ b/scripts/package/mkdebian
@@ -95,7 +95,7 @@ else
 	revision=$($srctree/init/build-version)
 	packageversion=$version-$revision
 fi
-sourcename=$KDEB_SOURCENAME
+sourcename=${KDEB_SOURCENAME:-linux-upstream}
 
 if [ "$ARCH" = "um" ] ; then
 	packagename=user-mode-linux
-- 
2.34.1

Re: [PATCH v2 4/5] kbuild: srcrpm-pkg: create source package without cleaning

[Miguel Ojeda]

On Sun, Jan 29, 2023 at 7:46 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> -               include init io_uring ipc kernel lib mm net rust \

For Rust, it is early to deal with packaging, so removing this from
here should not hurt.

In any case, I quickly tried the series and noticed that the
`.src.rpm` does not end in the `SRPMS` folder (as it did before) -- is
that expected?

Thanks!

Cheers,
Miguel

Re: [PATCH v2 4/5] kbuild: srcrpm-pkg: create source package without cleaning

[Masahiro Yamada]

On Mon, Jan 30, 2023 at 8:20 AM Miguel Ojeda
<miguel.ojeda.sandonis@gmail.com> wrote:
>
> On Sun, Jan 29, 2023 at 7:46 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
> >
> > -               include init io_uring ipc kernel lib mm net rust \
>
> For Rust, it is early to deal with packaging, so removing this from
> here should not hurt.

I guess you are talking about kernel-devel-*.rpm
(and linux-headers-.deb).

They are not useful for building external modules
written in Rust since they do not contain *.rmeta etc.
I am not caring about that because Rust support is not
mature enough yet.


This series does not touch binary packages,
rather it just changes how the source package is created.

I stopped hard-coding the top-level directories.
The resulting source package still contains all check-in files
under rust/, so it is good from the source package perspective.




> In any case, I quickly tried the series and noticed that the
> `.src.rpm` does not end in the `SRPMS` folder (as it did before) -- is
> that expected?


5/5 changed the behavior because rpm-pkg re-uses the
*.src.rpm generated by srcrpm-pkg.


Having *.src.rpm in the kernel tree seems Redhat's preference.
Commit 8818039f959b2efc0d6f2cb101f8061332f0c77e
added --define='_srcrpmdir $(srctree)'.



In contrast, binary rpm files are generated under rpmbuild/RPMS/.
I want to fix this inconsistency, though.



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 4/5] kbuild: srcrpm-pkg: create source package without cleaning

[Miguel Ojeda]

On Mon, Jan 30, 2023 at 2:29 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> I guess you are talking about kernel-devel-*.rpm
> (and linux-headers-.deb).
>
> They are not useful for building external modules
> written in Rust since they do not contain *.rmeta etc.
> I am not caring about that because Rust support is not
> mature enough yet.

Yeah, that is what I meant, i.e. since the Rust ML was Cc'd, I checked
and wanted to say removing `rust` from there was OK (an `Acked-by`
seemed too much for just that line :).

> I stopped hard-coding the top-level directories.
> The resulting source package still contains all check-in files
> under rust/, so it is good from the source package perspective.

Sounds good to me.

> 5/5 changed the behavior because rpm-pkg re-uses the
> *.src.rpm generated by srcrpm-pkg.

(3/5?)

> Having *.src.rpm in the kernel tree seems Redhat's preference.
> Commit 8818039f959b2efc0d6f2cb101f8061332f0c77e
> added --define='_srcrpmdir $(srctree)'.

Thanks for the details! I just noticed it, so I thought I would let
you know just in case.

(Perhaps it could be useful to mention this change in the output in
the commit message.)

> In contrast, binary rpm files are generated under rpmbuild/RPMS/.
> I want to fix this inconsistency, though.

That would be nice.

Cheers,
Miguel

Re: [PATCH v2 4/5] kbuild: srcrpm-pkg: create source package without cleaning

[Masahiro Yamada]

On Mon, Jan 30, 2023 at 9:00 PM Miguel Ojeda
<miguel.ojeda.sandonis@gmail.com> wrote:
>
> On Mon, Jan 30, 2023 at 2:29 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
> >
> > I guess you are talking about kernel-devel-*.rpm
> > (and linux-headers-.deb).
> >
> > They are not useful for building external modules
> > written in Rust since they do not contain *.rmeta etc.
> > I am not caring about that because Rust support is not
> > mature enough yet.
>
> Yeah, that is what I meant, i.e. since the Rust ML was Cc'd, I checked
> and wanted to say removing `rust` from there was OK (an `Acked-by`
> seemed too much for just that line :).
>
> > I stopped hard-coding the top-level directories.
> > The resulting source package still contains all check-in files
> > under rust/, so it is good from the source package perspective.
>
> Sounds good to me.
>
> > 5/5 changed the behavior because rpm-pkg re-uses the
> > *.src.rpm generated by srcrpm-pkg.
>
> (3/5?)


Yes.


>
> > Having *.src.rpm in the kernel tree seems Redhat's preference.
> > Commit 8818039f959b2efc0d6f2cb101f8061332f0c77e
> > added --define='_srcrpmdir $(srctree)'.
>
> Thanks for the details! I just noticed it, so I thought I would let
> you know just in case.
>
> (Perhaps it could be useful to mention this change in the output in
> the commit message.)


Fair enough.

I updated the commit description in v3.


>
> > In contrast, binary rpm files are generated under rpmbuild/RPMS/.
> > I want to fix this inconsistency, though.
>
> That would be nice.
>
> Cheers,
> Miguel



-- 
Best Regards
Masahiro Yamada