deb-pkg: Cleaning of debian/*tmp files when running 'make (dist)clean'

deb-pkg: Cleaning of debian/*tmp files when running 'make (dist)clean'

[Sedat Dilek]

Hi,

I creating Debian/Ubuntu packages using builddeb script ('make deb-pkg').

While I was doing a 'make distclean' and grep-ed for some patterns I
saw that my debian/*tmp dirs were not deleted.

$ ls debian/*tmp
debian/fwtmp:
DEBIAN  lib  usr

debian/hdrtmp:
DEBIAN  lib  usr

debian/headertmp:
DEBIAN  usr

debian/tmp:
boot  DEBIAN  etc  lib  usr

Any chance that this is also done on 'make (dist)clean' (not only
within the builddeb script)?

111:rm -rf "$tmpdir" "$fwdir" "$kernel_headers_dir"
"$libc_headers_dir" "$dbg_dir"

Any other make (PHONY) target I don't know?

Or introduce sth. like clean_tmpdirs() executed before 'exit 0' (or
simply invoke above rm-line).

BTW, can we have some more meaningful var-names for both headers-dirs,
something like...

[ scripts/package/builddeb ]

-kernel_headers_dir="$objtree/debian/hdrtmp"
-libc_headers_dir="$objtree/debian/headertmp"
+kernel_headers_dir="$objtree/debian/k_hdrtmp"
+libc_headers_dir="$objtree/debian/c_hdrtmp"

...it's a bit confusing when you look at filesystem-level (dirs, files).

Thanks.

Regards,
- Sedat -

Re: deb-pkg: Cleaning of debian/*tmp files when running 'make (dist)clean'

[Sedat Dilek]

On Fri, Dec 26, 2014 at 1:02 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
[...]
>
> Any other make (PHONY) target I don't know?
>

I fell over "clean-dirs"...

scripts/package/Makefile:93:clean-dirs += $(objtree)/debian/

...but did not really understood how it works.
The main Makefile defines some clean-dirs PHONY#s.
I can only speculate, someone with more skillz in Makefile handling
should look at this.

Thanks.

- Sedat -

Re: deb-pkg: Cleaning of debian/*tmp files when running 'make (dist)clean'

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1032 bytes --]

On Fri, 2014-12-26 at 13:43 +0100, Sedat Dilek wrote:
> On Fri, Dec 26, 2014 at 1:02 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> [...]
> >
> > Any other make (PHONY) target I don't know?
> >
> 
> I fell over "clean-dirs"...
> 
> scripts/package/Makefile:93:clean-dirs += $(objtree)/debian/
> 
> ...but did not really understood how it works.
> The main Makefile defines some clean-dirs PHONY#s.
> I can only speculate, someone with more skillz in Makefile handling
> should look at this.

It looks like this has not worked for a long time.  The scripts
directory is not included in the main recursive processes; it seems to
be special-cased for build and not for clean.  This is partly deliberate
because clean is supposed to leave 'enough to build external modules'.
It's not obvious to include scripts/packages/ in clean.

Ben.

-- 
Ben Hutchings
Experience is directly proportional to the value of equipment destroyed.
                                                         - Carolyn Scheppner

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: deb-pkg: Cleaning of debian/*tmp files when running 'make (dist)clean'

[Sedat Dilek]

On Fri, Dec 26, 2014 at 7:46 PM, Ben Hutchings <ben@decadent.org.uk> wrote:
> On Fri, 2014-12-26 at 13:43 +0100, Sedat Dilek wrote:
>> On Fri, Dec 26, 2014 at 1:02 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>> [...]
>> >
>> > Any other make (PHONY) target I don't know?
>> >
>>
>> I fell over "clean-dirs"...
>>
>> scripts/package/Makefile:93:clean-dirs += $(objtree)/debian/
>>
>> ...but did not really understood how it works.
>> The main Makefile defines some clean-dirs PHONY#s.
>> I can only speculate, someone with more skillz in Makefile handling
>> should look at this.
>
> It looks like this has not worked for a long time.  The scripts
> directory is not included in the main recursive processes; it seems to
> be special-cased for build and not for clean.  This is partly deliberate
> because clean is supposed to leave 'enough to build external modules'.
> It's not obvious to include scripts/packages/ in clean.
>

[ CCing some more people (submitters to scripts/Makefile.clean) ]

Thanks for your reply and confirmation.

So, how to fix that issue?

Cleanup debian/*tmp dirs within builddeb script for now?

Fix that clean-dirs mechanism (or remove it)?
These 4 Makefile#s use it...

Documentation/DocBook/Makefile
Makefile
scripts/Makefile.clean
scripts/package/Makefile

Any help appreciated.

- Sedat -

Re: deb-pkg: Cleaning of debian/*tmp files when running 'make (dist)clean'

[Michal Marek]

On 2014-12-26 19:46, Ben Hutchings wrote:
> On Fri, 2014-12-26 at 13:43 +0100, Sedat Dilek wrote:
>> On Fri, Dec 26, 2014 at 1:02 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>> [...]
>>>
>>> Any other make (PHONY) target I don't know?
>>>
>>
>> I fell over "clean-dirs"...
>>
>> scripts/package/Makefile:93:clean-dirs += $(objtree)/debian/
>>
>> ...but did not really understood how it works.
>> The main Makefile defines some clean-dirs PHONY#s.
>> I can only speculate, someone with more skillz in Makefile handling
>> should look at this.
> 
> It looks like this has not worked for a long time.  The scripts
> directory is not included in the main recursive processes; it seems to
> be special-cased for build and not for clean.

It actually is special-cased for mrproper, which is a subset if
distclean. The real problem is that

clean-dirs += $(objtree)/debian/

expands to

clean-dirs += ./debian/

and scripts/Makefile.clean things that this is relative to the
scripts/package directory.

Michal

Re: deb-pkg: Cleaning of debian/*tmp files when running 'make (dist)clean'

[Michal Marek]

On 2014-12-26 19:46, Ben Hutchings wrote:
> On Fri, 2014-12-26 at 13:43 +0100, Sedat Dilek wrote:
>> On Fri, Dec 26, 2014 at 1:02 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>> [...]
>>>
>>> Any other make (PHONY) target I don't know?
>>>
>>
>> I fell over "clean-dirs"...
>>
>> scripts/package/Makefile:93:clean-dirs += $(objtree)/debian/
>>
>> ...but did not really understood how it works.
>> The main Makefile defines some clean-dirs PHONY#s.
>> I can only speculate, someone with more skillz in Makefile handling
>> should look at this.
> 
> It looks like this has not worked for a long time.  The scripts
> directory is not included in the main recursive processes; it seems to
> be special-cased for build and not for clean.

It is special-cased for mrproper and thus distclean:

  mrproper-dirs      := $(addprefix _mrproper_,Documentation/DocBook
scripts)

The problem is that

  clean-dirs += $(objtree)/debian/

results in

  clean-dirs += ./debian/

and scripts/Makefile.clean treats it as relative to scripts/package.

Michal

Re: deb-pkg: Cleaning of debian/*tmp files when running 'make (dist)clean'

[Sedat Dilek]

On Wed, Dec 31, 2014 at 2:56 PM, Michal Marek <mmarek@suse.cz> wrote:
> On 2014-12-26 19:46, Ben Hutchings wrote:
>> On Fri, 2014-12-26 at 13:43 +0100, Sedat Dilek wrote:
>>> On Fri, Dec 26, 2014 at 1:02 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>> [...]
>>>>
>>>> Any other make (PHONY) target I don't know?
>>>>
>>>
>>> I fell over "clean-dirs"...
>>>
>>> scripts/package/Makefile:93:clean-dirs += $(objtree)/debian/
>>>
>>> ...but did not really understood how it works.
>>> The main Makefile defines some clean-dirs PHONY#s.
>>> I can only speculate, someone with more skillz in Makefile handling
>>> should look at this.
>>
>> It looks like this has not worked for a long time.  The scripts
>> directory is not included in the main recursive processes; it seems to
>> be special-cased for build and not for clean.
>
> It is special-cased for mrproper and thus distclean:
>
>   mrproper-dirs      := $(addprefix _mrproper_,Documentation/DocBook
> scripts)
>
> The problem is that
>
>   clean-dirs += $(objtree)/debian/
>
> results in
>
>   clean-dirs += ./debian/
>
> and scripts/Makefile.clean treats it as relative to scripts/package.
>

Thanks for your reply.

IIRC I looked through some commits touching this area.
From your description, is the simplification ("kbuild: simplify build,
clean, modbuiltin shorthands") in [1] the cause for this?

[ I haven't tried to revert it, returned home today, still looking
through my inbox :-). ]

- Sedat -

[1] http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/scripts/Makefile.clean?id=5b2389b45d1a9c12b9f4b976e38c36264bfc42f5

Re: deb-pkg: Cleaning of debian/*tmp files when running 'make (dist)clean'

[Michal Marek]

On 2014-12-31 15:16, Sedat Dilek wrote:
> On Wed, Dec 31, 2014 at 2:56 PM, Michal Marek <mmarek@suse.cz> wrote:
>> The problem is that
>>
>>   clean-dirs += $(objtree)/debian/
>>
>> results in
>>
>>   clean-dirs += ./debian/
>>
>> and scripts/Makefile.clean treats it as relative to scripts/package.
>>
> 
> Thanks for your reply.
> 
> IIRC I looked through some commits touching this area.
> From your description, is the simplification ("kbuild: simplify build,
> clean, modbuiltin shorthands") in [1] the cause for this?

No, it broke with 7e1c0477 (kbuild: Use relative path for $(objtree)).

I'll problably change the logic in Makefile.clean to check for paths
starting with './' AKA '$(objtree)/', unless I find a more elegant solution.

Michal

Re: deb-pkg: Cleaning of debian/*tmp files when running 'make (dist)clean'

[Sedat Dilek]

On Wed, Dec 31, 2014 at 3:25 PM, Michal Marek <mmarek@suse.cz> wrote:
> On 2014-12-31 15:16, Sedat Dilek wrote:
>> On Wed, Dec 31, 2014 at 2:56 PM, Michal Marek <mmarek@suse.cz> wrote:
>>> The problem is that
>>>
>>>   clean-dirs += $(objtree)/debian/
>>>
>>> results in
>>>
>>>   clean-dirs += ./debian/
>>>
>>> and scripts/Makefile.clean treats it as relative to scripts/package.
>>>
>>
>> Thanks for your reply.
>>
>> IIRC I looked through some commits touching this area.
>> From your description, is the simplification ("kbuild: simplify build,
>> clean, modbuiltin shorthands") in [1] the cause for this?
>
> No, it broke with 7e1c0477 (kbuild: Use relative path for $(objtree)).
>
> I'll problably change the logic in Makefile.clean to check for paths
> starting with './' AKA '$(objtree)/', unless I find a more elegant solution.
>

Thanks /o\.

- Sedat -

[PATCH 1/2] kbuild: Fix removal of the debian/ directory

[Michal Marek]

scripts/Makefile.clean treats absolute path specially, but
$(objtree)/debian is no longer an absolute path since 7e1c0477 (kbuild:
Use relative path for $(objtree). Work around this by checking if the
path starts with $(objtree)/.

Reported-by: Sedat Dilek <sedat.dilek@gmail.com>
Fixes: 7e1c0477 (kbuild: Use relative path for $(objtree)
Signed-off-by: Michal Marek <mmarek@suse.cz>
---
 scripts/Makefile.clean | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/scripts/Makefile.clean b/scripts/Makefile.clean
index 1bca180..627f8cb 100644
--- a/scripts/Makefile.clean
+++ b/scripts/Makefile.clean
@@ -42,19 +42,19 @@ __clean-files	:= $(extra-y) $(extra-m) $(extra-)       \
 
 __clean-files   := $(filter-out $(no-clean-files), $(__clean-files))
 
-# as clean-files is given relative to the current directory, this adds
-# a $(obj) prefix, except for absolute paths
+# clean-files is given relative to the current directory, unless it
+# starts with $(objtree)/ (which means "./", so do not add "./" unless
+# you want to delete a file from the toplevel object directory).
 
 __clean-files   := $(wildcard                                               \
-                   $(addprefix $(obj)/, $(filter-out /%, $(__clean-files))) \
-		   $(filter /%, $(__clean-files)))
+		   $(addprefix $(obj)/, $(filter-out $(objtree)/%, $(__clean-files))) \
+		   $(filter $(objtree)/%, $(__clean-files)))
 
-# as clean-dirs is given relative to the current directory, this adds
-# a $(obj) prefix, except for absolute paths
+# same as clean-files
 
 __clean-dirs    := $(wildcard                                               \
-                   $(addprefix $(obj)/, $(filter-out /%, $(clean-dirs)))    \
-		   $(filter /%, $(clean-dirs)))
+		   $(addprefix $(obj)/, $(filter-out $(objtree)/%, $(clean-dirs)))    \
+		   $(filter $(objtree)/%, $(clean-dirs)))
 
 # ==========================================================================
 
-- 
2.1.2

[PATCH 2/2] kbuild: Drop support for clean-rule

[Michal Marek]

clean-rule has not been used since 94869f86 (kbuild: Accept absolute
paths in clean-files and introduce clean-dirs) ten years ago.

Signed-off-by: Michal Marek <mmarek@suse.cz>
---
 scripts/Makefile.clean | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/scripts/Makefile.clean b/scripts/Makefile.clean
index 627f8cb..55c96cb 100644
--- a/scripts/Makefile.clean
+++ b/scripts/Makefile.clean
@@ -71,9 +71,6 @@ endif
 ifneq ($(strip $(__clean-dirs)),)
 	+$(call cmd,cleandir)
 endif
-ifneq ($(strip $(clean-rule)),)
-	+$(clean-rule)
-endif
 	@:
 
 
-- 
2.1.2

Re: [PATCH 1/2] kbuild: Fix removal of the debian/ directory

[Sedat Dilek]

On Wed, Dec 31, 2014 at 4:35 PM, Michal Marek <mmarek@suse.cz> wrote:
> scripts/Makefile.clean treats absolute path specially, but
> $(objtree)/debian is no longer an absolute path since 7e1c0477 (kbuild:
> Use relative path for $(objtree). Work around this by checking if the
> path starts with $(objtree)/.
>
> Reported-by: Sedat Dilek <sedat.dilek@gmail.com>
> Fixes: 7e1c0477 (kbuild: Use relative path for $(objtree)
> Signed-off-by: Michal Marek <mmarek@suse.cz>

After applying your patchset, thing work as expected.

Feel free to add my Tested-by.

Hope this will get into v3.19-rc3?

- Sedat -

[1] https://patchwork.kernel.org/patch/5555911/
[2] https://patchwork.kernel.org/patch/5555931/

P.S.: Testing new patchset (especially 1/2).

$ make distclean
  CLEAN   .
  CLEAN   arch/x86/kernel
  CLEAN   arch/x86/realmode/rm
  CLEAN   arch/x86/vdso
  CLEAN   arch/x86/lib
  CLEAN   crypto/asymmetric_keys
  CLEAN   drivers/tty/vt
  CLEAN   kernel/debug/kdb
  CLEAN   kernel/time
  CLEAN   kernel
  CLEAN   lib/raid6
  CLEAN   lib
  CLEAN   security/apparmor
  CLEAN   security/selinux
  CLEAN   usr
  CLEAN   arch/x86/boot/compressed
  CLEAN   arch/x86/boot
  CLEAN   arch/x86/tools
  CLEAN   .tmp_versions
  CLEAN   scripts/basic
  CLEAN   scripts/genksyms
  CLEAN   scripts/kconfig
  CLEAN   scripts/mod
  CLEAN   scripts/selinux/genheaders
  CLEAN   scripts/selinux/mdp
  CLEAN   scripts
  CLEAN   include/config usr/include include/generated
arch/x86/include/generated
  CLEAN   .config .config.old .version
include/generated/uapi/linux/version.h Module.symvers signing_key.priv
signing_key.x509 x509.genkey

$ git am ../patches/kbuild-fixes-3.19-rc2/1*
Wende an: kbuild: Fix removal of the debian/ directory

$ git am ../patches/kbuild-fixes-3.19-rc2/2*
Wende an: kbuild: Drop support for clean-rule

$ ls debian/*tmp
debian/fwtmp:
DEBIAN  lib  usr

debian/hdrtmp:
DEBIAN  lib  usr

debian/headertmp:
DEBIAN  usr

debian/tmp:
boot  DEBIAN  etc  lib  usr

$ make distclean
  CLEAN   ./debian/

$ LC_ALL=C ls debian/*tmp
ls: cannot access debian/*tmp: No such file or directory
- EOT .


> ---
>  scripts/Makefile.clean | 16 ++++++++--------
>  1 file changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/scripts/Makefile.clean b/scripts/Makefile.clean
> index 1bca180..627f8cb 100644
> --- a/scripts/Makefile.clean
> +++ b/scripts/Makefile.clean
> @@ -42,19 +42,19 @@ __clean-files       := $(extra-y) $(extra-m) $(extra-)       \
>
>  __clean-files   := $(filter-out $(no-clean-files), $(__clean-files))
>
> -# as clean-files is given relative to the current directory, this adds
> -# a $(obj) prefix, except for absolute paths
> +# clean-files is given relative to the current directory, unless it
> +# starts with $(objtree)/ (which means "./", so do not add "./" unless
> +# you want to delete a file from the toplevel object directory).
>
>  __clean-files   := $(wildcard                                               \
> -                   $(addprefix $(obj)/, $(filter-out /%, $(__clean-files))) \
> -                  $(filter /%, $(__clean-files)))
> +                  $(addprefix $(obj)/, $(filter-out $(objtree)/%, $(__clean-files))) \
> +                  $(filter $(objtree)/%, $(__clean-files)))
>
> -# as clean-dirs is given relative to the current directory, this adds
> -# a $(obj) prefix, except for absolute paths
> +# same as clean-files
>
>  __clean-dirs    := $(wildcard                                               \
> -                   $(addprefix $(obj)/, $(filter-out /%, $(clean-dirs)))    \
> -                  $(filter /%, $(clean-dirs)))
> +                  $(addprefix $(obj)/, $(filter-out $(objtree)/%, $(clean-dirs)))    \
> +                  $(filter $(objtree)/%, $(clean-dirs)))
>
>  # ==========================================================================
>
> --
> 2.1.2
>

Re: [PATCH 2/2] kbuild: Drop support for clean-rule

[Sedat Dilek]

On Wed, Dec 31, 2014 at 4:35 PM, Michal Marek <mmarek@suse.cz> wrote:
> clean-rule has not been used since 94869f86 (kbuild: Accept absolute
> paths in clean-files and introduce clean-dirs) ten years ago.
>
> Signed-off-by: Michal Marek <mmarek@suse.cz>

Tested-by: Sedat Dilek <sedat.dilek@gmail.com>

- Sedat -

> ---
>  scripts/Makefile.clean | 3 ---
>  1 file changed, 3 deletions(-)
>
> diff --git a/scripts/Makefile.clean b/scripts/Makefile.clean
> index 627f8cb..55c96cb 100644
> --- a/scripts/Makefile.clean
> +++ b/scripts/Makefile.clean
> @@ -71,9 +71,6 @@ endif
>  ifneq ($(strip $(__clean-dirs)),)
>         +$(call cmd,cleandir)
>  endif
> -ifneq ($(strip $(clean-rule)),)
> -       +$(clean-rule)
> -endif
>         @:
>
>
> --
> 2.1.2
>

Re: [PATCH 1/2] kbuild: Fix removal of the debian/ directory

[Michal Marek]

On 2014-12-31 16:50, Sedat Dilek wrote:
> On Wed, Dec 31, 2014 at 4:35 PM, Michal Marek <mmarek@suse.cz> wrote:
>> scripts/Makefile.clean treats absolute path specially, but
>> $(objtree)/debian is no longer an absolute path since 7e1c0477 (kbuild:
>> Use relative path for $(objtree). Work around this by checking if the
>> path starts with $(objtree)/.
>>
>> Reported-by: Sedat Dilek <sedat.dilek@gmail.com>
>> Fixes: 7e1c0477 (kbuild: Use relative path for $(objtree)
>> Signed-off-by: Michal Marek <mmarek@suse.cz>
> 
> After applying your patchset, thing work as expected.
> 
> Feel free to add my Tested-by.
> 
> Hope this will get into v3.19-rc3?

I pushed the patch to kbuild.git#rc-fixes. The 'clean-rule' removal is
in kbuild.git#kbuild.

Michal

Re: [PATCH 1/2] kbuild: Fix removal of the debian/ directory

[Sedat Dilek]

On Fri, Jan 2, 2015 at 4:01 PM, Michal Marek <mmarek@suse.cz> wrote:
> On 2014-12-31 16:50, Sedat Dilek wrote:
>> On Wed, Dec 31, 2014 at 4:35 PM, Michal Marek <mmarek@suse.cz> wrote:
>>> scripts/Makefile.clean treats absolute path specially, but
>>> $(objtree)/debian is no longer an absolute path since 7e1c0477 (kbuild:
>>> Use relative path for $(objtree). Work around this by checking if the
>>> path starts with $(objtree)/.
>>>
>>> Reported-by: Sedat Dilek <sedat.dilek@gmail.com>
>>> Fixes: 7e1c0477 (kbuild: Use relative path for $(objtree)
>>> Signed-off-by: Michal Marek <mmarek@suse.cz>
>>
>> After applying your patchset, thing work as expected.
>>
>> Feel free to add my Tested-by.
>>
>> Hope this will get into v3.19-rc3?
>
> I pushed the patch to kbuild.git#rc-fixes. The 'clean-rule' removal is
> in kbuild.git#kbuild.
>

Thanks.
I had already pulled them into my local for-3.19/kbuild-fixes.

Can you take care of the long outstanding builddeb fix I posted a few
minutes ago ?
I you plan a git-pull request, can we have this in rc-fixes?
It's trivial - don't think we need any ACKs (which I got 2012/2013).

- Sedat -

Re: deb-pkg: Cleaning of debian/*tmp files when running 'make (dist)clean'

[Sedat Dilek]

On Wed, Dec 31, 2014 at 2:53 PM, Michal Marek <mmarek@suse.cz> wrote:
> On 2014-12-26 19:46, Ben Hutchings wrote:
>> On Fri, 2014-12-26 at 13:43 +0100, Sedat Dilek wrote:
>>> On Fri, Dec 26, 2014 at 1:02 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>> [...]
>>>>
>>>> Any other make (PHONY) target I don't know?
>>>>
>>>
>>> I fell over "clean-dirs"...
>>>
>>> scripts/package/Makefile:93:clean-dirs += $(objtree)/debian/
>>>
>>> ...but did not really understood how it works.
>>> The main Makefile defines some clean-dirs PHONY#s.
>>> I can only speculate, someone with more skillz in Makefile handling
>>> should look at this.
>>
>> It looks like this has not worked for a long time.  The scripts
>> directory is not included in the main recursive processes; it seems to
>> be special-cased for build and not for clean.
>
> It actually is special-cased for mrproper, which is a subset if
> distclean. The real problem is that
>
> clean-dirs += $(objtree)/debian/
>
> expands to
>
> clean-dirs += ./debian/
>
> and scripts/Makefile.clean things that this is relative to the
> scripts/package directory.
>
> Michal

For the sake of completeness... Issue fixed by...

commit a16c5f99a28c9945165c46da27fff8e6f26f8736
"kbuild: Fix removal of the debian/ directory"

- Sedat -

[1] http://git.kernel.org/cgit/linux/kernel/git/mmarek/kbuild.git/commit/?h=rc-fixes&id=a16c5f99a28c9945165c46da27fff8e6f26f8736