[RFC] Standard way of generating assembler offsets

[RFC] Standard way of generating assembler offsets

[Keith Owens]

Almost every architecture generates Assembler values to map the offsets
of fields in C structures, about the only exception is i386 and that is
because its offsets are hard coded into entry.S.  Every arch has done
it differently, none of them have got it exactly right.

As part of kbuild 2.5 I am standardizing on one method for generating
Assembler offsets.  This change is required for kbuild 2.5 but it can
be added to 2.4 without disturbing the current kbuild, I want to do
this gradually now instead of a single massive change in kernel 2.5.  I
will be issuing per architecture changes for generating Assembler
offsets against 2.4.

The kbuild 2.5 method for generating Assembler offsets satisfies these
requirements:

* No manual intervention required.  Many architectures rely on users
  running make dep after changing config options that affect the
  Assembler offsets.  If the user forgets to run make dep then the C
  and Assembler code is out of sync - totally unacceptable.  This is
  completely fixed in kbuild 2.5; I cannot do a complete fix in kbuild
  2.4 but it is still better than the existing manual system.

* Standard name for the related files.  There are 6+ different names
  for the files used to generate Assembler offsets, kbuild 2.5 uses
  asm-offsets.[csh] on all architectures.

* Allows for multiple parallel compiles from the same source tree.
  Writing the generated asm-offsets.h to include/asm is not an option,
  it prevents concurrent compiles.

* The method must work in native and cross compile mode and give
  exactly the same results.  Some 2.4 code only works in native mode,
  some architectures have different methods for native and cross
  compile with different output formats.  Yeuch!

* Standard scripts for generating the output.  Every arch does it
  differently in 2.4, standards are good!

* Correct dependency trees.  Because 2.4 make dep does not scan .S
  files, there is little or no dependency information.  Even if the
  offsets are regenerated, the affected Assembler code does not always
  get rebuilt.  kbuild 2.5 handles dependencies for Assembler as well
  as C; I cannot get kbuild 2.4 perfect but I can improve on the
  existing (or non-existent) 2.4 dependencies.

All architectures will define arch/$(ARCH)/asm-offsets.c.  This has a
standard prologue for the macros that convert offsets to Assembler,
followed by arch specific field references.

arch/$(ARCH)/asm-offsets.s is generated from arch/$(ARCH)/asm-offsets.c
using standard rules, although kbuild 2.4 needs some tweaking.

arch/$(ARCH)/asm-offsets.h is generated from arch/$(ARCH)/asm-offsets.s
by a semi-standard script.  Most of the script is common to all
architectures but the precise format of the Assembler output is arch
specific.

The final result is included in *only* the Assembler programs that need
it, as #include "asm-offsets.h" with -I arch/$(ARCH) in the relevant
Makefiles.  Hard coding relative paths in source files is a pet hate,
use #include "localname.h" and -I instead.  Including the generated
file in C code is not allowed, it severly pollutes the dependency
chain, to the extent that any config change can force a complete
recompile, unacceptable.


Example from i386:

arch/i386/asm-offsets.c

/*
 * Generate definitions needed by assembly language modules.
 * This code generates raw asm output which is post-processed to extract
 * and format the required data.
 */

#include <linux/types.h>
#include <linux/stddef.h>
#include <linux/sched.h>

/* Use marker if you need to separate the values later */

#define DEFINE(sym, val, marker) \
  asm volatile("\n-> " #sym " %0 " #val " " #marker : : "i" (val))

#define BLANK() asm volatile("\n->" : : )

int
main(void)
{
  DEFINE(state,        offsetof(struct task_struct, state),);
  DEFINE(flags,        offsetof(struct task_struct, flags),);
  DEFINE(sigpending,   offsetof(struct task_struct, sigpending),);
  DEFINE(addr_limit,   offsetof(struct task_struct, addr_limit),);
  DEFINE(exec_domain,  offsetof(struct task_struct, exec_domain),);
  DEFINE(need_resched, offsetof(struct task_struct, need_resched),);
  DEFINE(tsk_ptrace,   offsetof(struct task_struct, ptrace),);
  DEFINE(processor,    offsetof(struct task_struct, processor),);
  BLANK();
  DEFINE(ENOSYS,       ENOSYS,);
  return 0;
}

asm-offsets.s to asm-offsets.h.

# Convert raw asm offsets into something that can be included as
# assembler definitions.  It converts
#   -> symbol $value source
# into
#   symbol = value /* 0xvalue source */

echo '#ifndef __ASM_OFFSETS_H__'
echo '#define __ASM_OFFSETS_H__'
echo '/*'
echo ' * DO NOT MODIFY'
echo ' *'
echo " * This file was generated by arch/$(ARCH)/Makefile.in."
echo ' *'
echo ' */'
echo ''
awk '
  /^->$/{printf("\n")}
  /^-> /{
    sym = $2;
    val = $3;
    sub(/^\$/, "", val);
    $1 = "";
    $2 = "";
    $3 = "";
    printf("%-20s = %3d\t/* 0x%x\t%s */\n", sym, val, val, $0)
  }
'
echo '#endif'

Generated arch/i386/asm-offsets.h

#ifndef __ASM_OFFSETS_H__
#define __ASM_OFFSETS_H__
/*
 * DO NOT MODIFY
 *
 * This file was generated by arch/i386/Makefile.in.
 *
 */

state                =   0      /* 0x0     offsetof(struct task_struct, state) */
flags                =   4      /* 0x4     offsetof(struct task_struct, flags) */
sigpending           =   8      /* 0x8     offsetof(struct task_struct, sigpending) */
addr_limit           =  12      /* 0xc     offsetof(struct task_struct, addr_limit) */
exec_domain          =  16      /* 0x10    offsetof(struct task_struct, exec_domain) */
need_resched         =  20      /* 0x14    offsetof(struct task_struct, need_resched) */
tsk_ptrace           =  24      /* 0x18    offsetof(struct task_struct, ptrace) */
processor            =  52      /* 0x34    offsetof(struct task_struct, processor) */

ENOSYS               =  38      /* 0x26    ENOSYS */
#endif

Re: [RFC] Standard way of generating assembler offsets

[george anzinger]

This sound great.  It has been a struggle for me for some time.

A couple of comments:

The symbol name IMHO should contain both the member name and the
structure name.  Otherwise there may be a problem if two structures use
the same member name (flags comes to mind).

This is way down on the list, but is it possible to generate a separate
file for each *.S AND put the "required symbols" in the *.S.  

When I last did this we put a comment in the *.S that had a special
format and changed the make rule for *.S->*.o to first build the
required file.  The *.S then used #include "sym*.h" to get the symbols. 
I think the standard make rule even removed the sys*.h file after the
*.o was built.  The actual section in the *.S that defined the required
symbols was written as cpp macros which were stripped with a sed
script.  The result (with a standard header and tail) was compiled and
run to generate the sym*.h file.  Even the includes needed to generate
the symbols were in the *.S so a dep script can generate the needed
stuff for the *.S.

The nice thing about this method is that it is easy to keep track of the
required symbols and the make dependencies are simple (and easily
derived).

George

Keith Owens wrote:
> 
> Almost every architecture generates Assembler values to map the offsets
> of fields in C structures, about the only exception is i386 and that is
> because its offsets are hard coded into entry.S.  Every arch has done
> it differently, none of them have got it exactly right.
> 
> As part of kbuild 2.5 I am standardizing on one method for generating
> Assembler offsets.  This change is required for kbuild 2.5 but it can
> be added to 2.4 without disturbing the current kbuild, I want to do
> this gradually now instead of a single massive change in kernel 2.5.  I
> will be issuing per architecture changes for generating Assembler
> offsets against 2.4.
> 
> The kbuild 2.5 method for generating Assembler offsets satisfies these
> requirements:
> 
> * No manual intervention required.  Many architectures rely on users
>   running make dep after changing config options that affect the
>   Assembler offsets.  If the user forgets to run make dep then the C
>   and Assembler code is out of sync - totally unacceptable.  This is
>   completely fixed in kbuild 2.5; I cannot do a complete fix in kbuild
>   2.4 but it is still better than the existing manual system.
> 
> * Standard name for the related files.  There are 6+ different names
>   for the files used to generate Assembler offsets, kbuild 2.5 uses
>   asm-offsets.[csh] on all architectures.
> 
> * Allows for multiple parallel compiles from the same source tree.
>   Writing the generated asm-offsets.h to include/asm is not an option,
>   it prevents concurrent compiles.
> 
> * The method must work in native and cross compile mode and give
>   exactly the same results.  Some 2.4 code only works in native mode,
>   some architectures have different methods for native and cross
>   compile with different output formats.  Yeuch!
> 
> * Standard scripts for generating the output.  Every arch does it
>   differently in 2.4, standards are good!
> 
> * Correct dependency trees.  Because 2.4 make dep does not scan .S
>   files, there is little or no dependency information.  Even if the
>   offsets are regenerated, the affected Assembler code does not always
>   get rebuilt.  kbuild 2.5 handles dependencies for Assembler as well
>   as C; I cannot get kbuild 2.4 perfect but I can improve on the
>   existing (or non-existent) 2.4 dependencies.
> 
> All architectures will define arch/$(ARCH)/asm-offsets.c.  This has a
> standard prologue for the macros that convert offsets to Assembler,
> followed by arch specific field references.
> 
> arch/$(ARCH)/asm-offsets.s is generated from arch/$(ARCH)/asm-offsets.c
> using standard rules, although kbuild 2.4 needs some tweaking.
> 
> arch/$(ARCH)/asm-offsets.h is generated from arch/$(ARCH)/asm-offsets.s
> by a semi-standard script.  Most of the script is common to all
> architectures but the precise format of the Assembler output is arch
> specific.
> 
> The final result is included in *only* the Assembler programs that need
> it, as #include "asm-offsets.h" with -I arch/$(ARCH) in the relevant
> Makefiles.  Hard coding relative paths in source files is a pet hate,
> use #include "localname.h" and -I instead.  Including the generated
> file in C code is not allowed, it severly pollutes the dependency
> chain, to the extent that any config change can force a complete
> recompile, unacceptable.
> 
> Example from i386:
> 
> arch/i386/asm-offsets.c
> 
> /*
>  * Generate definitions needed by assembly language modules.
>  * This code generates raw asm output which is post-processed to extract
>  * and format the required data.
>  */
> 
> #include <linux/types.h>
> #include <linux/stddef.h>
> #include <linux/sched.h>
> 
> /* Use marker if you need to separate the values later */
> 
> #define DEFINE(sym, val, marker) \
>   asm volatile("\n-> " #sym " %0 " #val " " #marker : : "i" (val))
> 
> #define BLANK() asm volatile("\n->" : : )
> 
> int
> main(void)
> {
>   DEFINE(state,        offsetof(struct task_struct, state),);
>   DEFINE(flags,        offsetof(struct task_struct, flags),);
>   DEFINE(sigpending,   offsetof(struct task_struct, sigpending),);
>   DEFINE(addr_limit,   offsetof(struct task_struct, addr_limit),);
>   DEFINE(exec_domain,  offsetof(struct task_struct, exec_domain),);
>   DEFINE(need_resched, offsetof(struct task_struct, need_resched),);
>   DEFINE(tsk_ptrace,   offsetof(struct task_struct, ptrace),);
>   DEFINE(processor,    offsetof(struct task_struct, processor),);
>   BLANK();
>   DEFINE(ENOSYS,       ENOSYS,);
>   return 0;
> }
> 
> asm-offsets.s to asm-offsets.h.
> 
> # Convert raw asm offsets into something that can be included as
> # assembler definitions.  It converts
> #   -> symbol $value source
> # into
> #   symbol = value /* 0xvalue source */
> 
> echo '#ifndef __ASM_OFFSETS_H__'
> echo '#define __ASM_OFFSETS_H__'
> echo '/*'
> echo ' * DO NOT MODIFY'
> echo ' *'
> echo " * This file was generated by arch/$(ARCH)/Makefile.in."
> echo ' *'
> echo ' */'
> echo ''
> awk '
>   /^->$/{printf("\n")}
>   /^-> /{
>     sym = $2;
>     val = $3;
>     sub(/^\$/, "", val);
>     $1 = "";
>     $2 = "";
>     $3 = "";
>     printf("%-20s = %3d\t/* 0x%x\t%s */\n", sym, val, val, $0)
>   }
> '
> echo '#endif'
> 
> Generated arch/i386/asm-offsets.h
> 
> #ifndef __ASM_OFFSETS_H__
> #define __ASM_OFFSETS_H__
> /*
>  * DO NOT MODIFY
>  *
>  * This file was generated by arch/i386/Makefile.in.
>  *
>  */
> 
> state                =   0      /* 0x0     offsetof(struct task_struct, state) */
> flags                =   4      /* 0x4     offsetof(struct task_struct, flags) */
> sigpending           =   8      /* 0x8     offsetof(struct task_struct, sigpending) */
> addr_limit           =  12      /* 0xc     offsetof(struct task_struct, addr_limit) */
> exec_domain          =  16      /* 0x10    offsetof(struct task_struct, exec_domain) */
> need_resched         =  20      /* 0x14    offsetof(struct task_struct, need_resched) */
> tsk_ptrace           =  24      /* 0x18    offsetof(struct task_struct, ptrace) */
> processor            =  52      /* 0x34    offsetof(struct task_struct, processor) */
> 
> ENOSYS               =  38      /* 0x26    ENOSYS */
> #endif
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] Standard way of generating assembler offsets

[Keith Owens]

On Thu, 04 Oct 2001 08:36:54 -0700, 
george anzinger <george@mvista.com> wrote:
>The symbol name IMHO should contain both the member name and the
>structure name.  Otherwise there may be a problem if two structures use
>the same member name (flags comes to mind).

The asm symbol name is arch defined, I am defining the standard method,
not the asm names.  It is up to the arch maintainers to pick suitable
names, e.g. ia64 does

DEFINE(IA64_SWITCH_STACK_AR_BSPSTORE_OFFSET, offsetof(struct switch_stack, ar_bspstore),);

>This is way down on the list, but is it possible to generate a separate
>file for each *.S AND put the "required symbols" in the *.S.  

That works for a small number of mappings but not when there are a
large number that are required in several places.  Take a look at
arch/ia64/tools/print_offsets.c, 130+ mappings used by 5 or 6 different
asm sources.  There are also technical reasons (to do with the kernel
CONFIG system) why a single asm-offsets file is easier to maintain.

Re: [RFC] Standard way of generating assembler offsets

[Keith Owens]

First patch to clean up Assembler offsets, as described in
http://marc.theaimsgroup.com/?l=linux-kernel&m=100219616028442&w=2

This patch is larger than normal because my testing highlighted a long
standing race in the top level kernel Makefile.

i386 patch.  Yes, I know that doing asm-offsets.c for i386 is overkill,
it only needs a few values that "never" change so they can be hard
coded.  However I need a reference implementation, plus the race needs
to be mitigated.

BTW, this is *much* cleaner in kbuild 2.5.  Bits of the patch are
workarounds for recursive make problems and to handle mkdep not
processing Assembler dependencies :(.

Index: 11-pre4.1/Makefile
--- 11-pre4.1/Makefile Fri, 05 Oct 2001 15:05:09 +1000 kaos (linux-2.4/T/c/50_Makefile 1.1.2.15.1.2.2.25.2.2.1.17.1.4.1.29 644)
+++ 11-pre4.1(w)/Makefile Sat, 06 Oct 2001 15:07:25 +1000 kaos (linux-2.4/T/c/50_Makefile 1.1.2.15.1.2.2.25.2.2.1.17.1.4.1.29 644)
@@ -205,7 +205,8 @@ CLEAN_FILES = \
 	drivers/scsi/53c700-mem.c \
 	net/khttpd/make_times_h \
 	net/khttpd/times.h \
-	submenu*
+	submenu* \
+	arch/$(ARCH)/asm-offsets.[sh]
 # directories removed with 'make clean'
 CLEAN_DIRS = \
 	modules
@@ -245,9 +246,9 @@ export	CPPFLAGS CFLAGS AFLAGS
 export	NETWORKS DRIVERS LIBS HEAD LDFLAGS LINKFLAGS MAKEBOOT ASFLAGS
 
 .S.s:
-	$(CPP) $(AFLAGS) -traditional -o $*.s $<
+	$(CPP) $(AFLAGS) $(AFLAGS_$@) -traditional -o $@ $<
 .S.o:
-	$(CC) $(AFLAGS) -traditional -c -o $*.o $<
+	$(CC) $(AFLAGS) $(AFLAGS_$@) -traditional -c -o $@ $<
 
 Version: dummy
 	@rm -f include/linux/compile.h
@@ -273,29 +274,63 @@ symlinks:
 		mkdir include/linux/modules; \
 	fi
 
-oldconfig: symlinks
+# There is a race here against anything built by the top level Makefile or by
+# arch/$(ARCH)/Makefile (directly included by the top level Makefile).
+
+# split-include changes timestamps on config settings without telling make.
+# Because make is not aware that config files have been changed, the timestamp
+# checks in .hdepend (see Rules.make) use the old times for config files.
+# Therefore anything built at the same time that split-include is running gets
+# wrong timestamps for config settings and for the include files that depend on
+# config settings.
+
+# I cannot completely fix this in kbuild 2.4 without moving most of the top
+# level Makefile into a second file and doing make -C on that file after
+# split-include has run.  Explicitly running split-include after *config instead
+# of delaying until make dep mitigates the race, although it does not completely
+# remove it.
+
+# The race is not a problem for SUBDIRS processed from the top level Makefile
+# because split-include has completed before they are entered.  However it is a
+# problem for top level builds and for directories invoked from
+# arch/$(ARCH)/Makefile without using SUBDIRS.
+
+# kbuild 2.5 has a clean separation between configuration (top level and
+# scripts/Makefile) and the rest of the kernel build, so it does not suffer from
+# this race.  Also kbuild 2.5 splits arch/$(ARCH)/Makefile into definitions
+# (Makefile.defs.{no}config) and kernel building commands (Makefile.in).
+# KAO
+
+split_include := scripts/split-include include/linux/autoconf.h include/config && touch include/config/MARKER
+
+oldconfig: symlinks scripts/split-include
 	$(CONFIG_SHELL) scripts/Configure -d arch/$(ARCH)/config.in
+	@$(split_include)
 
-xconfig: symlinks
+xconfig: symlinks scripts/split-include
 	$(MAKE) -C scripts kconfig.tk
 	wish -f scripts/kconfig.tk
+	@$(split_include)
 
-menuconfig: include/linux/version.h symlinks
+menuconfig: include/linux/version.h symlinks scripts/split-include
 	$(MAKE) -C scripts/lxdialog all
 	$(CONFIG_SHELL) scripts/Menuconfig arch/$(ARCH)/config.in
+	@$(split_include)
 
-config: symlinks
+config: symlinks scripts/split-include
 	$(CONFIG_SHELL) scripts/Configure arch/$(ARCH)/config.in
+	@$(split_include)
 
 include/config/MARKER: scripts/split-include include/linux/autoconf.h
-	scripts/split-include include/linux/autoconf.h include/config
-	@ touch include/config/MARKER
+	@$(split_include)
 
 linuxsubdirs: $(patsubst %, _dir_%, $(SUBDIRS))
 
-$(patsubst %, _dir_%, $(SUBDIRS)) : dummy include/linux/version.h include/config/MARKER
+$(patsubst %, _dir_%, $(SUBDIRS)) : dummy include/linux/version.h include/config/MARKER before_subdirs
 	$(MAKE) CFLAGS="$(CFLAGS) $(CFLAGS_KERNEL)" -C $(patsubst _dir_%, %, $@)
 
+.PHONY: before_subdirs
+
 $(TOPDIR)/include/linux/version.h: include/linux/version.h
 $(TOPDIR)/include/linux/compile.h: include/linux/compile.h
 
@@ -356,7 +391,7 @@ endif
 modules: $(patsubst %, _mod_%, $(SUBDIRS))
 
 .PHONY: $(patsubst %, _mod_%, $(SUBDIRS))
-$(patsubst %, _mod_%, $(SUBDIRS)) : include/linux/version.h include/config/MARKER
+$(patsubst %, _mod_%, $(SUBDIRS)) : include/linux/version.h include/config/MARKER before_subdirs
 	$(MAKE) -C $(patsubst _mod_%, %, $@) CFLAGS="$(CFLAGS) $(MODFLAGS)" MAKING_MODULES=1 modules
 
 .PHONY: modules_install
Index: 11-pre4.1/arch/i386/asm-offsets.c
--- 11-pre4.1/arch/i386/asm-offsets.c Sat, 06 Oct 2001 15:07:38 +1000 kaos ()
+++ 11-pre4.1(w)/arch/i386/asm-offsets.c Fri, 05 Oct 2001 21:53:18 +1000 kaos (linux-2.4/q/f/50_asm-offset  644)
@@ -0,0 +1,32 @@
+/*
+ * Generate definitions needed by assembly language modules.
+ * This code generates raw asm output which is post-processed to extract
+ * and format the required data.
+ */
+
+#include <linux/types.h>
+#include <linux/stddef.h>
+#include <linux/sched.h>
+
+/* Use marker if you need to separate the values later */
+
+#define DEFINE(sym, val, marker) \
+  asm volatile("\n-> " #sym " %0 " #val " " #marker : : "i" (val))
+
+#define BLANK() asm volatile("\n->" : : )
+
+int
+main(void)
+{
+  DEFINE(state,        offsetof(struct task_struct, state),);
+  DEFINE(flags,        offsetof(struct task_struct, flags),);
+  DEFINE(sigpending,   offsetof(struct task_struct, sigpending),);
+  DEFINE(addr_limit,   offsetof(struct task_struct, addr_limit),);
+  DEFINE(exec_domain,  offsetof(struct task_struct, exec_domain),);
+  DEFINE(need_resched, offsetof(struct task_struct, need_resched),);
+  DEFINE(tsk_ptrace,   offsetof(struct task_struct, ptrace),);
+  DEFINE(processor,    offsetof(struct task_struct, processor),);
+  BLANK();
+  DEFINE(ENOSYS,       ENOSYS,);
+  return 0;
+}
Index: 11-pre4.1/arch/i386/kernel/entry.S
--- 11-pre4.1/arch/i386/kernel/entry.S Mon, 01 Oct 2001 12:23:40 +1000 kaos (linux-2.4/S/c/24_entry.S 1.1.5.5 644)
+++ 11-pre4.1(w)/arch/i386/kernel/entry.S Sat, 06 Oct 2001 13:04:03 +1000 kaos (linux-2.4/S/c/24_entry.S 1.1.5.5 644)
@@ -68,20 +68,7 @@ IF_MASK		= 0x00000200
 NT_MASK		= 0x00004000
 VM_MASK		= 0x00020000
 
-/*
- * these are offsets into the task-struct.
- */
-state		=  0
-flags		=  4
-sigpending	=  8
-addr_limit	= 12
-exec_domain	= 16
-need_resched	= 20
-tsk_ptrace	= 24
-processor	= 52
-
-ENOSYS = 38
-
+#include "asm-offsets.h"
 
 #define SAVE_ALL \
 	cld; \
Index: 11-pre4.1/arch/i386/kernel/Makefile
--- 11-pre4.1/arch/i386/kernel/Makefile Wed, 19 Sep 2001 14:59:20 +1000 kaos (linux-2.4/S/c/21_Makefile 1.1.2.1 644)
+++ 11-pre4.1(w)/arch/i386/kernel/Makefile Fri, 05 Oct 2001 22:37:10 +1000 kaos (linux-2.4/S/c/21_Makefile 1.1.2.1 644)
@@ -8,7 +8,7 @@
 # Note 2! The CFLAGS definitions are now in the main makefile...
 
 .S.o:
-	$(CC) $(AFLAGS) -traditional -c $< -o $*.o
+	$(CC) $(AFLAGS) $(EXTRA_AFLAGS) $(AFLAGS_$@) -traditional -c $< -o $@
 
 all: kernel.o head.o init_task.o
 
@@ -40,5 +40,8 @@ obj-$(CONFIG_SMP)		+= smp.o smpboot.o tr
 obj-$(CONFIG_X86_LOCAL_APIC)	+= mpparse.o apic.o nmi.o
 obj-$(CONFIG_X86_IO_APIC)	+= io_apic.o
 obj-$(CONFIG_X86_VISWS_APIC)	+= visws_apic.o
+
+AFLAGS_entry.o += -I $(TOPDIR)/arch/$(ARCH)
+entry.o: $(TOPDIR)/arch/$(ARCH)/asm-offsets.h
 
 include $(TOPDIR)/Rules.make
Index: 11-pre4.1/arch/i386/Makefile
--- 11-pre4.1/arch/i386/Makefile Wed, 18 Apr 2001 11:00:10 +1000 kaos (linux-2.4/T/c/37_Makefile 1.1.2.1.2.1 644)
+++ 11-pre4.1(w)/arch/i386/Makefile Sat, 06 Oct 2001 15:07:31 +1000 kaos (linux-2.4/T/c/37_Makefile 1.1.2.1.2.1 644)
@@ -145,3 +145,45 @@ archmrproper:
 
 archdep:
 	@$(MAKEBOOT) dep
+
+# Convert raw asm offsets into something that can be included as
+# assembler definitions.  It converts
+#   -> symbol $value source
+# into
+#   symbol = value /* 0xvalue source */
+
+arch/$(ARCH)/asm-offsets.h: arch/$(ARCH)/asm-offsets.s
+	@echo Creating $@
+	@set -e; \
+	  (echo "#ifndef __ASM_OFFSETS_H__"; \
+	   echo "#define __ASM_OFFSETS_H__"; \
+	   echo "/*"; \
+	   echo " * DO NOT MODIFY"; \
+	   echo " *"; \
+	   echo " * This file was generated by arch/$(ARCH)/Makefile."; \
+	   echo " *"; \
+	   echo " */"; \
+	   echo ""; \
+	   awk "/^->\$$/{printf(\"\\n\");} \
+	     /^-> /{ \
+	       sym = \$$2; \
+	       val = \$$3; \
+	       sub(/^\\\$$/, \"\", val); \
+	       \$$1 = \"\"; \
+	       \$$2 = \"\"; \
+	       \$$3 = \"\"; \
+	       printf(\"%-20s = %3d\\t\\t\\t/* 0x%x\\t%s */\\n\", \
+	         sym, val, val, \$$0) \
+	     }"; \
+	   echo ""; \
+	   echo "#endif"; \
+	  ) < $< > $@;
+
+arch/$(ARCH)/asm-offsets.s: include/config/MARKER \
+	include/linux/sched.h \
+	include/linux/types.h \
+	include/linux/stddef.h
+
+# Kludge to prevent subdirs from being processed before asm-offsets.h is created.
+# I hate recursive make!  KAO.
+before_subdirs: arch/$(ARCH)/asm-offsets.h

Re: [RFC] Standard way of generating assembler offsets

[Pantelis Antoniou]

Hi there.

If anyone is interested I have already made a perl
script that produces assembler offsets from structure
members.

It doesn't need to run native since it reads the
header files, extract the structures and by using
objdump calculates the offsets automatically.

Maybe it needs some more work for what you describe, 
but it's exactly what you describe.

If you're interested please email me directly for
more information.

Regards

Re: [RFC] Standard way of generating assembler offsets

[george anzinger]

Pantelis Antoniou wrote:
> 
> Hi there.
> 
> If anyone is interested I have already made a perl
> script that produces assembler offsets from structure
> members.
> 
> It doesn't need to run native since it reads the
> header files, extract the structures and by using
> objdump calculates the offsets automatically.
> 
> Maybe it needs some more work for what you describe,
> but it's exactly what you describe.
> 
> If you're interested please email me directly for
> more information.
> 
One of the problems with this sort of thing is that it has a hard time
getting the CPP macros right.  The best way to do this sort of thing is
to actually compile the header file with all the CONFIG defines and a
set of tools (read macros) that produce the required offsets.  This way
you get what you want and don't have to reinvent the CPP stuff.  It also
allows production of #define constants and other constructs that folks
push into CPP, in a very simple and straight forward manner.

Been there, done that.

George

Re: [RFC] Standard way of generating assembler offsets

[Georg Nikodym]

At the risk of sticking my foot in it, is there something wrong with the
ANSI C offsetof() macro, defined in <stddef.h>?

--Georg

Re: [RFC] Standard way of generating assembler offsets

[george anzinger]

Georg Nikodym wrote:
> 
> At the risk of sticking my foot in it, is there something wrong with the
> ANSI C offsetof() macro, defined in <stddef.h>?
> 
> --Georg
No, and it could have been (and was) written prio to ANSI C defining
it.  Something like:

#define offsetof(x, instruct) &((struct instruct *)0)->x

The issues that CPP resolves have to deal with the following sort of
structure:

struct instruct {
	struct foo * bar;
#ifdef CONFIG_OPTION_DIDDLE
	int diddle_flag;
	int diddle_array[CONFIG_DIDDLE_SIZE];
#endif
	int x;
}

Or for the simple need for a constant:

#define Const (CONFIG_DIDDLE_SIZE * sizeof(int))

Of course you could have any number of constant operators in the
expression.  Note also, that the array in the structure above is defined
by a CONFIG symbol.  This could also involve math, i.e.:

#define CONFIG_DIDDLE_SIZE CLOCK_RATE / HZ

and so on.  All in all, it best to let CPP do what it does best and
scarf up the result:

#define GENERATE_CONSTANT(name,c) printf(#name " equ %d\n",c)

then:

GENERATE_CONSTANT(diddle_size,CONFIG_DIDDLE_SIZE);

In the code we did, we put all the GENERATE macros in a .h file.  The
the code looked like:

#include.... all the headers needed....

#include <generate.h>

GENERATE....  all the generate macro calls...

} // all done (assumes that the "main(){" is in the generate.h file)

This whole mess was included as comments in the asm file.  The make rule
then used a sed script to extract it, compile and run it to produce a
new header file which the asm source included outside of the above
stuff.

George

Re: [RFC] Standard way of generating assembler offsets

[Pantelis Antoniou]

[-- Attachment #1: Type: text/plain, Size: 2041 bytes --]

george anzinger wrote:
> 
> Georg Nikodym wrote:
> >
> > At the risk of sticking my foot in it, is there something wrong with the
> > ANSI C offsetof() macro, defined in <stddef.h>?
> >
> > --Georg
> No, and it could have been (and was) written prio to ANSI C defining
> it.  Something like:
> 
> #define offsetof(x, instruct) &((struct instruct *)0)->x
> 
> The issues that CPP resolves have to deal with the following sort of
> structure:
> 
> struct instruct {
>         struct foo * bar;
> #ifdef CONFIG_OPTION_DIDDLE
>         int diddle_flag;
>         int diddle_array[CONFIG_DIDDLE_SIZE];
> #endif
>         int x;
> }
> 
> Or for the simple need for a constant:
> 
> #define Const (CONFIG_DIDDLE_SIZE * sizeof(int))
> 
> Of course you could have any number of constant operators in the
> expression.  Note also, that the array in the structure above is defined
> by a CONFIG symbol.  This could also involve math, i.e.:
> 
> #define CONFIG_DIDDLE_SIZE CLOCK_RATE / HZ
> 
> and so on.  All in all, it best to let CPP do what it does best and
> scarf up the result:
> 
> #define GENERATE_CONSTANT(name,c) printf(#name " equ %d\n",c)
> 
> then:
> 
> GENERATE_CONSTANT(diddle_size,CONFIG_DIDDLE_SIZE);
> 
> In the code we did, we put all the GENERATE macros in a .h file.  The
> the code looked like:
> 
> #include.... all the headers needed....
> 
> #include <generate.h>
> 
> GENERATE....  all the generate macro calls...
> 
> } // all done (assumes that the "main(){" is in the generate.h file)
> 
> This whole mess was included as comments in the asm file.  The make rule
> then used a sed script to extract it, compile and run it to produce a
> new header file which the asm source included outside of the above
> stuff.
> 
> George

My script already handles that, everything is first passed through
CPP and the member offset are varied correctly according to any
compilation options.

I included the script and the results of two runs.

1. ./h2inc tst.h >tst.inc

2. ./h2inc --cflags="-DSHOW_HIDDEN -I./" >tst-hidden.inc

Regards

[-- Attachment #2: h2inc --]
[-- Type: text/plain, Size: 18360 bytes --]

#!/usr/bin/perl -w

use integer;
use Getopt::Long;
use File::Basename;
use File::stat;

my $CC       = "cc";
my $CFLAGS   = "-I./";
my $CPPFLAGS = "-E -dD";
my $OBJCOPY  = "objcopy";
my $OBJFLAGS = "-O binary";

sub find_32bit_type;
sub target_endianess;
sub alligned_type_size;
sub base_type_size;
sub members_offset;
sub tmpfile;
sub inputfile;
sub decode_type;
sub decode_member;
sub find_complex_member_name;
sub find_matching_brace;

$Getopt::Long::ignorecase = 0;	# Don't ignore case

my @filelist = ();

GetOptions	(
	"cc=s" 			=> \$CC,
	"cflags=s" 		=> \$CFLAGS,
	"cppflags=s" 	=> \$CPPFLAGS,
	"objcopy=s" 	=> \$OBJCOPY,
	"objflags=s" 	=> \$OBJFLAGS
			);

`$CC 2>/dev/null -v`;
die "Compiler is not present", if $? != 0;

my $u32 		= &find_32bit_type();
my $endianess 	= &target_endianess($u32);

for ($i = 0; $i <= $#ARGV; $i++) {
	$_ = $ARGV[$i];
	# print STDERR "file: $_\n";
	push @filelist, $_;
}

$#filelist >= 0 || die "Filename missing\n";

my %members  = ();
my %typedefs = ();
my %structs  = ();
my %unions   = ();

my %ilist = ();	# hash of included files
my $ilist;

my $defines;

foreach $f (@filelist) {

	$f = basename($f);

	%ilist = ();
	undef $ilist;

	$defines = "";

	&inputfile($f);

	my @defines = split(/\n/, $defines);

	my $incfile = $f;
	$incfile =~ s/\.h$/.inc/g;
	$incfile =~ s/\S*\/(\S+\.inc)/$1/g;
	my $incfiledef = "_" . uc($incfile);
	$incfiledef =~ s/\./_/g;

	print "#ifndef $incfiledef\n";
	print "#define $incfiledef\n\n";

	if (defined ($ilist)) {
		my $if;
		foreach $if (split /\s/, $ilist) {
			$if =~ s/\.h$/.inc/g;
			$if =~ s/\S*\/(\S+\.inc)/$1/g;
			$if = basename($if);
			print "#include \"$if\"\n";
		}
		print "\n";
	}

	my @offsets;
	my $b;
	my $i;
	my $j;
	my $k;
	foreach $b (sort keys %members) {
		my @m = split /\s/, $members{$b};
		my $m;
		print "/****************************************************************\n\n";
		print "\tOffsets for $b\n\n";
		print "****************************************************************/\n\n";

		my $size_define;
		my $complete_type;

		if ($typedefs{$b}) {
			$complete_type = $b;
			$size_define = $b . "_SIZE";
		} elsif ($structs{$b}) {
			$complete_type = "struct $b";
			$size_define = "struct_" . $b . "_SIZE";
		} elsif ($unions{$b}) {
			$complete_type = "union $b";
			$size_define = "union_" . $b . "_SIZE";
		} else { die; }

		@offsets = &members_offset($endianess, $u32, "#include \"$f\"\n", $complete_type, \@m);

		$j = 0;
		foreach (@offsets) {
			my $cm = $m[$j++];
			my @cm = split(/\./, $cm);
			for ($k = 0; $k <= $#defines; $k++) {
				my @tt = split(/\s+/, $defines[$k]);
				if (defined($tt[2]) && $tt[2] eq $cm) {
					# print STDERR "define alias found for $cm\n";
					$defines[$k] = "/* $tt[1] removed as an alias for $cm */";
					@cm = ($tt[1]);
					last;
				}
			}
			$cm = join('_', @cm);
			printf("#define %-30s %3d\n", $cm, $_);
		}

		printf("#define %-30s %3d\n", $size_define, &base_type_size($endianess, $u32, "#include \"$f\"\n", $complete_type));

		print "\n";
	}

	$defines = join("\n", @defines);

	if ($defines ne "") {
		print "/****************************************************************\n\n";
		print "\tSimple defines list \n\n";
		print "****************************************************************/\n\n";
		print "$defines\n\n";
	}

	print "#endif\n";

}

sub inputfile {
	my $file = shift(@_);
	$file = basename($file);
	my @wf = ();	# whole file
	my $wf = '';
	my $size;
	my $align;

	# print STDERR "Processing : $file\n";

	my $tmp = &tmpfile();
	my $tmp_c = $tmp . ".c";
	my $tmp_o = $tmp . ".o";

	local $SIG{'INT'} =
		sub {
			unlink $tmp_c;
			unlink $tmp_o;
			die $_[0];
		};
	
	open(TMPFILE, ">$tmp_c") || die;
	print TMPFILE "#include \"$file\"\n";
	close(TMPFILE);
	# first verify that the header file is correct
	`$CC $CFLAGS -c $tmp_c -o $tmp_o`; 
	if ($? != 0) {
		# try system wide include
		open(TMPFILE, ">$tmp_c") || die;
		print TMPFILE "#include \<$file\>\n";
		close(TMPFILE);
		# first verify that the header file is correct
		`$CC $CFLAGS -c $tmp_c -o $tmp_o`; die, if ($? != 0);
	}
	unlink $tmp_o;	# remove object file

	# open(LOGFILE, ">log.i") || die;

	# print LOGFILE "$CC $CFLAGS $CPPFLAGS $tmp_c |\n";

	open(CPPPIPE, "$CC $CFLAGS $CPPFLAGS $tmp_c |") || die;

	$wf = "";
	my $cf = "";	# current file - only output for current file 

	my $last;
	my $lll;

	while (<CPPPIPE>) {
		# print LOGFILE $_;
		$lll = $_;
		chop;
		if (s/\\$//) {
			$_ .= <CPPPIPE>; redo;
		}	# check for escape at the end of line
		$last = $_;
		my $ll = $_;
		if (! /\s*\#\s*/) {
			if (basename($cf) eq basename($file)) {
				$wf .= "$_\n";
			}
			next;
		}
		# print STDERR "\$\_='$_'\n";
		# print STDERR "o: \$\_='$_', \$\`='$`', \$\&='$&', \$\'='$'\n";
		if (/\s+[0-9]+\s*\"([^\"].*)\"/) {
			# print STDERR "$cf - $1\n";
			my $c = basename($1);
			if (!defined($ilist{$c})) {
				next, if ($c eq basename($tmp_c));				# do not add the dummy 
				if ($c ne basename($file) &&
					basename($cf) eq basename($file)) { # do not add self, and only directly included
					# print STDERR "$ll: $1\n";	
					$ilist{$c} = 1;
					if (defined($ilist)) {
						$ilist .= " $c";
					} else {
						$ilist = $c;
					}
				}
			}
			$cf = basename($1);
		} elsif (/\s*define\s*([a-zA-Z_][a-zA-Z0-9_]*)/) {
			# $_ = $';
			next, if ($cf ne $file);
			# print STDERR "\$lll='$lll'\n\$last='$last'\n\$\_='$_'\n\$\`='$`'\n\$\&='$&'\n\$\'='$'\n";
			my $defname = $1;
			next, if (/\(.*\)/);	# ignore arguments
			$_ = $';
			if (/\S+.*$/) {
				my $what = $&;
				# next, if ($what =~ /[()]/);	# only simple defines pass
				# $defines .= "#define $defname $1\n";
				# print STDERR "1-> '$defname' '$what'\n";
				$defines .= sprintf("#define %-30s %s\n", $defname, $what);
			} else {
				# print STDERR "2-> $defname\n";
				$defines .= sprintf("#define %-30s\n", $defname);
			}
		} else {
			# print STDERR "$_";
		}
	}

	# close(LOGFILE);

	# print STDERR "FILE:\n" . $wf ."FEND:\n";
	$wf =~ s/([*;,.!~{}()+\-\\\/\[\]])/ $1 /gsx;	# add spaces
	$wf =~ s/\s+/ /gsx;
	# print STDERR $wf;
	@wf = split /\s/,  $wf;

	$i  = 0;
	OUTTER: while ($i < ($#wf + 1)) {
		# print STDERR "$i: '$wf[$i]'\n";
		if ($wf[$i] eq "typedef" && $wf[$i+1] eq "struct") {
			&decode_type(\@wf, \$i);
		} elsif ($wf[$i] eq "typedef" && $wf[$i+1] eq "union") {
			&decode_type(\@wf, \$i);
		} elsif ($wf[$i] eq "struct") {
			&decode_type(\@wf, \$i);
		} elsif ($wf[$i] eq "union") {
			&decode_type(\@wf, \$i);
		} else {
			$i++;
		}
	}
	close CPPPIPE;

	unlink $tmp_c;

	local $SIG{'INT'} = 'DEFAULT';

	return 1;
}

sub decode_member {
	my $wf = shift(@_);	# reference to the whole body of the file
	my $ms = shift(@_);	# start of member definition
	my $me = shift(@_);	# end of member definition
	my $ii;
	my $i;
	my $n;

	# decode right to left
	$i = $me;
	$i--, if ($$wf[$i] eq ";");
	while ($$wf[$i] eq "]") {	# array definition, find match 
		# print STDERR "'$$wf[$i]'\n";
		$n = 1;
		do {
			$i--;
			# print STDERR "'$$wf[$i]'\n";
			$n++, if ($$wf[$i] eq "]");
			$n--, if ($$wf[$i] eq "[");
		} while ($n > 0 || $$wf[$i] ne "[");
		$i--;
	}
	while ($$wf[$i] eq ")") {	# function pointer definition, find match 
		$n = 1;
		$ii = $i;
		# print STDERR "'$$wf[$ii]'\n";
		do {
			$ii--;
			# print STDERR "'$$wf[$ii]'\n";
			$n++, if ($$wf[$ii] eq ")");
			$n--, if ($$wf[$ii] eq "(");
		} while ($n > 0 || $$wf[$ii] ne "(");
		# now check if next token is a pointer then we've found it
		# print STDERR "i=$i, ii=$ii, '" . $$wf[$ii] . "' '" . $$wf[$ii+1] . "' '" . $$wf[$ii+2] . "'\n";
		if ($i - $ii == 3 && $$wf[$ii+1] eq "*") {	# found it!
			$i = $ii + 2;
		} else {
			$i = $ii - 1;
		}
	}
	# print STDERR "found; $$wf[$i]\n";
	return $$wf[$i];
}

sub find_complex_member_name {
	my $wf   = shift(@_);	# reference to the whole body of the file
	my $k    = shift(@_);	# start of member definition
	my $last = shift(@_);	# end of member definition

	my $ct = $$wf[$k];
	my $ccs = $k + 2;
	my $n = 0;
	do {
		$k++;
		$n++, if ($$wf[$k] eq "{");
		$n--, if ($$wf[$k] eq "}");
	} while ($n > 0);
	my $cce = $k;
	$k++;
	my $cs = $k;
	$k++, while ($$wf[$k] ne ";");
	my $ce = $k;
	$k++;
	my $cn = &decode_member($wf, $cs, $ce);
	return $cn;
}

sub find_matching_brace {
	my $wf   = shift(@_);	# reference to the whole body of the file
	my $k    = shift(@_);	# start of member definition
	my $last = shift(@_);	# end of member definition

	my $n = 0;
	do {
		$k++;
		$n++, if ($$wf[$k] eq "{");
		$n--, if ($$wf[$k] eq "}");
	} while ($n > 0);
	$k++;
	$k++, while ($$wf[$k] ne ";");
	$k++;
	return $k;
}

sub get_next_type_name {
}

sub decode_type {
	my $wf = shift(@_);	# reference to the whole body of the file
	my $i  = shift(@_);	# reference to the token index at the file
	my $n = 0;			# nesting level
	my $k;
	my $j;
	my $anchor;
	my $first_brace;
	my $last_brace;
	my $trailer;
	my @result = ();

	$anchor = $$i;		# keep this for later
	# now find the brace
	$j = $anchor;
	# first find opening brace or terminating semicolon
	$j++, while ($$wf[$j] ne "{" && $$wf[$j] ne ";");

	if ($$wf[$j] eq ";") {	# not a structure definition
		$$i = $j + 1;
		return "";
	}
	$first_brace = $j;

	$n = 1;
	do {
		$j++;
		$n++, if ($$wf[$j] eq "{");
		$n--, if ($$wf[$j] eq "}");
	} while ($n > 0 || $$wf[$j] ne "}");

	$last_brace = $j;

	# find terminating semicolon
	$j++, while ($$wf[$j] ne ";");

	my $is_typedef = $$wf[$anchor] eq "typedef";
	my $is_struct  = $$wf[$anchor] eq "struct";
	my $is_union   = $$wf[$anchor] eq "union";
	my $is_declaration = 0;
	my $base_type;

	if ($is_typedef) {
		my $ii = $j - 1;	# remove semicolon
		# print STDERR "is_typedef\n";
		# print STDERR "'$$wf[$ii]' ";
		while ($$wf[$ii] eq "]") {	# array definition, find match 
			$n = 1;
			do {
				$ii--;
				# print STDERR "'$$wf[$ii]' ";
				$n++, if ($$wf[$ii] eq "]");
				$n--, if ($$wf[$ii] eq "[");
			} while ($n > 0 || $$wf[$ii] ne "[");
			$ii--;
		}
		$base_type = $$wf[$ii];
		if ($$wf[$j-1] eq "]") {
			$$i = $j;
			print STDERR "typedef arrays not supported; skipping $base_type\n";
			return "";
		}
		$typedefs{$base_type} = $is_typedef;
		$structs{$base_type}  = $is_struct;
		$unions{$base_type}   = $is_union;
	} else {
		# print STDERR "!is_typedef\n";
		$base_type = $$wf[$anchor] . " " . $$wf[$anchor + 1];
		$base_type = $$wf[$anchor + 1];
		if ($base_type ne "{") {
			$typedefs{$base_type} = $is_typedef;
			$structs{$base_type}  = $is_struct;
			$unions{$base_type}   = $is_union;
		} else {
			$is_declaration = 1;
		}
	}

	# print STDERR "sizeof($base_type)\n";
	my $ms; my $me; my $member; my @nt = ();
	my $cs; my $ce; my $ct; my $cn; my $ccs; my $cce;

	$ct = "";
	$n = 0;
	for ($ms = $first_brace + 1, $me = $first_brace+1; $me < $last_brace; ) {

		# print STDERR "$$wf[$me-1] \>$$wf[$me]\< $$wf[$me+1]\n";

		if (($$wf[$me] eq "union" || $$wf[$me] eq "struct") && $$wf[$me+1] eq "{") {
			$cn = &find_complex_member_name($wf, $me, $last_brace);
			# print STDERR "found $ct $cn\n"; 
			if (! $is_declaration) {
				if ($ct eq "struct" || length($cn) > 2) {
					if (defined($members{$base_type})) {
						$members{$base_type} .= " $cn";
					} else {
						$members{$base_type} = $cn;
					}
				}
			}
			$ms = $me + 2; $me = $ms;
			push @nt, $cn;
			# print STDERR "pushed $cn, \@nt = @nt\n"; 
			next;
		}
		if ($$wf[$me] eq "}") {
			$me++;
			$cn = pop @nt;
			# print STDERR "poped $cn, \@nt = @nt\n"; 
			$me++, while ($me < $last_brace && $$wf[$me] ne ";");
			$ms = $me + 1; $me = $ms;
			next;
		}

		$me++, while ($me < $last_brace && $$wf[$me] ne ";");
		last, if ($me >= $last_brace);

		$me--;	# remove semicolon
		last, if ($ms >= $me);	# protection for empty types

		if (! $is_declaration) {
			$member = &decode_member($wf, $ms, $me);
			# print STDERR "offsetof($base_type, $member)\n";
			$member = join('.', @nt, $member);
			if (defined($members{$base_type})) {
				$members{$base_type} .= " $member";
			} else {
				$members{$base_type} = $member;
			}
		}
		$ms = $me + 2; $me = $ms;
	}

	$$i = $j;

	return "";
}

sub tmpfile {
	my $tmp_dir  = -d '/tmp' ? '/tmp' : $ENV{TMP} || $ENV{TEMP};
	my $tmp_name = sprintf("%s/%s-%d-%d", $tmp_dir, basename($0), $$, time());
	return $tmp_name;
}

sub target_endianess {
	my $thirty_two_bits_type = shift(@_);
	my $extra_code = shift(@_);
	if (!defined($extra_code)) { $extra_code = ""; }
	my $tmp_dir  = -d '/tmp' ? '/tmp' : $ENV{TMP} || $ENV{TEMP};
	my $tmp_name = sprintf("%s/%s-%d-%d", $tmp_dir, basename($0), $$, time());
	my $tmp_c    = "$tmp_name.c";
	my $tmp_o    = "$tmp_name.o";
	my $tmp_bin  = "$tmp_name.bin";
	my $t_size;
	my $sb;
	my $fsize;
	my $i;

	unlink $tmp_c;

	local $SIG{'INT'} =
		sub {
			unlink $tmp_c;
			unlink $tmp_o;
			unlink $tmp_bin;
			die $_[0];
		};
	
	open(TMPFILE, ">$tmp_c") || die "Could not create temp file";
	print TMPFILE "$extra_code\n";
	print TMPFILE "const $thirty_two_bits_type tmp_val = 0x01234567LU;\n"; 
	close(TMPFILE);

	`$CC $CFLAGS -c $tmp_c -o $tmp_o`;		die, if $? != 0;
	`$OBJCOPY $OBJFLAGS $tmp_o $tmp_bin`;	die, if $? != 0;

	$sb = stat($tmp_bin);
	$fsize = $sb->size;

	my $read_bytes;
	my $buf;
	open(TMPFILE, "<$tmp_bin") || die "Could not open binary file";
	($read_bytes = read TMPFILE, $buf, 4) == 4 || die "File has illegal size";
	my @v;
	($v[0], $v[1], $v[2], $v[3]) = unpack("C4", $buf);
	close(TMPFILE);

	unlink $tmp_c;
	unlink $tmp_o;
	unlink $tmp_bin;

	local $SIG{'INT'} = 'DEFAULT';

	if ($v[0] == 0x01 && $v[1] == 0x23 && $v[2] == 0x45 && $v[3] == 0x67) {
		return("big");
	}
	if ($v[3] == 0x01 && $v[2] == 0x23 && $v[1] == 0x45 && $v[0] == 0x67) {
		return("little");
	}
	return("unknown");
}

sub alligned_type_size {
	my $type = shift(@_);
	my $type_val = shift(@_);
	if (!defined($type_val)) { $type_val = "0"; }
	my $array_size = shift(@_);
	if (!defined($array_size)) { $array_size = 16; }
	my $extra_code = shift(@_);
	if (!defined($extra_code)) { $extra_code = ""; }

	my $tmp_dir  = -d '/tmp' ? '/tmp' : $ENV{TMP} || $ENV{TEMP};
	my $tmp_name = sprintf("%s/%s-%d-%d", $tmp_dir, basename($0), $$, time());
	my $tmp_c    = "$tmp_name.c";
	my $tmp_o    = "$tmp_name.o";
	my $tmp_bin  = "$tmp_name.bin";
	my $t_size;
	my $sb;
	my $fsize;
	my $i;

	# first find out the size of the size_t type

	unlink $tmp_c;

	local $SIG{'INT'} =
		sub {
			unlink $tmp_c;
			unlink $tmp_o;
			unlink $tmp_bin;
			die $_[0];
		};

	open(TMPFILE, ">$tmp_c") || die "Could not create temp file";
	print TMPFILE "$extra_code\n";
	print TMPFILE "const $type tmp_array[$array_size] = {\n";
	for ($i = 0; $i < $array_size; $i++) {
		print TMPFILE "$type_val,\n";
	}
	print TMPFILE "};\n";
	close(TMPFILE);

	`$CC $CFLAGS -c $tmp_c -o $tmp_o`;		die, if $? != 0;
	`$OBJCOPY $OBJFLAGS $tmp_o $tmp_bin`;	die, if $? != 0;

	$sb = stat($tmp_bin);
	$fsize = $sb->size;

	$t_size = $fsize / $array_size;

	unlink $tmp_c;
	unlink $tmp_o;
	unlink $tmp_bin;

	local $SIG{'INT'} = 'DEFAULT';

	return $t_size;
}

sub members_offset {
	my $endianess	= shift(@_);	# big or little
	my $thirty_two_bits_type = shift(@_);
	my $extra_code  = shift(@_);	# 
	my $base   		= shift(@_);	# base type
	my $members		= shift(@_);	# member to find the offset
	my $offset = 0;
	my @offsets = ();
	my $i;
	my $j;

	my $tmp_dir  = -d '/tmp' ? '/tmp' : $ENV{TMP} || $ENV{TEMP};
	my $tmp_name = sprintf("%s/%s-%d-%d", $tmp_dir, basename($0), $$, time());
	my $tmp_c    = "$tmp_name.c";
	my $tmp_o    = "$tmp_name.o";
	my $tmp_bin  = "$tmp_name.bin";

	unlink $tmp_c;

	local $SIG{'INT'} =
		sub {
			unlink $tmp_c;
			unlink $tmp_o;
			unlink $tmp_bin;
			die $_[0];
		};

	open(TMPFILE, ">$tmp_c") || die "Could not create temp file";
	print TMPFILE "$extra_code\n";
	print TMPFILE "#include <stddef.h>\n";

	$j = $#$members + 1;
	print TMPFILE "const $thirty_two_bits_type offset_table[$j] = {\n";
	foreach (@$members) {
		print TMPFILE "\toffsetof($base, $_),\n";
	}
	print TMPFILE "};\n";

	close(TMPFILE);

	`$CC $CFLAGS -c $tmp_c -o $tmp_o`;		die, if $? != 0;
	`$OBJCOPY $OBJFLAGS $tmp_o $tmp_bin`;	die, if $? != 0;

	my $read_bytes;
	my $buf;
	open(TMPFILE, "<$tmp_bin") || die "Could not open binary file";
	
	for ($i = 0; $i < $j; $i++) {
		($read_bytes = read TMPFILE, $buf, 4) == 4 || die "File has illegal size";
		if ($endianess eq "big") {
			($offset) = unpack("N", $buf);	# big endian 32 bits
		} else {
			($offset) = unpack("V", $buf);	# little endian 32 bits
		}
		push @offsets, $offset;
	}

	close(TMPFILE);

	unlink $tmp_c;
	unlink $tmp_o;
	unlink $tmp_bin;

	local $SIG{'INT'} = 'DEFAULT';

	return(@offsets);
}


sub base_type_size {
	my $endianess	= shift(@_);	# big or little
	my $thirty_two_bits_type = shift(@_);
	my $extra_code  = shift(@_);	# 
	my $base   		= shift(@_);	# base type
	my $size = 0;

	my $tmp_dir  = -d '/tmp' ? '/tmp' : $ENV{TMP} || $ENV{TEMP};
	my $tmp_name = sprintf("%s/%s-%d-%d", $tmp_dir, basename($0), $$, time());
	my $tmp_c    = "$tmp_name.c";
	my $tmp_o    = "$tmp_name.o";
	my $tmp_bin  = "$tmp_name.bin";

	unlink $tmp_c;

	local $SIG{'INT'} =
		sub {
			unlink $tmp_c;
			unlink $tmp_o;
			unlink $tmp_bin;
			die $_[0];
		};

	open(TMPFILE, ">$tmp_c") || die "Could not create temp file";
	print TMPFILE "$extra_code\n";
	print TMPFILE "#include <stddef.h>\n";
	print TMPFILE "const $thirty_two_bits_type size = sizeof($base);\n";
	close(TMPFILE);

	`$CC $CFLAGS -c $tmp_c -o $tmp_o`;		die, if $? != 0;
	`$OBJCOPY $OBJFLAGS $tmp_o $tmp_bin`;	die, if $? != 0;

	my $read_bytes;
	my $buf;
	open(TMPFILE, "<$tmp_bin") || die "Could not open binary file";
	($read_bytes = read TMPFILE, $buf, 4) == 4 || die "File has illegal size";
	if ($endianess eq "big") {
		($size) = unpack("N", $buf);	# big endian 32 bits
	} else {
		($size) = unpack("V", $buf);	# little endian 32 bits
	}

	close(TMPFILE);

	unlink $tmp_c;
	unlink $tmp_o;
	unlink $tmp_bin;

	local $SIG{'INT'} = 'DEFAULT';

	return($size);
}

sub find_32bit_type {
	my $sizeof_int   = &alligned_type_size("int",    "0", 16, "");
	my $sizeof_short = &alligned_type_size("short",  "0", 16, "");
 	my $sizeof_long  = &alligned_type_size("long",   "0", 16, "");
	my $u32;

	if ($sizeof_int  == 4) {
		$u32 = "unsigned int"; $s32 = "signed int"; 
	} elsif ($sizeof_long == 4) {
		$u32 = "unsigned long"; $s32 = "signed long"; 
	} elsif ($sizeof_short == 4) {
		$u32 = "unsigned short"; $s32 = "signed short"; 
	} else {
		die "Not one 32 bit type found!\n";
	}
	return $u32;
}

[-- Attachment #3: tst.h --]
[-- Type: text/plain, Size: 201 bytes --]

#ifndef TST_H
#define TST_H

#include "tst2.h"

#define TST_DEF	0x10

struct tst {
	int tst_member_a;
	union {
		char *tst_u_str;
	} tst_member_b;
#ifdef SHOW_HIDDEN
	int tst_hidden;
#endif
};

#endif

[-- Attachment #4: tst2.h --]
[-- Type: text/plain, Size: 38 bytes --]

#ifndef TST2_H
#define TST2_H

#endif

[-- Attachment #5: tst.inc --]
[-- Type: text/plain, Size: 629 bytes --]

#ifndef _TST_INC
#define _TST_INC

#include "tst2.inc"

/****************************************************************

	Offsets for tst

****************************************************************/

#define tst_member_a                     0
#define tst_member_b                     4
#define tst_member_b_tst_u_str           4
#define struct_tst_SIZE                  8

/****************************************************************

	Simple defines list 

****************************************************************/

#define TST_H                         
#define TST_DEF                        0x10

#endif

[-- Attachment #6: tst-hidden.inc --]
[-- Type: text/plain, Size: 672 bytes --]

#ifndef _TST_INC
#define _TST_INC

#include "tst2.inc"

/****************************************************************

	Offsets for tst

****************************************************************/

#define tst_member_a                     0
#define tst_member_b                     4
#define tst_member_b_tst_u_str           4
#define tst_hidden                       8
#define struct_tst_SIZE                 12

/****************************************************************

	Simple defines list 

****************************************************************/

#define TST_H                         
#define TST_DEF                        0x10

#endif

[-- Attachment #7: tst2.inc --]
[-- Type: text/plain, Size: 241 bytes --]

#ifndef _TST2_INC
#define _TST2_INC

/****************************************************************

	Simple defines list 

****************************************************************/

#define TST2_H                        

#endif

Re: [RFC] Standard way of generating assembler offsets

[David S. Miller]

I think your work is way over-engineered and that you really
need to look at the very simple method by which we do this
on sparc64.

Franks a lot,
David S. Miller
davem@redhat.com

Re: [RFC] Standard way of generating assembler offsets

[David S. Miller]

   From: Pantelis Antoniou <panto@intracom.gr>
   Date: Mon, 08 Oct 2001 12:35:27 +0300
   
   If anyone is interested I have already made a perl
   script that produces assembler offsets from structure
   members.
   
   It doesn't need to run native since it reads the
   header files, extract the structures and by using
   objdump calculates the offsets automatically.

BTW, I assume you have already taken a look at how we
do this on Sparc64.  See arch/sparc64/kernel/check_asm.sh
and the "check_asm" target in arch/sparc64/kernel/Makefile

It also works in all cross-compilation etc. environments.
And I bet it would work on every platform with very minimal
changes, if any.

Franks a lot,
David S. Miller
davem@redhat.com

Re: [RFC] Standard way of generating assembler offsets

[Pantelis Antoniou]

"David S. Miller" wrote:
> 
>    From: Pantelis Antoniou <panto@intracom.gr>
>    Date: Mon, 08 Oct 2001 12:35:27 +0300
> 
>    If anyone is interested I have already made a perl
>    script that produces assembler offsets from structure
>    members.
> 
>    It doesn't need to run native since it reads the
>    header files, extract the structures and by using
>    objdump calculates the offsets automatically.
> 
> BTW, I assume you have already taken a look at how we
> do this on Sparc64.  See arch/sparc64/kernel/check_asm.sh
> and the "check_asm" target in arch/sparc64/kernel/Makefile
> 
> It also works in all cross-compilation etc. environments.
> And I bet it would work on every platform with very minimal
> changes, if any.
> 
> Franks a lot,
> David S. Miller
> davem@redhat.com

I've look at your script and it kinda flew over my head.

Would you mind explain this a bit?

Thanks

Re: [RFC] Standard way of generating assembler offsets

[David S. Miller]

   From: Pantelis Antoniou <panto@intracom.gr>
   Date: Tue, 09 Oct 2001 10:25:19 +0300

   I've look at your script and it kinda flew over my head.
   
   Would you mind explain this a bit?

We generate the offsets as data items in an assembler file, then we
parse out those data section entries and spit them into the header.

It allows cross compilation setups to work, ie. even in cases when you
cannot generate and run a binary.

Franks a lot,
David S. Miller
davem@redhat.com

Re: [RFC] Standard way of generating assembler offsets

[Keith Owens]

On Mon, 08 Oct 2001 02:49:46 -0700 (PDT), 
"David S. Miller" <davem@redhat.com> wrote:
>BTW, I assume you have already taken a look at how we
>do this on Sparc64.  See arch/sparc64/kernel/check_asm.sh
>and the "check_asm" target in arch/sparc64/kernel/Makefile
>
>It also works in all cross-compilation etc. environments.
>And I bet it would work on every platform with very minimal
>changes, if any.

I finally had time to look at sparc64 check_asm.  The code suffers from
several problems:

* It is only built at make dep time so any patches after make dep that
  affect the asm offsets are not picked up.

* The code has to know about all the config options that affect the
  structures, it manually tests for SMP and SPIN_LOCK_DEBUG.  That is
  bad enough but other architectures are much worse, several config
  settings can change the asm offsets, each config setting would have
  to be manually defined.

* The asm output lists all fields in the input structures.  It is
  arguable if this is a good or bad thing, I prefer to explicitly
  define just the symbols required by asm so I view this as a bad
  feature.

* check_asm can only generate offsetof and sizeof fields.  So
  calculated fields like AOFF_task_fpregs have to be done elsewhere.

* The code is complex.  I thought my asm-offsets.c->.s->.h conversion
  was complicated but asm_check takes the cake.

Here is a first cut at doing sparc64 asm offsets the same way that i386
and ia64 are done in kbuild 2.5.

* It uses .c->.s->h so it plugs directly into the kbuild 2.5 dependency
  handling, any patch or config changes that affect asm-offsets.h will
  automatically rebuild asm-offsets.h.

* Only the fields actually used by asm are listed.  I may have missed a
  field or two but gcc will tell us that.

* asm-offsets.c DEFINE() can do any calculation.  So AOFF_task_fpregs
  is now calculated here.

* Less complex :).

* The addition of comments against each define makes the result more
  readable.

==== arch/sparc64/asm-offsets.c ====

/*
 * Generate definitions needed by assembly language modules.
 * This code generates raw asm output which is post-processed to extract
 * and format the required data.
 */

#include <linux/types.h>
#include <linux/stddef.h>
#include <linux/sched.h>

/* Use marker if you need to separate the values later */

#define DEFINE(sym, val, marker) \
  asm volatile("\n-> " #sym " %0 " #val " " #marker : : "i" (val))

#define BLANK() asm volatile("\n->" : : )

int
main(void)
{
  DEFINE(AOFF_mm_context,		offsetof(struct mm_struct, context),);
  BLANK();

  DEFINE(AOFF_task_blocked,		offsetof(struct task_struct, blocked),);
  DEFINE(AOFF_task_egid,		offsetof(struct task_struct, egid),);
  DEFINE(ASIZ_task_egid,		sizeof(((struct task_struct *)NULL)->egid),);
  DEFINE(AOFF_task_euid,		offsetof(struct task_struct, euid),);
  DEFINE(ASIZ_task_euid,		sizeof(((struct task_struct *)NULL)->euid),);
  DEFINE(AOFF_task_flags,		offsetof(struct task_struct, flags),);
  DEFINE(AOFF_task_fpregs,		(sizeof(struct task_struct) + (64 - 1)) & ~(64 - 1),);
  DEFINE(AOFF_task_gid,			offsetof(struct task_struct, gid),);
  DEFINE(ASIZ_task_gid,			sizeof(((struct task_struct *)NULL)->gid),);
  DEFINE(AOFF_task_need_resched,	offsetof(struct task_struct, need_resched),);
  DEFINE(AOFF_task_personality,		offsetof(struct task_struct, personality),);
  DEFINE(ASIZ_task_personality,		sizeof(((struct task_struct *)NULL)->personality),);
  DEFINE(AOFF_task_pid,			offsetof(struct task_struct, pid),);
  DEFINE(AOFF_task_p_opptr,		offsetof(struct task_struct, p_opptr),);
  DEFINE(AOFF_task_processor,		offsetof(struct task_struct, processor),);
  DEFINE(AOFF_task_ptrace,		offsetof(struct task_struct, ptrace),);
  DEFINE(AOFF_task_sigpending,		offsetof(struct task_struct, sigpending),);
  DEFINE(AOFF_task_thread,		offsetof(struct task_struct, thread),);
  DEFINE(AOFF_task_uid,			offsetof(struct task_struct, uid),);
  DEFINE(ASIZ_task_uid,			sizeof(((struct task_struct *)NULL)->uid),);
  BLANK();

  DEFINE(AOFF_thread_current_ds,	offsetof(struct thread_struct, current_ds),);
  DEFINE(AOFF_thread_fault_address,	offsetof(struct thread_struct, fault_address),);
  DEFINE(AOFF_thread_fault_code,	offsetof(struct thread_struct, fault_code),);
  DEFINE(AOFF_thread_flags,		offsetof(struct thread_struct, flags),);
  DEFINE(AOFF_thread_fork_kpsr,		offsetof(struct thread_struct, fork_kpsr),);
  DEFINE(AOFF_thread_fpdepth,		offsetof(struct thread_struct, fpdepth),);
  DEFINE(AOFF_thread_fpsaved,		offsetof(struct thread_struct, fpsaved),);
  DEFINE(AOFF_thread_gsr,		offsetof(struct thread_struct, gsr),);
  DEFINE(AOFF_thread_kernel_cntd0,	offsetof(struct thread_struct, kernel_cntd0),);
  DEFINE(AOFF_thread_pcr_reg,		offsetof(struct thread_struct, pcr_reg),);
  DEFINE(AOFF_thread_reg_window,	offsetof(struct thread_struct, reg_window),);
  DEFINE(AOFF_thread_rwbuf_stkptrs,	offsetof(struct thread_struct, rwbuf_stkptrs),);
  DEFINE(AOFF_thread_use_blkcommit,	offsetof(struct thread_struct, use_blkcommit),);
  DEFINE(AOFF_thread_utraps,		offsetof(struct thread_struct, utraps),);
  DEFINE(AOFF_thread_uwinmask,		offsetof(struct thread_struct, uwinmask),);
  DEFINE(AOFF_thread_w_saved,		offsetof(struct thread_struct, w_saved),);
  DEFINE(AOFF_thread_xfsr,		offsetof(struct thread_struct, xfsr),);

  return 0;
}

==== arch/sparc64/Makefile.in ====

# Convert raw asm offsets into something that can be included as
# assembler definitions.  It converts
#   -> symbol $value source
# into
#   #define symbol value /* 0xvalue source */

user_command(asm-offsets.h
	($(objfile asm-offsets.s))
	(set -e;
	  (echo "#ifndef __ASM_OFFSETS_H__";
	   echo "#define __ASM_OFFSETS_H__";
	   echo "/*";
	   echo " * DO NOT MODIFY";
	   echo " *";
	   echo " * This file was generated by arch/$(ARCH)/Makefile.in.";
	   echo " *";
	   echo " */";
	   echo "";
	   awk "/^->\$$/{printf(\"\\n\");}
	     /^-> /{
	       sym = \$$2;
	       val = \$$3;
	       sub(/^\\\$$/, \"\", val);
	       \$$1 = \"\";
	       \$$2 = \"\";
	       \$$3 = \"\";
	       printf(\"#define %-24s %4d\\t\\t\\t/* 0x%x\\t%s */\\n\",
	         sym, val, val, \$$0)
	     }";
	   echo "";
	   echo "#endif";
	  ) < $< > $@;
	)
	()
	)

==== arch/sparc64/asm-offsets.h output ====

This is partial output and was actually generated on i386 so the
numbers are wrong but it shows what the output will look like.  I do
not have access to a machine for compiling 2.4 sparc64 kernels so none
of the thread values are in this sample.

#ifndef __ASM_OFFSETS_H__
#define __ASM_OFFSETS_H__
/*
 * DO NOT MODIFY
 *
 * This file was generated by arch/sparc64/Makefile.in.
 *
 */

#define AOFF_mm_context           128			/* 0x80	   offsetof(struct mm_struct, context) */

#define AOFF_task_blocked        1632			/* 0x660	   offsetof(struct task_struct, blocked) */
#define AOFF_task_egid            572			/* 0x23c	   offsetof(struct task_struct, egid) */
#define ASIZ_task_egid              4			/* 0x4	   sizeof(((struct task_struct *)NULL)->egid) */
#define AOFF_task_euid            556			/* 0x22c	   offsetof(struct task_struct, euid) */
#define ASIZ_task_euid              4			/* 0x4	   sizeof(((struct task_struct *)NULL)->euid) */
#define AOFF_task_flags             4			/* 0x4	   offsetof(struct task_struct, flags) */
#define AOFF_task_fpregs         1728			/* 0x6c0	   (sizeof(struct task_struct) + (64 - 1)) & ~(64 - 1) */
#define AOFF_task_gid             568			/* 0x238	   offsetof(struct task_struct, gid) */
#define ASIZ_task_gid               4			/* 0x4	   sizeof(((struct task_struct *)NULL)->gid) */
#define AOFF_task_need_resched     20			/* 0x14	   offsetof(struct task_struct, need_resched) */
#define AOFF_task_personality     116			/* 0x74	   offsetof(struct task_struct, personality) */
#define ASIZ_task_personality       4			/* 0x4	   sizeof(((struct task_struct *)NULL)->personality) */
#define AOFF_task_pid             124			/* 0x7c	   offsetof(struct task_struct, pid) */
#define AOFF_task_p_opptr         148			/* 0x94	   offsetof(struct task_struct, p_opptr) */
#define AOFF_task_processor        52			/* 0x34	   offsetof(struct task_struct, processor) */
#define AOFF_task_ptrace           24			/* 0x18	   offsetof(struct task_struct, ptrace) */
#define AOFF_task_sigpending        8			/* 0x8	   offsetof(struct task_struct, sigpending) */
#define AOFF_task_thread          880			/* 0x370	   offsetof(struct task_struct, thread) */
#define AOFF_task_uid             552			/* 0x228	   offsetof(struct task_struct, uid) */
#define ASIZ_task_uid               4			/* 0x4	   sizeof(((struct task_struct *)NULL)->uid) */

/* Thread values would be here if I could compile for sparc64. */

#endif