* Getting rid of (many) dynamic link creations in the xen build @ 2020-10-15 7:58 Jürgen Groß 2020-10-15 10:09 ` Jan Beulich 0 siblings, 1 reply; 10+ messages in thread From: Jürgen Groß @ 2020-10-15 7:58 UTC (permalink / raw) To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Wei Liu, Ian Jackson, George Dunlap After a short discussion on IRC yesterday I promised to send a mail how I think we could get rid of creating dynamic links especially for header files in the Xen build process. This will require some restructuring, the amount will depend on the selected way to proceed: - avoid links completely, requires more restructuring - avoid only dynamically created links, i.e. allowing some static links which are committed to git The difference between both variants is affecting the public headers in xen/include/public/: avoiding even static links would require to add another directory or to move those headers to another place in the tree (either use xen/include/public/xen/, or some other path */xen), leading to the need to change all #include statements in the hypervisor using <public/...> today. The need for the path to have "xen/" is due to the Xen library headers (which are installed on user's machines) are including the public hypervisor headers via "#include <xen/...>" and we can't change that scheme. A static link can avoid this problem via a different path, but without any link we can't do that. Apart from that decision, lets look which links are created today for accessing the header files (I'll assume my series putting the library headers to tools/include will be taken, so those links being created in staging today are not mentioned) and what can be done to avoid them: - xen/include/asm -> xen/include/asm-<arch>: Move all headers from xen/include/asm-<arch> to xen/arch/<arch>/include/asm and add that path via "-I" flag to CFLAGS. This has the other nice advantages that most architecture specific files are now in xen/arch (apart from the public headers) and that we can even add generic fallback headers in xen/include/asm in case an arch doesn't need a specific header file. - xen/arch/<arch>/efi/*.[ch] -> xen/common/efi/*.[ch]: Use vpath for the *.c files and the "-I" flag for adding common/efi to the include path in the xen/arch/<arch>/efi/Makefile. - tools/include/xen/asm -> xen/include/asm-<arch>: Add "-Ixen/arch/<arch>/include" to the CFLAGS. It might be a nice idea to move the headers needed by the tools to xen/arch/include/tools/asm and use "-Ixen/arch/<arch>/include/tools" instead, but this would require either the same path added to the hypervisor's CFLAGS or a modification of the related #include statements. - tools/include/xen/foreign -> tools/include/xen-foreign: Get rid of tools/include/xen-foreign and generate the headers directly in xen/include/public/foreign instead. - tools/include/xen/sys -> tools/include/xen-sys/<OS>: Move the headers from tools/include/xen-sys/<OS> to tools/include/<OS>/xen/sys/ and add the appropriate path to CFLAGS. - tools/include/xen/lib/<arch>/* -> xen/include/xen/lib/<arch>/*: Move xen/include/xen/lib/<arch> to xen/include/tools/lib/<arch> and add "-Ixen/include/tools" to the CFLAGS of tools. - tools/include/xen/libelf/* -> xen/include/xen/*: Move the affected headers from xen/include/xen to xen/include/tools/libelf and reuse the above set CFLAGS. - tools/include/xen/xsm -> xen/include/public/xsm: No longer needed (see next item in the list). - tools/include/xen/* -> xen/include/public/*: See above discussion of the two possible variants. Either add a static link in git from tools/include/xen -> xen/include/public (now possible with all links below tools/include/xen gone), or add "-Ixen/include/public" to the tools' CFLAGS. - stubdom/include/* -> tools/include/*: Set "-Itools/include -Itools/include/MiniOS" for the CFLAGS. I hope I have covered everything. Thoughts (especially about the two possible variants)? Juergen ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Getting rid of (many) dynamic link creations in the xen build 2020-10-15 7:58 Getting rid of (many) dynamic link creations in the xen build Jürgen Groß @ 2020-10-15 10:09 ` Jan Beulich 2020-10-15 10:41 ` Jürgen Groß 2020-10-15 10:49 ` Jürgen Groß 0 siblings, 2 replies; 10+ messages in thread From: Jan Beulich @ 2020-10-15 10:09 UTC (permalink / raw) To: Jürgen Groß Cc: xen-devel, Andrew Cooper, Wei Liu, Ian Jackson, George Dunlap On 15.10.2020 09:58, Jürgen Groß wrote: > After a short discussion on IRC yesterday I promised to send a mail > how I think we could get rid of creating dynamic links especially > for header files in the Xen build process. > > This will require some restructuring, the amount will depend on the > selected way to proceed: > > - avoid links completely, requires more restructuring > - avoid only dynamically created links, i.e. allowing some static > links which are committed to git While I like the latter better, I'd like to point out that not all file systems support symlinks, and hence the repo then couldn't be stored on (or the tarball expanded onto) such a file system. Note that this may be just for viewing purposes - I do this typically at home -, i.e. there's no resulting limitation from the build process needing symlinks. Similarly, once we fully support out of tree builds, there wouldn't be any restriction from this as long as just the build tree is placed on a capable file system. As a result I'd like to propose variant 2´: Reduce the number of dynamically created symlinks to a minimum. This said, I have to admit that I haven't really understood yet why symlinks are bad. They exist for exactly such purposes, I would think. > The difference between both variants is affecting the public headers > in xen/include/public/: avoiding even static links would require to > add another directory or to move those headers to another place in the > tree (either use xen/include/public/xen/, or some other path */xen), > leading to the need to change all #include statements in the hypervisor > using <public/...> today. > > The need for the path to have "xen/" is due to the Xen library headers > (which are installed on user's machines) are including the public > hypervisor headers via "#include <xen/...>" and we can't change that > scheme. A static link can avoid this problem via a different path, but > without any link we can't do that. > > Apart from that decision, lets look which links are created today for > accessing the header files (I'll assume my series putting the library > headers to tools/include will be taken, so those links being created > in staging today are not mentioned) and what can be done to avoid them: > > - xen/include/asm -> xen/include/asm-<arch>: > Move all headers from xen/include/asm-<arch> to > xen/arch/<arch>/include/asm and add that path via "-I" flag to CFLAGS. > This has the other nice advantages that most architecture specific > files are now in xen/arch (apart from the public headers) and that we > can even add generic fallback headers in xen/include/asm in case an > arch doesn't need a specific header file. Iirc Andrew suggested years ago that we follow Linux in this regard (and XTF already does). My only concern here is the churn this will cause for backports. > - xen/arch/<arch>/efi/*.[ch] -> xen/common/efi/*.[ch]: > Use vpath for the *.c files and the "-I" flag for adding common/efi to > the include path in the xen/arch/<arch>/efi/Makefile. Fine with me. > - tools/include/xen/asm -> xen/include/asm-<arch>: > Add "-Ixen/arch/<arch>/include" to the CFLAGS. It might be a nice idea > to move the headers needed by the tools to xen/arch/include/tools/asm > and use "-Ixen/arch/<arch>/include/tools" instead, but this would > require either the same path added to the hypervisor's CFLAGS or a > modification of the related #include statements. Separating headers intended for tools consumption is okay with me, but I dislike the tools/ infix in the path you suggest. Since there can't possibly be any shared prototypes, how about defs/ or some such not specifically naming either of the consuming components (and thus visually excluding the other)? Of course, the further asm/ underneath is kind of ugly because of being largely unnecessary. Perhaps we could have just xen/arch/include/defs/ and use #include <defs/xyz.h>? > - tools/include/xen/foreign -> tools/include/xen-foreign: > Get rid of tools/include/xen-foreign and generate the headers directly > in xen/include/public/foreign instead. Except that conceptually building in tools/ would better not alter the xen/ subtree in any way. > - tools/include/xen/sys -> tools/include/xen-sys/<OS>: > Move the headers from tools/include/xen-sys/<OS> to > tools/include/<OS>/xen/sys/ and add the appropriate path to CFLAGS. Not very nice imo because of the otherwise pointless intermediate directories, but if we truly need to minimize symlink usage, then so be it. > - tools/include/xen/lib/<arch>/* -> xen/include/xen/lib/<arch>/*: > Move xen/include/xen/lib/<arch> to xen/include/tools/lib/<arch> and > add "-Ixen/include/tools" to the CFLAGS of tools. Why not -Ixen/include/xen without any movement? Perhaps because -Ixen/include/tools wouldn't work either, due to code using #include <xen/lib/<arch>/xyz.h> ? I.e. you really mean Move xen/include/xen/lib/<arch> to xen/include/tools/xen/lib/<arch>? Not very nice. I have to admit I can't see why the header in xen/include/xen/lib/<arch>/ don't use #include "xyz.h" But then this would leave the problem with xen/lib/<arch>/*.c using similar #include-s. Would dropping xen/ from the paths perhaps help, moving xen/include/xen/lib/* to xen/include/lib/*? Istr suggesting this when the lib/ subtrees were introduced ... > - tools/include/xen/libelf/* -> xen/include/xen/*: > Move the affected headers from xen/include/xen to > xen/include/tools/libelf and reuse the above set CFLAGS. Why not xen/include/libelf/ or xen/include/lib/elf/? libelf-private.h has distinct #include-s for Xen and the tools anyway. All that's needed is that these headers don't sit in a directory where headers also live which are not supposed to be visible. Jan ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Getting rid of (many) dynamic link creations in the xen build 2020-10-15 10:09 ` Jan Beulich @ 2020-10-15 10:41 ` Jürgen Groß 2020-10-15 20:52 ` Andrew Cooper 2020-10-16 6:58 ` Jan Beulich 2020-10-15 10:49 ` Jürgen Groß 1 sibling, 2 replies; 10+ messages in thread From: Jürgen Groß @ 2020-10-15 10:41 UTC (permalink / raw) To: Jan Beulich; +Cc: xen-devel, Andrew Cooper, Wei Liu, Ian Jackson, George Dunlap On 15.10.20 12:09, Jan Beulich wrote: > On 15.10.2020 09:58, Jürgen Groß wrote: >> After a short discussion on IRC yesterday I promised to send a mail >> how I think we could get rid of creating dynamic links especially >> for header files in the Xen build process. >> >> This will require some restructuring, the amount will depend on the >> selected way to proceed: >> >> - avoid links completely, requires more restructuring >> - avoid only dynamically created links, i.e. allowing some static >> links which are committed to git > > While I like the latter better, I'd like to point out that not all > file systems support symlinks, and hence the repo then couldn't be > stored on (or the tarball expanded onto) such a file system. Note > that this may be just for viewing purposes - I do this typically at > home -, i.e. there's no resulting limitation from the build process > needing symlinks. Similarly, once we fully support out of tree > builds, there wouldn't be any restriction from this as long as just > the build tree is placed on a capable file system. > > As a result I'd like to propose variant 2´: Reduce the number of > dynamically created symlinks to a minimum. This said, I have to > admit that I haven't really understood yet why symlinks are bad. > They exist for exactly such purposes, I would think. Not the symlinks as such, but the dynamically created ones seem to be a problem, as we stumble upon those again and again. > >> The difference between both variants is affecting the public headers >> in xen/include/public/: avoiding even static links would require to >> add another directory or to move those headers to another place in the >> tree (either use xen/include/public/xen/, or some other path */xen), >> leading to the need to change all #include statements in the hypervisor >> using <public/...> today. >> >> The need for the path to have "xen/" is due to the Xen library headers >> (which are installed on user's machines) are including the public >> hypervisor headers via "#include <xen/...>" and we can't change that >> scheme. A static link can avoid this problem via a different path, but >> without any link we can't do that. >> >> Apart from that decision, lets look which links are created today for >> accessing the header files (I'll assume my series putting the library >> headers to tools/include will be taken, so those links being created >> in staging today are not mentioned) and what can be done to avoid them: >> >> - xen/include/asm -> xen/include/asm-<arch>: >> Move all headers from xen/include/asm-<arch> to >> xen/arch/<arch>/include/asm and add that path via "-I" flag to CFLAGS. >> This has the other nice advantages that most architecture specific >> files are now in xen/arch (apart from the public headers) and that we >> can even add generic fallback headers in xen/include/asm in case an >> arch doesn't need a specific header file. > > Iirc Andrew suggested years ago that we follow Linux in this regard > (and XTF already does). My only concern here is the churn this will > cause for backports. Changing a directory name in a patch isn't that hard, IMO. > >> - xen/arch/<arch>/efi/*.[ch] -> xen/common/efi/*.[ch]: >> Use vpath for the *.c files and the "-I" flag for adding common/efi to >> the include path in the xen/arch/<arch>/efi/Makefile. > > Fine with me. > >> - tools/include/xen/asm -> xen/include/asm-<arch>: >> Add "-Ixen/arch/<arch>/include" to the CFLAGS. It might be a nice idea >> to move the headers needed by the tools to xen/arch/include/tools/asm >> and use "-Ixen/arch/<arch>/include/tools" instead, but this would >> require either the same path added to the hypervisor's CFLAGS or a >> modification of the related #include statements. > > Separating headers intended for tools consumption is okay with me, > but I dislike the tools/ infix in the path you suggest. Since there > can't possibly be any shared prototypes, how about defs/ or some > such not specifically naming either of the consuming components > (and thus visually excluding the other)? I have absolutely no preferences for the naming. defs is fine IMO. > > Of course, the further asm/ underneath is kind of ugly because of > being largely unnecessary. Perhaps we could have just > xen/arch/include/defs/ and use #include <defs/xyz.h>? Yes, that should work, too. > >> - tools/include/xen/foreign -> tools/include/xen-foreign: >> Get rid of tools/include/xen-foreign and generate the headers directly >> in xen/include/public/foreign instead. > > Except that conceptually building in tools/ would better not alter > the xen/ subtree in any way. I meant to generate the headers from the hypervisor build instead. > >> - tools/include/xen/sys -> tools/include/xen-sys/<OS>: >> Move the headers from tools/include/xen-sys/<OS> to >> tools/include/<OS>/xen/sys/ and add the appropriate path to CFLAGS. > > Not very nice imo because of the otherwise pointless intermediate > directories, but if we truly need to minimize symlink usage, then > so be it. > >> - tools/include/xen/lib/<arch>/* -> xen/include/xen/lib/<arch>/*: >> Move xen/include/xen/lib/<arch> to xen/include/tools/lib/<arch> and >> add "-Ixen/include/tools" to the CFLAGS of tools. > > Why not -Ixen/include/xen without any movement? Perhaps because This would open up most of the hypervisor private headers to be easily includable by tools. > -Ixen/include/tools wouldn't work either, due to code using > > #include <xen/lib/<arch>/xyz.h> > > ? I.e. you really mean Move xen/include/xen/lib/<arch> to > xen/include/tools/xen/lib/<arch>? Not very nice. I have to admit > I can't see why the header in xen/include/xen/lib/<arch>/ don't > use > > #include "xyz.h" > > But then this would leave the problem with xen/lib/<arch>/*.c > using similar #include-s. Would dropping xen/ from the paths > perhaps help, moving xen/include/xen/lib/* to xen/include/lib/*? > Istr suggesting this when the lib/ subtrees were introduced ... This would at least eliminate one directory level. > >> - tools/include/xen/libelf/* -> xen/include/xen/*: >> Move the affected headers from xen/include/xen to >> xen/include/tools/libelf and reuse the above set CFLAGS. > > Why not xen/include/libelf/ or xen/include/lib/elf/? > libelf-private.h has distinct #include-s for Xen and the tools > anyway. All that's needed is that these headers don't sit in a > directory where headers also live which are not supposed to be > visible. That is correct. Juergen ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Getting rid of (many) dynamic link creations in the xen build 2020-10-15 10:41 ` Jürgen Groß @ 2020-10-15 20:52 ` Andrew Cooper 2020-10-16 6:52 ` Jan Beulich 2020-10-16 6:58 ` Jan Beulich 1 sibling, 1 reply; 10+ messages in thread From: Andrew Cooper @ 2020-10-15 20:52 UTC (permalink / raw) To: Jürgen Groß, Jan Beulich Cc: xen-devel, Wei Liu, Ian Jackson, George Dunlap On 15/10/2020 11:41, Jürgen Groß wrote: > On 15.10.20 12:09, Jan Beulich wrote: >> On 15.10.2020 09:58, Jürgen Groß wrote: >>> After a short discussion on IRC yesterday I promised to send a mail >>> how I think we could get rid of creating dynamic links especially >>> for header files in the Xen build process. >>> >>> This will require some restructuring, the amount will depend on the >>> selected way to proceed: >>> >>> - avoid links completely, requires more restructuring >>> - avoid only dynamically created links, i.e. allowing some static >>> links which are committed to git >> >> While I like the latter better, I'd like to point out that not all >> file systems support symlinks, and hence the repo then couldn't be >> stored on (or the tarball expanded onto) such a file system. Note >> that this may be just for viewing purposes - I do this typically at >> home -, i.e. there's no resulting limitation from the build process >> needing symlinks. Similarly, once we fully support out of tree >> builds, there wouldn't be any restriction from this as long as just >> the build tree is placed on a capable file system. >> >> As a result I'd like to propose variant 2´: Reduce the number of >> dynamically created symlinks to a minimum. This said, I have to >> admit that I haven't really understood yet why symlinks are bad. >> They exist for exactly such purposes, I would think. > > Not the symlinks as such, but the dynamically created ones seem to be > a problem, as we stumble upon those again and again. We have multiple build system bugs every release to do with dynamically generated symlinks. Given that symlinks aren't a hard requirement, this is a massive price to pay, and time which could be better spent doing other other things. Also, they prohibit the ability to build from a read-only source dir. The asm symlink in particular prevents any attempt to do concurrent builds of xen. In some future, I'd love to be able to do concurrent out-of-tree builds of Xen on different architectures, because elapsed time to do this is one limiting factor of mine on pre-push sanity checks. Personally, I'd prefer option 1 in the long run, but I've got no problems with achieving option 2 as an intermediate goal. > >> >>> The difference between both variants is affecting the public headers >>> in xen/include/public/: avoiding even static links would require to >>> add another directory or to move those headers to another place in the >>> tree (either use xen/include/public/xen/, or some other path */xen), >>> leading to the need to change all #include statements in the hypervisor >>> using <public/...> today. >>> >>> The need for the path to have "xen/" is due to the Xen library headers >>> (which are installed on user's machines) are including the public >>> hypervisor headers via "#include <xen/...>" and we can't change that >>> scheme. A static link can avoid this problem via a different path, but >>> without any link we can't do that. >>> >>> Apart from that decision, lets look which links are created today for >>> accessing the header files (I'll assume my series putting the library >>> headers to tools/include will be taken, so those links being created >>> in staging today are not mentioned) and what can be done to avoid them: >>> >>> - xen/include/asm -> xen/include/asm-<arch>: >>> Move all headers from xen/include/asm-<arch> to >>> xen/arch/<arch>/include/asm and add that path via "-I" flag to >>> CFLAGS. >>> This has the other nice advantages that most architecture specific >>> files are now in xen/arch (apart from the public headers) and >>> that we >>> can even add generic fallback headers in xen/include/asm in case an >>> arch doesn't need a specific header file. >> >> Iirc Andrew suggested years ago that we follow Linux in this regard >> (and XTF already does). My only concern here is the churn this will >> cause for backports. > > Changing a directory name in a patch isn't that hard, IMO. Also git (if you throw it the correct runes) can cope with this automatically. >>> - xen/arch/<arch>/efi/*.[ch] -> xen/common/efi/*.[ch]: >>> Use vpath for the *.c files and the "-I" flag for adding >>> common/efi to >>> the include path in the xen/arch/<arch>/efi/Makefile. >> >> Fine with me. Something which has been irritating me for years is that cscope doesn't tollerate the efi symlinks. This would be a great solution. ~Andrew ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Getting rid of (many) dynamic link creations in the xen build 2020-10-15 20:52 ` Andrew Cooper @ 2020-10-16 6:52 ` Jan Beulich 0 siblings, 0 replies; 10+ messages in thread From: Jan Beulich @ 2020-10-16 6:52 UTC (permalink / raw) To: Andrew Cooper Cc: Jürgen Groß, xen-devel, Wei Liu, Ian Jackson, George Dunlap On 15.10.2020 22:52, Andrew Cooper wrote: > On 15/10/2020 11:41, Jürgen Groß wrote: >> On 15.10.20 12:09, Jan Beulich wrote: >>> On 15.10.2020 09:58, Jürgen Groß wrote: >>>> After a short discussion on IRC yesterday I promised to send a mail >>>> how I think we could get rid of creating dynamic links especially >>>> for header files in the Xen build process. >>>> >>>> This will require some restructuring, the amount will depend on the >>>> selected way to proceed: >>>> >>>> - avoid links completely, requires more restructuring >>>> - avoid only dynamically created links, i.e. allowing some static >>>> links which are committed to git >>> >>> While I like the latter better, I'd like to point out that not all >>> file systems support symlinks, and hence the repo then couldn't be >>> stored on (or the tarball expanded onto) such a file system. Note >>> that this may be just for viewing purposes - I do this typically at >>> home -, i.e. there's no resulting limitation from the build process >>> needing symlinks. Similarly, once we fully support out of tree >>> builds, there wouldn't be any restriction from this as long as just >>> the build tree is placed on a capable file system. >>> >>> As a result I'd like to propose variant 2´: Reduce the number of >>> dynamically created symlinks to a minimum. This said, I have to >>> admit that I haven't really understood yet why symlinks are bad. >>> They exist for exactly such purposes, I would think. >> >> Not the symlinks as such, but the dynamically created ones seem to be >> a problem, as we stumble upon those again and again. > > We have multiple build system bugs every release to do with dynamically > generated symlinks. Given that symlinks aren't a hard requirement, this > is a massive price to pay, and time which could be better spent doing > other other things. > > Also, they prohibit the ability to build from a read-only source dir. In which way? In an out-of-tree build (see Linux) this gets created in the build tree, not the source one. Or else ... > The asm symlink in particular prevents any attempt to do concurrent > builds of xen. In some future, I'd love to be able to do concurrent > out-of-tree builds of Xen on different architectures, because elapsed > time to do this is one limiting factor of mine on pre-push sanity checks. ... this wouldn't already be possible there (including varying arch-es built from the same source tree). Jan ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Getting rid of (many) dynamic link creations in the xen build 2020-10-15 10:41 ` Jürgen Groß 2020-10-15 20:52 ` Andrew Cooper @ 2020-10-16 6:58 ` Jan Beulich 2020-10-16 7:25 ` Jürgen Groß 1 sibling, 1 reply; 10+ messages in thread From: Jan Beulich @ 2020-10-16 6:58 UTC (permalink / raw) To: Jürgen Groß Cc: xen-devel, Andrew Cooper, Wei Liu, Ian Jackson, George Dunlap On 15.10.2020 12:41, Jürgen Groß wrote: > On 15.10.20 12:09, Jan Beulich wrote: >> On 15.10.2020 09:58, Jürgen Groß wrote: >>> After a short discussion on IRC yesterday I promised to send a mail >>> how I think we could get rid of creating dynamic links especially >>> for header files in the Xen build process. >>> >>> This will require some restructuring, the amount will depend on the >>> selected way to proceed: >>> >>> - avoid links completely, requires more restructuring >>> - avoid only dynamically created links, i.e. allowing some static >>> links which are committed to git >> >> While I like the latter better, I'd like to point out that not all >> file systems support symlinks, and hence the repo then couldn't be >> stored on (or the tarball expanded onto) such a file system. Note >> that this may be just for viewing purposes - I do this typically at >> home -, i.e. there's no resulting limitation from the build process >> needing symlinks. Similarly, once we fully support out of tree >> builds, there wouldn't be any restriction from this as long as just >> the build tree is placed on a capable file system. >> >> As a result I'd like to propose variant 2´: Reduce the number of >> dynamically created symlinks to a minimum. This said, I have to >> admit that I haven't really understood yet why symlinks are bad. >> They exist for exactly such purposes, I would think. > > Not the symlinks as such, but the dynamically created ones seem to be > a problem, as we stumble upon those again and again. Well, the machinery to get them put in place needs to be fixed (and adjustments / additions be done more carefully). Taking together with what Andrew has said, option 2´ would move us in the same direction then. >>> The difference between both variants is affecting the public headers >>> in xen/include/public/: avoiding even static links would require to >>> add another directory or to move those headers to another place in the >>> tree (either use xen/include/public/xen/, or some other path */xen), >>> leading to the need to change all #include statements in the hypervisor >>> using <public/...> today. >>> >>> The need for the path to have "xen/" is due to the Xen library headers >>> (which are installed on user's machines) are including the public >>> hypervisor headers via "#include <xen/...>" and we can't change that >>> scheme. A static link can avoid this problem via a different path, but >>> without any link we can't do that. >>> >>> Apart from that decision, lets look which links are created today for >>> accessing the header files (I'll assume my series putting the library >>> headers to tools/include will be taken, so those links being created >>> in staging today are not mentioned) and what can be done to avoid them: >>> >>> - xen/include/asm -> xen/include/asm-<arch>: >>> Move all headers from xen/include/asm-<arch> to >>> xen/arch/<arch>/include/asm and add that path via "-I" flag to CFLAGS. >>> This has the other nice advantages that most architecture specific >>> files are now in xen/arch (apart from the public headers) and that we >>> can even add generic fallback headers in xen/include/asm in case an >>> arch doesn't need a specific header file. >> >> Iirc Andrew suggested years ago that we follow Linux in this regard >> (and XTF already does). My only concern here is the churn this will >> cause for backports. > > Changing a directory name in a patch isn't that hard, IMO. It's not hard at all, no, but it still takes some of the most precious resource we have: time. >>> - tools/include/xen/foreign -> tools/include/xen-foreign: >>> Get rid of tools/include/xen-foreign and generate the headers directly >>> in xen/include/public/foreign instead. >> >> Except that conceptually building in tools/ would better not alter >> the xen/ subtree in any way. > > I meant to generate the headers from the hypervisor build instead. This would make the tools/ build dependent upon xen/ having got built first aiui, which I think we want to avoid. >>> - tools/include/xen/lib/<arch>/* -> xen/include/xen/lib/<arch>/*: >>> Move xen/include/xen/lib/<arch> to xen/include/tools/lib/<arch> and >>> add "-Ixen/include/tools" to the CFLAGS of tools. >> >> Why not -Ixen/include/xen without any movement? Perhaps because > > This would open up most of the hypervisor private headers to be > easily includable by tools. Without the xen/ prefix, yes. But if someone wants to violate the naming scheme to get at them, adding a suitable number of ../ will also work as soon as symlinks aren't being used, or symlinks of full directories are used instead of ones referencing individual files. Jan ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Getting rid of (many) dynamic link creations in the xen build 2020-10-16 6:58 ` Jan Beulich @ 2020-10-16 7:25 ` Jürgen Groß 2020-10-16 8:11 ` Jan Beulich 0 siblings, 1 reply; 10+ messages in thread From: Jürgen Groß @ 2020-10-16 7:25 UTC (permalink / raw) To: Jan Beulich; +Cc: xen-devel, Andrew Cooper, Wei Liu, Ian Jackson, George Dunlap On 16.10.20 08:58, Jan Beulich wrote: > On 15.10.2020 12:41, Jürgen Groß wrote: >> On 15.10.20 12:09, Jan Beulich wrote: >>> On 15.10.2020 09:58, Jürgen Groß wrote: >>>> - tools/include/xen/foreign -> tools/include/xen-foreign: >>>> Get rid of tools/include/xen-foreign and generate the headers directly >>>> in xen/include/public/foreign instead. >>> >>> Except that conceptually building in tools/ would better not alter >>> the xen/ subtree in any way. >> >> I meant to generate the headers from the hypervisor build instead. > > This would make the tools/ build dependent upon xen/ having got > built first aiui, which I think we want to avoid. Today we have a mechanism to build tools/include (i.e. setup the links) from the main Makefile. The same rule could be used to create the needed headers in xen/include/public/foreign. > >>>> - tools/include/xen/lib/<arch>/* -> xen/include/xen/lib/<arch>/*: >>>> Move xen/include/xen/lib/<arch> to xen/include/tools/lib/<arch> and >>>> add "-Ixen/include/tools" to the CFLAGS of tools. >>> >>> Why not -Ixen/include/xen without any movement? Perhaps because >> >> This would open up most of the hypervisor private headers to be >> easily includable by tools. > > Without the xen/ prefix, yes. But if someone wants to violate the > naming scheme to get at them, adding a suitable number of ../ will > also work as soon as symlinks aren't being used, or symlinks of > full directories are used instead of ones referencing individual > files. We'd need to be very careful regarding name collisions in this case (there is e.g. xen/list.h and we have at least one list.h in a local directory). I'm not sure we want to take that additional risk. Juergen ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Getting rid of (many) dynamic link creations in the xen build 2020-10-16 7:25 ` Jürgen Groß @ 2020-10-16 8:11 ` Jan Beulich 0 siblings, 0 replies; 10+ messages in thread From: Jan Beulich @ 2020-10-16 8:11 UTC (permalink / raw) To: Jürgen Groß Cc: xen-devel, Andrew Cooper, Wei Liu, Ian Jackson, George Dunlap On 16.10.2020 09:25, Jürgen Groß wrote: > On 16.10.20 08:58, Jan Beulich wrote: >> On 15.10.2020 12:41, Jürgen Groß wrote: >>> On 15.10.20 12:09, Jan Beulich wrote: >>>> On 15.10.2020 09:58, Jürgen Groß wrote: >>>>> - tools/include/xen/foreign -> tools/include/xen-foreign: >>>>> Get rid of tools/include/xen-foreign and generate the headers directly >>>>> in xen/include/public/foreign instead. >>>> >>>> Except that conceptually building in tools/ would better not alter >>>> the xen/ subtree in any way. >>> >>> I meant to generate the headers from the hypervisor build instead. >> >> This would make the tools/ build dependent upon xen/ having got >> built first aiui, which I think we want to avoid. > > Today we have a mechanism to build tools/include (i.e. setup the links) > from the main Makefile. The same rule could be used to create the needed > headers in xen/include/public/foreign. Oh, indeed. >>>>> - tools/include/xen/lib/<arch>/* -> xen/include/xen/lib/<arch>/*: >>>>> Move xen/include/xen/lib/<arch> to xen/include/tools/lib/<arch> and >>>>> add "-Ixen/include/tools" to the CFLAGS of tools. >>>> >>>> Why not -Ixen/include/xen without any movement? Perhaps because >>> >>> This would open up most of the hypervisor private headers to be >>> easily includable by tools. >> >> Without the xen/ prefix, yes. But if someone wants to violate the >> naming scheme to get at them, adding a suitable number of ../ will >> also work as soon as symlinks aren't being used, or symlinks of >> full directories are used instead of ones referencing individual >> files. > > We'd need to be very careful regarding name collisions in this case > (there is e.g. xen/list.h and we have at least one list.h in a local > directory). I'm not sure we want to take that additional risk. Well, header in local dirs aren't a big problem - they get included via #include "xyz.h" anyway. But I see your point, and this imo is an argument to stick to symlinks, as this avoids unnecessary dir levels and allows to be as selective as we want/need to be. Jan ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Getting rid of (many) dynamic link creations in the xen build 2020-10-15 10:09 ` Jan Beulich 2020-10-15 10:41 ` Jürgen Groß @ 2020-10-15 10:49 ` Jürgen Groß 2020-10-16 6:59 ` Jan Beulich 1 sibling, 1 reply; 10+ messages in thread From: Jürgen Groß @ 2020-10-15 10:49 UTC (permalink / raw) To: Jan Beulich; +Cc: xen-devel, Andrew Cooper, Wei Liu, Ian Jackson, George Dunlap On 15.10.20 12:09, Jan Beulich wrote: > On 15.10.2020 09:58, Jürgen Groß wrote: >> After a short discussion on IRC yesterday I promised to send a mail >> how I think we could get rid of creating dynamic links especially >> for header files in the Xen build process. >> >> This will require some restructuring, the amount will depend on the >> selected way to proceed: >> >> - avoid links completely, requires more restructuring >> - avoid only dynamically created links, i.e. allowing some static >> links which are committed to git > > While I like the latter better, I'd like to point out that not all > file systems support symlinks, and hence the repo then couldn't be > stored on (or the tarball expanded onto) such a file system. Note > that this may be just for viewing purposes - I do this typically at > home -, i.e. there's no resulting limitation from the build process > needing symlinks. Similarly, once we fully support out of tree > builds, there wouldn't be any restriction from this as long as just > the build tree is placed on a capable file system. > > As a result I'd like to propose variant 2´: Reduce the number of > dynamically created symlinks to a minimum. This said, I have to > admit that I haven't really understood yet why symlinks are bad. > They exist for exactly such purposes, I would think. Another option would be to create the needed links from ./configure instead of committing them to git. Juergen ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Getting rid of (many) dynamic link creations in the xen build 2020-10-15 10:49 ` Jürgen Groß @ 2020-10-16 6:59 ` Jan Beulich 0 siblings, 0 replies; 10+ messages in thread From: Jan Beulich @ 2020-10-16 6:59 UTC (permalink / raw) To: Jürgen Groß Cc: xen-devel, Andrew Cooper, Wei Liu, Ian Jackson, George Dunlap On 15.10.2020 12:49, Jürgen Groß wrote: > On 15.10.20 12:09, Jan Beulich wrote: >> On 15.10.2020 09:58, Jürgen Groß wrote: >>> After a short discussion on IRC yesterday I promised to send a mail >>> how I think we could get rid of creating dynamic links especially >>> for header files in the Xen build process. >>> >>> This will require some restructuring, the amount will depend on the >>> selected way to proceed: >>> >>> - avoid links completely, requires more restructuring >>> - avoid only dynamically created links, i.e. allowing some static >>> links which are committed to git >> >> While I like the latter better, I'd like to point out that not all >> file systems support symlinks, and hence the repo then couldn't be >> stored on (or the tarball expanded onto) such a file system. Note >> that this may be just for viewing purposes - I do this typically at >> home -, i.e. there's no resulting limitation from the build process >> needing symlinks. Similarly, once we fully support out of tree >> builds, there wouldn't be any restriction from this as long as just >> the build tree is placed on a capable file system. >> >> As a result I'd like to propose variant 2´: Reduce the number of >> dynamically created symlinks to a minimum. This said, I have to >> admit that I haven't really understood yet why symlinks are bad. >> They exist for exactly such purposes, I would think. > > Another option would be to create the needed links from ./configure > instead of committing them to git. Ah yes, this would indeed seem better to me. Not sure though whether that's conceptually a legitimate thing to do. Jan ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2020-10-16 8:12 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-10-15 7:58 Getting rid of (many) dynamic link creations in the xen build Jürgen Groß 2020-10-15 10:09 ` Jan Beulich 2020-10-15 10:41 ` Jürgen Groß 2020-10-15 20:52 ` Andrew Cooper 2020-10-16 6:52 ` Jan Beulich 2020-10-16 6:58 ` Jan Beulich 2020-10-16 7:25 ` Jürgen Groß 2020-10-16 8:11 ` Jan Beulich 2020-10-15 10:49 ` Jürgen Groß 2020-10-16 6:59 ` Jan Beulich
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).