unable to link to a static library present alongside a shared library

unable to link to a static library present alongside a shared library

[Shriramana Sharma]

I read recently that someone was not able to link statically to a 
library which was placed alongside a shared library. Having gained some 
knowhow from the good people on this list I checked first the ld manpage 
and found this:

ld will search a directory for a library with an extension of .so before 
searching for one with an extension of .a.

Does anyone know of a way to subvert this behaviour and force linking to 
the static library?

Also, why is this behaviour made this way? Is there a reason or is it 
arbitrary?

Thanks as ever, for all your help.

Shriramana Sharma.

RE: unable to link to a static library present alongside a shared library

[cyon.john]

Hi,

You can use the '-static' option to gcc for forcing a static link even
if a shared library is present.
I think the default behaviour is set so as to take advantages of shared
library. For example, a dynamically linked executables has smaller size
than their corresponding statically linked files. While executing
multiple programs dynamically linked to a shared library, there will be
only one instance of that particular library in the memory for serving
all the programs that use it.

Hope that was helpful.

Regards,
Cyon P.J.

-----Original Message-----
From: linux-c-programming-owner@vger.kernel.org
[mailto:linux-c-programming-owner@vger.kernel.org] On Behalf Of
Shriramana Sharma
Sent: Wednesday, April 18, 2007 5:39 PM
To: linux-c-programming@vger.kernel.org
Subject: unable to link to a static library present alongside a shared
library

I read recently that someone was not able to link statically to a
library which was placed alongside a shared library. Having gained some
knowhow from the good people on this list I checked first the ld manpage
and found this:

ld will search a directory for a library with an extension of .so before
searching for one with an extension of .a.

Does anyone know of a way to subvert this behaviour and force linking to
the static library?

Also, why is this behaviour made this way? Is there a reason or is it
arbitrary?

Thanks as ever, for all your help.

Shriramana Sharma.
-
To unsubscribe from this list: send the line "unsubscribe
linux-c-programming" in the body of a message to
majordomo@vger.kernel.org More majordomo info at
http://vger.kernel.org/majordomo-info.html

Re: unable to link to a static library present alongside a shared library

[Shriramana Sharma]

cyon.john@wipro.com wrote:

> You can use the '-static' option to gcc for forcing a static link even
> if a shared library is present.

So much for my being intelligent enough to have looked up the ld manual. 
Thanks for your pointing out and I believe your reason for preferring 
the shared libs must be the correct one.

BTW what does this mean (found it under man:ld for -static):

This option can be used with -shared. Doing so means that a shared 
library is being created but that all of the library's external 
references must be resolved by pulling in entries from static libraries.

Does it mean that while creating a shared library, static libraries 
which the shared library itself depends on must be integrated into the 
.so file?

So if libfoo depends on libgoo and libgoo.a is available (but not 
libgoo.so) I will have to specify both -static and -shared to pull the 
contents of libgoo.a into libfoo.so when it is created?

That must further mean that the default behaviour is NOT to integrate a 
static library into a shared library. Is that right? Seems somewhat 
contrary to the standard policy of integrating into the target whatever 
static libraries are linked against, no?

Thanks.

Shriramana Sharma.

RE: unable to link to a static library present alongside a shared library

[cyon.john]

Hi,

> So if libfoo depends on libgoo and libgoo.a is available (but not
> libgoo.so) I will have to specify both -static and -shared to pull the
contents of libgoo.a into libfoo.so when it is > > created?

That "-shared -static" option becomes relevant when both libgoo.so and
libgoo.a are present. By default, libfoo.so will be dynamically linked
with libgoo.so and will be a dependency. But if used "-static" along
with "-shared" while creating libfoo.so, then it will be statically
linked with libgoo.a and won't have a dependency on libfoo.so.

FYI -

I was not able to see such a statement in the man pages in my system !!!

I tried 'gcc -shared -static -o libfoo.so foo.c -L . -lgoo', but the
output libfoo.so was dynamically linked with libgoo.so though libgoo.a
was present.

Then after some googling, I tried 'gcc -shared -Wl,-static -o libfoo.so
foo.c -L . -lgoo -nostdlib' and it gave me libfoo.so which is statically
linked with libgoo.a. 

Regards,
Cyon P.J. 

-----Original Message-----
From: linux-c-programming-owner@vger.kernel.org
[mailto:linux-c-programming-owner@vger.kernel.org] On Behalf Of
Shriramana Sharma
Sent: Wednesday, April 18, 2007 6:03 PM
To: linux-c-programming@vger.kernel.org
Subject: Re: unable to link to a static library present alongside a
shared library

cyon.john@wipro.com wrote:

> You can use the '-static' option to gcc for forcing a static link even

> if a shared library is present.

So much for my being intelligent enough to have looked up the ld manual.

Thanks for your pointing out and I believe your reason for preferring
the shared libs must be the correct one.

BTW what does this mean (found it under man:ld for -static):

This option can be used with -shared. Doing so means that a shared
library is being created but that all of the library's external
references must be resolved by pulling in entries from static libraries.

Does it mean that while creating a shared library, static libraries
which the shared library itself depends on must be integrated into the
.so file?

So if libfoo depends on libgoo and libgoo.a is available (but not
libgoo.so) I will have to specify both -static and -shared to pull the
contents of libgoo.a into libfoo.so when it is created?

That must further mean that the default behaviour is NOT to integrate a
static library into a shared library. Is that right? Seems somewhat
contrary to the standard policy of integrating into the target whatever
static libraries are linked against, no?

Thanks.

Shriramana Sharma.
-
To unsubscribe from this list: send the line "unsubscribe
linux-c-programming" in the body of a message to
majordomo@vger.kernel.org More majordomo info at
http://vger.kernel.org/majordomo-info.html

Re: unable to link to a static library present alongside a shared library

[Glynn Clements]

Shriramana Sharma wrote:

> I read recently that someone was not able to link statically to a 
> library which was placed alongside a shared library. Having gained some 
> knowhow from the good people on this list I checked first the ld manpage 
> and found this:
> 
> ld will search a directory for a library with an extension of .so before 
> searching for one with an extension of .a.
> 
> Does anyone know of a way to subvert this behaviour and force linking to 
> the static library?

You can force all libraries to be linked statically using -static. If
you try to use a library for which only a shared library is present,
the link will fail.

If you want to link against a mixture of static and shared libraries,
you can use e.g.

	gcc ... -Wl,-static -lfoo -Wl,-dy -lbar

This will link against libfoo.a (static) and libbar.so (shared).

> BTW what does this mean (found it under man:ld for -static):
> 
> This option can be used with -shared. Doing so means that a shared 
> library is being created but that all of the library's external 
> references must be resolved by pulling in entries from static libraries.
> 
> Does it mean that while creating a shared library, static libraries 
> which the shared library itself depends on must be integrated into the 
> .so file?

Yes.

> So if libfoo depends on libgoo and libgoo.a is available (but not 
> libgoo.so) I will have to specify both -static and -shared to pull the 
> contents of libgoo.a into libfoo.so when it is created?

Note that the compiler will do this even without -static. Using
-static simply means that shared libraries will not be considered when
linking.

When you create a shared library with "gcc -shared ...", and you
specify -l switches, any shared libraries will be listed as
dependencies and any static libraries will have the relevant code
embedded in the target.

> That must further mean that the default behaviour is NOT to integrate a 
> static library into a shared library. Is that right?

No. If you specify a -l switch which refers to a static library, the
relevant code will be embedded in the shared library.

> Seems somewhat 
> contrary to the standard policy of integrating into the target whatever 
> static libraries are linked against, no?

Code which is contained in a shared library should be
position-independent (i.e compiled with -fPIC) so that the library can
actually be shared.

Code which isn't position-independent needs to be relocated at
run-time, which requires modifying the in-memory copy of the library,
which means that the memory cannot be shared.

The code in static libraries often isn't position-independent. This
doesn't matter when using it in an executable, as the position is
fixed at compile time and run-time relocation isn't required.

But for a shared library, the end result will be a library which
cannot actually be shared: effectively just a DLL (dynamically-linked
library) rather than a *shared* library.

Also, on SELinux systems, you typically need to modify the security
policy to allow such libraries to be used, as the default security
policy will prevent the loader from performing the relocation (the
default security policy prohibits memory which has been mapped as
writable from being subsequently mapped as executable).

-- 
Glynn Clements <glynn@gclements.plus.com>

Re: unable to link to a static library present alongside a shared library

[Shriramana Sharma]

cyon.john@wipro.com wrote:

> That "-shared -static" option becomes relevant when both libgoo.so and
> libgoo.a are present. By default, libfoo.so will be dynamically linked
> with libgoo.so and will be a dependency. But if used "-static" along
> with "-shared" while creating libfoo.so, then it will be statically
> linked with libgoo.a and won't have a dependency on libfoo.so.

You are talking about the ordinary behaviour. -static always is used to 
mandate linking to a static lib and deny linking to a shared lib. 
Whether the target being built is an executable or a shared lib is 
immaterial.

> I was not able to see such a statement in the man pages in my system !!!

Dunno why. I'm using Kubuntu Edgy and my binutils version is 
2.17-1ubuntu1. The statement about using -shared and -static together is 
there in the documentation for the -static option (end of the paragraph) 
and is probably intended to clarify to the users that -static and 
-shared are not conflicting and only -static and -call_shared are 
conflicting. It could be rewritten.

> I tried 'gcc -shared -static -o libfoo.so foo.c -L . -lgoo', but the
> output libfoo.so was dynamically linked with libgoo.so though libgoo.a
> was present.
> 
> Then after some googling, I tried 'gcc -shared -Wl,-static -o libfoo.so
> foo.c -L . -lgoo -nostdlib' and it gave me libfoo.so which is statically
> linked with libgoo.a.

What's the difference? gcc will transparently pass on the -static option 
to ld if I read my man:gcc page right, and -nostdlib is to prevent 
linking to standard system libraries which libgoo is certainly not! How 
do the two commands give a different effect?

Shriramana Sharma.

Re: unable to link to a static library present alongside a shared library

[Shriramana Sharma]

Please take some time to read the following as I have been reading some 
Wikipedia articles about libraries and have a few questions. You don't 
have to answer the entire mail at once but please don't ignore it 
because it's a bit long. You are my teachers here since I am neither 
neither a professional programmer nor do I have a computer science 
degree so I am trusting in you...

Glynn Clements wrote:

> But for a shared library, the end result will be a library which
> cannot actually be shared: effectively just a DLL (dynamically-linked
> library) rather than a *shared* library.

I thought dll : Windows :: so : *nix. Apparently it is not so. After 
reading your mail I read through the 
http://en.wikipedia.org/wiki/Library_(computing) article and learnt some 
things but I'm still not fully clear:

1)

A Windows DLL is a dynamic link library. I hope I am right in 
understanding that to "link" means to basically resolve external 
references in an object module. 
http://en.wikipedia.org/wiki/Position_independent_code#Windows_DLLs 
tells me that Windows DLLs would largely be shareable only on disk and 
not on memory were it not for the fact that they are premapped to 
certain addresses in memory.

But Unix SO files are truly shareable on memory since they use PIC.

Is that all the functional difference between Windows DLL-s and Unix SO-s?

2)

Is there a difference between a dynamic link library and a dynamic load 
library? Would I be right in understanding that a dynamic link lib needs 
to be loaded at the same time as (or previous to) the loading of the 
caller, but a dynamic load lib can be loaded *after* the loading of the 
caller? This might mean that the latter would be valid for functions. Is 
my understanding correct?

3)

Is there a conflict between 
http://en.wikipedia.org/wiki/Library_(computing)#Dynamic_linking and 
http://en.wikipedia.org/wiki/Linker#Dynamic_linking . They have 
different explanations of dynamic linking...

4)

Anyway, the first article has the phrase "index names or numbers" but it 
does not say names or numbers of what. (If you tell me I will edit the 
article myself.)

5)

The article http://en.wikipedia.org/wiki/Static_Library further confuses 
me. It says that a static lib is a lib *in which* references to external 
variables/functions are resolved at compile time. I thought it was a lib 
which *satisfies* the references present *in a caller* and where the 
linking is done at compile time. Should the wording be corrected?

6)

The article also says "... or at runtime by the linker or linking 
loader". I think the word "respectively" should be added after "loader", 
but still I don't understand how refs to symbols provided by a static 
lib can be resolved at runtime. To me it goes against the very nature of 
  a static lib. What am I missing?

> Also, on SELinux systems, you typically need to modify the security
> policy to allow such libraries to be used, 

Such libraries meaning?

Thanks, as always.

Shriramana.

Re: unable to link to a static library present alongside a shared library

[Glynn Clements]

Shriramana Sharma wrote:

> > But for a shared library, the end result will be a library which
> > cannot actually be shared: effectively just a DLL (dynamically-linked
> > library) rather than a *shared* library.
> 
> I thought dll : Windows :: so : *nix. Apparently it is not so. After 
> reading your mail I read through the 
> http://en.wikipedia.org/wiki/Library_(computing) article and learnt some 
> things but I'm still not fully clear:
> 
> 1)
> 
> A Windows DLL is a dynamic link library. I hope I am right in 
> understanding that to "link" means to basically resolve external 
> references in an object module. 
> http://en.wikipedia.org/wiki/Position_independent_code#Windows_DLLs 
> tells me that Windows DLLs would largely be shareable only on disk and 
> not on memory were it not for the fact that they are premapped to 
> certain addresses in memory.
> 
> But Unix SO files are truly shareable on memory since they use PIC.
> 
> Is that all the functional difference between Windows DLL-s and Unix SO-s?

That's most of it, although the situation isn't quite as clear-cut
(early Linux implementations of shared libraries also didn't use PIC,
so had to be pre-mapped).

Another significant difference is that Windows executables and DLLs
associate any unresolved symbols with the DLL from which they are
meant to be loaded. OTOH, the Linux loader doesn't care where a symbol
comes from, so long as something defines it.

> 2)
> 
> Is there a difference between a dynamic link library and a dynamic load 
> library? Would I be right in understanding that a dynamic link lib needs 
> to be loaded at the same time as (or previous to) the loading of the 
> caller, but a dynamic load lib can be loaded *after* the loading of the 
> caller? This might mean that the latter would be valid for functions. Is 
> my understanding correct?

The two are often just used as alternative expansions of "DLL".

There is no techinical difference between a library which is
explicitly linked against an executable (or another library) and one
which is meant to be loaded dynamically with e.g. dlopen().

> 3)
> 
> Is there a conflict between 
> http://en.wikipedia.org/wiki/Library_(computing)#Dynamic_linking and 
> http://en.wikipedia.org/wiki/Linker#Dynamic_linking . They have 
> different explanations of dynamic linking...

There's no conflict; the first one is just slightly more detailed.

> 4)
> 
> Anyway, the first article has the phrase "index names or numbers" but it 
> does not say names or numbers of what. (If you tell me I will edit the 
> article myself.)

Windows (and maybe other systems) allows numeric references to symbols
in an external DLL. The executable or DLL can refer to e.g. symbol #7
in foo.dll rather than to the name of the function.

At one time, this probably provided a significant saving in terms of
file size.

> 5)
> 
> The article http://en.wikipedia.org/wiki/Static_Library further confuses 
> me. It says that a static lib is a lib *in which* references to external 
> variables/functions are resolved at compile time. I thought it was a lib 
> which *satisfies* the references present *in a caller* and where the 
> linking is done at compile time. Should the wording be corrected?

You're reading too much into the use of the word "in". When linking an
executable or shared library against a static library, external
references (those external to the executable or shared library) are
resolved at compile time.

> The article also says "... or at runtime by the linker or linking 
> loader". I think the word "respectively" should be added after "loader", 
> but still I don't understand how refs to symbols provided by a static 
> lib can be resolved at runtime. To me it goes against the very nature of 
>   a static lib. What am I missing?

It is possible to use a static library as a dynamically-loaded library
(the XFree86/X.org module loader does this), but it's debatable
whether it's reasonable to call this "static linking".

Ultimately, if a linker can merge various object files and libraries
together and write the result to a file, it can do the same in memory
and execute the result.

> > Also, on SELinux systems, you typically need to modify the security
> > policy to allow such libraries to be used, 
> 
> Such libraries meaning?

Those which contain relocations as a result of including code which
isn't position-independent.

Search Google for the phrase:

	"cannot restore segment prot after reloc: Permission denied"

This is not all that uncommon; it usually arises from using a library
which is only available as a static library. E.g. the BLAS/LAPACK
linear algebra libraries are written in Fortran; on some systems, you
can't generate position-independent code from Fortran, so any shared
library which uses BLAS/LAPACK ends up requiring run-time relocation.

-- 
Glynn Clements <glynn@gclements.plus.com>

Re: unable to link to a static library present alongside a shared library

[Shriramana Sharma]

Thanks for your continuing patience.

Glynn Clements wrote:

> Another significant difference is that Windows executables and DLLs
> associate any unresolved symbols with the DLL from which they are
> meant to be loaded. OTOH, the Linux loader doesn't care where a symbol
> comes from, so long as something defines it.

And what would be the implications of that? I mean, what happens 
practically because of that?

>> http://en.wikipedia.org/wiki/Library_(computing)#Dynamic_linking and 
> 
> Windows (and maybe other systems) allows numeric references to symbols
> in an external DLL. The executable or DLL can refer to e.g. symbol #7
> in foo.dll rather than to the name of the function.

I edited the page accordingly. I hope it is correct what I have written.

>> The article http://en.wikipedia.org/wiki/Static_Library further confuses 
> 
>> The article also says "... or at runtime by the linker or linking 
>> loader". I think the word "respectively" should be added after "loader", 
>> but still I don't understand how refs to symbols provided by a static 
>> lib can be resolved at runtime.

I edited this page also to be clearer. Again I hope it is correct.

I again sincerely thank you for your continuing patience and support,

Shriramana Sharma.

Re: unable to link to a static library present alongside a shared library

[leslie.polzer]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

On Thu, Apr 19, 2007 at 03:40:04PM +0530, Shriramana Sharma wrote:

> >meant to be loaded. OTOH, the Linux loader doesn't care where a
> >symbol comes from, so long as something defines it.
>
> And what would be the implications of that? I mean, what happens
> practically because of that?

You can deceive the program by using LD_PRELOAD tricks:
You may trace a certain group of functions,
chosen by you.
You may customize applications without having access to their
source or modify them in other ways as you see fit (as long as
it's dependent on a dynamically bound library function).

And so on.

  Leslie

-- 
NEW homepage: https://viridian.dnsalias.net/~sky/homepage/
gpg --keyserver pgp.mit.edu --recv-keys DD4EBF83

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: unable to link to a static library present alongside a shared library

[Shriramana Sharma]

leslie.polzer@gmx.net wrote:

> You may trace a certain group of functions, chosen by you.

I was able to get an idea of how the other two uses follow from this 
property of the Linux loader, but can you please elaborate a bit on this 
one?

Thanks.

Shriramana.

Re: unable to link to a static library present alongside a shared library

[leslie.polzer]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

On Thu, Apr 19, 2007 at 07:18:15PM +0530, Shriramana Sharma wrote:
> leslie.polzer@gmx.net wrote:
>
> >You may trace a certain group of functions, chosen by you.
>
> I was able to get an idea of how the other two uses follow from this
> property of the Linux loader, but can you please elaborate a bit on
> this one?

You can just wrap open(), for example, and provide its full
functionality, but also print out a message how it is called.

  Leslie

-- 
NEW homepage: https://viridian.dnsalias.net/~sky/homepage/
gpg --keyserver pgp.mit.edu --recv-keys DD4EBF83

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]