Dynamic libraries

Dynamic libraries, or shared libraries, are another way to produce libraries for reuse. As their name implies, unlike the static libraries, dynamic libraries are not part of the final executable itself. Instead, they should be loaded and brought in while loading a process for execution.

Since static libraries are part of the executable, the linker puts everything found in the given relocatable files into the final executable file. In other words, the linker detects the undefined symbols, and required definitions, and tries to find them in the given relocatable object files, then puts them all in the output executable file.

The final product is only produced when every undefined symbol is found. From a unique perspective, we detect all dependencies and resolve them at linking time. Regarding dynamic libraries, it is possible to have undefined symbols that are not resolved at linking time. These symbols are searched for when the executable product is about to be loaded and begin the execution.

In other words, a different kind of linking step is needed when you have undefined dynamic symbols. A dynamic linker, or simply the loader, usually does the linking while loading an executable file and preparing it to be run as a process.

Since the undefined dynamic symbols are not found in the executable file, they should be found somewhere else. These symbols should be loaded from shared object files. These files are sister files to static library files. While the static library files have a .a extension in their names, the shared object files carry the .so extension in most Unix-like systems. In macOS, they have the .dylib extension.

When loading a process and about to be launched, a shared object file will be loaded and mapped to a memory region accessible by the process. This procedure is done by a dynamic linker (or loader), which loads and executes an executable file.

Like we said in the section dedicated to executable object files, both ELF executable and shared object files have segments in their ELF structure. Each segment has zero or more sections in them. There are two main differences between an ELF executable object file and an ELF shared object file. Firstly, the symbols have relative absolute addresses that allow them to be loaded as part of many processes at the same time.

This means that while the address of each instruction is different in any process, the distance between two instructions remains fixed. In other words, the addresses are fixed relative to an offset. This is because the relocatable object files are position independent. We talk more about this in the last section of this chapter.

For instance, if two instructions are located at addresses 100 and 200 in a process, in another process they may be at 140 and 240, and in another one they could be at 323 and 423. The related addresses are absolute, but the actual addresses can change. These two instructions will always be 100 addresses apart from each other.

The second difference is that some segments related to loading an ELF executable object file are not present in shared object files. This effectively means that shared object files cannot be executed.

Before giving more details on how a shared object is accessed from different processes, we need to show an example of how they are created and used. Therefore, we are going to create dynamic libraries for the same geometry library, example 3.2, that we worked on in the previous section.

In the previous section we created a static library for the geometry library. In this section, we want to compile the sources again in order to create a shared object file out of them. The following commands show you how to compile the three sources into their corresponding relocatable object files, with just one difference in comparison to what we did for example 3.2. In the following commands, note the -fPIC option that is passed to gcc:

$ gcc -c ExtremeC_examples_chapter3_2_2d.c -fPIC -o 2d.o

$ gcc -c ExtremeC_examples_chapter3_2_3d.c -fPIC -o 3d.o

$ gcc -c ExtremeC_examples_chapter3_2_trigon.c -fPIC -o trigon.o

$

Shell Box 3-15: Compiling the sources of example 3.2 to corresponding position-independent relocatable object files

Looking at the commands, you can see that we have passed an extra option,-fPIC, to gcc while compiling the sources. This option is mandatory if you are going to create a shared object file out of some relocatable object files. PIC stands for position independent code. As we explained before, if a relocatable object file is position independent, it simply means that the instructions within it don't have fixed addresses. Instead, they have relative addresses; hence they can obtain different addresses in different processes. This is a requirement because of the way we use shared object files.

There is no guarantee that the loader program will load a shared object file at the same address in different processes. In fact, the loader creates memory mappings to the shared object files, and the address ranges for those mappings can be different. If the instruction addresses were absolute, we couldn't load the same shared object file in various processes, and in various memory regions, at the same time.

To create shared object files, you need to use the compiler, in this case, gcc, again. Unlike a static library file, which is a simple archive, a shared object file is an object file itself. Therefore, they should be created by the same linker program, for instance ld, that we used to produce the relocatable object files.

We know that, on most Unix-like systems, ld does that. However, it is strongly recommended not to use ld directly for linking object files for the reasons we explained in the previous chapter.

The following command shows how you should create a shared object file out of a number of relocatable object files that have been compiled using the -fPIC option:

$ gcc -shared 2d.o 3d.o trigon.o -o libgeometry.so

$ mkdir -p /opt/geometry

$ mv libgeometry.so /opt/geometry

$

Shell Box 3-16: Creating a shared object file out of the relocatable object files

As you can see in the first command, we passed the -shared option, to ask gcc to create a shared object file out of the relocatable object files. The result is a shared object file named libgeometry.so. We have moved the shared object file to /opt/geometry to make it easily available to other programs willing to use it. The next step is to compile and link example 3.3 again.

Previously, we compiled and linked example 3.3 with the created static library file, libgeometry.a. Here, we are going to do the same, but instead, link it with libgeometry.so, a dynamic library.

While everything seems to be the same, especially the commands, they are in fact different. This time, we are going to link example 3.3 with libgeometry.so instead of libgeometry.a, and more than that, the dynamic library won't get embedded into the final executable, instead it will load the library upon execution. While practicing this, make sure that you have removed the static library file, libgeometry.a, from /opt/geometry before linking example 3.3 again:

$ rm -fv /opt/geometry/libgeometry.a

$ gcc -c ExtremeC_examples_chapter3_3.c -o main.o

$ gcc main.o -L/opt/geometry-lgeometry -lm -o ex3_3.out

$

Shell Box 3-17: Linking example 3.3 against the built shared object file

As we explained before, the option -lgeometry tells the compiler to find and use a library, either static or shared, to link it with the rest of the object files. Since we have removed the static library file, the shared object file is picked up. If both the static library and shared object files exist for a defined library, then gcc prefers to pick the shared object file and link it with the program.

If you now try to run the executable file ex3_3.out, you will most probably face the following error:

$ ./ex3_3.out

./ex3_3.out: error while loading shared libraries: libgeometry.so: cannot open shared object file: No such file or directory

$

Shell Box 3-18: Trying to run example 3.3

We haven't seen this error so far, because we were using static linkage and a static library. But now, by introducing dynamic libraries, if we are going to run a program that has dynamic dependencies, we should provide the required dynamic libraries to have it run. But what has happened and why we've received the error message?

The ex3_3.out executable file depends on libgeometry.so. That's because some of the definitions it needs can only be found inside that shared object file. We should note that this is not true for the static library libgeometry.a. An executable file linked with a static library can be run on its own as a standalone executable, since it has copied everything from the static library file, and therefore, doesn't rely on its existence anymore.

This is not true for the shared object files. We received the error because the program loader (dynamic linker) could not find libgeometry.so in its default search paths. Therefore, we need to add /opt/geometry to its search paths, so that it finds the libgeometry.so file there. To do this, we will update the environment variable LD_LIBRARY_PATH to point to the current directory.

The loader will check the value of this environment variable, and it will search the specified paths for the required shared libraries. Note that more than one path can be specified in this environment variable (using the separator colon :).

$ export LD_LIBRARY_PATH=/opt/geometry

$ ./ex3_3.out

Polar Position: Length: 223.606798, Theta: 63.434949 (deg)

$

Shell Box 3-19: Running example 3.3 by specifying LD_LIBRARY_PATH

This time, the program has successfully been run! This means that the program loader has found the shared object file and the dynamic linker has loaded the required symbols from it successfully.

Note that, in the preceding shell box, we used the export command to change the LD_LIBRARY_PATH. However, it is common to set the environment variable together with the execution command. You can see this in the following shell box. The result would be the same for both usages:

$ LD_LIBRARY_PATH=/opt/geometry ./ex3_3.out

Polar Position: Length: 223.606798, Theta: 63.434949 (deg)

$

Shell Box 3-20: Running example 3.3 by specifying LD_LIBRARY_PATH as part of the same command

By linking an executable with several shared object files, as we did before, we tell the system that this executable file needs a number of shared libraries to be found and loaded at runtime. Therefore, before running the executable, the loader searches for those shared object files automatically, and the required symbols are mapped to the proper addresses that are accessible by the process. Only then can the processor begin the execution.

Manual loading of shared libraries

Shared object files can also be loaded and used in a different way, in which they are not loaded automatically by the loader program (dynamic linker). Instead, the programmer will use a set of functions to load a shared object file manually before using some symbols (functions) that can be found inside that shared library. There are applications for this manual loading mechanism, and we'll talk about them once we've discussed the example we'll look at in this section.

Example 3.4 demonstrates how to load a shared object file lazily, or manually, without having it in the linking step. This example borrows the same logic from example 3.3, but instead, it loads the shared object file libgeometry.so manually inside the program.

Before going through example 3.4, we need to produce libgeometry.so a bit differently in order to make example 3.4 work. To do this, we have to use the following command in Linux:

$ gcc -shared 2d.o 3d.o trigon.o -lm -o libgeometry.so

$

Shell Box 3-21: Linking the geometry shared object file against the standard math library

Looking at the preceding command, you can see a new option, -lm, which tells the linker to link the shared object file against the standard math library, libm.so. That is done because when we load libgeometry.so manually, its dependencies should, somehow, be loaded automatically. If they're not, then we will get errors about the symbols that are required by libgeometry.so itself, such as cos or sqrt. Note that we won't link the final executable file with the math standard library, and it will be resolved automatically by the loader when loading libgeometry.so.

Now that we have a linked shared object file, we can proceed to example 3.4:

#include <stdio.h>

#include <stdlib.h>

#include <dlfcn.h>

#include "ExtremeC_examples_chapter3_2_geometry.h"

polar_pos_2d_t (*func_ptr)(cartesian_pos_2d_t*);

int main(int argc, char** argv) {

void* handle = dlopen ("/opt/geometry/libgeometry.so", RTLD_LAZY);

if (!handle) {

fprintf(stderr, "%s\n", dlerror());

exit(1);

}

func_ptr = dlsym(handle, "convert_to_2d_polar_pos");

if (!func_ptr) {

fprintf(stderr, "%s\n", dlerror());

exit(1);

}

cartesian_pos_2d_t cartesian_pos;

cartesian_pos.x = 100;

cartesian_pos.y = 200;

polar_pos_2d_t polar_pos = func_ptr(&cartesian_pos);

printf("Polar Position: Length: %f, Theta: %f (deg)\n",

polar_pos.length, polar_pos.theta);

return 0;

}

Code Box 3-8 [ExtremeC_examples_chapter3_4.c]: Example 3.4 loading the geometry shared object file manually

Looking at the preceding code, you can see how we have used the functions dlopen and dlsym to load the shared object file and then find the symbol convert_to_2d_polar_pos in it. The function dlsym returns a function pointer, which can be used to invoke the target function.

It is worth noting that the preceding code searches for the shared object file in /opt/geometry, and if there is no such object file, then an error message is shown. Note that in macOS, the shared object files end in the .dylib extension. Therefore, the preceding code should be modified in order to load the file with the correct extension.

The following command compiles the preceding code and runs the executable file:

$ gcc ExtremeC_examples_chapter3_4.c -ldl -o ex3_4.out

$ ./ex3_4.out

Polar Position: Length: 223.606798, Theta: 63.434949 (deg)

$

Shell Box 3-22: Running example 3.4

As you can see, we did not link the program with the file libgeometry.so. We didn't do this because we want to instead load it manually when it is needed. This method is often referred to as the lazy loading of shared object files. Yet, despite the name, in certain scenarios, lazy loading the shared object files can be really useful.

One such case is when you have different shared object files for different implementations or versions of the same library. Lazy loading gives you increased freedom to load the desired shared objects according to your own logic and when it is needed, instead of having them automatically loaded at load time, where you have less control over them.