This is part of a series where I write down things I know or I've learned while trying to keep myself sharp in various subjects that I don't use so often anymore.
This post is quite sparsely structured however due to the lack of posts about the subject of packaging and specifically about how to handle weird conda-forge things, I thought it might be useful for someone anyway!
A given package will only work for a given architecture. Packages compiled to macOS Intel x86 won’t work in the M1 architecture’s ARM64, for example.
Dynamically linked libraries contain functions, classes, variables, and resources that can be used by multiple programs simultaneously. This promotes code reuse and efficient memory usage. They enable 'dynamic linking' where the linking happens at run time when both executable files and libraries are loaded into memory.
nm are used to work with binary files.
otool is the OSX tool
ldd command prints the shared libraries required by each program or shared library passed to it.
nm is used to inspect the symbols within object files or binaries and is available as part of the GNU binutils package.
nm to find out whether a symbol is defined or used in an object file, and whether the symbol is exported (global) or static.
nm is useful for debugging linking problems by showing which symbols are undefined or multiple defined in an object file.
otool -L to achieve similar functionality to
ldd on macOS. This command lists the dynamic library dependencies of a binary.
In the context of compiled languages like C and C++, symbols refer to these entities (function, classes, variables..) once the code has been compiled into object files (.o), static libraries (.a), shared libraries (.so, .dylib, etc.), or executables.
Object Files (.o/.obj)
Static Libraries (.a/.lib)
ar in Unix) from one or more object files.
Shared Libraries (.so/.dylib/.dll)
Questions and answers:
How does the operating system use shared libraries at runtime?
What must be ensured for a shared library to be used by an executable at runtime?
LD_LIBRARY_PATH on Linux).
Explain the implication of linking an application with a static library for the final executable. How is this achieved?
How are object files used in creating an executable?
What are the benefits of using dynamically linked libraries over statically linked libraries?
How to create a pip package with a fair load of c++ code on it?
ext_modules = [
'my_module.my_cpp_extension', # Name of the module
['my_module/my_cpp_extension.cpp'], # C++ source files
get_pybind_include(), # Path to pybind11 headers
'/usr/include/eigen3', # Path to other headers needed
description='A minimal package with a C++ extension',
install_requires=['pybind11>=2.5.0'], # Depend on pybind11
pip install setuptools wheel and then compile it
python setup.py build_ext --inplace. This will generate shared objects. The kinds of objects that will be generated are platform specific.
python [setup.py](http://setup.py/) sdist bdist_wheel to create a wheel and twine to distribute it
What is a wheel? And what’s an egg?
Describing each option from a
Host: The "host" environment contains the dependencies that your package will link against during the build process. When you’re cross-compiling your package you want to make sure that the target system has all the libraries necessary to build your software in their OS. The dependencies that need to be present for the package to compile or run during the build process. These are typically included when the package is installed by an end-user, as they are required for the package to function correctly.
This is where the build scripts will look for dependencies to link against or execute during the build process. If you're not cross-compiling, the "host" and "build" platforms are the same.
Use selectors in
You can use selectors like
# [win] or
# [linux] to include dependencies or run scripts conditionally depending on the platform. Here's a simple example:
- toolchain # [linux]
- m2-make # [win]
Can I have pip packages, or conda packages from other channels on my recipe?
As a general rule: all dependencies have to be packaged by conda-forge as well. This is necessary to assure ABI compatibility for all our packages.
Some dependencies are so close to the system that they are not packaged with conda-forge. These dependencies have to be satisfied with Core Dependency Tree (CDT) packages.
A CDT package consists of repackaged CentOS binaries from the appropriate version, either 6 or 7 depending on user choice and platform. We manage the build of CDT packages using a centralized repo, conda-forge/cdt-builds, as opposed to generating feedstocks for them. (Note that historically we did use feedstocks but this practice has been deprecated.) To add a new CDT, make a PR on the conda-forge/cdt-builds repo.
What the hell does this cross-compilation stuff is about anyway?
Cross-compilation is a technique where you compile a program on one type of system (the host) to run on another type (the target). Cross-compilation is more commonly used when targeting a platform that is not easily accessible or when the build process is much faster or more convenient in this platform. For example:
Embedded systems often use processors with different architectures than those in the developers' own computers. For instance, a developer might be using an x86_64 architecture computer but needs to build an application for an ARM architecture-based embedded system, like a Raspberry Pi or a smartphone.
Here's how it works in this scenario:
The developer will use a cross-compiler on their x86_64 machine that generates executable code for the ARM architecture. This allows the developer to take advantage of the more powerful development machine, speeding up the compile-test-debug cycle. Once the application is ready, the executable can be transferred to the ARM-based device for execution.
Cross-compilation is often faster than native compilation in such cases because the development machine has more resources (CPU, memory, etc.) than the target device. It also avoids the need to set up a full development environment on the target device, which might be constrained in terms of resources or might not be designed for such tasks.