This is the continuation of the initial post about new Docker images for the Conan Center that will help you understand the problem, the motivation and the proposed solution for Conan Docker Tools. In this post we will continue the proposed implementation, highlight the problems encountered during development,and show how this solution improves many aspects of the way we use Docker images for Conan. Also, this blog is a bit longer than usual and contains explanations about the decisions made, mistakes committed, and details the technical part of working with Dockerfiles. We divided this blog post in 2 parts:

The Importance of the Community

We want to thank the community for all of the great and important feedback we received involving Conan and Docker images. Your feedback lets us know how important these Docker images are not only for Conan Center to generate official packages for general distribution, but also for people who own their personal projects and use the images to validate and build their Conan recipes, as well as the companies that build packages through sources.

After long feedback and new formulations we understood that it was necessary to make changes to the initial concept. We realized that we could not modify the existing images for better maintenance given the risk of breaking the behavior between users, so we started from scratch again, formulating a more balanced version between Conan Center and users. We can’t keep an old version long enough to support older compilers, this is the price of progress. To keep moving forward we need to remove older versions for better maintenance for newer versions.

We are grateful to the great tribe that surrounds Conan and helps make it more and more complete. We would not be where we are without your support, feedback, and trust in our ability to continue to improve Conan.

Part 1: The Long Journey: Ideas, Motivation, Mistakes and Challenges

In this section we will describe how we revisited the initial proposal, found problems, proposed solutions and embarked on an entirely new idea. If you are more interested in the technical part of our journey please skip ahead to Part 2.

Conan Docker Images: Revisited

The pull request #204 showed us some weak points to consider:

  • Installing Clang via LLVM’s APT repository does not guarantee full compatibility with libstdc++ version used to build GCC.
  • APT packages that were requirements for GCC and Clang were still present, further inflating the final image.
  • The packages provided through Ubuntu do not have older versions available, in case a new version comes along. This affects the reproducibility requirement.
  • Older compilers were always an issue in terms of maintenance once they arrived in the EOL (end-of-life) state. It was necessary to update the PPA address and build the images again.
  • The continuous integration service used, although it was considerably fast, it was not possible to customize and prioritize the build, if necessary.

Noticing the listed issues, we tried a radical solution. This solution took more time to implement but resulted in something better in terms of maintainability and practicality.

This being the case we decided to abandon PR #204 and start again from scratch, considering the items listed above.

A New Plan: Using the Same Base Image

We first need to understand the objective behind the project before we start implementing new dockerfiles with their proper corrections. We want to understand what the expectation is for these new images 2 years from now. With the current approach we are already maintaining over 40 different dockerfiles which can be problematic. We also want to address one of the points discussed previously: there will be rotation to avoid the accumulation of old images and their restrictions in terms of maintenance.

The Plan

For current and already available images on hub.docker these will be kept but no longer supported (new versions will not be introduced unless it’s needed for ConanCenter). Your recipes and images will remain available, as their immediate removal would result in catastrophic failure for many users. They will eventually be removed, but at a distant and yet to be defined date.

As for the new images, these will be adopted as official and widely promoted for use by all users. The transition to the new images in Conan Center should take place in a few months after release because it will be necessary to rebuild the packages that already exist in the Conan Center and replace them to ensure full compatibility between packages. As for rotation and maintenance, we believe it is necessary to rotate supported compilers over time, to avoid a large build, effort and maintenance load for old images and packages that are not always used by the community. Therefore, the following rule will be adopted:

  • Clang will be supported from 10.0 to the newest version.
  • GCC, on the other hand, is widely used for the Linux environment and we will generate images from version 5 to the latest.
  • For both compilers we will keep updating all new compiler versions and Conan client version, according to new releases.
  • The multilib support was discarded as we are only interested in producing packages with 64-bit support.
  • Fortran support has been added, thus producing gfortran together in the image. Currently the Conan package for gfortran is totally broken and has a complex dependencies chain to fix.

We always wanted to be independent of the Linux distribution so one of the problems we would like to solve is the compiler used and its libraries. In the initial pull request, we built the GCC from sources, while the Clang uses prebuilt. To solve this problem we chose to build both from sources in order to have more control over the compiler used. Packages generated using these images (packages in ConanCenter) should work out-of-the-box in as many as possible different distros.

The library libstdc++ is distributed along with the GCC project. The intention was to use a single recent version of the library. This decision would allow older distributions to continue to use the library while also allowing new features to be consumed by newer compilers. The version chosen was libstdc++.so.6.0.28, the same distributed with GCC 9 and 10, but also is the default version in Ubuntu 20.04 LTS (Focal). Once GCC 10 was built we presumed it would be possible to copy this library to the rest of the images. That was the original intent but as we soon learned it was not possible. While we were developing the new recipes GCC 11 was released and with it a new libstdc++ version (6.0.29). It was not possible to use the previous version with this new compiler. We were left with the following dilemma:

  • Using the same libstdc++ version, except for GCC 11.
    • Conan Center becomes homogeneous (except GCC 11): all binaries will be built and linked using the same libstdc++ version, which guarantees that all can run in any image.
    • Binaries can hardly be used outside of Conan Center because they need the newest version of libstdc++ library that is not yet available in the official PPA. All executables built inside ConanCenter won’t work in the users’ machines.
    • A possible solution would be to statically link libstdc++ in all binaries, but this solution has a number of risks.
  • Each image uses the corresponding version of libstdc++ provided by the compiler:
    • Better than the current scenario, where it is dependent on PPA and we have no control over it.
    • All images still use the same version of glibc, another advantage over the current scenario.
    • We will need to take care of the executables, as they will only be compatible with later versions (as they are now).
    • Better for users, than the current scenario. The requirements related to libstdc++ are the same, but the glibc version is the same version for all the packages.

Given the conditions and risks, we chose to go the second way: Use the libstdc++ version available together with the compiler.

As Clang also supports libstdc++ we choose libstdc++.so.6.0.28 to be its default version. As commented before, that version has some advantages due to its age and compatibility. It is also the default for the latest LTS Ubuntu version, so we really think it should satisfy most of the use-cases.

Ubuntu 16.04 Xenial LTS is still the base used and it will be supported until April 2024. After that date we will need to update the images to a newer version of the distribution in addition to rebuilding all available official packages.

To summarize: forward thinking is having fewer images, but better support without the drastic breakage and incompatibility issues.

The Revised Plan:

  • Ubuntu 16.04 LTS as base Docker image
  • Build Clang and GCC from source
  • Use libstdc++ provided by the compiler
  • Use glibc 2.23 for all new Docker images
  • Images for old compilers will be built as long as their build script is compatible with the one for the newer compilers.

Training the Dragon: Building Clang from Sources

We want to use only one version of libstdc++ so we found a way to build the Clang without the direct dependency on GCC. By building the Clang with another Clang already installed it avoided libgcc_s, libstdc++ and used libc++, libc++-abi, libunwind, compiler-rt and ldd instead. The libstdc++ will only be used for Conan packages - not as a Clang requirement.

Challenges & Action Items:

  • The LLVM project uses CMake support, which facilitates the configuration of its construction, even customization if necessary.
  • We chose to use Clang 10 as a builder, as it is current and still compatible with the chosen Ubuntu version. The compiler is pre-built and distributed by the official LLVM PPA.
  • From version to version, options are added or removed, reflecting the evolution of project features and legacy deprecation. With these changes, it was inevitable to study the CMake files of each version to understand which options do not work in subsequent versions or which option should be used to specify the preferred library.
  • Unlike GCC, LLVM has a huge range of parameters and a longer build time, around 1h depending on the host. So, for each attempt, a long wait was needed to get the result.
  • Until Clang 9 release, libc++ was not automatically added to be linked when using Clang. As a solution, the project supports a configuration file, where libc++ can be specified by default. However, this behavior changes between versions 6, 7 and 8, requiring different standards and making it difficult to use the same Docker recipe for all versions.
  • With the removal of the GCC dependency, it was necessary to use libunwind during the build. It is already internalized in LLVM, but used as a dynamic library only. So a question arises, what happens if a project uses the image with Clang and installs Conan’s libunwind package? A big mess when linking is the answer. Clang tries to link the version distributed by the Conan package, resulting in several errors. As a workaround, we renamed the original LLVM libunwind to libllvm-unwind.

It became quite difficult to maintain from Clang 6 to 12 with the discovery of these issues and limitations. After a lot of discussions and advice from some of the LLVM maintainers we decided to limit Clang support to starting from version 10 because it is not necessary to apply as many modifications including the configuration file. Also, in the Linux environment Clang is not the primary compiler so we believe its use is always tied to newer versions.

Part 2: Under the Hood of Dockerfiles and Technical Details

Here we will be more focused on the final product, Dockerfiles, tests and CI.

From Blueprint to Prototype: Writing the New Docker Recipes

During prototyping we realized that we could divide the process of building a Docker image into 3 phases:

  • Phase 1:The base where all common packages are installed to all images such as Python, git, svn, etc, in addition to the non-root user configuration.
  • Phase 2: An image where only the compiler is built. In this container can be installed packages referring to the compiler build only, which will not be present in the final image, for example, Ninja, which is used for LLVM.
  • Phase 3: Merge the base to the produced compiler into a single image without adding extra packages but still reusable between each compiler version.

For the case of the base image, this one is still quite modular, just changing the variables file to update the package to be installed. The complete recipe can be obtained here, but let’s look at a few pieces:

ARG DISTRO_VERSION FROM ubuntu:${DISTRO_VERSION}

ENV PYENV_ROOT=/opt/pyenv \
    PATH=/opt/pyenv/shims:${PATH}

ARG CMAKE_VERSION ARG CMAKE_VERSION_FULL ARG PYTHON_VERSION ARG CONAN_VERSION

Now, both the distribution version and the installed packages are configurable in terms of version used. Previously, a script was used to update all 42 recipes as needed!

RUN printf '/usr/local/lib64\n' >> /etc/ld.so.conf.d/20local-lib.conf \
    && printf '/usr/local/lib\n' > /etc/ld.so .conf.d/20local-lib.conf \ ...

In order to not be affected by system packages or Conan packages that invoke apt-get, the compiler and its artifacts are installed in /usr/local. However, this is not enough to prioritize the order libstdc++ used, for that we need to update ldconfig with the local directories. Until then, this was not necessary in the previous images, as everything was either consumed directly from the system, or installed directly in /usr.

These are the main features of the base image, which is used in all final images.

For the construction of Clang, we tried to make it available from version 6.0 to 12, but we had a series of obstacles and challenges that made us change our mind. Here we will share a little bit of this long journey of CMake files and compilation hours.

Building GCC from Source

Now let’s look at the GCC build image, the full recipe can be found here, but let’s highlight a few points:

RUN cd gcc-${GCC_VERSION} \
    && ./configure --build=x86_64-linux-gnu --disable-bootstrap --disable-multilib ...

No matter the version, GCC continues to use the same lines for its build. Some factors were configured in this version used:

  • Bootstrap has been disabled to reduce build time to just 20 minutes.
  • Fortran is enabled, but it barely increase the building time and final

The last part of the image uses the concept of Docker multistage-build, a technique that avoids creating separate recipes and images to take advantage of common parts.

FROM ${DOCKER_USERNAME}/base-${DISTRO}:${DOCKER_TAG} as deploy

ARG GCC_VERSION ARG LIBSTDCPP_PATCH_VERSION

COPY --from=builder /tmp/install /tmp/install

RUN sudo rm -rf /usr/lib/gcc/x86_64-linux-gnu/* \
    && sudo cp -a /tmp/install/lib/gcc/x86_64-linux-gnu/${GCC_VERSION}
    /usr/lib/gcc/x86_64-linux-gnu/ \ && sudo cp -a /tmp/install/include/* /usr/local/include/ \ &&
    sudo cp -a /tmp/install/lib64/ /usr/local/ \ && sudo cp -a /tmp/install/libexec/ /usr/local/
    \ && sudo cp -a /tmp/install/lib/* /usr/local/lib/ \ && sudo cp -a /tmp/install/bin/*
    /usr/local/bin/ \ && sudo rm -rf /tmp/install \ && sudo update-alternatives --install
    /usr/local/bin/cc cc /usr/local/bin/gcc 100 \ && sudo update-alternatives --install
    /usr/local/bin/cpp cpp /usr/local/bin/g++ 100 \ && sudo update-alternatives --install
    /usr/local/bin/c++ c++ /usr/local/bin/g++ 100 \ && sudo rm /etc/ld.so.cache \ && sudo
    ldconfig -C /etc/ld.so.cache \ && conan config init --force

In this section the base image used is the same one we created before through a caching mechanism that drastically reduces the final image build time. All artifacts generated from GCC are now copied to their respective locations. Finally, the compiler becomes the default in the image and the libraries are listed and cached. And here’s the icing on the cake, the same recipe works from GCC 5 to the latest version. You just need to modify some arguments. The maintenance has been drastically simplified compared to current Conan Docker Tools.

Conan Meets the Wyvern: Building Clang C/C++ Compiler from Source

To see the full recipe, it is available here.

Let’s go a step further and detail the Clang deployment step.

FROM ${DOCKER_USERNAME}/gcc${LIBSTDCPP_MAJOR_VERSION}-${DISTRO}:${DOCKER_TAG} as libstdcpp

FROM ${DOCKER_USERNAME}/base-${DISTRO}:${DOCKER_TAG} as deploy

ARG LIBSTDCPP_VERSION ARG LIBSTDCPP_PATCH_VERSION

ARG DOCKER_USERNAME ARG DOCKER_TAG ARG DISTRO

COPY --from=builder /tmp/install /tmp/clang COPY --from=libstdcpp /usr/local /tmp/gcc

RUN sudo mv /tmp/gcc/lib64 /usr/local/ \
    && sudo ln -s -f /usr/local/lib64/libstdc++.so.6.0.${LIBSTDCPP_PATCH_VERSION}
    /usr/local/lib64/libstdc++.so.6 \ && sudo ln -s -f /usr/local/lib64/libstdc++.so.6
    /usr/local/lib64/libstdc++.so \ && sudo cp -a /tmp/gcc/include/*
    /usr/local/include/ \ && sudo rm -rf /usr/lib/gcc/x86_64-linux-gnu/* \ && sudo cp -a
    /tmp/gcc/lib/gcc/x86_64-linux-gnu/${LIBSTDCPP_VERSION} /usr/lib/gcc/x86_64-linux-gnu/ \ &&
    sudo cp -a /tmp/gcc/lib/* /usr/local/lib/ \ ...

Similar to what was done with GCC, in Clang we also use the same base image and copy the artifacts generated by the compiler to the /usr/local directory. However, the libstdc++ library was extracted from the GCC 10 image. This is a necessity of the possible configurations supported by Conan (possible values of compiler.libcxx setting).

In addition Clang requires some interesting CMake definitions:

  • LLVM_ENABLE_PROJECTS: Only enable what we want, otherwise we will have tons of binaries and hours of build
  • LLVM_USE_LINKER: We enforce LLVM linker (lld). It’s faster than GNU ld and reduces the total building time
...  && ninja unwind \ && ninja cxxabi \ && cp lib/libc++abi* /usr/lib/ \ && ninja cxx \ &&
ninja clang \

If we run ninja command alone, it builds more projects than we want configured as enabled, so we build one by one. Also, libcxx has a limitation when building using libc++abi, it searches on the system library folder, not the internal folders first.

Tests and more tests: A CI pipeline to test Docker images

To ensure that the images produced met our requirements we needed to add new tests that cover in addition to what was already tested in Conan Docker Tools. Until then, a single script was used which validated a series of builds, versions of installed binaries, and user permissions. The content of the new tests can be seen here.

We introduced greater modularization in the tests, dividing the steps into separate scripts to serve each compiler closer. With the support for Fortran it was necessary to adapt a test that covered it. Furthermore, many applications that are now bundled with Conan are no longer part of the base image and this has also been validated.

If you want to test locally a produced Docker image, you can easily run:

$ cd modern && pytest tests --image conanio/gcc10-ubuntu16.04 --service deploy -vv --user 0:0

The CI Service Change: From Travis to Azure and the Jenkins Arrival

Since the beginning of the project, Conan Docker Tools has always used CI services such as Travis and Azure. However, this did not give us the full power to prioritize the build in the queue, customize the host, or customize the build lines to use Docker-in-Docker if necessary.

With that in mind we started using Jenkins to also build the new Docker images. The big advantage in this is the use of features for cache in Docker. Previously a single job took up to 2 hours if it was on other services without the use of caching. Now using Jenkins and Docker --cache-from, updating a package from base image to final image takes just 4 minutes per job. The Jenkinsfile file used can be viewed here.

Although the file looks complicated at first glance, it is still possible to use docker-compose to build an image from scratch. As an example, let’s use Clang 12:

$ cd modern 
$ docker-compose build base 
$ docker-compose build clang12-builder 
$ docker-compose build clang12

The produced image will be named as conanio/clang12-ubuntu16.04:1.39.0, where 1.39.0 is the image tag and version of Conan installed. But it’s totally configurable by the .env file.

In the case of legacy images, they will continue to be built in Azure when needed, we have no intention of moving them to Jenkins due to effort and maintenance.

How to build a Conan package with new Docker images

After building our new images, we are ready to build our Conan packages. Let’s take Boost as our example.

$ docker run --rm -ti -v ${HOME}/.conan/data:/home/conan/.conan/data conanio/gcc10-ubuntu16.04:1.39.0
conan@148a77cfbc33:~$ conan install boost/1.76.0@ --build
conan@148a77cfbc33:~$ exit

Here, we start a temporary Docker container with interactive support. Also, we share our Conan cache data as volume. After starting, we build Boost 1.76.0 and its dependencies from source. All packages will be built and installed to the shared volume, so we can use it after closing the container. To finish and remove the container, we just need to exit.

$ docker run -d -t -v ${HOME}/.conan/data:/home/conan/.conan/data --name conan_container conanio/gcc10 
$ docker exec conan_container conan install boost/1.76.0@ --build 
$ docker stop conan_container 
$ docker rm conan_container

Similar execution, same result. Instead of creating a temporary Docker container, we executed it in the background. All container commands are passed by conan exec command. Also, we need to stop and remove manually after finishing.

Finals words and feedback

We invite everyone who uses Conan Docker Tools images to use this new formulation, which should become official soon. These new images are the result of a long journey where we learned from our mistakes and listened to the community to achieve what we have today. We believe improvements should be added continuously to keep the CDT progressing so please let us your feedback on issue #205.