In the fast-paced world of software development, reproducibility and traceability are becoming increasingly crucial. However, what happens when the third-party sources you rely on suddenly become unavailable? With the new Conan Source Backup feature, you can ensure that your builds are not only reproducible but also resilient against the uncertainties of external dependencies.
Thanks to this feature, you can configure Conan so that when a recipe downloads the sources of a library, it allows you to fetch sources from a file server within your infrastructure. This is done in a completely transparent manner for the developer and can be set up to be used only as a last resort in case the URL of the original sources fails.
Some of the advantages of using this feature are:
- Reproducibility: Ensures long-term reproducibility in your builds.
- Traceability: Keeps a record of the origin of downloaded files.
- Robustness: Stores the sources in your own infrastructure, reducing reliance on third-party services.
- Improved Speed: Hosting source files internally can yield faster download speeds compared to fetching from external servers.
- Cost Efficiency: Having sources within the same network can reduce egress charges, thus saving on networking costs.
In this post, we’ll show you how to make the most of the sources backup feature, using our experience at Conan Center as an example. This will give you a clear picture of how you can seamlessly integrate this feature into your workflow for better reliability and efficiency.
How the Conan team uses this feature in Conan Center
This feature has been used for some time now in a production environment like Conan Center. Conan Center hosts over 6000 distinct references with thousands of different source origins. Hence, it’s highly susceptible to situations where some of these source origins might become unavailable or even disappear. Under these conditions, it becomes an ideal production environment for the internal use of Conan’s Sources Backup feature within the CI server.
Using the feature on the developer side
To use this feature, the first step is to have a file server. In Conan Center, we use a
generic Artifactory repository which seamlessly fits with our infrastructure, but you
could use any other file server that supports HTTP
GET methods. The first
thing to configure is the server URL in the
global.conf file of
the Conan client:
# global.conf core.sources:download_urls=["origin", "https://url/for/my-backup-sources-download-server/"]
As illustrated, a list of different file servers can be provided, and Conan will refer to
this list each time it encounters a call to
in a recipe. To designate the original source, we use the reserved word
this configuration will first attempt to find the sources on the original server, and if
that fails, it will fall back to our server (you can list them in the configuration by
order of preference). For instance, in Conan Center, we use this server:
Upon accessing it, we can view the information regarding the copies of the different
Bear in mind, if server credentials are needed, they should be configured with the
configuration file. Another crucial point is that for the feature to be activated, the
sha256 of the files used in
download() methods must be defined (as seen in
the image above, the sources of the packages are saved in files with the binary hash as
the name, accompanied by a metadata
With the explanation provided, a developer or continuous integration system would have enough information to benefit from this feature for downloading sources. Now, we’ll look into the configuration needed on the client to facilitate the upload of those sources to the server.
Configuring the source upload
The source upload is typically executed from the CI server. In Conan Center, for example,
the backup of sources is uploaded every time a PR for a new version passes all checks
(turns green). This upload is performed transparently during a
conan upload if the
following configuration is defined in the
# global.conf core.sources:upload_url=https://url/for/my-backup-sources-upload-server
It’s important to note that the upload server could differ from the download servers.
Additionally, in some scenarios, it may be needed to exclude certain origins from the backup sources upload (imagine it’s code you don’t want all users to have access to). In such cases, the following configuration can be utilized to skip them if a call is made, and any URL that starts with the listed ones in the configuration will be skipped:
# global.conf core.sources:exclude_urls=["https://url/mycompanystorage/", "https://url/mycompanystorage2/"]
To explore the backup sources feature further, you can check the Conan documentation.
Conan’s Source Backup feature is a potent tool for ensuring long-term reproducibility and traceability of your builds. By taking a few simple steps to configure this feature, you can shield your projects from the uncertainties associated with relying on third-party source repositories.