Conan 2.0 launches bulk upload, download and remove operations with 'packages lists'
In Conan 1.X, uploading the exact set of packages that were built from source in the last conan install --build=missing
command was not a straightforward task.
It was necessary to have some script that would get the output of the command, and then iterate the result, executing conan upload <ref>
one reference at a time.
We are happy to launch the “packages lists” feature in Conan 2.0.7, which makes this highly requested functionality built-in.
Introduction to packages-lists files
A “packages list” is a json file that contains a list of Conan artifacts, both recipes and package binaries.
This file can be generated as the --format=json
of several commands. For example a conan list
command like the following:
# The output file name is user defined
$ conan list "*:*" --format=json > pkglist.json
will generate a pkglists.json
file that looks like this (simplified, not complete):
{
"conancenter": {
"zlib/1.2.12": {
"revisions": {
"b1fd071d8a2234a488b3ff74a3526f81": {
"timestamp": 1667396813.987,
"packages": {
"ae9eaf478e918e6470fe64a4d8d4d9552b0b3606": {
"revisions": {
"19808a47de859c2408ffcf8e5df1fdaf": {}
},
"info": {
"settings": {
"arch": "x86_64",
"os": "Windows"
}
}
}
}
}
}
},
"zlib/1.2.13": {}
}
}
The first level in the pkglist.json
is the “location”, it can be a remote or the local cache. In this case, we were listing
packages in the cache, so the origin is “Local Cache”.
The conan remove
, conan upload
and conan download
commands can also generate “packages lists” with the --format=json
formatter. For example conan upload ... -r=myremote --format=json
will generate a packages list for the “myremote” location
as the first level in the resulting json file.
Furthermore, it is also possible to generate a packages list from the result of a conan create
, conan install
and conan graph info
commands, as we will see in the examples below.
One of the advantages of packages lists files is that they can be used as inputs to perform bulk operations in several commands:
$ conan upload --list=pkgs_to_upload.json -r=myremote
$ conan download --list=download.json -r=myremote
$ conan remove --list=pkglists.json # From the cache or -r=remote
Let’s see some more complete and useful examples:
Downloading from one remote and uploading to a different remote
This is something that can be useful to populate or to promote packages in an air-gapped environment, where there are 2 servers. One server is the public one containing some packages that we want to introduce in the isolated environment (with a due diligence). This is fairly common in enterprise environments.
For example, let’s say we want to download some packages from ConanCenter, disconnect from the internet, check the packages locally, and finally upload them to our own private internal server. (Note: This is an example, for ConanCenter packages a better approach when security, reproducibility and robustness are very important would be building the packages from source, from a conan-center-index fork)
With the conan download
command, we can download some packages, for example, all the Windows binaries for zlib/1.2.12
(by default only the latest revisions), and generate a downloaded.json
packages list file:
$ conan download zlib/1.2.12:* -p "os=Windows" -r=conancenter --format=json > downloaded.json
The resulting downloaded.json
file will look like:
{
"Local Cache": {
"zlib/1.2.12": {
"revisions": {
"b1fd071d8a2234a488b3ff74a3526f81": {
"timestamp": 1667396813.987,
"packages": {
// full list of package binaries (Windows)
}
}
}
}
}
}
That means that we can now upload this same set of recipes and binaries to a different remote:
$ conan upload --list=downloaded.json -r=myremote -c
# Upload the artifacts listed in the "Local Cache" location to my own remote
One of the important points is that this works irrespective of the cache contents. It doesn’t matter if the local cache
contains other artifacts from other commands. The packages uploaded to the myremote
repository will be only the ones listed in the downloaded.json
.
Note that this would be a slow mechanism to run promotions between different server repositories. Some servers provide ways to directly copy packages from one repository to another without using a client, that are orders of magnitude faster because of file deduplication, so that would be the recommended approach if the 2 repositories can directly talk to each other.
Building from source and uploading packages
One of the most interesting applications of “packages lists” is the one when some packages are being built in the local cache, with a
conan create
or conan install --build=xxx
command. Typically, we would like to upload the locally built
packages to the server, so they don’t have to be re-built again by others. But we might want to upload only
the built binaries, but not all others transitive dependencies, or other packages that we had previously in
our local cache.
It is possible to use the graph from any of the commands conan install
, conan create
and conan graph info
to compute a package list which can then be used with upload
commands. Then, that package list can be used for the upload.
For example we can build from source the latest version of spdlog
and its transitive dependencies (this includes fmt
) with:
$ conan install --requires="spdlog/[*]" --build="*" --format=json > build.json
This will create a json representation of the graph, with information of what packages have been built (that is, they contain "binary": "Build"
),
the build.json
will look like (simplified):
{
"graph": {
"nodes": {
"0": {
"ref": "conanfile",
"recipe": "Cli",
"dependencies": { "1": {"ref": "spdlog/1.11.0"} }
},
"1": {
"ref": "spdlog/1.11.0#d0fdbaa523550b89156084bf42b41c90",
"binary": "Build",
"dependencies": { "2": {"ref": "fmt/9.1.0"}}
},
"2": {
"ref": "fmt/9.1.0#44302d39c5a4bf7de8a39adc50bb4568",
"binary": "Build"
}
}
}
}
We can compute a package list from this file, and then upload those artifacts to the server with:
$ conan list --graph=build.json --graph-binaries=build --format=json > pkglist.json
The resulting pkglist.json
will contain both spdlog
and fmt
recipes and built binaries, and
we can now use this list to upload these exact packages (and not other existing ones in the cache, or other things that
were not built from sources in my install
command) to my own server:
$ conan upload --list=pkglist.json -r=myremote -c
Conclusions
One of the advantages of the new Conan 2.0 architecture is that it is enabling the quick development of demanded features. This new “packages lists” feature is a new powerful addition to the Conan CLI, allowing to do bulk operations over sets of packages that previously required user custom automation (script files or Continuous Integration jobs), and now are conveniently provided as built-in.
Feedback is very welcome, please create a Github issue for any question, comment or suggestion about it.