# curator **Repository Path**: mirrors_mongodb/curator ## Basic Information - **Project Name**: curator - **Description**: Curator -- a build and package automation tool - **Primary Language**: Unknown - **License**: Apache-2.0 - **Default Branch**: main - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2020-08-09 - **Last Updated**: 2025-06-11 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README ================================================= ``curator`` -- Artifact and Repository Management ================================================= Overview -------- Curator is a tool that we use at MongoDB to generate our package repositories (e.g. ``repo.mongodb.org`` and ``repo.mongodb.com``). Additionally, curator provides tooling to support the automated publication and management of build artifacts as part of our continuous integration system. Components ---------- Please refer to the ``--help`` output for ``curator`` and its sub-commands. The following sections provides overviews of these components and their use. S3 Tools ~~~~~~~~ Curator includes a basic set of AWS S3 operations on the command line. Although the get, put, and delete operations are faily basic the ``sync-to`` and ``sync-from`` operations provide parallel directory tree sync operations between S3 buckets and the local file system. These operations compare objects using MD5 checksums, do multi-threaded file uploads, and retry failed operations with exponential backoff, for efficient and robust transfers. Repobuilder ~~~~~~~~~~~ The repobuilder is a helper package to submit Barque jobs to build RPM and DEB package repositories in S3. These jobs: sync files from an existing repository, add packages from the local filesystem to the repository, sign packages (for RPM), regenerate package metadata, sign package metadata, generate html pages for web-based display, and sync the changed files to the remote repository. The current implementation of the repobuilder process depends on external repository generation tools (e.g. ``createrepo`` and ``apt`` tools.) Additionally, the repobuilder currently depends on MongoDB's internal signing service. Artifacts ~~~~~~~~~ The artifacts functionality uses the release metadata feeds (e.g. ``https://downloads.mongodb.org/full.json``) to fetch and extract release build artifacts for local use. It is particularly useful for fetching artifacts for and maintaining local caches of MongoDB builds for multiple releases. Set the ``CURATOR_ARTIFACTS_DIRECTORY`` environment variable or pass the ``--path`` option to a flag, and then use the ``curator artifacts download`` command to download files. The ``artifacts`` command also includes two exploration subcommands for discovering available builds: Use the ``list-map`` for specific lists of available edition, target, and architecture combinations and ``list-all`` for a list of available target and architectures. Both list operations are specific to a single version. Combine the artifacts tool with the prune tool to avoid unbounded cache growth. Prune ~~~~~ Prune is based on the `lru `_ library, and takes a file system tree and removes files, based on age, until the total size of the files is less than a specified maximum size. Prune uses modified time for age, in an attempt to have consistent behavior indepenent of operation system and file system. There are two modes of operation, a recursive mode which removes objects from the tree recursively, but skips directory objects, and directory mode, which does not collect objects recursively, but tracks the size for the contents--recursively--of top-level directories. Development ----------- Design Principles and Goals ~~~~~~~~~~~~~~~~~~~~~~~~~~~ - All operations in the continuous integration environment should be easily reproducible outside of the environment. In practice, curator exists to build repositories inside of Evergreen tasks; however, it is possible to run all stages of the repository building process by hand. The manual abilities are useful and required for publishing package revisions and repairing corrupt repositories. - Leverage, as possible, third party libraries and tools. For example, the cache pruning and artifact management functionality is entirely derived from third-party repositories maintained separately from curator. - Major functionality is implemented and executed in terms of `amboy `_ jobs and queues. Not only does this provide a framework for task execution, but leaves the door open to provide curator functionality as a highly available service with minimal modification. APIs and Documentation ~~~~~~~~~~~~~~~~~~~~~~ See the `godoc API documentation `_ for more information about curator interfaces and internals. Issues ~~~~~~ Please file all issues in the `EVG project `_ in the `MongoDB Jira `_ instance.