# dr-ci

**Repository Path**: mirrors_pytorch/dr-ci

## Basic Information

- **Project Name**: dr-ci
- **Description**: Diagnose and remediate CI jobs
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2020-08-18
- **Last Updated**: 2025-10-04

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

[![CI](https://github.com/pytorch/dr-ci/workflows/CI/badge.svg)](https://github.com/pytorch/dr-ci/actions?query=branch%3Amaster)

# A log analyzer for CircleCI


## Intro

An organization would like to determine what the most common causes of intermittent build
failures/flaky tests are in the a repository so that effort can be prioritized
to fix them.

## Outputs

The Dr. CI project entails two distinct user-facing outputs:

* [Automatically-posted GitHub PR comments](application-logic/pull-request-comments)
* The Dr. CI website

The latter has several distinct utilities:
* Annotation interface for determinisitic `master` failures
* Flakiness review tool
* Stats dashboards

# Codebase

See [docs/CODEBASE-OVERVIEW.md](docs/CODEBASE-OVERVIEW.md).

# Repository assumptions

Dr. CI assumes a linear history of the `master` branch.
This can be enforced on GitHub via the following setting under the "Branches" -> "Branch protection rule" section for `master`:

![GitHub setting](docs/images/github-linear-history-option.png)

## Functionality
This tool obtains a list of CircleCI builds run against a GitHub repository for
a master branch, downloads their logs (stripped of ANSI escape codes) from AWS, and scans the logs for a
predefined list of labeled patterns (regular expressions).

These patterns are curated by an operator.  The frequency of occurrence of each
pattern are tracked and presented in a web UI.

The database tracks which builds have been already scanned for a given pattern,
so that scanning may be performed incrementally or resumed after abort.

### Tool workflow

* A webhook listens for build status changes on a GitHub PR
* For each failed build, that build's log will be scanned for any of the patterns in the database tagged as "flaky"
* If all of the failures were flaky, the indicator will be green.  There will be a link in the status box to dive into the details.
* likewise for failures marked with my tool as "known problems"

### Known Problem reporting

Requiring that failures in the master branch be annotated will facilitate tracking of the frequency of "brokenness" of master over time, and allow measurement of whether this metric is improving.

It is possible for only specific jobs of a commit to be marked as "known broken", e.g. the Travis CI Lint job.

## Log scanning data flow diagram

![flow diagram](docs/data-flow.svg)


Deployment
-------------

### Development Environment Setup

See: [docs/development-environment](docs/development-environment)


### AWS dependencies and deployment

See: [docs/aws](docs/aws)


### Ingestion overview

1. A small webservice (named `gh-notification-ingest-env` in Elastic Beanstalk, and hosted at domain `github-notifications-ingest.pytorch.org`) receives GitHub webhook notifications and stores them (synchronously) in a database.
2. A periodic (3-minute interval) AWS Lambda task `EnqueSQSBuildScansFunction` queries for unprocessed notifications in the database, and enqueues an SQS message for each of them.
3. Finally, an Elastic Beanstalk Worker-tier server named `log-scanning-worker` process the SQS messages as capacity allows.

We want a cool-off period during which multiple builds for a given commit can be aggregated into one task for that commit.
This is accomplished via an SQS deduplicating queue, where multiple instances of the same commit are consolidated while in the queue.


Optimizations
-------------

* We can skip inspecting *all* of the "previously-visited" builds if the master "scan" record points to the newest pattern ID.
    * Better yet, use a single DB query to get the list of out-of-date "already-visited" builds, instead of a separate query per build to obtain the unscanned pattern list.


## Other Features

* Periodically fetches builds directly from CircleCI API to catch up on GitHub notifications that may have been dropped


## Source attribution

Aho-Corasick implementation is from here: https://github.com/channable/alfred-margaret


### [Code of Conduct](https://code.fb.com/codeofconduct)

Facebook has adopted a Code of Conduct that we expect project participants to adhere to. Please read [the full text](https://code.fb.com/codeofconduct) so that you can understand what actions will and will not be tolerated.


### License

Dr. CI is [MIT licensed](./LICENSE).