Find flaky tests in affiliate-supporting projects
Run a recurring CircleCI flaky test check so tests that pass and fail on the same commit are easier to find and address.
Trigger
Recurring schedule
Notis starts this workflow on a schedule, such as daily, weekly, or during business hours.
Action
Get Flaky Tests
Finds tests that passed and failed in the same commit, across branches.
Why this helps
Flaky tests waste focus because they make it harder to tell whether a failure signals a real regression.
- Surface tests that both pass and fail in the same commit
- Give reliability cleanup a regular review point
- Reduce repeat investigation of inconsistent failures
Setup
Build it in a few focused steps.
- 1Connect Adtraction and CircleCI once in the Notis portal.
- 2Create an automation in Automations, New Automation, or tell Notis the outcome in plain language from any channel.
- 3Describe which project's flaky tests to report and how to summarize them for follow-up.
- 4Pick the Notis cron trigger and choose the schedule.
- 5Choose where run reports go, then test with one real example.
Questions about this workflow
What qualifies as a flaky test in this report?
CircleCI identifies tests that passed and failed in the same commit. The action provides branch-agnostic insights.
Can it fix the flaky tests?
No. It identifies them for review; the workflow does not modify tests or project code.
When this happens · Trigger
Do this · Action
Supported Triggers and Actions
Notis builds workflows that link Adtraction to Circleci. A trigger fires from one place; an action lands in another.
Adtraction triggers
Circleci actions
Recurring trigger
Notis starts this workflow on a schedule, such as daily, weekly, or during business hours.
Get Job Artifacts
Retrieves files produced by a job.
Webhook trigger
Notis starts this workflow when an external tool or custom backend sends an HTTP request.
Create Context
Creates a context for sharing environment variables securely across projects.
Get Job Details
Retrieves status, timing, and executor details for a job.
Create Context (GraphQL)
Creates a context for storing environment variables; returns an error if creation fails.
Get Pipeline Config
Retrieves the source or compiled configuration for a pipeline.
Create Context Restriction
Creates a context restriction to limit access by project, rules, or group.
Get Test Metadata
Retrieves test results, run times, and failure details for a job.
Create Organization Orb Allowlist
Allows URL-based orb references with matching prefixes in an organization’s pipelines.
Connect any two apps with Notis in the middle.
Adtraction and Circleci, or any other combination from 1,000+ integrations.
When this happens · Trigger
Do this · Action
Save your first hour today.
7-day trial of any paid plan, with 20$ of usage included.
No card. Works with personal or business Circleci.