Bring flaky test insights into a scheduled website review
See website observations alongside CircleCI's flaky test insights, so recurring reliability checks are easier to scan and act on.
Trigger
Recurring schedule
Notis starts this workflow on a schedule, such as daily, weekly, or during business hours.
Action
Get Flaky Tests
Finds tests that passed and failed in the same commit, across branches.
Why this helps
Flaky tests and website observations are easy to review separately, leaving reliability signals scattered across tools.
- Surface tests that passed and failed on the same commit.
- Review test reliability with website observations in one report.
- Make follow-up candidates easier to spot.
Setup
Build it in a few focused steps.
- 1Connect AgentQL and CircleCI once in the Notis portal.
- 2Create an automation in Automations, New Automation, or tell Notis the outcome in plain language from any channel.
- 3Ask Notis to summarize an AgentQL website inspection alongside CircleCI flaky tests for the relevant project.
- 4Pick a recurring Notis schedule as the trigger and choose where run reports should go.
- 5Test with one real scheduled run and a project that has flaky test insights.
Questions about this workflow
What does CircleCI consider in this flaky test result?
The action identifies tests that passed and failed in the same commit, with branch-agnostic insights.
Can this show whether flaky tests caused a website problem?
No. It provides test reliability insights and site observations separately. A causal link needs evidence beyond these results.
What must be known to retrieve flaky tests?
The relevant CircleCI project must be identified in the automation instruction.
When this happens · Trigger
Do this · Action
Supported Triggers and Actions
Notis builds workflows that link Agentql to Circleci. A trigger fires from one place; an action lands in another.
Agentql triggers
Circleci actions
Recurring trigger
Notis starts this workflow on a schedule, such as daily, weekly, or during business hours.
Get Job Artifacts
Retrieves files produced by a job.
Webhook trigger
Notis starts this workflow when an external tool or custom backend sends an HTTP request.
Create Context
Creates a context for sharing environment variables securely across projects.
Get Job Details
Retrieves status, timing, and executor details for a job.
Create Context (GraphQL)
Creates a context for storing environment variables; returns an error if creation fails.
Get Pipeline Config
Retrieves the source or compiled configuration for a pipeline.
Create Context Restriction
Creates a context restriction to limit access by project, rules, or group.
Get Test Metadata
Retrieves test results, run times, and failure details for a job.
Create Organization Orb Allowlist
Allows URL-based orb references with matching prefixes in an organization’s pipelines.
Connect any two apps with Notis in the middle.
Agentql and Circleci, or any other combination from 1,000+ integrations.
When this happens · Trigger
Do this · Action
Save your first hour today.
7-day trial of any paid plan, with 20$ of usage included.
No card. Works with personal or business Circleci.