Give every voice-agent PR a regression check
Catch conversational regressions while the code is still fresh in everyone's mind.
Trigger
New Pull Request Created
Triggers when a new pull request is created in GitHub, matching the optional configured filters. Emits an event for each newly created pull request. Supports filtering by repository, owner/organisation, author, assignee, label, state, base branch, head branch, and draft status.
Action
Create Eval
Tool to create an eval for testing conversation flows. Use when you need to validate that an AI assistant responds correctly to specific conversation scenarios.
Why this helps
Founders often remember test coverage only when a customer notices a problem.
- Fewer forgotten tests
- Faster feedback loops
Setup
Build it in a few focused steps.
- 1Connect both integrations to Notis once.
- 2Create an automation in the portal.
- 3Ask Notis to create a regression eval for relevant new pull requests.
- 4Choose the new-pull-request trigger and where run reports arrive.
- 5Test it on one real PR.
Questions about this workflow
Can irrelevant PRs be ignored?
Yes. Describe which voice-agent work should receive an eval.
When this happens · Trigger
Do this · Action
Supported Triggers and Actions
Notis builds workflows that link GitHub to Vapi. A trigger fires from one place; an action lands in another.
GitHub triggers
Vapi actions
New Workflow Artifact Created
Triggers when a new workflow artifact is created in a GitHub repository. Monitors for newly created GitHub Actions workflow artifacts. Optionally filters by artifact name to restrict monitoring to specific artifacts.
Update Assistant
Tool to update an existing Vapi assistant configuration. Use when you need to modify assistant properties such as name, voice settings, transcriber configuration, model settings, messages, or other behavior parameters. Only include fields you want to update.
Branch Changed
Triggers when a GitHub branch changes. Monitors a specific branch for: - New commits pushed (head commit SHA changes) - Protection status toggled (branch becomes protected or unprotected) - Protection settings changed, including: required status checks and their enforcement level, admin enforcement, required pull request reviews (dismiss stale reviews, code owner reviews, approving review count, last push approval), required linear history, force push allowance, deletion allowance, conversation resolution, branch locking, and fork syncing.
List Calls
Tool to list calls from Vapi. Use when you need to retrieve multiple calls with optional filtering by id, assistantId, phoneNumberId, or date ranges. Returns an array of call objects with details including status, costs, messages, and artifacts.
New Branch Created
Triggers when a new branch is created in a GitHub repository. Detects newly created branches. Deleted branches do not fire events.
Delete Chat
Tool to delete a chat by its ID from Vapi. Use when you need to permanently remove a chat conversation.
Check Run Status / Conclusion Changed
Triggers when a specific GitHub check run changes its status or conclusion. Monitors a single check run for changes to: status (queued, in_progress, completed, etc.), conclusion (success, failure, neutral, cancelled, skipped, timed_out, action_required), started_at, and completed_at.
Get Chat
Tool to fetch chat details by ID. Use when you have a chat ID and need full chat information including messages, costs, and configuration.
Check Suite Status / Conclusion Changed
Triggers when a GitHub check suite changes its status or conclusion for a given ref. Monitors all check suites associated with a git reference (branch, tag, or commit SHA) for changes to status (queued, in_progress, completed, etc.) and conclusion (success, failure, neutral, cancelled, skipped, timed_out, action_required, startup_failure, stale). Optionally filters by GitHub App ID.
Create Analytics Queries
Tool to create and execute analytics queries on VAPI data. Use when you need to analyze call or subscription metrics with aggregations like count, sum, average, min, or max. Supports grouping by various dimensions and time-based analysis with custom time ranges.
New Code Scanning Alert Created
Triggers when a new code scanning alert is created in a repository. Fires an event for each newly created code scanning alert detected in the configured repository. Alerts can be filtered by Git reference, scanning tool, state, and severity. The payload includes the alert number, rule details, tool information, state, severity, and the location of the most recent instance.
Create Assistant
Tool to create a new Vapi assistant with specified transcriber, voice, and AI model configurations. Use when setting up a conversational AI assistant for voice interactions. The assistant requires transcription (speech-to-text), voice (text-to-speech), and AI model (conversation logic) configurations at minimum.
New Repository Collaborator Added
Triggers when a new collaborator is added to a GitHub repository. Monitors the full list of collaborators on a repository and fires an event for each newly added collaborator. The payload includes the collaborator's GitHub username, account ID, profile URL, avatar URL, permission flags (pull, triage, push, maintain, admin), and assigned role name.
Create Eval
Tool to create an eval for testing conversation flows. Use when you need to validate that an AI assistant responds correctly to specific conversation scenarios.
Commit Event
Triggered when a new commit is pushed to a repository.
Create OpenAI Chat
Tool to create an OpenAI-compatible chat using the Vapi API. Use when you need to send a chat message to an assistant or squad and receive a response. Supports both streaming and non-streaming modes.
Connect any two apps with Notis in the middle.
Not just GitHub and Vapi. Any combination from 1,000+ integrations.
When this happens · Trigger
Do this · Action
Save your first hour today.
7 days free trial with 20$ free usage included.
No card. Works with personal or business Vapi.