Wes shared a personal story that highlights the dangers of ignoring test coverage. He once worked on a decade-old software product. This code was written in PHP. The automated test coverage for this product was literally zero.
The results of this lack of testing were staggering. The Change Failure Rate (CFR), which is the percentage of deployments that cause an incident or rollback, hovered near 100%. This meant almost every release introduced at least one new bug.
The team had adapted to the chaos instead of fixing it:
- Engineers braced for constant bug alerts.
- Sales representatives learned how to navigate around broken features during demos.
- Customer support needed scripted responses handy for common issues.
Executives asked for root causes. However, the only proposal available at the time was to manually write a massive test suite. This required adding a lot of new employees. Today, thanks to new technology, the path to fixing this problem looks completely different. GitAuto was started to give teams a fast, affordable way to escape this old nightmare.
Understanding Software Testing Fundamentals
We need to cover the basics of testing before discussing AI agents. Software testing falls into two main categories: manual and automated.
Manual Testing
Manual testing uses humans to find issues. This includes:
- Scripted manual testing.
- Exploratory testing (aggressively searching for bugs).
- Ad hoc testing (unplanned testing done by reviewers).
- User acceptance testing (UAT).
Manual testing is still very important. Humans are still great at finding nuanced and unexpected issues. However, relying only on manual testing is not scalable or reliable long-term.
Automated Testing Types
Automated testing is done by machines. It generally falls into three main types:
- Unit Tests: These check small, isolated pieces of code, typically individual functions or methods. They are fast and numerous. Unit tests are crucial for catching basic issues early in the process. Companies like Amazon demonstrate their importance by automatically blocking code that does not include adequate unit tests.
- Integration Tests: These verify that different components or modules work correctly together. For example, a unit test might simulate an external API response. An integration test actually makes that real API call and confirms real-world interactions.
- End-to-End (E2E) Tests: These simulate entire user workflows. Examples include signing in, signing up, making a payment, or installing and uninstalling software.
The key is finding a balance. While manual testing provides value, automated testing provides reliability and scale.
The Core Benefits of Automated Testing
Why is automated testing so important? The two biggest reasons are quality and savings in time and cost.
Quality Benefits
Automated tests provide these specific benefits for code quality:
- Catch Bugs Before Production: Tests catch problems early, before they reach actual users. This leads to a smoother experience for customers and fewer public issues.
- Psychological Safety for Developers: As your codebase grows, developers often edit code they did not write. Without automated tests, developers feel anxious about breaking things. Reliable tests provide immediate feedback if something breaks on a pull request.
- Living Documentation: Test suites act as a reliable form of documentation. Written documents often go stale. Tests stay current because if the code doesn’t behave as expected, the test fails. This makes tests the most reliable documentation you have.
Time and Cost Benefits
Automated tests save teams valuable resources:
- Speed and Repeatability: Automated tests are extremely fast and repeatable. Tasks that might take hours manually can finish in just seconds.
- Focus on Features: Less manual effort means your team can spend more time building features.
- Accelerated Release Cycles: Automated tests allow for frequent, confident releases. Without them, you must bundle changes and release infrequently. With automation, you can test every comment and merge event.
- Scalability: Automated tests run every time, even after tiny code tweaks. They scale easily, whether your team has two engineers or 200.
How to Track and Measure Test Progress
Once you start implementing automated testing, you need a way to measure your progress. The most practical way to do this is to track test coverage.
Most teams talk about statement coverage (also called line coverage). This shows what percentage of your executable code runs during tests. Other important types of coverage include:
- Function Coverage: This checks how many functions get called in your test suite.
- Branch Coverage: This is for conditional statements, like if statements. It checks if both the true and false paths are executed. If you cover both conditions, you achieve 100% branch coverage.
Test coverage is considered the “king” of test metrics because it is simple and powerful. If your team does not have a coverage goal yet, pick one and add it to your pipeline.
Test Coverage Reporting
When GitAuto or other AI agents work, they need clear data on what to test. Coverage tools provide detailed reports. While HTML formats are human-friendly, AI agents often use the LCOV code format.
LCOV reports are:
- Minimal and standardized.
- Easy for AI systems to consume.
- Supported across almost all programming languages.
This report is essential because it tells the AI exactly which lines and branches are currently untested.
The Rise of AI Coding Agents
AI agents are changing how development teams operate. There are different types of AI assistants:
- IDE Pet Programmers: Tools like GitHub Copilot or Cursor work inside your Integrated Development Environment (IDE). They help you write code or tests while you work on a project. However, they need you to ask them to do something.
- Autonomous Coding Agents: These run on the server side and are fully autonomous. They are allowed to edit all files in your repository. Examples include Devin from Cognition Labs.
Wes believes AI agents will follow the same path as human engineers: specialization. A general full-stack engineer is not always the best choice. Specialists, like QA coding agents, often outperform general AI agents in their specific focus area.
GitAuto is a specialized QA coding agent. It focuses on unit and integration testing. This specialization allows GitAuto to deliver a high impact in a focused area.
Introducing GitAuto: The QA Agent Workflow
GitAuto is a GitHub application that improves test coverage by generating test cases. It is triggered by events like manager assignments, code comments, or pull request events.
Here is the basic workflow for using GitAuto to improve coverage:
- Identify Low Coverage Files: The GitAuto dashboard displays all files in your repository with low test coverage scores.
- Target a File: A user selects the file they want to improve and instructs GitAuto to create a new issue for it.
- GitAuto Creates an Issue: The agent automatically creates a GitHub issue. This issue includes all the necessary information, such as current coverage reports, the specific lines to uncover, and the branch coverage details.
- Assign the Agent: The user assigns GitAuto to the issue by checking a box.
- Autonomous Testing: GitAuto starts working. It explores the repository, identifies relevant files, creates a new branch, generates the necessary test cases, and runs the tests using the existing GitHub CI/CD pipeline.
- Self-Correction: If the newly created tests fail, GitAuto attempts to self-correct the code or the test cases.
- Review and Merge: Once the tests pass successfully, GitAuto opens a pull request (PR). The user only needs to review the PR, check that the test results are green, and merge the changes.
This entire process can take as little as 10 minutes to get a file to 100% test coverage.
Reusable Test Prompts
To get the best results from any AI agent, you must be explicit about your preferences. Before asking the agent to write tests, you can provide reusable test prompts. These prompts cover rules like:
| Category | Preferences to Set |
|---|---|
| Test Coding Rules | Descriptive test names, concise variable names, function names to use. |
| External Dependencies | Whether to mock external APIs or use real dependencies. |
| Test Cases | If the agent should only cover happy paths or include all possible edge cases. |
| Comments & Documentation | If you want comments or reference URLs included in the tests. |
| File Structure | Where to place the new tests (e.g., in a dedicated test directory or next to the source file). |
| Constants | Where frequently used constants (like API tokens or webhook URLs) are defined. |
Demonstrating Real-World Test Generation
GitAuto can generate several types of tests automatically. Wes showed demos covering simple unit tests, API integration tests, and even component tests.
1. Generating Unit Tests
In a simple example, GitAuto targeted a file with a function that sends text to a Slack webhook URL. The original file had 90% statement coverage. After assigning the issue, GitAuto generated a pull request that included multiple tests:
- One happy path test (successful sending).
- Three edge case tests, including what happens if the Slack webhook URL is not set or if the request method throws an exception.
The tests ran immediately using GitHub CI/CD and passed. This took only a few minutes from issue creation to a ready-to-merge PR.
2. Integration Tests (API and Database)
GitAuto handles complex integration scenarios, too:
- API Integration: When testing a function that checks if a repository is forked, GitAuto generated six unit test cases and four integration test cases (10 total) with a single click. When a mistake caused the first test suite to fail, GitAuto was provided the error log and self-corrected the issue, demonstrating its autonomous capabilities.
- Database Integration: The agent can also generate tests for database interactions. In one demonstration, GitAuto created two test cases (one success case and one edge case) that successfully passed validation.
3. Component and E2E Testing
GitAuto can also help with higher-level testing, although it specializes in unit and integration tests:
- Component Tests: The agent successfully created unit tests specifically for a component, such as an “install button” on a website.
- E2E Tests: GitAuto generated an End-to-End test that ensured clicking the button brought the user to the specified URL. These E2E tests are placed in separate file paths, keeping the structure organized.
Conclusion: Integrating GitAuto into Your Project
GitAuto improves test coverage by acting like a specialized QA engineer on your team.
Installation and Integration
GitAuto is installed as a GitHub application.
It is not a dependency that you install directly into your project code. Instead, it is installed into your GitHub Actions (CI/CD environment). This setup is the same way a human engineer would use the CI/CD pipeline. GitAuto accesses your repository with necessary permissions and uses GitHub Actions to run the tests it generates.
Platform Support
Currently, GitAuto supports the GitHub platform only. It uses GitHub Actions for its CI/CD process.
However, GitAuto is programming language agnostic. It works with Python, Typescript, PHP, Ruby, and Swift.
While GitAuto is tightly integrated with GitHub actions now, it will soon support other CI/CD pipelines, such as Jenkins or Circle CI. This is because GitAuto only needs the test coverage report (in LCOV format), which any CI/CD platform can generate. GitAuto will support other platforms as soon as possible.
