Skip to main content

Analytics & Reporting

The Analytics & Reporting section of the platform provides users with powerful tools to monitor, understand, and act on testing activity and project health. This area is designed to help you make informed decisions by visualizing key metrics, trends, and detailed execution results. Below, you'll find guidance on using the main dashboard, exploring project-specific analytics, and generating or viewing detailed execution reports.

Main Dashboard​

Overview​

The main dashboard serves as your central hub for understanding the overall state of your testing efforts across all projects. It presents high-level statistics, visual charts, and recent activity feeds to give you a quick yet comprehensive snapshot of your organization's testing landscape.

Key Features​

  • Statistics Cards: Instantly see counts of passed, failed, and pending test cases, as well as the overall pass rate.
  • Recent Projects & Test Runs: Quickly access a list of your most recently updated projects and test runs.
  • Charts & Visualizations: View bar and pie charts that break down test execution statuses and project health.

How to Use​

  1. Accessing the Dashboard: Upon logging in, you are greeted by the dashboard. You can always return here via the sidebar navigation.
  2. Interpreting Metrics: Review the statistics cards at the top to gauge your testing performance at a glance.
  3. Exploring Recent Activity: Scroll through the recent projects and test runs to jump directly to items that need your attention.
  4. Analyzing Charts: Use the visual charts to spot trends, such as a spike in failed tests or improvements in pass rates.
  5. Navigating to Details: Click on any project or test run in the lists to drill down for more information.

Tip: The dashboard aggregates data from all your projects, making it ideal for managers or users overseeing multiple teams.

Project Analytics​

Overview​

Project analytics allow you to focus on a single project and understand its unique trends, strengths, and areas for improvement. This feature provides time-based visualizations and detailed breakdowns of test case creation, execution results, and project health.

Key Features​

  • Time-Series Trends: See how test case creation and execution results evolve over time.
  • Project Health Metrics: Get a health score based on the ratio of passed to failed tests and other factors.
  • Detailed Charts: Visualize distributions of test statuses, priorities, and types within the project.
  • Tab-Based Navigation: Easily switch between different aspects of the project, such as milestones, test plans, suites, cases, and runs.

How to Use​

  1. Selecting a Project: From the dashboard or projects list, click on a project to open its detailed view.
  2. Navigating Tabs: Use the tabbed interface to switch between overview, milestones, test plans, test suites, test cases, and test runs.
  3. Viewing Analytics: In the overview tab, review charts and statistics that show:
    • Test case creation trends (how many new cases are added over time)
    • Execution trends (how many tests are run, and their outcomes, over time)
    • Health metrics (such as pass rate and defect counts)
  4. Exploring Details: Click on any chart segment or list item to see the underlying data, such as which cases failed or which runs had the most issues.
  5. Comparing Versions: Use the version manager to view or revert to previous versions of the project data.

Tip: Use the analytics to identify bottlenecks, track progress toward milestones, and prioritize areas for improvement within your project.

Execution Reports​

Overview​

Execution reports provide detailed insights into individual test runs. These reports help you understand what happened during a specific run, including which tests passed or failed, who executed them, and any defects that were discovered.

Key Features​

  • Run Summary: See the overall status of the test run, including start/end times, duration, and pass/fail rates.
  • Test Execution Table: Review a detailed table of all test cases included in the run, their execution status, assignee, and any linked defects.
  • Status Distribution: Visual charts show the breakdown of test results (passed, failed, blocked, etc.).
  • Defect Tracking: Identify which test cases resulted in defects and track their resolution status.
  • Sorting & Filtering: Organize the table by status, assignee, or other criteria to focus on the most relevant information.

How to Use​

  1. Accessing a Test Run: Navigate to a project, then select the "Test Runs" tab. Click on a specific run to open its details.
  2. Reviewing the Report: The report displays a summary at the top, including key metrics and a status chart.
  3. Examining Test Results: Scroll through the table to see the outcome for each test case. Use sorting options to group by status or assignee.
  4. Investigating Failures: Click on failed or blocked test cases to view their details, including steps, comments, and any attached defects. Tip: Use execution reports during retrospectives or quality reviews to identify recurring issues, assign follow-up actions, and demonstrate testing coverage to stakeholders.

Additional Guidance​

  • Role-Based Access: Some analytics and reporting features may be restricted based on your user role. Administrators have access to all data, while other roles may see only the projects and runs they are assigned to.

Scheduled Reports​

Automate report generation on a recurring schedule.

Report Scheduling Options​

  • Daily Reports: Automatically generated reports every day at 6 AM
  • Weekly Reports: Weekly reports generated every Monday at 6 AM
  • Monthly Reports: Monthly reports generated on the 1st of each month at 6 AM

Report Output​

  • PDF Export: Reports are generated in PDF format
  • Report Types: Milestone, Test Plan, and Test Run reports can be scheduled

How to Use​

  1. Navigate to the Reports section within your project
  2. Create a new report and enable the scheduling toggle
  3. Select the frequency (daily, weekly, or monthly)
  4. The system will automatically generate the report on schedule

AI Usage Analytics​

Track and analyze AI credit consumption for cost management.

Credit Usage Tracking​

  • Operation-Level Analytics: Detailed breakdown by AI operation type (project_import, chunk_processing, heal_test_case, aggregate_chunks, github_integration, generate_automation_script)
  • AI Provider & Model Tracking: Each operation records the AI provider (Vertex AI, Anthropic via Vertex AI, Llama) and specific model used
  • Cost Analysis: Estimated USD cost per operation based on input/output token counts
  • User-Level Tracking: Individual user consumption patterns and quota utilization with per-user quota enforcement

Quota Management​

  • Usage Progress Bar: Visual indicator showing current quota utilization
  • Monthly Reset: Monthly credit quotas reset on the 1st of each month; purchased credits persist across resets but expire after 365 days
  • Dual-Level Monitoring: Track both organization-level pool and individual user allocations
  • Org Pool View (Tenant Owner): See the organization's total monthly allocation, allocated credits, available pool, and purchased credit balance

How to Use​

  1. Navigate to the Credits & Billing page in the sidebar
  2. View your monthly quota, used credits, remaining balance, and reset date
  3. Review the operation breakdown to see where credits are being consumed
  4. Tenant owners can view the organization pool and adjust user allocations via the Manage Allocations modal

By leveraging the Analytics & Reporting features, you can monitor testing progress, track AI usage costs, and make informed decisions about testing strategy and resource allocation.