list-unstable-tests

Name

ci:list-unstable-tests

Synopsis

/ci:list-unstable-tests <version> <keywords> [sippy-url]

Description

The ci:list-unstable-tests command queries OpenShift CI test results from Sippy and lists all tests matching the keywords that have a pass rate below 95%. This is useful for quickly identifying unstable tests that need attention.

By default, it queries the production Sippy instance at sippy.dptools.openshift.org. You can optionally specify a different Sippy instance URL to query alternative environments (e.g., QE component readiness).

This command is useful for:

• Identifying unstable tests with inconsistent pass rates
• Finding regression candidates for investigation
• Generating reports of unstable test cases
• Prioritizing test stabilization efforts
• Quality gate checks before releases

Arguments

• $1 (version): OpenShift version to query (e.g., "4.21", "4.20", "4.19")
• $2 (keywords): Keywords to search in test names (e.g., "olmv1", "sig-storage", "operator")
• $3 (sippy-url) [optional]: Sippy instance base URL. Defaults to "sippy.dptools.openshift.org" if not provided. Examples: "qe-component-readiness.dptools.openshift.org"

Implementation

1. Parse Arguments

   • Extract version from $1 (e.g., "4.20")
   • Extract keywords from $2 (e.g., "olmv1")
   • Extract Sippy URL from $3 if provided, otherwise use default "sippy.dptools.openshift.org"
   • Normalize URL to extract base domain for API endpoint (strip "/sippy-ng/" suffix if present)
   • Add "https://" prefix if not already present

2. Build Sippy API Request

   • Construct filter JSON for the /api/tests endpoint:

     filters = {
         "items": [
             {
                 "columnField": "name",
                 "not": False,
                 "operatorValue": "contains",
                 "value": keywords
             }
         ],
         "linkOperator": "and"
     }
     

   • Set query parameters:
     • release: The OpenShift version
     • filter: JSON-encoded filter object
     • sort: "asc"
     • sortField: "current_pass_percentage"