Position Overview:
The Test Engineering team is the quality assurance backbone of the CMS Medicare Systems Solutions (CMSS) portfolio, responsible for protecting the stability and integrity of FISS, MCS, DME, CWF, and Crosscutting programs through every software release. These systems process hundreds of millions of claims annually across 15+ million lines of legacy COBOL. The Supervisor, Test Engineering owns the end-to-end test strategy for this environment, with a primary mandate to ensure that every change, enhancement, and modernization effort is validated against the full depth of legacy system behavior before it reaches production.
In this role, the Supervisor applies an expert understanding of how the test engineering function integrates with development, architecture, operations, and business teams to protect system stability and support CMS's modernization objectives. The Supervisor identifies and solves technical and operational testing problems, particularly those arising at the intersection of legacy mainframe systems and modern integration layers. The Supervisor develops, coaches and directs test engineers and functional testers to adopt consistent regression and automation standards, adapting test plans and priorities to address resource and operational challenges, leading the test engineering team, and focusing on achieving quality, test coverage, and release readiness objectives.
Key Responsibilities:
Mainframe & Regression Testing Leadership
- Develop and implement a comprehensive testing strategy grounded in Agile/DevOps practices and automation-first principles.
- Own and continuously mature the regression testing strategy for COBOL-based mainframe systems, ensuring comprehensive coverage of all claims processing logic, edits, and business rules across every release cycle.
- Maintain and expand the regression test suite to reflect changes introduced by enhancements, new payment models, requirements updates, and modernization activities.
- Establish and enforce regression baseline standards: define what "passing" looks like for each system, document known behaviors, and ensure deviations are flagged, triaged, and resolved before production deployment.
- Lead mainframe-specific test planning for batch processing, JCL job streams, VSAM file handling, and claims adjudication logic.
- Coordinate regression test execution across all 12 Medicare Administrative Contractors (MACs) to ensure consistent validation of system behavior in distributed environments.
- Ensure regression test assets are version-controlled, traceable to requirements and business rules, and maintained as living documentation of system behavior.
Test Planning & Execution
- Create, maintain, and manage test execution plans spanning multiple testing phases, environments, and release cycles.
- Coordinate and oversee automated and manual testing activities across integrated teams and subcontractors.
- Ensure test environments, data sets, and automation assets support scalable, repeatable testing.
- Participate in requirements and user story reviews and design walk-throughs to ensure testability.
Systems Stability & Release Readiness
- Serve as the quality gate for every production release, including Go/No-Go recommendations grounded in complete, documented test evidence.
- Develop and maintain release-specific test plans that account for the full scope of changes, including interactions between legacy components and any modernized or integrated services.
- Identify and escalate systemic risks to system stability, including gaps in test coverage, untested code paths, and regression exposure introduced by modernization work.
- Lead defect triage and resolution analysis, ensuring defects are prioritized by business impact and that production readiness decisions are made with full visibility into open risk.
- Maintain production readiness checklists, test completion criteria, and sign-off documentation for each release.
Test Automation & Modernization Support
- Design and maintain automation frameworks that extend coverage into mainframe testing — including batch job validation, file comparison, output verification, and claims adjudication result checking.
- Support modernization initiatives by developing integration and regression test strategies that validate new components against legacy system behavior, ensuring functional equivalence and data integrity.
- Promote automation-first principles while maintaining the discipline to recognize where manual or exploratory testing is required in legacy environments.
- Identify, evaluate, and implement AI-enabled testing tools and optimization techniques to improve coverage efficiency and predictive quality insights.
- Ensure test automation assets are integrated into CI/CD pipelines where applicable, while maintaining compatibility with mainframe batch and release schedules.
Quality Management & Reporting
- Develop, maintain, and enforce Quality Management Plan (QMP) standards, processes, and test governance across the portfolio.
- Track and analyze test metrics, defect trends, regression pass/fail rates, and execution results; deliver clear dashboards and quality insights to program leadership.
- Identify, communicate, and mitigate risks, including business impacts related to defects, coverage gaps, or test strategy constraints.
- Collaborate with the Delivery Support Services team to ensure quality metrics are reflected accurately in program status reporting and deliverables.
Team & Stakeholder Leadership
- Supervises and assigns work for a team of test automation engineers and business analyst professionals (10–15 staff).
- Conduct performance monitoring and formal evaluations.
- Build partnerships with development, architecture, operations, and business teams to ensure aligned test coverage and timely issue resolution.
- Collaborate effectively across distributed teams and subcontractors within a fast-paced, multi-release environment.
- Foster a culture of quality, regression discipline, continuous improvement, and mission focus.
