I would walk away if corrections depend on manually repairing generated code one application at a time, if results cannot be reproduced consistently, or if the vendor treats every exception as additional services work. At portfolio scale, small exceptions become a boa constrictor. They accumulate slowly, then consume the schedule, the budget, and the business case.
I want an answer that defines functional equivalence in measurable terms and explains how it will be demonstrated across business outputs, numeric precision, interfaces, performance, error handling, and restart behavior. The vendor should describe how it builds the baseline, generates or expands test coverage, compares results, handles exceptions, and maintains traceability from the original system to the target.
This is where the demo ends and factory-scale delivery begins. I want clear answers on traceability, repeatability, model and prompt controls, human review, regression testing, exception management, retransformation, and ownership of the generated code. The vendor should also explain what happens when the source system changes during a multi-year program.
The biggest warning sign across all three questions is certainty without evidence. A modernization vendor should be able to show not only how it generates a result, but how it proves it, reproduces it, operates it, and stands behind it. The tools will keep improving. That discipline is what separates a modernization that holds up in production from one that quietly becomes a stabilization project.
2. “What evidence do you use beyond the source code to understand how the application operates in production?”
1. “How will you prove that the modernized system behaves like the existing production system?”
A credible answer includes runtime behavior, data characteristics, transaction and batch volumes, interfaces, scheduling, performance requirements, failure recovery, and operational procedures — and how those findings shape the modernization scope and target design.
3. “When the generated result is wrong, how will we detect it, correct it, and reproduce the correction across the rest of the portfolio?”
I would walk away if the approach is primarily code ingestion followed by AI-generated documentation and conversion. Static analysis can tell you what the application could do. It does not reliably tell you what it does in production, how often it does it, or under which data and operating conditions.
I would walk away if the answer is essentially “the generated code compiles,” “our AI is highly accurate,” or “your existing test suite will validate it.” Compilation is not validation, accuracy percentages are meaningless without a defined denominator, and legacy test suites are almost always incomplete.
