Assessing the Credibility of Computational Modeling and Simulation in Medical Device Submissions
This guidance provides a general risk-informed framework for assessing the credibility of computational modeling and simulation (CM&S) used in medical device regulatory submissions. It applies specifically to first principles-based models (e.g., physics-based or mechanistic models) but not to standalone statistical or data-driven models like machine learning or AI models. The guidance aims to help improve consistency and transparency in reviewing CM&S, increase confidence in using CM&S in regulatory submissions, and facilitate improved interpretation of CM&S credibility evidence.
Recommended Actions
- Define question of interest and context of use for the computational model
- Perform model risk assessment considering decision consequence and model influence
- Identify and categorize planned credibility evidence across verification, validation and other categories
- Define credibility factors and set goals commensurate with model risk
- Generate credibility evidence through planned studies and/or analysis of existing data
- Perform adequacy assessment to determine if evidence supports model use
- Document limitations and rationale for model credibility
- Prepare comprehensive credibility assessment report following recommended format
- Consider submitting Q-submission to get FDA feedback on credibility assessment plan
- Include both model details report and credibility assessment report in regulatory submission
Key Considerations
Clinical testing
- In vivo validation results can be used as credibility evidence, including clinical or animal data
- For patient-specific models, subject-level comparison between simulation and clinical data should be performed
- Population-based validation comparing model predictions to clinical datasets at a population level can be used
- Clinical trial results can provide validation evidence if appropriate statistical techniques and regulatory requirements are followed
Non-clinical testing
- Bench test validation under well-controlled laboratory conditions is recommended when possible
- Both prospectively planned validation and validation against retrospective datasets can be used
- Previously generated validation results may be applicable if relevance to current context of use can be demonstrated
- Calculation verification and uncertainty quantification should support validation results
Software
- Code verification must demonstrate correct implementation of numerical algorithms
- Software quality assurance and numerical code verification should be performed
- For models in device software, both software and model verification/validation are important but differ in scope
Labelling
- Context of use should be clearly described
- Model limitations should be explicitly stated
- Any assumptions should be documented and justified
Safety
- Model risk assessment should consider:
- Decision consequence (potential severity and probability of harm)
- Model influence relative to other evidence
- Risk mitigation procedures should be documented
- Safety thresholds and model predictions relative to these should be evaluated
Other considerations
- Model calibration evidence can support credibility but is weaker than validation
- Emergent model behavior can provide supporting evidence
- Model plausibility rationale should justify governing equations, assumptions and parameters
- Multiple categories of credibility evidence are recommended when possible
- Adequacy assessment should evaluate if evidence is sufficient given the model risk
Relevant Guidances
- Reporting of Computational Modeling and Simulation Studies in Medical Device Submissions
- Requests for Feedback and Meetings for Medical Device Submissions: The Q-Submission Program
- Technical Considerations for Physiologic Closed-Loop Control Medical Devices
Related references and norms
- ASME V&V 40: Assessing Credibility of Computational Modeling through Verification and Validation: Application to Medical Devices
- ISO 14971: Medical devices — Application of risk management to medical devices
- ISO/TR 24971: Medical devices — Guidance on the application of ISO 14971
Original guidance
This post is licensed under CC BY 4.0 by the author.