Assessing the Credibility of Computational Modeling and Simulation in Medical Device Submissions
This guidance provides a general risk-informed framework for assessing the credibility of computational modeling and simulation (CM&S) used in medical device regulatory submissions. It applies specifically to first principles-based models (e.g., physics-based or mechanistic models) but not to standalone statistical or data-driven models like machine learning or AI models. The guidance aims to help improve consistency and transparency in reviewing CM&S, increase confidence in using CM&S in regulatory submissions, and facilitate improved interpretation of CM&S credibility evidence.
What You Need to Know? 👇
What is the scope of FDA’s guidance on computational modeling credibility assessment?
This guidance applies to first principles-based models (physics-based or mechanistic) used in medical device submissions, including electromagnetics, fluid dynamics, and physiological models. It does not cover standalone statistical, machine learning, or AI-based models.
How does model risk assessment work in computational modeling submissions?
Model risk combines two factors: decision consequence (potential patient harm from incorrect decisions) and model influence (how much the model contributes versus other evidence). Higher risk typically requires more rigorous credibility evidence and validation activities.
What are the eight categories of credibility evidence for computational models?
The categories include: code verification results, model calibration evidence, bench test validation, in vivo validation, population-based validation, emergent model behavior, model plausibility evidence, and calculation verification/uncertainty quantification using context-of-use simulations.
When should manufacturers use the Q-submission process for computational modeling?
FDA recommends Q-submissions when planning credibility assessment activities, seeking feedback on model risk assessment, or using alternative approaches to the framework. This helps ensure alignment with FDA expectations before conducting expensive validation studies.
What is the difference between verification and validation in computational modeling?
Verification confirms the software correctly implements mathematical equations (code verification) and estimates numerical errors (calculation verification). Validation compares model predictions against real-world experimental data to assess accuracy for the intended use.
How does the adequacy assessment determine if credibility evidence is sufficient?
Adequacy assessment evaluates whether collected evidence supports using the model for its context of use, considering model risk. It includes prospective planning assessment and post-study evaluation, incorporating credibility goals achievement and proximity to safety thresholds.
What You Need to Do 👇
Recommended Actions
- Define question of interest and context of use for the computational model
- Perform model risk assessment considering decision consequence and model influence
- Identify and categorize planned credibility evidence across verification, validation and other categories
- Define credibility factors and set goals commensurate with model risk
- Generate credibility evidence through planned studies and/or analysis of existing data
- Perform adequacy assessment to determine if evidence supports model use
- Document limitations and rationale for model credibility
- Prepare comprehensive credibility assessment report following recommended format
- Consider submitting Q-submission to get FDA feedback on credibility assessment plan
- Include both model details report and credibility assessment report in regulatory submission
Key Considerations
Clinical testing
- In vivo validation results can be used as credibility evidence, including clinical or animal data
- For patient-specific models, subject-level comparison between simulation and clinical data should be performed
- Population-based validation comparing model predictions to clinical datasets at a population level can be used
- Clinical trial results can provide validation evidence if appropriate statistical techniques and regulatory requirements are followed
Non-clinical testing
- Bench test validation under well-controlled laboratory conditions is recommended when possible
- Both prospectively planned validation and validation against retrospective datasets can be used
- Previously generated validation results may be applicable if relevance to current context of use can be demonstrated
- Calculation verification and uncertainty quantification should support validation results
Software
- Code verification must demonstrate correct implementation of numerical algorithms
- Software quality assurance and numerical code verification should be performed
- For models in device software, both software and model verification/validation are important but differ in scope
Labelling
- Context of use should be clearly described
- Model limitations should be explicitly stated
- Any assumptions should be documented and justified
Safety
- Model risk assessment should consider:
- Decision consequence (potential severity and probability of harm)
- Model influence relative to other evidence
- Risk mitigation procedures should be documented
- Safety thresholds and model predictions relative to these should be evaluated
Other considerations
- Model calibration evidence can support credibility but is weaker than validation
- Emergent model behavior can provide supporting evidence
- Model plausibility rationale should justify governing equations, assumptions and parameters
- Multiple categories of credibility evidence are recommended when possible
- Adequacy assessment should evaluate if evidence is sufficient given the model risk
Relevant Guidances 🔗
- Reporting of Computational Modeling and Simulation Studies in Medical Device Submissions
- Q-Submission Program: Strategic Framework for FDA Interactions in Medical Device Development
- Technical Considerations for Physiologic Closed-Loop Control Medical Devices
Related references and norms 📂
- ASME V&V 40: Assessing Credibility of Computational Modeling through Verification and Validation: Application to Medical Devices
- ISO 14971: Medical devices — Application of risk management to medical devices
- ISO/TR 24971: Medical devices — Guidance on the application of ISO 14971