Things to consider
- classifying something as a model carries with it important consequences related to cost and productivity.
- procedures can be both expensive for the institution and onerous for the model owner.
- Model validations pull model owners away from their everyday work, adversely affecting productivity and, sometimes, quality of work.
- Too many unnecessary reports containing findings that are comparatively unimportant can bury enterprise risk managers and distract them from the most urgent findings.
Definition of a model
A quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates
- Models as have three components
- An information input component
- A processing component,
- A reporting component,
- Quantitative estimate implies a level of uncertainty about the outputs.
On Spreadsheet models
- Probably two types:
- a model that transforms inputs into quantitative estimates or
- a non-model spreadsheet that generates defined arithmetic results.
- Ask oneself if back-testing required to gauge the accuracy of the spreadsheet’s outputs
- Ask if the spreadsheet is simply applying a defined set of business rules (then not a model)
- Spreadsheets should be classified as models (and validated as such) when they produce forward-looking estimates that can be back-tested.
On vendor models
- Vendor documentation is not a substitute for model documentation:
- Documentation should cover:
- Discussion of the model’s purpose and specific application,
- Discussion of model theory and approach,
- Description of the model’s structure
- Identification of model limitations and weaknesses
- Comprehensive list of inputs and assumptions, including their sources
- Comprehensive list of outputs and reports
- List Model settings
- Description of testing
- Testing results should be requested of the vendor: (per OCC 2011-12) including performance monitoring and outcomes analysis,
- Model validators should attempt to replicate the results of these studies, where feasible
- Developmental evidence should be requested of the vendor: (per OCC 2011-12)
- Banks should have contingency plans for the vendor model
- Implementation of reliable challenger model (for contingency)
On general model documentation
- Model validations frequently seem to occur at the most inopportune moments for model owners.
- Documentation would ideally be prepared during periods of lower operational stress.
- Documentation follows these basic criteria:
- Identifies the model’s purpose,
- Comprehensively lists and justifies the model’s inputs and assumptions
- Describes the model’s overall theory and approach,
- Lays out the developmental evidence supporting choices
- Identifies the limitations of the model
- Explains how the model is controlled—who
- Comprehensively identifies and describes the model’s outputs, how they are used, and how they are tested
On Model validation
- Comprehensive model validations consist of three main components:
- Conceptual soundness
- Monitoring and benchmarking,
- Outcomes analysis and back-testing.
- Performance testing is the core of any model validation
- Benchmarking
- Back-testing
- Sensitivity analysis
- Stress testing
- Validate model inputs based on:
- Complexity of inputs
- Manual manipulation of inputs from source system prior to input into model
- Reliability of source system
- Relative importance of the input to the model’s outputs (i.e., sensitivity)
On Benchmarking and Back-testing
- Model developers as best practice should always list any alternative methodologies, theories, or data which were omitted from the model’s final version.
- Back-testing measures a model’s outcome and accuracy against real-world observations,
- Benchmarking measures those outcomes against those of other models or metrics.
- Back-testing and Benchmarking should ideally be performed together