Hello Courtney,
I would agree more with your second point – the AI model should be evaluated for inherent risk. Since AI is still relatively new and evolving, it's important to recognize that the models used can be flawed with biased and/or incorrect data, or intentionally manipulated. For example, there's been evidence of some organizations using biased algorithms for their hiring systems. Their models were trained on resumes that were mostly submitted by men, so the algorithm was unintentionally biased against women. Validating this model may not have shown any errors since the data it was drawing from was technically correct, but biased.
However, AI risk can expand beyond model validation and include other risk areas like privacy, cybersecurity, ethics, and even business continuity and compliance. Before you perform due diligence, it may help to collaborate with a qualified subject matter expert and consider some questions to ask the vendor who's providing the AI model. You might ask questions such as where the data is sourced from, how often the system is updated with additional training data, and how they're protecting the integrity of their system.
I hope this can give you a good starting point and I'm eager to know how others are dealing with evaluating AI models, as this can be a very complex issue.