We are a credit union and created a position for a Vendor Manager. It's basically a department of one until we get it completely built. Then we will decide if we need to add staff.
For Due Diligence, I work with our ERM to review financial related risk questions (He is also an accountant), and our AVP of Information Security on the IT and Cloud related questions. Our Compliance Officer reviews/edits any new contracts we engage in.