What Is a Vendor Evaluation Study?
What Is a Vendor Evaluation Study?
A vendor evaluation study selects technology vendors against your written requirements, using a weighted scoring model applied identically to every candidate — and documents the reasoning so the decision survives scrutiny later.
At DAG Tech, the vendor evaluation study is a CxO™ Virtual CTO deliverable. We are not a reseller for the platforms we evaluate, which is the only thing that makes an evaluation worth commissioning.
Why Vendor Selection Goes Wrong
Most selections are decided before the evaluation starts, and the evaluation is built to justify it:
- The demo decides it. Vendors demonstrate strengths. Nobody demonstrates their weaknesses, so the differentiator becomes presentation quality.
- Criteria are written after seeing the products. The requirements end up describing one vendor’s feature list.
- Only licence cost is compared. Implementation, migration, training, integration and exit costs are where the real difference sits.
- References are supplied by the vendor. Curated references tell you the product can work somewhere, not that it will work here.
- Exit is never considered. Nobody asks how data comes back out until the relationship has soured.
How DAG Tech Runs a Vendor Evaluation Study
1. Fix the Criteria Before Looking at Vendors
Objective: Prevent the requirements from being shaped by the options.
- Carry criteria forward from the technology requirements analysis where one exists.
- Weight each criterion with leadership, and lock the weights before scoring begins.
- Separate must-have from scored criteria — a missing must-have eliminates rather than costs points.
- Define how each criterion will be evidenced, so scoring is not impression-based.
2. Build the Longlist and Screen It
Objective: Consider the market, then narrow defensibly.
The longlist includes vendors nobody in the room has heard of, because market familiarity is not a selection criterion. Screening against must-haves reduces it to a shortlist, and the study records which vendors were excluded and on which specific criterion.
3. Score Against Identical Criteria
Objective: Produce a comparison, not a set of impressions.
- Structured demonstrations driven by your scenarios, not the vendor’s script.
- The same questions to every vendor, with answers recorded verbatim.
- Technical validation of integration claims against your actual environment.
- Scores recorded per criterion with the evidence noted, not a single overall impression.
4. Compare Total Cost, Not Licence Cost
Objective: Show the multi-year commitment.
Licensing, implementation, data migration, integration effort, training, ongoing administration, support tiers, and the cost of leaving. Modelled across three to five years, because the cheapest first year is frequently the most expensive third.
5. Assess Risk and Verify References
Objective: Test what the sales process does not cover.
Vendor viability and ownership, roadmap credibility, lock-in and data portability, security posture and certifications, support quality at your size and in your timezone, and reference conversations with customers we identify rather than only those the vendor offers.
What the Deliverable Actually Looks Like
The study is the document that justifies a multi-year commitment, so its structure matters as much as its conclusion. Ours runs in eight sections.
Sections 1–3: Purpose, method and criteria
What is being selected and the decision it supports, the scoring model and how bias was controlled, then the criteria and weights — published up front so the ranking can be challenged on its weighting rather than its conclusion.
Sections 4–7: Longlist, scoring, cost and risk
Who was considered and why some were excluded, then the scored comparison. Every vendor entry carries the same fields:
- Vendor and product — with the specific edition or tier evaluated.
- Must-have compliance — met or not met, per requirement.
- Score per weighted criterion — with the evidence that produced it.
- Three to five year total cost — broken out, not a single figure.
- Implementation effort and timeline — including your internal effort.
- Lock-in and exit assessment — how data comes back out, and at what cost.
- Reference findings — including from customers the vendor did not nominate.
- Risks — viability, roadmap, support and security.
Section 8: Recommendation and decision page
The recommended vendor, the runner-up, and the specific conditions under which the runner-up would be the better choice. That last part matters: if circumstances change, the study tells you whether the decision changes with them.
Where the Vendor Evaluation Study Fits
The vendor evaluation study is a CxO™ Virtual CTO deliverable that follows the requirements analysis and is tested against the enterprise technology architecture, since a vendor that violates your architectural principles creates cost that never appears in its pricing.
A vendor decision you cannot explain in writing two years later was not a decision. It was a preference that won.
Contact DAG Tech today to scope a vendor evaluation study.
Related reading: What Is an IT E-Learning Program? · What Is an End-user Technology Policy? · What Is an Organization Technology Structure?
