Why this is hard to get right
The Scenario: When "Let's Score the Vendors" Becomes a Politics Problem
Mariana is a Director of Sales Operations at a 400-person B2B software company. Her team has been running on a patchwork CRM for three years — two different tools, inconsistent data, and reps who've built their own spreadsheet workarounds. Leadership finally approved a proper CRM consolidation and set a hard deadline: vendor selected and contract signed in six weeks.
Mariana owns the evaluation. She has buy-in from Sales, IT, Security, and Finance. What she doesn't have is a shared framework.
The first meeting was a disaster. Sales wanted ease of use. IT wanted API flexibility. Security wanted SOC 2 and data residency guarantees. Finance wanted the lowest cost. Every stakeholder had a different top criterion and no one had agreed on weights. The scoring sheet Mariana inherited from a prior vendor review didn't define what a "3" meant versus a "4." One evaluator gave a vendor top marks for UI. Another gave the same vendor a 1 because it lacked a native Salesforce connector.
She spent two weeks running demos and collecting scores that were statistically incomparable.
Mariana tried asking an AI assistant to "create a vendor scoring matrix for CRM selection." The output was generic: a five-column table with criteria like "functionality," "support," and "pricing." No weights. No definitions. No compliance checklist. No process steps. Nothing her executive team could use to justify a $110,000 annual decision.
The root problem wasn't the AI. It was the prompt. Mariana hadn't told the AI what business goals she was solving for, what constraints were non-negotiable, who the evaluators were, or what format the output needed to match. Without that context, the AI defaulted to the most average possible answer.
When she restructured her request — specifying the role, the cross-functional panel, the business goals (cost reduction and admin time savings), the compliance requirements (SOC 2, GDPR, SSO), the budget ceiling, and the one-page board-ready format — the AI returned a complete evaluation package: a weighted scoring matrix with criterion definitions, a compliance checklist, a three-week process timeline, a demo script, and tie-break rules.
Stakeholders stopped arguing about opinions. They scored against shared definitions. The evaluation wrapped in 18 days. The final recommendation fit on one slide.
That's what a structured vendor evaluation prompt actually does. It doesn't just generate a template — it forces you to align on the criteria that matter before a single vendor walks in the door.
Common mistakes to avoid
Skipping Criterion Definitions and Scales
Asking for a scoring matrix without defining what each score means produces a table that looks complete but functions inconsistently. Different evaluators interpret "3 out of 5" differently — one grades generously, another harshly. Always specify that each criterion needs a definition and that each score level (0–5 or 1–10) needs an anchor description to ensure comparable results across the panel.
Omitting Weights and Business Rationale
Requesting criteria without percentage weights forces stakeholders to treat all factors as equal — which almost never reflects reality. Budget, compliance, and integration typically outweigh UI preferences. Specify that weights must total 100% and each weight should tie back to a stated business goal. Without this, AI produces a flat list that invites post-hoc manipulation by whoever scores last.
Forgetting Compliance and Non-Negotiables
Generic vendor prompts rarely surface must-have requirements like SOC 2, GDPR, SSO, or data residency. If you don't list them, the AI won't include them — and you'll score a vendor highly before discovering it fails a legal requirement. Always state your compliance checklist explicitly so it appears as a hard gate before weighted scoring begins.
Leaving Out Stakeholder Roles and Process Steps
A scoring matrix is useless without a process for using it. If the prompt doesn't specify who scores, by when, and how conflicts get resolved, the AI returns a static document rather than an operational workflow. Include roles (Sales Ops, IT, Security, Finance), timeline milestones, and tie-break rules to get a complete evaluation package, not just a table.
Using Vague Business Context
Prompts that say 'pick the best vendor' without specifying company size, user count, integration requirements, or budget force the AI to generate average criteria for a generic company. Specific context produces specific criteria. A 250-user B2B SaaS firm evaluating a $120k CRM needs completely different scoring logic than a 5,000-seat enterprise doing a multi-region ERP rollout.
Not Specifying Output Format
Without a format instruction, AI often returns a wall of prose or an oversized table that's impossible to share with executives. Specify the deliverable format explicitly — a one-page summary, a board-ready slide structure, or a downloadable matrix — so the output matches how it will actually be used in your review process.
The transformation
Create a vendor scoring template for our RFP and help pick a winner.
You are a Procurement Operations lead. Create an RFP evaluation package for a mid-market B2B SaaS CRM selection. 1) Audience: cross-functional panel (Sales Ops, IT, Security, Finance). 2) Goals: reduce CRM cost by 15%, cut admin time by 30% in 6 months. 3) Deliverables: - **Scoring matrix** with 8–10 criteria, definitions, and 0–5 scale. - **Weights** totaling 100%, with rationale. - **Compliance checklist** (SOC 2, GDPR, SSO, data residency). - **Process**: roles, timeline (3 weeks), demo script, Q&A log. - **Tie-break rules** and risk register. 4) Constraints: budget cap $120k/year, 250 users, Salesforce integration, 99.9% SLA. 5) Tone: concise, board-ready. Provide a one-page summary.
Why this works
Role Anchors Output Depth
The After Prompt opens with 'You are a Procurement Operations lead,' which signals to the AI that the response requires operational rigor, not surface-level advice. This role framing raises the baseline quality of every deliverable — the scoring matrix, compliance checklist, and risk register all reflect procurement-grade thinking rather than a generic template.
Measurable Goals Drive Relevant Criteria
Stating 'reduce CRM cost by 15%, cut admin time by 30% in 6 months' gives the AI a benchmark for criterion relevance. Every scoring weight and deliverable can now be traced back to a business outcome. Without these numbers, the AI invents generic criteria; with them, it builds criteria that actually test whether a vendor can deliver the stated goals.
Numbered Deliverables Eliminate Gaps
The structured list — scoring matrix, weights, compliance checklist, process, tie-break rules, risk register — forces the AI to treat each artifact as a required output. Nothing gets omitted because ambiguity has been removed. Each numbered item maps to a real evaluation step, so the full package is operational, not decorative.
Hard Constraints Filter Before Scoring
Specifying '$120k/year budget cap, 250 users, Salesforce integration, 99.9% SLA' gives the AI the data it needs to build a compliance gate that runs before weighted scoring. Vendors that fail hard constraints get eliminated early, which protects the panel's time and prevents a low-cost outlier from skewing weighted totals.
Tone and Format Instructions Match Stakeholder Needs
'Concise, board-ready. Provide a one-page summary' tells the AI exactly how the output will be consumed. This prevents a data dump and produces executive-friendly output — the kind that can be attached to a Slack message or presented in a 10-minute leadership review without additional editing.
The framework behind the prompt
The Theory Behind Vendor Evaluation Frameworks
Vendor evaluation is a structured decision-making problem, and the research on structured decision-making is clear: explicit criteria set before evaluation produces more consistent and defensible decisions than intuitive or consensus-based approaches.
The foundational framework here is the Analytic Hierarchy Process (AHP), developed by Thomas Saaty in the 1970s. AHP decomposes a complex decision into a hierarchy of criteria, assigns numerical weights through pairwise comparisons, and produces a mathematically consistent ranking. Most corporate vendor evaluations don't run full AHP, but the core principle — separate criterion definition from scoring, and separate scoring from interpretation — is embedded in every well-designed RFP matrix.
The MCDM (Multi-Criteria Decision Making) family of methods adds another key insight: scoring validity depends on criterion independence. If two criteria measure the same thing (e.g., 'ease of use' and 'user interface quality'), they artificially inflate a vendor's score in that area. Good prompt construction forces you to audit criteria for overlap before scoring begins.
From a procurement standpoint, the CIPS (Chartered Institute of Procurement and Supply) framework distinguishes between qualifying criteria (must-have gates) and award criteria (weighted scoring). This mirrors the structure of the After Prompt on this page — compliance and budget constraints act as hard gates, while the 8–10 weighted criteria determine relative ranking among qualified vendors.
Behavioral research on group decision-making (notably work by Daniel Kahneman on System 1 vs. System 2 thinking) shows that panels default to narrative reasoning — whoever tells the best story in the demo wins. Structured scoring matrices force System 2 thinking by requiring evaluators to assess each criterion independently before comparing overall scores.
The implication for AI prompting is direct: the more your prompt encodes decision-science principles — criterion definitions, weight rationale, compliance gates, tie-break logic — the more the AI's output functions as a real decision tool rather than a formatted list. Vague prompts produce vague outputs because they don't encode the structure that makes evaluation rigorous.
Prompt variations
You are an IT Security Manager conducting a vendor risk assessment for a cloud identity provider (IdP) selection at a 1,200-employee financial services firm.
Audience: IT, Security, Compliance, and Legal.
Business goals: Consolidate identity management across 14 SaaS tools, achieve SAML/SCIM compliance by Q3, and reduce helpdesk password-reset tickets by 40%.
Deliver:
- A weighted scoring matrix (8 criteria) covering SAML 2.0 support, SCIM provisioning, MFA enforcement, audit logging depth, uptime SLA, vendor SOC 2 Type II status, pricing per seat, and implementation timeline.
- A compliance gate checklist: FedRAMP readiness, SOC 2 Type II, data residency in the US, and GDPR Article 28 DPA availability.
- A three-round evaluation process: RFI shortlist, technical demo, security review.
- Scoring definitions for each criterion on a 0–5 scale with anchor descriptions.
- A one-page risk register for the top two finalists.
Constraints: Budget under $18/user/year, must integrate with Azure AD and Okta workflows, 99.99% SLA required.
Tone: Technical and precise. Flag any criterion where vendor self-reporting is insufficient and third-party verification is required.
You are a Marketing Operations Manager evaluating three email service providers (ESPs) for a B2C e-commerce brand with 2.5 million subscribers.
Audience: Marketing Ops, CTO, and Finance.
Business goals: Improve deliverability rate from 91% to 96%, reduce cost per send by 20%, and enable behavioral segmentation within the CDP integration.
Deliver:
- A scoring matrix with 9 criteria: deliverability infrastructure, CDP connector quality, segmentation depth, A/B testing capabilities, dedicated IP availability, send volume limits, reporting granularity, support tier, and contract flexibility.
- Weights totaling 100%, with each weight tied to one of the three business goals above.
- A proof-of-concept (POC) test protocol: 30-day trial with defined success metrics for deliverability, open rate lift, and segment sync latency.
- A side-by-side comparison format for final stakeholder presentation.
- Decision criteria for choosing between two finalists if scores fall within 5 points of each other.
Constraints: Budget ceiling $8,000/month, must support Klaviyo migration path, and provide a dedicated deliverability consultant during onboarding.
Tone: Concise and data-driven. Frame outputs for a non-technical CMO audience.
You are a Head of Procurement running a competitive RFP for a payment processing partner at a Series B SaaS company processing $40M annually.
Audience: Finance, Legal, and the CEO.
Business goals: Reduce blended processing fees by 0.3%, cut chargeback rate below 0.5%, and support international expansion into the EU and APAC within 12 months.
Deliver:
- A scoring matrix with 10 criteria: blended fee structure, chargeback dispute tooling, international currency support, settlement timing, PCI DSS Level 1 certification, fraud detection accuracy, API documentation quality, enterprise SLA, contract term flexibility, and references from comparable SaaS companies.
- Weights totaling 100%, with Finance and Legal signing off on the final weighting rationale.
- A two-stage RFP process: initial bid submission and finalist oral presentation with a standardized Q&A script.
- A commercial terms comparison table covering fee structures, minimums, and exit clauses.
- A risk register covering concentration risk, regulatory exposure by region, and data portability requirements.
Constraints: Require PCI DSS Level 1, support for ACH, wire, and card, and a 72-hour settlement window as non-negotiable.
Tone: Formal and board-ready. All scoring rationale must be documented for audit purposes.
You are a small business owner selecting a project management tool for a 12-person creative agency.
Context: The team currently uses email and shared folders. You need a tool that works for client project tracking, internal task assignment, and basic time logging. Budget is $50/month maximum. No dedicated IT support.
Deliver:
- A simple scoring matrix with 6 criteria: ease of use for non-technical staff, client-facing features (portals or sharing), time tracking capability, mobile app quality, customer support responsiveness, and price.
- Weights totaling 100%, prioritizing ease of use above all else.
- A 2-week free-trial checklist with 5 specific tasks the team should test during the trial period.
- A final recommendation format: one paragraph per finalist with a clear go/no-go recommendation.
Constraints: Must work on Mac and iOS, no per-project pricing models, and must not require a minimum 12-month contract to start.
Tone: Plain language. Avoid technical jargon. Write as if explaining to someone with no procurement experience.
When to use this prompt
Marketing Operations
Evaluate email service providers using criteria for deliverability, integration with the CDP, and campaign automation depth.
Product Management
Compare analytics vendors with weighting for event tracking performance, privacy features, and cost per monthly tracked user.
Sales Leadership
Select a sales engagement platform using criteria for dialer quality, sequence analytics, and Salesforce sync reliability.
IT and Security
Assess identity providers with a compliance checklist (SAML, SCIM, MFA), uptime SLAs, and audit reporting depth.
Finance and Procurement
Run a competitive bid for payment processors focused on fees, chargeback tooling, and settlement timing.
Pro tips
- 1
Define hard constraints first to eliminate unqualified vendors early and protect the timeline.
- 2
Weight criteria by business impact to avoid overvaluing nice-to-haves; tie each weight to a goal.
- 3
Specify stakeholder roles and deadlines so scoring happens on time with clear ownership.
- 4
Include tie-break rules and a risk register to handle close scores and surface hidden tradeoffs.
Most scoring matrices fail not because the criteria are wrong, but because stakeholders didn't agree on the weights before scoring started. One advanced approach is to build a two-phase prompt sequence.
In the first prompt, ask the AI to generate a draft matrix with placeholder weights and a rationale column for each criterion. Send that to your stakeholder panel and ask each function to assign their preferred weight distribution independently. Average the results. This is a simplified version of the Analytic Hierarchy Process (AHP), a decision science method that converts pairwise comparisons into mathematically defensible weights.
In the second prompt, feed the agreed weights back to the AI and ask it to finalize the matrix, add scoring anchor definitions for each 0–5 level, and generate the compliance gate checklist and tie-break rules.
This two-phase approach solves a real organizational problem: weight negotiation during scoring is corrupted by anchoring bias. Once evaluators have seen vendor demos, their preferred weights shift to favor the vendor they liked. Locking weights before scoring begins protects the integrity of the process.
For high-stakes decisions (above $250k annual contract value or multi-year commitments), consider asking the AI to include a sensitivity analysis table — showing how the final ranking changes if the top criterion's weight shifts by plus or minus 10 percentage points. This surfaces fragile decisions before you commit.
Vendor evaluation criteria are not universal. The criteria that matter for a marketing technology platform differ substantially from those for a logistics provider or a professional services firm. Here's how to adapt your prompt by vendor category:
SaaS Software: Prioritize integration depth (API quality, native connectors), uptime SLA, data portability, and vendor financial stability. Include a compliance gate for SOC 2 Type II and GDPR Article 28.
Professional Services and Agencies: Replace technical criteria with delivery quality metrics — case study relevance, team seniority, project methodology (Agile vs. Waterfall), and client reference quality. Weight cultural fit and communication cadence more heavily than in product evaluations.
Infrastructure and Logistics: Focus on capacity, geographic coverage, SLA penalties for missed delivery windows, insurance and liability terms, and business continuity documentation. Compliance gates should cover ISO certifications and carrier licensing.
Staffing and Talent Vendors: Score on time-to-fill rates, quality of candidate screening, replacement guarantee terms, geographic reach, and sector specialization. Include a background check and compliance verification as a hard gate.
When you build your prompt, name the vendor category explicitly and ask the AI to surface category-specific criteria that generic frameworks miss. This single instruction dramatically improves the relevance of the output.
Before you submit your vendor evaluation prompt to an AI assistant, run through this pre-flight checklist:
Context and Goals
- Business goals are stated with specific, measurable targets (not 'improve efficiency')
- Company size, user count, and industry are included
- The vendor category and contract scope are named
Criteria and Weights
- You've specified how many criteria you want (8–10 is recommended)
- You've requested weights totaling 100% with rationale
- You've asked for scoring anchor definitions at each scale level
Compliance and Constraints
- Hard budget ceiling is stated as a non-negotiable gate
- Required certifications and integrations are listed explicitly
- SLA minimums are defined
Process and Stakeholders
- Evaluator roles are named (not just 'the team')
- Timeline milestones are included
- Tie-break rules are requested
Output Format
- You've specified the format (matrix, one-pager, slide-ready summary)
- You've indicated the audience for the final output (board, cross-functional team, single decision-maker)
- You've asked for a risk register or risk summary
If any of these are missing, the AI will fill in the gap with a generic assumption — usually the wrong one for your context.
When not to use this prompt
When This Prompt Pattern Is Not the Right Tool
This structured RFP scoring approach works well for high-stakes, multi-stakeholder vendor decisions. But it's not always the right fit.
Don't use it for:
- Low-cost, easily reversible tool decisions. If you're picking a $15/month Slack plugin or a free design tool for personal use, a formal weighted matrix adds process overhead with no proportional benefit. A simple pros-and-cons list is faster and sufficient.
- Emergency or sole-source procurement. When a vendor is chosen because no alternative exists (a critical incumbent, a patented solution, or a regulatory mandate), a scoring matrix creates the appearance of competition where none exists. This can create legal and compliance risk rather than reducing it.
- Early-stage exploration with no defined requirements. If your team hasn't aligned on what problem you're solving, running a vendor evaluation will produce a matrix that measures the wrong things. Use a requirements-gathering prompt first, then return to the evaluation framework once criteria are stable.
- Highly technical decisions requiring deep domain expertise. AI can generate a security vendor matrix, but it cannot assess whether a vendor's zero-trust architecture actually matches your threat model. Use the AI output as a starting scaffold, then have your domain expert validate and refine every criterion and anchor definition before using it with a live panel.
When in doubt, ask whether the decision is reversible and whether the cost of a wrong choice justifies the evaluation process overhead.
Troubleshooting
The scoring matrix has no definitions — just criterion names and a blank number scale
Add an explicit instruction for anchor descriptions. Append this to your prompt: 'For each criterion, provide a one-sentence definition and a description of what a score of 0, 2, and 5 looks like in practice.' Without anchor descriptions, evaluators interpret the scale differently and your scores become incomparable. Definitions are what make the matrix replicable across your panel.
The AI generated criteria that don't reflect my industry or tech stack
Name your stack and industry explicitly in the prompt. Add a line like: 'Our current environment includes Salesforce, Workday, and AWS. We operate in a HIPAA-regulated healthcare setting.' Generic criteria appear when the AI has no industry context to work from. The more specific your environment, the more specific the criteria — especially for integration and compliance requirements.
The output is a long document but lacks a one-page summary for leadership
Request the summary format explicitly and separately. Add to your prompt: 'Also provide a one-page executive summary with the top 3 criteria, final vendor rankings, and a one-sentence recommendation rationale per finalist.' AI tools don't default to executive brevity — you have to instruct it. If you already ran the prompt, ask in a follow-up: 'Condense the above into a board-ready one-page summary.'
Weights don't add up to 100% or don't connect to stated business goals
Require traceability in the prompt. Add: 'Each criterion weight must be justified by one of the stated business goals. Weights must total exactly 100%. Show your weighting rationale in a column next to each criterion.' Without this instruction, AI distributes weights evenly or arbitrarily. The traceability requirement forces logical alignment between criteria and outcomes.
The compliance checklist is missing or too shallow
List your specific compliance requirements by name. Replace vague instructions with explicit standards: 'Include a compliance gate for SOC 2 Type II, GDPR Article 28 DPA, SSO via SAML 2.0, and data residency within the EU.' Then add: 'Mark each compliance item as pass/fail with no partial scoring.' Generic prompts produce generic checklists. Naming the standards forces the AI to treat them as hard requirements.
How to measure success
How to Evaluate the Quality of Your AI-Generated Vendor Matrix
Before you share the output with your evaluation panel, check it against these quality signals:
Structure completeness:
- Does every criterion have a name, a definition, and a weight?
- Do weights sum to exactly 100%?
- Does each weight connect to a stated business goal?
- Is there a compliance/hard-gate section separate from the weighted criteria?
Scoring precision:
- Does each criterion include anchor descriptions for at least the minimum, midpoint, and maximum scores?
- Are the anchor descriptions concrete and observable — not vague adjectives like "good" or "excellent"?
Process completeness:
- Does the output include a timeline with named milestones?
- Are evaluator roles assigned to specific functions (not just 'the team')?
- Is there a tie-break rule for close scores?
Executive readiness:
- Can a stakeholder who wasn't in the evaluation understand the recommendation from the one-page summary?
- Does the risk register flag at least two specific risks per finalist?
Red flags to watch for:
- Criteria that overlap or measure the same thing
- Weights distributed evenly across all criteria (a sign the AI defaulted to a generic response)
- A compliance checklist that's empty or lists only generic items not specific to your industry
Now try it on something of your own
Reading about the framework is one thing. Watching it sharpen your own prompt is another — takes 90 seconds, no signup.
Build a complete RFP scoring matrix, compliance checklist, and evaluation timeline for your next vendor decision.
Try one of these
Frequently asked questions
Eight to ten criteria is the practical sweet spot for most cross-functional evaluations. Fewer than six criteria oversimplifies the decision; more than twelve creates scoring fatigue and dilutes weights to the point of meaninglessness. Group related factors (e.g., security and compliance into one weighted cluster) if your evaluation has more dimensions than the matrix can handle cleanly.
Tie every weight back to a stated business goal. If cost reduction is your primary objective, pricing-related criteria should carry the most weight. A useful rule: hold a brief stakeholder alignment session before scoring begins where each function nominates their top three criteria. The aggregate ranking then informs the initial weight distribution, which is easier to defend than numbers assigned by one person.
Yes. The same framework applies to logistics providers, consulting firms, marketing agencies, and facilities vendors. Swap the compliance checklist for relevant certifications (ISO 9001, GDPR, or industry-specific), adjust the criteria to reflect service-based metrics like turnaround time or account management quality, and the core structure holds. The key is always to specify constraints, goals, and stakeholder roles upfront.
Build tie-break rules before you start scoring — not after. Good tie-break options include: weighting the single highest-priority criterion more heavily, requiring a second demo focused on the contested area, or running a short paid proof-of-concept. Ask the AI to include a formal tie-break protocol in the matrix so it's agreed on before results create stakeholder pressure.
Add industry-specific context directly to the prompt. Name your compliance requirements (HIPAA, SOC 2, FedRAMP), your integration stack (Salesforce, SAP, NetSuite), and your user profile (sales reps, security analysts, finance teams). Generic output almost always traces back to a generic prompt — the more specific your constraints and goals, the more specific the criteria the AI generates.
Remove the multi-stakeholder role assignments and replace the panel scoring columns with a single-evaluator scorecard plus a structured rationale field for each criterion. Ask the AI to add a 'Decision Log' section where the evaluator records why each score was assigned. This creates auditability even when only one person runs the evaluation — important for procurement compliance in many organizations.
Never share weightings before scoring — vendors will optimize demos around your highest-weighted criteria, which distorts the evaluation. Sharing broad evaluation categories (functionality, security, support, pricing) is standard practice and expected. The specific weights and scoring definitions should remain internal until after you've made a shortlist decision and begun final negotiations.
Treat budget as a hard gate, not a weighted criterion. If a vendor exceeds your budget cap, it should be eliminated before the scoring matrix runs — not penalized within the matrix. Ask the AI to separate compliance gates (must-pass filters) from scored criteria in the matrix structure. This keeps the weighted scoring clean and defensible.