How to Choose Useful Decision Criteria

Audit every criterion for independence, observability, and the ability to change the decision before assigning a weight.

Audit the criteria list

Purpose
Write the successful outcome in one sentence.
Gates
List non-negotiable conditions and their evidence.
Candidate criteria
List consequences, not product features.
Overlap test
Compare every pair and merge duplicates.
Anchors
Define observable low, middle, and high performance.

The test for whether a criterion belongs

Keep a criterion only if it describes a distinct consequence and a realistic difference could change the choice.

Test the criteria in a matrix

Begin with the outcome

Complete the sentence: this decision succeeds if... Then list the outcomes that must be true, not the features each option happens to advertise.

Criteria should describe consequences such as reliable task completion, predictable monthly cost, or access within a required time. This makes unlike options easier to compare.

Write each criterion as an outcome, not a product label or vague quality. Easy to use becomes time to complete the regular task with the required support. Good service becomes response time and resolution path for a defined problem. This forces the comparison to state what success means and gives both options a fair chance to provide evidence.

Separate gates from scores

A gate is pass or fail: legal, safe, compatible, affordable inside a hard ceiling, or available by the deadline. Do not let excellent performance elsewhere compensate for failing a genuine gate.

Preferences belong in the scored comparison only after both options pass the gates.

For every proposed gate, ask what failure would require you to do. A true gate removes the option or requires a change before it can proceed. A strong preference merely lowers the score. Record how the gate will be verified and do not use an average to pass it. This prevents an attractive bundle of benefits from compensating for one unacceptable condition.

After the list is independent and measurable, the weighted decision matrix guide shows how to assign weights without moving the goalposts.

Remove overlap

Test every pair of criteria. If a high score on one almost guarantees a high score on another, combine them or define the boundary more clearly.

For example, ease of use and onboarding time may overlap, while daily workflow friction and administrator setup time may affect different people and remain separate.

Use the remove-one test. Temporarily delete a criterion and ask whether another remaining score already captures the same consequence. Also check whether one cause appears through several labels, such as speed, productivity, and time saved. Merge overlap or narrow each definition. Independent criteria make the weight total easier to interpret and harder to manipulate.

Make the scale observable

Define what low, middle, and high mean using evidence you can collect. Avoid labels such as good fit unless you can name the behavior or result that demonstrates fit.

Define at least three observable points on the scale before rating either option. The middle should describe acceptable performance, not an unexplained average feeling. Use the same anchors, period, and evidence for both choices. If the available evidence cannot place an option on the scale, mark it unknown and decide whether that missing score can change the result.

To decide whether an input belongs in a score, range, or scenario, review the evidence types in the comparison methodology.

Handle criteria owned by different stakeholders

Mark who experiences each consequence and who has authority to accept it. A manager may value reporting while frontline users bear workflow friction. Combining both into a generic usability score hides the distribution of costs and benefits.

Ask affected people for observable examples before finalizing the list. Keep separate criteria when the consequences fall on different groups or occur at different times, even if they sound related. Document the final definitions so later scoring discussions do not reopen the meaning of the criteria.

Criteria that make a matrix look stronger than it is

Starting from a vendor feature list
Counting the same benefit under several names
Using vague criteria that cannot be scored consistently

Choosing criteria without counting one benefit twice

A small team comparing two scheduling systems starts with 14 desired features. It reduces them to data export, client booking completion, administrator time, accessibility, and annual cost. Mobile access is removed as a separate criterion because it is already part of booking completion.

When different stakeholders own the consequences

When several people own different consequences, agree on definitions before weighting. A shared label can hide conflicting meanings.