How to examine evidence
Learn to check a source, compare outcomes and recognise what the data do not yet allow us to conclude.
What each source allows us to claim
Law or regulation
What it allows us to examineWhat rule was established, for whom and from when.
The limitThe wording of a rule does not demonstrate that it was followed.
An institution's webpage
What it allows us to examineHow the institution describes a programme, its actions and its records.
The limitThe description alone does not prove that the programme caused the reported outcomes.
Impact evaluation
What it allows us to examineHow much of a change can be attributed to the intervention, according to the method used.
The limitThe conclusion depends on the quality of the data, the comparison and the context. It does not guarantee the same effect in another country, period or at another scale.
Simulation
What it allows us to examineWhat would happen within the assumptions and relationships adopted.
The limitThe calculation does not turn its assumptions into facts. Changing an assumption may change the conclusion.
A success story helps formulate a hypothesis. To decide, we also need to know about failures, alternatives and implementation conditions. Look for the full document and the passage supporting the claim, not just its title or a news story about it.
The outcome changed. Was it because of the policy?
An announcement presents an intention. A record may demonstrate delivery. Completing construction work does not automatically demonstrate that it solved the problem. An indicator improving after a policy does not, by itself, demonstrate that the policy caused it.
- Baseline
- The situation measured before the change. State the definition, population, period and source so that comparisons can be made later.
- Comparison
- Ask what would probably have happened without the intervention. This situation is called the counterfactual. The method needs to justify the comparison; selecting just any other municipality or group is not enough.
- Other explanations
- Demand for the service, the composition of the population served and other measures may change during the same period. The evaluation should examine competing explanations.
An appropriate comparison depends on the question and the available data. Monitoring targets helps identify changes, but attributing an effect to the policy requires further investigation. When that attribution is not possible, the observed outcome remains useful, with the limitation stated.
Who was counted and who was left out?
A total may hide the proportion served. A proportion may hide differences between people. Before comparing, examine three points:
- Denominator
- The total used as the basis of a proportion. Check who could have been counted, who was excluded and how cancellations or missing records were treated.
- The same measure
- Keep the definition, unit, period and criteria comparable. If the measurement changes, record the change; a difference in the number may come from the method.
- Distribution
- Examine locations, groups and more difficult situations. Overall improvement may coexist with worsening conditions for some people.
Hypothetical example: follow two groups of requests for the same period, using the same definition of completion and no exclusions. At the end, each service completed 80 requests from its group. The numbers serve only to explain the proportion.
Service A
Group followed80 requests completed out of a total of 100.
Proportion completed80 ÷ 100 = 80%.
Service B
Group followed80 requests completed out of a total of 200.
Proportion completed80 ÷ 200 = 40%.
The number completed is the same; the proportion is different. This still does not tell us which service performs better: complexity, resources, quality and response time also matter. People still waiting must remain visible.
Which uncertainty could still change the conclusion?
- Missing information
- State what was searched for, in which sources and up to what date. Failing to find evidence of delivery does not, by itself, prove inactivity or fraud.
- An imprecise result
- An evaluation may be consistent with effects of different sizes. An inconclusive result demonstrates neither that the policy worked nor that it had no effect.
- Another context
- Before transferring an experience, compare the population, institutions, capacity, costs and scale. Knowing about unfavourable cases helps identify when the mechanism fails.
A useful conclusion states what was observed, which interpretation is supported and what remains open. The strength of the conclusion should match the strength of the evidence.
How evidence helps correct the course
Evaluation also helps decide whether to maintain, expand, correct or stop an intervention. Examine costs, quality, effects on different groups and unintended consequences. A single target does not settle all these choices.
- Investigate the failure
- Distinguish a measurement problem, an implementation difficulty, a change in context and a design failure. Each cause may call for a different response.
- Compare the next decision
- Keeping things as they are also has consequences. Compare alternatives, the costs of change and the conditions needed to maintain the service.
- Justify and record
- In the proposed model, corrections or interruptions would require identified responsible parties, justification, an opportunity to respond and protection of service continuity. The institutional rules are still under study.
The references guide the analysis and do not prove OpenGov3's feasibility. The distinctions concerning records, responsibilities and review are also developed in sections 3 to 6 of whitepaper 2.0.
This topic in the model
In the proposed model, assessing a project requires distinguishing delivery, observed change and an effect attributable to the intervention. These questions make it possible to challenge conclusions and decide what needs correcting.