Scoring method

Principles

Inputs

The assessment has a level (facility, ward, LGA or state), a direction (improve weak performance, or learn from strong performance), a reporting period (month, quarter, year, campaign round or custom dates) with a target convention (annual target, target for the period only, or a period-specific target), the target population each antigen is measured against, the places, and any disagreements the team kept.

Each place has a level (state, LGA, ward, facility or settlement), the place it sits inside, annual target populations (live births, surviving infants, children in the second year of life, girls), dose counts by antigen (or coverage percentages with their source), whether coverage above 100% was reviewed, the programme answers below, the findings (barriers and enablers, each mapped to a CFIR 2.0 construct, with who said it) and the Learning Log rows (issue, likely cause, construct, evidence status, who said it, date, and whether the row is about a barrier or an enabler).

Yes and no answers can also be Not known or left blank. A number is read with commas removed; a negative number counts as not entered. An answer counts as recorded unless it is blank, not applicable, or an empty list.

The period and its fraction of a year

The target convention is the one recorded, or by default: target for the period only for a month, a quarter or custom dates; period-specific for a campaign round; annual for a year.

The period fraction f is days ÷ 365.25 for custom dates (days = end − start + 1), and months ÷ 12 otherwise (a month 1, a quarter 3, a year 12). For a campaign round, or custom dates without dates, f is not known.

The figures are a share of an annual target when the convention is annual and the period is not a full year (shorter than 365 days or longer than 366 days, a month, a quarter, a round, or custom dates not entered).

The reference date is the period’s end date, or else the date of the observation. The reference year is the year of the end date, or of the start date, or a four-digit year in the period label, or the year of the observation.

Coverage, pace, dropout and children without Penta3

Places inside places

A parent with figures of its own is scored on them. A parent without figures is pooled from the places inside it that have Penta1 doses; when some of them have a target population, only those are pooled. Places on a data-quality hold, with percentages only, or without Penta1 doses are left out and named. When the places inside are of two kinds (for example a ward’s facilities and settlements), only one kind is pooled, facilities first.

In a pooled parent’s coverage, a place whose coverage above 100% was reviewed counts at no more than its target (for a share of an annual target, the annual target × f); the dose counts stay as entered.

A pooled parent with no programme answers of its own takes the system gap, access and equity, ease of acting and the answer-based data-confidence penalties of the places it was pooled from: their mean, weighted by each place’s annual surviving-infant target, or 1 when it has none, applied level by level [design choice].

When nothing can be pooled and the parent has no figures or programme answers of its own, it stands in for the highest-placed place inside it (see "How places are ordered") and takes its status and IPI.

System gap (service-delivery capacity)

The system gap (sg) is the sum of the points below, capped at 100. A blank or Not known answer adds no points. These are programme signals, not CFIR determinants. All point values are design choices.

The last two rows follow two of the five components of the Reaching Every District approach, "Linking services with communities" and "Monitoring and use of data for action" (WHO, Reaching Every District (RED), https://www.who.int/teams/immunization-vaccines-and-biologicals/essential-programme-on-immunization/implementation/reaching-every-district-(red)).

Access and equity (populations at risk of being missed)

Access and equity (E) = the larger of 0.60 (severely hard-to-reach), 0.35 (hard-to-reach) and 0.35 (riverine or island catchment), whichever apply, plus 0.30 for a nomadic population, 0.25 for an urban slum, 0.25 for an access or security problem and 0.25 for an IDP camp or host community; at most 1. All values are design choices.

Data-quality hold and safety

Status rules, applied in this order

A condition on a figure that is not available is false. The first status whose rule holds is the place’s status.

Where the cut-offs come from

Implementation Priority Index

IPI = 100 × (0.35 G + 0.35 S + 0.15 R + 0.15 E) ÷ (the sum of the weights of the parts available), rounded half up to a whole number. G is not available when p1, dO and m1 are all unavailable; S is not available when U is not known; R and E are always available. A place on a data-quality hold has no IPI.

The aggregation is a linear weighted mean, with weights that express trade-offs between the parts (OECD and the European Commission Joint Research Centre, Handbook on Constructing Composite Indicators, 2008, p. 34). The weights, and the equal weight of a rate (G) and a number of children (S), are design choices.

Nothing else enters the IPI: no counts of barriers, findings or signals, no ease of acting and no data confidence.

How places are ordered

Inside a group (the same parent and the same kind of place; the places at the top level form one group), places are ordered by status (Urgent, Very high, High, Moderate, Sustain / learn), then IPI (whole number, highest first), then U (highest first), then access and equity (highest first), then the order in which they appear in the assessment. Places on a data-quality hold are listed apart and not ranked.

In lists that mix levels (the places to act on first on the home screen, and in the brief), every scored place is included except places standing in for the places inside them, places on a hold, and parents that are Urgent only because a place inside them is Urgent. They are ordered by status, then U (highest first), then IPI (highest first), then the order in the assessment.

Shown beside the IPI

Ease of acting = 0.45 × operational + 0.30 × financial + 0.25 × political [design choice]. Operational starts at 1 and loses 0.25 when cold chain equipment is not working (0.10 when that is not recorded as working), 0.35 for an access or security problem, and up to 0.30 in proportion to the share of settlements over 5 km away; it is kept between 0.15 and 1. Financial is 0.40 with no operational funding, 0.70 with partial funding, 1 with funding, otherwise 0.70. Political is 0.50 with low support from local leaders, 0.75 with medium, 1 with high, otherwise 0.75. Bands: easier (0.70 or more), moderate (0.40 to under 0.70), harder (under 0.40). Ease of acting describes the place’s conditions; whether a chosen action is feasible is recorded for each action and judged again at review.

Data confidence = 0.40 + 0.20 for each of three items that is available (a target for Penta1 for the period, a Penta1 figure, a Penta3 figure), less the penalties below, kept between 0 and 1 [design choice]. A missing target simply scores 0 on its item.

Verify the figures before acting is shown, with its reasons, when any of these hold: data confidence below 0.40; Penta1, Penta3 or MCV1 above 100% reviewed, here or in a place pooled into it; a later dose above an earlier one; reported and survey Penta3 coverage 15 or more points apart; another antigen above 100%.

Sensitivity of the order

Status does not depend on the weights, so these checks concern the order within a status. They are run for the main comparison: the top level when it has two or more ranked places, otherwise the first group with two or more.

Weights: 1,000 sets of weights (G, S, R, E) are drawn from a Dirichlet distribution with parameters 20 × (0.35, 0.35, 0.15, 0.15) [design choice]. The random numbers come from the mulberry32 generator, seeded with the 32-bit FNV-1a hash of the assessment’s identifier, the parent’s identifier (or "top") and the level, joined with "|", so the result repeats. Gamma variates are drawn by the method of Marsaglia and Tsang, each normal variate as √(−2 ln(1 − u1)) × cos(2π u2) from the next two uniform numbers, and the four gamma variates of each draw are taken in the order G, S, R, E and divided by their sum. In each draw the places are ordered again as above, with the unrounded weighted score in place of the IPI. The app shows how often the first place stays first (rounded half up) and, for each place, the 5th to 95th percentile of its position (the 50th and 950th of its 1,000 sorted positions).

Construction: the order is worked out again, as the method would, with the cap halved, with the cap doubled, with the performance-gap anchor at 80% instead of 90%, and with U replaced by annual surviving infants × (1 − Penta3 ÷ 100). The app names the places whose position changes, and under which choice.

Target population: each place’s own target populations (live births, surviving infants, second-year cohort, girls) are multiplied by 0.8, 0.9, 1.1 and 1.2, one place at a time, and the whole analysis is run again. The app shows whether the place, and every place above it, keeps its status and position.

These follow the robustness step of the OECD and Joint Research Centre handbook (2008), which covers the normalisation scheme, the treatment of missing data, the choice of weights and the aggregation method.

Change between observations and in package reviews

For Penta1 coverage (or pace) and Penta1 to Penta3 dropout, with Δ = after − before in percentage points: SE = 100 × √(p_before × (1 − p_before) ÷ n_before + p_after × (1 − p_after) ÷ n_after), where p is the figure ÷ 100, counted at no more than 1; n is the target for the period in whole children for coverage, and the Penta1 doses for dropout. The threshold is τ = the larger of 1 and 1.96 × SE. When either n is not known, τ = 1 and the reading says the size was not known.

A figure "rose" when Δ is at least τ, "fell" when Δ is at most −τ, and is otherwise "within normal variation for a place this size". 1.96 is the two-sided 95% point of the normal distribution; treating routine counts as binomial is a design choice.

In a package review, the indicators compare the package’s baseline with the later observation: Worse when coverage fell or dropout rose; otherwise Improved when coverage rose or dropout fell; otherwise No change; No later figures when there is no later observation. The same threshold is used in the mechanism check (dropout fell) and in "done as agreed, and the indicators improved".

A jump in Penta1 coverage since the previous observation is flagged as a possible data problem when |Δ| is at least the larger of 20 and 3 × SE.

Determinants (barriers)

A determinant is a CFIR 2.0 construct that a barrier finding was mapped to. It is a hypothesis about why the figures look as they do until a cause is recorded with evidence.

Weight = the sum, over places, of the strongest link at that place: 1 when the construct is a finding’s primary construct, 0.6 when it is its second construct. A place counts once, however many wordings it holds. It is shown as "recorded at N places", and as high when it is at least 0.66 of the top determinant’s weight, medium when at least 0.33, otherwise low [design choice].

Causes: Learning Log rows about a barrier (or, when the row does not say, rows in an assessment that improves weak performance) with this construct, a written cause and an evidence status other than Rejected. A cause counts only for a construct that has a finding. Evidence weights: Confirmed 1, Observed 0.9, Reported 0.6, Inferred 0.4, Hypothesis to verify or no status 0.2. A rejected cause changes nothing and is listed as ruled out.

Confidence = 0.25 + 0.10 × the smaller of (P − 1) and 3 + 0.10 × the smaller of (G − 1, at least 0) and 2 + 0.30 × E, where P is the number of places with a finding or a counted cause, G the number of voice groups among them (caregivers: mothers, fathers, other family; the health system: health workers, officials; records; the community), and E the highest evidence weight among the counted causes (0 when there is none). Less 0.10 when one of the construct’s own places has Penta1, Penta3 or MCV1 above 100%, a later dose above an earlier one, a target population likely too small, or a survey gap of 15 points or more; less 0.05 when one of its own places has a target population source that is not known, estimated or projected; kept between 0.15 and 1. Confidence of 0.66 or more needs E of at least 0.9 and at least two voice groups; otherwise it is kept at 0.65. An open disagreement about the whole assessment or about one of its places also keeps it at 0.65. Bands: high (0.66 or more), moderate (0.40 or more), low [design choices].

Explanation status: "Cause confirmed on the ground" when at least one counted cause is confirmed (how much it explains the figures is still the team’s judgement); "Supported, not confirmed" when causes were observed or reported; otherwise "Hypothesis to verify". Rival explanations are drawn only from the construct’s own places.

Determinants (enablers)

Enablers (facilitators) are ranked with the same weight and confidence formulas, from enabler findings and from Learning Log rows about an enabler, which ask why it works. They form their own ranking, in both kinds of assessment, and are never added to the barrier ranking. A construct recorded at a place both as a barrier and as an enabler stays in both rankings, and both views are shown.

An enabler is "confirmed on the ground" only when its counted rows span at least 30 days between the earliest and the latest observation date [design choice]; otherwise a confirmed row counts as 0.9 and the status is "Supported, not confirmed".

Matched strategies and programme actions

Implementation strategies are matched to CFIR 2.0 determinants using expert endorsement percentages from Waltz TJ, Powell BJ, Fernández ME, Abadie B, Damschroder LJ. Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implementation Science 2019;14:42 (Additional file 1), mapped to CFIR 2.0 constructs following Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated Consolidated Framework for Implementation Research based on user feedback. Implementation Science 2022;17:75 (Additional file 4). Strategy names and definitions: Powell BJ et al. Implementation Science 2015;10:21.

Only strategies endorsed by at least 20% of experts are listed: Level 1 (50% or more) and Level 2 (20% to 49.9%). All strategies tied with the seventh are kept, in alphabetical order. For Implementation Leads, the two columns of the tool are pooled as (formal leaders × 28 + champions × 27) ÷ 55, the numbers of experts who rated each. A subconstruct without a column of its own takes the list of the construct it belongs to, and says so. Strategies about fees, billing, payment schemes, liability law and financial penalties are left out, because routine immunization is free. Endorsement is expert opinion, not evidence that a strategy works.

A strategy’s score is the sum, over the lists that carry it and the places behind each list, of the strongest link at that place (1 or 0.6) × the share of experts who endorsed it. A place counts once per list, however many barriers it holds. Strategies are listed by score.

Constructs with no expert-endorsed match are labelled "No expert-endorsed match". Any strategy suggested there, and every strategy offered to protect and spread enablers, is Mataki AI’s own suggestion, labelled as not from the CFIR-ERIC tool, never ranked with endorsement percentages and never added to any score.

Programme standard actions (Reaching Every Ward, vaccine supply and cold chain practice, surveillance and safety practice) are matched by the words of each barrier and likely cause, by the place’s programme answers, and by its figures (for a place with figures of its own; a parent pooled from its places is not given them again), and are listed apart from the CFIR-ERIC strategies. A place on a data-quality hold gets the data check first and nothing else until it is reviewed.

References

Method version 2.0, Mataki AI release 58. The app shows the same text under Insights, How priorities are decided.