Adapting KAP methodology under evaluation constraints in a CSR education programme
| Sector | Education / CSR |
| Intervention type | Science lab set-up for school-going children |
| Evaluation type | Retrospective impact assessment |
| Methodology | Adapted KAP (Knowledge, Attitude and Practice) |
| Practice area | Monitoring, Evaluation & Learning (MEL) |
| Conducted by | NuSocia Delivery Team |
Methodology Case Study | During an impact assessment of a CSR-funded science lab intervention for school-going children, the NuSocia delivery team encountered a challenge common across the sector: no baseline data, no comparison groups, and a single point-in-time window to understand what had changed. This article captures a case where we adapted our methodology to generate meaningful findings without overstating impact.
THE CONTEXT
A familiar problem in CSR evaluation
Project implementation in the CSR ecosystem often begins under conditions that diverge considerably from planned settings. They are usually a lot faster, require more quantitative visibility, and need to be communicated to a board of directors with very different priorities than your traditional funders. For this reason, programme timelines compress. Implementation priorities take over. Processes like baseline studies or need assessments remain incomplete, not because of negligence, but because of practical constraints of time, funding, and operations.
Yet when an impact assessment is eventually commissioned, the expectation remains: to understand what changed.
This is the methodological tension NuSocia encountered while assessing an education intervention focused on science lab set-up for school-going children. The absence of baseline data, comparison groups, or historical records raised a direct question.
| Without a baseline, how do we understand progress responsibly, without overstating impact? |
THE CHALLENGE
KAP without a pre-post design
For decades, Knowledge, Attitude and Practice (KAP) studies have been used to understand not only what people know, but how knowledge translates into perception and eventually into behaviour. In a standard design, KAP tools are administered at multiple points in time, typically before and after an intervention, to capture shifts across these dimensions.
Field realities do not always permit this ideal design.
In many CSR evaluations, tools must be administered retrospectively, often at a single point in time. This introduces two compounding challenges:
- Respondents may feel encouraged, consciously or unconsciously, to share positive responses that reflect well on the programme.
- With children specifically, the tendency to provide what they believe are the “correct” or expected answers is heightened, particularly within school environments.
THE APPROACH
Reframing assessment as perception, not evaluation
The objective was simple: if a conventional pre-post design was not possible, could the conditions of administration still allow students to reflect honestly on perceived change?
NuSocia reframed the exercise as an anonymous perception test. Students were not required to write their names or provide identifying details. Questions were simplified into reflective before-and-now statements, supported through tick-mark responses rather than written explanations.
| Design shift: from evaluative questioning to reflective promptingInstead of:“Did the programme improve your confidence in science?”Students reflected through prompts such as:“Earlier, how comfortable were you answering questions in science class?”“How do you feel now?”Response options described recognisable experiences, not abstract scales:“I never raised my hand because I feared being wrong”“I answered sometimes but was unsure”“I now feel comfortable participating and sharing my thoughts” |
The shift moved the exercise away from proving programme success and closer to understanding personal experience. The perception tests were circulated without verbal instruction or framing. Students answered through tick marks, anonymously. This was akin to children identifying emotions through faces, offering more relatable data, than open form responses.
THE FINDINGS
Variation as signal, not noise
What emerged was not a uniformly positive picture. And that, in many ways, strengthened confidence in the responses.
Some indicators reflected visible improvement. Others showed only modest movement. Students reported increased comfort with classroom participation and independent learning activities. Certain areas showed little change.
| A uniformly optimistic dataset would have been easier to celebrate but harder to trust. The variation suggested students were responding through reflection rather than obligation. |
The findings did not establish causal impact in the strict experimental sense. But they offered meaningful signals of perceived change that might otherwise have remained invisible.
THE LEARNING
Responsible listening under methodological constraints
This case is not presented as a model for universal replication, nor as a replacement for rigorous baseline-driven evaluation. It is a reflection from practice.
Perception-based approaches administered at a single point in time carry important caveats. They cannot independently establish causal impact. When designed carefully, however, they can still provide useful insight into how people experience change.
| Key methodological considerations for practitionersAnonymity reduces socially desirable responding, particularly with children in institutional settings.Reflective before-and-now framing can approximate pre-post thinking without requiring multi-point administration.Experience-based response options ground abstract questions in recognisable behaviour.Variation in responses is a quality indicator, not a failure of the methodology.Transparency about limitations is part of rigour, not a departure from it. |
When monitoring and evaluation operates within tight methodological constraints, the task is not to abandon rigour but to adapt thoughtfully while remaining transparent about what the evidence can and cannot say.
Sometimes the task of assessment is not to measure impact with certainty, but to responsibly listen for early signals of change when certainty is still out of reach.
