Can local staff reliably assess their own programs? A confirmatory test-retest study of Lot Quality Assurance Sampling data collectors in Uganda

Beckworth, Colin, Anguyo, Robert, Kyakulaga, Francis Cranmer, Lwanga, Stephen and Valadez, Joseph ORCID: https://orcid.org/0000-0002-6575-6592 (2016) 'Can local staff reliably assess their own programs? A confirmatory test-retest study of Lot Quality Assurance Sampling data collectors in Uganda'. BMC Health Services Research, Issue 16:396.

	Text Can local staff reliably assess their own programs A confirmatory test retest study of Lot Quality Assurance Sampling data collectors in Uganda BMC Article RESUBMISSION.docx - Submitted Version Available under License Creative Commons Attribution. Download (71kB)
Preview	Text art%3A10.1186%2Fs12913-016-1655-4.pdf - Published Version Available under License Creative Commons Attribution. Download (540kB) | Preview

Abstract

Background
Data collection techniques that routinely provide health system information at the local level are in demand and needed. LQAS is intended for use by local health teams to collect data at the district and sub-district levels. Our question is whether local health staff produce biased results as they are responsible for implementing the programs they also assess.

Methods
This test-retest study replicates on a larger scale an earlier LQAS reliability assessment in Uganda. We conducted in two districts an LQAS survey using 15 local health staff as data collectors. A week later, the data collectors swapped districts, where they acted as disinterested non-local data collectors, repeating the LQAS survey with the same respondents. We analysed the resulting two data sets for agreement using Cohens’ Kappa.

Results
The average Kappa score for the knowledge indicators was κ=0.43 (SD=0.16) and for practice indicators κ=0.63 (SD=0.17). These scores show moderate agreement for knowledge indicators and substantial agreement for practice indicators. Analyses confirm that respondents were more knowledgeable on retest; no evidence of bias was found for practices indicators.

Conclusion
The findings of this study are remarkably similar to those produced in the first reliability study. There is no evidence that using local healthcare staff to collect LQAS data biases data collection in an LQAS study. The bias observed in the knowledge indicators was most likely due to a ‘practice effect’, whereby respondents increased their knowledge as a result of completing the first survey; no corresponding effect was seen in the practice indicators.

Item Type:	Article
Uncontrolled Keywords:	LQAS, Lot Quality Assurance Sampling, Test retest, Cohen’s kappa, Bias
Subjects:	W General Medicine. Health Professions > W 21.5 Allied health personnel. Allied health professions WA Public Health > Health Administration and Organization > WA 546 Local Health Administration. Community Health Services WA Public Health > Statistics. Surveys > WA 900 Public health statistics
Faculty: Department:	Clinical Sciences & International Health > International Public Health Department
Digital Object Identifer (DOI):	https://doi.org/10.1186/s12913-016-1655-4
Depositing User:	Helen Fletcher
Date Deposited:	10 Aug 2016 14:05
Last Modified:	06 Sep 2019 11:29
URI:	https://archive.lstmed.ac.uk/id/eprint/6047

Statistics

View details

Actions (login required)

Edit Item