Skip to main content

Table 5 Critical appraisal of quantitative component of studies evaluating Indigenous community development projects (n = 10)

From: A systematic review of studies evaluating Australian indigenous community development projects: the extent of community participation, their methodological quality and their outcomes

1st author, year Selection bias (A) Study design (B) Confounds (C) Blinding (D) Data collection methods (E) Withdrawal & drop-outs (F) Intervention integrity (G) Analysis (H) Summary rating
Mixed method studies (n = 8)
 Smith (2004) [53] Moderate Moderate NA Weak Weak Moderate Collection of quantitative data stopped before real community action started. Community-level allocation, individual-level analysis. No appropriate analysis of change in child growth over time. Weak
 Lee (2008) [42]a Weak Weak NA Moderate Weak Moderate Many youth involved in the interventions, no information on consistency, other community initiatives were running simultaneously (including stricter supply controls and rewards linked to school attendance). Community-level allocation and analysis. Statistical methods described in other publication. Dates of data collection (2001–2004) do not line up with dates of intervention (2003–2005), no post-test data. Weak
 Tyrell (2003) [43]a Weak Weak NA Moderate Weak Moderate No description of who was exposed to the project and who weren’t, nor of possible external influences on outcomes. Allocation on community and individual level. Evaluation on community, organisational and individual level. No statistical analysis (outcomes as percentages only). Weak
 Guenther (2011) [58] Strong Weak NA Weak Weak N/A All participants were part of the project; not all participants attended every session; it is likely that the results were influenced by other interventions put on the families. Individual-level allocation and analysis; statistical analyses (frequencies and t-test) were appropriate; analysis performed on actual intervention status. Weak
 Salisbury (1998) [44]a Weak Moderate NA Weak Weak Moderate No description of exposure or consistency; no mention of other interventions influencing outcomes; tested for population growth (which didn’t grow) Unit of allocation and analysis are on organizational level. No statistical analysis. Weak
 Hunt (2010b) [59] Weak Weak Weak Weak Weak Weak No description of exposure or consistency, potential influence of other interventions running in the communities at the same time. Unit of allocation community and organisational level, unit of analysis individual level, no statistical analysis. Weak
 Moran (2003) [45]a Moderate Moderate NA Weak Moderate Weak 92 % of participants reported awareness of town plan. Outcomes may be influenced by the cycle of optimism and pessimism. Unit of allocation is community level, analysis is done on community and individual level. No statistical analysis. Weak
 McCalman (2005) [54] Moderate Moderate NA NA Weak Weak All evaluation participants were exposed to intervention; consistency was not measured; outcomes likely influenced by other factors. Project was allocated at organizational level, data were collected on community level, cannot be sure whether changes at community level are caused by changes at the organizational level. No statistical analysis Weak
Quantitative only studies (n = 2)
 Jarvie (2008) [46]a Weak Weak Weak Weak Weak Strong One community withdraw temporarily, others stayed. There’s a chance that outcomes are influenced by other developments going on at the same time No statistical analysis, intervention offered at community level, data gathered at population level. Weak
 Shannon (2001) [47]a Weak Moderate N/A Weak Weak Moderate No description of exposure to intervention or consistency in delivery. Outcomes may be influenced by other factors Community-level allocation and analysis. Appropriate statistical analysis. Weak
  1. Note: Appraised using the Dictionary for Effective Public Health Practice Project Quality Assessment tool [38]
  2. a Published in peer-reviewed literature
\