Examining the measurement precision of behavior problems among a sample of primarily rural youth on juvenile probation and their parents

•Early detection and accurate assessment of youth behavioral health needs are essential to their short- and long-term outcomes.•Caregivers reported significantly higher internalizing and externalizing problems compared to youth.•When using brief screening tools to assess, youths’ internalizing and externalizing behaviors, both youth and caregiver perspectives are important for identifying intervention targets.•The Brief Problem Checklist (BPC) is a brief, feasible, and free tool that can produce reliable scores for a variety of problem behaviors, however more research is needed to confirm its psychometric properties when used to identify intervention needs among vulnerable populations (e.g., youth involved in the justice system, youth residing in rural areas). Accurate and timely intervention in the justice system is particularly critical in rural communities, given documented barriers to accessible, evidence-based services for youth. As youth in the juvenile justice system have a high prevalence of behavioral health needs, accurate assessment of those needs is a critical first step in linking youth to appropriate care. The goal of the current study is to examine the reliability of a brief assessment (the Brief Problem Checklist [BPC]) among a sample of 222 justice-involved youth and their caregivers who primarily reside in rural communities in the United States. Using a series of reliability analyses and tests of agreement, we examined whether youth and caregiver BPC produces reliable scales, the strength of the convergence among each of the BPC scales, and youth and caregiver agreement on the BPC scales. Findings support the reliability of the BPC, but not inter-rater reliability. Poor agreement between youth and caregiver reports exists for both youth internalizing and externalizing problems. Additionally, the BPC was significantly related to several theoretically relevant constructs, including treatment, substance use disorder severity, and family history of substance use. These findings lend merit to discussions about the need for more research on the reliability and validity of assessment instruments before their widespread use in guiding youth- and agency case planning decisions, along with informing conclusions about program effectiveness.