The agreement, reliability and accuracy of a subjective-rating for technique measurement in rugby union after video-based training

Abstract

Background: Rugby union is a high-intensity, intermittent sport, characterised by numerous contact events, such as rucks, mauls, scrums, and tackles. Specifically, the tackle is the most common action to occur during a rugby game. Subsequently, the tackle (tackler and ballcarrier) is responsible for the highest number of total injuries during a season. The most effective tackle technique is associated with a lower risk of injury than poorly executed tackles. Considering the prevalence of tackles, their propensity to cause injury, and the link between safety and proper tackle technique, a tool for assessing individual tackling and ball-carrying technique would be useful for rugby union researchers and practitioners. In particular, the assessment of tackling and ball-carrying technique would aid in monitoring adaptations to training, evaluating training programs and the prescription of training, assessing player qualities, and predicting performance and identifying future talent. Objectives: The primary objectives of this study were 1) to determine if undergoing videobased training would improve the agreement, reliability, and accuracy of raters using a subjective-rating measure to assess contact technique in rugby union, and 2) to establish if a self-selected viewing pace resulted in improved agreement, reliability, and criterion-validity over a real-time pace. Methods: Data were collected from 30 participants, with no prior experience in rugby union, who were randomly divided into training and non-training groups. After completing the training or non-training visits, all participants conducted tackling and ball-carrying technique assessments in 4 subsequent visits. Participants viewed video footage of players tackling and carrying the ball into contact during a full contact rugby drill on a laptop computer. Participants assessed tackling and ball-carrying technique using the standardised list of technical criteria. Technical proficiency scores were calculated out of 12 for tackling technique and out of 10 for ball-carrying technique, respectively. Agreement and reliability of the assessments were measured using the proportion of observed agreement (Po) and Kappa statistics (K). Criterion validity (accuracy) was measured using the proportion of observed agreement (Po) by comparing the raters’ assessments to a criterion. Between groups comparisons for technical proficiency scores and accuracy were done with hypothesis testing and effect sizes. Results: Over the real-time and self-selected paces, the training group produced 61-73% (K=0.24-0.38, Fair) and 63-73% (K=0.12-0.17, Poor) inter-rater agreement for tackling and ball-carrying technique, respectively. The non-training group produced 61-72% (K=0.26-0.38, Fair) and 59-71% (K=0.15-0.19, Poor) inter-rater agreement for tackling and ball-carrying technique, respectively. Including both real-time and self-selected paces, the within session intra-rater agreement for the training group ranged from 71-84% (K=0.43-0.65, Moderate-Substantial) and 74-83% (K=0.39-0.48, Moderate) for tackling and ball-carrying technique assessment, respectively. Then, the intra-rater agreement for the non-training group ranged from 68-83% (K=0.40-0.62, Fair-Moderate) and 67-84% (K=0.31-0.55, Fair-Moderate) for tackling and ball-carrying technique assessment, respectively. In terms of between session reliability, over the real-time and self-selected paces, the training group achieved intra-rater reliability that ranged from 74-83% (K=0.50-0.62, Moderate-Substantial) and 72-82% (K=0.35- 0.45, Fair-Moderate) for tackling and ball-carrying technique assessment, respectively. The non-training group achieved intra-rater reliability ranging from 72-82% (K=0.46-0.59, Moderate) and 69-81% (K=0.33-0.48, Fair-Moderate) for tackling and ball-carrying technique assessment, respectively. The technical proficiency scores between the groups displayed variation, and in some cases it was statistically significant. Compared to the criterion scores, the training group produced observed agreement of 68% and 67% for real-time pace and 74% and 72% for self-selected pace on tackling and ball-carrying technique assessment, respectively. The non-training group produced observed agreement of 66% and 67% for realtime pace and 74% and 73% for self-selected pace on tackling and ball-carrying technique assessment, respectively. Conclusion: Whether trained or untrained, the individual should be able to produce the same results if the footage is assessed within the same viewing period (agreement) or over two separate viewings (reliability). The tool is accurate for both groups when assessing at real- time or self-selected paces, with the latter being more accurate. However, it would appear that the tool is insufficiently robust when it comes to agreement between different raters. Future research should explore the inter-rater dynamics of assessing tackling and ball-carrying technique.