In the present study, the effect of training on inter-observer reliability was studied for a 5-category lameness scoring system used for routine on-farm surveys of welfare in dairy cattle. The inter-observer agreement between an experienced and an initially inexperienced observer was determined during an initial training phase and at specific time points in the course of data collection in 46 herds. During the training phase on three farms, inter-observer reliability increased to an acceptable level for both the 5-category gait scoring system and the distinction between lame and non-lame cows.

The 4th testing after 17 on-farm visits revealed a considerable increase in inter-observer reliability which was further improved in the course of the on-farm visits.

In conclusion, acceptable inter-observer agreement for differentiating between non-lame and lame cows was achieved after only a brief introduction. In order to obtain high inter-observer repeatability with the 5-category gait scoring system used in this study, (more) intensive training procedures are required.

Obtaining reliable welfare outcome measures from commercial farms can be challenging. We developed a training programme to train observers to score injuries of the tarsal joint, carpal joint and neck on dairy cows as part of an on-farm study. Twelve trainees were trained using protocols and photographs in a classroom session and on-farm visits. Continued repeatability checking was carried out during a refresher and mid-way assessment. Two trainers were used as the reference standard to which all trainees were compared. The study demonstrated that methods of scoring tarsal joint, carpal joint and neck injury can be learned by trainees from different backgrounds and high repeatability can be achieved and maintained at a very large regional or national level. Successful learning of injury scoring is dependent on protocols with strong definitions and photographs as well as repetitive training sessions. Additionally, continued repeatability checks are essential to ensure the reference standard continues to be met. This training programme can be used as a model to successfully train on-farm assessors.

Faecal soiling is one of the welfare indicators in the AWIN welfare assessment protocol for sheep (Ovis aries) and is measured by dag scores. Studies on dag scoring for ewes with docked and undocked tails have given rise to contradictory results. The aim of this study was to compare faecal soiling between ewes with docked and undocked tails and evaluate inter-rater reliability for faecal soiling of ewes. This study was conducted on a farm in Southern Brazil and included 66 undocked and 94 docked ewes. Dag score was recorded by three assessors on a scale of 1 to 5. There was no significant difference on faecal soiling for docked compared to undocked ewes and the median dag score was 3 (1-5). Repeatability amongst assessors by intra-class correlation coefficient of dag scores on docked and undocked ewes was 0.49 and 0.40, respectively; however, these repeatabilities showed no significant differences. The modest repeatability between three assessors on dag scoring indicates caution as regards the use of faecal soiling as an indicator and suggests a need for further studies. The best field results may be obtained by increasing the assessment sample to at least 160 ewes or by raising the number of assessors to five to promote better inter-observer repeatability. Results suggest that taildocking did not promote cleanliness on the breech area. Considering the negative impact on welfare, it seems reasonable to reverse the burden of proof and desist from recommending tail-docking in the absence of clear scientific evidence of any benefit.

The effect of Farm Assurance (FA) assessors’ attitude to farm animal welfare on the inter-observer reliability of some welfare outcome measures achieved following training was investigated as part of a larger project examining the feasibility and benefits of the incorporation of some on-farm welfare outcome assessments into UK Pig Farm Assurance Schemes. A total of thirty-one FA assessors were trained in three training sessions to assess the following welfare outcome measures: body lesions, tail lesions, severe tail lesions, lameness and pigs requiring hospitalisation. Assessment of photographs, live observations of individual pigs and pens of pigs were used to generate inter-observer reliability data. A previously validated farm animal welfare questionnaire was used to assess the FA assessors’ attitudes to farm animal welfare. Principal Component Analysis of FA assessor scores for this questionnaire resulted in two major components, with component 1 termed ‘pigs have mental welfare’ and component 2 termed ‘people-centric, pigs as profit’. FA assessors demonstrated a range in attitudes to farm animal welfare and, when assessing the same pigs, recorded a range in prevalence of welfare outcome measures and degree of agreement with a gold standard following training. There were only seven out of a possible 98 significant correlations between the FA assessor scores for components 1 and 2 and their recorded prevalence of welfare measures and levels of agreement with a gold standard. In particular, FA assessors’ scores for component 1 were significantly positively correlated with the recorded prevalence for pigs requiring hospitalisation in two of the three training sessions although there was no effect on the agreement with a gold standard for this measure. These results indicate that training in welfare outcomes, defined by a standard protocol, is relatively unconfounded by observer attitudes to farm animal welfare. To obtain better levels of agreement between assessors, and therefore more reliable data, it is recommended that FA schemes concentrate their resources on providing good quality training in a well-defined protocol and reliability testing and that they do not need to attempt to account for the attitudes of the FA assessors to farm animal welfare.

Qualitative Behavioural Assessment (QBA) is a whole-animal approach used to quantify the expressive style of animals’ behaviour. The aim of this study was to evaluate the inter-observer reliability of principal components and individual descriptors from QBA of housed sheep in Norway using a fixed list of descriptors. In part 1, eight animal welfare inspectors from the Norwegian Food Safety Authority scored 12 two-minute video clips, using 12 pre-defined terms. In the second part, three of these observers visited ten sheep farms and scored the behaviour using a modified list of eight terms. The scores for all assessors within each part were analysed using Principal Component Analysis (PCA). The level of agreement for the first two components and for individual terms was assessed using Kendall's coefficient of concordance (W). In part 1, only seven observers were assessed for reliability of principal components due to missing data, and for these observers the agreement was high for PC1 and moderate for PC2. For the sub-group of three inspectors that also participated in part 2, the reliability was high for both components. In the on-farm assessments in part 2, these three observers obtained only moderate agreement for PC1 and low agreement for PC2. Results illustrate that good reliability can be achieved with QBA for housed sheep using videos, but that equally satisfactory reliability is more difficult to obtain using the method on-farm. The result in part 2 may be related to less-controlled circumstances concerning exactly what the observers see when sheep are kept indoors with high stocking densities, observer drift, and limited between-farm variation, which is a challenge for many types of reliability studies in field conditions.

This study was conducted within the context of the Animal Welfare Indicators (AWIN) project and the underlying scientific motivation for the development of the study was the scarcity of data regarding inter-observer reliability (IOR) of welfare indicators, particularly given the importance of reliability as a further step for developing on-farm welfare assessment protocols. The objective of this study is therefore to evaluate IOR of animal-based indicators (at group and individual-level) of the AWIN welfare assessment protocol (prototype) for dairy goats. In the design of the study, two pairs of observers, one in Portugal and another in Italy, visited 10 farms each and applied the AWIN prototype protocol. Farms in both countries were visited between January and March 2014, and all the observers received the same training before the farm visits were initiated. Data collected during farm visits, and analysed in this study, include group-level and individual-level observations. The results of our study allow us to conclude that most of the group-level indicators presented the highest IOR level (‘substantial’, 0.85 to 0.99) in both field studies, pointing to a usable set of animal-based welfare indicators that were therefore included in the first level of the final AWIN welfare assessment protocol for dairy goats. Inter-observer reliability of individual-level indicators was lower, but the majority of them still reached ‘fair to good’ (0.41 to 0.75) and ‘excellent’ (0.76 to 1) levels. In the paper we explore reasons for the differences found in IOR between the group and individual-level indicators, including how the number of individual-level indicators to be assessed on each animal and the restraining method may have affected the results. Furthermore, we discuss the differences found in the IOR of individual-level indicators in both countries: the Portuguese pair of observers reached a higher level of IOR, when compared with the Italian observers. We argue how the reasons behind these differences may stem from the restraining method applied, or the different background and experience of the observers. Finally, the discussion of the results emphasizes the importance of considering that reliability is not an absolute attribute of an indicator, but derives from an interaction between the indicators, the observers and the situation in which the assessment is taking place. This highlights the importance of further considering the indicators’ reliability while developing welfare assessment protocols.