Measuring what matters: a descriptive participatory case evaluation of a tool for measuring outcomes of assistive technology after acquired brain injury

Abstract

Acquired Brain Injury (ABI) can lead to the need to plan for housing, assistive technology and support to meet a person’s goals. Recent Australian policy reforms have brought about opportunities for person-centred approaches to assistive technology (AT) product selection, and the opportunity for users of supports such as assistive technology and environmental adaptations to describe and measure their own outcomes. My AT Outcomes Framework (MyATOF) is an Australian tool designed for use by AT users and their support networks to articulate supports, outcomes, costs, and service delivery experiences. This article aims (i) report on use of MyATOF in a case study of ABI and AT-enabled housing and (ii) critically evaluate MyATOF tools and administration for this use case from the perspectives of the MyATOF author, an occupational therapist, and a lived experience expert with an ABI. An iterative development and test design was used to adapt the MyATOF administration procedures and conduct a case report evaluation. Results support the relevance of MyATOF conceptual domains, and detailed data about outcomes made possible by an AT-enabled home was captured. Recommendations are made to further refine the questions for ease of use by persons with ABI including simplifying the costing tool and enabling staggered administration. This case report suggests MyATOF can ‘measure what matters’ for ABI and AT-enabled housing and is worthy of further evaluation.

Introduction

Having a home, and having support, enables all of us to live our best lives. Acquired brain injury (ABI) can lead to the need for decisions about housing, technology and support design (Callaway, Tregloan, Williams, & Clark, 2016; Sloan, Callaway, Winkler, McKinley, & Ziino, 2012; Wright et al., 2017); an important part of that is user evaluation of both experiences and outcomes. People living with the impacts of health conditions such as ABI, may need specific features in their home and their supports, to live and to enable community living.

Human rights charters lay out the foundations of a good life. The United Nations Convention on the Rights of Persons with Disabilities (United Nations, 2006) specified many rights relevant to housing, technology and support design and to information access. The right to participate in, for example, work and employment, political, spiritual and cultural life, recreation, leisure and sport is influenced by access to both environmental interventions in housing, as well as assistive technology (United Nations, 2006). Notwithstanding universal rights, people living with disability around the world have very different experiences of support related to access to universal health care (Boisselle & Grajo, 2018; WHO & UNICEF, 2022). In Australia, all people living with disability receive some support, but the amount of support and type of support is dependent upon funding scheme eligibility (Friesen et al., 2014).

Enabling environments are effective in supporting people with ABI (Callaway et al., 2016) and may include inclusive communities (Watchorn & Layton, 2011), specific housing designs (Sloan et al., 2012), home modifications or adaptations (Cho et al., 2016), and assistive technology (AT). Author Chris Le Cerf (CLC) selected the phrase ‘AT-enabled home’ to describe his specific enabling environment, which is a type of integrated housing and AT support termed Specialist Disability Accommodation (SDA under the National Disability Insurance Scheme) in Australia (NDIA, 2021).

Specifying exactly what aspects of an enabling environment ‘work’ for an individual can be challenging. AT is an umbrella term for both assistive products and the services required to prescribe them (WHO & UNICEF, 2022). There are over 650 subclasses of assistive product classified in the international standard (AS/ISO 9999, 2018) which includes products relevant to life with ABI such as memory supports, information and communication technologies and mobility products (R. O. Smith et al., 2018). AT, home modifications and personal support are most effective when provided as a bundle, which may comprise a dozen or more individual elements (Layton & Wilson, 2009; Layton, Wilson, Colgan, Moodie & Carter, 2010). This makes it important to consider the relationship of supports as a whole, when considering the outcomes they may achieve. CLC utilises door and blind automation, a single-button call system to access 24-h shared onsite support, and various home modifications, aids and equipment. The ‘technology chain’ between built environment, AT and support is vital to be able to spend time alone at home, and do activities both at home and out in the community, like home or gym, gardening, going to the football, and eating out at local cafes.

Evidence also demonstrates the process of identifying and obtaining AT and the experience of autonomy as a consumer will influence a person’s satisfaction (Federici & Scherer, 2017; de Witte, Steel, Gupta, Ramos, & Roentgen, 2018; Desmond et al., 2018; Larsson Ranada & Lidström, 2019; E. M. Smith et al., 2018). In 2018, the National Disability Insurance Scheme commenced in the geographic location of CLC. The impacts of changed service delivery policy has been ‘bloody brilliant’ and provided opportunity to set goals and plan for housing, technology and support design every year, as part of an NDIS plan review. Thinking about what has been achieved, how to measure outcomes and the next goals for discussion in NDIS planning meetings, requires useable outcome measures.

Outcome measurement

AT outcomes research is defined as ‘systematic investigation aimed at identifying the changes that are produced by AT in the lives of users and their environments’ (Fuhrer et al., 2003, p. 1244). However published measurement tools either focus on one type of AT or modification; one type of disability, or one outcome area such as satisfaction, or independence; are usually designed to be administered by health care professionals; and often focus on outcomes of interest to funders or policymakers, such as ‘value for money’ or ‘hospitalisations avoided’ (Federici & Scherer, 2017). Some approaches aim to capture the impacts of sets of AT and quantify support needs with AT as a contributing factor (Arnold et al., 2014, 2015), but are not designed for use by the AT user, with their support network and their allied health professional if they choose.

My AT Outcomes Framework (MyATOF)

Comprehensive mapping of AT outcomes is provided through the My AT Outcomes Framework (MyATOF) (Table 1). MyATOF was developed by the first author with members of Australia’s disability and AT community (Layton, Doyle & Volkert, 2019; Layton et al., 2018).

Table 1. MyATOF Operational Framework and supporting references

MyATOF is based upon international classification systems, research evidence, regulatory and service delivery principles (Table 1). These evidence-based structures are translated into six optional tools. Table 1 lists the data sources used to operationalise each Tool within MyATOF.

MyATOF is person-centred, and intended to enable people and the health professionals who support them to describe the unique supports which deliver their valued outcomes (Layton & Callaway, 2020). The Framework’s tools identify which AT is used or required, the quality of service delivery, to what degree AT impacts upon their lives now and into the future, whether human rights are being realised, what unmet or undermet outcome areas remain to be fulfilled and at what likely cost.

MyATOF is intended for use across disability types and for all sorts of AT and related supports. MyATOF has been applied to prosthetics (Layton, Noonan & O’Connor, 2019) and to seating and positioning (Layton et al., 2019). A Delphi study run in southern Africa validated the domains for a set of low- and middle-income countries, and for stakeholders with dual sensory loss (Bell et al., 2021). For each use case, all optional tools were offered, but the subset of guiding categories and examples were modified, for example prosthetic examples included for consumers with amputation, and vision and hearing product examples offered for people with sensory loss.

This article aims to (i) report on use of MyATOF in a case study of ABI and AT-enabled housing and (ii) critically evaluate MyATOF tools and administration in the context of ABI and AT-enabled housing.

Method

A descriptive case study approach is used as this enables authors to illustrate topics within an evaluation (Yin, 2016), The method is informed by the fundamental principles of co-design (Arnstein, 1969) and consumer engagement in healthcare (Hill, 2011). Inclusive research principles were followed, including ownership through authorship, participatory engagement from project inception and accessible materials (Layton, Bould, Buchanan, Bredin, & Callaway, 2022). These principles, as utilised by NL in conjunction with lived experience experts have been cited in recent reviews of inclusive research (O’Brien, 2022).

Participants and Setting

The participant Authors comprise NL, an occupational therapist who advises on AT policy and practice globally (Layton & Borg, 2019) and lead author of the MyATOF. CLC is a lived experience expert, a person living with ABI in an AT-enabled home. CLC consults as a paid lived experience expert and contributes to an occupational therapy school training programme, working with occupational therapy students who attend the SDA setting on clinical placements. The authors are colleagues through Monash University and collaborated on this project in the first quarter of 2022. Four occupational therapy students on placement at the time with CLC were invited to participate in MyATOF completion.

Materials

MyATOF is available as an online platform (https://www.at-outcomes.org). The MyATOF tools were downloaded into a pen and paper format by NL (Supplementary File). Both Authors designed this MyATOF iteration so it could be (a) used in an interview context and (b) administered in sections, over time.

Procedure

1. 1. NL briefed the occupational therapy students who would administer the interviews, providing:- background on the MyATOF – MyATOF: interview format (Appendix A) – Follow-up question set to elicit user experience regarding MyATOF (Appendix B).

2. 2. The students familiarised themselves with MyATOF and negotiated an interview schedule with CLC whereby 2 students at a time would conduct the interviewer and scribe the results.

3. 3. Interview schedule to complete (a) MyATOF: interview format and (b) follow-up questions set, once MyATOF completed. CLC elected to complete the MyATOF over three 30-min sessions, answering the follow-up questions at the end of session three. The students recorded the interviews with permission of CL, deleting records once they had finalised data collection; and the information collected by the students was aggregated for the report on the framework.

4. 4. The completed MyATOF; interview format was structured into results tables by NL, in accordance with the online results layout offered by the MyATOF online tool.

5. 5. Both authors discussed (a) the MyATOF results and (b) the follow-up questions regarding the experience of using MyATOF. Both authors arrived at the discussion and conclusion points described below.

Results

Firstly, the case study results with MyATOF are reported. Secondly, the question set evaluating the tools and administration are reported.

The My Supports tool (Figure 1) considers the supports that are in use. CLC identified 27 different individual supports including environmental adaptations (8), assistive products (16), and forms of personal support (3) (see Supplementary File). This comprehensive list of AT included mainstream technologies with features which suit CLC’s needs. Open ended questions captured rich descriptions of what works ‘I love the layout of my home’ and what could be improved’.

The My Outcomes tool considers what can be done with these supports. CLC recorded responses for every domain and demonstrated the repeated utility of AT and other supports across multiple different activities and participations. Mobility was enabled by AT (walker and wheelchair), as was being part of the community, participating in recreation, and having a political life by going to vote. AT and support work also enabled CLC’s spiritual life, by attending Buddhist temples with his support workers and mobility aids (4WW and wheelchair) and performing seated yoga and meditation in his accessible chair. AT, support work and environmental adaptations, enabled self-care and managing general domestic and household tasks. Communication, relating to others, learning new skills and having an educational life were supported by AT such as the audiobook player (DAISY player) and computer setup. For some participations, AT in an adapted environment were all that was required, ‘I enjoy the independence in making coffee in the morning with the three-button coffee maker, without the need for a support worker’.

The My Rights tool considers human rights. CLC identified the detailed contributions that his supports make to the realisation of his human rights under the UN. The stories of achievement, or not yet realising rights echo those shared in the ‘supports’ tool. Table 2 specifies the ways in which AT and related supports enable a human right to be realised. Overall, CLC found the article of the UN Convention to have meaning and to clearly map to his supports. He found the question regarding Article 25: Health to be unclear and proposed a clearer question would be: ‘What specific AT allows you to get into health services if you need them’. Some questions led to consideration of future goals and plans. For example, in considering Article 30 about participation in leisure, CLC reflected that:

Table 2. MyATOF Results: Realisation of human rights

Due to COVID-19 it has been hard to participate in leisure, however, I would like to go to the cinema and theatre more with the help of my support worker… and to use audio cues to access the museum and the new dinosaur exhibition when it comes out.

The My Costs tool asks how much do my supports cost and how much do my supports save?. Prompts ask people to consider the cost of the AT and getting it set up and maintained, to think about money you don’t have to spend because you have the AT; and to think of any costs you may save later on because you currently have AT. In response to this question, When reflecting on this tool as we wrote this article, CLC spoke of his ‘medium to long term goal’ of moving to the country, and how he thinks about costs (cheaper land, thinking about partnering with government to build a suitable dwelling), as well as cost offsets such as the ability to have solar panels for power or to grow food. The question was therefore answered at the broadest level, recognising that supports do cost money and may save money. In terms of specific costs and savings, CLC described this tool as difficult to complete, and suggested the person who provides him financial support would need to be involved.

The My AT Service Delivery Pathway tool considers the experience of receiving supports, benchmarked to international good practice and the empowerment and satisfaction dimensions associated with this. Figure 2 contains key quotes from CCL in relation to each service delivery step, demonstrating the importance of knowledgeable supporters to support the process and the value of each step. Reflecting on the service delivery pathway, CLC recalled one instance where he is unable to use the device without further support and training, ‘I would like to use the voice recorder more to help with short-term memory, however, have forgotten how to use the voice recorder and would like to be retaught’.

Figure 1. MyATOF ‘My Supports’ lists the activity and participation domains of the World Health Organisation ICF.

Figure 2. AT service delivery pathway.

The My Customer Experience Tool comprises a set of statements that AT users tell us are important about the experience of getting AT (De Jonge et al, 2015). CLC provided data for and affirmed the importance of all steps. Additional commentary was provided on three statements in particular. CLC feels it is ‘absolutely important’ to consider AT needs across the lifespan and as needs change. Regarding support throughout the process of getting AT, including product trial, training and maintenance he states, ‘Some products are no longer useful anymore or no longer work, for example, the brakes on my old walker no longer work…I would like more support with maintenance of AT’.

Having assistive products that work well and don’t interfere with other supports was seen as important, and an ongoing compromise, ‘My walking frame doesn’t fit through some doors as it’s wider than most walkers… this is an issue when going to the podiatrist as the doors are narrow’.

Consideration of personal preferences and identity so that AT is chosen to suit lifestyle and participation was a valuable question from the perspective of CLC, and one which is rarely asked, ‘It is absolutely important to be involved in the consideration of personal preferences when choosing AT, for example, my Richmond Football Team coloured keyboard’.

This tool also captured future goals, for example working to change between free-to-air TV and putting on a movie independently without support. This is currently challenging as the remote controls are all individual.

Discussion

Furthering the exploration of MyATOF across populations and settings, this article describes the iterative development and testing of an interview-based version of MyATOF, applied in an SDA setting to evaluate AT-enabled housing. MyATOF was used by a person with ABI, and the process of completing MyATOF was supported by occupational therapy students as interviewers and scribes.

The results demonstrate the applicability of MyATOF with ABI in describing life with an AT-enabled home. In terms of utility, the My Supports Tool open text fields, supported by a taxonomy of technologies and adaptations, was exhaustively able to capture all supports identified by CLC. The ‘My Outcomes’ tool comprehensively mapped sets of AT to the activity and participation outcomes they support. Explaining the role of, for example, mobility products in supporting a cultural life, or the role of information and communication technologies in supporting a political life, is of value in that it ‘makes the case’ for the full impact and outcomes of AT in a person’s life. My Human Rights contained some theoretical categories that CLC felt would benefit from further explanation, as noted. My Service Delivery and My Customer Experience Tools elicited related sets of comments but were not felt to be repetitive. My Costs Tool could effectively only be answered at the broadest level, that is, a narrative statement about overall costs and perceived overall benefit or return. This is in contrast to the update of this tool by cohorts for example prosthesis users, who took the opportunity to specify market costs for various prosthetic limbs and contrast this with cost offsets such as fewer environmental modifications, increased productivity or decreased health conditions.

In terms of measuring what matters, MyATOF enabled comprehensive descriptions of life with AT to be mapped against standardised criteria, turning an AT user narrative into an organised data ‘story’. Using MyATOF, CLC generated a written document containing comprehensive information about the supports used, their impact and outcomes, and the nuances of service provision. Further, the full benefit of supports was spelled out across multiple domains of activity and participation. Such a granular data capture is valuable in arguing that, for example, a walking aid does not just deliver on mobility outcomes, rather, it may deliver recreational, social, exercise and volunteering engagement. The ‘My Rights’ tool placed stories of participation in a ‘rights realised/rights not yet realised’ framework and in so doing provides an impetus to argue for full supports. Costs is a challenging area and one where specific prompts have been useful in other applications of MyATOF, for example listing the prices of prosthetic products and consumables, and identifying common cost offsets and social return on investment evidence.

Unsurprisingly, the constellation of supports CLC uses or envisions needing is not static – there are goals around tailoring his ICT supports for watching TV and requirements for repair and maintenance of his four-wheeled walker. Some MyATOF users run the question set with different types of AT, or even envisioning optimal AT, to quantify the likely changes in participation, in cost, and in outcome.

Overall, the applicability of MyATOF has been demonstrated as a multidimensional outcome framework to capture the impact AT-enabled housing. AT users value participation outcomes and the opportunity to explain what life is like with and without AT. Policy stakeholders seek data on costs and on meeting human rights obligations. Product data is captured by the lens on supports, while personnel and provision issues are captured through the service delivery and customer experience tools. Such integrative measures from a consumer standpoint are key data sources according to the newly published Global Report on Assistive Technology (WHO & UNICEF, 2022).

Limitations

This article describes use of MyATOF by one individual only, in the context of collaborative participatory enquiry. It will be important to test MyATOF use more widely with a diverse set of people with brain impairment to ascertain wide useability. Indeed, CLC makes an important point about the way MyATOF questions are asked, noting if questions are too difficult, some AT users might not understand the question. As it stands, the straightforward high level question for each tool was comprehensible for this user. The levels of executive functioning needed to respond to theoretical concepts such as rights, or technical questions regarding costs, are an important consideration for use by people with cognitive impairment. A further observation from both authors is the challenge of providing ‘concrete’ questions for each of the MyATOF Tool to guide completion, particularly for rights and costs tools. These are key learning for MyATOF development, and emphasise the importance of enabling a broad ‘high level’ responses as well as optional detailed responses, to be captured. The presence of a support person, or, indeed, AT professional, to assist in completing the tools may be required to use MyATOF most effectively.

Conclusion

Australian policy reforms mean people with significant ABI can now be provided with government funding for environmental adaptations, assistive technology and other supports, based on their participation goals and associated support needs. In order to be heard and understood in these new funding contexts, AT consumers and the professionals who support them must ‘make the case’ for adequate funding of supports in new ways. The MyATOF is based upon evidence that provide a blueprint or guidance about human rights, outcomes, supports, costs, and service delivery expectations, designed for use by AT users. The MyATOF: interview format enabled an AT user with ABI to provide data via interview. Five of the six tools captured were found to be useable by CLC. One tool, concerning costs, could only be answered at the broadest level, and would require further plain language adaptation or strategies to bring information from key others, such as people who support individuals with ABI with financial matters. This project has high potential impact as consumers and policymakers from the disability and aged care sectors demonstrate significant interest in these tools for self-advocacy, data collection and policy performance purposes.