The past few decades have witnessed a marked increase in our understanding of how congenital heart disease (CHD) impacts neurodevelopment. We now know that certain forms of CHD are associated with differences in brain development that begin in utero, Reference Limperopoulos, Tworetzky and McElhinney1–Reference Peyvandi and Rollins6 and that these abnormalities in brain development can predict adverse neurodevelopmental outcomes. Reference Sadhwani, Wypij and Rofeberg7–Reference Peyvandi, Latal and Miller8 Neurodevelopmental challenges threaten quality of life across the lifespan for many individuals with CHD, Reference Marelli, Miller and Marino9–Reference Marino, Lipkin and Newburger13 and variability in neurodevelopmental outcomes is often more strongly predicted by sociodemographic and patient-specific (e.g., prematurity and genetics) factors than medical/surgical management strategies. Reference Gaynor, Stopp and Wypij14–Reference Bucholz, Sleeper and Sananes17 With few exceptions, this growth in knowledge has been fuelled by data from single-centre studies. Existing studies are characterized by small to moderate sample sizes, heterogeneous patient populations, inconsistent neurodevelopmental assessment approaches, underrepresentation of socio-economically disadvantaged and culturally diverse children, and only scattered attempts at replication. As a result, generalizability and applicability from any one study’s findings to the broader CHD population are limited. Further, it is concerning that, despite overall improvements in medical care for individuals with CHD, and release of the 2012 American Heart Association scientific statement on neurodevelopmental care of these cardiac survivors, the prevalence of neurodevelopmental disabilities has not decreased. Reference Marino, Lipkin and Newburger13,Reference Loblein, Vukmirovich and Donofrio18,Reference Feldmann, Bataillard and Ehrler19 In fact, there are currently no empirically supported neuroprotective medications, Reference Stegeman, Lamur and van den Hoogen20 and there are only limited strategies Reference Newburger, Jonas and Soul21 to mitigate the risk of brain injury/dysmaturity and neurodevelopmental disabilities in this vulnerable population.
In an effort to address the inherent limitations of the existing research landscape, some countries have formed consortiums to promote and standardize neurodevelopmental research in CHD, Reference Stegeman, Feldmann and Claessens22,Reference Feldmann, Hagmann and de Vries23 whereas other countries have created prospective registries examining neurodevelopmental outcomes of individuals with CHD. Reference Natterer, Schneider and Sekarski24,Reference Marshall, d'Udekem and Winlaw25 In service of the same goal, the Cardiac Neurodevelopmental Outcome Collaborative, Reference Marino, Sood and Cassidy26 a multi-centre, multi-national, multi-disciplinary group of healthcare professionals from over 40 member institutions, set out to establish a data registry capturing clinically relevant neurodevelopmental outcome data from routine clinical care. Launched in 2019, the Cardiac Neurodevelopmental Outcome Collaborative’s registry has already become the largest repository of its kind and will provide opportunities for investigators to conduct novel, large-scale research and quality improvement studies, the results of which are likely to be more representative and therefore more beneficial to children and families affected by CHD. As the number of centres contributing data to the Cardiac Neurodevelopmental Outcome Collaborative registry continues to grow, so too will the Cardiac Neurodevelopmental Outcome Collaborative’s capacity to provide neurodevelopmental benchmarking across the broader collaborative.
The purpose of this paper is to describe the development of the Cardiac Neurodevelopmental Outcome Collaborative registry, components of the registry, initial research projects proposed using data from the registry, and the valuable lessons that have been learned through the process.
Collaboration with Cardiac Networks United and ArborMetrix, Inc
Characterising the neurodevelopmental outcomes of individuals with CHD and potential predictors of neurodevelopmental risk require historical information about cardiac diagnosis and hospital course. Many Cardiac Neurodevelopmental Outcome Collaborative member institutions already collect detailed clinical data as a part of the Pediatric Cardiac Critical Care Consortium Reference Gaies, Cooper and Tabbutt27 and the Pediatric Acute Care Cardiology Collaborative. Reference Kipps, Cassidy and Strohacker28 Under the auspices of Cardiac Networks United, Reference Gaies, Anderson and Kipps29 these groups created an efficient method to link a patient’s neurodevelopmental follow-up data with their medical and surgical data. Data needed for the Pediatric Cardiac Critical Care Consortium and Pediatric Acute Care Cardiology Collaborative registries are entered by hospitals into a contracted software system and then are ultimately transferred to the Cardiac Networks United Data Core at the University of Michigan for storage and analysis. The Cardiac Neurodevelopmental Outcome Collaborative developed an additional data collection module with a healthcare software vendor, ArborMetrix, Inc., which provides web-based reporting platforms for other Cardiac Networks United collaboratives. This pathway allowed easy linkage to the clinical data at the Cardiac Networks United Data Core, enabling centres to efficiently use their resources for data collection, reduce data entry burden, and promote data standardization. In addition, the partnership of the Cardiac Neurodevelopmental Outcome Collaborative with Cardiac Networks United simplified the contractual and regulatory processes and greatly reduced the start-up time compared to building a stand-alone registry.
Process of registry development
The Cardiac Neurodevelopmental Outcome Collaborative registry was developed to capture information from cardiac neurodevelopmental follow-up clinics and programs. Aiming to support the administrative, quality, and research priorities of the Cardiac Neurodevelopmental Outcome Collaborative and its member institutions, the registry needed to not only permit comparison of results within and across centres but also to compare how assessments are performed, the timing of evaluations, and follow-up rates. Over the last decade, as congenital heart centres have built their neurodevelopmental follow-up programs, the approach to both in-patient and out-patient multi-disciplinary neurodevelopmental teams, testing schedules, testing batteries, and neurodevelopmental follow-up have varied widely. Much of this variability is dependent on existing infrastructure, clinical champions, and administrative support. Reference Miller, Sadhwani and Sanz30,Reference Miller, Lisanti and Witte31 Although the Cardiac Neurodevelopmental Outcome Collaborative published and disseminated recommended assessment batteries, Reference Ware, Butcher and Latal32,Reference Ilardi, Sanz and Cassidy33 admittedly the value of different approaches had not been tested or objectively compared. The registry, therefore, needed to have the capacity to objectively capture practice variation to make evidence-based decisions and continually reassess practice recommendations that support programs of differing sizes and resources. Stakeholders in the development of the registry included providers across multiple disciplines (e.g., psychology and cardiology), centre administrative leaders from programs of varying size, as well as representatives from the Steering Committee and select Standing Committees of the Cardiac Neurodevelopmental Outcome Collaborative, including Database and Implementation, Research, and Quality Improvement. Reference Marino, Sood and Cassidy26 Fig 1 provides an overview of the organizational structure of the Cardiac Neurodevelopmental Outcome Collaborative, including the different standing committees and cores.
Figure 1. Organizational structure of the Cardiac Neurodevelopmental Outcome Collaborative.
Development of the registry occurred in a stage-wise approach, first with higher-level conceptual discussions within the Steering Committee regarding the purpose, value, and return on investment (of both time and capital) for the member institutions and the Cardiac Neurodevelopmental Outcome Collaborative. Once the Steering Committee determined the scope of the registry and ownership of data, the Database and Implementation Committee Reference Marino, Sood and Cassidy26 took over the operationalization, working closely with the Data Coordinating Center, Neurodevelopmental Core Lab, the Data Analysis Core, the Research Committee, and the Quality Improvement Committee. Together this group developed a data dictionary and manual of operations. 34 Limiting duplicate entry was prioritized, and data definitions were closely aligned with already developed fields in other Cardiac Networks United datasets. Given the longitudinal nature of the datasets, however, some demographic fields were populated from the other Cardiac Networks United registries with the option to update any fields that may have changed over time. The components of the registry are outlined in Table 1.
Table 1. Components of the cardiac neurodevelopmental outcome collaborative data registry.
Data management, monitoring, and analysis
The Cardiac Neurodevelopmental Outcome Collaborative Steering Committee understood that efficient, effective management, and monitoring of data fidelity would be key to the registry’s success. To this end, three entities were established to assist in data monitoring and analysis, the Data Coordinating Center, the Neurodevelopmental Core Lab, and the Data Analysis Core. These entities work in concert to support regular monitoring of data accuracy, improvement of data entry, creation of datasets, and data analyses to address research aims. The Neurodevelopmental Core Lab and Data Analysis Core were selected by a competitive application process in 2018, supervised by the Steering Committee with renewal every 5 years. Fig 2 summarises how data flow from member institutions into the registry and how these entities work together to export data, monitor for errors, correct errors, and create datasets for analyses.
Figure 2. Anticipated data flow within the various components of the Cardiac Neurodevelopmental Outcome Collaborative.
To capitalize on existing infrastructure and simplify regulatory protocols, the Cardiac Neurodevelopmental Outcome Collaborative enlisted the existing Cardiac Networks United Data Coordinating Center at the University of Michigan to support data collection and exporting of data. There is an embedded Cardiac Neurodevelopmental Outcome Collaborative data manager on the Data Coordinating Center’s data management team. The primary role of the Data Coordinating Center revolves around data quality and integration. The Data Coordinating Center facilitates data collection and integration of data from other data sources and works closely with the Neurodevelopmental Core Lab to develop and maintain data integrity. The Data Coordinating Center also creates datasets for regular quality reporting, investigator-initiated research, and smaller-scale data requests for study planning or other purposes.
The Neurodevelopmental Core Lab is responsible for ensuring the fidelity and quality of neurodevelopmental data collection, facilitating access to key measures and appropriate training of site providers in the administration and scoring of measures, and supporting the integrity of scientific applications of these data in research. The Neurodevelopmental Core Lab is comprised of a principal investigator and research assistant with expertise in developmental and neuropsychological assessments. It focuses on orienting new sites to existing protocols, working with the Data Coordinating Center to establish automated data accuracy checks, and screening data exports for possible data errors or unusual patterns of data. Reports are provided to the Data Coordinating Center, which communicates errors directly to sites for clarification or correction. Feedback will be provided to individual sites with recurrent errors in order to improve data capture and reduce errors.
The Data Analysis Core coordinates with the Data Coordinating Center and Neurodevelopmental Core Lab to ensure quality and integrity of the neurodevelopmental data and to conduct data analyses. The Data Analysis Core, led by a principal investigator with a background and expertise in statistical analyses for multi-centre research studies, provides statistical and methodological input for grant applications and research and quality improvement projects, including supporting analyses for papers and abstracts. The principal investigator is responsible for recruiting additional staff members and overseeing their work within the Data Analysis Core.
Quality control and access to data
As with any clinical registry, security of data, and efficiency and accuracy of data entry were a priority. The registry was built and maintained around NIST 800-53: a cybersecurity standard and compliance framework developed by the National Institute of Standards in Technology. 35 Data encryption, multi-factor authentication, and multi-site disaster recovery plans help ensure that patient health information is kept private and secure within the registry. A robust system of range checks for standardized tests were also built into the Cardiac Neurodevelopmental Outcome Collaborative module to improve data integrity and limit potential inaccuracies. Finally, certain data points were programmed as required fields to limit missing data. In addition, prior to any data analyses, data are cleaned and queried for irregularities and missingness and sent back to sites to rectify or complete.
In October 2021, the Database and Implementation Committee introduced bimonthly “User Feedback Sessions” to continually educate and update end users as well as collaborate to improve the efficiency and accuracy of data entry into the registry. The Data Coordinating Center also developed a system to run data quality checks at periodic intervals to confirm whether raw scores align with standardized scores according to test norms. All errors are flagged, submitted to the Neurodevelopmental Core Lab to review, and then sent back to each institution to resolve. In addition, when a dataset is prepared for statistical analysis, the required data are cleaned, queried for irregularities, and, if needed, sent back to the sites to rectify errors and/or inconsistencies. Depending on the extent of irregularities, a site audit by the Neurodevelopmental Core Lab may also occur.
Member institutions have full access to all of their own data that they input into the registry in real time. Mechanisms are set in place to enable a site to export their own data as needed. Data can be downloaded from the Cardiac Networks United platform using a secure file transfer system. In addition, the Cardiac Neurodevelopmental Outcome Collaborative module on the Cardiac Networks United platform has a dashboard where institutions can access comparative analytics indicating how their institution compares to other sites, albeit blind to other sites’ names. These data can be used for internal monitoring, benchmarking, and quality improvement initiatives.
Initial goals and projects of the data registry
The Research Committee of the Cardiac Neurodevelopmental Outcome Collaborative identified two primary aims for the registry’s first Cardiac Neurodevelopmental Outcome Collaborative-initiated research studies. The overarching goal of these projects was to inform a broader understanding of the patterns in neurodevelopmental follow-up and risk factors for adverse outcomes, beyond what single-centre studies or clinical trials can provide. The first project aimed to identify the rate of neurodevelopmental follow-up at 11–30 months of age for children who underwent cardiopulmonary bypass in the first year of life. This age range was chosen because it is a critical period for toddler development and encompasses the Cardiac Neurodevelopmental Outcome Collaborative’s 18-month follow-up recommendations. Reference Ware, Butcher and Latal32 Patient-level (e.g., race/ethnicity, cardiac diagnosis, age at surgery, genetic diagnosis, and distance to centre) and centre-level (e.g., centre’s referral criteria, scheduling process for the neurodevelopmental visit, administrative and staff support, and surgical volume) variables were examined to identify factors that predict who did and did not attend a neurodevelopmental evaluation. Analyses were conducted with sites de-identified. The findings from this study are integral for understanding the global implementation of the 2012 Scientific Statement recommendations made by the American Heart Association. Reference Marino, Lipkin and Newburger13 The second project, which is in progress, aims to identify the association between clinical factors and neurodevelopmental outcome at ≤ 30 months of age for those undergoing cardiopulmonary bypass in the first year of life. The Bayley Scales of Infant and Toddler Development, Third and Fourth Editions Reference Bayley36 and the Adaptive Behavior Assessment System, Third Edition, Reference Harrison, Oakland and Angeles37 are outcome measures, and medical, surgical, sociodemographic, and de-identified centre-level variables are possible predictor variables. While prior research studies provided important information about risk factors for adverse neurodevelopmental outcomes in CHD, Reference Marino, Lipkin and Newburger13,Reference Gaynor, Stopp and Wypij14 the size and diversity of the Cardiac Neurodevelopmental Outcome Collaborative registry will contribute invaluable data across a large spectrum of patients undergoing routine neurodevelopmental evaluations. This may lend novel insights for those at highest risk for poor neurodevelopmental outcomes and inform future clinical practice recommendations.
Challenges and lessons learned
Finding the optimal balance between standardized and flexible approaches to neurodevelopmental assessment and data entry was a challenge throughout the development and early implementation of the registry. Initially, variables in the registry aligned closely with age-specific assessment batteries that were recommended by the Cardiac Neurodevelopmental Outcome Collaborative. Reference Ware, Butcher and Latal32,Reference Ilardi, Sanz and Cassidy33 In practice, however, cardiac neurodevelopmental follow-up programs vary widely with regard to resources and structure, Reference Miller, Sadhwani and Sanz30 and some programs partner with other high-risk clinics (e.g., Neonatal Intensive Care Unit and Down Syndrome) that may administer different tests or assess patients at different time points. To facilitate the use of the registry by all of the Cardiac Neurodevelopmental Outcome Collaborative member institutions and to eliminate the need for duplicate site-specific registries and double data entry, greater flexibility was needed. Variable constraints were loosened to allow for data entry across the age ranges specified for each neurodevelopmental test (as opposed to only those age ranges recommended by the Cardiac Neurodevelopmental Outcome Collaborative), and variables corresponding with alternate test measures that are lower cost, time-efficient, or available across a broader range of member institutions were also included.
Another challenge was finding the optimal balance between responsiveness to user feedback and consistency in registry variables over time. For the first two years following the registry launch, iterative changes were made to the registry structure and the individual variables in response to user feedback and review of entered data. For example, duplicate demographic and clinical variables across serial assessments were described by users as burdensome and yielded inconsistent responses. To allow for the possibility that these variables could change over time (e.g., newly detected genetic diagnosis and change in parental education level) while reducing user burden and inconsistent responding, these variables were moved to a patient-level tab within the registry, with the requirement that users review this tab and attest that the information remains current. Variables captured by the Pediatric Cardiac Critical Care Consortium and the Pediatric Acute Care Cardiology Collaborative, which have robust auditing processes, were imported into the Cardiac Neurodevelopmental Outcome Collaborative module, with the option to change the response if needed. The emergence of the global COVID-19 pandemic also necessitated additional variables, including location of the neurodevelopmental assessment (virtual/in person) and whether the clinician and child wore personal protective equipment during the in-person assessment, as both could affect child engagement and interpretation of test findings. After two years of iterative changes to the database, the current version was locked in August 2021 to facilitate greater consistency in registry variables moving forward. Subsequent versions incorporating necessary changes, including future revisions of testing items, ongoing use of telehealth in ND assessment, Reference Cox, Butcher and Sadhwani38 and other improvements based on user feedback will be released on an annual basis.
Current status and future directions
The Cardiac Neurodevelopmental Outcome Collaborative registry officially launched on 15 May, 2019 with the original release limited to birth to 5-year-old patient assessment entries. In July, 2020, the ≥ 6-year-old, school age module was launched. Version 1.0 of the clinical data registry was closed on 31 August , 2021, as the registry transitioned to Version 2.0. During those two years of Version 1 updates and enhancements, a total of 3,597 assessments were entered into the registry from 29 contributing member institutions (28 in the United State and 1 in Canada) on 2,903 unique patients. As of 1 January, 2023, there are 34 North American member institutions contributing data with a total of 7,918 assessments in the registry. Data sharing contractual issues have made it challenging for some international sites to “share” their data under the constraints of their country’s data privacy protection constraints. Ongoing discussions and strategies have been introduced to collaborate with institutions outside of North America on data sharing agreements.
In addition to the two Cardiac Neurodevelopmental Outcome Collaborative-initiated research studies previously described, in 2021, the Cardiac Neurodevelopmental Outcome Collaborative announced an Investigator Award to provide funding support for members to leverage the organization’s infrastructure and registry to conduct research and/or quality improvement projects. As the data registry continues to expand, it will be an ever-growing repository of data for clinical research and quality improvement projects among member institutions. Formal processes are in place such that individual institutions or multi-centre collaborative projects can request to mine data from the registry for clinical research and quality improvement proposals.
Conclusions
The mission of the Cardiac Neurodevelopmental Outcome Collaborative is to develop and implement best practices related to neurodevelopmental outcomes for individuals with pediatric and congenital heart disease through clinical, quality improvement, and research initiatives. The Cardiac Neurodevelopmental Outcome Collaborative is a founding member organization of Cardiac Networks United, and this partnership facilitates efficiency and ease for linking neurodevelopmental outcome data to medical and surgical data collected through the Pediatric Cardiac Critical Care Consortium, the Pediatric Acute Care Cardiology Collaborative, and other learning networks within Cardiac Networks United. We hope this data registry will serve the Cardiac Neurodevelopmental Outcome Collaborative’s mission and facilitate multi-centre collaborations that improve our understanding of the neurodevelopmental and psychosocial challenges experienced by individuals with CHD. Using the Cardiac Neurodevelopmental Outcome Collaborative’s large registry, we hope to gain insight into medical and sociodemographic factors impacting child and family outcomes and quality of life, and to learn from the variability in neurodevelopmental follow-up practices across a growing number of institutions to ultimately improve the quality of care and outcomes.
Acknowledgements
The authors thank ArborMetrix for the development of the data module. The authors appreciate the work of all data collection team members at the member institutions who are contributing data.
Financial support
None.
Competing interest
None.
