Accurate, complete data on congenital cardiac surgery activity is now the expectation of patients, families, payers, government, and even the media. The oversight of this obligation has been appropriately assumed by major cardiothoracic surgery organizations:
• The Society of Thoracic Surgeons in North America,
• The European Association for Cardio-Thoracic Surgery in Europe, and
• The Central Cardiac Audit Database in the United Kingdom.
It has been previously shown that patients not included in medical audit have a worse outcome than those included.Reference Elfstrom, Stubberod and Troeng1 A report from the United Kingdom Central Cardiac Audit DatabaseReference Gibbs, Monro, Cunningham and Rickards2 reveals that hospital databases under-reported 42 operative deaths out of a total of 194 (21.6%). Similarly, the European Association for Cardio-Thoracic Surgery implemented a data verification process and discovered that 7 hospital deaths out of 68 (10.3%) were not reported.Reference Maruszewski, Lacour-Gayet, Monro, Keogh, Tobota and Kansy3 These three manuscriptsReference Elfstrom, Stubberod and Troeng1–Reference Maruszewski, Lacour-Gayet, Monro, Keogh, Tobota and Kansy3 illustrate the need for verification of the completeness and accuracy of data in congenital cardiac registries.
The purpose of this article is to review the current strategies used for verification of the data in the congenital databases of The Society of Thoracic Surgeons, The European Association for Cardio-Thoracic Surgery,Reference Maruszewski, Lacour-Gayet, Monro, Keogh, Tobota and Kansy3 and The United Kingdom Central Cardiac Audit Database.Reference Gibbs, Monro, Cunningham and Rickards2 Because the results of the initial efforts to verify data in the congenital databases of the United KingdomReference Gibbs, Monro, Cunningham and Rickards2 and EuropeReference Maruszewski, Lacour-Gayet, Monro, Keogh, Tobota and Kansy3 have been previously published, this article will provide a more detailed look at the current efforts in North America, which prior to this article have not been published. The discussion and presentation of the strategy for the verification of data in the congenital cardiac surgery database of The Society of Thoracic Surgeons will then be followed by a review of the strategies utilized in the United KingdomReference Gibbs, Monro, Cunningham and Rickards2 and Europe.Reference Maruszewski, Lacour-Gayet, Monro, Keogh, Tobota and Kansy3 The ultimate goal of sharing the information in this article is to provide information to the participants in the databases that track the outcomes of patients with congenitally malformed hearts. This information should help to improve the quality of the data in all of our databases, and therefore increase the utility of these databases to function as a tool to better the management strategies provided to our patients.
Verification of data in the Congenital Heart Surgery Database of the Society of Thoracic Surgeons
Background: Although examination of Society of Thoracic Surgeons congenital data for intrinsic flaws has been conducted by the Duke Clinical Research Institute for several years, it has been recognized that this strategy alone is not adequate to detect many data collection system issues, software problems, or intentional under-reporting of poor outcomes. In 2007, the Society of Thoracic Surgeons Congenital Database administration working with a professional medical audit firm, the Iowa Foundation for Medical Care, implemented a formal on-site audit program.
Methods: Data harvested annually by the Society of Thoracic Surgeons is immediately examined electronically for intrinsic flaws. Certain consistency edits are performed automatically on harvested data. Consistency edits are modifications to field values to make them consistent with other field values in the data record. Consistency edits are performed on a field after comparison between the field and related fields. For example, if the field “Mortality Discharge Status” is “Alive” or missing, but “Mortality Date” is specified and it is between the “Date of Admission” and “Date of Discharge”, the field “Mortality Discharge Status” is changed to “Dead”. Consistency edits are only performed on procedure records where the field “Operation Type” contains either “Cardiopulmonary Bypass” or “No Cardiopulmonary Bypass Cardiovascular”.4 A “Data Quality Report”, which summarizes accepted data and itemizes missing and inconsistent data, is generated and sent by e-mail back to the submitting institution. The participant then makes corrections and resubmits the data. This process can be repeated multiple times during the harvest period.
In 2007, on-site data audits were also conducted. Iowa Foundation for Medical Care controlled and performed the on-site audits accompanied by an experienced congenital cardiac surgeon. Fifty-six North American sites submitted data to the Society of Thoracic Surgeons Congenital Heart Surgery Database in 2007. In late April 2007, Duke Clinical Research Institute randomly selected 20 sites that met the following criteria:
• Active Congenital Database Participant
• At least 30 admissions during the year of the audit, in other words, between January 1, 2006 and December 31, 2006.
The 20 selected sites were numbered consecutively. Five participating centres, beginning with number one, were scheduled for the year and were notified six to eight weeks in advance of the visit. None of the other 15 randomly selected centres were contacted.
Audits were conducted in a collegial manner and were not at all punitive. The focus was always on improving data quality. Each audit had five components:
• A scripted interview is performed with the manager of the database (also known as the “data manager”) and the surgeon responsible for the database. The goal of the scripted interview is to determine policies and practices related to the collection of the data, specifically targeting potential problem areas such as capturing “30 day mortality status”, and complications.
• A comparison is made between the hospital case log and the data submitted to the Society of Thoracic Surgeons database. Discrepancies are investigated and potential cause determined.
• Twenty data elements are abstracted from medical records of 20 randomly selected patients and compared with data submitted to the Society of Thoracic Surgeons database. Data abstracted by the auditor and the surgeon are listed in Table 1. Agreement rates are calculated.
Table 1 List of abstracted data elements.
• All operations with associated mortality are analyzed by the surgeon-auditor. Age, gender, and dates of birth, admission, surgery, and discharge/death are determined and compared with data submitted. The mortality classifications for each operation are independently determined and compared to data base submission. Agreement rates are again calculated. Four fields in the database concerning mortality are examined:
○ “Mortality Discharge Status” (alive or dead),
○ “Status at 30 days after surgery” (alive, dead, or unknown),
○ “Operative Mortality” (yes or no), and
○ “Mortality Assigned to this Operation” (yes or no).
• A summary conference is conducted with the manager of the database and surgeon responsible for the database to review preliminary findings and discuss problem areas and potential solutions. Audit report process is also discussed.
Audits were concerned with database issues only. No serious quality of care problems were inadvertently discovered. In the future, if significant quality concerns arise they will be referred to the appropriate Society of Thoracic Surgeons peer review process.
Results:
Intrinsic data verification
Although inconsistencies of data and automatic edits are an important part of the intrinsic process of verification of data, missing elements of data represent a much greater problem. To partially alleviate this problem, missing “yes/no” variables are assumed to be “no”. Continuous variables, however, cannot be imputed.
In most fields, missing data is decreasing, but significant elements remain problematic. In 2002 “ventilation time data” was missing in 38% of the records. This percentage rose to 42% in 2004, and is still missing from 22% of reported cases in 2006. The percent of participating centres with greater than 5% missing data about ventilation is 55.4% in 2006, but this percentage is an improvement upon the 80% value in 2002. Complication data was missing from 15% of the records in 2003, and although the missing data in this field improved to 11% in 2006, the incomplete data in this field remains a problem for the evaluation of morbidity. Also disturbing is the fact that the number of participants who have more than 5% missing data for post-operative length of hospital stay has risen from 0% in 2002 and 2003 to 3.6% in 2006.5
“Mortality Discharge Status” is currently the most important data element. Fortunately, it is also the field with the least missing data. In 2002, 2.5% of this data was missing, but that value has been less than 1% since 2002, and for 2006 is 0.4% (Fig. 1). The percent of participating centres with greater than 5% missing data for “Mortality Discharge Status” remains a concern. From 10% in 2002, this value dropped to 0% in 2005; however, in 2006, 3.8% of the centres were missing more than 5% of the data for “Mortality Discharge Status”.5
Figure 1 5This graph illustrates the percentage of missing data in the fields “Mortality Discharge Status” (alive or dead) and “Status at 30 days after surgery” (alive, dead, or unknown) in The Society of Thoracic Surgeons Congenital Database by year from 2002 through 2006, inclusive.
Thirty day mortality data has been much more difficult to capture. The percentage of this data element missing has dropped dramatically from 28% in 2002, but remains above the threshold for statistically reliable and useful data at 12% in 2006 (Fig. 1).5
On-site audit verification
1. Scripted interview with data manager and surgeon
Data collection was performed only by the manager of the database in three centres and by multiple personnel in two. In two institutions, all data was entered retrospectively. The remaining programs collected and entered data both concurrently and retrospectively. Four of the five participants that were audited collected data using a paper form, and the data was then entered into the computerized database by the data manager or clerical staff.
Surgeon involvement was quite variable. In one institution, the surgeon was solely responsible for collection of the diagnoses, procedures, complications and other intra and post-operative clinical data. In the remaining centres, the surgeons provided consultative input when asked or when a particularly complicated procedure was performed. One surgeon reviewed all data prior to submission to the Society of Thoracic Surgeons database.
All institutions relied heavily on the perfusion record for weight, height, and perfusion times. Extubation date and time was recovered from nurse’s notes, physician progress notes, or physician orders universally. Three programs recorded complications only while the patient was in the hospital. The other two had developed mechanisms of communication with the clinics to discover complications occurring after discharge.
Mortality status at 30 days was determined by relying on informal communications with referring physicians at two institutions. One of these used “alive” as the default and the other did not. Three centres had a relatively formal mechanism to routinely verify 30 day status either by accessing clinic notes electronically or calling referring doctor’s offices. Two of these three programs used “alive” as the default response.
A formal internal quality control process was employed by only one of the five audited participants. They routinely reviewed one of every ten records in detail for completeness and accuracy. One other centre compared their database with the operating room case log monthly to be certain that all cases were captured.
At all five institutions, an attempt was made to submit their data to the data warehouse early in the period of harvest of the data. The report generated by the Duke Clinical Research Institute about the quality of the data (the “Data Quality Report”) was then referenced by the data manager to identify any discrepancies in submitted data. Any needed corrections were made and the data re-submitted. All centres submitted data multiple times during the most recent harvest, with the number of submissions of data per harvest ranging from three to six.
2. Comparison of hospital case logs with submitted data
The time required to complete this comparison varied from one half hour to three hours. The two most protracted comparisons were due to a mid-year changes in the scheduling process combined with difficulty reconciling nomenclature and designation of the primary procedure.
Two participants had excellent correlation in the comparison of hospital case logs with submitted data. One failed to report only one case which was performed. The other had two patients representing six operations which were not submitted. The latter also reported five cases that were not in the hospital operating room log. It was determined that these procedures were not performed in the operating room.
The three other centres experienced multiple instances of operations entered in the local data base that did not appear on the list of submitted operations provided by Duke Clinical Research Institute. Part of the reason for this discrepancy was incorrect data entry, which caused the records to be rejected by the data warehouse. It is also possible that the software programs were not functioning properly.
There were 16 operations performed in one program that were not included in their submitted data. After investigation, it was found that seven of these were operated at an outside hospital but that the nine were just missed for no apparent reason. No indication of intentional under-reporting or failure to report poor outcomes was identified at any of the five audited centres.
3. Medical record abstraction
All five programs demonstrated excellent agreement rates for demographic, hospitalization, and intra-operative information with the exception of one element of data. Height was recorded correctly only 87% of the time across the five centres. One centre failed to report height in three patients and reported it incorrectly in two others (Table 2).
Table 2 Agreement rates for 17 abstracted data elements.
--.-% Indicates mean agreement rate less than 90%.
Two centres captured post-operative data very well. Extubation date and time was a significant problem for the two others, and centre number 5 only reported it once in the 20 records reviewed. One centre had zero agreement with 30 day mortality status due to missing data. The centre with 10% agreement for this element had a software problem which caused the preposterous reporting of 152 deaths at 30 days (Table 2).
All centres except number 2 transitioned from database version 2.30 or 2.31 to version 2.50 during 2006. This transition seemed to create a problem with reporting of Fundamental Diagnosis since the field is not available in versions 2.30 or 2.31 and is included on the Demographics page rather than with clinical information in version 2.50. Three programs had a significant amount of missing data in this field (Table 2).
Institution number 5 was missing the primary diagnosis on five records. The remaining mismatches across all centres were the result of entering a primary diagnosis which was not appropriate for the primary procedure. Often the fundamental diagnosis was entered as the primary diagnosis when the reason for the operation was clearly a separate diagnosis.
The primary procedure was a missing data element on 10 submitted cases across three centres. The other mismatches occurred when the procedure entered as primary was not the highest complexity procedure performed during the operation (Table 2).
The complications field was left blank in five instances at one program and one at another. Centre number 3 entered “Other Complication” on seven cases where the surgeon auditor could not find a complication worthy of reporting in the medical record. The other 14 mismatches occurred as a result of failure to enter significant complications that were discovered in the medical record (Table 2).
4. Mortality analysis
Centre number 5 had the poorest agreement rate for all elements in the mortality analysis. This disagreement was largely caused by the failure to report the mortality of a patient who had four operations and the failure to report a secondary procedure on another patient. Capture of “30 day mortality status” was a significant problem for three of the five programs. These centres depended largely on informal communication with referring physicians. The successful institutions employed more formal mechanisms to access follow-up information. Only centre number 2 achieved satisfactory agreement for the fields of data named “Operative Mortality” and “Mortality Assigned to this Operation”. The other programs failed to consistently employ definitions recently published by the Society of Thoracic Surgeons Congenital Database Task Force.Reference Jacobs, Mavroudis and Jacobs6
Combined verified data from all 5 centres documented a total of 148 operations associated with mortality involving 55 patients. One patient out of 55 (1.8%) who died in hospital was not correctly reported as a mortality. This patient had four operations. One other secondary operation associated with mortality went unreported, in another patient who was correctly reported as a mortality with the primary operation. A total of five operations out of 148 (3.4%), therefore, were not correctly reported as discharge mortality (Table 3).
Table 3 Mortality analysis agreement rates.
--.-% Indicates mean agreement rate less than 90%.
Discussion: Verification process of The Database of Society of Thoracic Surgeons
The data verification process in the Society of Thoracic Surgeons Congenital Database has for several years involved only intrinsic evaluation of data. On-site audits have just begun and techniques are evolving. Not surprisingly, the problem areas identified by intrinsic data analysis are similar to those uncovered during the audits, but on-site examination of these issues provides much more insight into the causes and potential solutions.
It is universally accepted that if greater than 10% of a data element is missing or corrupt, analysis of that particular field is unreliable. Average agreement rates across the audited programs were well over 90% for elements in the administrative, hospitalization, and intra-operative categories except for one field. The data in the field named “Patient height”, submitted to the Society of Thoracic Surgeons Database, was missing or incorrect in 13% of the records abstracted. This error seemed to occur much more commonly when the field named “Operation Type:” was filled out with “No Cardiopulmonary Bypass Cardiovascular” compared to when this field was filled out with “Cardiopulmonary Bypass”. Perhaps too much reliance is placed on the perfusion record for collection of this data element.
Collection of post-operative data was a significant problem for some centres. Three of the five failed to capture extubation date and time consistently and two had very poor reporting of 30 day mortality. The centres that did well in this category had set up specific mechanisms to actively acquire this data and did not rely on casual communication.
Although one program correctly and completely reported diagnoses and procedures, average agreement rates were poor. Problems capturing fundamental diagnosis are somewhat understandable because it is a new data element in version 2.50. Additionally, it is entered on the demographics page and not with the other clinical data. Often the fundamental diagnosis was listed as the primary diagnosis causing mismatches in that data field. Although the fundamental diagnosis may be the primary diagnosis, the fundamental diagnosis is not always the primary diagnosis of a given operation. A given operation may include multiple component procedures. The primary procedure is the one with the highest Aristotle Basic Complexity Score, that is, the most complex component procedure according to a panel of 50 experts.Reference Lacour-Gayet, Clarke and Jacobs7 Those unfamiliar with the score may not appreciate the relative complexity of procedures which may cause designation of an incorrect primary procedure.
The one program that achieved 90% agreement for reporting of complications also captured 30 day status and extubation date and time. This accomplishment is the result of having an active mechanism in place to collect this data. With the exception of centre number 3 that over reported complications, the remaining mismatches were due to under reporting.
Mortality analysis totals for the five audits revealed 148 operations associated with mortality involving 55 patients. One patient out of 55 (1.8%) who died in hospital was not correctly reported as a discharge mortality. A total of five operations out of 148 (3.4%) were not correctly coded with respect to discharge mortality, but, as discussed above, only one of these was the index operation for that patient. (As discussed below, at the Duke Clinical Research Institute, mortality that occurs for an admission with multiple operations is assigned to the first cardiac operation of that admission, in other words the first operation of that admission in which the field “Operation Type” is filled out with the choices “Cardiopulmonary Bypass” or “No Cardiopulmonary Bypass Cardiovascular”. This initial cardiac operation of the hospitalization is considered the index operation of the hospitalization.)
Mortality status at 30 days was inconsistently captured, and often “alive” was used as the default. A software problem at one centre resulted in the submission of 152 deaths at 30 days; actually there were only 9 patients who died. Specific mechanisms must be implemented to actively acquire the data for the field named “Status at 30 days after surgery” (alive, dead, or unknown). It is not acceptable to simply rely on casual communication.
Operative mortality was frequently not answered yes, or left blank for secondary procedures. The field named “Mortality Assigned to this Operation” was frequently answered no for cases that are not included in the Duke Clinical Research Institute calculation of mortality. Mortality data should be collected and submitted according to the definitions recently published by The Society of Thoracic Surgeons Congenital Database Task Force and the Joint European Association for Cardio-Thoracic Surgery – Society of Thoracic Surgeons Congenital Database Committee:Reference Jacobs, Mavroudis and Jacobs6
• “Mortality Discharge Status” (alive or dead): Was the patient alive or dead regardless of cause upon leaving the hospital? This element should be recorded for each and every cardiothoracic operation.
• “Status at 30 days after surgery” (alive, dead, or unknown): Was the patient alive or dead regardless of cause 30 days following the date of surgery? This element should also be recorded for each and every cardiothoracic operation.
• “Operative Mortality” (yes or no): This element should be marked as “Yes” for any operation whose status was “dead” for either or both of the above two elements – “Mortality Discharge Status” and “Status at 30 days after surgery”. This element should be recorded yes or no for each and every cardiothoracic operation.
• “Mortality Assigned to this Operation” (yes or no): This element should be recorded as “Yes” for the single operation during the hospitalization to which the mortality is attributed and “No” for all other operations. A patient undergoing multiple operations during a given hospitalization may have the field “Operative Mortality” answered “Yes” for each operation; however, the field “Mortality Assigned to this Operation” will be answered “yes” for only one operation. The surgeon will therefore assign the operative mortality to the most appropriate operation.
At the Duke Clinical Research Institute, mortality that occurs for an admission with multiple operations is assigned to the first cardiac operation of that admission, in other words the first operation of that admission in which the field “Operation Type” is filled out with the choices “Cardiopulmonary Bypass” or “No Cardiopulmonary Bypass Cardiovascular”. This initial cardiac operation of the hospitalization is considered the index operation of the hospitalization.
These issues related to assigning mortality are important and are explained further in the following quote taken from a recent publication of The Society of Thoracic Surgeons Congenital Database Task Force and the Joint European Association for Cardio-Thoracic Surgery – Society of Thoracic Surgeons Congenital Database Committee:Reference Jacobs, Mavroudis and Jacobs6
“The patient who has had multiple operations in a given hospitalization raises the next issue related to the numerator in the fraction quantifying death. The STS currently employs the rule that ‘mortality that occurs for a patient with multiple admissions is assigned to the latest admission’, and ‘mortality that occurs for an admission with multiple operations is assigned to the first operation during that admission’. Occasionally the first operation of a given hospital admission may not be the most appropriate operation to assign the mortality; nevertheless, we now believe that less error will result from the application of this rule (ie, assigning mortality to the first operation of an admission) than from a policy that allows the individual surgeon or other data entry personnel to choose the operation to which a given mortality is assigned.”
Thus, the field named “Mortality Assigned to this Operation” is not used at the Duke Clinical Research Institute to assign mortality. This field is, nevertheless, still an element of the current database. In a patient who has had an operative mortality, this field should be marked as “Yes” for the single operation during the hospitalization to which the mortality is attributed and “No” for all other operations. This field may be used by individual institutions for individual programmatic tracking of mortality data, but at the current time, this field is not used at the Duke Clinical Research Institute.
Only operations in which the field “Operation Type” is marked as “Cardiopulmonary bypass” or “No cardiopulmonary bypass cardiovascular” are considered as “Cardiac operations” and are included in the mortality analysis done at the Duke Clinical Research Institute. Operations in which the field “Operation Type” is marked as “Extra-corporeal membrane oxygenation”, “Thoracic”, “Interventional Cardiology”, or “Other” are not considered as “Cardiac operations” and are not included in the mortality analysis done at the Duke Clinical Research Institute. Operations that involve only insertion, maintenance, or removal of a ventricular assist device are also not considered as “Cardiac operations” and are not included in the mortality analysis done at the Duke Clinical Research Institute. These operations involving ventricular assist devices are now best coded as “Other” in the field named “Operation Type”. In the next upgrade of the specifications for the software of the database, an additional operation type will be added that will be termed “Ventricular assist device”. Operations that are marked with the operation type as “Ventricular assist device” will be treated similarly to those operations with the operation type marked as “Extra-corporeal membrane oxygenation”.
Even if an operation is not included in the calculation of programmatic mortality performed by Duke Clinical Research Institute, it should still be counted as a mortality in the individual institutional database. The exclusion of this case in the programmatic mortality calculation will be made by Duke Clinical Research Institute at the time of the analysis of the data. For example, patients weighing less than or equal to 2,500 grams undergoing ligation of the patent arterial duct as their primary procedure are not be included in the mortality calculations in the database reports of The European Association for Cardio-Thoracic Surgery and The Society of Thoracic Surgeons. These cases should, however, still be marked as a mortality in the individual institutional database if they indeed died. A similar example is operations in which the field “Operation Type” is marked as “Extra-corporeal membrane oxygenation”, “Thoracic”, “Interventional Cardiology”, or “Other”, which also are not included in the mortality analysis done at Duke Clinical Research Institute. Even though these operations are currently excluded from the analysis of mortality done for The Society of Thoracic Surgeons at Duke Clinical Research Institute, they should be reported strictly according to the criteria described above. The decision of how to analyze the data rests with the Society of Thoracic Surgeons Congenital Database Taskforce and not with the individual participants. The way the database committee decides to evaluate and report the data should not affect the way the data is entered or submitted.
Data verification audit site visits are very effective at drawing attention to the importance of high quality data. The audits can benefit the participating centres by validating methods that are effective and by identifying ineffective practices and providing suggestions for improvement. The visits can also provide “ammunition” for convincing institutional administration to commit appropriate resources to data management.
An effective data verification process also benefits the Society of Thoracic Surgeons and the congenital heart surgery community. It may forestall audits done using administrative data by individuals or agencies who do not understand congenital heart surgery. Verification of the Society of Thoracic Surgeons data also increases the accuracy and credibility of research done using the congenital database.
Verification of data in the Congenital Heart Surgery Database of the European Association for Cardio-thoracic Surgery
The European Association for Cardio-Thoracic Surgery database has examined all data for internal consistency since its beginning in 1999 and has conducted on-site audits for the past four years. Five audits per year, involving twelve different centres, have examined data from 2003 through 2006 inclusive. The process employed, called “Source Data Verification” reviews records on every operation performed. The number of verified procedures (5,892) represents 18.26% of the total procedures collected by the database over the four years.8
Data elements are examined by comparing original data submitted with corrected data after verification. No statistically significant difference was found for age, cross-clamp time, cardiopulmonary bypass time, hospital length of stay, or patient weight (Table 4).9 Mortality analysis also revealed no significant difference for hospital or 30 day mortality (Table 5).8 Still, in a 2005 publication, 7 hospital deaths out of 68 (10.3%) were not reported.Reference Maruszewski, Lacour-Gayet, Monro, Keogh, Tobota and Kansy3 More recent analysis reveals that 19 hospital deaths out of 234 (8.1%) were not reported and 24 out of 242 thirty-day mortalities (9.9%) were missed.8 No statistical difference can be demonstrated between verified and nonverified data, including the important fields of data named “Mortality Discharge Status” (p = 0.4972) and “Status at 30 days after surgery” (p = 0.3989). Still, the need for verification of data is clearly demonstrated.
Table 4 8European Association for Cardio-Thoracic Surgery Audit Results.
Table 5 8European Association for Cardio-Thoracic Surgery Mortality Audit.
Verification of data in the United Kingdom
Following the Kennedy report on the results of infant congenital cardiac surgery in Bristol, United Kingdom,10 the Central Cardiac Audit Database was established in 1999 to provide national analyses of outcomes after cardiovascular surgery and therapeutic catheterization. Data are collected electronically in an anonymous encrypted format with prospective tracking of mortality and re-intervention using up to a 40 field minimum dataset. The diagnoses and procedures are coded using the 2002 version of the Short List of the European Paediatric Cardiac Code.Reference Franklin, Anderson and Daniëls11 The database is centrally funded by the Department of Health and data submission is compulsory for all centres undertaking congenital cardiac disease interventions. Patients give informed consent for data submission.
The verification process begins at the congenital cardiac centre. Most of the 13 cardiac units in the United Kingdom have database managers who check for data accuracy with medical staff before the data is submitted. Independent validation of the patient’s status (alive or dead) is achieved by central tracking using the linkage of each patient’s National Health Service number to the Office of National Statistics, where the death of every resident in England and Wales is registered. Separate, similar systems exist in Scotland and Northern Ireland.Reference Gibbs, Monro, Cunningham and Rickards2 This system, therefore, allows for patients to be tracked at multiple different centres within the United Kingdom.
In addition, each unit is visited for one or two days each year by a specialist database nurse administrator from the Central Cardiac Audit Database and a volunteer surgeon or cardiologist from another unit. A detailed pre-visit proforma is completed by each centre covering such areas as security and confidentiality, in-house verification and quality assurance, training for data collection and accuracy, communication issues, accountability, health records management, and timeliness of central submission. The visits are scheduled in the year following data submission. At the visit, all operating room and catheter laboratory logbooks are scrutinized to ensure procedural data accuracy and that all procedures have been captured. Also, a random selection of 20 patient hospital records is requested in advance and compared to the dataset submitted for missing or incorrect data. A Data Quality Indicator score is then calculated.12 The results have been encouraging with the scores improving over time from an average of 79% to 91% currently (range 81–98%).13 At the end of the visit, the unit clinicians meet with the auditors to discuss areas of excellence and deficiencies. Within weeks, a formal report is submitted back to the hospital team and to higher management.
The visiting audit team is often able to successfully bring pressure on hospital managers to invest in the provision of manpower and/or quality data entry software to achieve higher standards. The visits are therefore seen by the congenital cardiac clinicians as very positive encounters.
Ideally, every medical record of the approximately 8,000 patients undergoing procedures each year should be examined. Lack of funding and skilled manpower for such an enterprise precludes implementation of this costly “time and resource consuming” strategy at present. It seems preferable to visit each hospital annually rather than visit a limited number of centres and examine all of the patient records. If a unit is under-performing in a particular area the hospital team is asked to re-examine this data element and resubmit their data. Investigations beyond this protocol have not yet been required. Centre specific results are now published on the World Wide Web allowing free access to families and the media.14 An additional incentive, therefore, exists to provide accurate and complete data, knowing that central tracking of mortality provides external monitoring of performance.
Summary and conclusions
The European Association for Cardio-Thoracic Surgery and the Central Cardiac Audit Database of the United Kingdom have developed relatively sophisticated and comprehensive systems that take advantage of their respective resources. The Society of Thoracic Surgeons is beginning an overall audit process, which is evolving and improving. On-site auditing of nearly 10% of participating centres per year is respectable and credible. In 2007, The Society of Thoracic Surgeons Congenital Heart Surgery Database audited five out of 56 (8.9%) of the North American sites who submitted data in 2007. Meanwhile, The Society of Thoracic Surgeons Adult Cardiac Surgery Database audits approximately 3% of participating centres. System problems which are not solely the responsibility of the participants are being identified and rectified. The resulting improvement of the Society of Thoracic Surgeons database will further increase the credibility of outcome research and performance evaluation.
The need for accurate, complete and high quality Congenital Heart Surgery outcome data has never been more pressing. The public interest in medical outcomes is at an all time high and “pay for performance” is looming on the horizon. Information found in administrative databases is not risk or complexity adjusted, notoriously inaccurate, and far too imprecise to evaluate adequately performance in congenital cardiac surgery. The Society of Thoracic Surgeons and European Association for Cardio-Thoracic Surgery databases contain the elements needed for assessment of quality of care provided that a mechanism exists within these organizations to guarantee the completeness and accuracy of the data. The Central Cardiac Audit Database in the United Kingdom has an advantage in this endeavour with the ability to independently track and verify mortality through their National Health Service.
A combination of site visits with “Source Data Verification”, in other words, verification of the data at the primary source of the data, and external verification of the data from independent databases or registries, such as governmental death registries, may ultimately be required to allow for optimal verification of data.Reference Jacobs, Wernovsky and Elliott15 Further research in the area of verification of data is also necessary. Data must be verified for both completeness and accuracy.
Acknowledgement
The Audits of The Society of Thoracic Surgeons Congenital Database are funded by the Society of Thoracic Surgeons. We thank Victoria Polich (Audit Director) and Pam Beck (Data Auditor) from the Iowa Foundation for Medical Care for organizing, coordinating, performing and reporting The Society of Thoracic Surgeons Congenital Database on-site audits referred to in this manuscript.
We thank The Children’s Heart Foundation (http://www.childrensheartfoundation.org/) for financial support of this research. We also acknowledge that The Society of Thoracic Surgeons Congenital Database Audits are funded by the Society of Thoracic Surgeons.
This manuscript was reviewed by the Access and Publications Committee of the Database of the Society of Thoracic Surgeons and approved for publication in this supplemental issue of Cardiology in the Young.
