Quality appraisal of clinical nutrition practice guidelines for critically ill adult patients: a systematic review using the Advancing Guideline Development, Reporting and Evaluation in Health Care instrument II (AGREE-II) and AGREE-Recommendation Excellence (AGREE-REX)

Abstract

Nutritional therapy should follow evidence-based practice, thus several societies regarding nutrition and critical care have developed specific Clinical Practice Guidelines (CPG). However, to be regarded as trustworthy, the quality of the CPG for critically ill patients and its recommendations need to be high. This systematic review aimed to appraise the methodology and recommendations of nutrition CPG for critically ill patients. We performed a systematic review (protocol number CRD42020184199) with literature search conducted on PubMed, Embase, Cochrane Library and other four specific databases of guidelines up to October 2021. Two reviewers, independently, assessed titles and abstracts and potentially eligible full-text reports to determine eligibility and subsequently four reviewers appraised the guidelines quality using the Advancing Guideline Development, Reporting and Evaluation in Health Care instrument II (AGREE-II) and AGREE-Recommendation Excellence (AGREE-REX). Ten CPG for nutrition in critically ill patients were identified. Only Academy of Nutrition and Dietetics and European Society of Intensive Care Medicine had a total acceptable quality and were recommended for daily practice according AGREE-II. None of the CPG recommendations had an overall quality score above 70 %, thus being classified as moderate quality according AGREE-REX. The methodological evaluation of the critically ill adult patient CPG revealed significant discrepancies and showed a need for improvement in its development and/or reporting. In addition, recommendations about nutrition care process presented a moderate quality.

Critical illness is associated with intense catabolism that is driven by a systemic inflammatory response⁽¹⁾, and nutrition therapy studies in this scenario have advanced over the years demonstrating that nutritional interventions could impact the clinical course of critical illness. Nutritional support could not only provide benefits but also may cause harm, thus it is part of therapies that should impact patient outcomes⁽²⁾. Nutritional therapy should follow evidence-based practice and involves three fundamental principles: awareness of the best available evidence, clinical experience and patient values and preferences⁽³⁾. In this sense, rigorously developed clinical practice guidelines (CPG) provide support to healthcare professionals by establishing standards across the healthcare continuum (including screening and diagnosis), supported by the strongest scientific evidence available. The development of high-quality CPG is a challenging task, and for it to be trustworthy, it should be unbiased, scientifically valid and incorporate grading systems for characterising quality of available evidence and strength of clinical recommendations⁽⁴⁾.

The AGREE collaboration defines guideline quality as ‘confidence that the potential biases inherent in guideline development have been addressed adequately and that the recommendations are both internally and externally valid, and are feasible for practice’⁽⁵⁾. The Appraisal of Guidelines for Research & Evaluation II (AGREE II) is a validated and reliable tool specially designed to assess the issue of variability in guideline quality^(5,6) , through which many critical appraisals of guidelines have been performed and published^(7–9). AGREE II has been shown to be the most widely used CPG assessment tool in a systematic review of twenty-four different tools⁽¹⁰⁾. Although meeting rigorous methodological requirements is necessary, it is not sufficient to ensure that guideline recommendations are clinically credible or implementable. Thus, AGREE-Recommendation Excellence (AGREE-REX) was proposed to address this gap and assess the quality of guideline recommendations, which is also used as a strategy to inform their development and reporting⁽¹¹⁾.

Several societies regarding nutrition and critical care in different countries have developed specific guidelines. A systematic review published in 2016 aimed to appraise the methodological quality of critically ill patients’ CPG using AGREE II and demonstrated overall suboptimal CPG quality in this field, i.e. only four of nine CPG were rated as ‘recommended.’ Major deficiencies were found in applicability, editorial independence, stakeholder involvement and rigor of development domains⁽¹²⁾. A lack of globally standardised procedures implies a vastly heterogeneous clinical management of nutrition therapy in the intensive care setting. Biased and divergent recommendations between different guidelines can also create a dilemma for professionals who choose not to follow the recommendations they distrust⁽¹³⁾.

In the past five years, at least four CPG have been published^(14–17) in which updated evaluations of the guidelines for the nutrition care of critically ill patients are needed. Moreover, to the best of our knowledge, there is no systematic review that has applied AGREE-REX to the clinical nutrition practice guidelines for critically ill adult patients. In order to provide valuable information for clinical practice and for the further development of CPG, the current systematic review aimed to critically appraise the methodology and recommendations of clinical nutrition CPG for critically ill patients.

Methods

Design and registration

The current systematic review was conducted based on the Cochrane Handbook recommendations⁽¹⁸⁾ and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses checklist⁽¹⁹⁾. The protocol of this study was registered in the International Prospective Register of Systematic Reviews (identifier: CRD42020184199).

Eligibility criteria

The most up-to-date version of CPG for critically ill adults that addressed recommendations for nutrition therapy were included if they (1) employed a grading system to rate the quality of evidence; (2) were published in a peer-reviewed journal or in a guideline database; (3) were the last version of CPG published and (4) were intended for health professionals and had included recommendations.

CPG were excluded if they were exclusively dedicated to the paediatric patient population. We also excluded position or consensus statements, as well as commentaries that summarised the evidence from a published CPG and made recommendations according to local factors. CPG with unavailable text were excluded

Clinical practice guideline identification

The literature search to identify guidelines according to the inclusion criteria was performed in May 2020, and the last update was conducted on October 2021.

Search strategy

The current systematic review addresses two research questions:

1. What is the quality of the evidence-based nutrition practice guidelines for critically ill adult patients?

2. What is the quality of the recommendations of the evidence-based nutrition practice guidelines that meet a minimum methodological threshold of moderate and high quality in AGREE II?

It was constructed according to the ‘PICAR’ framework – P: Population, clinical condition; I: Interventions, C: Comparator, A: Attributes of eligible CPG and R: Recommendation characteristics⁽²⁰⁾, which are detailed in Table 1.

Table 1. PICAR criteria for inclusion of Clinical Practice Guidelines (CPG)

Search database

The search for CPG was performed in PubMed, Embase, Cochrane Library; National Institute for Health and Clinical Excellence; National Guideline Clearinghouse – Agency of Healthcare Research and Quality until July 2018, since it was closed in that moment; the Intercollegiate Guidelines Network (SIGN) and Guideline International Network databases, with no restrictions on language or date of publication, using specific keywords according to each database.

Newly published articles were screened every month until the final version of the manuscript was submitted. The references of all eligible CPG for additional relevant guidelines were screened, and we performed a manual search through other potentially relevant guideline databases, such as the American Society for Parenteral and Enteral Nutrition (ASPEN), European Society for Clinical Nutrition and Metabolism, Federación Latinoamericana de Terapia Nutricional, Nutrición Clínica y Metabolismo and Parenteral and Enteral Nutrition Society of Asia.

Box 1 presents the full electronic search strategy used in PubMed. The search strategy adopted for Embase and the Cochrane Library is available in Online Supplements 1 and 2.

Box 1: Full PubMed electronic search strategy

(((((((((((((((((((((Critical Care[Title/Abstract]) OR Care, Critical[Title/Abstract]) OR Intensive Care[Title/Abstract]) OR Care, Intensive[Title/Abstract]) OR Surgical Intensive Care[Title/Abstract]) OR Care, Surgical Intensive[Title/Abstract]) OR Intensive Care, Surgical[Title/Abstract]) OR Intensive Care Units[Title/Abstract]) OR Care Unit, Intensive[Title/Abstract]) OR Care Units, Intensive[Title/Abstract]) OR Intensive Care Unit[Title/Abstract]) OR Unit, Intensive Care[Title/Abstract]) OR Units, Intensive Care[Title/Abstract]) OR Critical Illness[Title/Abstract]) OR Critical Illnesses[Title/Abstract]) OR Illness, Critical[Title/Abstract]) OR Illnesses, Critical[Title/Abstract]) OR Critically Ill[Title/Abstract]) OR Critically Ill Patients[Title/Abstract])) AND (((((((((((((((((((((((((((((((((((((((((((((((Nutrition Therapy[Title/Abstract]) OR Therapy, Nutrition[Title/Abstract]) OR Medical Nutrition Therapy[Title/Abstract]) OR Nutrition Therapy, Medical[Title/Abstract]) OR Therapy, Medical Nutrition[Title/Abstract]) OR Nutritional Support[Title/Abstract]) OR Support, Nutritional[Title/Abstract]) OR Artificial Feeding[Title/Abstract]) OR Feeding, Artificial[Title/Abstract]) OR Enteral Nutrition[Title/Abstract]) OR Nutrition, Enteral[Title/Abstract]) OR Enteral Feeding[Title/Abstract]) OR Feeding, Enteral[Title/Abstract]) OR Force Feeding[Title/Abstract]) OR Feeding, Force[Title/Abstract]) OR Feedings, Force[Title/Abstract]) OR Force Feedings[Title/Abstract]) OR Tube Feeding[Title/Abstract]) OR Feeding, Tube[Title/Abstract]) OR Gastric Feeding Tubes[Title/Abstract]) OR Feeding Tube, Gastric[Title/Abstract]) OR Feeding Tubes, Gastric[Title/Abstract]) OR Gastric Feeding Tube[Title/Abstract]) OR Tube, Gastric Feeding[Title/Abstract]) OR Tubes, Gastric Feeding[Title/Abstract]) OR Parenteral Nutrition[Title/Abstract]) OR Nutrition, Parenteral[Title/Abstract]) OR Parenteral Feeding[Title/Abstract]) OR Feeding, Parenteral[Title/Abstract]) OR Feedings, Parenteral[Title/Abstract]) OR Parenteral Feedings[Title/Abstract]) OR Intravenous Feeding[Title/Abstract]) OR Feeding, Intravenous[Title/Abstract]) OR Feedings, Intravenous[Title/Abstract]) OR Intravenous Feedings[Title/Abstract]) OR Nutrition Assessment[Title/Abstract]) OR Assessments, Nutrition[Title/Abstract]) OR Nutrition Assessments[Title/Abstract]) OR Nutritional Assessment[Title/Abstract]) OR Assessment, Nutritional[Title/Abstract]) OR Assessments, Nutritional[Title/Abstract]) OR Nutritional Assessments[Title/Abstract]) OR Assessment, Nutrition[Title/Abstract]) OR Nutrition care procedures[Title/Abstract]) OR Nutrition[Title/Abstract] OR Diet[Title/Abstract]))) AND (((Clinical pathway[mh] OR Clinical protocol[mh] OR Consensus[mh] OR Consensus development conferences as topic[mh] OR Critical pathways[mh] OR Guidelines as topic [Mesh:NoExp] OR Practice guidelines as topic[mh] OR Health planning guidelines[mh] OR guideline[pt] OR practice guideline[pt] OR consensus development conference[pt] OR consensus development conference, NIH[pt] OR position statement*[tiab] OR policy statement*[tiab] OR practice parameter*[tiab] OR best practice*[tiab] OR standards[ti] OR guideline[ti] OR guidelines[ti] OR ((practice[tiab] OR treatment*[tiab]) AND guideline*[tiab]) OR CPG[tiab] OR CPGs[tiab] OR consensus*[tiab] OR ((critical[tiab] OR clinical[tiab] OR practice[tiab]) AND (path[tiab] OR paths[tiab] OR pathway[tiab] OR pathways[tiab] OR protocol*[tiab])) OR recommendat*[ti] OR (care[tiab] AND (standard[tiab] OR path[tiab] OR paths[tiab] OR pathway[tiab] OR pathways[tiab] OR map[tiab] OR maps[tiab] OR plan[tiab] OR plans[tiab])) OR (algorithm*[tiab] AND (screening[tiab] OR examination[tiab] OR test[tiab] OR tested[tiab] OR testing[tiab] OR assessment*[tiab] OR diagnosis[tiab] OR diagnoses[tiab] OR diagnosed[tiab] OR diagnosing[tiab])) OR (algorithm*[tiab]))

Process of Clinical Practice Guideline selection and data extraction

The EndNote reference manager software program (version X7·17, New York City: Thomas Reuters, 2011) was used to coordinate the review and tracking process. Two trained reviewers independently screened the titles and abstracts (AC and PPT) and subsequently evaluated the full-text versions of all potentially relevant articles as well as the supporting materials of each included CPG. A third reviewer (FMS) resolved all cases of disagreement.

Data extraction was performed using Google Forms® and exported to Microsoft Office Excel®. It was guided by a standardised electronic form and independently performed by two reviewers (AC and PPT). Disagreements were discussed, analysed and resolved through the arbitration of a third reviewer (FMS). The characteristics extracted from each CPG included journal, publication year, location, society, first author’s name, first version of the guideline or its update, target audience, registered dietitian in the development team, methodology development, type of meta-analysis (direct, indirect only or mixed evidence) and the grading system to rate the quality of evidence. The main recommendations for nutrition care process steps were also extracted: nutrition risk screening, nutrition assessment and diagnosis, start of nutrition therapy (time, route and dose), monitoring tolerance and adequacy of nutrition therapy, selection of appropriate enteral formulation, parenteral nutrition (PN) and nutritional therapy for specific clinical conditions. Authors were contacted in case of doubts regarding the methodology of the development of their CPG by mail at three different instances, and if they did not reply, the available information in the publication was considered.

Quality assessment of clinical practice guidelines and their recommendations

The English version of the most recent version of the AGREE II tool (2017) was employed to evaluate the quality of the CPG. The tool comprises the following six domains: (1) scope and purpose, (2) stakeholder involvement, (3) rigour of development, (4) clarity and presentation, (5) applicability and (6) editorial independence. The domains were then refined to twenty-three items in total, and each of them was rated on a seven-point agreement scale from 1 (strongly disagree) to 7 (strongly agree)⁽²¹⁾. Online Supplement 3 material provides information on all questions evaluated for each domain.

Each CPG was independently rated by four reviewers (AC, PPT, ICE and FMS), who held degrees in clinical nutrition and had experience in performing systematic reviews and methodological evaluations of literature. All reviewers were trained to apply the AGREE II tool by studying its manual and performing the online training offered by the AGREE PLUS platform. Reviewers first read the full version of the CPG and reviewed all relevant information regarding the guideline development process, including the supplementary material related to the CPG. Using two clinical guidelines^(17,22) , a pilot appraisal trial was performed in order to rehearse the experience of using AGREE II, and then the results were discussed. All reviewers were instructed to share relevant supplementary materials with the group to ensure that all reviewers could evaluate the same information.

The score for each domain was obtained by summing all the scores of each reviewer for all items in a domain, and then standardised as follows: (obtained score – minimum possible score)/(maximum possible score – minimum possible score), and an example is provided in Online supplements 4 ⁽²²⁾. One reviewer summed all scores, and each domain score was calculated and presented as percentage in a table of results. Considering that the AGREE II manual does not provide cut-off points for the interpretation of the scores, after the assessment of the twenty-three items and the comprehensive judgment by reviewers, the selected guidelines were evaluated by their quality, followed by the decision if the CPG were to be recommended for use in clinical practice. For overall quality assessment, mean domain scores were categorised as good (≥ 80 %), acceptable (60 %–79 %), moderate (40 %–59 %) or low (< 40 %), as adopted in previous publications^(7,9) . With regard to these recommendations, the following criteria were considered: if four of the six domains were categorised as ≥ 60 %, including domain 3 (rigor of development), the CPG was recommended; if at least two domain scores were categorised above 60 %, the CPG was recommended with modifications and if three of the six domain scores were categorised as less than 30 % or none of the domains were above 60 %, the CPG was not recommended⁽⁷⁾. We opted by these criteria since we believe that the rigor of development is a crucial domain to determine the quality of a guideline.

As a complement to AGREE II, the AGREE-REX tool was used for quality evaluation of the CPG recommendations that meet a minimum methodological threshold of moderate and high quality and were at least recommended (with or without modifications) in AGREE II. This tool comprises the following three domains: (1) clinical applicability, (2) values and preferences and (3) implementability. The domains were then refined to nine items in total, and each of them was rated on a seven-point agreement scale from 1 (strongly disagree) to 7 (strongly agree)⁽¹¹⁾. A detailed description of all items is provided in the online supplement 3.

AGREE-REX was applied to a group of recommendations (steps of nutrition care processes) defined a priori by four independent reviewers (AC, PPT, ICE and FMS) that also applied the AGREE II tool. The evaluations were performed independently, and evaluators were blinded to the other evaluators’ assessments. The scoring system for each domain was equal to that of AGREE II, as described previously and represented in Online supplements 4. For overall quality assessment, the cut-off points suggested by the AGREE-REX manual were adopted: guidelines with overall scores > 70 % were defined as high quality, those with overall quality scores < 30 % as lower quality, and all others as moderate quality⁽¹¹⁾.

Data synthesis

Data were analysed using the Statistical Package for the Social Sciences (SPSS) version 22.0. The means and standard deviations or medians and interquartile amplitudes (according to the domain distribution) were computed for the domain scores and overall scores. The normality of the domains was tested using the Shapiro Wilk test.

The inter-rater agreement was computed using the intraclass correlation coefficient with a two-way random effects model for each domain and total score. The level of agreement (intraclass correlation coefficient) was classified according to commonly cited cut-offs as follows: poor (< 0·40), fair (0·40–0·59), good (0·60–0·74) or excellent (0·75–1·00)⁽²³⁾.

Results

Selection and general characteristics of eligible Clinical Practice Guidelines

A total of 7358 articles were initially identified through database searches, of which 834 were duplicates. Additional records through manual search and review of grey literature amounted to five articles. Figure 1 illustrates the flow of the guideline selection. The full texts of twenty-four articles were assessed for eligibility, and eighteen of them were included in the current systematic review^{(14–17,22,24–36)} , which comprised nine CPG for critically ill adults addressing recommendations for the nutrition care process. The guidelines for specialised nutritional and metabolic support in critically ill patients published by the Spanish Society of Intensive Care Medicine and Coronary Units-Spanish Society of Parenteral and Enteral Nutrition (SEMICYUC-SENPE) comprised ten articles^(24–33). Considering that newly published articles were screened every month until the final version of the manuscript was submitted, more one guideline of ASPEN published in January 2022 had its eligibility checked⁽³⁷⁾. It was included in the current review; however, the previous guideline of ASPEN published in 2016 was not removed because the most recent guideline restricted the GRADE process to five questions that trials had explored and the version published in 2016 were more complete in relation to the recommendations for more steps of nutrition care process and differed in the methods.

Fig. 1. Study flow diagram.

The CPG included were published between 2012 and 2021 in six different countries, most of them in the USA (four CPG)^{(14,22,36,37)} and in the English language (eight CPG)^{(14,16,17,22,24–34,36,37)} . Only the European Society of Intensive Care Medicine (ESICM) guideline⁽¹⁴⁾ was an original version. Most CPG reported funding^{(14,16,22,24–34,36)} , based their recommendations in meta-analyses^{(14–17,22,24–34,36,37)} , and used the GRADE system to grade their evidence^{(14,15,17,22,24–33,35,37)} . In nine guidelines, the names of all authors had been described, and the mean number of authors was 16 (ranging from six⁽³⁵⁾ to 24⁽¹⁴⁾). Eight CPG listed the references used in its content. The mean number of CPG references was 293 (ranging from 42⁽³⁷⁾ to 502⁽¹⁶⁾). The general features of eligible CPG are presented in Table 2.

Table 2. General features of eligible Clinical Practice Guidelines (CPG)

Academy of Nutrition and Dietetics (AND); American Society for Parenteral and Enteral Nutrition (ASPEN); Association of the Scientific Medical Societies in Germany (AWMF); Canadian Critical Care Society (CCCS); Canadian Institutes of Health Research (CIHR); Canadian Society for Clinical Nutrition (CSCN); European Society for Clinical Nutrition and Metabolism (ESPEN); European Society of Intensive Care Medicine (ESICM); German Society for Nutritional Medicine (DGEM); Grading of Recommendations Assessment, Development and Evaluation (GRADE); International Fluid Academy (IFA); Not Reported (NR); Randomized Controlled Trials (RCT); Registered Dietitian (RD); Registered Nurses (RN); Sociedad Española de Medicina Intensiva, Crít ica y Unidades Coronarias (SEMICYUC); Sociedad Española de Nut rición Parenteral y Enteral (SENPE); Sociedade Brasileira de Nutrição Parenteral e Enteral (BRASPEN); Société francaise d’anesthésie et de réanimation (SFAR); Society of Critical Care Medicine (SCCM).

Quality appraisal of Clinical Practice Guidelines

Table 3 presents the AGREE II quality scores for each of the ten CPG included in the current review. The mean domain scores for overall quality of the CPG was equal to 48·3 % (sd = 16·5). None of the CPG had an overall quality score above 80 %, two CPG scored between 60 and 79 %^(14,36) , five CPG scored between 40 and 59 %^{(16,17,22,34,37)} and three CPG scored below 40 %^{(15,24–33,35)} . Only two CPG achieved acceptable quality and were recommended^(14,36) , while four were recommended with modifications^{(16,17,22,37)} , and four were not recommended for use in clinical practice^(15,24–35) . Online Supplement 5 presents the individual scoring of AGREE II for each CPG.

Table 3. Standardized scores of each domain by Appraisal of Guidelines for Research & Evaluation II (AGREE II) of Clinical Practice Guidelines (CPG)

Academy of Nutrition and Dietetics (AND); American Society for Parenteral and Enteral Nutrition (ASPEN); European Society for Clinical Nutrition and Metabolism (ESPEN); European Society of Intensive Care Medicine (ESICM); German Society for Nutritional Medicine (DGEM); Intraclass correlation coefficient (ICC); Sociedad Española de Medicina Intensiva, Crítica y Unidades Coronarias (SEMICYUC); Sociedad Española de Nut rición Parenteral y Enteral (SENPE); Sociedade Brasileira de Nutrição Parenteral e Enteral (BRASPEN); Société francaise d’anesthésie et de réanimation (SFAR); Society of Critical Care Medicine (SCCM).

The domains with higher scores were the ‘clarity of presentation’ domain (domain 4), with a mean score of 81·1 %, and the ‘editorial independence’ domain (domain 6), with a median score of 69·4 %. The lowest score was in the ‘applicability’ domain (domain 5), with a median score of 8·1 %.

The inter-rater reliability was excellent for the overall mean score (> 0·900), and it was higher than 0·750 for the majority of the domains. A description of the evaluation performed for each domain of AGREE II is presented below.

The quality scores for domain 1 (scope and purpose) ranged from 13·9 %⁽³⁵⁾ to 86·1 %⁽³⁷⁾, and the scores of the six CPG were greater than 60 % for this domain^{(14,16,17,34,36,37)} . Low scores were given to four CPG^{(15,22,24–33,35)} due to the lack of providing detailed and adequate descriptions of overall objectives, health questions and clarity as to which population the CPG was meant to be applied to.

For domain 2 (stakeholder involvement), it ranged from 8·3 %⁽¹⁵⁾ to 79·6 %⁽³⁶⁾. The majority of the CPG did not provide adequate information about expertise and description of the member’s role in the development of the guideline, as well as the patients’ preferences, so they were assigned low scores. Only the SEMICYUC-SENPE^(24–33), DGEM⁽¹⁶⁾, ASPEN^(22,37) and AND⁽³⁶⁾ guidelines described the target users. AND was a unique CPG that reached a score higher than 60 % in this domain⁽³⁶⁾.

The AGREE II quality scores for domain 3 (rigor of development) ranged from 17·7 %⁽¹⁵⁾ to 81·9 %⁽³⁶⁾. Eight CPG scored less than 60 % in this domain^{(15–17,22,24–35,37)} . Procedures for updates, external reviews and lacks in the descriptions of the systematic search methods and criteria for selecting the evidence were the most common weaknesses across all included CPG. Only two CPG received a score higher than 60 %: ESICM⁽¹⁴⁾ and AND⁽³⁶⁾.

The quality scores of domain 4 (clarity of presentation) ranged from 61·1 %^(24–33) to 94·4 %⁽¹⁴⁾. This domain was well addressed in all CPG, with scores higher than 60 % for all CPG.

On the other hand, the quality scores for domain 5 (applicability) ranged from 0 %^(15,24–33) to 47·2 %⁽³⁶⁾. The AND guideline received the highest score; however, it did not achieve a score of 60 %⁽³⁶⁾. Most CPG did not explicitly mention barriers to its application and did not provide advice and/or tools to assist its application in clinical practice. In addition, the potential resource implications of applying the recommendations and monitoring criteria were not described. All CPG received scores below 60 %.

Finally, the quality scores for domain 6 (editorial independence) ranged from 0 %^(15,35) to 91·7 %⁽¹⁴⁾. This domain yielded scores above 60 % for most CPG, except for the Canadian CPG⁽³⁴⁾, SFAR⁽³⁵⁾ and BRASPEN⁽¹⁵⁾ – competing interests and funding were poorly addressed in these guidelines.

Quality appraisal of Clinical Practice Guideline recommendations

Table 4 presents the AGREE-REX quality scores for each of the six^{(14,16,17,22,36,37)} CPG classified as moderate/high quality and were at least recommended (with or without modifications) by AGREE II in the current review. None of the CPG recommendations had an overall quality score above 70 %, and all CPG scored between 30 and 70 %, thus resulting in these being classified as moderate quality. Online Supplement 6 presents the individual scoring of AGREE-REX for each CPG. Recommendations made by these four guidelines regarding the nutrition care process of critically ill patients are presented in Online Supplement 7. Recommendations made by these guidelines regarding nutrition therapy for specific conditions in critically ill patients are presented in Online Supplement 8.

Table 4. Standardized scores of each domain of AGREE-Recommendation Excellence (AGREE-REX) of high and moderate quality and recommended Clinical Practice Guideline (CPG)

Academy of Nutrition and Dietetics (AND); American Society for Parenteral and Enteral Nutrition (ASPEN); European Society for Clinical Nutrition and Metabolism (ESPEN); European Society of Intensive Care Medicine (ESICM); German Society for Nutritional Medicine (DGEM); Intraclass Correlation Coefficient (ICC); Society of Critical Care Medicine (SCCM).

The quality scores for domain 1 (Clinical Applicability) ranged from 63·0 %⁽¹⁶⁾ to 86·1 %⁽¹⁴⁾; the scores of four CPG were greater than 70 % for this domain^{(14,17,36,37)} .

The quality scores for values and preferences (domain 2) ranged from 2·1 %⁽²⁾ to 21·9 %⁽¹⁴⁾; no CPG recommendations received scores above 70 %. The lack of adequate descriptions of the values and preferences of target users, decision-makers and guideline developers resulted in low scores. Patient preference items were scarcely addressed in the ESICM⁽¹⁴⁾ and AND⁽³⁶⁾ recommendations.

Regarding the implementability (domain 3), the quality scores ranged from 35·4 %⁽³⁷⁾ to 72·2 %⁽³⁶⁾. The AND guideline received the highest score and was the only one that achieved a score above 70 %⁽³⁶⁾. The other CPG recommendations did not explicitly mention local applications (i.e. provide tools and resources to facilitate the implementation of the recommendations as well as mentioning barriers to its application).

Discussion

Principal findings

In this systematic review, we conducted a methodological evaluation and overall assessment of nutritional care procedures of nutritional support CPG for critically ill patients using the AGREE II and AGREE-REX tools. Ten CPG that provided recommendations for at least one of the steps of the nutrition care process were identified. According to AGREE II, only the ESICM⁽¹⁴⁾ and AND⁽³⁶⁾ guidelines achieved acceptable quality and were recommended. None of the CPG recommendations had a high overall quality according to the AGREE-REX tool.

Quality of guidelines according to Appraisal of Guidelines for Research & Evaluation II

Lower quality scores were observed in the ‘applicability’, ‘stakeholder involvement’ and ‘rigor of development’ domain methodologies. According to previous CPG systematic reviews, the ‘rigor of development’ domain is considered crucial to produce evidence-based recommendations^(7,9,38) . However, in the current study, eight of ten CPG did not achieve a minimum score of 60 % in this domain^{(15–17,22,24–35,37)} . Similarly, four systematic reviews of CPG observed lower scores in the methodology domains. The mean reasons for decreases in the CPG’ scores were they were not based on systematic reviews; there was no clear criteria used to select the evidence; there was no link between the supporting evidence and the recommendations; there was not an external review; there was no involvement of all the required members in the guideline development group or they lacked descriptions of the procedures for updating the guidelines^(8,9,38,39) . ‘Rigor of development’ is a key component in the process of developing a CPG, and it is linked with confidence and credibility⁽⁴⁰⁾. CPG would benefit from a more rigorous and standardised methodology and quality of evidence while preventing the use of ambiguous recommendations (i.e. ‘may be used’ or ‘uncertain recommendations’)⁽⁴¹⁾.

Similar to other studies conducted for quality evaluation of the pharmacological management of chronic diseases in primary care⁽³⁸⁾, diabetes mellitus guidelines created by Chinese researchers⁽³⁹⁾, asthma guidelines⁽⁴⁰⁾, rehabilitation after anterior cruciate ligament reconstruction⁽⁴¹⁾ and physical activity and safe movement in osteoporosis⁽⁴²⁾ In CPG, we observed that the ‘applicability’ domain was the domain with the lowest mean score (8·1 %). Making CPG easy to implement is a crucial step towards increasing their rate of use in clinical practice⁽⁴²⁾. Gao et al. suggested that most guideline development agencies ignored their applications, favouring the use of relevant supporting documents and as emphasising the promotion and hindrance factors in the application process⁽³⁹⁾ instead. The potential resource implications of applying the recommendations were also poorly reported by the CPG for critically ill patients.

The stakeholder involvement domain addresses the degree to which the guideline represents the views of its intended users⁽²¹⁾. The implementation of CPG requires both the contribution and experience of the multidisciplinary medical team and the patients’ and/or family’ points of view⁽⁴⁰⁾. The Canadian CPG reinforces that patients’ perspectives could not be elicited because of the inability of most critically ill patients to participate in discussions about their nutrition, due to the therapy and factors such as sedation and/or mechanical ventilation⁽³⁴⁾. However, it is possible to access patient values and preferences via family and/or studies addressing this point of view. This aspect is crucial because the experiences of critically ill patients are linked with the quality of care in the ICU⁽⁴³⁾.

Low scores in the domains cited above are a problem for the overall quality of CPG. This was confirmed by a study that evaluated 206 Japanese CPG published between 2011 and 2015, which showed using a regression model that domain 3 (rigor of development), domain 4 (clarity of presentation), domain 5 (applicability) and domain 6 (editorial independence) all influenced the overall assessments of CPG. It was also revealed that item 8 (the criteria for selecting the evidence are clearly described), item 15 (the recommendations are specific and unambiguous), item 19 (the guideline provides advice and/or tools on how the recommendations can be put into practice) and item 22 (the views of the funding body have not influenced the content of the guideline) significantly influenced the overall assessment. These four items could explain 72·1 % of the variance⁽⁴⁴⁾ observed and should be improved during the development of CPG.

Quality of guideline recommendations according to the AGREE-Recommendation Excellence tool

In the current review, the CPG included in the AGREE-REX evaluation did not meet the threshold of high quality. Lower quality scores were observed in the ‘values and preferences’ domain, which was also observed in the few reviews that applied the AGREE-REX tool^(45–47). The AGREE-REX user manual reinforces the importance of considering target users, patients/population, decision-makers and guideline developers’ values and preferences⁽¹¹⁾. As an alternative to access this information, Gillespie et al. highlighted CPG that conducted an online survey of patient consumers and made stakeholders’ drafts of recommendations available as notable exceptions and point out that CPG are more implementable when they address these aspects⁽⁴⁵⁾.

Although barriers exist for the inclusion of stakeholder groups in the development process, successful engagement can be obtained using approaches such as discussion and knowledge exchange between groups (e.g. multiple stakeholders in small panels), practical support (e.g., online or face-to-face meetings) and reassessment and feedback⁽⁴⁸⁾. As an alternative for future CPG, a checklist was developed to help identify specific operational strategies to meet AGREE-REX quality criteria a priori. It can allow the CPG development group to prioritise when there is an absence of rigorous and feasible operational methods so that efforts can be directed to address these gaps⁽⁴⁹⁾.

Quality of guidelines on nutrition care processes

A systematic review of nine CPG for nutrition of critically ill patients published in 2016 recommended four CPG; three were recommended with modifications, while two were not recommended⁽¹³⁾. We agree with this publication in recommending the AND⁽³⁶⁾ and not recommending the SFAR CPG⁽¹³⁾. On the other hand, the authors recommended the ASPEN and Canadian CPG, while we recommended the former with modifications and did not recommend the latter. In addition, we did not recommend the SEMICYUC-SENPE CPG, while the above review recommended it with modifications. These divergences can be explained by the fact that we obtained lower scores in most AGREE II domains because we strictly followed the user’s manual guidance. As an example, in the ‘rigor of development’ domains, we observed that the ASPEN CPG^(22,37) did not perform systematic reviews, and both the ASPEN⁽²²⁾ and Canadian⁽³⁴⁾ CPG lacked a detailed evidence-based search strategy, explicit description of criteria for excluding evidence and descriptions of the methods used to formulate the recommendations. They also did not detail how final decisions were arrived as, nor did they provide details regarding external review. In addition, the cited systematic review included two other CPG that were not selected by us (the Australian and New Zealand Intensive Care Society CPG and University of Pennsylvania Medical Center CPG), since they were specifically designed for surgical ICU and did not fulfill our inclusion criteria^(50,51) .

A study including eight CPG for the nutrition management of severely burned adult patients showed an overall quality score between 29·2 % and 100 %, and most of them also presented a score methodological rigor score that was lower than 60 %⁽⁵²⁾. Similarly, a study including seventeen CPG for the nutrition management of cancer patients showed an overall quality of CPG that ranged from 24·4 % to 94·5 %, in which most of the CPG presented scores lower than 60 % in the ‘rigor of development’ domain (range from 13·0 % to 93·2 %). Of the seventeen CPG included in this review, only two were strongly recommended, eleven were recommended with modifications and four were not recommended. The authors also found that the nutritional care procedures recommended by different CPG varied greatly⁽⁸⁾. Recently, a quality appraisal review of eleven guidelines for the nutritional management of patients with chronic kidney disease showed that none of them presented high quality according to the AGREE II tool, while three presented moderate quality and were recommended with modifications by the reviewers⁽⁵³⁾. To the best of our knowledge, no systematic review of nutrition CPG applied the AGREE-REX tool⁽¹¹⁾.

Strengths and limitations

The present review was conducted following Cochrane protocols⁽¹⁸⁾, and search was performed in seven large electronic databases. It presents a protocol registration and did not restrict the literature search for specific languages. We appraised CPG using validated tools. To our knowledge, this is the first systematic review of clinical nutrition practice guidelines for critically ill adult patients using the AGREE-REX tool. Four independent evaluators in both the AGREE II and AGREE-REX tools assessed all included guidelines, and the agreement between evaluators was excellent for most domains. The inter-rater reliability between reviewers was excellent. Regardless of this, there are intrinsic limitations related to the tools used for CPG quality assessment. The AGREE II tool does not provide a cut-off to distinguish between high- and low-quality guidelines, nor does it suggest the more relevant domains to be considered. Regarding the AGREE-REX tool, even if CPG are based on solid methods, the overall quality will be reduced when information about the values of patients, health care professionals and funding sources is insufficiently described. In addition, although the AGREE II and AGREE-REX tools aim at objectively scoring certain questions, inevitably, there is some subjectivity in the scoring.

Implications for clinical practice

In order to improve the future development of more rigorous and high-quality CPG, it should be noted that the aim of the current systematic review was not to criticise the existing CPG but rather to critically appraise their development; the AGREE II tool is designed to evaluate the quality of reporting and not the clinical applicability of the guidelines.

Guidelines are intended to promote evidence-based decisions as a way to implement them in clinical practice. It is well known that non-adherence to guidelines results in insufficient healthcare, great discrepancies in the provided care, worse disease outcomes and increased medical costs^(54–57). However, health professionals need simple, current, reliable and accessible guidelines, while, in parallel, the demand for training health professionals in guideline usage is a priority^(58–60). These features are not fulfilled by the AND CPG⁽³⁶⁾ because it is only available for registered users that can sign in to view the topic and access it via several links on the AND website. On the other hand, the ESICM CPG⁽¹⁴⁾ achieved more simplicity, which can be explained by the limited number of recommendations regarding early enteral nutrition in critically ill patients. However, it does not address all steps of the nutrition care process.

Conclusions

The methodological evaluation of the critically ill adult patient CPG revealed significant discrepancies, especially in terms of their applicability, stakeholder involvement and rigor of development methodologies, which exhibited poor performance and showed substantial room for improvement. Only two out of ten CPG were identified as acceptable quality and could be recommended for daily practice. According to the AGREE-REX tool, the recommendations for these CPG were classified as moderate quality. The findings of the current systematic review emphasised methodological issues that could improve the quality of reporting of future guidelines concerning nutrition support among critically ill adult patients.