Article
Insights from ISPOR 2024: Looking for improvements in tools used in systematic review assessments
This research explores methods of assessing the quality of economic evaluations included in systematic reviews. The author, drawing from personal experience and a background in health economics, found minimal changes in checklist usage since a 2019 review. It emphasizes the confusion between reporting quality and methodological quality and calls for enhanced training and a unified tool for quality assessment in systematic reviews.
5 questions with Fiona Stewart
At ISPOR 2024, Cencora team members presented research posters on a variety of topics in the health economics and outcomes research space. We took the opportunity to chat with them about their work and its potential impact. Here, Fi Stewart, Assistant Director, Evidence Generation and Value Communications for Cencora answers questions about the poster, “Methodological quality checklists in systematic literature reviews of health economic evaluations, 2022-2024.” Nicole Fusco, Evelyn Gomez-Espinosa, Nicole Szydlowski, and Maria Koufopoulou served as co-authors.
What inspired this research?
This research was partly inspired by my personal experience as a systematic reviewer over many years. I am familiar with the methods used in standard systematic reviews and had just begun studying for a master's degree in health economics. So as I explored systematic reviews in the field of health economics and reviews of economic evaluations, I realized that the methodologies were not as clearly defined.
In this context, an economic evaluation examines the cost effectiveness of a new drug compared to the standard of care. A systematic review of economic evaluations analyzes various evaluations that assess the same issue. Assessing the quality of the studies included in systematic reviews is essential for interpreting the overall findings. This process involves not only analyzing data from the different economic evaluations, but also assessing the quality of the methods used in each individual economic evaluation included in the review. This careful interpretation of the findings is one of the crucial steps that make a systematic review so valuable.
In terms of systematic reviews of economic evaluations, I was investigating current practices compared to the established standards. I was aware of a study conducted a few years ago that addressed a similar question, and I wanted to determine whether there had been changes, improvements or declines in the field since then.
In this context, an economic evaluation examines the cost effectiveness of a new drug compared to the standard of care. A systematic review of economic evaluations analyzes various evaluations that assess the same issue. Assessing the quality of the studies included in systematic reviews is essential for interpreting the overall findings. This process involves not only analyzing data from the different economic evaluations, but also assessing the quality of the methods used in each individual economic evaluation included in the review. This careful interpretation of the findings is one of the crucial steps that make a systematic review so valuable.
In terms of systematic reviews of economic evaluations, I was investigating current practices compared to the established standards. I was aware of a study conducted a few years ago that addressed a similar question, and I wanted to determine whether there had been changes, improvements or declines in the field since then.
Was there a hypothesis that was confirmed through the research?
I was updating previous work published in 2019 that showed there is a lack of consistency among the quality assessment methods used in systematic reviews of economic evaluations. For my research, I identified systematic reviews of economic evaluations by searching Medline and Embase, which are the standard bibliographic databases that systematic reviewers typically consult first due to their extensive range of medical literature.
I did not have a concrete hypothesis; rather, my primary question was whether the findings of the review published in 2019 had changed. Specifically, I wanted to know if researchers had begun using checklists more appropriately. My initial feeling was that conditions may have improved, and I aimed to test that presumption. However, I assumed that changes would be more significant than they ultimately were.
I did not have a concrete hypothesis; rather, my primary question was whether the findings of the review published in 2019 had changed. Specifically, I wanted to know if researchers had begun using checklists more appropriately. My initial feeling was that conditions may have improved, and I aimed to test that presumption. However, I assumed that changes would be more significant than they ultimately were.
What are the key takeaways from your research?
We currently lack standardized methods for assessing the methodological quality of economic evaluations, which differentiates them from other research methods. There remains a persistent confusion between reporting quality and methodological quality, a trend that has not changed significantly in recent years.
Checklists are used for assessing both reporting quality and methodological quality. Reporting quality pertains to how well the study methods are described. In contrast, methodological quality focuses on how well the study was designed and conducted, and whether it achieves its intended purpose. This discussion of methodological quality is essential for inclusion in a systematic review.
When assessing cost-effectiveness or other types of economic evaluations, there are numerous checklists available, but there is little clarity regarding which ones to use. I believe that providing training on research methods for systematic reviews would be highly beneficial in guiding researchers toward the appropriate checklists for their evaluations.
Checklists are used for assessing both reporting quality and methodological quality. Reporting quality pertains to how well the study methods are described. In contrast, methodological quality focuses on how well the study was designed and conducted, and whether it achieves its intended purpose. This discussion of methodological quality is essential for inclusion in a systematic review.
When assessing cost-effectiveness or other types of economic evaluations, there are numerous checklists available, but there is little clarity regarding which ones to use. I believe that providing training on research methods for systematic reviews would be highly beneficial in guiding researchers toward the appropriate checklists for their evaluations.
Was there anything in the research that was surprising, that you didn't expect?
I was surprised that the reporting checklist was used far more frequently than the methodological quality checklist. Additionally, I was taken aback by the variety of different checklists used. While there were common checklists, many other versions served similar purposes, and I discovered tools that I had not previously encountered.
Another unexpected finding was that some studies utilized both a reporting checklist and a methodological quality checklist, which I believe is unnecessary.
Another unexpected finding was that some studies utilized both a reporting checklist and a methodological quality checklist, which I believe is unnecessary.
What are the next steps from this research?
We aim to develop this research into a full paper, as there appears to be a significant need for it. I would like to issue a call to action for academic groups that have the capacity to influence and reform the guidance on methodological quality for systematic reviewers. One potential direction could be the development of a single tool for assessing methodological quality in economic evaluations.
There is compelling evidence that these methods are not applied correctly. If a systematic review does not consider the quality of the studies within that review, we can be less confident in its findings. We need to be more thoughtful about the methods we employ, ensuring they are robust and that the outcomes we produce are meaningful.
There is compelling evidence that these methods are not applied correctly. If a systematic review does not consider the quality of the studies within that review, we can be less confident in its findings. We need to be more thoughtful about the methods we employ, ensuring they are robust and that the outcomes we produce are meaningful.
This article is intended to communicate Cencora’s capabilities, however, Cencora strongly encourages readers to review the links provided herein and all available information related to the topics mentioned and to rely on their own experience and expertise in making decisions related thereto as the article may contain certain marketing statements and does not constitute legal advice.
Cencora.com is providing automated translations to assist in reading the website in languages other than English. For these translations, reasonable efforts have been made to provide an accurate translation, however, no automated translation is perfect nor is it intended to replace human translators. These translations are provided as a service to users of Cencora.com and are provided "as is." No warranty of any kind, either expressed or implied, is made as to the accuracy, reliability, or correctness of any of these translations made from English into any other language. Some content (such as images, videos, Flash, etc.) may not be accurately translated due to the limitations of the translation software.
Any discrepancies or differences created in translating this content from English into another language are not binding and have no legal effect for compliance, enforcement, or any other purpose. If any errors are identified, please contact us. If any questions arise related to the accuracy of the information contained in these translations, please refer to the English version of the page.