Article
NICE problems: Limited clinical evidence
Generating robust evidence to show a new technology’s clinical efficacy in an HTA can be challenging, particularly for rarer conditions and complex treatment pathways. This article examines practical examples of this issue in NICE HTAs, and solutions and lessons to aid similar submissions in the future.
Introduction
Gathering evidence to demonstrate the clinical efficacy of a product, and cost-effectiveness against comparators, poses significant problems when there are limitations in the availability of credible patient data. For example, inability to recruit eligible patients, pathway placement restrictions, ethical or practical problems in designing robust trials, or pooling small trials can lead to issues such as increased uncertainty in statistical tests, immature survival data, and heterogeneity between participants. This uncertainty leads committees to question whether to accept evidence for decision-making, risking the likelihood of a product being recommended.
Data limitations are an expected and understood part of technology development and companies who successfully navigate National Institute for Health and Care Excellence (NICE) technology appraisals (TAs) demonstrate alignment with the NICE manual, adherence to the reference case, and constructive work with the External Assessment Group (EAG) through technical engagement. These factors are critical in presenting evidence in the most supportable way, and committee decisions recorded in TA guidance documents reveal discussions where a lack of transparency or deviations from available support and guidelines have led to failure to recommend.
To examine the problem, and gather evidence to aid manufacturers in optimising a NICE submission which involves limited clinical evidence, NICE TAs over the nine months ending 31 March 2025 were examined (n=43; 69 TA guidance notes were published in the period, seven were for terminated technologies, and 19 were summaries making decisions following additional evidence NICE committees had requested for appraisals from earlier periods). It should be noted that no consideration was given to the rarity of the condition, nor orphan or other status, when reviewing the guidance notes; rather evidence was recorded only where there was a note of the effects of data limitations. From the 43 guidance notes, 23 were identified where committee discussions focused on limitations arising from clinical evidence.
To examine the problem, and gather evidence to aid manufacturers in optimising a NICE submission which involves limited clinical evidence, NICE TAs over the nine months ending 31 March 2025 were examined (n=43; 69 TA guidance notes were published in the period, seven were for terminated technologies, and 19 were summaries making decisions following additional evidence NICE committees had requested for appraisals from earlier periods). It should be noted that no consideration was given to the rarity of the condition, nor orphan or other status, when reviewing the guidance notes; rather evidence was recorded only where there was a note of the effects of data limitations. From the 43 guidance notes, 23 were identified where committee discussions focused on limitations arising from clinical evidence.
Understanding the problem
As noted, limited evidence is sometimes unavoidable, but companies will try to gather as much information as possible to make samples big enough for robust statistical analysis and, therefore, ostensibly, to aid committees in drawing conclusions. However, these attempts to improve participant numbers can lead to criticism of poor generalisability to the applicable population if key differences between studies are not carefully considered.
Between-trial variation
Pooling trials to produce a higher response-evaluable sample is a key theme. This technique was observed not only in single-arm trials with fewer than 100 participants but also randomised controlled trials with more than 1,000. Committee discussions, however, merge around a common issue; namely company assumptions of homogeneity are challenged because of inconsistencies between baseline characteristics. Between-trial variations, including study populations, treatment practices, and trial design, are not reflected in the naïve pooling (‘treat-as-one-trial’) method employed by some companies, which is deemed to introduce excessive heterogeneity and high uncertainty in results.
Alignment with NHS population and clinical practice
Even when trials with limited data are presented individually, companies must be sure that the trial is designed to produce the most relevant results possible and not just sufficient numbers of participants. Drawing on sizeable multicentre studies, for example, has also led to discussions regarding the applicability of the sample that the company presents. Discussions highlight analyses where the data (or subgroups extracted from the full studies) do not reflect National Health Service (NHS) patients, clinical practices, or pathway placement. This was particularly stressed in a case where post hoc analysis was used to show clinical efficacy in subgroups that were narrower than the population for which market authorisation was granted. The committee expressed concern when data from a multicentre trial identified subgroups in post hoc analysis, questioning whether the results would be applicable to NHS practice. Clinical expert support was required to contextualise the evidence within the complex NHS pathway and give the committee confidence.
Alignment to marketing authorisation
Population alignment with the marketing authorisation is also seen as an important theme—the trial must represent the population for which authorisation was granted. Committees criticise any deviation from market authorisation and, furthermore, note their willingness to take into account patient and clinical expert evidence of severity and practical restrictions in recruiting to a trial when considering clinical efficacy; even where the trial has been narrowed to a very small sample, committees look favourably on a technology which focuses on unmet need. Although no evidence was seen where NICE allowed expansion beyond the market authorization, there are documented cases accepting trials designed for a subset of the authorisation population, again specific to a particular unmet clinical need, and where conflicting clinical expert opinion in care guidance led to further data gathering under a managed access agreement.
Uncertainty in long-term outcomes
Issues with long-term outcomes from trial data are also recorded, particularly relating to immature data when presenting overall and progression-free survival evidence. Committee discussions show reluctance to accept results for rare conditions, where median overall survival and progression-free survival have not been reached, even when companies have gone through data augmentation and have added further data cuts requested at initial committee meetings.
Application of indirect treatment comparisons
The most commonly documented issue relating to limited data is the application of indirect treatment comparison (rather than choice of technique—when to apply a matched adjusted indirect comparison [MAIC], for example). EAG comments and committee uncertainty frequently focused on small effective sample sizes (with limited covariates available for adjustment), heterogeneity between studies and care guidelines, and misidentification of treatment modifiers and prognostic variables. Although techniques were not commonly questioned, some evidence of criticism of selecting pairwise studies from multiple sources was found. For example, companies might be reluctant to conduct many MAIC studies across combinations of comparators and lines of treatment, but committees noted the opportunity to use emerging techniques such as multi-level network meta-regression to provide wider sensitivity-based results—or at least test the feasibility.
Solutions
To address these problems, some key factors have emerged, informed by strong mitigations by companies in response to data limitation concerns, which result in success, and unresolved issues leading to products not being recommended. These factors are relevant across all problem types described above.
Firstly, and as noted initially, it is important for health technology assessment (HTA) applications to align with the NICE HTA process, expectations, and decision problem PICO (population, intervention, comparators, and outcomes): namely, the NICE technology appraisal guidance manual and market authorization but also following guidance from the technical support documents. As obvious as this may seem, committee discussions frequently focus on failure to do so. EAG guidance and open interaction through the technical engagement process can help ensure alignment with these expectations where unclear.
Where issues remain following technical engagement or initial committee meetings, evidence submitted for the final committee decision meeting should include as much evidence as possible to support company responses to data uncertainties. Strong expert input (eg, key opinion leader [KOL] interviews and Delphi group exercises) should be added where possible. Although NICE includes patient and clinical experts in their meetings (and, in fact, is likely to involve them far more than company representatives in discussions), submitting strongly supported evidence in the dossier will aid discussions and decision-making within the meeting. The use of real-world evidence (RWE) is also welcomed in support; however, it is essential to ensure that RWE is generalisable to the relevant population and clinical practices and does not contribute further to issues discussed above.
Examples of committee guidance include suggesting pre-specification of an intention to pool during the engagement process, clear communication with the EAG during technical engagement of the HTA, and the use of KOLs to contextualise the clinical and practical importance of heterogenic factors. Working with these stakeholders to create scenarios and statistical analysis plans which explore alternatives and include such context in the dossier would aid discussions in committee meetings.
Although this additional evidence might not remove uncertainties, the recommendation to use the managed access programme allows further evidence to be gathered in NHS clinical settings when technology ‘has potential to be cost-effective’—a common phrase. Managed access, although not a common route, has been used where committees see high potential in a new treatment; however, even getting access to the managed access programme relies on demonstrating that the comparative clinical impacts have been fully explored and on clear contextualisation of how the condition impacts clinical practice and patient lives. Committees want fair and generalisable data and are willing to take into account patient and clinical expert evidence.
Firstly, and as noted initially, it is important for health technology assessment (HTA) applications to align with the NICE HTA process, expectations, and decision problem PICO (population, intervention, comparators, and outcomes): namely, the NICE technology appraisal guidance manual and market authorization but also following guidance from the technical support documents. As obvious as this may seem, committee discussions frequently focus on failure to do so. EAG guidance and open interaction through the technical engagement process can help ensure alignment with these expectations where unclear.
Where issues remain following technical engagement or initial committee meetings, evidence submitted for the final committee decision meeting should include as much evidence as possible to support company responses to data uncertainties. Strong expert input (eg, key opinion leader [KOL] interviews and Delphi group exercises) should be added where possible. Although NICE includes patient and clinical experts in their meetings (and, in fact, is likely to involve them far more than company representatives in discussions), submitting strongly supported evidence in the dossier will aid discussions and decision-making within the meeting. The use of real-world evidence (RWE) is also welcomed in support; however, it is essential to ensure that RWE is generalisable to the relevant population and clinical practices and does not contribute further to issues discussed above.
Examples of committee guidance include suggesting pre-specification of an intention to pool during the engagement process, clear communication with the EAG during technical engagement of the HTA, and the use of KOLs to contextualise the clinical and practical importance of heterogenic factors. Working with these stakeholders to create scenarios and statistical analysis plans which explore alternatives and include such context in the dossier would aid discussions in committee meetings.
Although this additional evidence might not remove uncertainties, the recommendation to use the managed access programme allows further evidence to be gathered in NHS clinical settings when technology ‘has potential to be cost-effective’—a common phrase. Managed access, although not a common route, has been used where committees see high potential in a new treatment; however, even getting access to the managed access programme relies on demonstrating that the comparative clinical impacts have been fully explored and on clear contextualisation of how the condition impacts clinical practice and patient lives. Committees want fair and generalisable data and are willing to take into account patient and clinical expert evidence.
Sources listed below.
Disclaimer:
This article summarises Cencora’s understanding of the topic based on publicly available information at the time of writing (see listed sources) and the authors’ expertise in this area. Any recommendations provided in the article may not be applicable to all situations and do not constitute legal advice; readers should not rely on the article in making decisions related to the topics discussed.
Entrez en contact avec notre équipe
Notre équipe d’experts de la valeur se consacre à la transformation des données probantes, des informations sur les politiques et des renseignements sur le marché en stratégies efficaces d’accès au marché mondial. Laissez-nous vous aider à naviguer dans le paysage complexe des soins de santé d’aujourd’hui en toute confiance. Contactez-nous pour découvrir comment nous pouvons vous aider à atteindre vos objectifs.
Sources:
- The University of Sheffield. NICE Decision Support Unit. Technical support documents. Published January 22, 2026. Accessed February 12, 2026. https://sheffield.ac.uk/nice-dsu/tsds
- NICE. NICE technology appraisal and highly specialised technologies guidance: the manual. www.nice.org.uk. Updated December 17, 2025. Accessed February 12, 2026. https://www.nice.org.uk/process/pmg36/
- NICE. Highly Specialised Technologies (HST) criteria checklist Maralixibat for treating cholestatic disease in Alagille Syndrome [ID3941]. February 2022. Accessed February 12, 2026. https://www.nice.org.uk/guidance/gid-ta10832/documents/supporting-documentation
