Network Meta-Analysis for Decision-Making

12Validity of Network Meta-Analysis

12.1 Introduction

Doubts have been expressed about the validity of network meta-analysis from the outset, and there has been a steadily growing literature around this that deserves to be evaluated and reviewed. Certainly, any statistical method has its limitations. In the case of network meta-analysis, the concerns have revolved mainly around its assumptions. There have been two main lines of empirical research.

Firstly, there is a series of reviews of applied literature that set out to document whether the assumptions of network meta-analysis have been checked or even mentioned. Unfortunately, there has been little consensus on what the assumptions actually are, and in a number of papers, the assumptions are stated in a form that is subtly different from how they have been stated in this book. For this reason we begin the chapter by reviewing the terminology surrounding network meta-analysis assumptions (Section 12.2) in the hope of dispelling some of the confusion.

A persistent assumption running throughout the clinical and methodological literature has been that direct evidence is superior to indirect evidence, although there seems to be no formal analysis that supports this claim. In Section 12.3 we present some ‘thought experiments’ that aim to clarify in quantitative terms the threats that unrecognised effect modifiers pose to valid inference in evidence synthesis. We ask whether indirect comparisons are more vulnerable, less vulnerable ...