Updated: Nov 5, 2021
As I embark upon a large qualitative research project, I've been thinking a lot about metrics. I've been thinking about what we measure (and how sometimes reality differs from intention!), about how we measure it, about the value that this imparts onto it and about what it means to be a qualitative researcher in a quantitative world.
In this post, I wanted to share some of my initial thoughts about the pros and cons of metrics in global health. There's a short bibliography at the end, but I'd really appreciate your recommendations for further reading on the subject. I'd also love to hear your thoughts and feedback on my summary so far.
What do we mean by metrics?
Global Health metrics seek to translate the status (and trends) of complex, shifting and multi-faceted ethnographic circumstances into a manageable, malleable and memorable quantitative format through the use of indicators.
It occurs to me that the process of reducing rich and nuanced occurrences into a handful of digits is akin to a siren-like beauty: the idea is beguiling but can be treacherous if trusted without question.
Keeping Count The discipline of global health has evolved in conjunction with the rise of quantitative data (i.e. inventories, statistics, metrics, indicators and numerical goal-setting). Before dissecting the relative merits and problems with this, I think it's useful to reflect upon the sheer extent to which metrics are woven into the fabric of our world’s health systems, as one can become blind to normative practices when they are familiar. So let's have a quick think about some of the myriad ways in which we use (and are used by?) numbers:
Indicators and metrics guide governments in both determining and communicating priorities and resource allocation to national health providers.
In a global setting, metrics are used for measuring the baseline of, and progress beyond, key health issues (and a plethora of non-health related issues such as poverty or corruption, for example). This information not only directs the policy and priority of global health organisations, it influences investments, trading, and the global economic market as a whole.
Metrics are utilised by donors (institutional and philanthropic alike) to hold recipient organisations to account, frequently in combination with hefty incentives and reprisals. The Bill & Melinda Gates Foundation are a high-profile example of this: a powerful donor with a zealot-like belief that quantitative data and financial incentives are key to solving global health challenges.
Metrics are a powerful tool of advocacy groups, think-tanks and researchers for influencing change. Consider the Access to Medicines Index, which has impressively shifted the behaviour of pharmaceutical behemoths in a short span, through the use of indicators, analysis, public reporting and thereby the generation of competition.
Metrics are used heavily in research and academia where there is sometimes a propensity for quantitative studies to be considered more serious and scientific than qualitative ones. This is reflected in journals like the Lancet, for example: a highly-respected and widely-read journal, which is a strong proponent of metrics.
Pervasive, to say the least. With this in mind then, let us explore the positive and detrimental impact of metrics.
The benefits of Global Health metrics
As pithily stated by Shiffman and Shawar, “the inherent allure of metrics [is] their ease of use and their alleged capacity to render legible complex worlds” (1). The value of this is undeniable (by all but the most ardent critics) because it enables information to be utilised and applied in a manner that is not feasible (or at least practical) with qualitative data. Through the generation of a number, our “complex world” is reified into a single reference that can be manipulated through calculation and communicated with ease.
Based on my experiences, and through my initial readings, I think the benefits of metrics could be summarised and grouped as follows. So, what's all the fuss about numbers?
Firstly, they allow us to conduct processes (e.g. budgeting, or risk management) that would otherwise be less effective, or even unworkable, without numbers. Such fundamental activities are core to “business as usual” for most health organisations. Moreover, certain disciplines are entirely dependent on, or characterised by, their use of statistical analysis (epidemiology, or quantitative research, for example).
Secondly, information becomes compelling. Numbers are simple, memorable, and easy to communicate (relative to other forms of data). There is also evidence to indicate that individuals tend to be more willing to trust quantitative data, as it is more likely to be viewed as scientific (1). As a result, information shared in this manner tends to resonate with the audience and thus it is a powerful tool for those wishing to gain attention or exert change.
Thirdly, it facilitates comparison. Distillation of an issue into a single, solid figure, removes extraneous factors, bypasses ambiguity, and creates a tangible and actionable output. An obvious application of this is to track the progress of an issue over time. Another benefit is that it creates scope for rankings and competition. This benefit is readily used by employers, campaigners, regulators and numerous global actors to, often publicly, urge change or alignment with a cause.
Fourthly, it is emancipatory. Impartial measurement can map issues and shine a light on problems that may otherwise be marginalised, in a manner that may be challenging to deny (1).
Fifthly, it enables engagement with a non-technical audience. While an individual may lack the competence or time to engage with a subject, complex ideas, issues and circumstances can be rapidly communicated and discussed. In my experience, this is often of particular value when dealing with the public or when liaising with senior management.
Finally, it creates accountability. Numbers allow a systematic and objective consideration of an activity or outcome that provides a dispassionate and apparently unbiased picture of performance.
"When counting, try not to mix chickens with blessings." Leonardo da Vinci
The problem with Global Health metrics
However, and rather obviously, there are drawbacks to metrics that not only undermine all of these benefits, but also that may have an actively detrimental impact on global health. These could be described as follows:
Similarly to Plato’s Allegory of the Cave, metrics are a shadow of reality, dancing on the cave wall of our understanding. The methods of data collection, and of imputation, are often opaque. In addition, as discussed, the numbers (and indicators) themselves are “products of social processes heavily reliant on interpretation” (1). However, through ignorance or inability, there is a risk that these abstracted figures are confidently used and applied as if they are complete and specific.
Metrics are reductive by intention. However, it strikes me that quantifying human experience – particularly human suffering – is a somewhat grotesque and heartless activity. It, according to Fukuda-Parr, “redefines [the concept] to a utilitarian perspective” (2) and in so doing, it undermines an individual’s lived experience. She exemplifies this statement by considering poverty: metrics may describe it in terms of income, but this is wholly unsuited to conveying the dehumanisation experienced by begging or prostitution (2).
Metrics involves gathering data from an array of cases and analysing them as a cohort. The results are used to make generalised observations, declarations and decisions and, in so doing, ignore nuance, quirks and specificities. Tichenor and Sridhar question “the larger ramifications of practices of standardization, data correction, and imputation… particularly with the goal of making local contexts readable from a satellite’s view of the world” (3). They point to the problem of universalising experience and applying the results to national or district health services without local adaptation. This issue is particularly problematic for the Global South and may result in the presumptive implementation of inappropriate interventions (3).
Counterintuitively, metrics can both be generalised and individualised. Since the 1960s there has, broadly, been a reframing of metrics and global goals into person-centred measurements and targets. Although this humanisation of data is not entirely regrettable, some birds-eye analysis of (and interest in) global issues has been lost (2). For example, moving the narrative from economic development to an impact on livelihoods can mean that a macroeconomic understanding of the issue is neglected.
Global data sets have the potential to subvert the establishment of national information systems (1). Over the last three decades, as globilisation, humanitarianism and economic development have matured, metrics have shifted from an organisational tool to a world-wide industry. Massive data sets are gathered, analysed and communicated on a global stage and, whilst this data may not be intended for national decision-making, it is predictable that it has disincentivised the creation of individual governmental health information systems in some lower income countries (1).
As a result, effected countries may lack the national or local knowledge required to make suitably adapted decisions (regarding intervention, priority, or resource allocation, for example) (1). In addition, this contributes to a lack of local data analysis skills, due to a lack of opportunity and investment to build this capacity (3). Finally, it also means that the open-source data sets used by a country are not actually owned by it, which can have implications for the interpretation and analysis of it (as full knowledge of collection, imputation and calculation methods are unlikely to be available).
Whilst considering large data sets (used to measure and track global burden of disease, international development goals, or country comparators, for example) we should question who determines the indicators that are used. These are, inevitably, large organisations (such as a UN agency or the World Bank), wealthy organisations (such as donors or private firms), or High-Income Countries situated in the Northern hemisphere.
Even if we assume that indicators are selected by these parties for purely altruistic purposes, the outcome is a shift of power: from designated to designator, from low-income to high-income entity. This results in global attention being focussed on those conditions and causes that the privileged consider to be worthy – and, similarly, to be measured by indicators that they believe to be representative. This paternalistic exclusion of relevant actors creates, according to Shiffman and Shawar, an “uneven playing field” (1). It is this same “uneven playing field” that prevents these metrics being corrected and optimised through scrutiny and iteration. Not only will differing perspectives have a smaller platform and be less likely to be heard, but alignment around them is driven via financial incentives or social pressure. Public ratings and comparisons create the potential for reputational damage, which in turn coerces a State or organisation to align with these designated priorities.
According to Shiffman and Shawar, critics of metrics feel that they “present a scientific veneer to a contingent undertaking, and thus acquire an authoritativeness [that] they do not deserve” (1). While their compelling and motivating nature is acknowledged as a benefit in this paper, the flip side is a skewing effect that it holds on the sector (as is manifest in several of the points already mentioned).
While anyone can digest metrics and graphs at face value, the critical thinking and interrogative scepticism needed to interpret and act upon them are learned skills. This, inevitably, leads to the politicisation of metrics. Numbers can be presented and manipulated to great effect to mobilise support and resources for a campaign, to undermine a competitor’s reputation, or to disingenuously present results, for example. The scale of this unearned legitimacy is perhaps most simply summarised by the adage: “Lies, damn lies, and statistics”.
Finally, and as is evident across all of these points, metrics possess a force (for influence seems too weak a term). The aforementioned negative aspects result in a host of negative consequences and – as forceful as they are – not only measure global health, but shape it in their own right.
First, as stated, metrics can create alignment around, and have a substantial influence on, strategy, prioritisation and the allocation of resource. This has the effect of supplanting national/organisational strategic processes of prioritisation and therefore interfering with internal structures, mechanisms, and outputs.
Secondly, and equally significantly, metrics impact things that are not measured. As only a limited number of indicators can be tracked, this effectively both narrows and silos global focus (thereby further influencing the global agenda). Furthermore, regarding what is not measured, it sends a tacit message in respect of its worth(lessness) – and therefore of the issues and experiences that it seeks to represent. For example, when the Millennium Development Goals (MDGs) chose not to include access to reproductive health as a measure (believing that that this was sufficiently covered by poverty reduction and maternal mortality) they failed to recognise the struggle and significance that this has on the rights and capabilities of women (2). Given the influence of global targets in setting national agendas, it is easy to see how such an omission can (negatively) impact initiatives and subsequently alter their course.
Thirdly, the existence of data (or the potential for it) is influential. Where it exists, it begs to be researched. This has an impact on the popularity of quantifiable issues, which in all likelihood leads to their increased discussion, readership and scientific progress. Furthermore, where an intervention can be measured, it may be preferentially implemented. For example, Cognitive Behavioural Therapy is a cornerstone of NHS mental health intervention. Its measurability has undoubtedly contributed to this status – a trait that more traditional (yet potentially more effective) forms of talking therapy do not share. This creates a “technocratic agenda” and can medicalise issues that should instead be progressed via social, educational or alternative interventions (1).
Fourthly, there is a general human (and organisational) preference for metrics over other forms of data. This privileges that “form of knowledge…[over] those that cannot be quantified” (1). This has a directing effect on interventions, research, publications and public support.
Fifthly, finally, and most profoundly, metrics have the potential to shape our perspective on reality. They categorise the world and in doing so they define and shape it, by delineating how it should be categorised, considered and valued.
Given the pervasiveness of metrics in Global Health, it is not a stretch to assert that they (and the indicators from which they are derived) have the potential to shape every facet of healthcare, in every corner of the world. However, how this potential is realised is, of course, dependent upon how the numbers are interpreted and applied.
References & Bibliography
(1) Shiffman J, Shawar YR. Strengthening accountability of the global health metrics enterprise. The Lancet (British edition) 2020;395(10234):1452-56. doi: 10.1016/S0140-6736(20)30416-5
(2) Fukuda-Parr S. Global Goals as a Policy Tool: Intended and Unintended Consequences. Journal of Human Development and Capabilities 2014;15(2-3):118-31. doi: 10.1080/19452829.2014.910180
(3) Tichenor M, Sridhar D. Metric partnerships: global burden of disease estimates within the World Bank, the World Health Organisation and the Institute for Health Metrics and Evaluation. Wellcome Open Research 2020;4(35) doi: 10.12688/wellcomeopenres.15011.2
Adams V. Metrics : what counts in global health. Durham, [North Carolina] ; London, [England]: Duke University Press 2016.
Whittemore R, Chase SK, Mandle CL. Validity in Qualitative Research. Qualitative Health Research 2001;11(4):522-37. doi: 10.1177/104973201129119299
Thanks for reading. I'm currently researching these issues at the Ethox Centre of the University of Oxford, with the kind support of the Wellcome Trust. If you'd like to stay in the loop, please get in touch or join the community on this site.