What matters, what's researched, what's measured

I've been thrown into the world of the 'validation of research instruments'. Or what you and I might call 'surveys and questionnaires that we are told can be trusted to do the job they say they will do'.

My whole self dislikes and distrusts any tool that says it's been scientifically validated, but is then used by humans on humans to find out about things that can't be observed - like wellbeing or health.

My visionary self has a dream of a different way of living together as humans, and my top priority is thinking how to get from the currently reality to something closer to this dream. Dreams usually take lifetimes or more, however hard and however strategically you fumble and work towards them.

My pragmatic self accepts that within the way things currently happen in the policy, practice and research world of health and social care, it is impossible to escape validated research instruments as they have power and credibility with those in power. Validated research tools are therefore key to improving things within the social and public service systems we currently have. It matters to people's lives now that things improve now within systems. We have to live in our present, not just dream for the future.

For validated research tools, therefore, the questions become ones of harm reduction, and opening up the debate about these tools.

I am very grateful to an academic* for linking me to a great 'primer' on the validation of research tools, as I've never really known what validation involves. For a published paper with so much information about such an obscure topic, I found it immensely readable https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6004510/
*She's nameless only because I'm writing this early on a Saturday morning and want to get this blog up before coproduction week finishes, ie before I can show it to her and check if she'd like to be named

Reading that paper made it clear that developing a validated research instrument is not a task to be undertaken lightly. It is time, brain and cost intensive.

But - some research funders for some types of research require the use of validated research instruments. So if there isn't a suitable validated research instrument, you can't get the funding, so can't do the research. Or to put it another way, you can only research something if there is already a research instrument that has been through the validation process. No instrument, no research.

It took a while for the significance of that to sink in for me. Some things that really matter to people's lives cannot be researched because the research cannot be funded because there is no suitable validated research instrument.

And then, add in another two layers of challenge.

1. I have yet to see a validated research instrument that can be used inclusively, across the diversity of people who experience whatever it is that is being researched. If used in line with the validation, this means research will perpetuate the exclusion of people who are already unheard, or heard but ignored when decisions are made that affect their lives. If used outside of the validation, for example by rephrasing and reframing questions or changing the response options in order to allow a more diverse group of people to take part, the tool is no longer validated.

2. Step 1 of developing a validated research instrument, according to Boateng and the rest of his research team (ie the people who wrote the paper that I found so helpful), is having a clear conceptual grasp of the topic. Say the topic is 'quality of life'. Whose conceptual grasp of the topic should provide the conceptual framework for the tool? I will always prioritise the conceptual framework of the potential participants whose lives will be affected by any decisions made on the basis of the research. And it is possible to do that. But it is also possible to use pre-existing academic conceptual frameworks. Yep, you can design a new tool based on the frameworks that are based on research that is shaped by what could be done using a previously validated research instrument. I suspect that route is more common, as it is much cheaper and easier to do a literature review than brand new qualitative research with people recruited from your pool of potential future participants.

So what?

So those of us who would prefer to avoid validated research instruments need to engage with the creation of validated research instruments if we are to avoid the perpetuation of exclusion and silencing of some groups of people from the kind of research that policy makers and governments listen to.

And it would be doable. But it is a big investment of effort. So it would be important to make sure that the effort is being invested in developing a tool that would allow research to be carried out on a topic of crucial or strategic importance to those who are usually excluded or silenced.

And that means starting with the research on 'what matters', so we know what is worth that investment.

And that is where coproduction comes in.

Doing research to identify 'what matters' cannot be done just by academic researchers, or just by people with lived experience, or just by think tanks, or just by government research teams. Doing this research needs the skills of academics, the skills of people with lived experience who are already involved in policy or practice work, the skills of policy makers, the skills of people responsible for providing public services, the skills of members of the public. What everyone in that list has in common is possessing a set of expertise, experience and perspective that others on that list do not have, but that are needed for top quality health and social care research. In other words, to do top quality health and social care research means developing our skills as research coproducers. And by 'our skills', I mean the skills of everyone in that list.

I would argue that everyone involved in coproducing research needs to learn the skills that are needed to be research coproducers. It does not come naturally to any of us. For those of us with professional training, it won't have been part of that training so we need training. For those of us without professional training, we need training too.

And I would argue that we need the same training, a training that helps us work out for ourselves how to apply the principles of coproduction to the professional training or lived experience we bring. And a training that helps us see other people's skill sets and ways that the skill sets work together to make us and our research stronger.

Where do we go from here?

I don't know!

I'm putting this out in Coproduction Week, because it is probably the best week of the year to wave a flag saying 'over here!' about something that I've not seen discussed. PLEASE come and find me if you are already grappling with this. I won't be doing anything practical about any of this until I've submitted my doctoral thesis. But this is something that would benefit from a long brew on the back burner. Here's to Coproduction Week 2021 when I will hold myself accountable for what I have (or haven't) done to start developing these rough ideas into actual life changing research changing work.

Research questions and rivers

In the recent flooding, more than one river burst its banks so thoroughly that it lost any sense of having banks, and certainly no traces of the banks are visible to an observer. At this point, it is more a mass of water than anything identifiable as a river.

I can play for hours in water and I enjoy getting to understand the currents, the feel, the properties of that untrammelled water. But eventually, for the sake of the countryside, and in particular for the sake of making sure the water gets from A to B, we need that river back. In places, the water will find its own new channels through which to flow. In places, I will need to rebuild, channel and trammel the water into a river that flows where I want it to flow.

As with water, so with research questions. There’s a time for flooding – after all, the land can be much richer and productive after a good flood. As academics, we need space to let our brains overflow if we are going to be creative, imaginative researchers. But there’s also a time for safe, well-contained rivers. And crafting a research question is definitely one of those times.

It’s been a hard and frustrating couple of months with the PhD. I’ve felt as if I was getting somewhere, only to discover I was going round in overlapping circles. In the last few days, I think I’ve worked out both the source of the problem and a way to conceptualise the solution.

I have been filling my head with epistemology, ontology, ethics, standpoints, theoretical perspectives, methodologies, approaches, ways of seeing the world, ethical stances for a long time now. For a while, my river held as the water levels rose. Then it burst its banks – not hard as I tend not to build high flood defences. But then I continued adding a deluge of not-terribly-critical-but-highly-creative thinking to the swirling and increasingly murky water. I’ve been paddling around in the flood – sometimes swimming, sometimes half-drowning – for the last few months.

And yesterday I began to realise that this is why I cannot define my all-important research question.  I’d worked out over the last few weeks that the water was getting very murky and that my thinking might be creative but wasn’t very “researcher-like”.  But it was only in the last week, thanks to an amazingly perceptive (and patient) supervisor, that I caught sight of the serious threat posed by the flood and remembered I was supposed to be navigating a river. In practical terms, the realisation came via reflecting on a set of immensely helpful questions from my supervisor that she had sent to help me think through why I was going round in circles.  

The source of the problem, and the germ of the solution is this: If a research question acts as the river banks that determine the edge and course of the river, then it is impossible to craft a research question while still splashing and half-drowning on a flood plain.

So, back to basics. Yesterday, force myself to use a flow chart/table to funnel my thinking from an overarching topic down to a specific question, and from that the questions I will need to be able to ask of my data to answer that specific question. That really helped me work out the direction the river should be flowing. It also showed me that I’d been forgetting my overarching topic in the hunt for a workable research question. Simply looking wider broke through the blockage to focusing my sight more narrowly.

Today, I’ve gone back to a rigorous and focused overview of epistemology, theoretical perspective, methodology to remind myself where the river banks could lie. Next challenge is to work out where the banks need to be to get the river flowing in the direction I want – in other words to pick a suitable way of seeing the world and stick to it. In the process, I accept that I may realise I need the river to flow in a slightly different direction because of where I feel the need to put the banks.

Tomorrow, it’s time to double check that my methods suit the data I will need in order to answer the questions that my theoretical perspective allow me to ask. And if I manage that, then I reckon that’s the research question sewn up and I can knock off happily for the Christmas break. The challenge in the new year will be to reduce the volume of water (new ideas, creative stimuli) that fill up the river so it doesn’t go into flood again.