The “Hidden Universe Of Idiosyncratic Researcher Variability” Or: Why Research May Never Have The “Right” Answer

Last year, a team of researchers published a study that cast light into the uncertainty or subjectivity of research.‍

The researchers structured a large-scale crowdsourced research effort, in which 73 research teams analyzed the same hypothesis and the same data set, to come up with numerical findings and conclusions. The methodologies, decision making, expertise, prior beliefs, and expectations of the research team were all tracked.  

What did they discover?  No two teams came back with the same results.  

90% of the differences in the numbers could not be attributed or defined even after analyzing how the teams conducted their research, or their individual beliefs and expectations of the researchers.  

You read that right.  Same hypothesis.  Same data.  Different results every time. 

So what are the implications of that for researchers?  Should we never trust research?  Is research even worth doing if results cannot be replicated and we are not aware of our own subjectivities and unconscious assumptions that are shaping our work?  Can research ever be objective?  Can we ever know the truth? 

My point of view: 

‍

1. I love the willingness of researchers to be so self-analytical, reflective and vulnerable so as to conduct such a study, with its potentially condemning results. 

I think this is where true progress and growth is derived - from doing this hard work to point out flaws or unconscious assumptions.  Processes like this are actually exactly why we need research - to uncover the truth, even when the results may (seem to) hurt us.  

‍

2. As researchers, it would serve us to spend more time contemplating the role of the observer. 

I have always been fascinated by the double slit experiment in physics.  

In this experiment, electrons pass one at a time through a screen containing two slits.  If there was no observer when the particles passed through the screen - the particles appeared to have passed through both slits simultaneously, like an electromagnetic wave.  This was the first mystery - why a particle would act like a wave, and not like matter.  It gets more mysterious.  

When an observation tool was added to monitor the experiment in order to understand why matter was behaving as a wave, the particle passed through one slit or the other -  acting as we expect matter to act.  The very act of observing made a particle behave differently - as though it was aware it was being watched.  

At the subatomic level - the very act of observing will change results.  As researchers, when we observe, we shape and define what we are observing.  

I think Richard Feynman says it best, saying this experiment “contain(s) the general mystery… baring nature in her most elegant and difficult form.”

‍

3. I wonder what Plato would think about the double slit experiment. 

Objectivity, and the idea that there is a truth that is independent of my individual perspective or experience is one of my own cognitive biases - like Plato, I believe in a universal truth.  

I also believe in the limitation of my experience and consciousness to understand and interpret truth through my own subjective lens.  As my college philosophy professor once stated - ‘the naked eye is blind’, asserting that a lens that is not subjective, cannot capture any information.  

While there may be an objective truth, I am not an objective observer.  This is why research methodology attempts to create structures to avoid biases is critical.  As a researcher, I must persistently be on a journey to - as the Delphic maxim reads -  know (my)self.  Being self-analytical, attempting to understand and make conscious my subjective biases and striving to not let them influence my research methodologies is the unending work of a researcher in search of truth.

‍

4. If the very act of observation changes reality, and if no one is an objective observer, what role does research play in creating value in this world?  

This takes my back to one of my favorite reads - Principles by Ray Dalio.  While he wasn’t necessarily writing about research, I think he sums up well why conducting research is so important: 

• Having questions is better than having answers because it leads to more learning.
• Finding out about one’s weaknesses is good and it is the first step toward finding out what to do about them and not letting them stand in your way.
• Pain (or painful insights) are required to become stronger.

He concludes: “Ask yourself what do you want, then ask ‘what is true’ — and then ask yourself ‘what should be done about it.’ If you do this you will move much faster toward what you want to get out of life.”

This is often the job of strategic research: 

• To define goals.  
• To ask questions that lead to learning.  
• To craft experiments that test our assumptions and gain knowledge. 
• To be humble about our biases, limitations, and results.
• To be accountable to the truth, even when the results are difficult. 
• To empower our clients with knowledge to move more effectively towards their goals. 

Research begets more research, more questions, more learnings.  Embracing this process, embracing humility about our limitations, and always striving to learn and grow is what we’re here to do.  

‍

Ever in search of truth - 

Vina Rathbone Falvey

‍