We’ve seen in the book Truth Decay discussed in the previous post that there’s a declining trust in traditional institutions and people tend to have more confidence in what their friends or favourite media outlets say than in proper argumentation. It would then be understandable to question whether science itself is a fraud and whether researchers are actually just spending the whole day playing Solitaire and googling random findings.

Fortunately, the scientific method has proven reliable for centuries and, as we’ve seen in the last post, there are tried-and-tested ways to asses the confidence of findings starting from primary research.

What is the Scientific Method?

The scientific method is a process used by scientists to study and understand natural phenomena. It involves several steps:

1. Observation: Scientists use their senses to observe a phenomenon or a problem they want to study.
2. Question: Scientists ask questions about what they have observed, formulate hypotheses, and make predictions.
3. Hypothesis: Scientists develop a tentative explanation, or hypothesis, to explain the observed phenomenon. The hypothesis is usually based on prior knowledge and must be testable and falsifiable.
4. Prediction: Scientists use the hypothesis to make predictions about what will happen under certain conditions.
5. Experiment: Scientists design and conduct experiments to test the hypothesis and the predictions. The experiments are designed to control for variables that could influence the outcome.
6. Analysis: Scientists analyze the data collected from the experiments to determine whether the results support or refute the hypothesis.
7. Conclusion: Scientists draw conclusions based on the results of the experiments. If the hypothesis is supported by the evidence, it may become accepted as a scientific theory. If it is refuted, scientists will modify or discard the hypothesis and develop a new one.

The scientific method is an iterative process, and the results of one study often lead to new questions and hypotheses, which can then be tested with new experiments.

Coming to the main topic of this post, research can be classified in various categories which fall under two important types:

• Experimental: more accurate but costlier and sometimes impossible to carry out
• Observational: cheaper but less reliable.

We won’t copy-paste the various definitions as they have been listed in other places, including here. We’ll just include the diagram the author has created, which shows how the most common types of studies can be classified.

Another type of study, which sits outside of this classification, is the meta-analysis, which is not primary research but compiles multiple studies to discover commonalities and calculates a weighed average of the quantitative results found in the selected studies. One source that specifies the methodology for performing a meta-analysis is Cochrane Training. Of course, this type of research is not a silver bullet because it can be corrupted by applying unfair selection criteria (selection bias) to exclude certain studies that don’t support a pre-ordered conclusion.

Perhaps the most relatable example of how the scientific method is applied in real life is the process of drug development and certification. The Multiple Sclerosis Trust in the UK provides a step-by-step description of the 10-15 years it takes for a drug to reach the medicine cabinet. A key part of this process is made of the clinical trials (Phases I, II and III), which involve first testing on healthy patients, then treating patients who suffer from a condition in randomised control trials and, in the end, testing with larger number of patients over multiple regions. Such rigorous testing, which can end up eliminating all but one or two out of 10,000 compounds, should make most rational persons demand similar evidence from purveyors of “alternative” medicine.

Thus, since we want to put a score tag, experimental studies will rank higher than observational ones in Argumentful. We also have some controls put in place to ensure findings are not reported only partially or that the selection criteria when conducting a meta-analysis is fair. A study with a funding conflict or one that has never been replicated will score lower regardless of the methodology.