Thank you for your interest in YYD.

The Seven Dimensions of the Dissertation Research Validity Check

Hu Lehao

Validity refers to the correctness or reliability of a description, conclusion, explanation, understanding, or other idea. However, it is often the case in studies that there is a threat to the validity of the study due to the bias of the investigator and the influence of the investigator on the individuals being studied. Harvard University professor Joseph A. Maxwell, in his book Qualitative Research Design, provides a checklist for looking for evidence that challenges the conclusions of the study or excludes potential threats to validity. Each of these methods is described below.

1. Highly and long-term investment

Long-term observations can provide more complete data about specific situations and events than any other method. Not only does this approach provide more types and richer data, but at the same time, the data can be more straightforward and there are fewer things to deduce. Repetitive observations, interviews, and the researcher's continued presence in the context can help avoid false connections and immature theories. In addition, these methods provide a better opportunity to test other hypotheses in the course of research. For example, in a long-term observational study of medical students, the researchers not only learned more about the cynicism of the medical profession and found a typical point of view, but also learned how different perspectives manifested themselves in different social situations and how these students faced the conflict between these ideas.

2. "Rich" data

Long-term investment and in-depth interviews allow us to obtain "rich" data, which is very detailed and varied, and can provide a comprehensive and insightful picture of the research phenomenon. In interview studies, this type of data usually records the interviews verbatim, rather than just those that you think are important. Rich data comes from detailed and vivid records (or videos and transcriptions) of specific, specific events.

3. Subject approval

Subject approval (also known as "member checks") is a systematic solicitation of feedback from study participants, primarily about our data and conclusions. It's the easiest and most effective way to prevent misunderstandings about what they say, do, and hold, and it can also help us identify our own biases and misunderstandings about what we observe. However, the participants' feedback was no more effective than their interview responses. Both of these sources should be seen only as evidence to validate their own opinions.

4. Intervention

Although some qualitative researchers believe that experimental manipulation is inconsistent with the idea of qualitative research, formal interventions are often applied to traditional qualitative research that lacks formal "treatment". For example, in a study on the reading development of two children and the role of teachers' expectations and behaviors in children's reading development, the researcher shared his views on one of the students not meeting the teacher's expectations. The result changed the teacher's behaviour towards the student and subsequently promoted the development of the student's reading skills. The interventions employed by the teacher, the subsequent changes in behaviour and the increased level of development of the students all support the researchers' view that the teacher's behaviour – rather than the teacher's expectations – is the main factor in the development or non-development of the student.

5. Look for inconsistent evidence or negative cases

The search for and analysis of inconsistent evidence and negative cases is a key part of the validity test of qualitative research. Examples that cannot be explained by concrete ideas can point out obvious flaws in the theory. However, sometimes inconsistent evidence is not convincing, or there may be flaws in the understanding of inconsistent evidence. There are many experimental results in physics that seem "subversive" but are later proven to be problematic. The rule we need to adhere to is to look for evidence for and against to assess whether to stick to the conclusion or revise it, and we must not ignore the results that are inconsistent with our own research conclusions. Asking others to comment on the logic and methodology of the research is also a good way to test your biases and assumptions. In specific complex situations, it is best to report evidence that is inconsistent with our own, and let the reader judge to reach their conclusions.

6. Quasi-statistical analysis

The conclusions of many qualitative studies are quantitative. Any idea that a particular phenomenon is typical, rare, or common in a particular context or group is quantified in nature and, therefore, needs to be supported by quantitative evidence. Quasi-statistical analysis can not only help to test and support quantitative opinions, but it can also help us assess how well our findings support a conclusion and how much it threatens validity. For example, how many inconsistent pieces of evidence there are and how many different sources of this evidence have.

7. Comparison

In quantitative variable theory studies, it is common to employ explicit comparisons (e.g., a comparison between the intervention and control groups) to assess the threat of validity. However, many qualitative studies, particularly multi-case or multi-site studies, also use comparative methods. These comparisons (including comparisons of the same situation at different times) can address the objection to the use of qualitative research to explain causation, which holds that qualitative research does not definitively answer the "opposite" question, i.e., what would happen if there were no hypothetical causes.

In addition, qualitative studies in a single context or interviews with a relatively homogeneous group of people usually do not require much comparison to draw conclusions. There may be literature on "typical" situations or individuals studied, so it will be relatively easy to find the relevant characteristics and processes of a particular case and understand their importance. In other cases, the study participant may have had some prior experience in other situations or situations, and the researcher should use this experience to identify important factors and the implications of those experiences.

8. Summary

It is important to emphasize that some validity threats are unavoidable, and we need to acknowledge these issues in the study application or study conclusion, and no one can expect a perfect answer to every validity question anyway. The key question is how likely and serious these inevitable validity problems are.

(This article was first published on the website of the Institute of Research Writing, May 6, 2024)