Critically reading scientific papers is critical for Data Scientists working some areas – especially those working in health. With that in mind, here are some key considerations in reading scientific (peer-review, grey literature) papers:
Theory: Is the theory sound? Are there theoretical issues in the design that cause problems? Implementation: Are there concerns about the implementation that cause you to question the conclusions?
Methodology best practices: Consider the best practices for doing that particular study type. Did the authors follow these practices? Did they perform the quality checks that have been discussed? How did the study perform on those quality checks?
Threats to validity: What are the “threats to validity?” How have the authors addressed these concerns or are some of these concerns not issues in this study?
Data: Are there concerns about the data acquisition, data quality, etc.? Does the raw data seem compelling or does it have any fundamental flaws?
Analysis: Are the models logically sound? Are the models strong representations of the data? Are correlations/collinearities impacting the conclusions? Models with low predictive power are a concern.
Author bias: Are the authors selectively focusing their results or their discussion and ignoring key elements that are not supporting their thesis? If there are reasons to question the author’s bias, that doesn’t mean we reject the study but it does raise concerns.
Further research/ research limitations: How could the study have been improved? What changes would you have made to the data acquisition, study design, analysis, variable inclusion, etc. in order to draw conclusions? Are there reasons that the authors didn’t do the steps you are considering?
Conclusions: After thinking through all of the steps above, do you believe the author’s conclusion? Why or why not?
