Welcome to critical review paper page

home page

REQUIRED SKILLS

Knowledge

1. Understanding of the most appropriate design and methodology to examine the hypothesis proposed in a research investigation.

2. Recognition of the key features, and common sources of problems or bias, of different types of research methodology, including randomised controlled trials, cohort studies, case-control studies, single case studies, studies involving economic analysis, and qualitative studies.

3. Understanding of basic statistical concepts (confidence intervals and probability) with sufficient knowledge to be able to interpret the results from common statistical tests used for parametric data (t-tests, ANOVA, multiple regression) and categorical data (chi-square test, logistic regression).

4. Understanding of the definition and meaning of measures important in critical appraisal (including relative and absolute risk reduction, sensitivity, specificity, likelihood ratio, odds ratio, and number-needed- to-treat).

5. Knowledge of the methodology of systematic reviews, including meta- analysis and the potential sources of bias in the interpretation of such analyses and overviews.

Skills

1.  The ability to critically examine a published paper so as to determine its scientific value. The ability to determine the reliability and validity of information and results presented in such a paper.

2. The ability to determine the clinical importance and relevance of results from the information given in a paper.

3. The ability to detect errors in design and methodology that render the stated conclusions invalid or affect their impact.

4. The ability to understand and assess logically the process and results of critical appraisal of such a paper.

5. The capacity to suggest further experiments which would confirm or expand understanding in the field under investigation.

6. The ability to place the results of the paper in clinical context and assess how far clinical practice may be altered as a consequence.

A worked example

Geriatricians are trying to identify the patients in their ward, who have phobic disorders. They want to know whether there is an easy way of identifying them.

You find a study which investigates the usefulness of the tool Phobia-Screen (a six item questionnaire) which screens for phobias.

The Phobia-Screen results were divided into phobia / no phobia. These results have been compared with a diagnostic gold standard namely Structured Clinical Interview for Axis 1 DSM-IV Disorders (SCID).

Total number of participants in the study was 84

Phobia-Screen, phobia (P+) : 22
Phobia-Screen, no phobia (P)- : 62

This is shown in fig. 1

Q. How would you make this a 2X2 table?

A.

The paper appears to be methodologically sound and you decide to find out whether Phobia-Screen would suit for the geriatrician's purpose.

Q.  What proportion of patients identified as having phobic disorders by the phobia-screen really have a phobia?

A.  20/22 = 0.90 (x 100 = 90%)

Q.  What is this measure?

A.  Positive predictive value

Q.  What proportion of patients identified as not having a phobic disorder really don't have a phobic disorder?

A. 60/62 = 0.96 (x 100 =96%)

Q.  What is this called?

A.  Negative predictive value

Q.  How would you interpret these findings clinically?

A.  Test is somewhat better at excluding phobic disorders than confirming it.

After a survey of the geriatric wards, you find that the actual prevalance is 23.00 %.

Q.  In the study what is the prevalance?

A.  22/84 x 100 = 26.19% (all SCID +ves / total number of patients x100)

Q.  What difference does this make to the use of the test on the geriatric wards?

A.  When Phobia-screen is used in the geriatric wards +ve predictive value will be higher and the -ve predictive value will be lower than those found in the study.

Q.  There are 2 characteristics of the test which can be assumed to be constant, and would enable you to use the results of the study in the geriatric wards. What are these?

A.  Sensitivity and specificity.

Q. What is sensitivity?

A.  True + ves / true +ves + false -ves (out of all the true cases what proportion does the test pick up)

Q. What is specificity?

A.  True -ves / true -ves + false +ves (out of all the true non cases what proportion does the test pick up)

Q.  Calculate  sensitivity.

A.  20/22 = 0.90

Q.  Calculate the likelihood ratio and explain what it is?

A.  0.90/1 - 0.96  (60/60+2) = 22.5

The likelihood that a positive test result will be observed in a patient with, as opposed to one without the disorder.

Sensitivity/ 1 - Specificity

Q.  How would you use likelihood ratio to help the geriatricians decide if someone is likely to have a phobic disorder?

A.  Likelihood ratio can be multiplied by the pre test odds to produce post test odds, and therefore post test probability of having a phobic disorder.

Q.  What does pre test odds mean?

A.  Pre test odds = pre test probability / (1 - pre test probability). (pre test probability is the
prevalance)

Q.  What is post test odds?

A.  Pre test odds x likelihood ratio.

Q.  Because probabilities are easier to interpret than odds, post test odds are usually converted back to post test probability. How do you do this?

A.  Use the formula, Post test probability = Post test odds / (1 + post test odds)

Q.  Is there any method of avoiding calculating pre test odds?

A.  Yes. Use a normogram.

Q.  If the geriatricians want a test which will detect more patients with phobic disorders, how are you going to achieve this using Phobia-Screen?

A.  Use a higher cut off point in the Phobia-Screen.

Q.  What would happen to the specificity?

A.  Fall. (there would be more false positives, sensitivity and specificity are inversely related).

(We are thankful to Dr M Sayeed Haque, Applied Statistician, Queen Elizabeth Psychiatric Hospital,
Birmingham, for valuable comments, about this worked example).

*****************************************

Look out for new model answers in this page

******************************************