Conducting a Critical Review of the Literature

See the questions from the Critical Review Workshops to help you to intelligently tap into and integrate research to inform your master's question.

The purpose of the handout below is to anticipate some of the questions you might have as you read the more statistical journal articles either on intrinsic motivation or on your particular masters project topic.  My goal is to give you a glossary of sorts which you can use as a basic tool for mining the purpose of the study and the conclusions you can or shouldn’t draw from a study.

WHAT’S GOING ON IN THE STUDY?

Below are questions to support you in making sense of the details of the study.

HOW IS IT RELEVANT TO ME?

Below are questions to support you in thinking about how to integrate what you are reading to your particular research question.

I usually read the abstract *of the article first.  That’s the short paragraph at the beginning of the article that usually summarizes the goals, procedures and findings of the study.  Here are questions I ask the abstract:

    

 

 

  • Keep your master’s question in mind as you read!  Keep asking “how is what I am reading helping me to think about my research question.” 
  • Take notes to summarize your answer to this question.

 

Abstract  (an opportunity to pre-read the article)

  • What is the goal of the study? 
  • What questions/hypotheses are they investigating?
  • Who are the subjects?
  • What did they do?
  • What did they find out?

* Note: not all abstracts give all this information.  Not all descriptions of studies have an abstract.

Remember how Terry Ford said that if readers understand the context, then they are more likely to be able to make sense of ambiguous details?  I usually read the sections of a study out of the order in which they are presented to create that context for myself.

 

Introduction (a more in depth look at the rationale and purpose of the study)

  • What is the goal of the study?
  • Why do they think it’s an important goal? (The answer to this question is often along the lines to the one that you are writing in the introduction to your master’s project)
  • What are the hypotheses they are testing?

Discussion -- What are they going to try to conclude from this study?  By addressing this question along side the questions above and below you will help yourself distinguish between foreground and background in this study.

  • What did they find out?  Were their hypotheses supported?
  • What seemed important to them about their findings?
  • In what ways are they hesitant to be confident about their findings?
  • Notice if the study if closely related to your master’s question, it might provide you with rationale/history or useful references to these parts in the introduction.


  • Continue asking  and taking notes on  how what you are reading is helping you to think about your research question.” 

Okay now I’m usually ready to tackle the more technical parts of the article.  I always take a deep breath and accept that I may not understand every detail.  Here are the basic questions I puzzle around with.

 

Methodology ( Here is the place to wonder how the researchers converted their question into something they could look at and measure)

  • Who are they looking at? (participants/subjects)
    Age group, gender, income level, race.  Is there anything about who participated in the study that makes you wonder if you can generalize the findings?  For example, if you are interested in considering factors that affect middle schoolers’ motivation are there things that you need to consider before generalizing the findings from a study that used college students ? (Use the Statistics Concepts and Controversies book, ch2 & 3, or some other methods book on samples, to look at some of the issues you might consider here).
  • What are they asking the subjects to do? (procedure)
    Were they in a lab or in realistic setting?  Where they completing questionnaires, interviews, or some tasks in an experiment?
  • What tools are they using?  (operational definitions, validity) 
    What tools are the authors using to represent the constructs they are measuring?  Remember Mismeasure of Man.  Some people’s operational definition of intelligence was head circumference.  Others’ used head volume.  Other’s used answers on the Stanford Binet.  Other’s used answers on the Wechsler.   

    If the authors use a questionnaire to measure intrinsic motivation, what questions are they asking?  Do the questions relate to something that represents your idea of intrinsic motivation?        

    Remember how you defined your terms in the introduction of your master’s projects?   In a study they often define their terms in two places – the general conceptual definition in the introduction (like you did) and the more specific definition that ended up being what they looked at.  These two ways of looking at things in a given study are not always perfectly matched.
  • Is this an experiment, a quasi experiment, a survey?
    See handout on "Guide to Research Designs, Methods, & Strategies," (Isaac & Michael, 1995)
  • How is what they are asking the subjects to do going to help the authors address their question?  Trying to come up with the answer to this question helps me to construct what’s going on in the study and helps me bump into what I might need to look at a little more.

 

 

 

 

 

 

  • Have you seen people use other tools to represent the same concept?  Does it matter what they use?   

    Sometimes when there is a debate about something (e.g. whether rewards impact intrinsic motivation), some of the discrepancies in the findings can be explained by how and when the different researchers measured the rewards and the motivation of their subjects.
 

The parts of the RESULTS that I usually focus on are the tables, the graphs, and the charts.  I use the text of the results sections to (i)make sure that I know what I am looking at; and (ii) to see whether the authors see the same interesting things in these figures that I see. 

Note:  The results are merely a description of the findings.  They are not usually an interpretation of the findings; the interpretation and meaning making usually happens in the discussion section of the study.  This is a place where you can see, “hmm… would I draw the same conclusions if I got their data?”

Below are some questions to help you consider what you are looking at in table and charts.  Knowing me, I haven’t anticipated everything you might encounter.  So...

If you have something odd or need some demonstrative oral and visual support– come ask

     

 

Results (Here is the place to wonder how the researchers organized their findings/data in such a way that they could address their hypotheses. 

  • Look back at the hypotheses and the methods sections.  Are the researchers interested in comparing groups of people or are they interested in seeing if there are relationships between experiences, or both

    If, comparing groups of people  -- (e.g., comparing girls to boys; or a group that had one teaching experience compared to a group that had another.)  You will probably see somewhere a TABLE or a GRAPH which has columns of numbers to compare. 

    Ask yourself these questions as you look at the title, the headings, the footnotes around the table/graph:

    What do the numbers represent?
    -           a proportion or percentage of something?
    -           a score?
    -           a period of time (second, minutes, etc)
    -           a number of objects
    -           is the number a straight number or a mean of some sort? (A mean is a type of  average; e.g. the mean of 2,4,6= (2+4+6)/3=4  ).

    If looking at relationships between experiences – (e.g. the relationship between how interested someone is and the amount of time they pursue a difficult task; or the relationship between years in school and income level; or the relationship between hunger and how much people end up eating). 

    You will probably encounter a table of a particular types of numbers called either (i) correlation coefficients (r); or (ii) regression coefficients.  [See chapter on correlation in Statistics Concepts and Controversies, or some other statistics/methods textbook.  Page 270 – 274 has a succinct summary on correlation to rely on].

     A correlation (r) can be positive, negative or non-existent, that doesn’t mean good and bad.  I’ll try to explain this through an example: 

    Let’s say I was interested in the relationship between someone’s interest and the amount of time they would persist with a difficult task. 

    -           If from looking at the data I got a positive correlation, I would find out that there was a tendency that people who were more interested would spend more time on the task.  People who were less interested would spend less time on the task. 

-           If from looking at the data I got a negative correlation, I would find out that there was a tendency that people who were more interested would spend less time on the task.  People who were less interested would spend more time on the task. 

-           It’s also possible to get a zero correlation, that means that I would have a hard time predicting people’s time on task from their interest.  They might spend a lot of time or no time at all.  That is, there may be no relationship between interest and time on task.

A correlation (r) is always a number between –1.00 and +1.00, The closer to a 1.0, be it positive or negative, the larger the relationship.

So, a correlation of .40 represents a positive relationship that is stronger than a relationship .20.  A correlation of -.65 would be a stronger relationship than a -.36. 

Let’s use our example from above with some extra variables so that you can see how you might encounter and make sense of these correlations in a table:

Table 1: Correlations to examine the relationship between interest, time spent on task, and number of puzzles completed.

You’ll notice that the relationship between interest and and time spent on task is a positive correlation of .60.  That means that the more interested people were, they were also likely to  spend more time working on the task.

The relationship between interest and degree of irritation is a negative correlation of -.25That means that the people who were interested were less likely to be irritated.  Another way of saying this is that people who were more irritated were likely to be less interested.

The relationship between degree of irritation and time spent on task was .02.  That means that how long people spent on the task probably had nothing to do with how irritated they were.

I’ll let you figure out how to interpret the last row of correlations with prior knowledge

 

Important considerations: “correlation does not equal causation”  -- You cannot usually conclude that just because something is correlated/related that one thing caused the other.  Here are some examples that illustrates why:

Apparently there is a strong relationship between ice-cream sales in New York City and crime.  Does that mean that the more ice-cream you eat the more likely you are to be a victim or perpetrator? Or that if you’ve just committed a crime that you are likely to have a sudden hankering for ice-cream?  Sounds absurd right?  It turns out that ice-cream sales and crime rate do have some things in common.  Summer time, warm weather and more people out in the streets.   These things in common are called CONFOUNDS.  They are other things that might explain a relationship but that aren’t immediately visible or measured.

How about in our example with interest and time spent on task? 

- It might be tempting to quickly assume that interest caused people to spend more time on the task.  Whoa though -- What would you want to know before interpreting such a relationship between these variables?  For example, were people asked about their interest before or after they finished working on the task?  If it was measured after completing the task, could it be then that people became interested as a result of working on the difficult task? 

- Thinking about the ice-cream/crime story above, could it be that interest and persistence share something in common?  For example, could it be that people who are interested tend also to have more prior knowledge about the types of task they are being asked to do for the experiment, and as a result have more tools to draw on, so you see them persisting more on difficult tasks because they are in fact better equipped?  So, it’s not interest that impacts time on task, but something that people who showed these qualities have in common – prior knowledge about the types of tasks they were being asked to do.

Is there ever a time when you can assume something caused something else?  ONLY if the study is a real experiment.  [See the handout I gave out at the beginning of the library workshops in the fall quarter on the descriptions of different types of studies]

A real experiment is one where the participants in the study were randomly assigned to treatment groups.  For example, if you were interested in whether aspirin could eliminate headaches.  You’d need to take a bunch of people with headaches and RANDOMLY assign them to one of two groups, either a treatment group that gets aspirin, or a control group that doesn’t get aspirin.  Then you can compare whether all other things being equal, the treatment group gets relief any sooner than the control group which didn’t get the aspirin.

Imagine if you allowed people to self-select into the aspirin groups.  That is allow people to decide themselves which group they wanted to be in.  What else might explain changes in their headaches?  Perhaps the group that chose not to take the aspirin rely on a range of their own alternative remedies (sleep, water, cucumber slices over their eyes).  Perhaps they don’t take aspirin because they assume it won’t work?  Perhaps for that matter those that chose to take the aspirin have a strong belief in it’s effectiveness, and what ends up curing them is that belief (or placebo) not the aspirin itself. 

As educators you will read a lot about studies that compare age groups, sex, race, cultural backgrounds, language.  None of these variables are chosen by the subjects.  You as an experimenter cannot randomly assign a participant to be a girl, a ten-year old, a Mexican, or a native Vietnamese speaker.  So, as you compare, let’s say a girl from a boy in their interest in science, there are so many, many things that differentiate a girls experience from a boys.   Girls may be treated differently, maybe have different potentialities, maybe need different pedagogies to access their interest, maybe see different adult models of what’s possible for themselves, etc.  Some of the differences may be physical, social, environmental, etc.  Since you did not control the presence of these experiences, and since boys and girls are systematically different on so many different levels you’d have a hard time explaining any difference between girls and boys simply on account of their biological sex.  There are many confounding variables that could help to explain a difference.

 

Now, MULTIPLE REGRESSIONS are a type of correlation where the researchers are trying to predict how much a set of variables rather than just a single variable predicts something.  It’s a pretty involved and complex statistic which I am about to completely oversimplify.  Many statisticians will probably groan.  Oh well, here it goes: 

There are a few types of questions you might encounter.  I’ll use our example above to illustrate some of the variations:

·   One might wonder how much time on task, prior knowledge and irritation combined predict interest.

·   Another might be how much does time on task predict interest above and beyond anything that irritation or prior knowledge predicts. 

I will add information about how to read these kinds of tables here soon…

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

  • As you read the discussion, look out for how the researchers explain their findings.  Do they mention any possible “confounds”?

At this point I usually REREAD the DISCUSSION. Now I am in a position to wonder:

  • How well do these interpretations and conclusions match what I saw in the study?
  • What alternative explanations or critiques do the authors provide of their own project?
  • What other explanations might I have given what I've read or learned somewhere else?