WHAT’S GOING ON IN THE STUDY?
Below are questions to support you in making sense of the details of
the study. |
HOW IS IT RELEVANT TO ME?
Below are questions to support you in thinking about how to integrate
what you are reading to your particular research question. |
I
usually read the abstract *of the article first. That’s
the short paragraph at the beginning of the article that usually summarizes
the goals, procedures and findings of the study. Here are questions
I ask the abstract: |
- Keep
your master’s question in mind as you read! Keep asking “how is
what I am reading helping me to think about my research question.”
- Take
notes to summarize your answer to this question.
|
Abstract (an opportunity to pre-read the article)
-
What is the goal of the study?
- What
questions/hypotheses are they investigating?
- Who
are the subjects?
- What
did they do?
- What
did they find out?
*
Note: not all abstracts give all this information. Not all descriptions
of studies have an abstract. |
Remember
how Terry Ford said that if readers understand the context, then
they are more likely to be able to make sense of ambiguous details?
I usually read the sections of a study out of the order in which they
are presented to create that context for myself.
|
|
Introduction
(a more in depth look at the rationale and purpose of the study)
- What is the
goal of the study?
-
Why do they think it’s an important goal? (The answer to this
question is often along the lines to the one that you are writing
in the introduction to your master’s project)
-
What are the hypotheses they are testing?
Discussion -- What are they going to try to conclude
from this study? By addressing this question along side the questions
above and below you will help yourself distinguish between foreground
and background in this study.
- What did they
find out? Were their hypotheses supported?
-
What seemed important to them about their findings?
-
In what ways are they hesitant to be confident about their
findings?
|
-
Notice
if the study if closely related to your master’s question, it might
provide you with rationale/history or useful references to these
parts in the introduction.
-
Continue
asking and taking notes on how what you are reading is helping
you to think about your research question.”
|
Okay now I’m usually ready to tackle the more technical
parts of the article. I always take a deep breath and accept that I
may not understand every detail. Here are the basic questions I puzzle
around with.
|
|
Methodology ( Here is the place to wonder how the
researchers converted their question into something they could look
at and measure)
- Who are
they looking at? (participants/subjects)
Age group, gender, income level, race. Is there anything about who
participated in the study that makes you wonder if you can generalize
the findings? For example, if you are interested in considering factors
that affect middle schoolers’ motivation are there things that you
need to consider before generalizing the findings from a study that
used college students ? (Use the Statistics Concepts and Controversies
book, ch2 & 3, or some other methods book on samples, to look
at some of the issues you might consider here).
- What
are they asking the subjects to do? (procedure)
Were they in a lab or in realistic setting? Where they completing
questionnaires, interviews, or some tasks in an experiment?
- What
tools are they using? (operational definitions, validity)
What tools are the authors using to represent the constructs they
are measuring? Remember Mismeasure of Man. Some people’s
operational definition of intelligence was head circumference. Others’
used head volume. Other’s used answers on the Stanford Binet. Other’s
used answers on the Wechsler.
If the authors use a questionnaire to measure intrinsic motivation,
what questions are they asking? Do the questions relate to something
that represents your idea of intrinsic motivation?
Remember how you defined your terms in the introduction of your master’s
projects? In a study they often define their terms in two places
– the general conceptual definition in the introduction (like you
did) and the more specific definition that ended up being what they
looked at. These two ways of looking at things in a given study are
not always perfectly matched.
- Is this
an experiment, a quasi experiment, a survey?
See handout on "Guide to Research Designs, Methods, &
Strategies," (Isaac & Michael, 1995)
- How is
what they are asking the subjects to do going to help the authors
address their question? Trying to come up with the answer
to this question helps me to construct what’s going on in the study
and helps me bump into what I might need to look at a little more.
|
- Have
you seen people use other tools to represent the same concept? Does
it matter what they use?
Sometimes when there is a debate about something (e.g. whether rewards
impact intrinsic motivation), some of the discrepancies in the findings
can be explained by how and when the different researchers measured
the rewards and the motivation of their subjects.
|
The
parts of the RESULTS that I usually focus on are the tables, the graphs,
and the charts. I use the text of the results sections to (i)make sure
that I know what I am looking at; and (ii) to see whether the authors
see the same interesting things in these figures that I see.
Note: The results are merely a description
of the findings. They are not usually an interpretation of the
findings; the interpretation and meaning making usually happens in the
discussion section of the study. This is a place where you can see,
“hmm… would I draw the same conclusions if I got their data?”
Below are some questions to help you
consider what you are looking at in table and charts. Knowing me, I
haven’t anticipated everything you might encounter. So...
If you have something odd or need some
demonstrative oral and visual support– come ask
|
|
Results (Here is the place to wonder how the
researchers organized their findings/data in such a way that they could
address their hypotheses.
- Look
back at the hypotheses and the methods sections. Are the researchers
interested in comparing groups of people or are they interested
in seeing if there are relationships between experiences, or
both?
If,
comparing groups of people
-- (e.g., comparing girls to boys; or a group that had one
teaching experience compared to a group that had another.) You
will probably see somewhere a TABLE or a GRAPH which has columns
of numbers to compare.
Ask
yourself these questions as you look at the title, the headings,
the footnotes around the table/graph:
What do the numbers represent?
-
a proportion
or percentage of something?
-
a score?
-
a period of time (second, minutes, etc)
-
a number of objects
-
is the number a straight number or a mean of some sort? (A mean
is a type of average; e.g. the mean of 2,4,6= (2+4+6)/3=4
).
If
looking at relationships between experiences
– (e.g. the relationship between how interested someone is
and the amount of time they pursue a difficult task; or the relationship
between years in school and income level; or the relationship between
hunger and how much people end up eating).
You will probably
encounter a table of a particular types of numbers called either
(i) correlation coefficients (r); or (ii) regression coefficients.
[See chapter on correlation in Statistics Concepts and Controversies,
or some other statistics/methods textbook. Page 270 – 274 has a
succinct summary on correlation to rely on].
A correlation (r) can be positive,
negative or non-existent,
that doesn’t mean good and bad. I’ll try to explain this through
an example:
Let’s
say I was interested in the relationship between someone’s interest
and the amount of time they would persist with a difficult task.
-
If from looking at the data I got a positive
correlation, I would find out that there was a tendency
that people who were more interested would spend more
time on the task. People who were less interested
would spend less time on the task.

-
If from looking at the data I got a negative
correlation, I would find out that there was a tendency
that people who were more interested would spend less
time on the task. People who were less interested would spend
more time on the task.
-
It’s also possible to get a zero correlation,
that means that I would have a hard time predicting people’s time
on task from their interest. They might spend a lot of time or
no time at all. That is, there may be no relationship between
interest and time on task.

A correlation (r) is always a number between –1.00 and +1.00,
The closer to a 1.0, be it positive or negative, the larger the relationship.
So, a correlation
of .40 represents a positive relationship that is stronger than
a relationship .20. A correlation of -.65 would be a stronger relationship
than a -.36.
Let’s
use our example from above with some extra variables
so that you can see how you might encounter and make sense
of these correlations in a table:
Table
1: Correlations to examine the relationship between interest,
time spent on task, and number of puzzles completed.
You’ll notice
that the relationship between interest
and and time spent on task is a positive correlation of .60.
That means that the more interested people were, they were also
likely to spend more time working on the task.
The relationship between interest and
degree of irritation is a negative correlation of -.25.
That means that the people who were interested were less
likely to be irritated. Another way of saying this is that people
who were more irritated were likely to be less interested.
The relationship between degree of irritation and time spent
on task was .02. That means that how long people spent
on the task probably had nothing to do with how irritated they
were.
I’ll let you figure out how to interpret the last row of
correlations with prior knowledge
Important
considerations: “correlation does not equal causation”
-- You cannot
usually conclude that just because something is correlated/related
that one thing caused the other. Here are some examples
that illustrates why:
Apparently
there is a strong relationship between ice-cream sales in New York
City and crime. Does that mean that the more ice-cream you eat
the more likely you are to be a victim or perpetrator? Or that if
you’ve just committed a crime that you are likely to have a sudden
hankering for ice-cream? Sounds absurd right? It turns out that
ice-cream sales and crime rate do have some things in common. Summer
time, warm weather and more people out in the streets. These things
in common are called CONFOUNDS.
They are other things that might explain a relationship but that
aren’t immediately visible or measured.
How about in
our example with interest and time spent on task?
- It might be tempting to quickly assume that interest
caused people to spend more time on the task. Whoa though
-- What would you want to know before interpreting such a relationship
between these variables? For example, were people asked about
their interest before or after they finished working
on the task? If it was measured after completing the task,
could it be then that people became interested as a result
of working on the difficult task?
-
Thinking about the ice-cream/crime story above, could
it be that interest and persistence share something in common?
For example, could it be that people who are interested tend also
to have more prior knowledge about the types of task they are
being asked to do for the experiment, and as a result have more
tools to draw on, so you see them persisting more on difficult
tasks because they are in fact better equipped? So, it’s not
interest that impacts time on task, but something that people
who showed these qualities have in common – prior knowledge about
the types of tasks they were being asked to do.
Is
there ever a time when you can assume something caused
something else? ONLY if the study is
a real experiment. [See the handout I gave out at the beginning
of the library workshops in the fall quarter on the descriptions of
different types of studies]
A real experiment
is one where the participants in the study were randomly assigned
to treatment groups. For example, if you were interested in whether
aspirin could eliminate headaches. You’d need to take a bunch of
people with headaches and RANDOMLY
assign them to one of two groups, either a treatment
group that gets aspirin, or a control group that doesn’t
get aspirin. Then you can compare whether all other things being
equal, the treatment group gets relief any sooner than the control
group which didn’t get the aspirin.
Imagine if
you allowed people to self-select into the aspirin groups. That
is allow people to decide themselves which group they wanted to
be in. What else might explain changes in their headaches? Perhaps
the group that chose not to take the aspirin rely on a range of
their own alternative remedies (sleep, water, cucumber slices over
their eyes). Perhaps they don’t take aspirin because they assume
it won’t work? Perhaps for that matter those that chose to take
the aspirin have a strong belief in it’s effectiveness, and what
ends up curing them is that belief (or placebo) not the aspirin
itself.
As
educators you will read a lot about studies that compare age groups,
sex, race, cultural backgrounds, language. None of these variables
are chosen by the subjects. You as an experimenter cannot randomly
assign a participant to be a girl, a ten-year old, a Mexican, or
a native Vietnamese speaker. So, as you compare, let’s say a girl
from a boy in their interest in science, there are so many, many
things that differentiate a girls experience from a boys. Girls
may be treated differently, maybe have different potentialities,
maybe need different pedagogies to access their interest, maybe
see different adult models of what’s possible for themselves, etc.
Some of the differences may be physical, social, environmental,
etc. Since you did not control the presence of these experiences,
and since boys and girls are systematically different on so many
different levels you’d have a hard time explaining any difference
between girls and boys simply on account of their biological sex.
There are many confounding variables that
could help to explain a difference.
Now,
MULTIPLE REGRESSIONS are a type of correlation where the researchers
are trying to predict how much a set of variables rather than
just a single variable predicts something. It’s a pretty involved
and complex statistic which I am about to completely oversimplify.
Many statisticians will probably groan. Oh well, here it goes:
There are a
few types of questions you might encounter. I’ll use our example
above to illustrate some of the variations:
·
One might wonder how much time on task, prior knowledge
and irritation combined predict interest.
·
Another might be how much does time on task predict
interest above and beyond anything that irritation or prior
knowledge predicts.
I
will add information about how to read these kinds of tables here
soon…
|
- As
you read the discussion,
look out for how the researchers explain their findings. Do they
mention any possible “confounds”?
|
At this point I usually REREAD the DISCUSSION. Now I am in a position
to wonder:
- How
well do these interpretations and conclusions match what I saw in
the study?
- What alternative explanations
or critiques do the authors provide of their own project?
- What
other explanations might I have given what I've read or learned somewhere
else?
|