Establishing a Cause-Effect Relationship PowerPoint Presentation

Slide 1 -

Establishing a Cause-Effect Relationship

Slide 2 -

Internal Validity The “treatment” and the “outcomes” The independent and dependent variables. Observation Treatment Outcomes What you do What you see Is the relationship causal between... Alternative cause Alternative cause Alternative cause Alternative cause In this study

Slide 3 -

Establishing Cause and Effect Temporal precedence

Slide 4 -

Establishing Cause and Effect Temporal precedence Cause Effect then Time It can get complicated through: -sloppiness (campaign contributions - Chicken and egg cyclical functions (democracy and GDP)

Slide 5 -

Establishing Cause and Effect Temporal precedence Covariation of cause and effect

Slide 6 -

Establishing Cause and Effect Temporal precedence Covariation of cause and effect if X, then Y if not X, then not Y if treatment given, then outcome observed (usually) if program not given, then outcome not observed

Slide 7 -

Establishing Cause and Effect Temporal precedence Covariation of cause and effect if X, then Y if not X, then not Y if program given, then outcome observed if program not given, then outcome not observed Dosage effects or comparative statics: If more of treatment, then more of outcome observed if less of treatment given, then less of outcome observed

Slide 8 -

Establishing Cause and Effect Temporal precedence Covariation of cause and effect No alternative explanations if X, then Y if not X, then not Y Treatment Outcome Micromediation

Slide 9 -

Establishing Cause and Effect Temporal precedence Covariation of cause and effect No alternative explanations if X, then Y if not X, then not Y Treatment Outcome Micromediation Alternative cause Alternative cause (nuisance) Alternative cause Alternative cause (substantive)

Slide 10 -

In Lab or Field Experiments… Is taken care of because you intervene before you measure outcome Is measured by comparing treated and untreated groups Is the central issue of internal validity -- usually taken care of through random assignment Temporal precedence Covariation of cause and effect No alternative explanations

Slide 11 -

Single-Group Threats to Internal Validity

Slide 12 -

The Single Group Case Two designs:

Slide 13 -

The Single Group Case Two designs: “Post-test only single-group design” X is the treatment O is the observation

Slide 14 -

The Single Group Case Two designs: Measure baseline O “pre-test, post-test single-group design” or “interrupted time-series”

Slide 15 -

The Single Group Case Two designs: Measure baseline O Alternative explanations Alternative explanations Alternative explanations

Slide 16 -

Example After the 2003 recall election, did Democrats in the California Assembly move to the center? California ran a full legislative “season” before the October, 2003 election, then ran another “season” afterward. We can look at roll call vote behavior

Slide 17 -

Example: What Kind of Design?

Slide 18 -

History Threat Any other event that occurs between pretest and posttest Perhaps the nation was just shifting to the center at this time. How might we rule it out?

Slide 19 -

Maturation Threat Normal growth between pretest and posttest. Coming into an election year, state legislators always shift to the center.

Slide 20 -

Ruling Out a Maturation Threat

Slide 21 -

Testing Threat The effect on the posttest of taking the pretest Legislators may have learned that the state was watching them. When real tests are given, this is a big problem.

Slide 22 -

Instrumentation Threat Any change in the test from pretest and posttest A different test may have been used if a different roll call estimation technology used.

Slide 23 -

Mortality Threat Nonrandom dropout between pretest and posttest If some legislators had been recalled along with Gray Davis, this would be a problem.

Slide 24 -

Regression Threat Group is a nonrandom subgroup of population. The 2003 session was particularly extreme, any other session would look more centrist.

Slide 25 -

Multiple-Group Threats to Internal Validity

Slide 26 -

The Central Issue When you move from single to multiple group research the big concern is whether the groups are comparable. Usually this has to do with how you assign units (for example, persons) to the groups (or select them into groups). If you are not careful, may mistake a selection effect for a treatment effect.

Slide 27 -

The Multiple Group Case Administer treatment Measure outcomes Measure baseline Alternative explanations Alternative explanations Do not administer treatment Measure outcomes Measure baseline

Slide 28 -

Example Suppose USAID looked before and after at countries where it did and didn’t run governance programs in the last decade Pre-post program-comparison group design Measures (O) are all of the things Clark hates, but let’s set that aside for now.

Slide 29 -

Selection Threats Any factor other than the program that leads to posttest differences between groups. USAID did not randomly select the countries in which it ran programs, and sent aid to those with the lowest-rated governments

Slide 30 -

Selection-History Threat Any other event that occurs between pretest and posttest that the groups experience differently. For example, countries that begin with more stable democracies faced fewer challenges in the past decade.

Slide 31 -

Selection-Maturation Threat Differential rates of normal growth between pretest and posttest for the groups. It is easier to move from a semi-democracy to a full democracy than it is to move from a non-democracy to a semi-democracy

Slide 32 -

Selection-Testing Threat Differential effect on the posttest of taking the pretest. At least these measures are “unobtrusive,” so this probably is not a grave threat

Slide 33 -

Selection-Instrumentation Threat Any differential change in the test used for each group from pretest and posttest For example, the Polity measures may give some countries credit for having a USAID program

Slide 34 -

Selection-Mortality Threat Differential nonrandom dropout between pretest and posttest. Perhaps the countries with weak governments are more likely to cease being a country over the past decade.

Slide 35 -

Selection-Regression Threat Different rates of regression to the mean because groups differ in extremity. For example, the countries that USAID chooses may have nowhere to go but up.

Slide 36 -

“Social Interaction” Threats to Internal Validity

Slide 37 -

What Are “Social” Threats? All are related to social pressures in the research context, which can lead to posttest differences that are not directly caused by the treatment itself. Most of these can be minimized by isolating the two groups from each other, but this leads to other problems (for example, hard to randomly assign and then isolate, or may reduce generalizability).

Slide 38 -

Types of Designs

Slide 39 -

Types of Designs Random assignment?

Slide 40 -

Types of Designs Random assignment? Yes

Slide 41 -

Types of Designs Random assignment? Yes Randomized or true experiment?

Slide 42 -

Types of Designs Random assignment? Yes No Randomized or true experiment?

Slide 43 -

Types of Designs Random assignment? Control group or multiple measures? Yes No Randomized or true experiment?

Slide 44 -

Types of Designs Random assignment? Control group or multiple measures? Yes No Yes Randomized or true experiment?

Slide 45 -

Types of Designs Random assignment? Control group or multiple measures? Yes No Yes Randomized or true experiment? Quasi-experiment

Slide 46 -

Types of Designs Random assignment? Control group or multiple measures? Yes No Yes No Randomized or true experiment? Quasi-experiment

Slide 47 -

Types of Designs Random assignment? Control group or multiple measures? Yes No Yes No Randomized or true experiment? Quasi-experiment Nonexperiment

Slide 48 -

Design Notation Example R O X O R O O Os indicate different waves of measurement.

Slide 49 -

Elements of a Design Observations and measures Treatments Groups Assignment to group Time

Slide 50 -

Design Notation Example R O X O R O O Vertical alignment of Os shows that pretest and posttest are measured at same time.

Slide 51 -

Design Notation Example R O X O R O O X is the treatment.

Slide 52 -

Design Notation Example R O X O R O O There are two lines, one for each group.

Slide 53 -

Design Notation Example R O X O R O O R indicates the groups are randomly assigned.

Slide 54 -

Design Notation Example R O1 X O1, 2 R O1 O1, 2 Subscripts indicate subsets of measures.

Slide 55 -

Design Notation Example Pretest-posttest (before-after) Treatment versus comparison group Randomized experimental design R O X O R O O

Slide 56 -

Design Example Posttest Only Randomized Experiment

Slide 57 -

Design Example Posttest Only Randomized Experiment R X O R O

Slide 58 -

Design Example Pretest-Posttest Nonequivalent Groups Quasi-Experiment

Slide 59 -

Design Example Pretest-Posttest Nonequivalent Groups Quasi-Experiment (note multiple groups or multiple observations are REQUIRED to have a quasi-experiment) N O X O N O O

Slide 60 -

Design Example Posttest Only Nonexperiment

Slide 61 -

Design Example Posttest Only Nonexperiment X O