Skip to main content

Causal Inference Course

Discussion 2. Analyzing an Experiment in R

STSCI/INFO/ILRST 3900: Causal Inference

September 2, 2026

Announcements

  • Office Hours throughout the week (see Syllabus or website)
    • Filippo: Thursday 4-5pm in 325A CIS Building
    • Shira: Monday 5-6 pm in 325A CIS Building
  • No need to submit, encouraged to complete the file and knit
  • In-class assignments will start later in the semester

Generative AI and Math Learning Experiment

Experimental Design

  • Nearly 1,000 high school students in Turkey were randomly assigned 1 of 3
  • High school math students were randomly assigned to one of three treatment arms:
    1. Control: Standard practice material with no AI assistance access
    2. GPT Base: Access to standard GPT-4
    3. GPT Tutor: Access to a customized GPT-4 (GenAI with guardrails)
  • Following this practice session, all students took an exam without AI access to evaluate learning

Goal for Today

Replicate Something Similar to Table A.1 (in appendix) and Table 1

  • Table A.1 demonstrates balance of covariates across treatment arms
  • This ensures treatment assignment is randomized so exchangeability holds
  • Table 1 shows the main results of student performance in the practice and exam problems
  • Identify the average Treatment effect (ATE) utilizing randomization \[ACE = E[Y^\text{a=1}]- E[Y^\text{a=0}] = E[Y \mid A=1]-E[Y\mid A=0]\]

Resources for Markdown

Step 1: Download the .Rmd file here

  • Start by running the code in Section “Necessary packages”
  • If you get an error, you may need to install the package

Step 2: Import and Clean the Data 

genAI_dta <- read_csv("https://raw.githubusercontent.com/causal3900/causal3900.github.io/refs/heads/main/assets/discussions/discussion02_files/GenAI_Learning.csv")
  • Quick peek at the dataset using the function glimpse
  • Notice the variable Honors
  • Notice the treatment variables GPTBase and GPTTutor 
glimpse(genAI_dta)
  • …students are randomly assigned to classrooms (with the exception of honors-designated classrooms)…
  • Exclude honors-designated classrooms from the sample
  • For this, you will want to use the function filter
  • The general syntax is filter(condition)
  • A condition would be Honors == 0 
genAI_dta <- genAI_dta %>%
  filter(...)
  • Create a single treatment variable with word labels:
    • “Control”: If GPTBase == 0 and GPTTutor == 0
    • “GPTBase”: If GPTBase == 1 and GPTTutor == 0
    • “GPTTutor”: If GPTBase == 0 and GPTTutor == 1
  • For this, you will want to use the function case_when which is described here
  • The general syntax is case_when(condition ~ output-value)
  • A condition would be (GPTBase == 0) & (GPTTutor == 0) and an output value would be "Control" 
genAI_dta <- genAI_dta %>%
  mutate(treatment = case_when(...))

Step 3: Table A.1

  • Is the data balanced on covariates?
  • We want to check that the treatment groups are balanced on covariates
  • For each treatment arm/group, calculate the mean for each of the designated covariates in table A.1 (list given below)
  • Use group_by() to calculate separate means for each treatment arm
  • Use summarise() to compute the mean of each covariate in covariates 
covariates <- c("education_parent_college", "n_household_members",
                "n_household_children", "class_enjoyment",
                "class_participation_likelihood", "math_hw_completion", "hw_help",
                "private_tutorship","visit_training_center","female",
                "n_weekday_study_hours","n_weekend_study_hours","gpa_prev")

genAI_balance <- genAI_dta |>
  distinct(`Student ID`, .keep_all = TRUE) |>
  group_by(...) |>
  summarise(across(all_of(...),\(x) mean(x, na.rm = TRUE)))


genAI_balance |>
  pivot_longer(-treatment, names_to = "Description", values_to = "value") |>
  pivot_wider(names_from = treatment, values_from = "value") |>
  select(Description, GPTTutor, Control, GPTBase) |>
  print()

Step 4: Table 1

  • What are the main results of the experiment?
  • For each treatment arm, calculate the average grade of students in the:
    • Practice problems (Part2Tot)
    • Exam problems (Part3Tot)
  • Use group_by() to calculate separate means for each treatment arm
  • Use summarise() to do the following:
    • Create a column Practice_perf- the average grade in step 2 of each group
    • Create a column Exam_perf- the average grade in step 3 of each group
  • Note that your numbers will not match up exactly with Table 1
genAI_means <- 
  genAI_dta %>%
  group_by(...) %>%
  summarise(Practice_perf = mean(...),
            Exam_perf = mean(...))

print(genAI_means)
  • Use the sample means to estimate the average treatment effect
ATE_practice <-
  genAI_means$Practice_perf[genAI_means$treatment!="Control"] -
  genAI_means$Practice_perf[genAI_means$treatment=="Control"]

ATE_exam <-
  genAI_means$Exam_perf[genAI_means$treatment!="Control"] -
  genAI_means$Exam_perf[genAI_means$treatment=="Control"]


print(cbind(ATE_practice,ATE_exam))

Example from Lecture

  • Given potential outcomes, you chose a diet
  • Are \((Y^{a=0},Y^{a=1}) \perp\!\!\!\perp A\)?
  • Observational setting: no access to potential outcomes, only observed outcome
\(Y_i = \text{Disease } (0)\) \(Y_i = \text{No Disease } (1)\)
\(A_i = \text{Delicious } (0)\) 20 54
\(A_i = \text{Healthy } (1)\) 0 64

\[\widehat{ATE} = E(Y \mid A = 1) - E(Y \mid A = 0) = \frac{64}{0+64}- \frac{54}{20+54} \approx 0.27\]

True ATE

  • 50% of students have \(Y_i^{\text{Delicious}} = 1\), \(Y_i^{\text{Healthy}} = 1\)
  • 50% of students have \(Y_i^{\text{Delicious}} = 0\), \(Y_i^{\text{Healthy}} = 1\)
  • \(Y_i^{\text{Delicious}} \sim Bernoulli(0.5)\)
  • \(Y_i^{\text{Healthy}} \sim Bernoulli(1)\)

\[ATE = E(Y^{\text{Healthy}}) - E(Y^{\text{Delicious}}) = 1-0.5 = 0.5\]

Assume a Randomized Trial 

n <- 138

Y1 <- rep(1,n)
Y0 <- c(rep(0,n/2),
        rep(1,n/2))

set.seed(1231)
A <- rbinom(n, 1, 0.5)

Y <- ifelse(test = (A==1), yes = Y1, no = Y0)

print(table(A,Y))
##    Y
## A    0  1
##   0 37 41
##   1  0 60
\(Y_i = \text{Disease } (0)\) \(Y_i = \text{No Disease } (1)\)
\(A_i = \text{Delicious } (0)\) 37 41
\(A_i = \text{Healthy } (1)\) 0 60

\[\widehat{ATE} = E(Y \mid A = 1) - E(Y \mid A = 0) = \frac{60}{0+60}- \frac{41}{41+37} \approx 0.47\]