clean-survey-data

Clean Survey Data

A skill for cleaning survey data in R, handling common patterns like missing value codes, variable recoding, and Stata label conversion.

Quick Start

library(data.table)
library(rio)
library(haven)

# Load data
dt <- as.data.table(import("data.dta"))
names(dt) <- tolower(names(dt))

Key Patterns

1. Convert Stata Labels to R Factors

Use this function to extract value labels from Stata variables and convert them to proper R factors:

attach_label_to_variable <- function(x, na_exclude = TRUE) {
  var_lab <- attr(x, 'labels')
  if (na_exclude) {
    # Remove common missing value codes
    var_lab <- var_lab[!var_lab %in% c(91, 92, 97, 98)]
  }
  if (!is.null(var_lab)) {
    x <- factor(x, levels = var_lab, labels = names(var_lab))
  }
  return(x)
}

# Usage
dt[, education := attach_label_to_variable(education)]
dt[, marital_status := attach_label_to_variable(marital_status)]

2. Handle Missing Value Codes

Survey data often uses special codes for missing values (e.g., 91=refused, 92=don't know, 97=not applicable, 98=skip):

# Single variable
dt[variable %in% c(91, 92, 97, 98), variable := NA]

# Multiple variables at once
missing_codes <- c(91, 92, 97, 98)
vars_to_clean <- c("var1", "var2", "var3")
for (var in vars_to_clean) {
  dt[get(var) %in% missing_codes, (var) := NA]
}

# Using data.table syntax for all numeric columns
dt[, (numeric_vars) := lapply(.SD, function(x) {
  ifelse(x %in% c(91, 92, 97, 98), NA, x)
}), .SDcols = numeric_vars]

3. Recode Categorical Variables

Create meaningful categories from coded responses:

# Race/ethnicity recoding from multiple binary indicators
dt[, race_single := NA_character_]
dt[race_white == 1, race_single := "White"]
dt[race_black == 1, race_single := "Black"]
dt[race_asian == 1, race_single := "Asian"]
dt[ethnicity_hispanic == 1, race_single := "Hispanic"]

dt[, race_single := factor(race_single,
    levels = c("White", "Black", "Hispanic", "Asian", "Other"))]

# Age groups
dt[, age_group := cut(age,
    breaks = c(0, 30, 40, 50, 60, 70, 80, 110),
    include.lowest = TRUE,
    labels = c("18-29", "30-39", "40-49", "50-59", "60-69", "70-79", "80+"))]