## Thursday, December 5, 2013

### Strata Weights from Duflo, Glennerster, Kremer 2008

Written with StackEdit.

On pg 3935 of **USING RANDOMIZATION IN DEVELOPMENT ECONOMICS
RESEARCH: A TOOLKIT**, the authors discuss how to weight an average treatment effect, when the probability of treatment varies within strata.

I had such an issue, and so looked to this page for help. But the information provided on this page does not seem to be correct. Essentially, the authors say that weighting the treatment effect (namely, the average difference between treated and untreated) in each strata by the probability of being in that strata, conditional on treatment is equivalent to running a regression of the outcome on the treatment dummy controlling for strata dummies (and all interactions between strata dummies, if the sample was stratified along more than one dimension, say city and gender).

This is likely not the case, as the variance covariance matrix from a weighted regression versus one with additional dummies will not be the same.

The ATE is measured by:
[E(YT|X,T)E(YC|X,C)]$[E(Y{^T}|X,T)- E(Y{^C}|X,C)]$ and we are interested in the overall effect:

When the data are stratified, Duflo et. al. suggest the following:
Ex[E(YT|X,T)E(YC|X,C)]$E_x[E(Y{^T}|X,T)- E(Y{^C}|X,C)]$= Σx(E(YT|X,T)E(YC|X,C))P(X=x|T)$\Sigma_{x}(E(Y{^T}|X,T)- E(Y{^C}|X,C))P(X=x|T)$
(duflo_toolkit.pdf, pg 3935), where, using data,

• Y–>outcome
• X–>strata
• T–>treatment==1 and C–> treatment==0

The authors use a continuous strata in their equation (an integral sign). I’m not sure why, as strata are usually discrete. The authors leave it at the above, but I’ll walk you through what those weights mean first.

So , what is P(X=x|T)? It would be great if the authors spelled this out.

P(X=x|T)=P(T|X=x)P(X=x)P(T)$P(X=x|T)=\frac{P(T|X=x)*P(X=x)}{P(T)}$
where P(T)=P(T|X)P(X)$P(T)=P(T|X)P(X)$

An example of computing these weights follows with my own data.

An Example

Here is a summary of observations from an experiment. You can see from this simple table, that the probability of being treated in each strata is very different. In strata 1, 107 out of 648 individuals were treated, whereas in strata 2 98 out of 1,075 individuals were treated. So we need to correct for this, if we’d like an average effect of the treatement variable.

strata1234totaltreatment=06481,0751,5512,0215,295treatment=1107989998402total7551,1731,6502,1195,697

So, P(X=x|T)=P(T|X=x)P(X=x)P(T)$P(X=x|T)=\frac{P(T|X=x)*P(X=x)}{P(T)}$, where, for example, the weight in strata 1 is: 1077551414(107755+981173+991650+982119)=0.427495568$\frac { \frac{107}{755}*\frac{1}{4} }{\frac{1}{4}* (\frac{107}{755}+\frac{98}{1173}+\frac{99}{1650}+\frac{98}{2119} ) }=0.427495568$

Handbook’s Suggested Methodologies

Supposed we’d like to estimate the effect of the treatment on an outome. Our hypothesis is:

H0: The effect of the treatment on outcome is zero.

Duflo et. al. gives two methods of estimating the treatment effect correcting for the fact that the probability of treatment depends on the strata as follows:

1. Run a regression with controls for each strata

2. Use weights in the form of: P(X=x|T)=P(T|X=x)P(X=x)P(T)$P(X=x|T)=\frac{P(T|X=x)*P(X=x)}{P(T)}$

Both (1) and (2) can be done in a regression form. Given that, how does the OLS estimator (or probit, since my primary outcome variable is binary) and it’s standard errors (clustered at the strata level) change for (1) and (2)? Is β^$\hat{\beta}$ the same? Is the variance covariance matrix and therefore inference the same for (1) and (2)? Probably not.

General OLS estimators:

β=(XX)1(XY)$\beta=(X'X)^{-1}(X'Y)$ be the effect of treatment on the outcome, and let

sd(β)=Σs(XsXs)1(XsΩsXs)(XsXs)1$sd(\beta)=\Sigma_s (X_s′X_s)^{-1}(X_s'\Omega_s X_s)(X_s′X_s)^{-1}$, where s denotes strata. Note that X1$X_1$ is 775x1, while X2$X_2$ is 1173x1, etc.

where….

Ω=$\Omega=$

Ω10000Ω20000Ω30000Ω4

and

Ωs=E(ϵsϵs)$\Omega_s=E(\epsilon_s \epsilon_s')$, for strata s, with Ns$N_s$ individuals, and vector of error terms ϵs$\epsilon_s$, is:

ϵ1sϵ1s.ϵ1sϵNs...ϵNsϵ1s.ϵNsϵNs

1.If we use controls, X=[treatment, strata1, strata2, strata3,strata4], Y=[outcome] (I guess we can have a dummy for each strata and suppress the intercept or, take one strata dummy out). β$\beta$ and sd(β$\beta$) are as given above.

2.If we use weights, then X=[treatment]’, Y=[outcome]’, and W=[vector of strata weights]’, for example, w1=P(X=1|T)=.42$w_{1}=P(X=1|T)=.42$, and is constant within stratas 1

β=(XW1X)1(XW1Y)$\beta=(X'W^{-1}X)^{-1}(X'W^{-1}Y)$, and

sd(β)=(XWX)1(XWΩWX)(XWX)1$sd(\beta)=(X′WX)^{-1}(X'W \Omega^* W'X)(X′WX)^{-1}$, where:

Ωs=E(ϵsϵs)$\Omega_{s}^* =E(\epsilon_{s}^* \epsilon_{s}^{*'})$, for strata s, with Ns$N_s$ individuals, a vector of error terms ϵs$\epsilon_{s}^{*}$,

Perhaps (1) and (2) could produce equivalent results for β$\beta$ if the added controls in (1) create a weighted sum of the x’s for each strata that is equivalent to the weights.

In terms of the variance of beta, Ωs$\Omega_{s}$ should not be the same as Ωs$\Omega_{s}^*$, given that the error terms come from different linear models between (1) and (2).

More General Issues on Heterogenous Treatment Effects

Stratifying a random sample can often be used to get a representative sample within strata, allowing the research to look at the treatment effect by strata. Without stratification, if we subset the data to a certain city, for example, we can’t be sure that our sample within that city is representative of the population.

From a Bayesian perspective, this would be equivalent to saying that our prior distribution of the data is incomplete–we’re missing a whole matrix of individuals within that city who would respond differently to the treatment than the ones we happened to pick up in our sample that was not stratified by city.

In fact, some great Bayesians, like Andrew Gelman, have writen about this issue from a Bayesian perspective: http://www.stat.columbia.edu/~gelman/research/published/bayes_management.pdf
http://www.stat.columbia.edu/~gelman/stuff_for_blog/feller.pdf

There are several other methodologies out there that relate to heterogenous treatment effects that use machine learning methods rather than the chop, dice and data mine methods:

Imai’s work at Princeton essentially says, rather than look at all the interaction effects between treatments and strata to decide on which treatment is best for which strata, let’s reduce the problem to just a few treatments and strata a priori. The algorithm is here, but it’s essentially an optimization problem with an added constraint that dampens the effect of some interaction effects (hopefully, I’m getting this right):
Imai: http://imai.princeton.edu/research/svm.html

Yet, one more method that I haven’t delved into is by Grimmer et. al.:
Grimmer: http://www.stanford.edu/~jgrimmer/het.pdf

The traditional go-to method in the sciences of throwing in many interaction effects between treatment(s) and a strata (or whatever you may be conditioning) is wrought with two major issues:
1. It expands your parameter space, requiring a much larger sample to maintain the same power as before.
2. Designs don’t exogenously vary both the treatment AND the strata conditioned on, so the interaction cannot necessarily be interpreted as causal.

It’s time to consider new methods, I think.