Tuesday, September 27, 2011

Interpreting the "trend effect" when we include a marginal effects

Let's say we have a model:

y=bX+ error

b is negative

And when we add an interaction term:

y=bX+ c(X*Z) + error

now b become spositive but c is negative.
Does the coefficent of b have any meaning in this case, should I just at the overall effects?

Person B:
Yes, b definitely has meaning?
I don't know any papers off hand that just deal with interactions, but any d-d paper (like duflo education) would interpret interactions, since rct's are always an interaction of time and the program.

If you want the marginal effect of X, they dy/dx= b+ cZ^, where Z^ is a specified value, like the mean of Z or whatever you decide.
if Z is a dummy, then it's just when the dummy is turned on.

If b changed sign, I'd be worried about multicollinearity (how collinear is X with X*Z?), or Z having been omitted such that Z and X are negatively correlated.

The latter seems likely if c is negative and b goes to positive.

Person A:
No b has meaning yes as it cnotributes to the caclulation of the overall effects. but b alone does not have meaning.

Person B:
In program evaluation b definitely has meaning, e.g. if X were time, and Z were the program, then b would tell you the average time trend effect, and c would tell you the program's effect (over time).

Person A:
Would you care about the average time trend effect and not the program effect?

Person B:
Yes-it would change the interpretation of what the program effect is doing.
If b>0, then program effect is enhancing, if b<0, then the program effect is minimizing a crisis.

You'd also care about the b, because it measures whether your randomization was done properly (namely, if X were program, and be measured just the effect of the program, not over time...good randomizaiton should make b insig...but that's a different story. I don't know what you're regression is.)

Person A:

Hmm. So these politcial scientists are wrong? Look at bottom page 71 and beginning of page 72 in attached document.
(Understanding Interaction Models: Improving Empirical Analyses, Brambor et al 2005).

Person B:
They're right, and I'm saying the same thing.

I guess I can be more specific by saying b is the average/trend effect holding all else constant, but the full marginal effect of X is: b+cZ

Person A:

I thought what they are saying is that if you have the interaction term you cannot say b is the average trend effect. This is only true if you dont have the the interaction effect, but only have b.

"Scholars should refrain from interpreting the constitutive elements of interaction terms as unconditional or average effects—they are not...As a consequence, the coefficient on the constitutive term X must not be interpreted as the average effect of a change in X on Y as it can in a linear-additive regression model. As the above discussion should have made clear, the coefficient on X only captures the effect of X on Y when Z is zero."

I thought this menas b is only valid in interpretation if you DO NOT have an interaction term in the estimation.

Person B:
To me avg conditional effect is the same as saying avg effect holding all else constant.
They're saying b is not the avg undconditional effect, which is true, ie. it's conditional.

But I think they're main point is that the marginal effect of X on Y is not just b if an interaction effect is present.
Of course, it's sort of self enforcing. It is the researcher that decides that the marginal effect be introduced or not.
I suppose you can introduce it, see if it's sign, and then if it's not dump it. But, one probably has reason to believe that the interaction should/shouldn't be there.

That's my take.

Monday, September 12, 2011

Over-Use of Randomization?

I just watched A. Deaton's talk on RCTs. It is very good and I think we should all take the time to watch/listen:


The basic idea he advocates is not blindly implementing RCTs and knowing what you are getting when you do (or even when you implement them without total blindness). From my recent experience trying to design an impact evaluation, I find myself wondering if randomization was necessary, and asking why, if we do use randomization, do we need a baseline? D-in-D requires a baseline of a treatment and a comparable control group. For RCTs we should technically only have to compare the outcome of interest after the treatment -- if the randomization worked we do not need to control for initial conditions. But not controlling for initial conditions seems like a very bad idea to me, in social sciences, since different initial conditions can lead to drastically different outcomes. And, after all, we cannot randomize on unobservables, the very thing we are trying to correct for.

Since we cannot observe the unobservables, there is no way to tell if your randomization really worked. We use means of observables of treatment and control as a way to suggest whether or not randomization worked for unobservables, but that same distribution according to observables can be achieved with matching techniques and the like. I find myself wanting to use randomization AND D-in-D so that I can achieve decent comparison groups, but D-in-D actually was born because economists could not randomize. Is randomization plus D-in-D an overkill? Do I spend a lot of money for an extensive baseline if randomization means I do not need outcome measures from the baseline?