Seth M. Noar, Ph.D.
School of Journalism and Mass Communication
University of North Carolina at Chapel Hill
Transcript
The title of this talk is What do survey data really mean? Considering issues of causality and temporality in survey research.
So, why test theoretically-based factors and theories of health behavior? Ultimately, what we’re interested in is causal mechanisms for behavior change. What factors cause someone to change their behavior from an unhealthy to a healthy behavior? For example, starting to floss or do so more often, starting to brush one's teeth or do so more often, starting to regularly visit the dentist. So, if we can understand what the causal mechanisms are that underlie these kinds of behaviors we can then develop behavioral interventions that are affective in changing behavior.
This is a conceptual model that shows that entire process together. We develop a behavioral intervention, let’s say it’s a one-to-one counseling intervention or maybe it’s a web-based intervention that is designed to change the mediators of behavior change in the orange bubble. Those mediators change and result in change in the behavior, and ultimately what we’re interested in is change in behavior to reduce disease and improve quality of life. Note that this entire model rests on the assumption that we have correctly identified those mediators, those causal factors that, when changed, affect behavior change.
So, the $99,000 question then is, how do we discover those causal mechanisms? How do we discover those mechanisms which underlie and causally affect behavior?
The most commonly applied approach has been survey research and this is the case for several reasons. First, we can assess theoretical factors and behaviors that are not readily observable. While we can’t readily observe factors such as attitudes, self-efficacy, and norms, we can assess them through survey research. Also, behaviors that take place in the privacy of one's home, for example, brushing and flossing, can be assessed through surveys. Secondly, we can robustly measure theoretical constructs, constructs such as attitudes and norms or require multiple-item scales that can be applied in survey research. Third, it is both feasible and cost-effective to implement surveys especially with new technologies such as web surveys, iPads, and even smart phones. And, fourth, we can achieve high external validity through surveys by garnering generalizable samples in the research.
But, there are some cautions with this type of research. One caution is that this approach, that is, using survey research to try to understand these theoretical mediators, will not tell us 100 percent of what we want to know. The second caution is that the approach actually could be misleading at times. And, number three here, this approach will be strengthened to the extent that we can combine it with other approaches. So, I’m going to talk about each of these in more detail. For right now though, let's think about how we can use the survey research approach most effectively.
Let’s talk about a common application in health behavior theory testing. A student of mine conducted a cross-sectional study focused on preventive dental visitation behavior among young adults. She was interested in what factors were associated with preventive dental visitation behavior. She applied an expanded theory of planned behavior. This is a theory that suggests that factors such as attitudes, social norms, perceived behavioral control, and attentions affect behavior. She also added in additional variables that the literature suggested might be important. She assessed dental behavior, dental visitation behavior, actually in the past year, and she conducted multivariate analyses examining how those factors were associated with dental visitation. Her results suggest that the most strongly associated variables were injunctive norms, perceived behavioral control, and environment constraints.
Unfortunately, there are several common misconceptions with this approach. One is that statistically significant variables actually predicted dental visitation. While we can say in the statistical sense that something like injunctive norms predicted visitation. In a real sense it was associated with it, not necessarily causally related. So, while we can say there is an association between those variables, we cannot say that one variable caused the other.
The second misconception is the idea that changing these factors, the ones that were found statistically significant in the intervention, will lead to changes in the behavior. We haven’t actually demonstrated this and we can't be sure that changing these variables will actually lead to changes in the behavior.
Third, and finally, we can’t say that these factors, that is, the ones that were found to be statistically significant, actually led to the behavior. There are several causality related issues as well as temporal issues that we need to consider and that would show us that we can't make that assumption.
So, let’s talk about temporality for a minute. In typical cross-sectional survey research, we’re asking people about their behavior over a past period of time. In this case, we asked about the past one year since this behavior is not engaged in very often, and tends to be something like twice per year. But other variables, such as visit satisfaction, provider communication, were measured with regard to their last visit.
So, we asked people, at your last visit to the dentist for preventive reasons, please rate your satisfaction, etcetera. But of course we don't know exactly when that last visit was. For some people it might have been last week. For some people it might have been nine months ago. Still, other variables, attitudes, injunctive norms, perceived behavioral control, were essentially measured today. We are asking people for a current assessment of their attitudes, their norms, their perceived behavioral control.
And, finally, the dental fear measure that we used is actually framed about the future. It asks about the next time you visit the dentist how would you feel with regard to different fear-related items. So, when we look back now we realize wow, we really had a lot of different temporal periods in this study and that was probably not ideal. It likely had the effect of adding some noise into the study and one implication here is to be very thoughtful about time periods that one’s assessing in survey research.
So, let’s think about temporality some more. What we’re often interested in is how does a factor today affect one's behavior in the future. So, how do my attitudes today about dental visitation affect my dental visitation behavior over the coming year? And, so, if I believe that going to visit the dentist is going to reduce future cavities, reduce future dental bills, perhaps I will be more likely to go. That’s the hypothesis of what we’re trying to test, but in a cross-sectional study it actually plays out in kind of the opposite way. What we’ve measured, we’ve measured visitation behavior over the past year related to attitudes today. So, in cross-sectional research it's sort of upside down in terms of instead of looking at attitudes leading to behavior, in many ways we’re looking at behavior leading to attitudes.
So, let’s think about this for a minute. How could our attitudes about dental visitation today have led to our visits to the dentist, let’s say a year ago. Clearly they couldn’t have. So, what are we doing in this type of research? What we’re essentially doing is we’re using our attitudes today as a proxy for past attitudes because our hypothesis is that our formed attitudes before the behavior took place are what may have led to the behavior. So, we should recognize that in essentially all cross-sectional survey research, a lot of these kinds of variables we’re measuring are imperfect and we’re using them as proxies or previously held attitudes, perceived social norms, self-efficacy, etcetera.
What about a longitudinal survey? Maybe that’s a better way to do things. So, rather than a single point in time, a cross-sectional survey, we assess the person at multiple points in time, a longitudinal survey. Here we’re focusing on prospective associations rather than retrospective associations. So, let’s say we measure our key variables at Time 1 today and one year later we measure the behavior. Here, our temporality is improved because we are measuring the variables that we think are going to affect future behavior before the behavior takes place. Though, it’s still not perfect. We still have a long time lag before the behavior takes place and the time lag is still going to vary for each individual.
Still, our measurement here is more in line with our conceptual model. Our conceptual model suggests that our attitudes today are going to affect our behavior in the future, and our measurement now is measuring our attitudes first and subsequently measuring the behavior. Whereas in the cross-sectional research, it was looking at the behavior previous and the current attitudes.
But, even in the case of longitudinal research, and even if you could find a way to make the temporality between attitudes and behavior perfect, we still would not have demonstrated causality. That’s the case because there is no guarantee that factors associated with the behavior actually caused the behavior to happen. The other way to say this is that association is a necessary but not sufficient factor in demonstrating causation. The issue here is that both the theoretical factor, in this case attitudes, and the behavior, could be caused by a third factor. So, there could be a spurious relationship between those two variables. So, the implication is that cross-sectional and longitudinal survey research by themselves cannot determine causal mechanisms.
Let’s think about a concrete example of this. Political research demonstrates that car preference is associated with political affiliation. Democrats are actually more likely to drive Volvos, Subaru’s, Hyundai’s, minivans, and hybrids. While Republicans are more like to drive American cars, Porsches, Jaguars, and Land Rovers. If we interpret these data in a causal way, it would indicate that your car causes you to vote a particular way, which is not the case. Or, does your political affiliation cause you to buy a particular car? Neither of these explanations is likely to be true but those are the conclusions we would reach if we interpreted these data causally. In fact, it's much more likely that demographic, cultural, and socioeconomic explanations are at play here.
So, what’s actually going on are that the true causal factors affect both one's political affiliation as well as one's car preference, but that there is a spurious relationship between political affiliation and car preference which leads to the association. So, we have to be very careful that when finding an association we don’t automatically jump to the conclusion that a causal mechanism is at play.
This may also be the case in areas that we study. So, for example, one’s attitudes about dental visitation may be associated with dental visitation but it may not be the causal factor. Among young adults, for example, parents still seem to have a big influence and a big role and so injunctive norms may actually be a strong causal factor and attitudes may end up being simply an associated variable.
So, let’s take a step back for a minute and consider the requirements for demonstrating causation. First, we need a theoretically plausible mechanism by which causation operates. There are theories that suggest that our attitudes influence our behavior, and so this requirement is met.
Secondly, we need to demonstrate an association between variables. As we’ve already talked about, survey research is well-equipped to handle that. The problem becomes with requirements 3 and 4. Number 3 has to do with ruling out extraneous variables; that is, demonstrating that it wasn't a host of other variables that influenced the behavior, but the variable that we’re focusing on. We can try to do this with statistical controls but it is very difficult. Number 4 has to do with demonstrating that one variable causally preceded the other; that is, that the variable we believe caused the behavior took place before the behavior itself. In survey research this can be often difficult or even impossible to demonstrate.
If we looked at the causation experts, this is how they put it. Correlational designs refer to situations in which the presumed cause and effect are identified and measured. Many commentators doubt the potential of such designs to support strong causal inferences in most cases. So, in another words, we have to remember that even though we’re interested in causal mechanisms and trying to understand them, correlation is not causation.
One other consideration related to association, the association of a presumed theoretical mediator with behavior is often interpreted as the effect on behavior. But we must remember that several theories also suggest that our behavior affects the mediator. So, for example, my attitude may indeed affect my dental visitation behavior, but if I have a terrible experience at the dentist, that may affect my dental visitation attitudes, which may in turn, affect behavior in the future. So, there may be a dynamic interplay between those variables.
Also, we sometimes change our attitudes to fit our behavior, also known as rationalizing. This may especially be a problem in cross-sectional research because we are measuring all these variables at the same time and thus, we cannot get at these different kinds of interplays between these variables.
Going back to what I said earlier, the survey research approach will not tell us 100 percent of what we want to know. While it can demonstrate association it cannot demonstrate causation. Secondly, the approach could be misleading at times. It may lead us to think a particular variable plays a causal role when it does not. Temporality in measurement issues also will affect this.
Moreover, the survey research approach will be strengthened by the extent to which we can combine it with other approaches. Approaches such as experimentation are much stronger at demonstrating causal mechanisms. Thus, we can take variables that appear promising in survey research and apply them in experiments. Other techniques, such as quasi-experiments and mediation analysis interventions, also can help.
The traditional paradigm for health behavior theory has been completely reliant on survey research, mostly cross-sectional surveys, as well as meta-analyses of those surveys.
We’ve recently suggested a new paradigm, where survey research still plays a role but we also have several other methodological techniques that are used to test health behavior theory, including lab experiments, field experiments, mediational analysis and interventions, and using meta-analyses of these different techniques. This way, survey research gives us some clues as to what those important factors may be but we can test them with stronger designs using experimental approaches.
So, let’s look at that model from earlier. We designed a behavioral intervention to change particular mediators. Changing those mediators should lead to change in behavior. Often times we do this in intervention studies but we don't actually go back and analyze first, whether the mediators changed, and second, whether change in the mediators led to change in the behavior. Often we simply look at, did the intervention work, did behavior change, and then we say we had success or failure. What's really important is for us to measure the mediators at several places in the study and to conduct meditational analyses to see, first of all, did the mediators change and secondly, did changing the mediators affect change in the behavior.
In conclusion, neither cross-sectional nor longitudinal survey research can definitively determine causal mechanisms. So, why do it at all? Is there even a role for it? Let’s remember, association is one of the requirements for causation, and survey research is very good at demonstrating association. In most cases though, it will be better for ruling causal factors out than for definitively ruling them in. Factors that are ruled in can be the subject of future experimental research and other research that is stronger at demonstrating causal mechanisms. Also, whether cross-sectional or longitudinal, survey design and temporal dimensions should be carefully thought out in order to conduct the most accurate study possible.
Thank you.