Michael Davern, Ph.D.
NORC at the University of Chicago
Transcript
Hello, I’m presenting a slide show of collaborating with survey centers to conduct survey research.
The first question you should ask yourself when you are thinking about conducting a survey is, surveys are hard to do well and they are expensive. So, always ask yourself, is there existing survey data that I can use to answer my research question? Or, is the research question appropriate for survey research? And, finally, do I have an appropriate level of funding to conduct the research that I’m interested in doing. After you have reached a conclusion or would like to discuss these issues with someone, then you should seek out an experienced survey professional, who are often housed within survey centers.
The first question that I posed at the beginning that you should ask yourself—is there an existing source of data to answer my research question? In other words, do I actually need to conduct a survey or can I just go get existing data to answer the question I’m interested in knowing more about. Good places to find existing survey microdata and documentation of those survey systems can be found at ICPSR at the University of Michigan. The web address for the ICPSR is on the screen. Also, IPUMS at the University of Minnesota, and the National Bureau for Economic Research, NBER, also has many datasets and very good documentation of those datasets.
Survey centers can help you find and analyze these data sources as well if you are unable to find what you want or would like to know if there is existing data. So, survey data centers often don’t just collect survey data but they are often able to point you in the right direction of finding existing data and also help you analyze that data, because analyzing survey data often takes specific skills that are not universally available to folks and it would be useful to work with people who know survey data and [Cuts off].
The second question I posed at the beginning—is your question appropriate for survey research? Is my research question appropriate to the methodology for survey research? Some work well and others do not. I’ve traffic-lighted using green and yellow and red. The green worked well, the yellow somewhat more difficult, the red not likely to be answerable by surveys. Examples of questions would be, What proportion of the population has visited the dentist in the past year? This is something you could ask someone and theoretically they should be able to get a good response to the question of Did you visit a dentist in the past year?
Questions that may be a little harder to answer with surveys, but still possible is Why did a person visit the dentist in the past year? How does behavior influence the likelihood of a cavity, crown, or root canal or dental emergency? So, asking questions about health behaviors that may be associated with those types of procedures or those types of need often are very hard to ask people about. For example, how often they may drink soda or how often they may eat food that has high content of sugar or how often do they brush their teeth, and do they floss and what types of tools do they use for oral hygiene. These are all health behavior issues that need to be, or information collected on, to understand how behavior influences outcomes.
The next question is, What proportion of the population is in need of dental services? This gets harder as you may want to define need as real needs, as someone who actually has a cavity and needs it filled, or someone who has the perceived need that they know that they need to go see a dentist but have not been able to go in perhaps due to the expenses associated with going to see a dentist. So, you really need to understand what need is, and if need is a perceived need you can ask about it. If it is a clinical need, it gets harder to know without doing a clinical examination and that may be out of the realm of a survey.
The next questions are very hard to answer using surveys, if not impossible, and that is why they are in red, What proportion of the population has a cavity and does not know it?, What proportion of the population has gum disease?, What proportion is at risk of experiencing serious pain due to an unmet dental need in the next year? All of these would require clinical examination of some sort, not that it is impossible to do within the survey realm, but it’s very expensive if you are going to have a clinical examination associated with your data collection.
Next question that I posed at the beginning is—do I have the appropriate level of funding to conduct a survey? To do a survey well is more expensive than most people think. I often use a guideline of a minimum of $50,000 is generally needed and cost only goes up from there.
I would consult a survey center about the costs of various components you are thinking of. I have seen survey costs range from anywhere from $20 per completed survey to over $2,000 for a completed survey. This cost variance depends on the types, modes, you are willing to use, for example, web or mail intercept surveys where you just take people who are available to you and you don’t care about having a representative sample of the population on the low end, to a survey that has a biomarker collection associated with it and also sending those specimens off to labs to have them processed and get the data back and make that all part of the existing survey system and combining that with the answers on the survey that the respondent had.
Those can be very, very expensive. And as we talked about earlier, some of questions that are much harder to be answered by survey research may involve a clinical exam which would drive that price up even higher.
Researchers are often tempted to get surveys done cheaply by using graduate students and ignore their own time. In general, I would resist this because the quality of the work is often not reproducible and suffers, and science is based on the idea of replicability. And, you need to make sure that your survey is up to the scientific standards and has the appropriate documentation. So, it is best to work with people who have experience conducting surveys and documenting those surveys in order to make sure your survey will make a contribution to the scientific literature.
Basics on collecting survey data, it’s important to remember that we can ask people about things they are likely to know. Which of the following will people be able to provide a valid answer to, do you think? Again, I have put these in green and red. Do you have dental insurance coverage? People should be able to answer that, although there is, there are issues with knowing whether or not someone has dental insurance. They may confuse it with health insurance, in general, but people are fairly accurate with these kinds of questions.
Questions that people typically aren’t good at answering will be, What is your premium for your dental insurance? or What is your deductable? Perhaps someone who has reached their deductable or has used their deductable would know, but people who don’t or haven’t would not. And so, this becomes a question that you won’t get answered accurately by everyone and it turns out to not be a very easy question to get answers through survey data or survey collections.
How much did your last dental visit cost in total? That may not be something they know. They may know, though, what their total out of pocket expenses were for their last dental visit. You could ask them about that, but they may not know the total, especially if they have insurance and the total could vary by what was billed to the insurance company versus what would have been billed to the individual who doesn’t get the discount that the insurance company does. So it is a very confusing concept. But most people could probably give you an idea of what the out-of-pocket expenses were for the last dental visit or the last dental visit of their children or spouse.
During the last year, were you told you had a cavity or needed a crown, or root canal? There again is a good example of a question that most people would be able to give you an answer to and the [inaudible].
Survey centers can help you set up processes in addition to the questions that people can answer, that you can ask people that they can answer that we just talked about. Survey centers can also help you set up the processes to link respondents to other sources of information; for example, results from a clinical exam, x-rays, digital image, or claims data themselves.
Survey centers can also help you collect biomarkers such as DNA, weight, blood, such as draws or spots, blood pressure, and urine, and for the specimen we can have those sent out to a lab to be processed and put those data back on the survey. That, combined with data [and prepare that for analysis.] Cut off. On slide 7.
So, why should you engage a survey center? Surveys may look simple, but many things look simple when you are looking at it from the outside. But survey centers know how to engage potential sources of error. They’ve been through the process before, they know how to anticipate the problems that you are going to have and face, and they know how to minimize them and their potential impact, and also the potential impact of those errors on scientific inference. Survey centers know how to create the appropriate survey documentation and methodology reports, which are essential reference materials when writing up results for scientific audiences. These materials are absolutely essential to have as you’re writing journal articles and/or reports based off your data and drawing scientific inferences.
Survey centers also know how to look and analyze existing sources of survey data. We can do that in conjunction with the survey we conduct for you or we can do that as a whole study in and of itself; that is, just the study of existing sources or we can compare some of the items that you would have on your survey that are also on other surveys to look to make sure that we are getting similar answers and that your survey, that we, that survey centers helped you conduct is valid.
Survey centers can perceive the problems that you will have and provide you with the appropriate counsel to make an informed decision about the trade-offs and there’s always trade-offs in surveys. You need to have someone help you make those decisions about whether you should contact people 16, 17, 18 times on the telephone to get a response or should you replace that person who hasn’t responded yet with a new telephone number and call it. It gets very, very expensive to re-contact folks and at some point, you need to say, okay, I’m going to enter a new piece of sample and try to contact that person. Of course, because you have a non-respondent, your response rate, your total number of people sampled who participate in the survey, will go down. And, at some point, you need to make those kinds of informed decisions, and people at survey centers can help you do that and understand how that’s likely to impact your final research and your final analysis as you write it up, to make a scientific contribution.
We have established processes and protocols that have been scientifically tested and improved for over 80 years of scientific survey research. And, most survey centers have experts who can help you [make the best decisions for your specific research design.] Cut off. On slide 8.
So, when should you engage a survey center? You should contact a survey researcher or survey center as soon as you can and as soon as you are thinking of conducting a survey. If you are applying for a grant, for example, or asking for funding from a foundation, you want to make sure to get a realistic process before submitting it. The reason you want to do this is because a) it makes your grant application look stronger to those on the review panel who will be able to know what the real cost of the survey and doing it well will be; and also it allows you to make sure you have enough money, you’ve asked for enough money for the grant or foundation proposal to actually conduct a high quality scientifically rigorous survey.
Most survey centers are more than happy to talk with you at the foundation stages to building a budget and understanding what that budget should look like and needs to look like in order to answer your research question.
And, that brings us to how to get pricing from a survey vendor. You can meet with us to discuss the ballpark figure and costs and trade-offs of various designs. You can informally request pricing from several survey vendors. Often they will comply; not everyone will for various reasons. But, you’ll be able to get some vendors who will certainly supply you with some ballpark cost figures. And then, also, you can go through a formal request for proposal process in which you send vendors a [cut off].
There’s different types of survey centers that you should know about. There are those that are based at universities. There’s an association called the Academic Association of Survey Research Organizations. The web link is here. And, there are several survey centers within academic homes within universities. There are those that do Federal government contracting, primarily, such as the one I work for, which is NORC at the University of Chicago. There are those that predominantly do market research, for example, Gallup which you may have heard of. There are some that do a mix of market research, and federal, state, and government work, and a very good source on a lot of different survey centers and one that potentially is by you, in your neighborhood, is the AAPOR Blue Book, as it is called. It used to be a blue book but now it is a web reference, but the cover on the web is still blue. And, you can find it at this web link. AAPOR is the America Association for Public Opinion Research, and is the main scientific body that oversees survey research. And these are the vendors of which, that are members of AAPOR.
So, what survey centers can do for you. This is what survey centers are very good at thinking about that you may not be because you are someone who just wants to use surveys to answer your research questions but not an expert in designing and understanding surveys. Survey centers can design a sample and a sampling plan and know how to design an efficient sample in an optimum allocation of sample based on the resources and constraints that you provide for the survey centers. So, it is very important for you to engage with someone who knows how to do that early on so that you understand how much sample you are going to need, what’s the best way for you to draw your sample, and what are the various constraints and/or trade-offs.
Survey centers are also very experienced with IRBs and if you are doing Federal contracting work, Office of Management and Budget applications as well. So, both IRB and OMB packages, putting them together and getting them approved and knowing how to do that can save you a great deal of hassle and time and knowing what the IRB and OMB is interested in and writing an appropriate response to any issues that may come up is something that survey centers know how to do very well.
Survey centers also know how to help you with instrument development. They know how to develop questionnaires, scales, whether you want to measure mental health or behavioral health issues. We have tools, we know how to do it, and we can tailor it to your specific purposes.
We also can help you select modes of surveys and this could be an in-person survey where we actually knock on people’s doors. It could be through telephone, where we call people on the telephone, and the telephone could be cell phone or landline telephone. We also do web surveys and mail surveys. We also like to mix all these things together and try to contact people in a variety of different ways in order to increase our participation among the respondents, offering them a variety of ways to respond to the survey.
We also know how to develop the operational procedures which are necessary to conduct high end survey research, such as dialing landlines and cell phones, mixing survey modes; for example, doing a survey that can be done either over the web or mail or both or other variety of modes as well, such as telephone and in-person. And, also the collection of paradata, such as the number of phone calls to elevate your response rate, such as how many attempts did you make to a specific number, the amount of time to complete phone interviews, and the number of surveys mailed out to a phone number, you also may be making more than one mailing, more than one attempt to get them to respond to the survey.
You want to keep track of all that information so that you have it and can use it in your analysis to understand maybe what the difference was between the people who responded early, the people who responded later, to understand the people who you didn’t get any response from, the people you did get responses from, to understand in a computerized instrument how long it took people to answer certain survey questions relative to other survey questions. Some questions are much more difficult to answer and take people longer periods of time. The computer automatically records this [in your analysis.] Cut off. On slide 11.
What else can the survey centers do for you? We can help develop best design for your research question so that you optimize the design. We also know how to conduct interviewer trainings. We can train interviewers if we’re conducting a telephone or in-person survey to understand and lead people through the instruments. We can obviously interview the respondents and provide the interviewers to do that. We can prepare the data. We prepare data sets for analysis by editing inconsistent data and imputing missing data. And, if people refuse or don’t answer, we can replace those values using statistical modeling techniques that are common within the survey research field. We can also develop coding schemes for open-ended responses that you may have in your survey. This helps a great deal so that we can take our experience in doing this kind of work to help you get to an analytic data file very quickly.
We also know how to conduct disclosure limitation editing and review the data for threats in confidentiality. So, we can make sure that you have a safe data set in which you can share with others or create a public use file, and it will not threaten the privacy of any individual respondent on the data file. We know how to edit the data file to make sure to take off the sensitive items, certainly personally identifying information, and other pieces of information which may pose a threat to re-identification.
One last slide of what survey centers can do for you. We can link data to other sources of data, such as claims, x-rays, digital images, enrollment data, environmental sources of data. We can prepare methodology reports for you in great detail, allowing you to document everything, the process of which how we did everything so that when you go through the report writing process and you write your book or journal article, you have reviewers asking questions—well, did you conduct your interviews in Spanish and English, and you can’t remember, we have written the methodology report so that you can go back and cite the reference and find out, yes, we did conduct the survey in both Spanish or English.
We also collect file markers. We can help you set up file marker collection protocols and we can also set up labs to process the specimens and put that data back on to data files and then give it to you prepared for analysis.
We also know how to include within our documentation the appropriate metadata and documentation that follows the Data Documentation Initiative if you would like, which is a unified way of presenting survey data documentation.
We know how to conduct analysis including the appropriate handling of an analysis of complex survey data which, due to the nature of the sample designs in most surveys, it’s not adequate to just open up SAS or SPSS or R or Stata or other statistical software and begin working with the data file as those software default settings generally speaking, the data were not collected as part of a complex sample design and in most cases survey data are collected as part of a complex sample design and that sample design needs to be taken into account and those software packages can take it account. You just have to tell them how to, using survey procedures and we can show you how to do that and we can write up procedures for how to do that and put examples into the documentation as well.
We can evaluate the quality of the data. We can investigate anomalies in the data and we can write up results and reports and help you write up research papers of the data that can be put into the public [or the professional literature.] Cut off. On slide 13.
We, survey centers are here to help and we look forward to assisting you in both collecting and analyzing survey data. Survey research centers are service-related organizations that are in the business of helping you conduct your research, and we know how to do our surveys very, very well. And, we can assist you in attempting to gather the information you need to answer the research question you have.
If you have a research question for which you would like to conduct a survey and there is no existing data that can be used to answer your survey question, and you have potential sources of funding available to you, we would very much look forward to working with you, and we can also help you identify some of those potential sources of funding as well, if you would like to discuss that with us, as survey centers do have experience in applying for the work.
If you are just not sure, not sure about the answers to these questions and would like to talk to an expert in trying to help you understand these questions, survey centers are generally happy to spend time with you and develop your ideas and try to find the best procedure for you to move forward to answer your research questions. So, thank you very much and we look forward to hearing from you soon.
[No audio this slide]