Survey / Questionnaire Design
This topic has two objectives:
1) To provide an overview of the process of designing a questionnaire
2) To illustrate good and bad practice in questionnaire design.
It will cover the following topics:
Identify the information needed
Decide upon the type of survey method
Determine the content of individual questions
Design the questions to overcome the respondents inability/ unwillingness to answer
Decide on the structure of the questionnaire
Determine the question wording
Arrange the questions in a proper order
Consider the form and layout of the questionnaire
Reproduce the questionnaire
Eliminate bugs by pre-testing.
IDENTIFY THE INFORMATION NEEDED
For example, is the information needed to:-
Classify e.g. Identify roles of leaders
Quantify e.g. Determine market shares
Describe Behaviour e.g. Understand the impact of high levels of emotional intelligence on staff motivation.
Discover attitudes or perceptions i.e. What are people’s views on the ethical stand taken by a particular organisation. Would it impact on the decision to seek employment within that organisation?
The type of information required is usually determined by the objectives of the survey/dissertation.
The type of information will in turn affect the design of the questionnaire. For example, if we are looking for information to discover attitudes we need to use open-ended questions because closed questions may prevent us from gathering all the information we seek.
Most questions are designed to determine:
· whether the informants are aware of an issue at all, or whether they have thought about it.
· the respondents general feelings towards an issue. This is invariably of the free response variety.
· answers on specific parts of the issue. This is usually achieved by pre-coded and structured questions.
· the reasons for the respondent’s views.
· how strongly a respondent’s views are held.
DECIDE UPON THE METHOD
No particular survey method is the best in all cases. Depending upon the problem, none, one, two, or many methods may be appropriate.
When deciding the type of survey to use the following need to be considered:
Who is the respondent? E.g. If we want a response from a company do we want the reply from a salesperson, the accountant, the managing director or someone else?
Nature of survey and motivation to respond. Depending upon the nature of the survey people may have different motives for responding. Would everyone answer a personal interview on their sexual relations honestly? People are more likely to respond if they are interested in the topic being discussed.
Simplicity of questions. Not everyone has a degree or even GCSEs so if you want everyone to respond make the questions easy.
Length of questionnaire. Respondents and interviewers get bored especially if the questionnaire is impersonal and there is no other person to keep them going so consider the format and make the length appropriate.
Response bias. This cannot be avoided but should be considered, as it can vary. For example, the questions in a mail questionnaire are likely to be biased in the same way for all, but in a personal interview the interviewer can bias the questions in a different way every time he/she asks the question, through their body language, comments, intonation etc.
Sample selection. Some bias will always be introduced where a sample, rather than a census, is chosen. Care should be taken to ensure you reach the sample intended. With some techniques this is more difficult. It is easier, for example to determine you are interviewing the correct person in a personal interview than it is with a postal questionnaire where someone other than the named respondent may have completed the form.
Accuracy of sample frame. How easy is it for a student to get a correct sample frame? For example is the mailing list up-to-date? Is it best to use the BT phone directory or are the respondents on cable? Are all businesses in a particular field in the trade directory? etc.
Reaching the sample. How easy are they to contact? For example, how easy is it to contact a business person by e-mail compared to reaching the average pensioner?
Cost. This is a key consideration for students and may well limit the method used.
Speed. Another key consideration, which may eliminate possible approaches is, if time is short.
Importance of interviewer credibility. Consider, for example, which is better to send a smart e-mail to a CEO in a large multi-national, or to turn up in jeans and sweat-shirt? These may be the only clothes available to a poor student, but are they appropriate?
Multiple respondents e.g. What is best if you have a fax questionnaire but want a number of responses from an organisation? Should you tie up their fax machine for ages? Expect them to copy the fax a number of times? Or do you consider a postal questionnaire to each individual instead?
Where is the respondent e.g. If you are trying to contact someone like a salesperson who moves around a lot how can we contact them. A land line to an office phone may not reach them, but we may not have mobile phone numbers. Letters may be slow while they may not even have a fax or e-mail. You therefore need to consider which approach is suitable for your study.
How will the survey be administered? e.g. If this is being analysed by an optical copy reader then there is little benefit from using a telephone study as it will then need to be transferred to hard copy.
Do the respondents need to be treated anonymously? Again this would affect how the survey is coded and whether a personal or impersonal technique were more appropriate.
The characteristics of the main methods of contact with respondents, which should be considered when deciding which approach is most suitable, are.....
MAIL
Relatively slow; Have you time to get replies?
Cheap.
Impersonal.
Need reminders and cover letters.
Poor response rate. Will you have enough replies for analysis?
Needs a good sample frame.
Sample bias; do respondents differ from non-respondents?
Good for long questionnaires.
Reaching the respondent? ie. does a letter addressed to a financial director reach that person, or is it redirected by a secretary to another person in the organisation? One who cannot answer all the questions.
TELEPHONE
Quick
Cheap
Relatively Impersonal compared to personal interview. (Consider credibility of interviewer)
Limitations on length of questionnaire. Respondents will tire.
Problems with complex questions.
PERSONAL INTERVIEWS
Expensive
Interviewer bias is easily introduced.
Reaching the respondent - may be difficult to locate.
Requires skilled interviewers
Good for complex questions
Few limits to length apart from the comfort of the participants. Compare for example the responses likely from a person in the comfort of their own sitting room to an interview carried out in the street on a cold January day.
E-MAIL
Quicker than mail but slower than the phone for individual interviews. However multiple mailings can be sent making it much faster where larger sized samples are required.
Cheap
It is relatively easy to send reminders to boost response rates.
Impersonal
Sample limited to those with an e-mail address, which may cause biased results.
Relatively easy to contact respondents anywhere in the world
Cannot convey senses like touch and taste, but can send sound and moving pictures (given enough memory) otherwise only available through video-conferencing or personal interviews.
FAX
· Similar pros and cons to e-mail but cannot transmit sound, and cannot handle multiple mailings.
· Cost is likely to be greater than e-mail but less than other methods, including the telephone.
DETERMINE THE CONTENT OF INDIVIDUAL QUESTIONS
Is the question necessary? Ask yourself ’Why am I asking this question?’
· Write specific questions only after you have thoroughly thought through your research questions. Write down the research questions.
· When you are working on the questionnaire, constantly refer to your research questions.
· For each question you write, explain how the information obtained from responses will help in answering your research questions.
Don’t reinvent questions, especially on generic issues.
Repeating questions from other studies that have demonstrated acceptable levels of reliability and validity can allow you to
· reduce the time needed for testing
· compare results across a number of studies
· establish response reliabilities for the study at hand.
Are several questions needed instead of one?
· For example, if a respondent agrees that they buy a product because it is high quality and/or value for money, how do you know whether they buy it for quality or value or both? The answer is that you cannot tell unless you split the question and ask about each issue separately.
DESIGN THE QUESTIONS TO OVERCOME THE RESPONDENTS INABILITY/ UNWILLINGNESS TO ANSWER THE QUESTIONS CORRECTLY
When designing the questions ask yourself:
Is the respondent able to answer?
· Is the respondent informed about this issue?
· Can the respondent remember? We all forget things.
· Can the respondent articulate the answer? Maybe they do not know the technical terms.
· Could the respondent reinterpret the question to their own understanding, thereby giving inaccurate data to the researcher.
Is the respondent unwilling to answer?
· Does it require an effort to respond? Like having to search through papers when they have other things to do.
· Is the question context valid? Do they feel it infringes upon their liberty?
· Is there a legitimate purpose to the question? If a respondent cannot see why you want to know something they may feel justified in keeping the information to themselves.
· How sensitive is the information? Will it force them to give embarrassing information?
· Does the question impact upon an area where prestige is involved, such as age, income, status, alcohol consumption, etc. Many respondents will try to impress and inflate or deflate their responses to impress the interviewer.
HOW COULD YOU INCREASE A RESPONDENT’S WILLINGNESS TO ANSWER?
If a respondent is unwilling / unable to answer we may get one of the following reactions:-
· Item non-response. i.e. No reply to a particular question
· Refusal to complete the questionnaire
· Distortion of the truth. How honest are survey respondents? (This is worse than the other two outcomes as you may not realise the respondent is lying and use the data to produce inaccurate results.)
DECIDE ON THE TYPES OF QUESTION
There are only two main types:
Open Ended (where the respondent has a Free Response to answer in any way they please)
Closed (where the respondent is forced to choose an answer from a fixed list)
These may be further broken down into the following categories
+Dichotomous e.g. Yes/No
+ Multiple Choice – when using these types of question you must ensure that they are mutually exclusive and collectively exhaustive.
For example, when asking how much gross income a family has you should not use 0-£5k: £5-10k: £10-£20k: £20-£50k: as the categories. What if the family earn more than £50? (this is not exhaustive). What if the earn £20k? They could tick either of the last two categories 10-20 or 20-50. They are more likely to tick the higher options. This is an example of not being exclusive.
+ Rankings/Ratings – when asking a respondent to rank / rate something you need to decide whether to give them a middle value or force them to make a preference.
+ Check Lists
Some common examples of these question styles include:
Select 1 option
+the respondent is given a list of (r) options and is required to choose one option only.
Select n options
+ the respondent gets a set of (r) options to select from but this time chooses up to ‘n’ options (n<r).
Select and rank
+ similar to select n options, but in addition to selecting a number of options from a list the respondent is then asked to rank two or more of the options selected.
Select and select
+ the respondent is asked to select options from category one and different options from category two.
Rank
+ the respondent gets r options and is asked to rank the top n (n<r)
Integer Rating
+ the respondent is asked to rate on a linear scale of 1 to ‘n’ a description e.g. 1 strongly disagree 5 strongly agree. Only integer responses accepted.
Continuous Rating
+ similar to integer rating except the response can be any number within the range e.g. 5.2 on a scale of 0 to 10
Constant sum
+the respondent is provided with a set of attributes and is asked to distribute a total of p points across these attributes.
Either / or
+ questions entailing a simple either/or response eg. Yes / No
Integer
+ the respondent is asked for a fact that can be expressed in an integer number form e.g. age. a valid range would then be provided for error checking.
Real
+similar to integer, except that the answer expected is in the form of a real (not necessarily integer) number e.g. Income. Again a valid range may be provided for checking
DETERMINE THE QUESTION WORDING
When developing a questionnaire always consider the following standard questions prompts to help you begin a question:
Who
What
When
Where
Why
Which
How
When wording a question you should pay particular attention to the following:
· Define the issue - Consider the question “Which toothpaste do you use?” It is poorly defined. For example, does this mean ‘you’ individually or ‘you’ the household? Does it refer to brand, or flavour, or something else? Do you answer in respect of the last toothpaste you used, your preferred brand, or any you have ever used?
· Use ordinary words – or at least those that are ordinary to the respondents
· Do all words have the same meaning - Ambiguous words are those that can be interpreted differently by people. For example, what does regularly mean? It could be hourly, monthly, annually etc. Give a respondent a specific choice e.g. Have you cleaned your teeth today?
· Avoid leading or biasing questions- Gives clues to the respondent as to what the answer should be. E.g. An obviously leading question is ‘Do you agree that sensible people would not recommend smoking pot?’, but a less obvious example may be ‘Research suggests that smoking pot eases the pain for MS sufferers. Do you believe pot should be legalised for medical purposes?
· Avoid Implicit Alternatives - An alternative that is not explicitly expressed e. g. Compare, ‘Do you like to eat vegetarian food?’ with, ‘Do you like to eat vegetarian food, or would you prefer meat?’ The first question is likely to elicit a higher positive response.
· Avoid Implied Assumptions - Questions whose answer is dependent on an implicit assumption e.g. compare, ‘Are you in favour of older teenage children making their own way to nightclubs?’ with, ‘Are you in favour of older teenage children making their own way to nightclubs, if it means they borrow their parents cars?’ Again, the first question is likely to elicit a higher positive response in favour.
· Avoid Generalisations and Estimates - Avoid the general make specific e.g. To answer the question ‘What is your families average annual spend on clothes?’ will involve you having to determine both the number of people in the family and each members spend (probably per week or month as it is unlikely they will know the annual spend) then you will have to calculate the answer. It would be better to ask these simpler questions instead.
· Positive and Negative statements - Use either dual statements or be neutral in questioning (attitudes) Evidence shows that directional statements can influence responses.
ARRANGE THE QUESTIONS IN A PROPER ORDER/SEQUENCE
Question Order
¨ Start with easy, salient, non-threatening, but necessary questions that provide motivation. Never start a mail questionnaire with an open-ended question that requires a great deal of writing.
¨ Where ever possible put sensitive / threatening or difficult questions at the end. They may elicit a negative response or a refusal to continue. This becomes less important towards the end when the majority of the data has been collected.
¨ Since some demographic questions are threatening, put these questions at the end of the interview. If at all possible, avoid asking demographic questions first.
¨ For personal interviews use-funnelling procedures to minimise order effects and provides a logical order. Start with the general questions and move to specific questions. For low-salience topics, however, it may be necessary to ask questions about specific dimensions of the topic before asking for a summary view.
¨ Consider the effect of one question on subsequent questions. Could it affect the response
¨ Avoid sequences of questions that are likely to elicit the same response.
¨ If questions deal with more than one topic, complete all questions on a single topic before moving to a new topic.
¨ When switching topics, use a transitional phrase to make it easier for respondents to switch their trains of thought.
¨ To ensure that all contingencies are covered, make a flowchart for filter questions. Filter questions should not require extensive page flipping by the interviewer or respondent or require remembering answers to earlier questions.
¨ If multiple filter questions will be asked, try to ask all of them before asking the more detailed questions. Otherwise, respondents may learn how to avoid answering detailed questions.
¨ Design branching questions with care (these are questions which direct respondents to different places in the questionnaire based on their responses to the question at hand.)
¨ A simple way to account for all contingencies is to prepare a flow chart of the logical possibilities and then develop the branching questions and instructions based upon it.
NB. The following guidelines should be followed when creating branching questions....
The question being branched (the one to which the respondent is being directed) should be placed as close as possible to the question causing the branching.
The branching questions should be ordered so that respondents cannot anticipate what additional information will be required. Otherwise the respondents may discover that they can avoid detailed questions by giving certain answers to branching questions.
CONSIDER THE FORM AND LAYOUT OF THE QUESTIONNAIRE
The key aim of the form and layout is to help people reply and make them want to complete the questionnaire. To do this:
· Make questionnaires easy to read e.g. use tick boxes, spacing
· Consider the number of questions. Too few and you may not get the data you need. Too many and you will put people off from replying.
· Avoid splitting questions over two pages or columns. It is confusing and will result in incomplete answers.
· Do not print on both sides of a sheet of paper. Unless it is part of a book or very clearly marked there will always be some respondents who do not complete the back page.
· Place directions close to questions, preferably in a different typestyle to identify them as instructions.
· Avoid overcrowding questions and answer sections as this leads to errors in data collection.
· Remember that some styles of typeface, colour, size etc are easier to read than others, and can affect responses. .
· Pre-code answers (i.e. assign a code to every conceivable response before data collection.)
NB. Coding
§ The assignment of a code to each question response, usually a number to give it a unique identifier.
§ The codes improve the level of analysis and the speed of input into computers.
§ Help to make complex questionnaires accessible.
REPRODUCTION OF THE QUESTIONNAIRE
· Quality of the paper - Does it tear easily? Can you write on it with a pen? Does the copy show through making it difficult to read? Do you need to scan it in an OCR? If so, will it scan?
· Appearance - shabby/professional. Make it appropriate for the circumstances. What is suitable for a student funding their own dissertation may be completely inappropriate for Rolls Royce!
· Presentation of long questionnaires - loose sheets / stapled/clipped/booklet etc which is best? Most cost effective?
· Making questionnaires easy to read - large/clear type
ELIMINATE BUGS BY PRE-TESTING.
When Pre-testing Ask:-
· What items should be pre-tested? Test the whole questionnaire and any prompts etc to be used by interviewers.
· How should they be pre-tested? Test the survey in conditions as close to the real survey as possible.
· Who should conduct the pre-test? If possible use the same people / instruments as will be used in the real survey.
· Which respondents should be involved in the pre-test? Use respondents who closely match the real sample. This could be another sample from the population or it may have to be a matched sample from another population. Try to avoid asking the real sample as the pre-test my bias their responses.
· How many respondents should be used? Normally between 10 and 20 but this will vary depending upon the size and composition of the population. Be prepared to repeat the pilot test is major problems are found with a pre-test. While there is no such thing as the perfect questionnaire you should continue until it is as good as you can make it.
NB. In some cases a pre-test is not possible because of lack of time, a very small population size or the destructive nature of the testing. If this is the case mock simulations in non ideal conditions are better than no test.
QUESTIONNAIRE DESIGN FOR INTERNATIONAL STUDIES
Factors which you should consider because they add to the complexity of research in international markets include:
· Understanding the culture
· Lack of secondary data
· Cost of collecting primary data.
INTERNATIONAL METHODOLOGICAL CONSIDERATIONS
· These arise chiefly because of the complex cultural differences in the meaning of products and activities, and the paucity of standardised data-collection instruments to measure marketing related variables in different countries.
+Functional equivalence e.g. Jewellery in India displays social status, in the US it shows wealth and security.
+ Conceptual equivalence e.g. Family in the US means parents & children, in India it also includes aunts, uncles and other relatives.
+ Ethnic influences e.g. cultural taboos may prevent discussion of issues such as personal hygiene products in some cultures.
+ Instrument equivalence Can you use the same measuring instrument in different countries e.g. response categories to closed questions may differ between countries.
+ Instrument translation how does the questionnaire translate? eg. Boeing translated “rendezvous lounge” into “prostitution chamber” in Portuguese.
SUMMARY
· The Design Process is not scientific, but by adhering to the principles suggested you will improve the quality of your results
· Quality is improved by experience, but you get little chance to do this in the course of a dissertation.
· Process is improved if the needs of the respondent are considered.
· Common sense is appropriate
· Must have a clear idea of the information needed, for the study. This should be based on your objectives.
Bibliography
¨ Malhotra N K & Birks Market Research - An Applied Approach , 2007, European Edition, Prentice Hall
¨ Chisnall PM, Marketing Research, 1997, 5th edition, The McGraw-Hill Marketing Series
¨ Proctor, T, Essentials of Marketing Research, 4th Edition, 2005, Prentice Hall
Please complete each of the following tasks:
QUIZ 1 – Multiple Choice Questions
Instructions:
Tick the box(es) that you think gives the correct answer to the questions. There may be more than one answer to a question. The maximum score is 16.
1. Which is the most appropriate technique for conducting a questionnaire when time is short, and money is tight?
· Personal Interviews
· Telephone Surveys 4
· Mail Surveys
2. Which of the following statements is incorrect
· Faulty questionnaire design is a major contributor to sampling errors. 4
· Poor design will result in biased responses.
· Incorrect conclusions drawn from a badly designed questionnaire will result in management making the wrong decisions.
3. Quantitative research
· Uses unstructured surveys
· Provides a statistical result 4
· Recommends a final course of action 4
4. Mystery shopping is an example of
· Experimentation
· Survey
· Observation 4
5. Observation is only suitable as a research method when
· The data is accessible 4
· The behaviour being observed is unpredictable
· The time span involved is long
6. _____________ surveys are expensive; require skilled interviewers but are good for long and complex studies
· Mail
· Telephone
· Personal Interviews 4
7. In _________ interviews the questions must be shorter and simpler than in __________ surveys.
· personal, telephone
· telephone, mail 4
· mail, telephone
8. The first question in a survey should be
· simple 4
· challenging
· interesting 4
9. Sensitive questions should be
· Asked at the beginning of the questionnaire to get them out of the way
· Mixed in with the other questions
· Left to the end of the questionnaire 4
10. Questions are coded to
· Identify respondents
· Make it easier to ask the questions
· Help in the analysis of the survey 4
11. Which of the following should you not do when designing a questionnaire
· Avoid splitting questions over two pages
· Use small type faces to reduce the length of questionnaires 4
· Place directions close to questions
12. When pretesting a survey it is important that the sample are similar to the full survey in terms of
· Ages 4
· Background 4
· Interest in the topic 4
QUIZ 2 Spot the Errors
Instructions
The following questionnaire shows examples of various common errors found in questionnaire design. Below the questions is a list of errors. Identify which questions are examples of which errors by ticking the appropriate boxes. Some questions may exhibit more than one type of error. The maximum score is 12.
a) Sensitive question
b) Uses generalisations/estimates
c) Ambiguous wording
d) Leading question
e) a double barrelled question
f) Implied assumption
g) Implicit alternative
h) Non-Exclusive question
i) Non-Exhaustive question
1. Studies show that four out of five males prefer to go to a football match than to eat out in a restaurant. Do you prefer to watch a football match rather than eat out at restaurant? d
2. Where did you buy your last car? Main dealer ð
Local garage ð
Private sale ð f & i
3. Do you eat dinner every day ð
only at weekends ð
only on weekdays ð
never ð c & i
4. How much do you earn? £5000-10000 ð
£10000-15000 ð
£15000-20000 ð
£20000-25000 ð
more than £25000 ð a,h & i
5. Do you like or admire Tony Blair? Yes ð No ð e
6. Do you like to walk when going to the local pub? Yes ð No ð g
7. In the last year how often have you been to the supermarket? ____ b
8. How often have you visited this Internet site before? Never ð Occasionally ð Sometimes ð Often ð Regularly ð
