Designing Effective Survey Instruments

Today law firm marketers, executive directors and managing partners are experiencing the power of marketing research. According to a 2001 study by Concord, MA. based Expertise Marketing, professional service firms that conduct formal market research report they are two to nearly five times more effective at attracting and retaining clients. The goal of any research methodology is to produce information to guide decisions for the purpose of creating higher market share and earnings. Survey research is just one research tool that savvy decision-makers are using to answer a variety of strategic questions such as:

• What legal issues will be outsourced in the coming years? Which issues will be handled internally?
• What issues do clients want to handle on-line?
• How do clients and prospects feel about outsourcing their legal issues? How do they choose a law firm?
• Which firms do my clients and prospects think of first when outsourcing legal work?
• How do my clients perceive my firm? What do they perceive to be my firm’s strengths and weaknesses? What must I do to increase my client’s loyalty?
• To what firm’s do my clients outsource work? What prevents my client from awarding that work to my firm? What must I do to get more or higher fees?
• Do my clients and prospect recall my advertising? What is the message they recall? Are my communication strategies effective?
• How do law students, associates and partners feel about my firm? What do I need to do to retain and attract the top talent?
• Is the market willing to pay for our new service? How should I structure my price? How much should I charge for this service?
• How, if at all do my clients use my web site? What do they want to do on my site? How much can I charge my clients to access information on my web site?
• What kind of legal work is most likely to produce the highest earnings in my proposed market expansion? What is the price elasticity for my areas of expertise?
• Should I change the venue for a trial? How might I support a change in venue to the court?
• What survey data do I need to support that my client’s trademark is being infringed upon?
• What motivated the jury to convict? Acquit?
• How will jurors react to specific evidence? What demographics characterize potential jurors that are most likely to be sympathetic to my case?
• How do the clients in our recently merged firm feel about the new firm? What are their fears? Hopes?

If you find yourself wondering about these and other questions, survey research may well be the tool that can help power your decisions. But, where and how do you begin? This article will explain some of the basic guidelines to questionnaire design.

What is Survey Research?

Survey Research is, “the systematic gathering of information from respondents for the purpose of understanding current attitudes and behavior, or to predict future behavior of the population of interest.”

Survey research usually involves administering a pre-scripted questionnaire to a large sample. It is considered to be quantitative and usually measures attitudes or behaviors such as when, where, how, how often, etc. Often preceded by qualitative research, survey research is not the best methodology to answer questions about why respondents feel or behave a certain way. (Email Marketing & Research Partners at MandRP@sbcglobal.net for a detailed explanation of when you should consider qualitative vs. quantitative research).

Most survey research projects undergo six basic steps:

• Planning and methodology design
• Survey instrument design
• Data collection (sometimes referred to as fielding the survey)
• Data tabulation and statistical analysis
• Analysis
• Report or presentation of findings
• This article deals only with the second phase, survey instrument design.

Getting Started

Before the survey instrument can be designed, the following questions must have been answered in the planning phase:

• What decisions will be made?
• What specific information is needed to make these decisions?
• Who should be interviewed to obtain this information?
• In some cases, hurdle measurements will also be defined. Hurtle rates are those scores or percent of respondents answering that must be achieve in order to make a specific decision (e.g., What score will I need to achieve in order to expend into the specified territory? A top-two-box score of 60% or 70% that indicate they will consider my firm for trademark work?)

Most surveys contain at least four sections:

• Introduction and screening
• Main questionnaire
• Classification
• Close and verification statement
• A brief overview of each step follows.

Introduction and Screening

The introduction is simply a brief sentence or two that contains the following information:

• Name of interviewer and name of company interviewing
• Purpose
• Approximate time or interview length ( Interviews in excess of 15 minutes have much higher refusal rates and therefore will increase the cost)
• Statement that this is not a sales call and no lead generation will take place (Important: Research is NOT lead generation. If you consider fielding your own studies please familiarize yourself with privacy and research guidelines. A copy of these can be found at: http://www.casro.org/codeofstandards.cfm).Example introduction when client is not identified and purpose is general:Hello, my name is _______ with Marketing & Research Partners, an independent research firm located in Dallas, TX. We are conducting a brief survey on legal services and would like to include your opinions. The survey takes approximately 10 minutes. We are not selling anything or obtaining sales leads. This study is for research purposes only. Is this a convenient time?

  Example introduction with purpose and client identified:

  Hello, I’m ________________ with Marketing & Research Partners, an independent marketing research company located in Dallas TX. I am calling on behalf of XYZ. XYZ has commissioned research to learn how they are performing on your behalf. They want to understand how their clients perceive their strengths and weaknesses, so they can improve their performance and working relationship with you.

  The interview will take approximately 20 minutes. Please be assured that all your answers are confidential. We will write a report of the findings for XYZ based on everyone’s responses. No single person or company will be identified in the report.

  Is this a convenient time to conduct the interview? [If not,] when would be a good time to call you back?

The first questions are designed to screen respondents for eligibility. Before you begin, refer to the question, ” Who should be interviewed?” Using this information you will draft screening questions like the example below:

Example:

Are you primarily or jointly responsible for deciding which attorneys or law firms will handle your AB Co. environmental litigation?

Yes….….1 [continue]

No………2 [ask to speak with the attorney who is primarily responsible and re-screen]

Note: Interviewer instructions are in brackets and bolded.

Main Questionnaire

This section will include those questions designed to answer the research issues outlined the in planning stage. Again, you will refer to the two questions you answered earlier, ” What decisions will be made?” and “What specific information is needed to make these decisions?” You will use both close-ended and open-ended questions.

Closed-ended questions are those questions that provide a finite answer set and commonly include rating scales or forced choice questions. (To learn more about scales for answer sets, refer to Marketing Scales Handbook by Bruner and Hensel)

Example:

Next, I would like to read you a list of statements. For each statement, please tell me how well you feel that statement describes AB firm. We will use a scale of 0 – 10 where “0” means the statement does not describe AB at all and “10” means the statement describes AB completely. You may use any number between 0 and 10 depending on how you feel. The first statement is…

Open-ended questions provide an opportunity for the respondent to answer anyway he or she wishes, but requires the interviewer to probe the responses. Open-ended questions are more productive when an interviewer is administering the survey; respondents do not provide adequate details when asked to “fill in the blank.”

Example:

What, if anything, do you specifically like about AB firm? What else? [Record verbatim response. Probe until all responses are exhausted]

Thinking about the last time you “fired” a law firm, what were the circumstances that lead to this decision? [Probe: What other circumstance lead to your decision not to continue working with a firm] [Tell me more about that] [Probe until all responses are exhausted]

Basic Rules to Remember

Regardless of the types of questions, the writer should follow several basic rules when drafting the questionnaire:

Keep the questionnaire short.

• Start with easy, non-threatening questions; threatening or difficult questions should be placed towards the end of the survey. NEVER start a self-administered questionnaire with an open-ended question that requires a detail response. Self-administered surveys (electronic and paper) will never provide in-depth answers to open-ended questions. You must consider this when selecting the data collection method in your planning stage.
• Use language that is respondent-friendly. Conduct qualitative interviews prior to quantitative surveys to make sure your audience understands the language and that you are including issues on which they can comment.
• Questions and terms should be free from ambiguity. Check to make sure the answer lends itself to an actionable solution.

Example:

Inexperienced questionnaire writers might use the word “Quality”. An experienced writer would ask, “What do I need to know about the quality? How does my target audience define quality? Will quality mean the same thing to everyone?

I NEVER use the word “quality” in any survey. Instead, I begin with qualitative research to understand how the target audience defines quality. In legal services, I find that the definition can vary by respondent group.

• Make sure questions are not leading the respondent.

Example:

Wrong : Aren’t the lawyers at XYZ excellent?

Right : Overall how would you rate the attorneys at XYZ? Would you say the attorneys are..
[Read list. Record one best answer]?

Excellent.…….5
Very good……4
Good…………3
Fair, or.………2
Poor………….1

• Arrange questions so that order bias is minimized. Typically questions flow from general attitudes > general behaviors > specific brand attitudes > specific behaviors.
• Unaided questions should always come before aided questions. Advertising awareness should always come before brand awareness, etc.

Example:

“When you think about law firms, which firms come to mind?” should be placed before “Have you ever heard of XYZ firm”

• Group questions by topic before skipping to the next topic.
• Arrange questions and answer sets to prevent automatic response. (Don’t always word responses to obtain a positive rating)
• Closed-ended answer sets requiring a single response should be exhaustive and mutually exclusive.

Example:

Wrong: Are you…?
18 and under
18 – 29
29 – 39
39 – 49
49 and older

Right: Are you…?
18 and under
19 – 29
30 – 39
40 – 49
50 and older

• Avoid overwhelming recall.

Example:

“How many times in the past ten years have you fired a law firm?”

• Interviewer instructions and skip patterns should be clear. Skip patterns are instructions that direct the interviewer/respondent to skip questions based on a previous response.

Classification

This section will include those questions that help the analyst classify respondents into categories that will provide additional insight. Usually, you will include a transition statement. This notifies the respondent that the interview is coming to a close and that the questions, although of a personal nature, are to be used only as an analytical tool. This will also decrease respondent refusals. Something simple will do, such as, (“Finally, I have a few additional questions so that I can sort my interviews into groups.”)

Typically, this section will include the following:

• Respondent title
• Name of business
• Standard industrial code, or other industry classification
• Location
• Number of employees
• Annual revenues
• Number of offices
• Other classification data you feel might be important when analyzing data

Close and Verification

You will close the questionnaire by thanking the respondent and asking to verify the contact information. Most often, a professional interviewing firm will verify a portion of each interviewer’s work. I choose not to validate interviews when I am obtaining client satisfaction measures from C-level executives and I have identified my client. If you decide not to validate interviews, you must be very comfortable with your interviewer’s skill and integrity.

Example close:

This concludes our survey. Thank you for your participation, your opinions are important. (If appropriate), My supervisor may wish to verify my work. May I have your name, phone number and address? If my supervisor were to call you, what time would be best? Thank you for your time.

Checking it Twice

When you have completed your fist draft, review each question to make sure each meets the following criteria:

Need for the Data: Inexperienced questionnaire designers frequently make the mistake of asking a large number of questions without considering how each will be used. The researcher must ensure that each question contributes insight to the analysis.

Ability of the Questionnaire to Produce the Data: The question must generate the needed information in a clear, unbiased manner.

Ability and Willingness of the Respondent to Answer: Inability to answer a question usually arises from four major sources: 1) never having been exposed to the answer, 2) forgetful respondent, 3) request for overwhelming recall and 4) inability to articulate an answer. Assuming the respondent can answer the question, the researcher must still assess the likelihood that he/she will answer it. Questions may be refused for one of three common reasons: 1) none of the interviewer’s business, 2) seen as embarrassing or 3) reflects on his/her prestige. Lastly, if you have included critical open-ended questions on a self-administered survey, you may want to reconsider you data collection method.

Wrap-up

Law firm leaders have many useful research tools at their disposal and, like their corporate clients, are using research to crave a competitive edge. Survey research is just one tool that quantifies the insight that powers strategic decisions. Properly executed surveys take time and expertise, and should be designed and conducted by experienced researchers who know the pit falls. A poorly executed survey may miss-lead decision-makers. If you only have a small budget, I implore you to seek advice. A properly designed and executed research project will save your firm countless dollars and put increased profits at your discretion.

In today’s market data demonstrates … the firm who has the most insightful information and can execute strategically wins.