We’ve had an exciting year – our first! – here at Research Access. So as we pause to celebrate the holidays and put the finishing touches on 2010, we’d like to take a moment to extend our wishes for a happy and healthy holiday season to all of you!

It’s been an exciting year for our industry,  and we know there’s much more to come in 2011. Stick with us for more in-depth and practical coverage of market research trends, tools and techniques. Until then, have a wonderful holiday season!

Allow me to introduce myself. My name is Joshua Hoffman, and I work for you.

I’m very excited to be joining Research Access as the new Editor-in-Chief, but when I say I work for you, I mean it. My role at Research Access is to bring you the latest, most interesting, innovative, and exciting information that exists in the market research industry. My hope in the end is to challenge and inspire you, to spark the discussion of new ideas, and to build a richer market research community, which I know would benefit us all.

Of course, what kind of market researcher would I be if I didn’t include a MicroPoll in my first post? Please share your thoughts below, but also, please consider leaving your comments on this post, describing what you’d like to see from Research Access in the future. Your input and feedback will be paramount in shaping the topics, authors, and resources you’ll see from Research Access going forward.

Cheers,
-Josh

This information is useful for people who want to understand a couple times when it is not appropriate to use conjoint analysis.

Conjoint analysis is a gold standard technique for measuring feature preference, particularly in relationship to price.  I’m particularly fond of Adaptive Choice-Based Conjoint (but that’s a topic for another post).  There’s a lot of buzz around conjoint as a tool to help product managers choose features that will help their products better compete in the marketplace, so I often get calls from companies thinking it would be a good idea to do a conjoint project.

In this post, I’ll show two common times when it is NOT appropriate to use conjoint analysis.

Definitions:

An “attribute” is something like brand, number of licenses, amount of storage, color, package size, etc.  A “level” is the degree of an attribute.  For example, brand A, B or C; 5, 10, or 20 licenses; 1, 2 or 3 TB of storage; blue, red, or black color; 12 ounce, 18 ounce, or 24 ounce package.

When NOT to Use Conjoint:

1. Your product features are already locked in – you just want to test prices. If your product is fully baked, you don’t want to use conjoint.  Conjoint is all about looking at the inter-relationship between various levels of product attributes and price.

If your product is locked in as a 10 license product with 1 TB of storage and other features set, conjoint is not for you.

So, how can you test price on your fully baked product?

Note: each of these methods deserves its own post, but here’s a taste.

Monadic designs:

• Break your respondent sample into groups that each see a single price associated with the product and ask their likelihood to purchase.  Plot the probabilities against the prices.

The  van Westendorp Price Sensitivity Meter:

• Ask “too inexpensive”, “inexpensive”, “expensive”, and “too expensive” questions.  Plot the data to obtain lower and upper bands and optimal price point.

The Newton-Miller-Smith variant of van Westendorp:

• Add purchase probability follow-up questions based on the inexpensive and expensive answers.  Build consideration curves.

2. Your attributes don’t vary (don’t have levels) – you’re just testing preference/importance of a number of items. You are not looking at the inter-relationship of various levels of brand, size, quality, durability, package, price, etc.  Instead, you want to understand the importance of, or preference for, a number of features/attributes that each have a single (constant, not varying) level.  Perhaps you want to test the general importance of brand vs size vs quality, etc.  Or, you may want to understand the importance of the specific, fixed features that make up your product (e.g., is having 10 licenses more important than having 1 TB of storage or the other features that make up your product?).

So, how can you test preference/importance of these features?

Note: A full description of MaxDiff can be found on the Outsource Research website.

Maximum Difference Scaling (MaxDiff)

• Force respondents to make trade-offs between (usually) 4 of your items at a time.  They indicate which item is most and least preferred (important, etc.).  The output yields all the items on a 100-point scale, where you can truly say that a given item is “twice” as preferred as another item with half its value.

Note: MaxDiff can be used to help reduce the number of attributes that you carry forward into a conjoint.  For example, if your product has a lot of potential features to test, it would be wise to reduce the number that you bring into conjoint, so that the respondent is not overwhelmed.  MaxDiff can show you the most important attributes, which can then be further explored in the conjoint.

Conclusion:

Conjoint analysis is a powerful technique that can help you configure your feature-price mix to create a product that will be most preferred by your market.  However, if your feature set is already locked and you just want to test prices, or if your attributes don’t have any variation (levels) to them, then conjoint is not for you and you’ll need other techniques to solve your research problem.

This information is useful for people who use panel sample for online surveys, and who want to make sure their survey data is truly clean.

Online Survey Panels Tell Us Their Panelists Are Clean

It’s hard to open a marketing magazine without seeing an ad from an online survey panel company proclaiming how clean and high quality their panel is.  A few years ago, this claim was a big deal – it was the Wild West of online survey panels, and buyers of sample had to be very careful as to who they worked with.  Today, however, most major online survey sample companies have adopted measures to get rid of professional respondents, prevent over-surveying, and make sure that respondents are who they say they are.  So, whether the sample is “true” or “pure”, or there’s “attention to detail”, most reputable panel companies are doing a decent job of giving those of us who field surveys a good product.

But Survey Data is Still Dirty

However, and here’s a big however, the data from most online surveys using panel sample still comes in with some dirty responses.  My research shows that between 1 and 5% of survey data from panel sample is garbage.  Garbage – throw it out; don’t bring it into your final dataset to analyze.  Sure, one can blame some of these dirty responses on frustrated respondents dealing with poor survey writing (bad questions, too long, etc.), but the fact remains that you had better clean that survey data before it goes in for analysis.

So, How Do I Clean the Data?

Here’s a plan you can use to clean your data.

When I say “flag” below, I mean that you create a new variable in your dataset next to the variable you are examining, and you place a “1” in a cell if the respondent’s case is flagged.

1. Flag speeders. Look at time to completion and flag those respondents who took the survey in an unrealistically short time.  Check the median time to completion and establish rules that you feel comfortable with – I often flag those taking <1/3 of median time with a “1″ (“speeder”), and those taking < 1/4 of the median time with a “2″ (“super speeder”).  You might consider removing outliers (at the slow end) before calculating your median.
2. Flag straightliners. If you having any grid/matrix questions, flag those respondents who gave the same response to every item (unless it makes sense that they could do so).
3. Flag gibberish or garbage responses. If you have any open-ended responses, look for text such as “asdf” or “…..”; flag these responses, and any other “colorful, yet meaningless” responses you find.
4. Flag incongruent combinations. If a respondent says their company size is 1000 and the number of PCs in the company is 5, something’s wrong here.  Flag it.
5. Trap questions. Did you include any questions such as “Please choose the third response below”, or “Please type the word “attention” below”?  If you did, check them, and flag those respondents who didn’t follow the directions.
6. Sum up your flags. Compute a new variable that sums all the flags.
7. Sort your dataset by summed variable. Bring cases to the top that have suspicious answers on a number of your checks.
8. Inspect and delete cases with flags. Delete those cases that are too “dirty” to be included.  Review with key stakeholders to agree on deletions.
9. Notify your vendor of any bogus respondents. All the vendors I work with do not charge for any respondents I have flagged for deletion.  Show them the IDs of the respondents you threw out, and they’ll take action on their side to warn and/or remove these panelists from their database.

Following the steps above will insure that the data you analyze is as clean as possible.  Yes, it takes a bit of time, but the effort is clearly worth it when compared to making decisions based on the analysis of data that includes bogus responses.

One last note: if you really need your final sample size to hit a specific number, and you can’t go below that number, you can over-sample, in anticipation of throwing out some respondents.

Feel free to contact me for more details about some of the specific techniques I have found useful to clean data, or follow me on Twitter @NicoPeruzziPhD to hear about other marketing research topics.