So, you’ve come up with a cool idea, but how do you test the market for your new product or service?

First, you need to figure out what is essential and what might be helpful. Before we get into the detailed recommendations, do you need research? Some entrepreneurs discount the idea of market research for various reasons:

• You may think that your new idea is so special that research won’t tell them anything useful. With rare exceptions, this is simply not true.  Even if your new idea is so different from what’s currently on the market that people won’t understand it immediately, you can gather information about current pain points, needs, wants, and motivations.

Perhaps you believe that people will see the product and rush to buy it (the better mousetrap fallacy.)  Maybe, but usually not unless you’ve done your homework on how to reach prospects and their influences and decision making.  Are you sure they want a better mousetrap?  Perhaps they just acquired a cat and don’t need a mousetrap.

• Maybe you are convinced that you only need a fraction of a big market in order to succeed.  Get in front of an angel investor or a V.C. and you’ll learn that they aren’t impressed by that idea.  But even if you are bootstrapping without outside investment, you should understand that the riches are in the niches.  As has been well demonstrated in theory and practice, a niche is important so that product development, marketing, and sales costs are minimized; greater market share in a niche should lead to more profitability and better defense against competition.  Even if your product really could address a broad market, it is important to identify your starting niche.

• And of course, some people have the perception that market research is too expensive.  Don’t forget that making a mistake is even more expensive; it is much better to find out that your idea isn’t really going to succeed, or needs some changes, before launching.  Out of pocket costs can still be a concern, but hopefully you’ll get some ideas from this article that will help save money.

With that out of the way, the most important market research for startups is generally market sizing, concept validation, and pricing.  These aren’t necessarily separate projects, but the research plan should be written to give valuable information in these areas.  We tend to use both qualitative and quantitative techniques, and often secondary research before the primary custom work.

Secondary research is likely to give some general information about the broad market, such as the overall size and large subgroups (for example classes of hotels, or types of users for a software product).  Hopefully you’ll also learn something about terminology used by your prospects, which may well be different from your own jargon. Start by looking for free sources of information such as the Census Bureau (for both business and consumer), or the Pew Research Center.  The best free sources are also well documented, with published methodologies that give you confidence that the results will be applicable to your situation. Other secondary sources include investment analysts and industry groups. But the information isn’t likely to give you all the answers you need to identify and size your niche, much less to give you much to go on as you create the product that solves a problem, which is one reason to think hard before you spend money on a published report.  My ex-colleagues building high-technology startup products tell me that they rarely trust any of the secondary sources for the industry because much of the data about projections is simply gathered by asking a bunch of insiders who exaggerate so the overall numbers will be higher.  Maybe that is a thing of the past given the current state of the economy; you be the judge.

Sizing the niche for a new product is usually a matter of triangulation with inputs coming from a few secondary research sources, and also from primary research that you commission or conduct yourself.  How much effort (and cost) is needed for the sizing exercise is somewhat dependent on your need for outside investment.  An angel or a VC is likely to want to know that the market niche is large enough to justify their investment, as well as whether competition makes the niche less attractive.  Don’t forget that competition includes the status quo.  Your potential customers may just choose to live with an inferior approach if they don’t have sufficient incentive to change.

The level of reliability needed for accurate sizing requires a sample that is representative of the overall target population.  For example, if you are building an iPhone application that you think might appeal to a wide range of types of people you should sample all iPhone owners.  Research can tell you whether your seemingly general app should be targeted at, say, college students rather than business users and also the percentage of students.  To find out the percentage of college students you can turn back to secondary research, combining iPhone general demographics with census data for college enrollment.  But you might also want to know some other things about the target that could only come from surveying them; perhaps your product would be particularly useful to students whose grades have declined in the past semester.

Your representative sample could most likely come from a panel company.  Assuming you do an online survey, there are a range of online panel providers to choose from.  The most reputable suppliers follow industry standard protocols for recruiting and maintaining their panels, and the results aren’t likely to come from people who cheat on surveys.  In addition to vendors who recruit people specifically for surveys, sample can be obtained from companies like Peanut Labs who have relationships with social media sites or LinkedIn who can send invitations to members.  Obtaining sample like this should generate representative results, but sending invitations to lists you are on or posting in social media sites without control will not.  Panel companies charge by the complete, based on incidence.  The cost may seem high to a bootstrapping startup, but it should be money well-spent, and you’ll get results quickly.  Self-service survey tool companies, including QuestionPro and SurveyGizmo, generally have easy integration with panels.

For a representative sample for a consumer product you’ll want to gather at least 400 completes to give reliable results (±5% at 95% confidence). Larger samples allow reliable analysis of subgroups, and also support more advanced analysis such as segmentation.  For B2B research, sample costs tend to be higher and the target more limited so a smaller sample is often acceptable; 200 completes will give ±7% at 95% confidence.  Not only do these confidence intervals give the ability to project to the entire market, they also allow you to know that one result is statistically different from another.

It may be OK to use a non-representative sample if you don’t need overall market sizing but you still need to validate the concept.  For example, if you have a fairly good idea of the target and don’t need to convince an investor, you can refine your understanding by just tapping the right people.  In this case, posting your survey to one or more email lists, or just posting yourself in social media sites can be effective.

Another possibility for sample is a customer list.  This is a great way to invite people to take surveys, but startups don’t usually have existing customers.  If your company already has customers and you are producing a new product, maybe the customer list will work, but the results won’t tell you much about market size, and may give you false impressions if the products don’t appeal to the same people.  You might also think about using pop-under invitations on your website, but this is unlikely to be representative, to say nothing of the chicken and egg problem from the likely small traffic.

Before I describe some of the important content of the survey, let’s take a look at qualitative research for startups.  Qualitative research can give you a rich understanding of pain points, needs and motivations and other intangibles.  It is exploratory, so you don’t need to know all the questions before you start, but doing early research can help create a more effective survey. Just remember that qualitative means not projectable. One of the cardinal sins is to take results from a few people and assume that they reflect the overall market.  Useful qualitative techniques include in-depth interviews, focus groups, and idea generation and evaluation.

You’ve probably already heard the maxim that startup founders should talk to prospects.  If you make those in-depth interviews structured, you’ll be able to get useful information that is comparable from one discussion to the next.  A structured discussion guide makes this work.  Use mostly open-ended questions, and allow for probing to dig deeper than the initial response. You may need help to write the guide, and you may prefer to have someone else do some of the interviews, or to sit in on the conversations.

Focus groups have been used for many years, generally conducted at custom designed facilities allowing audio and video recording and two-way mirrors for real-time observation.  Focus groups are also held online, using real-time or bulletin board approaches.  Although online might seem to be a natural progression and a way to cut costs, most groups are still face-to-face.  Much like in-depth interviews, focus groups use structured discussion guides to ensure that topics are covered in the right order and with opportunity for open-ended discussion and probing.  The group setting creates opportunities for more active discussion where what one person says can stimulate new ideas from others.  Some worry that the setting creates a potential for groupthink or domination by a few participants, but skilled moderation avoids the issue and instead uses group dynamics for positive outcomes.   The moderator is the most important element in successful focus groups.  Most startups benefit from using an experienced independent moderator because entrepreneurs are generally too close to the topic and too passionate.   As with in-depth interviews, it is important to remember that this is research; it is more important to hear from participants rather than try to sell the concept.  While focus groups are ideally held at a dedicated facility, it is possible to conduct the groups in other settings that may be less expensive.

Ideation techniques are often used in focus groups, but similar techniques can be used in an online setting too.  Private discussion areas are set up, perhaps as part of a broader community environment, or perhaps just to discuss ideas about a product or idea.  Ideas may be posted by the organizer for comments and voting by participants, and in some cases discussion threads may be started by regular users. These tools are great for gathering freeform input, but don’t treat them as quantitative.

Armed with information from the qualitative research, and/or your own knowledge of the issues and what you are building to solve the problem, you can create or commission the survey with questions about the situation, pain points (problems), products currently being used and what’s missing to validate the concept. If the new product is easily described, you might be able to ask about features in a concrete fashion, but in any case you should be able to ask about importance of specific issues, as well as how well current products solve the issues.  Include questions about information sources (both online and offline), and key influences for new product choice. You should include demographics relevant to the topic, but also attitudinal questions. These days, psychographics (attitudes, interests, opinions) tend to be more significant because people aren’t defined simply by demographics. In addition (and particularly relevant to startups), it is important to understand where prospects fit on the adoption life-cycle for that product.  You’d naturally expect early adopters to be more interested in a radically new solution, but if you can’t distinguish between them and mainstream buyers you might underestimate the market size or make the mistake of assuming that messaging for everyone should look the same.

Finally, what about pricing?  Research can generate some guidance for pricing decisions, but it will rarely provide definitive answers, especially in startup situations.  Pricing decisions incorporate many factors including competition, cost, purchase authority, share of wallet, etc., with research providing some of the inputs.  Pricing research works better when the product is easily understood, and may not work at all when the new product is a big innovation.  In this case you could try asking about the value of fixing various problems, but results are likely to be inconclusive at best. I generally steer startups to Van Westendorp analysis because it is easy to implement, analyze and explain, and of course useful! My article (http://www.5circles.com/wordpress/blog/2009/05/van-westendorp-pricing-the-price-sensitivity-meter/mike-pritchard/) includes a comparison of other pricing approaches including why you shouldn’t just ask “what would you be prepared to pay?”  Using Van Westdendorp’s 4 questions, you can generate a range of acceptable prices which is a useful input to the pricing decision.  An additional Likert scale question(Very unlikely, Unlikely, Unsure, Likely, Very Likely) is often used to assess willingness to purchase – with the results modeled to generate purchase likelihood at each price point.  Further tweaking can be used to generate simulated demand curves, but at this point using conjoint analysis techniques may be more appropriate.  Conjoint is powerful, but tends to be more expensive because of the complexity of setup and analysis, as well as increased survey length (meaning higher sample costs).  Because so many factors are often moving targets in a startup situation, a simpler and cheaper method for pricing research is generally preferred.

As you can see, there are a number of types of research that benefit the startup.  Fortunately, you don’t have to worry about customer satisfaction (although you might want to assess satisfaction with competitors), brand awareness, advertising effectiveness or any of the other applications for market research.  The most important thing is to do some research, whether you do it yourself or work with professionals.

Idiosyncratically,

Mike Pritchard