Apr 15

Lean Experimentation: What to Do When Customers (Internal and External) Won’t Tell You What They Want!

Today Bruce Winner writes about an extraordinary method for trainers, developers, and training managers to use, in order to experiment their way to greater results and success. This method, pioneered in the Silicon Valley, is changing the way that organizations innovate.

Public and private sector organizations spend thousands of dollars and hours on surveys and other means of collecting information, but often find that people do a poor job of telling them what it is they want or need. This blog will introduce you to “Lean Startup”, a method that many Silicon Valley startups and other forward thinking organizations are using to find out what their customers “really” want. The method has the potential to change the way you develop new products and services and could turn your operation into an innovation machine. Lean Startup will decrease your reliance on survey responses or second-hand information and provide you a means to allow your clients to “show you what they want”.

The Lean Startup Method

The Lean Startup Method originated in the Silicon Valley and has launched startups worth billions. Larger companies have also adopted the method to innovate and speed up development of new products and services. It was first proposed in 2011 by Eric Ries, the author of “The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses.”

Lean Startup encourages an organization to experiment or:

  1. Build – Start small, with a “minimum viable product” or MVP
  2. Measure – Follow each step with a measured test (often by a very small groups of users),
  3. Learn – Learn lessons from each step (by way of rigorous measurement)

The first step is interesting and differs from a traditional approach for building a new product or service. Most training organizations would spend several months or more building the “perfect” training product or service and try to assure it was absolutely market-ready before launch. This approach dictates starting with a small idea and testing it first to see what you can learn about the viability of the concept.

The goal of the method is to learn more at each iteration or experiment. At the end of each short cycle your group will learn from the experiment and continue to move in the same direction, if the learning was positive. If the learning was negative or indicated a misstep, then you will know to change directions or “pivot” in the language of Lean. In Lean, your internal or external customers are “showing you what they want” by responding to your experiments.

Validated Learning

  • Lean Startup is also known as “validated learning” because the goal of each test or experiment is to learn one thing, validate or invalidate the lesson, by paying unbiased attention to the outcome.

The cycle is to build a minimum viable product, test and measure how it works, and then learn from the results. It allows organizations to capture the results of experiment after experiment and build on this learning to achieve greater results and success. A popular phrase among lean practitioners is “fail fast”. Remember, the goal is to learn. The faster you fail, the more you learn, and the sooner you develop a product or service that your audience wants from you.

Can Experimentation like this Really Make a Difference?

Lean Startup originated in the Silicon Valley, so much of the experimentation has been conducted by testing web sites, web-based offers, and examining web-based analytics. If you are like most learning organizations, you too struggle to maximize the use of the web analytics at your disposal.

A few examples from Lean Online Businesses

  • An example from an online wine sales site showed a 41% increase in sales, simply by testing and then choosing the top performing web design. All the tested versions had the same information; there were simply design differences between tested versions.
  • A recent publication on lean analytics cited a seller of tickets who changed the wording on a call to action button from “Get started free” to “Try it out free”. The test and final selection resulted in a 376% increase in sales!

How can Your Training Organization use this Method?

A/B Tests – The simplest test is the A/B or split test. In this test your group tests one offer or product variation against a control group, that receives the old or original offer or product. For example, Group A receives the standard email offer for a training product or service (the same one or similar to the one you routinely use), but Group B receives the offer with one modification.

Segmentation – In this method you test the same message, product, or service by sending test messages to different demographic segments. This is not a new method and has been used effectively for years by direct mailers. Through the test, the organization can determine if the offer is equally effective from one “segment to another (women vs. men, gen X vs. boomers, etc.).

Cohort Testing or Longitudinal Testing – In this method a number of changes can be introduced over time and the group’s responses (one time period versus the others) are compared.

What Could Training and Development Groups Learn through Testing?

  • Consider learning through testing various marketing and promotional offers for your training products or services. This sort of testing can work for internal promotion of courses in a large organization or sales to external clients.
    • Examples:
      • Experiment with various email offers designed to promote your training products or services. (you can test designs, language, or various offers for effectiveness)
      • Test web descriptions or designs that vary the focus on benefits to recipient (student) vs benefits to the department, supervisor, or the person(s) paying or approving the training
  • Instead of launching your next training product or service fully formed, consider testing the idea with a low cost 30-minute webinar or even a three-minute YouTube video. Take the product (video or webinar) you have developed and test it with small groups of users. Ask managers or execs who view the webinar or video if they would be willing to commit X number of their employees to a full training session on the topic. (Essentially you are asking them to “purchase” a program or “show you what they want”.) Remember, you can continue to improve or iterate the product if it is received well or go in another direction, “pivot”, if the product or service is rejected by the test audience.
    • You could conduct a similar test for end-users or training participants. Using the same video or webinar, ask the participants if they would be willing to sign up (at the time of viewing) for the course to be offered at a later time. If they are internal customers, ask if they would be willing to send a request to take the course to their immediate supervisor. You are searching for commitment or their willingness to “show you what they want”.

Again, what could testing or rigorous experimentation do for your organization?

Your training group or firm could discover a powerful and attractive product or service that is currently not being offered, and that you may NEVER offer, unless you find audience interest by doing some innovative small group testing.

What do you think? Is the Lean Startup Method a good fit for your organization?

  • It is a low-cost method
    • The method allows you to conduct small and inexpensive experiments for marketing, promotion, and especially for innovating or building new training products or services. Do you have a similar mechanism for doing this now?
  • Learning is the goal
    • Since learning is the goal, the method seems ideally suited to learning and development organizations. Go ahead, don’t be afraid to take some risks and learn from them. Your organization will appreciate the fact that new things are happening.
  • It is truly cutting edge
    • This is a progressive technique that is being used by some of the most forward thinking companies and organizations in the world. Don’t your colleagues or clients deserve the same?

Here are a Few Resources if you Want to Learn More:

  • Eric Ries’ book, “The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses”, published in September, 2011. The “bible” for practitioners of the method, this is the best place to start.
  • “Lean Analytics: Use Data to Build a Better Startup Faster”, Alistair Croll and Benjamin Yoskovitz 2013. For my money, after you read and absorb “The Lean Startup”, this is the book to read. The heart of the Lean Startup Method is disciplined experimentation and an unbiased approach to using the data you collect. This book provides a stepwise approach to building this expertise.
  • “Running Lean: Iterate from Plan A to a Plan that Works”, Ash Maurya. This is the second book in the Lean Book Series, started by Eric Reis. Running Lean will introduce you to the “lean canvas”, an innovative one-page visual business model or plan that that will give you a tool to brainstorm business models, see where to start, and assist you in tracking your learning. The book also provides more detail on doing face-to-face MVP interviewing. This interviewing is the first qualitative step in making sure you don’t have a “solution in search of a problem”, a sure road to failure. I will blog about this qualitative first step in an upcoming blog.
  • Eric Ries website and a goldmine of information
  • Lean Startup Meetup Groups exist all over the United States and the world. Tap into one of the “meetups” to learn more about how to use the process in your association.

Leave a reply