A Tale of Two Teams – When The Data Isn’t There – An Alternative to Kirkpatrick

Linking results to a learning event can be difficult. Perhaps it is because the right systems are missing to track the right data to understand the impact the learning event has had. This dilemma led me on a hunt to find alternatives to Kirkpatrick’s Levels of Evaluation, the standard in corporate training. On my search I found a guy named Brinkerhoff and his Success Case Method (SCM) which he outlines in several articles and a few books, including Telling Training’s Story.

I utilized the methodology to evaluate a course for which Net Promoter Score (NPS) is the best indicator, and a measure that has very low adoption in the organization under study. My objective was to understand if this course was proving successful and what adjustments might be made. While the results of the study weren’t surprising, they provide a very compelling illustration of how to achieve results from learning.

At its core, the SCM seeks to find individuals who were the most and least successful at applying what they had learned. Brinkerhoff outlines a quite simple way to find these people, which I was skeptical of, however found it to be simpler than expected. Once those individuals are identified, the practitioner conducts structured interviews with each of those individuals to understand the success factors or barriers to application.

While the information uncovered from each of the interviews was quite valuable, there were three interviews that stood out. Two of them were high success cases and one was a low success case. The combined stories tell us something that isn’t a surprise to experienced learning practitioners. However, it is often hard to get managers to really understand without an illustration such as this.

The three individuals all attended the same learning event in the same location at the same time. All three individuals said they enjoyed the class and that the instructor did an excellent job. Why was it that two of the individuals identified they had been extremely successful in applying what they learned, yet the third individual claimed to have not used what they had learned at all. Let’s look at each of their stories.

The design of the course was set up to introduce and define four different levels of client relationships for sales individuals. Participants analyze their current client portfolio and classify each of the clients in the relationship category in which they believe they sit. Participants are then introduced to a formula for building trust within a client relationship and provided examples about how they might go about achieving it. Participants then look at their client portfolio and begin to create a plan for actions they can take with each client to move them up the ladder of client relationships. Lastly, participants are given examples of power questions that will help develop their client relationships. The end result is intended to be higher NPS and increased revenue.

Big Revenue Ryan

Before I dig into the details of Ryan, it’s important to caveat that each of the high success interviews looks at one specific success the individual has had, to get a deeper look at how and why they were successful. Ryan had been attempting to land some business with a particular client and kept running into walls. They weren’t looking to take on any additional suppliers. He was constantly getting shut down by procurement, a challenge that many sales people can identify with. Ryan persisted, mapped out several individuals in the organization and was able to finally identify some individuals who were disappointed in their current suppliers. Ryan convinced them to give him a shot at their business and was able to fill their needs in a fraction of the time the other suppliers were. Ryan continued to build relationships with a few of those individuals and continued to win more business and each time was able to beat the competition. It didn’t take long until those individuals went to procurement and asked to officially add him as a preferred vendor. Ryan continued to build more relationships across the business and is now the go-to person for several across the organization. Managers at this company continually consult Ryan on various facets of their business because they value Ryan as an advisor. Goal achieved! To date, with just this one client, Ryan has closed nearly $1M in revenue and has an average Net Promoter Score 9.8, nearly perfect.

Let’s look at what specifically Ryan did to achieve this result.

  1. Ryan respected internal processes. Often in sales, the sales people go looking for a backdoor to go around the decision makers. Ryan respected their process and earned the trust of everyone involved because they knew he wouldn’t circumvent their processes. The initial point of building trust was by adhering to the process that procurement had set out.
  2. Ryan set appropriate expectations and then exceeded them. He found that hiscompetition wasn’t delivering at a very fast pace, sohe was able to exceed all expectations by filling the roles quickly which then earned more trust from the client.
  3. Because Ryan wasfilling roles quicker than any of the other preferred vendors were, the client began giving them the heads up on new roles before they were sent to the preferred vendors. This allowed himto be ready to fill any new roles that came up before any other vendor, resulting in increased response time.
  4. Once the trust level of this client increased, other internal stakeholders began approaching himfor advice before they approached other preferred vendors. “It wasn’t me asking questions anymore, it was me being able to give advice…”.
  5. Ryanremembered intimate details about the clients,he would talk and ask questions about their family, or how their home remodeling project was going. He showed a true interest with each of the individuals with this client. They in turn would ask how his wedding planning was going and asked intimate details about his life.
  6. Something that helped in application was being able to recognize the various parts of a relationship and how to build those, from credibility,to reliability, to intimacy. He began to analyze what it was going to take for that client to trust him.
  7. Another benefit ofmoving up the ladder for Ryanwas that he didn’t feel the need to have regular meetings with the client just to maintain the relationship or connection. They both proactively reached out to each other as necessary, allowing Ryan to focus on building new business in other areas of his market.

Business Savvy Sarah

Sarah’s story is a little different. Sarah hasn’t closed any business yet with the client she told me about, but to quote her: “Come back to me in June, if I haven’t closed over $1M in revenue, make sure I’m not working here anymore.” That’s a lot of confidence!

Sarah began talking to a new potential client who didn’t have an immediate need. She kept up the conversations though because she could sense there was some opportunity lurking. In one of the conversations that Sarah had with this client, they had both expressed a mutual interest in Argentina. The client had mentioned that he was considering expanding his operations in Argentina, but wasn’t sure if it was a good decision so he asked Sarah for her opinion. Now, it’s important to remember that Sarah isn’t a business analyst. However, Sarah thought this was an interesting challenge and performed a good amount of research and came back to the client with a presentation, a SWOT analysis if you will, of expanding operations in Argentina. Sarah’s recommendation was that the business climate in Argentina was not conducive to expansion and she recommended to the client that he not move forward with his expansion plans. The client appreciated this advice and the relationship deepened. The conversations continued until finally the client indicated that they had intentions to move the operations of their local business an hour from its current location. Because of this move they anticipated that they would lose 30 to 50% of their staff and would need the services of Sarah. Guess who’s on the docket to do business with them now? This executive began to make introductions all across the business for Sarah so that she could begin building relationships in anticipation for the need to replace several employees who wouldn’t make the move.

Let’s look at what specifically Sarah did to achieve this result.

  1. Sarah asks lots of questions to understand the industry and businesses she is working with. As an individual new to working in her assigned industry, Sarah didn’t know a lot about the businesses she was working with. The only way to learn is to ask questions and then use that knowledge to ask more informed questions of other potential clients which increases her credibility bybeing able to have an informedconversation with her clients.
  2. Sarah began by doing one small deal with this client. This opened the door for her to have more conversations. In those conversations, it wasn’t about her finding more business, but a genuine interest to learn more about the client.
  3. Sarah went above and beyond by doing research into a country the client was interested in doing business This is something that isn’t a part of Sarah’s every day job, but the research she did built a tremendous amount of credibility with the client. Delivering on this type of work that wasn’t a part of Sarah’s every day job also built a sense of reliability with the client.
  4. Sarah continued to build relationships with this client as the first executive she worked with made introductions.

Busy Becky

My interview with Becky was a low success interview. Becky agreed that the class she attended was great, but when asked why she hadn’t seen any success in applying what she learned, she articulated that it was probably just her. It takes time and focus to think about applying something new, and she was just too busy to think about it. She took on all the responsibility herself and agreed that maybe she could see a lift in her business if she actually focused on it. I also asked her what her manager’s involvement was in discussing what she had learned and helping her to apply it and she admitted that they hadn’t even had a single conversation about it. The manager was clueless to what the class was about, not a single conversation about the learning event she attended. When asked what improvements Becky would recommend to the class to help her apply what she learned, she admitted that more focused time and follow-up would have been helpful.


So why is it that these three people had such different experiences when they were all in the same classroom at the same time and work in the same location? We’ll start with the fact that Ryan and Sarah also had their manager present in the class that they attended, Becky’s was not – nor had he ever attended this class before. Ryan and Sarah’s manager had weekly meetings they called “keystone meetings” in which they would discuss a relationship with a current client that they had improved and the steps they took to make it better. They then took those lessons learned as a team and chose another client they would focus on advancing a relationship with. It wasn’t the learning event that created the change, it was only the initial seed. It was the daily and weekly focus of the manager and team in applying what they had learned that made it blossom into something great. As stated before, Becky’s manager didn’t even have a conversation with her after the learning event to see what she had learned or how she might apply it.

We learning professionals have known this for some time: manager support after the learning event is what determines the success of learning application. But as much as we scream it from the rooftops, they don’t seem to understand the importance of this. They think that going to training is about being inspired and having fun in a classroom and then magically their people will come back transformed and able to apply what they learned. This just isn’t the case.

My personal conclusion was that the SCM process has provided much more valuable information than the typical interventions learning professionals typically employ. Brinkerhoff claims that his process can be more successful at identifying why a program was or was not successful because it looks at the extreme cases and looks to learn from them. Whereas Kirkpatrick’s model looks at averages and you lose the robust and valuable stories that sit on the perimeter. When I look at the base survey data which I have employed with many other programs, there are no major trends which emerge that would tell a practitioner this level of detail in creating success.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s