Management Knows Best: Or Do They? Outcomes from the real world.
Introduction: Surveys have been around since at least the beginning of recorded history. Today many organizations use surveys to assess job satisfaction, reaction to a planned change in a product or service, and customer satisfaction, to name a few applications. While there have been advances in both the design/administration and analysis/interpretation of surveys, too many are haphazardly done and yield little more than ‘interesting’ outcomes. Debacles like the Literary Digest survey in 1936 forecasting an Alf Landon landslide win over Franklin Roosevelt, or Gallup’s 1948 prediction that Thomas Dewey would easily win over Harry Truman, or the decision based on extensive consumer surveys by Ford to make the Edsel, have had little impact on deterring poorly designed and executed surveys.
Encouragingly there is an increase in professionally done, psychometrically sound surveys that result in reliable, actionable data. These are characterized by:
• properly drawn samples,
• effective items with realistic response alternatives,
• pilot testing to ensure accurate end-user interpretation,
• outcomes linked to other variables, such as financials, etc.
In job satisfaction surveys, a common analysis has been to look at the management versus non-management sample. Over the years these studies repeatedly indicate management tends to be more satisfied than non-management on most job related issues. The ‘scientific’ reaction to this outcome for most practitioners is a “no duh.” But lest I digress that is NOT the focus of this article. Rather than job satisfaction, this looks at customer satisfaction and how well organizations predict it- with a twist.
Specifically, we wanted to compare how well management could anticipate what their customers’ level of satisfaction was compared to what non-management thought it would be.
Background: Typically when we do customer satisfaction surveys we conduct a parallel internal survey designed to measure what the client anticipates the customer will say. This allows us to evaluate not only the key message from the customer, but also how well the client knows the customer. Actually we take this same approach with our 3600 tools as well. It quickly minimizes the “I knew that” reaction to feedback data and helps focus on not only the message but the significant gaps where the message was truly not anticipated. Further, obtaining an internal measure of what they anticipate the respondent will say avoids the common response of devaluing the data because it is ‘not accurate’ or ‘cannot be right because here are the facts...’
Real World outcomes: We recently completed an annual customer satisfaction survey program for an international client. The survey was primarily administered via the web and responded to electronically, hosted on one of our secure servers. [Small digression: This was a change for this client. Previously we had done their surveys using traditional hard copy media and first class mail or airmail. We found return rates were slightly better for electronic versus paper versions, but in both media the return rates consistently ran at twice industry average. We attribute that to careful piloting and our administration process. Non-significant differences were found on item response characteristics when comparing paper to electronic for matched samples. This is important to check any time you have two different mediums. In some applications we have found consistent differences that preclude combining the data without appropriate adjustment.] As with prior surveys for this client, the instrument was available in half a dozen languages in parallel forms, content adjusted for cultural differences and to ensure close if not identical interpretation by the end-users.
Overall results documented significant differences in response patterns based on the location of the respondent, specific customer group and product line, as it had in the past. Looking at the big picture, both management and non-management’s estimates were highly correlated with that of their customers’ responses (.810 and .910, respectfully, significant at the .01 level). That is, this client continued their trend of knowing the direction of their customers’ satisfaction across the broad spectrum of customer satisfaction components measured. However there were numerous differences between the management and non-management data that were not expected.
Considering all items on the survey, management underestimated customer satisfaction by about 10% while non-management overestimated customer satisfaction by about 3%. This difference becomes more dramatic when you look at individual items. Management’s estimate of their customers’ satisfaction was five or more percentage points lower than the actual customer rating on over 50% of the survey items, as compared to only 15% of the items for non-management. Conversely management overestimated by five or more percentage points what their customers actually said on 21% of the items as compared to 38% for non-management.
Conclusion: Management does not necessarily know best. In general the results document that non-management had a better (more accurate) handle on their respective customers’ specific levels of satisfaction. Management’s reaction to the difference between their results and non-management’s was one of surprise, especially in the degree of the divergence. Their explanation was that they tended to get brought in when things were going badly with a customer to a far greater degree than when things were going well. This in turn colored their overall perception of the customer’s level of satisfaction.
This client has a history of taking the survey outcomes seriously and setting action plans in place to achieve improvements. For example, data on technological competence came in lower than they expected one of our earlier surveys. They took specific steps based on the feedback and increased their customers’ perceptions of their technological capabilities by more than twenty percentage points in less than two years. Similarly they used one client’s positive outcomes to go back and increase the amount of work they were sourcing to them. They are taking a similar focus with the outcomes of this year’s survey. Also management is now working on ways to maintain a more accurate picture of their customers’ level of satisfaction (in addition to the annual survey). We are in the process of conducting and analyzing new global customer satisfaction surveys to see if this management non-management difference is replicated.