Decoding Data Series, Part III: Survey Challenges in Military Data Collection
In Part 3 of our Decoding Data Series, we examine how survey fatigue, methods, and trust challenges stand in the way of reliable military data collection.
The Uber Experience
Imagine you’re in an Uber, cruising downtown to meet friends at a local food spot.
You arrive, hop out of the car, and thank the driver. In the blink of an eye, the Uber app prompts you to share your ride experience. Five empty stars eagerly await your judgment.
Instinctively, you tap the star furthest to the right that perfectly matches your five-star-worthy journey. Behind the scenes, this data is stored and applied to the driver’s overall rating with all the other feedback that shapes the driver’s street cred (literally).
Simultaneously, the driver gets a chance to rate your behavior as a rider, also bestowing you five stars for your polite and safe behavior and shaping your reputation as a rider.
It’s an instant motivator to do things right: nasty and rude riders get less business and service, respectively; good and polite riders get better.
This system works by design and is a clear and straightforward example of good data collection and analysis.
Micro Surveys and Reliable Feedback
The example above is a micro survey: one of the super effective ways businesses receive reliable user feedback.
When it’s reliable, user feedback is a priceless way to get an edge over the competition.
X (the Twitter company going through its Prince phase) made a significant change when it ditched its infamous 256-character limit, a decision driven by a 2006 survey.
In its earliest days, Starbucks used feedback cards to discover customer favorites, transforming them into worldwide staples.
Big airlines like American Delta and United have all leveraged small surveys to improve customer flight experiences.
The sentiment is evident when you can get good, reliable user feedback; you can make significant changes.
The Military Experience
In the US Armed Forces, there are behavior surveys, peer surveys, subordinate surveys, retirement surveys, and many more. Unlike the concise 1-question Uber rating, military surveys are notoriously long. Even the short surveys usually have upwards of 20 questions.
It wouldn’t be so bad, but long surveys seem like the military’s go-to tool for gathering insights on virtually every topic.
The History of Military Surveys
In October 1941, the War Department created a research branch that began a program of surveys that December. The program’s broad purpose was to investigate the morale of soldiers.
The project grew and evolved over the next four years and served as a vital part of getting the perspective of Soldiers during the height of World War II.
Soldiers spoke about the need for less “boot-licking” and hated the culture of favoritism. Even then, they wanted more merit-based promotions and to be treated fairly. They also shared their positive outlook on the Army and appreciated their service as outstanding opportunities for upward mobility.
With the success of the surveys during World War II, it is no wonder that the military continues its tradition of surveys. The military conducts surveys on all aspects of military life, and when the survey data is reliable and the analysis is practical, surveys help drive policy decisions.
Survey Problems
The current state of surveys has its problems:
Survey Fatigue
During a year, the military asks service members to take 20–35 military surveys. Ask an Army friend to do a simple inbox query of “survey,” and they’ll likely smile at you. There is a possibility that they are a bit sick of taking surveys.
One of the reasons they may be tired of surveys is the overwhelming demand for their time. In the monograph Lying to Ourselves: Dishonesty in the Army Profession, Dr. Wong and Dr. Gerras make it clear that there are often more requirements than time to reasonably accomplish them at this point — to the point where military leaders can be dishonest. So when given a choice between taking an optional survey and doing the overwhelming amount of mandatory work that exists, surveys take a back seat every time.
Surveys are Too Long
The website SurveyMonkey did a study on surveys. They found that when surveys take longer than 7–8 minutes, survey completion dropped anywhere from 5 to 20 percent.
The average exit survey in the Army takes 20 minutes to complete. It is no surprise that the exit survey is mandatory.
To the survey-taker, a survey you are (1) forced to take and (2) takes twenty minutes to complete starts to feel more like a test than a survey.
Surveys Aren’t Accurate
To be clear, it is not fair to say that people are being dishonest on surveys. If people think that specific answers are more likely to elicit inevitable consequences, they may be more likely to respond with something other than what they truly feel.
If you’ve ever taken a hospital survey asking, “How many drinks do you consume a week” a week after a vacation or beer festival, it’s safe to see that honesty isn’t always our policy.
We do not like to feel judged. If there is a remote chance of reprisal or consequences for our honesty, we will likely be dishonest in our responses.
Sprinkle in pressuring emails from flag officers telling you why it is crucial to do a survey; all of these factors create a highly unreliable atmosphere of surveys compared to their industry counterparts.
Ultimately long military surveys risk becoming a game of bad data in and lousy analysis out.
The Impacts
We Can’t See Ourselves
The inability to collect data worth anything in surveys leads to a scenario of crap data in and crap data out. Not seeing ourselves has a significant impact on the military. For one, this means we cannot see ourselves accurately, which means leaders cannot make accurate decisions based on the information provided.
Data Aversion
It makes the military data averse because we ultimately do not believe that information can provide any value to our decisions because of how long it takes to gather the data, how inaccurate it is, or how few people they’ve gotten quality data from.
Relying Purely on Hunches
Without quantifiable data, anecdotes, intuitions, gut feelings, and experience are the only things people can fall back on. That is not to say that experience and gut feelings, and impulses aren’t valid, but we must be honest with ourselves. We make data-driven decisions every day of our lives. It might not look like a metric, line graph, or chart, but if we see it raining outside, it changes our plan of what we will do. If we can’t see and walk out, we become blind to what’s hit us.
Bad news does not get better with time, but if you cannot see it, you guarantee it will only get worse.
Solutions
So, what are the ways we can fix this problem?
Solution 1: Adopt Micro Surveys
The easiest of these would be to adopt micro surveys. The Army already does this in the “After Action Reviews” technique. In my experience, Soldiers give some of the best feedback in these impromptu sessions. Writing down that feedback is a challenge.
But the reality is 1–3 question surveys have a much higher completion rate than the average Army survey. As a rule of thumb, surveys should have five questions or less. It turns out people appreciate when we cut to the chase.
Solution 2: Incentivize Detailed Surveys
The world cannot run on micro surveys alone. But typically, big companies make participating in tailored surveys something beneficial. Some companies offer an “Insider” membership, perks, discounts, and other neat features for completing a lengthy survey.
Sure, “it’s the noble thing to do” is also a great motivator. But given that many people you’re asking to volunteer for the survey are already making huge sacrifices, it wouldn’t hurt to incentivize their desire to take a survey.
Solution 3: Make it Extremely Clear What the Survey’s Being Used For
The other day, one of my peers told me they were nervous that a diagnostic assessment might be used against them later in their careers. That sentiment is a realistic feeling in the Army, and when trust breaks, it is tough to believe that the military will keep its information promises with its service members.
The Verdict
Just like at any other time in history, there is a huge need for accurate, concise, and precise information. Nowhere is this true more than in the Armed Forces.
The sooner we start to create surveys that are competitive with the styles of surveys found in the industry, the sooner we begin to analyze that data as if we have a bottom line. The sooner we start to have quality insights into the mentality of our soldiers and items that senior leaders need to address so that our Armed Forces continue to be the best, the most powerful, and most cunning military n the world.
