Saturday, 4 February 2012

We are unable to process your response

Scientific surveys are distrusted by many people and organisations. Sadly, due to the fact that surveys are not always particularly well written, this is not entirely without reason.

Many years ago, when working for a health and safety consultancy, I was shown a newspaper article about the enforced closure of a care home after they had failed an inspection on safety grounds, such as their banisters being a couple of millimetres too wide apart. "These people aren't interested in whether or not you've got a loving home," one of the very upset care workers was quoted as saying, "they're only interested in ticking little boxes."

More recently, the IT firm Atos has been the recipient of the outsourcing of decisions on sick and disabled people's welfare, taking these decisions out of the hands of GPs and others who know the welfare recipients, and placing it in the hands of a survey for which you have to score points to be declared unfit for work. LatentExistence describes it in more detail here, and this is one of the results of this procedure. (By the way, if I say anything too critical of this company, my entire blog may be shut down - this happened to CarerWatch and it took a lot of fighting and correspondence to find out that the entire forum, which is a pillar of sanity and support for many exhausted, poor and desperate people, was closed due to a link someone had posted many months previously. But I recommend Margaret McCartney's writings on them, too - sadly the BMJ article I had in mind, and which I believe is linked to here, no longer seems to be available.)

In other words, a badly thought out survey can have horrific - and fatal - results. It can of course also be fairly hilarious to those who have the time and ability to pick it apart, as bloggers did to the BCA's "plethora of evidence" about chiropractic being effective back in 2009.

I'm currently earning my pennies by doing some scientific data entry, which involves a bit of database testing. I'm actually finding it both fun and fascinating, and also discovering just how much thought has to go into writing a survey and its results. A simple "N/A" in a box where an integer is required means that query after query gets generated, multiplying the poor data manager's work. When you create a survey, study, or report, you have to allow for various responses.

The problem a lot of people cite (in my experience, anyway) with surveys is that they "don't give a holistic picture", "ignore the real person", "don't treat anyone as an individual", "reduce important things to tick boxes" and so on. The trouble with this is that a really large survey can't treat everybody, or anybody, as an individual, except for case studies. You need to state exactly what you want to find out, and how much. No survey can find out everything about everybody! And if what people say isn't representative of what's really going on, or the results don't make any sense - that's when you've got a problem.

If your tick boxes make people feel like that, this doesn't mean that surveys involving tick boxes are the problem, it means that the wrong questions are being asked - or, if it's multiple choice, that the wrong range of answers are being offered.

It's a pretty good idea, I'd say, to do a trial run of a survey, and find out where these errors are coming from. No planning can possibly think of everything that will go wrong. So do a practice run, change what you need to, and then have another go. (This is much better than, say, adding a new question halfway through what you are doing - this makes the whole thing a mess.)

Still, I would have thought that even a mobile phone company would have had the imagination to forecast the problems they created themselves with the survey they just sent me. I won't name the mobile company, but they've just offered me a new contract. I've accepted it, because it's a lot better than my current one, and also there isn't a minimum time on it. Let me make clear that they telephoned me while I was at work and I asked if they could call me back another time, which they did, exactly when they said they would - and they did not much underestimate the time the call would take, which was a big bonus. Anyway, next day I got the following message as follows:

MOBILE: "From [X]: You recently spoke to us on [Day X] and we'd like to ask you 6 quick questions about your experience. All your feedback is free."

MOBILE: "Question 1: Was this the first time we've spoken about a specific problem or query? Reply with Y for yes and N for no."

So far so good . . .

ME: "Y".

MOBILE: "Question 2 of 6: Is your query or problem now resolved? Reply with Y for yes and N for no."

ME: "Y."

First problem. I thought I'd better add to that . . .

ME: "Except I didn't have a problem/query. [MOBILE COMPANY] did."

MOBILE: "Error: We were unable to recognise your response. Please enter a number between 0 and 9 where 0 is 'poor' and 9 is 'excellent'."

MOBILE: "Question 3 of 6: That's good to hear. Overall how would you rate our advisor on a scale of 0 to 9 where 0 is 'poor' and 9 is 'excellent'?"

I know a few people who work for these companies and know that if you answer anything other than the top number, they get a bollocking and lose their commission. (Mobile companies are not always kind to their staff - I recommend a read of this experience Dean had with a different one.) So although it was annoying being rung up and I had to listen to the standard waffle, the guy was a good listener and answered - even anticipated - all my questions. So I decided to be generous.

ME: "9."

MOBILE: "Question 4 of 6. How would you rate the time it took before you spoke to an advisor where 0 is poor and 9 is excellent?"

I think by now it's clear that they have started off this survey by making a very specific assumption which needed clearing up before questions and answers could helpfully continue.

ME: "N/A. I didn't call you."

MOBILE: "Error: We were unable to recognise your response. Please enter a number between 0 and 9 where 0 is 'poor' and 9 is 'excelent'."

ME: "'Excellent' is spelled with two 'L's. Please take a literacy course."

MOBILE: "Unfortunately we are unable to recognize the response in your message. Please try again at another time."

By not allowing for a few very simple different situations, or employing a proofreader, or even allowing a Ctrl-C Ctrl-V to be used in designing error messages, this company has messed up its own surveying ability and wasted its own time and money as well as mine.

And this is why it's important to learn how to design a survey before you do one.

12 comments:

Rebecca said...

Just in case you are interested, I found Norbert Schwarz's paper "self reports: how the questions shape the answers" enlightening for issues within qualitative research. It's long but the gist of it is that very simple changes in wording, question order, open/closed format, and the magnitude of the options (0-10 or "never" to "often") can drastically alter the responses that people give. Worrying stuff.

Sakib said...

At least astronomical allsky surveys get results! :-)

Kold_Kadavr_flatliner, sub/dude said...

Isn't it interesting, maam, you study the stars, yet, you have no idea who made'm nor do you care. And you're how old? 17?? I'm God-fearing because we A-L-L must someday sometime somewhere bite-the-dust. Then what? Then, our body becomes food for grubsNworms, while our indelible soul lives fo'eva. While we can escape Jesus while on earth, we can never escape after death: Jesus shall judge U.S. on how well WE have lived our finite existence, deserving Heaven or Hell. No inbetween. So, it wouldn't be a reeeeeeeel good idea to be a LeftyLib who only worships the whorizontal. God bless you with discernment. Meet me Upstairs, k, where we'll have a BIG-ol, kick-ass, party-hardy celebrating our resurrection for maaany eons (then, I gotta wanna vamoose to love more). You're most invited --- My suggestion? Go to a Catholic church and find-out when RCIA is. Yes, the Catholic church was founded by God; the rest were founded by mortals. See ya soon, bud.

Kold_Kadavr_flatliner, sub/dude said...

Sorry. Purgatory is there for those who don't know QUITE how to dress for the wedding banquet; you'll soon be released to enjoy the magnificent splendours of the length and breadth of forever. Won't that be kick-ass?

endless psych said...

Two things come to mind upon reading this.

The ATOS thing you mention isn't a scientific survey and neither is the market research you mention at the end.

The box ticking exercise you mention also doesn't appear to be a survey in the same sense you mean but rather a survey of a building and it's fixtures and fittings. Which isn't really comparable.

As to your criticism of surveys as not treating people as individuals. Well it depends on the survey. You can certainly argue that they aren't holistic and so on and so forth but at the end of the day the criticism levelled at "scientific surveys" is unfair.

As they do indeed pilot such measures and they go through innumerable and rigorous processes before being sent to a large scale population.

It is not scientific surveys that are the problem it is in fact psuedoscientific ones ;)

Alice said...

Hey! I get stalky style loony spam! (On a blogpost that doesn't even mention religion, much less the Catholic church!) Those are just too hilarious to delete.

midali said...

Religion vs Science is an old debate.
Here are mostly science addicted people. It's a question of... choice ?

Alice said...

Keir - I have not actually claimed that all surveys (particularly not Atos or the marketing nonsense) were scientific. I thought it was merely useful to compare types, aims, and qualities of surveys. Also I think I explained that I disagree with the charge that surveys can't be holistic etc, and explained why?

Anyway, if this wasn't clear, then many thanks for the feedback and it's good to get my arguments refined. Keep it coming!

Thanks all for the comments - I'm surprised that this off-the-cuff little snippet got so many! I really must try and write more here! :-)

Mike Hypercube said...

Some great management advice and pointers there - keep it coming. Looks like you might want to look for a new audience though ;)

Even if not many phone companies are reading this, the sort of failure of imagination your example highlights, the unquestioned assumption that gets hard coded in, is something we must all find ourselves guilty of at one time or another. This has to be one of the funnier ones though.

Mike Hypercube said...

Oh, and as if by magic, the tortuous procedure for publishing a comment here on this blog using my Google ID suffered from exactly the failure of context that your phone company example suffered from. Though less amusingly.

Anonymous said...

Keir,
Is the Catholic Church of yours will allow child abuse by priests and pay compensation after. is there one really ruled by the God ? Where it is ? In which Galaxy? On which earth? (Not on our one for sure)

Alice said...

Hey, Keir didn't post the stupid Catholic church comments! I should have said "Endless_Psych" - sorry, Twitter strikes the autopilot button again ;-)

Mike - sorry - I enable comment moderation when the post is >5 days old. That means I only have to keep track manually of what has happened recently . . . But two posts from you have got through fine - hopefully that was what you intended!

Much enjoying all these comments - except the obvious, which I enjoyed in a very different and less positive way.