Understanding Mode Effects between Mobile Web and Mobile SMS Surveys

By Brandon Poduska, SSI
Paul Johnson Senior Analyst, SSI
Introduction
With cell phone usage in the US an estimated 90-97% of adults and the increasing capabilities and availability of mobile technology, market researchers can now reach respondents almost anytime – almost anywhere. The ability to gather opinions closer to the “moment of truth” is an important benefit offered by mobile research. Understanding potential effects on responses and the nuance of mobile research is vital before we begin to benefit. To test possible mode effects between standard web survey formats, simplified mobile web formats, and standard text messaging (SMS) formats, SSI panelists were asked if they would be willing to participate in a research effort using their mobile device using SMS text message or mobile web browser to respond. The results were encouraging: 62% of panelists with a mobile device and 80% of panelists with smart phones were willing to engage using mobile survey technology.
Method
To test possible mode effects we planned to administer a simple questionnaire about respondents’ perceptions of a certain event, in this case the opening weekend of Iron Man 2. First, we pre-screened panelists using our dynamic profiling system to identify respondents planning to see Iron Man 2 on opening weekend. All 520 respondents who volunteered to participate and intended to see Iron Man 2 opening weekend were assigned to one of four groups. Each group was assigned a survey format: standard web format, simplified mobile web format, SMS text messaging format, and a choice to take the survey in any of the three formats.
E-mail invitations were sent to each group on the morning the movie entered theaters. Instructions for the SMS text message group included instructions to text the word “start” immediately after watching the movie. Those in the mobile web group were instructed to click on the survey link from their mobile device immediately after watching the movie. Those in the standard web group were provided the normal survey invitation that did not request an immediate response after seeing Iron Man. Those in the fourth group were presented a choice with instructions to complete using the link to the standard web survey, the link to the mobile web survey and instructions for completing via SMS text. The survey was closed on Monday morning and results were then tallied about the weekend reaction to the survey.
Results
Before we looked at data difference due to mode effects, we wanted to account for potential demographic differences. Based on the random assignment of groups and high response rate, most demographics were balanced across the three modes. The SMS text group had significantly more 18- 24 year olds. To avoid confounding demographic and mode effect, we weighted the data in each group to the total demographic distribution. After accounting for demographics we still found potential mode effects.
The overall rating of Iron Man 2 was significantly higher in the text message group. The same trend occurred on the rating of the special effects and the story line. The acting and character development ratings were similar with no differences by mode. More importantly to revenue, a significantly higher proportion of the text message group reported being more likely to recommend the movie to a friend. In contrast, a significantly smaller proportion of the text message group plan to see the movie again in the theaters or purchase the DVD. Surprisingly, there were no significant differences between the mobile and the standard web format.
The complete SSI White Paper can be downloaded here: SSI White Paper Understanding Mode Effects between Mobile Web and Mobile SMS Surveys