Fine-Tune Your Online Interventions with Paradata

Paradata provides insight into the process by which online interventions can affect outcomes

Online interventions allow you to reach large numbers of potential participants with cost-effective, tailored designs. But making something available on the Web doesn’t guarantee participation. Information about how participants interact with your online interventions—called paradata—can help you refine them, according to the results of a study published recently in the Journal of Medical Internet Research. That study found that capturing and analyzing paradata helped explain how tailoring an online intervention increases participant engagement.

The authors note that a key concern for designers of online interventions is that participants who are not engaged may drop out, be lost to follow-up, or experience an attenuated treatment effect. Enter paradata, which can include general information—what browser a visitor used or the connection speed—to specific information designed into the intervention—which pages participants visited, when, how often, and for how long. Paradata have not been used extensively in health interventions, so the authors looked at how participants engaged in a multi-arm randomized controlled trial of an online intervention designed to increase fruits and vegetable consumption, how engagement related to study retention, as well as the relationship between engagement and key outcomes of the trial.

The researchers analyzed the information from 2,513 men and women between the ages of 21 and 65 years who participated in the Making Effective Nutritional Choices for Cancer Prevention (MENU) study. MENU tested an interactive website designed to promote increased intake of fruits and vegetables. The intervention provided four education sessions over four months, including motivation, information, behavioral strategies, and supplemental features, including 300 fruit- and vegetable-based recipes and food preparation videos. Participants were randomly assigned to one of three arms: untailored, tailored, or tailored with email support. Patients were asked to report fruit and vegetable intake at baseline, and at three, six, and 12 months.

The researchers measured participant engagement by gathering paradata, specifically the time stamps for five actions: logging into the website; starting any of the online surveys; completing any of the surveys; loading the first page of any web session; and loading the first page of any special feature. The authors then created two indirect engagement measures: breadth, which measured how much of the offered activities a participant visited; and depth, which measured how long a participant spent on any one element of the intervention. The two tailored arms were significantly more engaged than the untailored arm. Breadth and depth, however, were significantly and strongly associated with completion of follow-up surveys and greater breadth was associated with greater increase in fruit and vegetable consumption.

The authors write that, “This paper provides a starting point to identify areas where online intervention design improvements may be required and, ultimately, may give us clues as to why a particular intervention may be more or less successful.”

Source: Couper MP, Alexander GL, Zhang N, et al. 2010. Engagement and retention: measuring breadth and depth of participant use of an online intervention. Journal of Medical Internet Research 12(4):e52.