Diffusion of treatment research: does open access matter?

Diffusion of Treatment Research: Does Open AccessMatter? Advocates of the Open Access movement claim that removingaccess barriers will substantially increase the diffusion of academicresearch. If successful, this movement could play a role in efforts toincrease utilization of psychotherapy research by mental healthpractitioners. In a pair of studies, mental health professionals weregiven either no citation, a normal citation, a linked citation, or a freeaccess citation and were asked to find and read the cited article. After1 week, participants read a vignette on the same topic as the articleand gave recommendations for an intervention. In both studies, thosegiven the free access citation were more likely to read the article, yetonly in one study did free access increase the likelihood of makingintervention recommendations consistent with the article. & 2008Wiley Periodicals, Inc. J Clin Psychol 64: 821–839, 2008.
Keywords: open access; diffusion; dissemination; accessibility;publishing Clinical researchers often express the concern that mental health practitioners do notpay sufficient attention to the results of treatment research. There is substantialevidence for a gap between clinical practice and the implications of clinical research.
For example, comprehensive reviews of controlled clinical trials of therapies foralcohol problems have shown that some of the most commonly utilized therapies havethe weakest empirical track records (Miller, Andrews, Wilbourne, & Bennett, 1998).
Many plausible suggestions have been advanced to try to improve dissemination andreduce research/practice gaps, such as conducting effectiveness rather than efficacy We are grateful to Brenda Reis for reviewing and revising the materials used in the second study.
Correspondence concerning this article should be addressed to: David Hardisty, Department ofPsychology, 419 Schermerhorn, Columbia University, 1190 Amsterdam Avenue, MC:5501, New York,NY, 10027; e-mail: [email protected] JOURNAL OF CLINICAL PSYCHOLOGY, Vol. 64(7), 821- 839 (2008) Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/jclp.20492 Journal of Clinical Psychology, July 2008 research to improve external validity (Westen, Novotny, & Thompson-Brenner, 2004),improving treatment-development practices to take advantage of existing clinicalwisdom (Morrison, Bradley, & Westen, 2003), and changing training-programorganization to emphasize research literacy (Nathan, 2000).
In the research reported in this article, we evaluated whether it would help to take a simpler step—that of improving practitioners’ literal access to journal articlesreporting the latest advances in treatment. After all, it does not matter howrepresentative the sample is or how well the reader understands the methodology ifthe article is not read. Busy practitioner schedules and the overwhelming output ofjournal articles reporting research on psychotherapy combine to make it highly likelythat simple unawareness of new literature, or lack of easy access to it, representimportant impediments to dissemination.
In the broader field of scientific publishing, researchers confronting similar dissemination issues with respect to their fellow researchers, applied scientists (e.g.,medical care providers), and the lay public have argued that dissemination could beimproved by increasing accessibility of research articles. The so-called Open Access(OA) movement seeks to make scholarly research results available on the Internet toanyone, anywhere, at any time, free of charge, and free of most copyright andlicensing restrictions (Harnad, 2003). This can take the form of delayed OA journalssuch as the New England Journal of Medicine, which provides free access 6 monthsafter initial publication; author-fee OA journals such as those found at BioMedCentral; and self-archived eprints, whereby authors deposit pre- and postprints in apublicly accessible Web site (Willinsky, 2003).
The case for OA is sometimes argued as a justice issue (Mattaini, 2004). Indeed, the broad consensus that taxpayers should not have to pay for access to publicly fundedresearch led to the federally mandated National Institutes of Health Public AccessPolicy, which requires all investigators to make their peer-reviewed manuscriptsfunded by the National Institutes of Health freely available to other researchers andthe public through PubMed Central (National Institutes of Health, 2005).
From another angle, OA proponents have argued that removing access barriers will increase the diffusion of scholarly research among researchers and the public,thus maximizing research impact. A citation study by Thomson Scientific (2004)found that available OA journals in the Web of Science and in the ISI Web ofKnowledge databases were more often among the lower ranking journals in the fieldby impact factor. Sixty-six percent of the OA journals ranked below the 50thpercentile overall on impact factor. Furthermore, only 6% of OA journals were in orabove the 91st percentile. However, these are correlational data that do not lendthemselves to causal interpretation. It could be that low-impact journals areparticularly likely to have taken the step of providing OA at an early stage of thistrend, as a means of maximizing their readership, while some publishers of journalswith larger impact factors might see OA as a threat to traditional sales of theirproduct and thus be less likely to offer it.
A different method of studying OA in relation to citation impact controls for preexisting journal status. Harnad and Brody (2004) compared the citation counts ofindividual OA and non-OA physics articles appearing in the same (non-OA)journals.1 They found citation advantages for OA articles of 200 to 300%, dependingon the publication year. Similar studies have compared OA and non-OA articles inastronomy, computer science, electrical engineering, mathematics, philosophy, 1The OA articles in non-OA journals were made OA by their authors through self-archived eprints.
and political science, finding OA impact advantage rates of 25 to 250% (Antelman,2004; Eysenbach, 2006; Hajjem, Harnad, & Gingras, 2005b; Kurtz et al., 2005a;Lawrence, 2001), with an average OA advantage of 93.2% in psychology (Hajjemet al., 2005a).
Aside from the effects of OA on citations by other scholars, some OA advocates have argued that OA will increase the spread of scientific research to professionalpractitioners and to the general population. No studies have empirically tested thispossibility. At face value, the theory that greater accessibility of published articleswill lead to greater readership and usage is compelling; however, a growing body ofdissemination research has documented that passive diffusion of new research islargely ineffective and unlikely to result in behavior change (Gotham, 2004;Grimshaw et al., 2001; Kerner, Rimer, & Emmons, 2005, Stirman, Crits-Christoph,& DeRubeis, 2004). Furthermore, there are methodological difficulties in measuringthe impact of particular articles on the general population; the traditional citationmeasure is not appropriate because nonresearchers do not write research articles.
Many researchers have proposed usage rates (Web hits or download rates) as analternative impact metric, the ‘‘Reading Factor’’ (Bollen, Van de Sompel, Smith, &Luce, 2005). Studies of the correlations between an article’s (citation-based) impactfactor and (download-based) reading factor have yielded mixed results—frominsignificant to strong correlations—depending on the discipline, journals, andtimeframe examined (Darmoni et al., 2000; Kurtz et al., 2000, 2005a, 2005b; Moed,2005). Therefore, it should not be assumed that increasing accessibility will have thesame effect on readership as on citation counts. To our knowledge, no studies havespecifically examined correlations between impact factor and reading factor inpsychology journals.
The present study aims to measure the influence of article accessibility on the diffusion of scholarly research towards mental health professionals. Theterm diffusion indicates a relatively passive process by which new knowledge iscommunicated through certain channels over time among the members of a socialsystem (Rogers, 1995), in contrast to the term dissemination, which denotesmore active, intensive efforts (Ellis et al., 2005). We measured diffusion impact intwo ways: (A) the proportion of participants accessing and reading a specificresearch article and (B) the degree to which participant responses to hypotheticaltreatment questions on the same topic as the article corresponded with thearguments advanced in that article. Rather than trying to compare equivalentOA and non-OA articles in the same journal, we experimentally manipulatedthe accessibility of a single article. Participants received a normal citation, a linkedcitation that facilitated finding the article on the Internet, a free access citationthat ensured participants could access the article without paying a fee, or a nocitation control. We predicted that greater levels of access would lead to higherreading rates and thus more treatment recommendations consistent with the findingsof the article.
We use the term free access instead of open access because OA removes both price barriers and permission barriers; an OA article must be permanently available toeveryone on the Internet without most usage restrictions while the article in thepresent study was available only for a limited time to a subset of the participants.
We requested that participants find and read the article, but asked them to continue with the study even if they were not able to find and read it. In this way, wehoped to simulate the real-world scenario in which a professional might see a citationthat he or she wanted to read, but was not required to read.
Journal of Clinical Psychology, July 2008 The study consisted of two sessions, 1 week apart. In Part 1, participants were givena normal citation, a linked citation, a free access citation, or no citation, and wereasked to find and read the article within a week. After 7 days, participants weree-mailed a link and asked to return for the second session of the study, in which theyread a clinical vignette and answered some related questions.
Three hundred participants were recruited through online classifieds and researchlistings2 over a period of 5 months, responding to an ad for ‘‘Research on Treatmentof Adolescent Substance Abuse’’ that involved reading a research article andanswering some questions about a clinical vignette. A gift-certificate lottery wasoffered as compensation; entry into this lottery was guaranteed regardless of whetherthe study was completed. Participants were self-identified mental health practitionersor professionals-in-training at least 18 years old who were not currently employed asmental-health researchers. A total of 194 participants did not return for the secondsession of the study. This rather high dropout rate can be attributed to the facts thatsome Session 2 invitations were probably filtered into junk-mail folders and thusnever read, some participants did not provide a valid e-mail address, and/or manyparticipants may have been busy or uninterested when they received the Session 2invitation. Dropout rates were not related to a difficulty in finding or reading thearticle, as no significant differences were found between groups on completion rate(see condition differences later in this article). Data from noncompleters weredropped from subsequent analyses, with a final sample size of 106. Participants listedtheir primary profession as organizational/industrial psychologist (1%), mentalhealth nurse (2%), mental hospital staff (5%), psychiatrist (0%), social worker(18%), therapist/counselor (30%), professional-in-training (33%), and other mentalhealth profession (11%). Their highest level of educational attainment was somehigh school (1%), some college (13%), 2-year degree (3%), 4-year degree (29%),some graduate school (18%), and graduate or professional degree (36%). Theiraverage age was 32.5 (SD 5 10.0, mdn 5 29), and years of professional experiencewas 5.5 (SD 5 6.1, mdn 5 3). Seventy-seven percent were female, and 79% wereCaucasian. Participants lived in the United States (94%), in Canada (4%), and inother countries (2%).
The study was conducted entirely on the Internet through psychologystudies.org, acustom Web site. Although the study was entirely automated, participants wereprovided with a contact e-mail address and phone number in case they hadquestions.
The research article used in the study was ‘‘Linking Session Focus to Treatment Outcome in Evidence-Based Treatments for Adolescent Substance Abuse’’ (Hogue,Liddle, Dauber, & Samuolis, 2004). This article analyzed psychotherapy sessions 2http://www.craigslist.org/, http://psych.hanover.edu/research/exponnet.html, and three others. Althoughno records were kept on referral sources, anecdotal evidence from correspondence with participants hassuggested that the majority came from craigslist.org with 51 inner city, substance-abusing adolescents. Its primary finding was that familyfocus (but not adolescent focus) predicted posttreatment improvement. We chose touse this article because it was available on the Internet, was characterized by a highlevel of scholarship, and had implications for evidence-based practice potentiallyapplicable by mental health professionals from all theoretical orientations.
A vignette describing a hypothetical adolescent with substance abuse issues (a situation relevant to the article) was developed (see Appendix A). Eleven possiblefoci for therapy were described, 3 of which were supported by the findings of thearticle, namely: ‘‘Mike’s relationship with his mother,’’ ‘‘Mike’s relationship with hisfather,’’ and ‘‘Core relational themes, such as trust, respect and independence.’’ In the first session, participants read the consent form, filled out demographicinformation, and entered their e-mail addresses. They were then randomly assignedto one of four conditions.
1. Normal citation: Participants were asked to read the Hogue et al. (2004) article, and were given a standard American Psychological Association (APA) stylecitation, with the additional instruction: ‘‘Please read the article at your leisure,sometime in the next week. Important: You will receive an email with instructionsto finish the study in one week. Please continue with the study at that time even ifyou are not able to read the article’’ (bold and italics as in the original).
2. Linked citation: Same as Condition 1, except that the citation was linked directly to the online copy of this article in the PsycARTICLES database, available for afee of $11.95 (or free for registered members).
3. Free access citation: Same as Condition 1, except that the citation was linked to a freely available copy of the article.
4. No citation: Participants received the instruction: ‘‘Part one of the study is complete. You will receive an automated email in 1 week with instructions tofinish the study.’’ The second session took place 1 week later, thus giving participants adequate time to find and read the article. Participants were sent an e-mail with a link to the secondsession, with explicit instructions to continue even if they had not read the article. Inthis session, they were asked to imagine they were going to provide therapy to a 16-year-old boy named Mike who was brought in for treatment by his mother. Theythen read a brief, fictional vignette that described Mike’s marijuana use, familyproblems, social life, and academic problems (see Appendix A). After this, they readthese instructions: Following is a list of 11 possible topics for therapy with Mike. Please lookat the whole list, and decide which topics you think would be mostimportant to focus on. Then, please type a number, 1–11, in the box nextto each item to indicate its rank—where ‘‘1’’ is the item you believe to bemost important, and ‘‘11’’ the least important.
Each item represented an adolescent topic or a family topic, as outlined in Hogue et al. (2004). Next, participants were asked to indicate how important they thought itwas to involve Mike’s mother as a participant in therapy, on a scale from 1 (not atall), to 7 (extremely). This directly paraphrased one of the suggestions in the article.
Journal of Clinical Psychology, July 2008 Next, participants were asked whether they read the Hogue et al. (2004) article. If they read it, they were asked how they found it and whether they thought itinfluenced their responses to the clinical vignette. If they did not read the article, theywere asked why not. Finally, participants were asked whether they believed thecurrent research publication system adequately makes research articles available toworking mental health professionals and, if not, how the system could be improved(free response).
As mentioned earlier, roughly 65% of the participants who completed Session 1 didnot return for Session 2. A chi-square test comparing the completion rate of the fourconditions found no significant differences (p4.5), indicating that experimentalcondition did not influence retention.
We compared the demographic characteristics of the four conditions with ANOVAs and chi-square tests, and found no significant differences across conditions in age, sex,education, years of experience, ethnicity, nationality, theoretical orientation, and beliefthat science is important. However, the professional distribution was not equivalent,w2(12, N 5 106) 5 25.71, po.05. The free citation condition had more mental hospitalstaff than did the other conditions, and the no-citation condition had moreparticipants with a profession of ‘‘other mental health field.’’ In the no-citation control condition, in which participants were not given the citationfor the Hogue et al. (2004) article nor were asked to read it, none of them reportedreading it. In the normal citation condition, 19% of the participants reported readingthe article. In the linked citation condition, 27% reported reading the article. In thefree access condition, 44% reported reading the article (see Figure 1). We comparedreading rates in the four conditions with an omnibus chi-square and found thatreading rates were affected by condition, w2(3, N 5 106) 5 15.66, po.001. Focused (1df) planned comparisons between conditions revealed that participants given thenormal citation were more likely to read the article than those not given a citation, Study 1: Effect of citation type on reported reading rates. Error bars represent one standard w2(1, n 5 55) 5 5.70, po.05. As predicted, participants given the free access citationwere more likely to read the article than those given a normal citation, w2(1,n 5 52) 5 3.96, po.05, and were marginally more likely to read it than were those inthe normal and linked conditions combined, w2(1, n 5 78) 5 3.73, p 5 .054). Nodifferences were found when directly comparing the linked and normal conditions,w2(1, n 5 53) 5 0.53, p 5 n.s. or the linked and free access conditions, w2(1,n 5 51) 5 1.63, p 5 n.s.
Participants who read the article reported that it influenced their responses to the clinical vignette somewhat, rating the influence a 3.1 (SD 5 1.4) on a scale from 1(not at all) to 5 (very much).
Participants indicated what to focus on in therapy with rank order (1–11) responses,so the Kruskal–Wallis nonparametric analysis of variance was used to compare thefour conditions on all 11 items. No differences were found between conditions(ps4.1). Collapsing across conditions, each of the three therapeutic foci supportedby the article were given a median rank of 5. Responses to the question aboutwhether to include the mother ‘‘as a participant in therapy and a target for change’’were compared with an ANOVA and also were not found to vary significantly acrossconditions, F(3, 97) 5 0.79, p 5 n.s., with an overall mean of 5.2 (SD 5 1.3) on the 7-point scale.
We subsequently collapsed the four conditions and compared the responses of participants who reported reading the article with those who did not. Each of the 11response items was compared with a two-tailed Mann–Whitney test, which revealeda significant difference: Participants who read the article focused more on theadolescent’s school work and grades (p 5 .02, uncorrected). Note that because articlereading was not directly manipulated experimentally, this result could be caused byself-selection rather than by article influence (i.e., participants who were more likelyto find and read a research article also were more likely to focus on academics).
Furthermore, the p value of this unplanned comparison may be misleading due tofamily-wise error.
Explanations of Barriers to Article Readership Participants who were given a citation but did not report reading the article indicatedthe main reason why they did not (see Table 1). We compared the reasons given inthese three conditions with an omnibus chi-square and found that the reasons givenwere affected by condition, w2(6, n 5 49) 5 27.455, po.001. Focused comparisonsbetween pairs of conditions all revealed significant differences (pso.01), indicatingthat while participants in the normal citation condition reported not being able tofind the article or not having time, participants in the linked condition generallyreported that the article was too expensive, and participants in the free accesscondition reported that they did not have time.
Fifty-two percent of all participants believed that the current research publication system adequately makes research articles available to working mental healthprofessionals. Those who responded negatively gave open-ended suggestions as tohow the current system could be improved, with 71% of these statements advocatingreducing or eliminating access charges.
Journal of Clinical Psychology, July 2008 Table 1Reasons Why Participants Reportedly Did Not Read the Article in Study 1, in the ThreeConditions in Which Participants Received a Citation I didn’t think it would be interesting.
As predicted, participants given the free access citation were more likely to reportreading the article. In fact, they were more than twice as likely to report reading thearticle as were those given a normal citation, thus demonstrating the potentially largeimpact that reducing access barriers can have on reading rates by mental healthprofessionals. Contrary to predictions, there was no difference in reading ratesbetween the normal citation and linked citation conditions—indicating that the timeor difficulty involved in locating an article from its citation may not have asignificant impact on reading rates.
Contrary to predictions, article accessibility had no effect on participants’ clinical vignette responses. Participants in all citation conditions gave the same recommen-dations for treatment focus as did a control group who received no citation at all.
Post-hoc comparisons of those who read the article and those who did not(irrespective of condition) revealed only one significant difference: Those who foundand read the article placed more importance on academics. This was probably a self-selected group bias because academic focus was not recommended by the article.
These results are consistent with the findings of previous dissemination studiesshowing that passive diffusion is unlikely to produce behavior change and that Web-based diffusion specifically is ineffective in this regard (Backer, 2000; Buller, Buller,& Kave, 2005; Lewis et al., 2005). The fact that participants reported that the articleinfluenced their answers, in contrast to the objective evidence that it did not,indicates that participants may not have fully understood or been able to effectivelyapply the findings of the article. Another explanation (suggested by a reviewer of thisarticle) as to why groups did not differ on vignette responses is that participantsmight, in deciding what to focus on in treatment, be more influenced by their overalltheoretical orientation than by information from a specific study.
One shortcoming of Study 1 is that it did not directly test participants’ comprehension and retention of the article. Participants were not asked to followthe conclusions of the article in their responses to the clinical vignette, so we do notknow whether the lack of observed effect resulted from (a) lack of comprehension ofthis fairly technical study; (b) memory decay between the time the article was readand the time of the clinical vignette responses up to a week later; or (c) adisagreement with the methods, conclusions, or applicability of the article withregard to the clinical vignette.
A second serious shortcoming of this study was the reliance on self-report for measuring article-reading rates. As participants in the free access condition hadfewer excuses for not reading the article, they may have felt a greater need to reportreading the article even if they had not in fact done so.
A third shortcoming was the high dropout rate. Although dropout was not affected by experimental condition, we could only speculate on the reasonsparticipants failed to return for the second session, so our final sample may havebeen biased and not reflective of the general population of mental healthprofessionals and professionals-in-training.
In an effort to address these concerns and assess the replicability of the results, we ran a second study, with some important changes. Participants were able toparticipate in the second study only after providing and confirming a valid e-mailaddress, thus ensuring that they would eventually receive the Session 2 invitation.
After asking participants whether they read the article, we also gave them a multiple-choice test assessing comprehension/retention of the reading, thus providing anobjective measure of diffusion.
We selected an article that we believed would be relevant to more participants and also more easily understood and applied, to assess whether this would lead to adifference between conditions on responses to the clinical vignette. Furthermore, weasked participants directly whether they planned to implement the findings of thearticle in their own practice.
Participants. Participants were recruited in the same way as for Study 1, but this time responding to an ad for ‘‘Research on Preventing Therapy Dropout.’’ A gift-certificate lottery was again offered as compensation.
Participants were eligible to complete the study only if they provided a valid e-mail address, reported being over 18 years of age, reported being a mental healthprofessional or a practitioner in training, and reported not being a mental healthresearcher. A total of 173 participants fit these criteria and began the study. Of these,2 attempted to participate in the study more than once (identified by a repeated e-mail address or a repeated IP address and demographic data). In these cases, the firstset of data for each participant was retained and analyzed while data from theirrepeated participation was excluded.
Fifty-eight participants dropped out after the first session, bringing the total number of valid completers to 115. While the completion rate in the no citationcondition (88%) was significantly higher than the completion rates in the freecitation (70%), linked citation (53%), and normal citation conditions (64%)(pso.05), there were no significant differences between the three conditions in whichparticipants were given a citation. All further analyses will concern only the 115participants who completed the study.
Participants’ average age was 33.7 (SD 5 10.3, mdn 5 32), and average years of professional experience was 7.8 (SD 5 8.1, mdn 5 5). Eighty-nine percent werefemale, and 80% were Caucasian. Ninety-two percent of the participants lived in theUnited States, 4% in Canada, and 4% in other countries. Participants listed theirhighest completed level of education as high school (1%), some college (7%), 2-yearcollege degree (2%), 4-year college degree (10%), some graduate school (24%), andgraduate or professional degree (57%). They listed their primary profession astherapist/counselor (33%), social worker (17%), mental health nurse (3%), mental Journal of Clinical Psychology, July 2008 hospital staff (2%), psychiatrist (0%), professional-in-training (37%), and othermental health profession (9%). The primary theoretical orientation was behavioral(11%), cognitive-behavioral (39%), eclectic (25%), humanistic/existential (4%),interpersonal (9%), psychodynamic (5%), other (3%), and none (3%). While 90% ofparticipants indicated they had given therapy or counseling at some point, only 64%said they were currently doing so. On average, those who had given counselingestimated that 23% of their clients dropped out of therapy prematurely. Of thosewho had given counseling, 56% believed that premature dropout was a problem intheir practice.
As before, the study was conducted entirely online, using psychologystudies.org, acustom Web site. The article used was ‘‘Preventing Therapy Dropout in the RealWorld: The Clinical Utility of Videotape Preparation and Client Estimate ofTreatment Duration,’’ by Reis and Brown (2006). The article was chosen because ithad clear recommendations for improving practice and was likely more relevant andaccessible to participants than was the article used in Study 1.
Knowledge test. A knowledge test about the reading was developed by generating seven multiple-choice questions with four response options for each question,and pilot testing them with a convenience sample of 10 graduate students—5 ofwhom read the article before taking the test. Five of the seven items were found toreliably distinguish between those who had read the article and those who had not,so these items were retained for the final version of the test used in the main study.
The test questions and answers were then sent to Brenda Reis, the first authorof the Reis and Brown (2006) article. She confirmed the validity of the questionsand answers, and made some minor suggestions for improvement, which wereimplemented.
Vignette and recommendations. A vignette describing a hypothetical problem involving a clinic with a high dropout rate (a situation relevant to the article) wasdeveloped (see Appendix B). Twelve possible solutions were described, 2 of whichwere mentioned in the article. Only 1 of the solutions was empirically supported bythe study described in the article: namely, a 12-min videotaped instruction (Tell ItLike It Is; Acosta, Yamamoto, Evans, & Skilbeck, 1983) prior to beginning therapy,introducing patients to behaviors considered desirable in psychotherapy. Thevignette and possible solutions were reviewed by Brenda Reis as well, who gave herapproval.
The procedure was the same as that for Study 1, with the following additions: Beforebeginning the study, participants were required to enter their e-mail address. Aconfirmation e-mail was then sent to that address, with a link participants had toclick to participate in the study, thus ensuring we had a valid e-mail address for eachparticipant. After reading the clinical vignette in Session 2 and giving their responses,participants were asked whether they had ever treated patients in psychotherapy, andif so, whether they had found therapy dropout to be a problem in their practice. Ifthey reported having conducted therapy, they also were asked which methods forpreventing therapy dropout they had tried in the past and which methods theyintended to try in the future. Finally, participants had to complete a test assessing their knowledge of the cited article. They were asked to complete the test as best theycould even if they had not read the article. They also were asked not to read thearticle as they took the test, even if the article was easily available to them.
A chi-square test revealed that article accessibility had a significant effect on reportedreading rates, w2(3, N 5 115) 5 32.31, po.001. As predicted, pairwise comparisonsconfirmed that a higher proportion of participants given the free citation (n 5 30)reported reading the article (70%) than those given the linked citation (n 5 28; 14%),po.001, the normal citation (n 5 29; 45%), p 5 .04, or no citation (n 5 28; 7%),po.001. Although those given the normal citation were more likely to report readingthe article than those given the linked citation or no citation (pso.01), there was nota significant difference between those given the linked citation and those given nocitation.
Knowledge tests were scored from 0 to 5, with 1 point for each correct answer. Aseach question contained four response choices, a person answering at random wouldscore an average of 1.25. An analysis of variance showed that article accessibility hada significant effect on knowledge test scores, F(3, 111) 5 7.78, po.001 (see Figure 2).
As predicted, participants given the free citation scored significantly higher on thereading test (M 5 2.80, SD 5 1.47) than those given the linked citation (M 5 1.54,SD 5 1.04), t(52) 5 3.80, po.001, the normal citation (M 5 1.86, SD 5 1.33),t(56) 5 2.57, p 5 .01, or no citation (M 5 1.46, SD 5 0.79), t(45) 5 4.34, po.001.
Comparing the free citation condition to each of the three other conditions, effectsizes (Cohen’s d) were large, ranging from 1.5 to 1.6. There were no significantdifferences in test scores between the linked, normal, and no citation conditions.
In all conditions, half of those who reported reading the article said they only skimmed it (rather than reading it thoroughly). Article reading was then coded as 0(didn’t read), 1 (skimmed), or 2 (read thoroughly) and found to be positivelycorrelated with reading test scores (r 5 .61, po.001), indicating that participants who Study 2: Article accessibility affects knowledge test scores. Error bars represent one standard Journal of Clinical Psychology, July 2008 reported reading the article and reading it more thoroughly were likely to do betteron the knowledge test.
Participants indicated which plans for reducing dropout they would recommendwith rank order (1–12) responses, so the Kruskal–Wallis nonparametric analysis ofvariance was used to compare the four conditions. Article accessibility had asignificant effect on level of support for using videotaped instruction to reducetherapy dropout, H(3, n 5 115) 5 14.52, p 5 .002 (see Figure 3). As predicted,participants given the free citation ranked videotaped instruction higher (mdn 5 2)than those given the linked citation (mdn 5 6.5), H(1, n 5 58) 5 9.59, p 5 .002, thenormal citation (mdn 5 6), H(1, n 5 59) 5 4.93, p 5 .03, or no citation (mdn 5 7),H(1, n 5 58) 5 12.31, po.001. There were no significant differences between thelinked, normal, and no citation conditions. Furthermore, there were no significantdifferences between conditions on level of support for any of the 11 other therapydropout reduction plans.
Collapsing across conditions, those who reported reading the article thoroughly ranked videotaped instruction significantly higher (mdn 5 1) than did those whoreported skimming the article (mdn 5 3.5) or not reading it (mdn 5 7), H(2,n 5 115) 5 27.37, po.001, all pairwise comparisons being significant at p 5 .02 orbetter.
Only 4% of participants reported having tried videotaped instruction. This meant videotaped instruction was the second least common intervention that participantsreported having tried (No participants reported having tried a finishing bonus.) Themost commonly attempted intervention to reduce therapy dropout was to ‘‘attemptto contact clients after they drop out and survey them to find out their reasons,’’which 46% of participants reported having tried.
A chi-square test showed that article accessibility had a significant effect on participants’ intentions to try videotaped instruction within the next 6 months, w2(3,n 5 115) 5 19.77, po.001. A greater proportion of participants given the free citationsaid they planned to try videotaped instruction (33%) than were those given thelinked citation (4%), w2(1, n 5 58) 5 8.35, p 5 .004, the normal citation (7%), w2(1,n 5 59) 5 6.36, p 5 .01, or no citation (0%), w2(1, n 5 58) 5 11.28, p 5 .001.
Study 2: Article accessibility affects support for videotaped instruction. Error bars represent the 95% confidence interval of the median.
Collapsing across conditions, participants were more likely to say they would tryvideotaped instruction if they reported reading the article (25%) than if they did not(4%), w2(1, n 5 115) 5 11.47, p 5 .001. There was not a significant difference betweenthose who reported reading the article thoroughly and those who reported skimmingit. Overall, the most popular intervention that participants said they would try in thenext 6 months was to ‘‘have each client & therapist mutually agree on a treatmentduration estimate during the intake session,’’ which was endorsed by 30% ofparticipants, and did not vary significantly by condition. Note that this interventionwas tested, but not supported, by Reis and Brown (2006).
There were no significant differences in sex, age, ethnicity, nationality, education,profession, or years of experience. Among participants who had given therapy atsome point, those who found therapy dropout to be a problem in their practice weremarginally more likely to report reading the article (n 5 57; 44%) than were thosewho did not think dropout was a problem (n 5 45; 27%), w2(1, n 5 102) 5 3.23,p 5 .07, and scored marginally higher (M 5 2.19 vs. M 5 1.76) on the knowledge test,t(98) 5 1.71, p 5 .09. They ranked videotaped instruction significantly higher(mdn 5 4 vs. mdn 5 7), H(1, n 5 102) 5 6.77, po.01, and were significantly morelikely to say they intended to try it themselves within the next 6 months (19 vs. 4%),w2(1, n 5 102) 5 4.99, p 5 .03.
Explanations of Barriers to Article Readership Those participants who reported having read the article indicated how they accessedit (see Table 2). Overwhelmingly, those participants who read the article found it onthe Internet and were able to access it for free. Only 1 participant reported paying forthe article. Those participants who did not read the article listed the main reason whythey did not (see Table 3). Broadly speaking, those who were not given a citationstated that they had never heard of the article, those given a normal citation statedthat they did not have time, those given a linked citation stated that it was tooexpensive, and those given a free citation stated that they did not have time.
Forty-five percent of the participants said the current research publication system adequately makes research articles available to working mental health professionals(This proportion did not vary by condition.) The other 55% gave free responses tosuggest how the current system could be better; 75% of these suggestions mentionedreducing the cost in one way or another.
In a replication of Study 1, participants given the free access citation weresignificantly more likely to report reading the article than were those in the otherthree conditions. Reading rates in the free access condition were roughly 2.4 timesthe average of those in the linked and normal citation conditions. Furthermore,participants given a free access citation also scored higher on a knowledge test thandid those in the other conditions, providing objective verification that they weremore likely to read and understand the article. This finding suggests that the resultsof Study 1 were not simply due to demand characteristics or other self-report biases.
Clearly, the article’s accessibility had a significant impact on mental healthprofessionals’ likelihood to read it.
Journal of Clinical Psychology, July 2008 Table 2How Participants Reportedly Found the Article in Study 2, in Each of the Four Conditions I found it on the Internet and was able to access I found it on the Internet but had to buy it.
Table 3Reasons Why Participants Reportedly Did Not Read the Article in Study 2, in Each of the FourConditions I didn’t think it would be interesting.
Note. Five participants did not give a reason.
Unlike Study 1, article accessibility had a significant effect on participants’ recommended responses to a relevant hypothetical scenario, and also influenced theirintentions to implement the findings of the article in their therapeutic practice in thenext 6 months. Why did we see this difference between the two studies? To be sure,research accessibility is necessary, but not sufficient, for influencing practice or evenpractice intentions. The article used in Study 2 was likely easier to understand, easierto apply, and relevant to more of the participants. In support of the hypothesis thatarticle relevance influenced diffusion, participants in Study 2 who believed therapydropout was a problem were significantly more likely to give interventionrecommendations and change their real-life practice in ways consistent with aresearch article on the same topic. Nevertheless, it must be remembered that there isno evidence whether participants’ stated intentions to change their therapeuticpractice actually translated into real change.
Unlike Study 1, participants given a normal citation were more likely to report reading the article than were those given a linked citation; however, this findingshould be taken with a grain of salt, as knowledge test scores did not differsignificantly between the two conditions. One explanation (suggested by a reviewer)for the difference in reported reading rates is that participants given the linkedcitation may have been less likely to search for free versions of the article (thinking it was only available at a price)—even if they could have accessed it through their usualsearch mechanisms, which might be free (e.g., access to databases through theirworkplaces). In support of this explanation, those given the linked citation were theonly group to report that cost was the main deterrent to reading the article (see Table3). Furthermore, only 1 linked citation participant reported finding the articlethrough a library or ‘‘other’’ means, compared with 5 normal citation participantswho reporting using these methods (see Table 2). This explanation, that perceivedcost may decrease search efforts, underscores the point that price is a barrier todiffusion. Dissemination efforts that come with a price tag may prove less effective,even if they are promoted more heavily, than dissemination efforts that are free.
Given that none of the results showed any advantage for the linked citation over the normal citation, it is clear that locating the target article was not a significantdiffusion barrier. Further supporting this notion was the data that 75% of thesuggestions for improvement to the research publication system concerned cost whilenone mentioned being unable to locate the article.
The dropout rate in Study 2 (34%) was much lower than that in Study 1 (65%), but still notable. It is likely that noncompleting participants were busy oruninterested when they received the Session 2 invitation. It also is possible thatsome of the noncompleters used an infrequently checked, secondary e-mail addressto sign up for the study, and so did not read the Session 2 invitation in time.
Participants in the no citation control condition had lower dropout rates than didthose participants who were asked to find and read a citation—even thoughparticipants given a citation were asked to complete the study even if they were notable to read the article. This may indicate that participants were self-conscious aboutreturning for the second session without fulfilling the request of the study or were putoff by the request to find and read the article; however, completion rates did not varysignificantly by citation type, so there is no evidence that the normal or linkedcitations were significantly more discouraging to participants than were the freeaccess citations. Future studies could maximize internal validity by running the studyin a single session, and requesting that participants find and read the articleimmediately and answer the questions right afterward. Additional dependentvariables could be added to this design, such as dropout rates and time spent findingand reading the article.
The aim of these analogue experiments was to determine the effect of articleaccessibility on reading rates by mental health professionals and on their treatmentrecommendations in a relevant context. The accessibility manipulation containedsome atypical features (A reference to one specific study was provided directly toparticipants, with the prospect of a 1-week follow-up inquiry related to it.), butoverall, we believe it had reasonably good external validity. Self-identified mentalhealth professionals completed the study from their home or office, had a week tofind and read the article in the course of their normal lives, and external incentiveswere not contingent upon their having read the article. Furthermore, as participantswere initially recruited for studies about ‘‘Treatment for Adolescent SubstanceAbuse’’ or ‘‘Preventing Therapy Dropout,’’ the sample was probably representativeof a population with some intrinsic interest in the subject matter.
As found in the first study and replicated in the second study, the cited article was read by roughly twice as many participants in the free access citation condition as Journal of Clinical Psychology, July 2008 those in the normal and linked citation conditions. This bolsters the positionof OA advocates, showing that OA may lead to greater research diffusion not just inthe research community but in the professional community as well. No advantageswere observed for participants given a linked citation over those given a normalcitation, indicating that locating articles may not be a significant barrier to researchdiffusion.
Changes in article accessibility did not consistently translate into altered treatment recommendations. In Study 1, neither article accessibility nor self-reported readingrates predicted meaningful changes in responses to a relevant scenario while in Study2 both did, a difference that may be attributed to the more straightforward, morerelevant content of the article used in Study 2. There was no evidence, however, as towhether participants’ attitudes and intentions in Study 2 translated into real-worldbehavior. According to Backer (2000), ‘‘the single, most common failure of pastdissemination strategies . . . was the assumption that getting information out wasenough to create change’’ (p. 364). Even among more active research disseminationefforts, no strong evidence yet exists to recommend any one strategy as effective in avariety of circumstances (Ellis et al., 2005).
Taken together, the data on readership and on treatment recommendations suggest that OA could increase consumption of treatment research, which is anecessary, but not sufficient, condition for that research to influence clinical practice.
Do particular types of articles influence practice more than do others? What wouldmaximize reading versus skimming of the articles? Future research needs to focus notonly on easing access to relevant research but also on understanding factors (e.g.,access to training in the approach, acceptability to consumers) associated withimplementation of the methods supported by research.
A more immediate practical implication of the present study is that scholars wishing to maximize the diffusion of their research among the professionalcommunity should deposit eprints of their work in OA archives. There are nocopyright or other legal barriers to this OA strategy, with 91% of research journals(including all APA and Wiley journals) already giving their explicit green light toauthors self-archiving of pre- or postprints (Eprints, 2008). One hundred percent OAis a reachable goal.
Complete Text of the Clinical Vignette Used in Study 1 Mike is brought in for his first visit by his mother. He has generally been a goodstudent in school, but his grades have fallen recently. He is active in the soccer club,and has a large group of friends. Even so, Mike reports chronic feelings of sadnessand loneliness.
He smokes marijuana almost every day, often with friends but sometimes alone.
He was arrested recently for marijuana possession, but charges were dropped. Mike’sparents divorced when he was 4 years old, and since then Mike sees his father onlyon weekends.
Mike has no brothers or sisters, but has a close friend named Luke whom he has known since early childhood. Mike considers Luke to be his closest attachment. 3weeks ago, Luke was convicted of drug dealing, and is now serving his sentence injail—this precipitated Mike’s coming to therapy.
The complete text of the clinical vignette used in Study 2 Imagine you are the supervisor of a mental health clinic in a large healthmaintenance organization. There are around 20 therapists under your supervision,coming from a variety of theoretical backgrounds: psychodynamic, cognitivebehavioral, family systems, and eclectic. The clients are mostly middle class, andthe primary problems they report are anxiety, depressed mood, parenting issues,marital problems, and relationship problems.
The clinic has been experiencing somewhat high client dropout rates for the last few years—according to therapist reports, around 60% of clients terminate therapyprematurely. Your boss is concerned about the situation, and has requested that youinstitute changes in the clinic to improve client retention, while minimizing the costsof the changes.
Acosta, F.X., Yamamoto, J., Evans, L.A., & Skilbeck, W.M. (1983). Preparing low-income Hispanic, Black, and White patients for psychotherapy. Evaluation of a new orientationprogram. Journal of Clinical Psychology, 39, 872–877.
Antelman, K. (2004). Do open access articles have a greater research impact? College & Research Libraries News, 65, 372–382.
Backer, T.E. (2000). The failure of success: Challenges of disseminating effective substance abuse prevention programs. Journal of Community Psychology, 28, 363–373.
Bollen, J., Van de Sompel, H., Smith, J., & Luce, R. (2005). Toward alternative metrics of journal impact: A comparison of download and citation. Information Processing andManagement, 41, 1419–1440.
Buller, D.B., Buller, M.K., & Kave, I. (2005). Web-based strategies to disseminate a sun safety curriculum to public elementary schools and state-licensed child care facilities. HealthPsychology, 24, 470–476.
Darmoni, S.J., Roussel, F., Benichou, J., Faure, G.C., Thirion, B., & Pinhas, N. (2000).
Reading factor as a credible alternative to impact factor: A preliminary study. Technologyand Health Care, 8, 174–175.
Ellis, E., Ciliska, D., Sussman, J., Robinson, P., Armour, T., Brouwers, M., et al. (2005). A systematic review of studies evaluating diffusion and dissemination of selected cancercontrol interventions. Health Psychology, 24, 488–500.
Eprints. (2008). Journal policies—Summary statistics so far. Accessed February 27, 2008, from Eysenbach, G. (2006). Citation advantage of open access articles. PLoS Biology, 4, e157.
Gotham, H.J. (2004). Diffusion of mental health and substance abuse treatments: Development, dissemination, and implementation. Clinical Psychology: Science andPractice, 11, 161–176.
Grimshaw, J.M., Shirran, L., Thomas, R., Mowatt, G., Fraser, C., Bero, L., et al. (2001).
Changing provider behavior: An overview of systematic reviews of interventions. MedicalCare, 39(Suppl. 2), II2–II45.
Hajjem, C., Brody, T., Gingras, Y., Harnad, S., Hitchcock, S., Oppenheim, C., Stamerjohanns, H., & Vallie`res, F. (2005a). E´tude de la variation de l’impact de citationsdes articles en acce`s libre. [Study of the variation in citation impact of open access articles.]Retrieved http://www.crsc.uqam.ca/lab/chawki/graphes/ Journal of Clinical Psychology, July 2008 Hajjem, C., Harnad, S., & Gingras, Y. (2005b). Ten-year cross-disciplinary comparison of the growth of open access and how it increases research citation impact. IEEE DataEngineering Bulletin, 28, 39–47.
Harnad, S. (2003). Open access to peer-reviewed research through author/institution self- archiving: Maximizing research impact by maximizing online access. Journal ofPostgraduate Medicine, 49, 337–342.
Harnad, S., & Brody, T. (2004, June). Comparing the impact of open access (OA) vs. non-OA articles in the same journals. D-Lib Magazine, 10(6). Available at: http://www.dlib.org/dlib/june04/harnad/06harnad.html Hogue, A., Liddle, H.A., Dauber, S., & Samuolis, J. (2004). Linking session focus to treatment outcome in evidence-based treatments for adolescent substance abuse. Psychotherapy:Theory, Research, Practice, Training, 41, 83–96.
Kerner, J., Rimer, B., & Emmons, K. (2005). Dissemination research and research dissemination: How can we close the gap? Health Psychology, 24, 443–446.
Kurtz, M.J., Eichhorn, G., Accomazzi, A., Grant, C.S., Demleitnerm, M., & Murray, S.S.
(2005a). The bibliometric properties of article readership information. Journal of theAmerican Society for Information Science and Technology, 56, 111–128.
Kurtz, M.J., Eichhorn, G., Accomazzi, A., Grant, C.S., Demleitner, M., & Murray, S.S.
(2005b). Worldwide use and impact of the NASA Astrophysics Data System Digital Library.
Journal of the American Society for Information Science and Technology, 56, 36–45.
Kurtz, M.J., Eichhorn, G., Accomazzi, A., Grant, C.S., Murray, S.S., & Watson, J.M. (2000).
The NASA astrophysics data system: Overview. Astronomy and Astrophysics SupplementSeries, 143, 41–59.
Lawrence, S. (2001). Free online availability substantially increases a paper’s impact. Nature, 411, 521. Available at: http://www.nature.com/nature/debates/e-access/Articles/lawrence.html Lewis, E., Mayer, J.A., Slymen, D., Belch, G., Engelber, M., Walker, K., et al. (2005).
Disseminating a sun safety program to zoological parks: The effects of tailing. HealthPsychology, 24, 456–462.
Mattaini, M.A. (2004). Open access journals as a justice issue [Editorial]. Behavior and Social Miller, W.R., Andrews, N.R., Wilbourne, P., & Bennett, M.E. (1998). A wealth of alternatives: Effective treatments for alcohol problems. In W.R. Miller & N. Heather(Eds.), Treating addictive behaviors (2nd ed., pp. 203–216). New York: Plenum Press.
Moed, H.F. (2005). Statistical relationships between downloads and citations at the level of individual documents within a single journal. Journal of the American Society forInformation Science and Technology, 56, 1088–1097.
Morrison, K.H., Bradley, R., & Westen, D. (2003). The external validity of controlled clinical trials of psychotherapy for depression and anxiety: A naturalistic study. Psychology andPsychotherapy: Theory, Research and Practice, 76, 109–132.
Nathan, P. (2000). The Boulder model: A dream deferred—or lost? American Psychologist, National Institutes of Health. (2005). Policy on enhancing public access to archived publications resulting from NIH-funded research. Federal Register, 70(26), 6891–6900.
Available at: http://grants.nih.gov/grants/guide/notice-files/NOT-OD-08-033.html Reis, B.F., & Brown, L.G. (2006). Preventing therapy dropout in the real world: The clinical utility of videotape preparation and client estimate of treatment duration. ProfessionalPsychology: Research and Practice, 37, 311–316.
Rogers, E.M. (1995). Diffusion of innovations (4th ed.). New York: Free Press.
Stirman, S.W., Crits-Christoph, P., & DeRubeis, R.J. (2004). Achieving successful dissemination of empirically supported psychotherapies: A synthesis of disseminationtheory. Clinical Psychology: Science and Practice, 11, 343–359.
Thomson Corporation, McVeigh, M.E. (2004). Open access journals in the ISI citation databases: Analysis of impact factors and citation patterns. White Paper. Available at:http://www.thomsonscientific.com/media/presentrep/essayspdf/openaccesscitations2.pdf Westen, D., Novotny, C.M., & Thompson-Brenner, H. (2004). The empirical status of empirically supported psychotherapies: Assumptions, findings, and reporting in controlledclinical trials. Psychological Bulletin, 130, 631–663.
Willinsky, J. (2003). The nine flavours of open access scholarly publishing. Journal of Postgraduate Medicine, 49, 263–267.

Source: http://davidhardisty.info/downloads/2008-DiffusionOfTreatmentResearch-HardistyHaaga.pdf