Systematic reviews —the most comprehensive type of literature review—should be taken into account before a clinical trial or a narrative review on a topic is undertaken. The objective of this study was to describe the use of systematic reviews in clinical trials and narrative reviews in dermatology.
Material and methodsThis was a descriptive cross-sectional study. We selected randomized clinical trials and narrative reviews from the dermatological clinical research journals identified as most important (according to impact factor) and from Actas Dermosifiliográficas, and studied the bibliographies to ascertain whether the authors made reference to existing systematic reviews and Cochrane reviews.
ResultsOf the 72 clinical trials for which a systematic review was available, 24 (33.3%) cited at least 1 review; reference was made to relevant Cochrane reviews in 15.6% of cases and to non-Cochrane reviews in 32%. In the case of the 24 narrative reviews for which a review was available, 10 (41.7%) cited at least 1 review; Cochrane reviews were cited in 20% and non-Cochrane reviews in 35.3%.In the case of Actas Dermosifiliográficas, very few clinical trials were found and the findings for narrative review articles were similar to those observed for the other journals.
ConclusionsSystematic reviews are not often taken into account by the authors of clinical trials and narrative reviews and this may lead to redundant studies and publications. Authors appear to use Cochrane reviews even less than non-Cochrane reviews and are therefore ignoring one of the main sources of available evidence.
Las revisiones sistemáticas son la forma de revisión más exhaustiva y deberían ser consideradas antes de realizar un ensayo clínico o revisión sobre un tema. El objetivo de este estudio es describir la utilización de revisiones sistemáticas en los ensayos clínicos y revisiones narrativas publicadas en dermatología.
Material y métodosSe diseñó un estudio descriptivo transversal. Se seleccionaron ensayos clínicos aleatorizados y revisiones narrativas de las revistas más relevantes (por factor de impacto) de investigación clínica de dermatología y de Actas Dermosifiliográficas y se evaluó si hacían referencia a revisiones sistemáticas y Cochrane (en caso de existir) en la bibliografía.
ResultadosEn el grupo de ensayos clínicos se hacía referencia a alguna de las revisiones sistemáticas existentes en el 33,3% de los artículos (15,6% de las revisiones Cochrane que existían y 32,2% de las no Cochrane sobre el tema). En el grupo de revisiones narrativas alguna de las revisiones sistemáticas existentes eran referenciadas en el 41,7% de los trabajos (20% Cochrane y 35,3% no Cochrane). En Actas Dermosifiliográficas existían muy pocos ensayos clínicos publicados; las revisiones narrativas reproducían lo que se observaba en el resto de revistas.
ConclusionesLas revisiones sistemáticas son poco tenidas en cuenta en la realización de ensayos clínicos y revisiones narrativas, lo que puede llevar a estudios y publicaciones redundantes. Además las revisiones Cochrane parecen ser incluso menos empleadas, obviando así los autores una de las principales fuentes de evidencia existentes.
An accurate account of existing knowledge is essential when planning a new study or compiling information that is already known. Authors or researchers who embark on a new study or review of a topic without searching for and analyzing pertinent systematic reviews run the risk of looking for an answer that is already known. Systematic reviews are the most effective means of identifying and appraising available evidence.1 Traditional or narrative reviews have been found to reach conclusions of little value, or even erroneous conclusions based on personal opinions and experiences, while systematic reviews produce results based on the best available evidence.2
The aim of this study was to analyze the percentage of clinical trial reports and narrative reviews in dermatology that use information from systematic reviews on the subject under investigation.
Material and MethodsWe conducted a cross-sectional descriptive study to investigate whether clinical trial reports and narrative reviews in the field of dermatology cite relevant Cochrane and non-Cochrane systematic reviews in the bibliography. We included randomized controlled trial (RCT) reports addressing treatment interventions published in a selection of dermatology journals in 2010 and 2011 and narrative reviews of treatments published in the same group of journals in 2011.
The journals chosen were the British Journal of Dermatology, the Journal of the American Academy of Dermatology, and Archives of Dermatology. The selection was determined by impact factor and a predominant focus on clinical research. Because of its relevance in Spain, we also performed an identical analysis of Actas Dermo-Sifiliográficas.
We collected all publications from the study period that were indexed in PubMed as “clinical trial” or “review” (narrative, i.e., not systematic). The articles retrieved were distributed among the authors, who recorded the following information: type of article, date of publication, citation of a Cochrane or non-Cochrane systematic review in the bibliography, existence in the literature of a Cochrane or non-Cochrane systematic review on the topic under investigation and the date of its publication. We used the Cochrane Library3 (http://www.thecochranelibrary.com) to search for Cochrane systematic reviews and the Centre for Reviews and Dissemination4 (part of the UK National Institute for Health Research) to search for non-Cochrane systematic reviews. Systematic reviews published at least 1 year before the publication of the article being analyzed were considered citable.
In each group, we excluded articles that were not RCT reports or narrative reviews, RCTs and narrative reviews that did not address a treatment, and duplicate publications.
We calculated the percentage of systematic reviews (Cochrane and non-Cochrane) cited in each group of articles, and also calculated separate figures for each type of review.
Before choosing the target journals and the years to be analyzed, we calculated the sample size needed to detect a margin of error of up to 25% with a statistical power of 0.9 and an alpha level of 0.5; the rate of citation of systematic reviews was estimated at 15%. The data used to calculate the number of years and journals to include in the study were taken from a pilot sample of dermatology journals.
The SPSS software package, version 16.0 (IBM), was used for data analysis. The flow of articles throughout the process was charted.
ResultsRandomized Clinical TrialsThe flowchart for the inclusion/exclusion of articles reporting RCTs is shown in Fig. 1.
Not counting data for Actas Dermo-sifiliográficas, we retrieved 137 articles, of which 111 (81%; 95% CI, 73%-87%) met the inclusion criteria for the study. Pertinent systematic reviews existed for 72 RCT publications (64.9%; 95% CI, 55.2%-73.7%) and were cited in 24 of these (33.3%; 95% CI, 22.7%-45.4%).
Cochrane reviews existed for 32 publications (28.8%; 95% CI, 20.6%-38.2%) and were cited in 5 cases (15.6%; 95% CI, 5.2%-32.8%). Non-Cochrane systematic reviews existed for 59 articles (53.2%; 95% CI, 42.4%-62.7%) and were cited in 19 of these (32.2%; 95% CI, 20.6%-45.6%).
Three articles were retrieved from PubMed for Actas Dermo-Sifiliográficas and 2 met the inclusion criteria. There were no systematic reviews on the topics addressed by these articles.
Narrative ReviewsThe flowchart showing the selection of narrative reviews is shown in Fig. 2.
Our search retrieved 123 narrative reviews, not counting articles from Actas Dermo-Sifiliográficas. Of these, 40 (32.5%; 95% CI, 24.1%-41.6%) met the inclusion criteria. Pertinent systematic reviews existed for 24 of the reviews (60%; 95% CI, 43.3%-75.1%) and were cited in 10 of these (41.7%; 95% CI, 22.1%-63.4%).
Cochrane reviews existed for 20 publications (50%; 95% CI, 33.8%-63.2%) and were cited in 4 (20%; 95% CI, 5.7%-43.7%). Non-Cochrane systematic reviews existed for 17 articles (42.5%; 95% CI, 27%-59.1%) and were cited in 6 cases (35.3%; 95% CI, 14.2%-61.7%).
Thirty-four narrative reviews were retrieved in Actas Dermo-Sifiliográicas, and of these 9 (26.5%; 95% CI, 12.9%-44.4%) fulfilled the study inclusion criteria. Citable systematic reviews were identified for 4 articles (44.4%; 95% CI, 13.7%-78.8%). These reviews were cited in 2 cases (50%; 95% CI, 7.6%-93.2%). Pertinent Cochrane reviews existed for 3 articles (33.3%; 95% CI, 7.5%-70.1%) and were cited in 1 of these (33.3%; 95% CI, 0.8%-90.6%). Finally, non-Cochrane reviews existed for 4 articles (44.4%; 95% CI, 13.7%-78.8%) and were cited in 2 of these (50%; 95% CI, 6.8%-93.2%).
The sum of articles with Cochrane and non-Cochrane systematic reviews does not match the total number of articles because more than 1 citable systematic review (of either type) existed for some articles.
DiscussionSystematic reviews were only taken into account by 1 in every 3 RCT reports (33%) published in the top clinical research journals in the field of dermatology. In the case of narrative reviews, the percentage was 41.7%.
The Consolidated Standards of Reporting Trials (CONSORT) statement specifies the minimum information that should be included when reporting the results of a clinical trial. The statement also recommends that the results of a clinical trial be related to results from previous trials and suggests including systematic reviews to facilitate this comparison.5 Nonetheless, studies indicate that adherence to the CONSORT statement and similar reporting guidelines is suboptimal, although more recent articles seem to be faring better.6
Clarke et al.7,8 collected data on clinical trial reports published in a small group of top medical journals in different years (1997, 2001, 2005, and 2009) and showed that very few reports cited systematic reviews. They also found that the information from the systematic reviews was not adequately linked to the results obtained. There was no evidence of an improvement over the years, although in 2009, there did appear to be a greater presence of systematic reviews.7,8
In our study, we distinguished between Cochrane and non-Cochrane reviews and found that the former were cited less often in both RCT reports (15.6% vs 32.2%) and narrative reviews (20% vs 42.5%). An earlier study of systematic reviews reported that non-Cochrane reviews were twice as likely as Cochrane reviews to reach a favorable conclusion on the intervention being studied, a tendency that was attributed to the fact that Cochrane reviewers apply stricter criteria and only report favorable results when there is no room for doubt.9 One study of dermatology reviews found that approximately 40% of Cochrane reviews find sufficient evidence to recommend clinical intervention.10 This percentage is sufficiently high to warrant consideration of these studies. Additionally, while systematic reviews might not find sufficient evidence to answer a clinical question, they do serve to identify research gaps.11 Conducting a systematic review can be a good first step towards identifying exact needs before launching a clinical trial. The results of this trial can then be used to update and improve the systematic review, leading to a cycle of “systematic reviews followed by clinical trials, followed by systematic reviews.”12
Just 2 RCT reports in Actas Dermo-Sifiliográficas met the inclusion criteria in our study and we are therefore unable to draw any conclusions. However, in the case of narrative reviews, the citation pattern appeared to be similar to that observed for the other journals.
One limitation of our study is that we did not assess the quality of the systematic reviews analyzed, and it is therefore possible that some authors decided not to include what they considered to be reviews of suboptimal quality. However, we believe that systematic reviews always provide information, even if only to identify existing literature. Furthermore, Cochrane reviews are underpinned by a complex editorial system that guarantees that the conclusions reported will be relevant. Also, because we did not conduct an exhaustive search, we may have overlooked some systematic reviews. However, even if this were the case, it would not alter our conclusions as the percentage of “citing” articles would be even lower. Finally, it may be questioned whether the journals selected for our study are the most representative journals. We included journals with the highest impact factors and excluded those focusing primarily on basic research. If we accept that impact factor is a marker of journal and publication quality, we can probably assume that the citation rates we report are higher than true rates for dermatology journals in general.
In conclusion, we believe that systematic reviews are not sufficiently taken into account in RCT reports or narrative reviews and this may result in redundant studies and publications. Furthermore, Cochrane reviews appear to be cited even less frequently than non-Cochrane systematic reviews, indicating that authors are ignoring one of the strongest sources of evidence available.
Ethical DisclosuresProtection of humans and animalsThe authors declare that no tests were carried out in humans or animals for the purpose of this study.
Confidentiality of dataThe authors declare that they have followed their hospital's protocol on the publication of data concerning patients and that all patients included in the study have received sufficient information and have given their written informed consent to participate in the study.
Right to privacy and informed consentThe authors declare that no private patient data appear in this article.
Conflicts of InterestThe authors declare that they have no conflicts of interest.
Please cite this article as: Conde-Taboada A, Aranegui B, García-Doval I, Dávila-Seijo P, González-Castro U. Empleo de las revisiones sistemáticas en ensayos clínicos y revisiones narrativas en dermatología: ¿se usa la mejor evidencia disponible?. Actas Dermosifiliogr. 2014. 105:295–299.
