Actas Dermo-Sifiliográficas (English Edition) Actas Dermo-Sifiliográficas (English Edition)
Actas Dermosifiliogr 2017;108:924-30 - Vol. 108 Num.10 DOI: 10.1016/j.adengl.2017.10.004
Original Article
Spanish Cultural Adaptation of the Questionnaire Early Arthritis for Psoriatic Patients
Adaptación cultural al español del cuestionario Early Arthritis for Psoriatic Patients
J. García-Gavína,, , L. Pérez-Péreza, I. Tinazzib, D. Vidalc, D. Mc Gonagled
a Clínica Pérez & Gavín dermatólogos, Vigo, Pontevedra, España
b Unidad de Reumatología, Ospedale Sacro Cuore Don Calabria, Verona, Italia
c Hospital de Sant Joan Despí Moisés Broggi, Barcelona, España
d Leeds Institute of Rheumatic and Musculoskeletal Medicine, University of Leeds, Leeds, Inglaterra
Received 25 January 2017, Accepted 18 June 2017
Abstract
Introduction and objectives

The Early Arthritis for Psoriatic patients (EARP) questionnaire is a screening tool for psoriatic arthritis. The original Italian version has good measurement properties but the EARP required translation and adaptation for use in Spain. This article describes the cultural adaptation process as a step prior to validation.

Material and methods

We used the principles of good practice for the cross-cultural adaptation of patient-reported outcomes measurement established by the International Society Pharmacoeconomics and Outcome Research. The steps in this process were preparation, forward translation, reconciliation, back-translation and review, harmonization, cognitive debriefing and review, and proofreading. During preparation the developers of the original questionnaire were asked for their permission to adapt the EARP for use in Spain and to act as consultants during the process.

Results

The original questionnaire was translated into Spanish by native Spanish translators, who made slight changes that were approved by the questionnaire's developers. The Spanish version was then back-translated into Italian; that version was reviewed to confirm equivalence with the original Italian text. The reconciled Spanish EARP was then tested for comprehensibility and interpretation in a group of 35 patients. All the patients answered all items without making additional comments.

Conclusion

This cultural adaptation of the EARP questionnaire for Spanish populations is the first step towards its later use in routine clinical practice. The application of a cross-cultural adaptation method ensured equivalence between the original and Spanish versions of the EARP. The Spanish questionnaire will be validated in a second stage.

Resumen
Introducción y objetivos

El cuestionario Early Arthritis for Psoriatic patients (EARP) es una herramienta de screening para artritis psoriásica con buenas propiedades de medición en su versión original en italiano que no ha sido todavía adaptado culturalmente al español. Este artículo expone la adaptación para población española como paso previo a su validación.

Material y método

Aplicación de la metodología recomendada por la International Society for Pharmacoeconomics and Outcomes Research para adaptaciones culturales de medidas centradas en el paciente y que consta de las siguientes fases: preparación, traducción, reconciliación, retrotraducción y su revisión, armonización, test de comprensión y su revisión y corrección de pruebas. En la preparación se obtuvo el permiso de los autores del cuestionario original para su adaptación cultural y colaboraron durante el proceso como asesores.

Resultados

La traducción del cuestionario original al español la realizaron traductores nativos que realizaron pequeñas modificaciones aceptadas por los autores de la versión original. Se realizó la retrotraducción al italiano, obteniendo una versión equivalente al EARP original. La versión española después de la retrotraducción se aplicó en el test de comprensión a 35 pacientes. Todos ellos contestaron a todos los ítems sin hacer aportaciones adicionales.

Conclusión

La adaptación cultural del cuestionario EARP para población española constituye la primera etapa para su posterior uso en práctica clínica habitual. La aplicación de una metodología estandarizada garantiza la equivalencia entre el EARP en español y el original. En una segunda etapa se realizará la validación en población española.

Keywords
Psoriatic arthritis, Psoriasis, Screening, EARP questionnaire, Cultural adaptation
Palabras clave
Artritis psoriásica, Psoriasis, Screening, Early Psoriatic Arthritis Screening Questionnaire, Adaptación cultural
Introduction

Psoriatic arthritis (PsA) is a chronic inflammatory disease in which psoriasis is associated with a variety of clinical and radiologic signs that are not exclusive to this diagnosis.1–3 The estimated prevalence ranges from 0.02% to 0.42%4–6 and the incidence from 3.4 to 8 cases per 100 000 population per year.7,8 About 30% of patients with psoriasis may have PsA.9

The diagnosis of this disease is complicated by the variety of clinical presentations and its similarity to other joint diseases. Two patient profiles can be defined. One type of patient develops early arthritis and is generally followed by a rheumatologist; another will initially present with skin involvement and is usually followed by a dermatologist but will develop arthritis later.10 A recent study in 7 countries found that a third of patients with psoriasis treated in dermatology clinics also had PsA.11 The dermatologist must therefore be able to identify signs suggestive of this disease so that the patient can be referred to a rheumatologist for early control of joint involvement.12–14 This multidisciplinary approach to PsA means that the main specialists treating these patients require valid diagnostic tools for use in routine clinical practice.14,15

The various self-administered questionnaires for detecting PsA cases include the Psoriatic Arthritis Screening and Evaluation and Toronto Psoriatic Arthritis Screening questionnaires.16,17 The former has been validated for use in Spanish populations.18 A study comparing the ability of these tools to detect cases found that their specificity in clinical use was lower than that reported in the initial validation studies and also lower than the levels considered clinically useful.19 Moreover, these instruments were less sensitive in patients with lower levels of disease activity, fewer joint symptoms, or recent onset of disease.19 This situation means they cannot contribute to early case detection.13

The Early Arthritis for Psoriatic Patients (EARP) questionnaire20 has been shown to have good measurement properties in its original version in Italian. The main advantages of the EARP over the previously mentioned questionnaires are that it is simple and can be administered quickly, features that make a tool useful to practicing dermatologists.21 Furthermore, the initial validation study for the EARP showed that it identified a large percentage of the patients who had not previously reported PsA symptoms. The Spanish version we present in this article has undergone such a validation study; the results are consistent with those findings and a report should be published soon. The availability of a culturally adapted and validated Spanish version of the EARP should prove useful for managing patients with psoriasis in Spain.

Literally translated questionnaires can contain erroneous interpretations because of cultural differences between countries. Before a translated questionnaire is used in a new population, it must undergo a process of cultural adaptation of the language and the translated tool's measurement properties must be appropriately validated. The International Society of Pharmacoeconomics and Outcomes Research (ISPOR) provides a standardized method for the translation, cultural adaptation, and linguistic validation of clinical tools.22

Our aim here is to report the process we followed to culturally adapt a version of the EARP questionnaire suitable for use in a Spanish population.

Material and MethodsThe Questionnaire

The EARP, whose purpose is to detect PsA in patients with psoriasis, was developed based on a review of the signs and symptoms typical of patients diagnosed with PsA.20 The Italian authors drafted the questionnaire with 14 yes/no and either/or questions, but after piloting and validating it in a group of patients, they discarded 4 items because of low correlation with other items (low internal consistency). Thus, the final version has 10 items (Fig. 1), and the results are scored from 0 (all negative responses) to 10 (all affirmative responses). The cut point is 3: a score of 3 or more indicates the possible presence of PsA. The authors concluded that the EARP is brief, easy to answer and reliably identifies the presence of PsA and can also detect incipient PsA.

Figure 1.
(0.28MB).

Items in the original Italian version of the Early Arthritis for Psoriatic Patients questionnaire.

Cultural Adaptation

The cultural adaptation of the EARP questionnaire for use in Spain followed the ISPOR guidelines,22 which stipulate the following processes: 1) preparation, 2) forward translation, 3) reconciliation, 4) back-translation, 5) back-translation review, 6) harmonization, 7) testing of the translation in relevant patients followed by cognitive debriefing, 8) review of cognitive debriefing findings and revision of the translation, and 9) proofreading (Fig. 2).

Figure 2.
(0.25MB).

Flow chart of the standardized process used to adapt the Early Arthritis for Psoriatic Patients (EARP) questionnaire for use in a Spanish population.

In the first, preparatory, phase we asked the EARP questionnaire developers for their permission to adapt it for Spain. At the same time we asked them to clarify our doubts about ambiguities in concepts and invited them to participate actively in our translation process. A working group was established and native translators to Spanish and Italian were recruited.

In the second step, two native speakers of Spanish each translated the questionnaire.

In the third step, the Spanish translators reconciled their translations to produce a single version after discussion of doubts and discrepancies to arrive at consensus. The Italian developers participated throughout.

In the fourth and fifth phases, the reconciled Spanish version was back-translated to Italian and reviewed by the Italian developers of the original.

In this process, involving much exchange of information among working group members and the Italian developers, harmonization was conceived as an ongoing quality control effort to ensure conceptual equivalence between the original and translated version. Thus, harmonization was not a separate phase in our process. Next, a group of patients representative of the target population answered the items on the translated EARP questionnaire and participated in a cognitive debriefing interview. We enrolled 35 patients for this purpose, although the ISPOR guidelines22 suggest fewer (between 5 and 8) and other cultural adaptation working groups have tested translations in groups that were smaller than ours.23,24 We felt that testing in a larger group would allow us to be more confident that items were comprehensible and that potential sources of confusion had been identified. The results of this test process were discussed among members of the working group.

The last phase, formal proofreading, was unnecessary because the questionnaire was very brief and quality checks in the previous phases ensured accuracy in grammar and spelling.

We complied with ethical requirements to protect the confidentiality of participants and obtain their informed written consent.

Results

Before translation, the developers of the EARP questionnaire gave their permission for the translation and cultural adaptation to Spanish, and they collaborated actively throughout the process. Examples of help they provided include discussion of alternative verb tenses to improve comprehension of items and the use of the phrase and/or when more than one part of the body was referred to. They also clarified whether the phrase dolore alla schiena in item 3 (Fig. 1) referred to pain in the entire back or only the lower back. The authors also clarified expressions with negative constructions that were potentially difficult to understand.

The translators in the second phase had university degrees in translation and were experienced with medical texts (2 translating into Spanish, 1 into Italian). The project supervisor also gave the Spanish translators the Italian developers’ comments and clarifications. After reconciliation (third phase) modifications were made (Table 1). Items were reordered so that the ones referring to joint pain were grouped first; then followed items referring to swollen joints and finally the remaining items. Verb tenses were changed from the present tense used in Italian to the present perfect for the Spanish version, and the introductory statement about the time frame was shortened. Item 10 of the Spanish version used the phrase tomar fármacos antiinflamatorios (take anti-inflammatory drugs) in order to introduce the familiar word fármacos even though it was unnecessary, rather than the shorter phrase used in the Italian. Another change was the placement of this item last in the questionnaire since it was the only one to use a time frame of 3 months; in addition, this statement of time frame was shifted to the beginning of the sentence, whereas it appeared at the end in the Italian. These last 2 changes were made to facilitate comprehension for patients when they encountered a shift in the time frame. All changes were approved by the developers of the original questionnaire.

Table 1.

Changes Made in Items of the Early Arthritis for Psoriatic Patients Questionnaire During Translation and Adaptation.

Item  Comment 
Introductory phrase  The introductory phase about the time frame for most questions was shortened. 
All items  Items were ordered differently in the Spanish version. 
All items  The verb tense was changed to present perfect for the Spanish version. 
Item 10  The Spanish item included the familiar word fármacos (“anti-inflammatory drugs” rather than just “anti-inflammatories”) in Spanish in the interest of using colloquial language. 
Item 10, list order  This item was placed last in the list because its 3-month time frame differed from the 6-month frame of other items. 

The reconciled Spanish version was back-translated to Italian (Fig. 3) by a single native Italian translator; like the Spanish translators, this individual had a degree in translation and was experienced with medicine. No important differences between the original and the back-translated Italian versions were found. The consensus of the working group and the Italian developers was that the reconciled Spanish translation was a valid version of the Italian questionnaire.

Figure 3.
(0.49MB).

Items in the Spanish version of the Early Arthritis for Psoriatic Patients questionnaire after translation and cultural adaptation. (Translator's note: The non-validated English translations shown below each Spanish item are for information purposes only.)

Harmonization (the sixth recommended phase) was carried out throughout the process by means of active exchange of opinions and evaluations within the working group, which included specialists in dermatology and rheumatology, and the Italian developers. Harmonization was thus accomplished through ongoing quality control measures. The 35 patients with psoriasis who answered the translated EARP questionnaire (seventh and eighth phases) were native speakers of Spanish; 20 lived in Vigo and 15 in Barcelona. The mean (SD) age in the group was 46.1 (16.7) years. More than half were men and the largest education subgroup had only primary schooling (Table 2). This phase is critical for ensuring that the questionnaire is correctly understood by the target population and to identify potentially confusing terms or expressions. All the patients answered all questions and none reported having problems understanding any of the items or any expressions or terms. Nor did they consider it necessary to make any changes in the wording.

Table 2.

Social and Demographic Characteristics of the Sample of Patients Testing the Spanish Translation (Phase 7).

Patient Characteristics  Mean (SD)  n (%) 
Age, y  46.1 (16.7)   
Gender
Male    11 (62.9) 
Female    8 (37.1) 
Higher education    9 (25.7) 
Secondary education    3 (8.6) 
Primary school    22 (62.9) 
No schooling    1 (2.9) 

At the end of the testing phase, the adaptation process was complete, and a version was ready for trial in a Spanish population (Fig. 3).

Proofreading, the last IPSOR-recommended phase, was deemed unnecessary because grammar and spelling in this very brief questionnaire were checked throughout the process.

Discussion

This article describes the cultural adaptation process used to translate the EARP questionnaire for use with psoriasis patients in Spain. The process has provided a version that is comparable to the original Italian one. If a standardized process22 is followed, the user can be confident that a questionnaire has not simply been literally translated. Instead, cultural aspects will have been taken into consideration in the interest of minimizing error due to inappropriate interpretation of item content. Various processes have been recommended in the literature on producing validated translations of measurement tools,25,26 but the minimally essential steps are as follows: translation, back-translation, and testing of comprehension with patients who represent the target population. This process was used to adapt the EARP questionnaire for use in Spain.

Adequate communication with the developers of the original version of a questionnaire is also a basic requirement for cultural adaptation and should be initiated from the start of a project. Dialog with the original developers facilitates understanding of the exact meaning of each item. In our process we explained our doubts to the original authors regarding subtle but important details such as the location of back pain (whole back vs lower back).

Inadequate cultural adaptation of a questionnaire can lead to erroneous interpretations affecting the treatment of individual patients during follow-up. It may also mean that reports of research findings for populations in different countries are not comparable. Given the cultural adaptation process we applied to translate the EARP questionnaire, it can be concluded that the Spanish and Italian versions are comparable.

Properly following the cultural adaptation process is also an essential condition for later validating the translated instrument by calculating sensitivity and specificity statistics. With a valid translation as the starting point, the user can be confident that discrimination performance statistics reflect proper interpretation of items that have been appropriately forward- and back-translated to safeguard against cultural misinterpretation. Furthermore, in a context in which patient measures are considered important data in the approval of new therapies,27 compliance with recommended procedures during cultural adaptation of measurement instruments acquires particular importance, as mentioned above.

We detected no remarkable difficulties in interpreting content during the translation and cultural adaptation of the EARP questionnaire to Spanish. Thus, we did not have to make great changes in item content, largely because the questions were brief, simple, and asked for yes/no and either/or answers that were easy for patients to give. Using colloquial words such as dedos (fingers) and caderas (hips) instead of more technical terms undoubtedly facilitated comprehension, and the test patients reported no interpretation problems.

Nevertheless, our process had certain limitations. Even though we tested the translation in more patients than the ISPOR recommendations22 require, and also more than the 9 to 16 patients other recommendations suggest,23,24 it would have been desirable to recruit patients representing more geographic areas within Spain. It would also have been desirable to include a second Italian back-translator in the process so that language choices could have been discussed. Another point to bear in mind is that the Spanish language version we produced is valid for a population whose mother tongue is a version spoken within Spain. It would be necessary to determine whether it would also be valid for use in other Spanish-speaking areas such as South America.

The cultural adaptation of an instrument is not the final step for determining whether a translated questionnaire is reliable for use in clinical practice or research. A validation study that assesses our translation's measurement properties (validity, sensitivity, specificity) must be done26 in an appropriate Spanish population sample to ensure the proposed tool behaves similarly to the original questionnaire. Our recently completed validation study of the Spanish EARP questionnaire should be published shortly.

Ethical DisclosuresProtection of human and animal subjects

The authors declare that no experiments were performed on humans or animals for this investigation.

Data confidentiality

The authors declare that they followed their hospitals’ regulations regarding the publication of patient information and that written informed consent for voluntary participation was obtained for all patients.

Right to privacy and informed consent

The authors declare that no private patient data are disclosed in this article.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

References
1
W.J. Taylor,D.E. Fellow,P.S. Helliwell
Case definition of psoriatic arthritis
Lancet, 16 (2000), pp. 2095
2
E. Soriano
Artritis psoriásica: una enfermedad prevalente con gran diversidad de manifestaciones clínicas
Rev Arg Reumatol, 23 (2012), pp. 8-9
3
W.H. Boehncke,A. Qureshi,J.F. Merola,D. Thaçi,G.G. Krueger,J. Walsh
Diagnosing and treating psoriatic arthritis: An update
Br J Dermatol, 170 (2014), pp. 772-786 http://dx.doi.org/10.1111/bjd.12748
4
T. O’Neill,A.J. Silman
Psoriatic arthritis. Historical background and epidemiology
Baillieres Clin Rheumatol, 8 (1994), pp. 245-261
5
P. Trontzas,A. Andrianakos,S. Miyakis,K. Pantelidou,E. Vafiadou,V. Garantziotou,ESORDIG Study Group
Seronegative spondyloarthropathies in Greece: A population-based study of prevalence, clinical pattern, and management. The ESORDIG Study
Clin Rheumatol, 24 (2005), pp. 583-589 http://dx.doi.org/10.1007/s10067-005-1106-9
6
R. De Angelis,F. Salaffi,W. Grassi
Prevalence of spondyloarthropathies in an Italian population sample: A regional community-based study
Scand J Rheumatol, 36 (2007), pp. 14-21 http://dx.doi.org/10.1080/03009740600904243
7
I.N. Bruce
Psoriasis arthritis: Clinical features
Rheumatology, 3rd ed, pp. 1241-1252
8
M. Shbeeb,K.M. Uramoto,L.E. Gibson,W.M. O’Fallon,S.E. Gabriel
The epidemiology of psoriatic arthritis in Olmsted County, Minnesota, USA, 1982-1991
J Rheumatol, 27 (2000), pp. 1247-1250
9
S. González,R. Queiro,J. Ballina
Actualización en la patogenia de la artritis psoriásica
Reumatol Clin, 8 (2012), pp. S1-S6 http://dx.doi.org/10.1016/j.reuma.2011.12.003
10
A. López-Ferrer,A. Laiz-Alonso
Psoriatic arthritis: An update
Actas Dermosifiliogr, 105 (2014), pp. 913-922 http://dx.doi.org/10.1016/j.ad.2013.10.020
11
P.J. Mease,D.D. Gladman,K.A. Papp,M.M. Khraishi,D. Thaçi,F. Behrens
Prevalence of rheumatologist-diagnosed psoriatic arthritis in patients with psoriasis in European/North American dermatology clinics
J Am Acad Dermatol, 69 (2013), pp. 729-735 http://dx.doi.org/10.1016/j.jaad.2013.07.023
12
J.D. Cañete,E. Daudén,R. Queiro,M.D. Aguilar,J.L. Sánchez-Carazo,J.M. Carrascosa
Elaboración mediante el método Delphi de recomendaciones para el manejo coordinado (reumatólogo/dermatólogo) de la artritis psoriásica
Actas Dermosifiliogr, 105 (2014), pp. 216-232 http://dx.doi.org/10.1016/j.adengl.2013.07.003
13
J.L. López Estebaránz,P. Zarco-Montejo,M.L. Samaniego,C. García-Calvo,PREVAL Study Group
Prevalence and clinical features of psoriatic arthritis in psoriasis patients in Spain. Limitations of PASE as a screening tool
Eur J Dermatol, 25 (2015), pp. 57-63 http://dx.doi.org/10.1684/ejd.2014.2449
14
J.D. Cañete,L. Puig
Multidisciplinary teams for psoriatic arthritis: On aims and approaches
Actas Dermosifiliogr, 105 (2014), pp. 325-327 http://dx.doi.org/10.1016/j.ad.2014.01.002
15
J. Luelmo,J. Gratacós,M. Moreno,M. Ribera,J. Romanía,J. Calvet
Experiencia de 4 años de funcionamiento de una unidad multidisciplinar de psoriasis y artritis psoriásica
Actas Dermosifiliogr, 105 (2014), pp. 371-377
16
P.L. Domíngez,M.E. Husni,E.W. Holt,S. Tyler,A.A. Qureshi
Validity, reliability and sensitivity-to-change properties of the psoriatic arthritis screening evaluation questionnaire
Arch Der Res, 301 (2009), pp. 573-579
17
D. Gladman,C. Schentag,B. Tom,V. Chandran,J. Brockbank,C. Rosen
Development and initial validation of a screening questionnaire for psoriatic arthritis: The Toronto Psoriatic Arthritis Screen (ToPAS)
Ann Rheum Dis, 68 (2008), pp. 497-501 http://dx.doi.org/10.1136/ard.2008.089441
18
J. Walsh,K. Callis,G. Krueger,D. Clegg
Limitations in screening instruments for psoriatic arthritis: A comparison of instruments in patients with psoriasis
J Rheumatol, 40 (2013), pp. 287-293 http://dx.doi.org/10.3899/jrheum.120836
19
L.G. Ferreyra,E.R. Soriano,J.E. Rosa,D.A. Navarta,C. Saucedo,M. Scolnik
Validation in Spanish of a screening questionnaire for the detection of psoriatic arthritis in patients with psoriasis
Rheumatology, 52 (2013), pp. 510-514 http://dx.doi.org/10.1093/rheumatology/kes306
20
I. Tinazzi,S. Adami,E.M. Zanolin,C. Caimmi,S. Confente,G. Girolomoni
The early psoriatic arthritis screening questionnaire: A simple and fast method for the identification of arthritis in patients with psoriasis
Rheumatology, 51 (2012), pp. 2058-2063 http://dx.doi.org/10.1093/rheumatology/kes187
21
O. FitzGerlad
Screening for arthritis in patients with psoriasis
Nat Rev Reumatol, 640 (2012), pp. 640-641
22
D. Wild,A. Grove,M. Martin,S. Eremenco,S. McElroy,A. Verjee-Lorenz
Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: Report of the ISPOR task force for translation and cultural adaptation
Value in Health, 8 (2005), pp. 94-104 http://dx.doi.org/10.1111/j.1524-4733.2005.04054.x
23
W. Castillo-Tandazo,A. Flores-Fortty,L. Feraud,D. Tettamanti
Spanish translation, cross-cultural adaptation, and validation of the Questionnaire for Diabetes-related Foot Disease (Q-DFD)
Vasc Health Risk Manag, 9 (2013), pp. 501-508 http://dx.doi.org/10.2147/VHRM.S49749
24
G. Gastaminza,M. Herdman,I. Baiardini,F. Braido,M. Corominas
Cross-cultural adaptation and linguistic validation of the Spanish version of the drug hypersensitivity quality of life questionnaire
J Investig Allergol Clin Immunol, 23 (2013), pp. 508-510
25
L.D. McLedo,D. Coon Ch,S.A. Martin,S.E. Fehnel,R.D. Hays
Interpreting patient-reported outcomes results: US FDA guidance and emerging methods
Expert Rev Pharmacoecon Outcomes Res, 11 (2011), pp. 163-169 http://dx.doi.org/10.1586/erp.11.12
26
U.S. Department of Health and Human Services Food and Drug Administration. Guidance for Industry. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims 2009 [cited 2016 Nov 9]. Available from: http://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/ucm193282.pdf
27
F. Pignatti,B. Jonsson,G. Blumenthal,R. Justice
Assessment of benefits and risks in development of targeted therapies for cancer--The view of regulatory authorities

Please cite this article as: García-Gavín J, Pérez-Pérez L, Tinazzi I, Vidal D, McGonagle D. Adaptación cultural al español del cuestionario Early Arthritis for Psoriatic Patients. Actas Dermosifiliogr. 2017;108:924–930.

Corresponding author. (J. García-Gavín jgavin@pgdermatologos.com)
Copyright © 2017. Elsevier España, S.L.U. and AEDV
Actas Dermosifiliogr 2017;108:924-30 - Vol. 108 Num.10 DOI: 10.1016/j.adengl.2017.10.004