[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article

Design and consensus content validity of the questionnaire for b-learning education: : A 2-Tuple Fuzzy Linguistic Delphi based Decision Support Tool

Published: 01 November 2023 Publication History

Abstract

Classic Delphi and Fuzzy Delphi methods are used to test content validity of data collection tools such as questionnaires. Fuzzy Delphi takes the opinion issued by judges from a linguistic perspective reducing ambiguity in opinions by using fuzzy numbers. We propose an extension named 2-Tuple Fuzzy Linguistic Delphi method to deal with scenarios in which judges show different expertise degrees by using fuzzy multigranular semantics of the linguistic terms and to obtain intermediate and final results expressed by 2-tuple linguistic values. The key idea of our proposal is to validate the full questionnaire by means of the evaluation of its parts, defining the validity of each item as a Decision Making problem. Taking the opinion of experts, we measure the degree of consensus, the degree of consistency, and the linguistic score of each item, in order to detect those items that affect, positively or negatively, the quality of the instrument. Considering the real need to evaluate a b-learning educational experience with a consensual questionnaire, we present a Decision Making model for questionnaire validation that solves it. Additionally, we contribute to this consensus reaching problem by developing an online tool under GPL v3 license. The software visualizes the collective valuations for each iteration and assists to determine which parts of the questionnaire should be modified to reach a consensual solution.

Highlights

A model to test content validity of questionnaires is proposed.
The model relies on linguistic 2-tuples, extended hierarchies, and the Delphi method.
A consensus process achieves a suitable degree of agreement between experts.
The model is applied in an actual blending learning experience in education.
We design a decision support system as an online tool for the model application.

References

[1]
Santiago-Delefosse M., Gavin A., Bruchez C., Roux P., Stephen S., Quality of qualitative research in the health sciences: Analysis of the common criteria present in 58 assessment guidelines by expert users, Soc. Sci. Med. 148 (2016) 142–151.
[2]
Saffie N.A.M., Rasmani K.A., et al., Fuzzy delphi method: Issues and challenges, in: 2016 International Conference on Logistics, Informatics and Service Sciences (LISS), IEEE, 2016, pp. 1–7.
[3]
Padilla-Rivera A., do Carmo B.B.T., Arcese G., Merveille N., Social circular economy indicators: Selection through fuzzy delphi method, Sustain. Prod. Consum. 26 (2021) 101–110.
[4]
Dawood K.A., Sharif K.Y., Ghani A.A., Zulzalil H., Zaidan A., Zaidan B., Towards a unified criteria model for usability evaluation in the context of open source software based on a fuzzy Delphi method, Inf. Softw. Technol. 130 (2021).
[5]
Bergmann J., Sams A., Flip Your Classroom: Reach Every Student in Every Class Every Day, International Society for Technology in Education, 2012.
[6]
Jaldemark J., et al., Editorial introduction: Collaborative learning enhanced by mobile technologies, Br. J. Educ. Technol. (2017) 201–206.
[7]
Al-Emran M., Mezhuyev V., Kamaludin A., Technology Acceptance Model in M-learning context: A systematic review, Comput. Educ. 125 (2018) 389–412.
[8]
Kaiser H.F., A systematic review of research on the flipped learning method in engineering education, Br. J. Educ. Technol. 49 (3) (2018) 398–411.
[9]
Tan X., Zhu J., Cabrerizo F.J., Herrera-Viedma E., A cyclic dynamic trust-based consensus model for large-scale group decision making with probabilistic linguistic information, Appl. Soft Comput. 100 (2021).
[10]
Mendel J.M., Zadeh L.A., Trillas E., Yager R., Lawry J., Hagras H., Guadarrama S., What computing with words means to me [discussion forum], IEEE Comput. Intell. Mag. 5 (1) (2010) 20–26.
[11]
Zadeh L.A., Computing with Words: Principal Concepts and Ideas, Vol. 277, Springer, 2012.
[12]
Montes R., Sanchez A.M., Villar P., Herrera F., A web tool to support decision making in the housing market using Hesitant Fuzzy Linguistic Term Sets, Appl. Soft Comput. 35 (2015) 949–957.
[13]
Montes R., Sanchez A.M., Villar P., Herrera F., Teranga Go!: Carpooling Collaborative Consumption Community with multi-criteria hesitant fuzzy linguistic term set opinions to build confidence and trust, Appl. Soft Comput. 67 (2018) 941–952.
[14]
Morente-Molinera J., Kou G., Pérez I., Samuylov K., Selamat A., Herrera-Viedma E., A group decision making support system for the Web: How to work in environments with a high number of participants and alternatives, Appl. Soft Comput. 68 (2018) 191–201.
[15]
Orfanou K., Tselios N., Katsanos C., Perceived usability evaluation of learning management systems: Empirical evaluation of the system usability scale, Int. Rev. Res. Open Distributed Learn. 16 (2) (2015) 227–246.
[16]
Carrasco R.A., Villar P., Hornos M.J., Herrera-Viedma E., A linguistic multi-criteria decision making model applied to the integration of education questionnaires, Int. J. Comput. Intell. Syst. 4 (5) (2011) 946–959.
[17]
Aggarwal L., Goswami P., Sachdeva S., Multi-criterion intelligent decision support system for COVID-19, Appl. Soft Comput. 101 (2021).
[18]
Herrera F., Martínez L., A 2-tuple fuzzy linguistic representation model for computing with words, IEEE Trans. Fuzzy Syst. 8 (6) (2000) 746–752.
[19]
Zhao S., Dong Y., Wu S., Martínez L., Linguistic scale consistency issues in multi-granularity decision making contexts, Appl. Soft Comput. 101 (2021).
[20]
Ponterotto J.G., Ruckdeschel D.E., An overview of coefficient alpha and a reliability matrix for estimating adequacy of internal consistency coefficients with psychological research measures, Percept. Motor Skills 105 (3) (2007) 997–1014.
[21]
Evers A., Muñiz J., Hagemeister C., Høstmælingen A., Lindley P., Sjöberg A., Bartram D., Assessing the quality of tests: Revision of the EFPA review model, Psicothema 25 (3) (2013) 283–291.
[22]
Kaiser H.F., An index of factorial simplicity, Psychometrika 39 (1) (1974) 31–36.
[23]
Bartlett M.S., Tests of significance in factor analysis, Br. J. Math. Stat. Psychol. 3 (2) (1950) 77–85.
[24]
Ding C.S., Hershberger S.L., Assessing content validity and content equivalence using structural equation modeling, Struct. Equ. Model. 9 (2) (2002) 283–297.
[25]
Hyrkäs K., Appelqvist-Schmidlechner K., Oksa L., Validating an instrument for clinical supervision using an expert panel, Int. J. Nurs. Stud. 40 (6) (2003) 619–625.
[26]
Walker K.D., Electronic Delphi method, in: Okoko J.M., Tunison S., Walker K.D. (Eds.), Varieties of Qualitative Research Methods: Selected Contextual Perspectives, Springer International Publishing, Cham, 2023, pp. 155–160.
[27]
Gómez-Parra M., Golubeva I., Espejo R., Design and validation of a questionnaire for the measurement of students’ perceptions of intercultural practices within bilingual secondary schools in the European context, J. Lang. Educ. 7 (3) (2021) 58–75.
[28]
Shah S., Solangi Y., Ikram M., Analysis of barriers to the adoption of cleaner energy technologies in Pakistan using modified delphi and fuzzy analytical hierarchy process, J. Clean. Prod. 235 (2019) 1037–1050.
[29]
Miller K., Collada B., Tolliver D., Audi Z., Cohen A., Michelson C., Newman L., Using the modified Delphi method to develop a tool to assess pediatric residents supervising on inpatient rounds, Acad Pediatr. 20 (1) (2020) 89–96.
[30]
Shinners L., Aggar C., Grace S., Smith S., Exploring healthcare professionals’ perceptions of artificial intelligence: Validating a questionnaire using the e-Delphi method, Digit. Health 7 (2021).
[31]
Partner R., Tee J., Darrall-Jones J., Jones B., Development of a rugby shoulder function (RSF) questionnaire: An online Delphi study, Phys. Ther. Sport 61 (2023) 185–191.
[32]
Djordjević B., Fröidh O., Krmac E., Determinants of autonomous train operation adoption in rail freight: knowledge-based assessment with Delphi-ANP approach, Soft Comput. 27 (11) (2023) 7051–-7069.
[33]
Okoli C., Pawlowski S.D., The Delphi method as a research tool: An example, design considerations and applications, Inf. Manag. 42 (1) (2004) 15–29.
[34]
Aengenheyster S., Cuhls K., Gerhold L., Heiskanen-Schüttler M., Huck J., Muszynska M., Real-Time Delphi in practice — A comparative analysis of existing software-based tools, Technol. Forecast. Soc. Change 118 (2017) 15–27.
[35]
Murray T.J., Pipino L.L., van Gigch J.P., A pilot study of fuzzy set modification of Delphi, Hum. Syst. Manag. 5 (1) (1985) 76–80.
[36]
Zadeh L.A., The concept of a linguistic variable and its application to approximate reasoning—I, Inform. Sci. 8 (3) (1975) 199–249.
[37]
Martinez L., Herrera F., An overview on the 2-tuple linguistic model for computing with words in decision making: Extensions, applications and challenges, Inform. Sci. 207 (2012) 1–18.
[38]
Wan S.-P., 2-Tuple linguistic hybrid arithmetic aggregation operators and application to multi-attribute group decision making, Knowl.-Based Syst. 45 (2013) 31–40.
[39]
Wei G., Zhao X., Some dependent aggregation operators with 2-tuple linguistic information and their application to multiple attribute group decision making, Expert Syst. Appl. 39 (5) (2012) 5881–5886.
[40]
Xu Y., Wang H., Approaches based on 2-tuple linguistic power aggregation operators for multiple attribute group decision making under linguistic environment, Appl. Soft Comput. 11 (5) (2011) 3988–3997.
[41]
Herrera F., Martínez L., A model based on linguistic 2-tuples for dealing with multigranular hierarchical linguistic contexts in multi-expert decision-making, IEEE Trans. Fuzzy Syst. 31 (2) (2001) 227–234.
[42]
Espinilla M., Liu J., Martinez L., An extended hierarchical linguistic model for decision-making problems, Comput. Intell. 27 (3) (2011) 489–512.
[43]
Awidi I.T., Paynter M., The impact of a flipped classroom approach on student learning experience, Comput. Educ. 128 (2019) 269–283.
[44]
Aznar-Díaz I., Hinojo-Lucena F.-J., Cáceres-Reche M.-P., Romero-Rodríguez J.-M., Analysis of the determining factors of good teaching practices of mobile learning at the Spanish University. An explanatory model, Comput. Educ. 159 (2020).
[45]
Irwanto I., Saputro A.D., Widiyanti W., Laksana S.D., Global trends on mobile learning in higher education: A bibliometric analysis (2002–2022), Int. J. Inf. Educ. Technol. 13 (2) (2023) 373–383.
[46]
Kim J., Developing an instrument to measure social presence in distance higher education, Br. J. Educ. Technol. 42 (5) (2011) 763–777.
[47]
Garrison D., Akyol Z., Toward the development of a metacognition construct for communities of inquiry, Int. Higher Educ. 17 (Supplement C) (2013) 84–89.
[48]
Arbaugh J., et al., Developing a Community of Inquiry instrument: Testing a measure of the Community of Inquiry framework using a multi-institutional sample, Int. Higher Educ. 11 (3–4) (2008) 133–136.
[49]
García-Lira K., Gutiérrez-Santiuste E., Montes-Soldado R., Questionnaires for the communication and satisfaction assessment upon applying flipped classroom methodologies combined with m-learning in higher education, in: III International Congress on Media Education and Digital Competence, 2017, pp. 1145–1163.
[50]
Zuheros C., Martínez-Cámara E., Herrera-Viedma E., Herrera F., Sentiment Analysis based Multi-Person Multi-criteria Decision Making methodology using natural language processing and deep learning for smarter decision aid. Case study of restaurant choice using TripAdvisor reviews, Inf. Fusion 68 (2021) 22–36.
[51]
Zuheros C., Martínez-Cámara E., Herrera-Viedma E., Herrera F., Crowd decision making: Sparse representation guided by sentiment analysis for leveraging the wisdom of the crowd, IEEE Trans. Syst., Man, Cybern.: Syst. 53 (1) (2022) 369–379.

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Applied Soft Computing
Applied Soft Computing  Volume 147, Issue C
Nov 2023
1683 pages

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 November 2023

Author Tags

  1. Decision support system
  2. Multicriteria decision making
  3. Extended linguistic hierarchies
  4. Linguistic 2-tuples
  5. Questionnaire validation
  6. Consensus
  7. Fuzzy delphi method

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 0
    Total Downloads
  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 15 Jan 2025

Other Metrics

Citations

View Options

View options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media