One of the great challenges in the current context is to find ways to cope with the avalanche of information to which we are exposed, seeking to select what is relevant and establish meaningful associations. The flow of knowledge production is greater than can be handled in formal educational training. In this scenario marked by the digital era, the knowledge society and the globalised world, we place the mind map as a learning strategy that allows us to develop the necessary competences to cope with this situation (Okada, 2008). Its effectiveness has been demonstrated, especially in the field of higher education and, particularly, in the area of health sciences and biology (D’Antoni & Zipp, 2006; D’Antoni et al., 2009, 2010; Wu & Wu, 2020; Badriah et al., 2023, among others).

By a mind map we mean a unique way of capturing information on a particular topic by relating/associating key concepts and/or ideas with images. In the words of Buzan (1996), its precursor in the educational field, the mind map is a graphic organiser that allows its users to use visual tools to illustrate relationships between concepts with different degrees of complexity, its expression being similar to radiant thinking and, therefore, a natural function of the human mind. It is, according to the same author, ‘a powerful graphic technique that provides a masterful key to unlocking the brain’s potential’ (Buzan, 1996: 59).

According to Sambrano and Steiner (1997: 23), ‘(...) keywords and images are used, bringing into action the left and right hemispheres respectively, to give great freedom and creativity to thought’. A mind map is a diagram that is constructed in a personal way using colours, logic, visual rhythm, images, numbers and key words to link ideas together and relate concepts, establishing dynamic rhythms of non-linear reading (Cervantes, 1999). In short, it is a creative and logical way of representing information (De Montes & Montes, 2019), with all its elements constituting ‘thematic information nodes’ around a single central element that may be represented by an image or keyword.

Elaborating on mind maps, Buzan (1996) introduces the so-called ‘laws of mind mapping’, i.e., a set of principles that serve as guidelines or references. These laws have been translated by Ontoria Peña et al. (2002: 43) as follows:

In addition to these laws that should guide mind map elaboration and the elements that every mind map should contain, Buzan (2018) proposes a series of steps to follow. 1. Place an image or theme in the centre using at least three colours (centre). 2. Use images, symbols, and codes of different dimensions throughout the mind map (use). 3. Select the keywords and play with their position, style and size. Also, each word/image should be associated with a line that connects to the central idea. The central lines should be thicker, organic and fluid, and become thinner as they radiate outwards from the centre (keywords). 4. Use colours—your code—throughout the mind map to develop your own style. 5. Use elements to emphasise and highlight the different associations between keywords and images and maintain the clarity of the mind map using the radial hierarchy and branching of ideas/concepts (clarity).

Once its principles and essential characteristics have been presented, it is worth highlighting the impact of mind mapping as a process that simulates the natural functioning of the human mind. In this sense, there are many studies that have demonstrated with different methodologies the existence of a positive relationship between the use of mind mapping and learning outcomes in terms of understanding of the content given by the teacher (D’Antoni & Zipp, 2006; D’Antoni et al., 2009; Holland et al., 2004; Mento et al., 1999; Buran & Filyukov, 2015; Frerichs et al., 2018; Indriani & Mercuriani, 2019) as well as in the development of metacognitive skills (Astriani et al., 2020; Tanriseven, 2014), the development of critical thinking (D’Antoni et al., 2010; Wu & Wu, 2020; Sánchez et al., 2019; Polat & Aydın, 2020), creativity (Zubaidah et al., 2017; Vijayakumari & Kavithamole, 2014; Taadi et al., 2019), and the spatial understanding of things, concepts, conditions, processes and events (Harley & Woodward, 1987; Okada, 2008), as well as in relation to student motivation towards learning (Jones et al., 2012; Aguilera, 2015).

In the Spanish literature, it is worth noting the work carried out by Muñoz and Ariza to introduce the mind map tool in initial teacher training and favour the development of competences, including learning to learn and teaching to learn (Muñoz González, 2010; Muñoz González et al., 2011, 2014, 2016), as well as its impact on students’ motivation to learn to learn (Ariza Carrasco, 2021; Ariza Carrasco & Muñoz González, 2020, 2021).

Regarding rubrics as instruments for assessing the quality of mind mapping, it should be noted that the first rubrics of this style were created during the first decade of this century. Specifically, the work carried out by D’Antoni et al. (2009), Evrekli et al. (2010) and Hua and Wind (2018) stand out. The first focused on measuring the reliability of their rubric based on the intraclass correlation coefficient (ICC) applied to the value (score) that each assessor gave to each of the aspects of the rubric (previously weighted). The aspects to be taken into account in the rubric created by these authors can be interpreted as follows: concept links (thematic nodes), cross-links (conceptual links or lines linking two different concepts), hierarchies (expressed through the number of sub-branches), examples (written text illustrating a concept), pictures (images illustrating or expressing a concept), colours (significant use of colours to communicate a concept) and invalid components (elements of the mind map that do not contribute to its effectiveness). The second study was an adaptation of previous work by D’Antoni et al. (2009). Its main difference with respect to the previous study is the breakdown of the concept link item into different levels (^1st, ^2nd and ^3rd level), assigning more value to thematic nodes representing more complex hierarchies. This scoring system, once applied by raters to a significant sample, showed higher reliability (using intraclass correlation) compared to the rubric and assessment system designed by D’Antoni et al. (2009).

Finally, Hua and Wind (2018) proposed an improved version of the rubrics of D’Antoni et al. (2009) and Evrekli et al. (2010) by applying Rasch’s multifaceted model to a sample of mind maps produced by secondary school students. Regarding the rubric, these authors selected eight indicators related to two general dimensions of learning: chunking skills and mnemonic techniques. By chunking skills, these authors refer to the students’ ability to articulate and represent the relationships between concepts in the mind map. By mnemonic skills, they refer to the different ways of giving meaning to a key idea or concept in ways that make it easier to memorise and more memorable. Within each dimension we find four indicators, whereby the first dimension is measured on the basis of: (1) the natural structure of the mind map, (2) the level of connections, (3) the taxonomy and (4) the meaning attached to the elements of the map. The second dimension of this rubric is expressed in terms of: (1) the selection of keywords, (2) the appropriate use of drawings, (3) the appropriate use of colours and format in general to represent these keywords and (4) the handling of the portfolio, meaning the material prepared by the students to elaborate the semantic categories or thematic nodes of the mind map.

The results obtained after psychometric testing, following the Rasch model developed by Hua and Wind (2018), were acceptable, although the authors recommended further research into the characteristics of their rubric from other methodological approaches. In this sense, this work continues the line of research of these authors and attempts to improve the instrument and the results by adding, among other things, descriptions by levels of achievement and extending the field of application to a sample of mind maps produced by university students.

The aim of this study is to analyse the validity and reliability of a rubric applied to mind maps produced by university students. Specifically, the aim is to study the construct and criterion validity, as well as to analyse the reliability of the rubric in terms of the consistency of the evaluators’ ratings and through the degree of agreement between evaluators by criterion. Thus, the question of whether the proposed rubric is valid, consistent and accurate is to be answered.

According to AERA et al. (2014) reliability is understood in a broad sense as the consistency of assessments subject to several replications. In other words, it refers to the correlation between the scores of the same test applied at various points in time or applied by different assessors. As for the validity of an instrument, following the same sources, this property has to do with the ability to measure the proposed theoretical construct, i.e., that the results obtained from applying the instrument serve to measure what it was actually intended to measure. There are three types of validity evidence which can be assessed in an instrument: content, construct and criterion. On this occasion, construct validity (the existence of latent factors in the items of the rubric) and criterion validity (the ability of the instrument’s results to predict contemporary or future results, i.e., whether the results can be generalised to other related activities) are measured. To measure these two parameters, factor analysis was used to extract latent factors (construct validity), and criterion validity was assessed by comparing the mind map scores with the grades obtained by the same group of students in the final project of another subject with similar content, which was evaluated independently from the mind map assessment.

Figure 1 and Table 2 show the average scores for each criterion according to each evaluator. At first glance it is evident that evaluator M assigned higher average scores on all criteria while evaluator R exhibits lower scores on most criteria. These differences, which a priori may be worrying and indicative of inconsistency, are subsequently assessed using other statistical tools to see if there is indeed a relevant degree of agreement and consistency between the three evaluators.

First, intra-rater reliability is measured by the internal consistency of the assessments according to Cronbach’s alpha statistic. Table 3 shows the values of the alpha statistic applied to each criterion and each rater. Overall (considering all three raters) all criteria show a high level of internal consistency, with alphas between 0.765 and 0.813. When considering each rater independently, acceptable levels of internal consistency are also obtained.

In order to measure inter-rater reliability, the aim is to determine the degree of correlation between the scores of each rater for each criterion. First, Spearman’s rank correlation is calculated, which is considered to be more appropriate because the scores are on a scale of 1 to 5. It is important to note that Spearman’s statistic does not measure the level of agreement or concordance, but rather quantifies the degree of association between two measurements. Table 4 shows these correlation coefficients together with their significance levels at 5%. It is worth noting a high association between pairs of raters for most criteria. As an exception to the above, criteria 4 and 5 exhibit a lower degree of association when considering rater M. Additionally, the last column of Table 4 shows a multi-way correlation coefficient that simultaneously incorporates the scores of all three raters (Taylor, 2020). Again the correlation is positive and significant, although criteria 4 and 5 have lower coefficients of association.

Second, to measure inter-rater reliability, a two-factor random-effects ANOVA model is assumed. In this model, a random sample of individuals is assumed, each assessed by the same set of randomly selected raters (Shrout & Fleiss, 1979).

Table 5 shows the intraclass correlation coefficients for each criterion. These are positive and significant for both consistency and agreement. A lower degree of agreement is only found for criteria 4 and 5, but in any case the correlation coefficients are higher than 0.5, which ensures an acceptable degree of reliability (Koo & Li, 2016).

Finally, other statistics generally used to measure the degree of agreement between raters are Krippendorf’s alpha and Cohen’s kappa. In both cases, the aim is to quantify the degree of agreement in the evaluators’ scores. Table 6 shows the results of both statistics for each criterion considering the ratings of the three raters. Both the alpha and kappa exhibit very similar values in their coefficients, with criteria 1, 3, 4, 5 and 7 standing out with values above 0.35, which represents a sufficient degree of agreement (Landis & Koch, 1977). On the other hand, the lowest degrees of agreement among the evaluators are found in criteria 2 and 6, although in both cases they are statistically significant at 5%.

As for the validity of the rubric, available evidence on its construct and criterion validity is analysed. In the first case, by means of factor analysis it is possible to confirm the existence of latent factors in the items of the proposed rubric. As for criterion validity, the aim is to relate the results of the proposed rubric with other results obtained by the student in related subjects independent of the mind map, in order to confirm the predictive capacity of the rubric with respect to an external criterion. For the factor analysis, we first confirmed the relevance of the available data for using this methodology by means of the Kaiser–Meyer–Olkin (KMO = 0.779) and Bartlett (χ2 = 385.1, p-value = 0.00) tests. In both cases, we obtained values that allowed factor analysis to be applied to the results of the rubric. Secondly, the factors are extracted by means of principal axis factorisation, provided that the eigenvalues are greater than unity, and with Varimax rotation. The application of factor analysis allows us to distinguish two latent factors to the rubric items: the first with an eigenvalue of 3.227 and 26.3% of variance explained, the second with an eigenvalue of 1.101 and 23.4% of variance explained. Together, these two factors explain almost 50% of the variance.

According to Table 7, for the first latent factor, items 1, 2, 3 and 4 are most important, while for the second latent factor, items 5, 6 and 7 are the most relevant. This grouping of items into two latent factors coincides with the two dimensions proposed in the rubric: chunking skills and mnemonic techniques.

Finally, criterion validity is studied by relating the results obtained from applying the proposed rubric with other results obtained by students in tests with similar content that was external to the mind map. Specifically, given the limited availability of data on student grades, the marks obtained in the ‘Essay’ and ‘Practical’ sections of the Curriculum Design and Development II course were used, as well as the final grade in the Curriculum Design and Development I (DD1) course. The following correlation coefficients between pairs of variables were obtained: 0.23 with Essay (p-value = 0.09), 0.35 with Practical (p-value = 0.01) and 0.46 with DD1 (p-value = 0.00).

A first step in determining whether a rubric for assessing mind maps is appropriate or not is to analyse its reliability, understood as the consistency shown by the assessment instrument in successive replications of the same test. Despite the proven usefulness of mind mapping in education, there is still little development of methods to assess the quality of mind maps.

The results presented in the previous section allow us to validate the reliability of the proposed rubric for assessing mind maps. On the one hand, the internal consistency of the scores is very high, with values of Cronbach’s alpha statistic close to 0.8 when considering the results of the three assessors. In the worst case, evaluator R showed a higher degree of inconsistency in his evaluations; even so, the alpha statistic still ranged between 0.61 and 0.76. In other evaluations of mind and concept mapping found in the literature, somewhat lower values of the alpha statistic are obtained. For example, D’Antoni et al. (2009) obtained a coefficient of 0.38, while Angell (2015) obtained an alpha of 0.51. Better results were obtained by Guzmán-Cedillo et al. (2017), with Cronbach’s alpha of 0.806.

On the other hand, a significant degree of agreement or concordance was found between the scores of the three assessors. Firstly, the correlation coefficients, both by pairs of assessors and considering all three assessors simultaneously, are positive and significant. This ensures a high level of association in the way each criterion is scored, which makes it possible to anticipate a significant degree of agreement. Secondly, inter-rater reliability is empirically quantified through intraclass correlation coefficients. These allow us to affirm that the reliability of the rubric is good, even obtaining similar results to other rubric evaluation studies.

Specifically, D’Antoni et al. (2009), West et al. (2002) and Guzmán-Cedillo et al. (2017) report intraclass correlation coefficients of 0.86, 0.88 and 0.91, respectively, in their studies. An important aspect to highlight is the result of consistency and agreement obtained in criterion 4 (NvCo). This is the criterion that at the individual level exhibits the lowest degree of agreement according to the intraclass correlation and according to Spearman’s correlations. This suggests that it is a criterion that requires more detailed explanation and understanding by the assessors. As for the validity of the rubric, the available evidence allows us to affirm that this instrument is valid in terms of construct and criterion. In the first case, because through the proposed items it is possible to find two latent factors that coincide with the two dimensions of skills and techniques that the rubric was intended to measure. In the second case, because the results obtained with the rubric have a positive and significant relationship with the grades in subjects of similar content.

So, this paper proposes a rubric for assessing mind maps and examines its psychometric properties to determine its reliability and validity. The proposed instrument exhibits good results in terms of consistency and inter-rater agreement, thus confirming that it is a reliable rubric. However, one criterion (NvCo-7) was found to consistently exhibit poorer psychometric results, so further detail and explanation of this criterion by the raters may be necessary. In terms of validity, it can be concluded that the instrument is valid in construct and criterion, having found latent factors that coincide with the dimensions assessed by the rubric, and that there is a significant relationship between the results of the rubric and the grades in another subject related to the subject matter contained in the mind map. With regard to the limitations of this study, a lack of knowledge on the part of teachers and students about the use of mind maps is noted, which means that time is needed to learn how to use the tool. Likewise, with regard to the validity procedure of the rubric’s content, it would be necessary to submit it to the opinion of experts in the field in order to give it more weight as well as dissemination.

Additionally, the study focuses on university students, however, introducing this mind mapping tool at earlier educational stages, particularly in secondary school, could be beneficial, as students are already required to assimilate substantial amounts of academic information.

In this way, an early commitment would be made to associative learning more in line with radial thinking rather than the linear learning of content which, in the long term, would favour the retention and understanding of information.

Finally, it should be noted that the rubric, in addition to being an assessment tool to be used by teachers to achieve a greater degree of objectivity and fairness, is an instrument that serves students as a self-guide for the achievement of skills and competences.