Study finds common creativity assessment in education may not be completely valid
LAWRENCE — Creativity is increasingly viewed as an essential part of education and vital for the future of a competitive nation. Yet not a lot is known about what makes a person creative, or how to identify and teach those traits. A new study from the University of Kansas has found that one of the most common methods used in studying creativity may be reliable, but it varies widely in how it is used, depends on subjective judgments and may not be completely valid.
Subjective creativity assessment is one of the most popular methods in creativity research. In essence, the approach depends on judges who study a work or topic, then provide a rating on whether it meets creativity criteria. Haiying Long, associate professor of educational psychology at KU, led research that analyzed 84 studies using the approach, also known as Consensual Assessment Technique, and found their approaches vary widely and do not necessarily prove its validity. That should warrant caution in depending on the approach or attempting to translate it to use in schools, she said.
Long compared the approach to voting for the Academy Awards, in which voters watch a film, then provide a rating. The CAT, introduced in 1982 and widely used since, is similar.
“I feel like there are a lot of things about this approach that are not consistent with what I learned about educational measurement. We don’t often use subjectivity to measure in education,” Long said. “A popular assessment that’s been used for 40 years without much evaluation or consideration for validity seems like it could be dangerous to me.”
Long’s study, co-written with Jue Wang of the University of Miami, was published in the journal Educational Psychology Review. The researchers analyzed the studies, finding that most focused on the reliability and consistency of judges. While some were rated as experts and some as quasi-experts in their fields, the studies mostly focused on the consistency of the judges’ ratings. The analysis showed the judges were reliably consistent about 70% of the time. But that reliability cannot be assumed to mean the assessment is valid.
“What I found was the 84 studies are very different in their fields,” Long said. “Some were in writing or the arts, or in science. Some are for elementary, some for college. And the raters are very different, too, in their characteristics, like how long they’ve worked in the field, their interests and their motivations.”
The creativity studies also rarely divulged whether judges gave a high score to a work they observed because they liked it, because they thought it truly met the criteria of creativity or both. That wide-ranging subjectivity and lacking consistency in approach means caution should be used in pronouncing subjective creativity assessments completely valid, according to the researchers.
The CAT is one of three dominant approaches in creativity research, along with creative and divergent thinking tests and questionnaires used to assess different aspects of creativity, such as creative activity and creative self-beliefs. Consensual Assessment Technique is the only one that relies on subjective judgments to rate creativity of a product or work.
Long and Wang wrote that the goal is not to discredit CAT or discourage its use, but to encourage further study and better understanding of the assessment and how it is used. That could lead not only to enhancing creativity research and understanding its role in education, but in helping develop curriculum and methods for educators to teach creativity and identify it in students. Currently, students are identified as creative most of the time by teachers who judge them as such or as having the potential to be creative, based on classroom observations.
“When we use this approach, we need to be aware of its weaknesses and use other approaches as well,” Long said. “At the same time, we need to study this approach more and get a better idea of how it works to better understand how it can be applied for classroom use.”
To the classroom use point, Long — who has previously published research on the pros and cons of the most popular classroom creativity assessments in education — hopes to continue research into developing creativity models to study students holistically, focusing on their strengths. That could help translate creativity research into tools educators can use to identify where a student’s creative potential is strongest, and build from that, while also providing support in areas that may be lacking.
“I want to do that in a modern way, with the best available technology. I don’t want teachers to feel like this is one more thing they have to do,” Long said. “Every student can be creative, but we need to support them. The tests shouldn’t be to see who’s creative and who’s not, but what each individual’s strengths are and how best to support them.”
Image credit: Pexels.com