Beyond the Output: Cultivating Critical Data Literacy with AI in the Math Classroom

Inspired by the World Economic Forum's emphasis on analytical thinking as a skill of the future, our recent math lesson aimed to equip students with the tools to critically evaluate information in the age of AI. Our goal was to enhance their understanding of core mathematical concepts and the limitations of AI in the classroom, demonstrating that while AI can generate content, it may not always deliver the precise results we need. This focus on verifying AI-generated information against original sources was central to the lesson.

The premise was straightforward: students, having just completed an assessment requiring them to write a two-page report from a given data summary, were tasked with asking AI to perform the same job. Their initial interest in seeing how AI would tackle their recent challenge was clear.

Unpacking AI's Strengths and Stumbles

This hands-on experimentation quickly provided some valuable insights. Students fed a limited data summary (on White Cell Count and Lean Body Mass in athletes) to various AI platforms, requesting a two-page report. The subsequent analysis of these AI-generated documents offered practical lessons in Accuracy, Bias, Completeness, Relevance, and Ethics (ABCRE), alongside the phenomenon of hallucination.

One of the direct discoveries was the AI's tendency to hallucinate written information. Despite the original summary containing no details on athlete ages, heights, or weights, reports from platforms like MagicSchool confidently incorporated these into their narratives. As one student observed, "Gave info about ages of people in the survey that was not in summary data" I observed this myself in my test example where it had given me information about "heights and weights" which I had not given it. This showed that AI, when faced with gaps, often invents details to create a seemingly coherent story, regardless of the information not being present on the data summary document. Some outputs also pushed reports "beyond Grade 8" complexity, despite specific instructions. This prompted students to question the accuracy and relevance of the report if it had included information not given.

Interestingly, where direct numerical transcription was possible, like basic counts or means, or ranges the AI generally got it right. However, more complex mathematical analysis proved challenging. A key learning moment came when students reviewed the AI's attempt at interpreting a line of best fit. 33% of students noted that the equation for the line of best fit was omitted from the AI generated report, despite being present in their original summary statistics. This highlighted that even seemingly objective mathematical outputs from AI need careful checking.

The Nuance of Data and the Power of Context

The lesson became more involved as we explored the AI's approach to completeness and relevance. While AI reports did attempt to draw a connection between White Cell Count and Lean Body Mass, these connections were often superficial. Students critically pointed out that while 2/3 of the time the equation of the line of best fit 'y=0.793x+56.3' was present, a discussion on causation versus correlation was consistently missing. This highlighted that the AI generated report may be incomplete. Students also noted, that because it only had a data summary, it could only present surface-level mathematical relationships without the necessary contextual depth for meaningful understanding. Students readily identified the need for additional contextual data, such as "sport type and their body fat percentage," to provide a more thorough analysis.

The discussion around bias and ethics also offered valuable insights. Students noticed that the AI reports often emphasized reasons for correlation using information not provided in the original summary – for example, making assumptions about athletes' "peak physical condition" or training frequency. This revealed how AI can implicitly introduce bias by filling narrative gaps with external, unverified assumptions, impacting the ethical presentation of data.

The potential negative impacts of such inaccuracies were a significant takeaway. Students articulated that flawed AI reports could lead to "misinformation" and even "negatively impact athlete health" if decisions were made based on incorrect interpretations of white cell counts. This directly connected the abstract concepts of AI flaws to practical, real-world consequences, emphasizing the importance for accuracy in data reporting.

Pedagogical Reflections and Forward Steps

This lesson confirmed the benefit of hands-on experimentation in teaching AI literacy. Students were engaged, challenged, and gained a more nuanced understanding of AI's capabilities and limitations than any theoretical discussion could have provided.

One pedagogical reflection from this experience reinforced the importance of explicitly addressing complex terminology. While the core ideas of Accuracy, Bias, Completeness, Relevance, and Ethics were central to the lesson, considering how we introduce and reinforce these ideas and terms to our students is something I'd do more of next time.

The challenges students faced, such as when using a"safe sandbox" AI like Magicschool eventually yielding a report despite limitations (raising questions about 'manipulating' even designed systems), or other platforms generating elaborate hallucinations, became clear teaching moments. These instances underscore the importance of careful critical thinking regardless of the AI tool used.

Ultimately, this lesson was a meaningful step in cultivating critical data literacy. Students didn't just learn about AI; they actively practiced verifying its outputs against original sources, analyzing its biases, and considering the ethical implications of data interpretation. This hands-on, problem-solving approach to AI integration helps our students become informed, confident, and responsible digital citizens, capable of navigating the complex data landscapes of the future.

How are you integrating Data Literacy in your classroom? I'd love to hear from you.

If you would like a copy of some of the resources used in this lesson, please message me on LinkedIn.

There is a video of the googlesite I created to go along with this lesson here.