Bias In The Machine: The Lack Of Equity And Inclusion In AI Tools

In November and December of 2022 when I started receiving some odd student submissions in my English 12 course I needed to figure out what was going on. At first I was annoyed, then intrigued, then annoyed again, maybe even a little angry. I made a lot of mistakes in those first two months with how I addressed students using ChatGPT. I dove in and did what I could to figure out what was happening, how the tools worked and how to address learners who were using them inappropriately. Within two more months I was facilitating table talks and sharing some of what I was learning at DLAC in 2023. That led to me writing some guides on how we might think about using them in education. Even after a year, I think they still hold up, but something is still bothering me.

Part of my job involves writing curriculum. It once took me a full calendar year to write one of the courses I teach now. How much time could I save if I used generative AI as an assistant? It became a big part of my day and continues to seep into many aspects of my life. I’ve now presented on some aspect of this technology over 50 times throughout the country and online and for as much as things evolve, some things have not changed. These tools are inherently biased, and I have enormous concerns about the level of adoption educators are navigating.

Don’t get me wrong. I’m a proponent of embracing these Large Language Models (LLM) in our work. I use them every day. I’m also a skeptic. It is in our best interests to figure out everything we can about these tools; the implications they may have on learning and the learners themselves. My guiding philosophy is that the default version of anything sucks and it is only when we customize something to suit our individual needs, or the needs of our learners, or organizations that we find its usefulness. So what are these default versions of LLM giving us?

Before I answer that, please consider for a moment the last time you experienced a new technology, that by default, supported and lifted up marginalized communities. Maybe you thought of one, I can’t. There is implicit bias baked into just about everything humans make. It’s not always even intentional, but it is unavoidable. Think about these LLM. Who selected the programmers, what biases do they come with? Choices made when coding often further perpetuate existing systemic inequities. Also think about how these models were trained. We uploaded a massive amount of information humans have written on all manner of topics. Biased humans, many with a propensity for racism, misogyny, homophobia, ableism, and more. We can put all the guardrails we want on these tools but at the end of the day, so much of what we’ve written as a species is littered with content that does not align with our current values.

We also know that these LLM have an inclination toward sycophancy. They tell us what we want to hear. Some of that is by design and some of it gets worse as the systems are updated. The LLM will also “make assumptions” about the person doing the prompting. This aids the tools in determining just how thorough the response should be. Above all else, humans want to see responses that align with their existing beliefs.

So why do I still encourage people to use generative AI when all of these things are true? It’s because I believe that we can prompt the LLM away from those tendencies by creating a clearer picture of what we are looking for and are not looking for. When I prompt these tools, I ask them to incorporate elements of Social Emotional Learning (SEL) and Universal Design for Learning (UDL) in resources for students; I ask for anti-racist examples with multiple perspectives included; I ask for the use of inclusive language that demonstrates care and support, and too many other prompts to get into with this writing. I also ask these tools to avoid suggesting what I don’t want. I don’t want antiquated and debunked practices around learning styles or left/right brain thinking, and so many other educational myths that persist. It also matters what we tell the tools about ourselves. If I tell them I am an educator, my responses will be more detailed and there are more ways to prompt for more comprehensive and inclusive results.

I quite literally include aspects of SEL in my prompts to teach the LLM within the context of each chat, the kind of language I expect in return. I want to see these five fundamentals incorporated into assignment directions and reflections: self-awareness, self-management, social awareness, relationship skills, and responsible decision-making. I do the same thing with UDL principles. If I do not see those things, I push back and have it try again.

When I was writing a 7th grade activity for students writing nature journals, I asked ChatGPT for a list of poets who write about nature. It gave me a nice long list, that included Ralph Waldo Emerson, and guess what? A bunch of other white men. That’s not good enough for me, or for my learners. In Minnesota we have a large Somali population, Latino population, Hmong population, we have 11 reservations representing Dakota and Anishinaabe peoples, I cannot put a lesson like this in front of learners who come to us across the state that do not include people from those groups, so I asked. It gave lists of more writers to me for each of those groups and I kept going, but there was one more group of people missing, women!

How many educators having AI tools writing lesson plans for them right now are continuing to push back until they see the faces of their students reflected in the lessons? I don’t know, but if there are some who are not, then this technology is setting us back. We know that not everyone is asking within their prompts to use inclusive language, give multiple perspectives, anti-racist examples (yes, add those things directly to your prompts!)

You will also get better responses if you tell the LLM more about your setting, your student population, and yourself. Lie if you need to. Tell it you are a national teacher of the year, known for your engaging lessons and demonstration of care for your students, etc. etc. Make yourself sound as important as you can and you will get better responses. Here’s an example I created just using something simple like making a BLT. Tell it about your setting, are you hybrid, asynchronous, those details help. Tell it that you are working with struggling students who are having difficulty understanding a specific benchmark and you will get more context and strategies. But be careful there too! It will often suggest things that have been debunked like learning styles.

None of this should surprise us, but it’s something we might not have thought of when we just jump in, click around and make some requests to make our workload a bit lighter. If we trained these tools on a bunch of our own biases, then of course that’s what these tools “think” we sound like and it will parrot those same biases back to us by default...unless we tell them not to. It’s up to us.

Do I care which of the hundreds of AI applications you choose or use? Not really, though some are better than others. What I care most about is that you do not turn your thinking over to a machine; and that we are responsible for the material we put in front of learners through a thorough vetting process by subject matter experts with training and background in equity and inclusion initiatives. I care that we are not giving a platform and agency to machines before we even make sure marginalized groups of humans have representation. There are so many other tasks we are giving AI tools to do for us that need thoughtful consideration (observations, data analysis, accommodations/modifications, etc.) Default responses from AI are not what we need, we need thoughtful, caring professionals that are going to use these tools collaboratively and responsibly. That can only be accomplished through much discussion and consideration about the problem that we are trying to solve when we look to these tools. The marketers of these tools aren’t putting those practices at the forefront of what they are doing, that responsibility falls on you. These tools have the potential to change lives; only you will determine if that’s for better or worse.

Jon Fila is an award winning educator and currently teaches English at Northern Star Online. He has written several books on the use of AI in education for students and educators. He provides workshops, and other trainings on how we should be thinking about incorporating these tools into our practices. You can find out more at jonfila.com, or learn more about his online modules to address these kinds of issues at inclusiveaistrategies.com.