Bias in AI systems
Executives in nonprofits and local government already know bias is part of human decision-making. You see it in who shows up to public meetings, whose stories get told in funder reports, and whose complaints reach your desk. Without thoughtful practices and policies, it would be easy to let our natural biases influence the way critical decisions are made.
When you ask a computer instead of a person, it can feel as if you’re getting an unbiased response. But AI is built out of its training data. That training data is either directly made by humans (e.g. the training data for Large Language Models like ChatGPT and Claude is human-written content online) or data traces from human activity. Further, what it is built for, how it was built, and how it is implemented all are subject to human biases.
When you add AI into funding, eligibility, hiring, or outreach, you’re plugging statistical patterns—learned from people—into work that touches real people and their lives. There is a very real possibility that using AI here is not avoiding human biases, but rather laundering those biases and presenting them back to us.
Bias can sneak in at every stage: how you define the problem, which data you collect, how the model is trained, which prompts staff use, and how the system shows up in daily work.
Where bias creeps into AI: a quick tour
We will look at four major ways that bias can sneak into our AI-augmented work:
The data that trains the system
The target the system tries to predict
The prompts and use cases people design around it
The way humans oversee and interpret its output
I’ll walk through each briefly with some relevant examples.
1. Biased training data: whose world does the AI “see”?
AI systems learn patterns from data. The data comes from people, their activities, and how and what they choose to measure. If those examples reflect an unequal world, the model learns that inequality as “normal.”
For LLMs like ChatGPT and Claude, training data comes largely from the public internet and other large text collections. That material over-represents powerful institutions and dominant groups and under-represents many communities that nonprofits and local governments serve. Researchers have warned that this leads language models to echo harmful stereotypes and one-sided viewpoints.
For classifiers and other machine learning algorithms, training data sets are subject to lots of human decisions: what data is included, what is measured and how. The models only know patterns, and they don’t know which are discriminatory. For example, because the training data didn’t include enough data to allow it to distinguish between the two, image classifiers have labeled images of Black people as “gorillas.” Although this was almost surely not the model developers’ intent, it is the result of them not reviewing the training data set for representativeness or testing the model once it was trained.
Why this matters for you:
If your AI learns from records that already reflect unequally distributed services, arrests, complaints, or donations, it can quietly treat that uneven pattern as the “right answer” and reinforce it.
2. Biased targets and labels: what you ask the AI to optimize
Even with perfect data, you can bake in bias when you decide what the system should predict.
Say for example, a funder scores community groups on “grant return on investment” using previous grant size as a proxy, even though many grassroots groups never had access to large grants in the first place. This would continue the existing pattern of exclusively large organizations getting large grants and the funder could develop a false sense of security, believing they are eliminating bias using math.
Why this matters for you:
The goal you encode into an AI (cost, grant size, click-through rate) or action the software is intended to prompt (purchase, hire, arrest) can quietly tilt benefits toward already-advantaged groups. You need to ask: “Predicting what, exactly. How is that measured and who does that help?”
3. Biased error: who bears the wrong answers?
Answers can be biased, but so can error:
Facial-analysis tools used for emotion recognition read Black men’s faces as angrier or more negative than white men’s faces.
The Gender Shades study found error rates in facial recognition as high as 30% percent for Black women and less than one percent for white men.
If tools with biased error like this were used, for example to monitor people at work, the higher error rates can compromise people’s careers.
For LLMs, a similar pattern can emerge: the overall error rate might look acceptable, but made-up case studies or missing context may cluster around certain topics (for example, communities with less online coverage, or policy areas with fast-changing law).
context
Why this matters for you:
Bias is not just about biased outputs. A low average error rate can still hide disparate impact.
4. Biased prompts and workflows: how people steer the system
For large language models like ChatGPT, Co-Pilot, and Claude, the prompts and workflows around it can introduce bias.
A grant officer might ask an LLM: “Write a justification for funding the strongest applicants,” and then define “strongest” in a way that reflects their own preconceptions, including some unconscious biases.
You might give an LLM a bunch of information about your program outcomes and ask “how did our program help?” It will tell you that it helped, even if it has to hallucinate to do it. It will not respond with something like, “you should have asked me ‘what are the outcomes of this program?’ or ‘please evaluate this program’ instead of ‘how did it help?’”
Prompts also shape what the model leaves out. If you never ask for impact by race, disability, language, or neighborhood, the model will not volunteer that analysis. That creates a form of “biased omission.”
Why this matters for you:
Bias does not only live in the model; it also lives in the questions your teams ask it to answer.
5. Biased oversight: when “human in the loop” is not enough
A common safeguard is to “keep a human in the loop:” to have a staff member review AI outputs before acting on them.
Automation bias. People tend to over-trust recommendations from an AI system, especially under time pressure. Reviews of decision-support tools in health care and other fields show that users often follow automated suggestions even when errors are obvious.
“Moral crumple zones.” Organizations sometimes design workflows so that, when something goes wrong, blame falls on the human reviewer instead of the system design or vendor contract. When problems arise, they can “fix” the problem by replacing the human reviewer and avoid acknowledging problems in the algorithm, policies, or workflow.
Unsafe or biased advice can slip through oversight, especially when staff receive little training and feel pressured to accept algorithmic recommendations.
Why this matters for you:
A human reviewer who has no time, no training, and no authority to push back becomes a rubber stamp, not a safeguard.
Takeaways for nonprofit and public-sector executives
1. Treat AI as a policy choice, not a gadget
Before approving any AI project, ask three plain questions:
What decision will this influence? Hiring, eligibility, outreach targeting, performance review, grant scoring, content drafting?
What values do we want it to uphold? Equity across race and neighborhood, language access, disability inclusion, harm reduction?
What would “harm” look like here? A family denied benefits, a community skipped in outreach, a staff member mis-labeled as “low performer”?
Write down the answers and keep them with the procurement or project charter. That gives you a yardstick to evaluate vendors and internal experiments.
2. Ask vendors (and your own teams) concrete bias questions
For any AI system you buy or build, require clear answers to questions like:
Data: What data trained this system? From what years, locations, and institutions? Whose experiences does it miss?
Target: What exactly does the model predict or score? Is that the outcome we actually care about, or just a convenient proxy?
Error distribution: Do you have evidence on how error rates differ by race, gender, age, disability, or neighborhood? If not, who will test that and when.
Update process: How often do you retrain or update the system? Who decides what counts as “good enough”?
Appeals: If someone believes the system harmed them, how can they contest the decision?
3. Start with “assistive, not adjudicative” use cases
When possible, begin with uses where the AI helps staff decide rather than decides for people.
Safer early use cases include:
Drafting first versions of emails, reports, or grant narratives that staff then edit.
Summarizing long documents for internal briefings.
Brainstorming outreach ideas, communication strategies, or survey questions.
Rewording existing content for another audience or purpose
Use cases that deserve far more caution include:
Eligibility screening
Risk scores for individuals or neighborhoods
Hiring or interview filters
Performance ratings
Be cautious about high-stakes decisions because subtle LLM errors—like omissions or mischaracterizations—are hard to detect and hard to fix after the fact.
4. Give humans real power to resist the AI
If you keep a human in the loop, make sure they are not just a “moral crumple zone.” To do that:
Design for disagreement. Build workflows where staff must actively accept or reject key AI recommendations, and capture reasons when they disagree. Ensure that disagreement does not punish staff, either socially or with more or more tedious work.
Train for skepticism. Teach staff about hallucinations, biased error, and automation bias so they know when to look for and understand the stakes.
Protect dissent. Make it clear in policy that disagreeing with the AI is welcome, not punishable, especially in high-stakes cases.
5. Build AI governance
For every AI tool in use, at least:
Keep a one-page “AI factsheet” with: purpose, data sources, target outcome, high-risk groups, and an owner inside your organization.
Schedule a regular review (for example, yearly) to ask: Is this still aligned with our mission and equity goals? Do we have any complaints or red-flag cases?
For use cases with high stakes and mission impact, more safeguards and monitoring will be needed; we’ll talk about that in another post.
6. Connect AI risk to your existing equity and ethics work
You already manage bias in human decisions: through hiring policies, civil-rights compliance, language-access plans, and community engagement. AI bias belongs in those same conversations, not in a separate “tech” bucket.
Practical moves:
Add AI systems to your equity impact assessments and civil-rights reviews.
Ask your HR, legal, and program leads to help set guardrails for high-risk AI uses.
When you report impact to funders or the public, include how you checked AI-enabled programs for fairness.
Closing thought
AI can help your teams work faster and uncover patterns that would be hard to see by hand. But it does not wash away human bias. It compresses that bias into code and probabilities, where it can become harder to spot.
As an executive, your job is not to perfect the algorithms. Your job is to decide where AI belongs in your organization, who it can help, who it might harm, and what values it must never override.
—
LLM disclosure: I asked both Claude (before the 4.5 drop) and ChatGPT 5.1 to draft this post based on a chapter from my upcoming book several times. It kept coming out structured in a way I didn’t like. To fix this, I drafted the introduction (including the signposting at the end of the introduction) myself. This worked perfectly to guide the structure.