Are We Trusting AI Too Much? New Research Calls for Real Accountability and Transparent Decisions

What if your bank froze your account overnight and the only explanation was “our system flagged unusual activity”? Or an AI assistant suggested a diagnosis that didn’t quite add up—but no one could say why. Would you accept those outcomes, or would you want to look under the hood?

A new study out of the University of Surrey—reported by ScienceDaily—posits that we’re trusting AI systems far more than we understand them, especially in life-altering arenas like banking, healthcare, and criminal justice. The research urges a wholesale shift in how we design, evaluate, and govern AI, moving away from black-box models that demand blind faith and toward systems that can actually show their work.

Read the ScienceDaily coverage: Are we trusting AI too much? Research highlights need for accountability

In this post, we’ll unpack what the study is warning about, explore why the explanation gap is so dangerous, and outline the concrete steps organizations can take today to build AI that earns trust the right way—through transparency, accountability, and meaningful human oversight.

The University of Surrey’s Warning: Blind Trust in Opaque AI Is a High-Stakes Gamble

The study’s core message is simple and urgent: as AI systems make decisions that can affect your health, your freedom, and your financial stability, it’s no longer acceptable for them to be inscrutable. The research highlights:

A growing pattern of AI systems producing decisions without explanations users can understand—leaving people confused and vulnerable.
Real-world harms, including healthcare misdiagnoses and erroneous banking fraud alerts that can be life-threatening or financially devastating.
A technical and governance gap: even where detection accuracy is strong (for example, in card fraud with extremely imbalanced data—fraud can be roughly 0.01% of transactions), too many systems can’t explain why a particular decision was made.

That last point is crucial. The imbalance in fraud data doesn’t just make learning hard; it also complicates how we communicate and justify model behavior. The study calls for an immediate shift in how AI models are built and judged. Black boxes—systems whose inner workings can’t be inspected or explained—are no longer “good enough” where the cost of error is severe.

The Problem with Blind Trust: Why “Black Box” AI Persists

We didn’t get here by accident. Black-box AI has flourished for understandable reasons.

1) The performance trap

Organizations are incentivized to chase performance metrics—reduce false positives, increase precision, bump up conversion, cut fraud. Complex models often do this well. But in high-stakes settings, performance without clarity is a liability, not a win.

2) Data complexity and scale

Modern datasets are messy, high-dimensional, and dynamic. Deep models can capture patterns that humans can’t articulate. That’s valuable—but it doesn’t absolve teams from explaining model behavior in human-relevant terms.

3) Human psychology: automation bias

Humans are prone to over-trusting automated outputs, especially when systems look confident or are wrapped in a polished UI. This “automation bias” magnifies the risk of opaque AI in critical decisions. Learn more: Automation bias.

The result is a precarious equilibrium: systems appear to work, people trust them, harms are under-reported or hard to contest, and accountability becomes diffuse. The Surrey study is a call to break that cycle.

Fraud Detection’s Paradox: Accurate Flags, Opaque Reasons

Fraud detection is a poster child for the explanation gap. When fraud represents about 0.01% of all transactions, most data shows “normal” behavior. That imbalance creates two interlocking problems:

Learning challenge: It’s hard for models to learn the rare class (fraud) without overfitting or missing subtle signals.
Communication challenge: When a model flags a transaction, “why?” is often buried inside complex interactions across hundreds of features.

Many fraud models do achieve high precision—that is, if they flag something, it’s often truly fraudulent. But that’s not the same as being able to articulate the rationale in a way a case agent or customer can understand.

This matters because:

Customers deserve actionable reasons when their funds are frozen or their card is declined.
Analysts need to triage alerts quickly and accurately.
Institutions face regulatory expectations to provide meaningful explanations and appeal pathways.

If your model is a bouncer that just points and says “No,” you don’t have a trustworthy system—you have an obstacle course.

For a refresher on evaluation in imbalanced settings, see Precision and recall.

Why explanations matter in finance

Customer rights: In many jurisdictions, people have the right to meaningful information about automated decisions, including under frameworks inspired by the GDPR’s Article 22. See: GDPR Article 22.
Operational speed: Clear explanations reduce back-and-forth, speeding up case resolution.
Model improvement: Transparent rationales expose failure modes and drift early, enabling faster fixes.

Healthcare and Criminal Justice: When the “Why” Saves Lives and Livelihoods

Healthcare AI can assist with diagnosis, triage, and treatment decisions. But a misdiagnosis without an explanation doesn’t just slow down care—it can send clinicians in the wrong direction. The safest path is augmented intelligence: AI that supports, not supplants, clinician judgment, and that offers transparent reasoning or evidence.

Clinical expectations: Clinicians need evidence pathways—key features, similar cases, and counterfactuals (“If this symptom were absent, risk would drop from X to Y”). Interpretability is not a nice-to-have; it’s part of clinical safety.
Regulatory context: The FDA continues to refine its approach to AI/ML-enabled medical devices, including the need for consistent performance and change control. More here: FDA on AI/ML-enabled medical devices.

In criminal justice, risk assessments and pattern recognition tools have been criticized for opacity and bias. When liberty is at stake, opaque algorithms can entrench systemic inequities. For background, see ProPublica’s seminal piece on risk assessment controversies: Machine Bias.

In both domains, the bottom line is the same: trust must be earned through evidence and explanation—not demanded.

From Trust to Trustworthiness: Redefining “Good AI”

Trust is an outcome. Trustworthiness is a design choice.

Leading frameworks emphasize that trustworthy AI must be:

Transparent and explainable: Not every detail, but enough for users to understand and challenge decisions.
Accountable: Clear responsibility for outcomes, with logs, audits, and remedies.
Robust and secure: Performs reliably across conditions; resists adversarial abuse.
Fair and respectful of rights: Minimizes and monitors harmful bias; upholds privacy.
Human-centered: Keeps a human in the loop where stakes are high; supports contestability.

Useful references: – NIST AI Risk Management Framework: NIST AI RMF – OECD AI Principles: OECD AI Principles – The White House Blueprint for an AI Bill of Rights: AI Bill of Rights

Building Explainability That Actually Explains

“Explainability” isn’t one thing. The right approach depends on the audience and the decision.

Global vs. local explanations:
Global: How the model behaves overall (feature importance, monotonic relationships).
Local: Why this individual prediction happened (contributions, counterfactuals).
Intrinsic vs. post-hoc:
Intrinsic: Using interpretable models (e.g., generalized additive models, monotonic gradient boosting) where possible.
Post-hoc: Applying explanation tools to complex models (e.g., SHAP, LIME).

Popular tools and methods: – SHAP: additive feature attributions with solid theoretical grounding. SHAP docs – LIME: local approximations around a point of interest. LIME on GitHub – Counterfactual explanations: “What minimal change would have flipped the decision?” Wachter et al. (2017)

But beware pitfalls: – Explanation illusions: Pretty plots that don’t faithfully reflect model logic. – Instability: Explanations that change wildly with small input perturbations. – Incomprehensibility: Overly technical output that end-users can’t act on.

Good explanations are: – Faithful: Accurately reflect the model’s reasoning. – Stable: Small input changes don’t yield wildly different rationales. – Sparse: Surface a few key drivers, not a laundry list. – Actionable: Suggest clear next steps (e.g., “Verify merchant location,” “Order lab test X”). – Audience-appropriate: Different views for clinicians, analysts, and consumers.

Designing for Accountability: What Teams Can Implement Now

Accountability is a system property, not just a model feature. Here’s how to build it into your stack.

For product and operations teams

Documented appeal pathways:
Provide customers with reasons, contacts, and timelines for review.
Track reversal rates and time-to-resolution as KPIs.
Human-in-the-loop policies:
Require human review for high-stakes or borderline cases.
Escalate when explanations are low-confidence or contradictory.
Decision logging:
Store inputs, model versions, thresholds, explanations, and reviewer notes.
Enable forensics and root-cause analysis.
Model Cards and user-facing transparency:
Summarize intended use, limitations, performance by segment, and known risks.
See: Model Cards
Incident response:
Treat harmful outcomes like security incidents: contain, learn, and prevent recurrence.

For data science and engineering

Imbalanced learning strategies:
Cost-sensitive training, focal loss, resampling with care (avoid leakage), anomaly detection hybrids, and one-/few-shot approaches.
Calibrate probabilities; don’t ship raw scores. See: Calibration in scikit-learn
Appropriate metrics and validation:
Focus on AUPRC, class-conditional error, calibration, subgroup performance, and decision-cost curves.
Test stability of explanations across seeds and perturbations.
Interpretable-by-design where feasible:
Consider monotonic constraints (e.g., risk should not decrease as evidence increases).
Explore GAMs or tree-based models with constraints for structured data.
Explanation rigor:
Evaluate explanation fidelity against the underlying model.
Provide counterfactuals and example-based rationales alongside feature attributions.
Dataset documentation:
Track provenance, known gaps, and consent constraints. See: Datasheets for Datasets

For executives and risk leaders

Risk-tiering and governance:
Classify AI systems by impact. Impose tighter controls for high-stakes use.
Independent assurance:
Periodic third-party audits for fairness, robustness, and security.
Board-level reporting on AI incidents and mitigations.
Policies aligned with emerging standards:
NIST AI RMF integration into enterprise risk.
Consider alignment to ISO/IEC 42001 (AI management system).
Culture and incentives:
Reward teams for reducing harm and improving explainability—not only for chasing headline metrics.

For regulators and policymakers

Encourage transparency and contestability by design:
Clear rights to an explanation and appeal in high-stakes uses.
Require record-keeping and impact assessments:
Ongoing monitoring for drift, bias, and performance degradation.
Harmonize with global initiatives:
For context on Europe’s direction, see the EU AI Act overview by the European Parliament.

Measuring “Explainability”: How Do We Know It’s Enough?

You can’t improve what you don’t measure. Consider an explainability scorecard that blends technical and human-centered criteria:

Fidelity: How well does the explanation reflect the model’s true decision logic?
Stability: Do explanations remain consistent under small input changes?
Sparsity: Are only the most relevant factors surfaced?
Consistency: Do similar cases yield similar explanations?
Simulatability: Can a domain expert predict the model’s output from the explanation?
Actionability: Do explanations lead to correct, faster decisions by users?
Comprehension time: How long does it take users to understand and act?
Confidence calibration: Do explanations align with uncertainty estimates?
User trust and satisfaction: Measured through structured surveys after decisions.
Outcome impact: Do better explanations reduce appeals, errors, and time-to-resolution?

Run A/B tests where some reviewers see explanations and others don’t; measure speed, accuracy, and reversals. Explanations should earn their keep.

The Roadmap: 10-Step Checklist for Trustworthy, Accountable AI

1) Define the stakes and risk tier of each AI use case.
2) Choose the simplest model that meets performance and explainability needs.
3) Design explanations for the audience from day one (not as a bolt-on).
4) Document data provenance, imbalance handling, and known gaps.
5) Use appropriate metrics: AUPRC, calibration, subgroup error, and cost curves.
6) Implement human-in-the-loop and escalation for high-stakes or low-confidence cases.
7) Provide user-facing reasons, counterfactuals, and clear appeal pathways.
8) Log everything: inputs, model versions, thresholds, explanations, overrides, outcomes.
9) Monitor continuously for drift, bias, and explanation stability; retrain responsibly.
10) Audit independently, publish Model Cards, and report incidents with remediation plans.

What This Means for You (Consumer, Patient, Citizen)

Ask why: If an AI-driven decision affects you, request a clear explanation in plain language.
Keep records: Save notices, timestamps, and any explanations you’re given.
Appeal: Use formal appeal channels and ask for human review.
Seek support: Consumer protection agencies and advocacy groups can help escalate issues.
Learn the basics: Understanding terms like precision, recall, and calibration empowers you to ask better questions.

The Bigger Picture: From “Can It Work?” to “Can It Be Accountable?”

The University of Surrey’s study lands at a pivotal moment. We’ve proven that AI can do extraordinary things. But in high-stakes contexts, performance is table stakes. What truly matters is whether AI can justify itself, be challenged, and be corrected.

Opaqueness is not sophistication; it’s risk. The path forward is not to abandon powerful models, but to upgrade our standards:

Show your work.
Share responsibility.
Support human judgment.

If an AI system can make a consequential decision, it must be able to explain it—and someone must be accountable for it. That’s how we transform blind trust into well-founded confidence.

FAQs

Q: What is a “black box” AI model?
A: It’s a system whose internal decision process is opaque or too complex to understand. Black boxes can deliver high accuracy but struggle to provide clear, faithful explanations for individual decisions.

Q: Are explanations always possible?
A: Yes, but not always equally useful. You can generate post-hoc explanations for most models, but the quality varies. In high-stakes settings, prefer models and methods that support faithful, stable, and actionable explanations—sometimes that means choosing more interpretable architectures.

Q: Does explainability reduce performance?
A: Not necessarily. For many structured-data problems, interpretable models with constraints can match or closely approach black-box performance. Even where there’s a small gap, the safety and governance benefits often outweigh marginal accuracy gains.

Q: What counts as a “good” explanation?
A: One that is faithful to the model, stable, concise, and actionable for the intended user. It should help a person verify, contest, or act on the decision. Ideally, it also communicates uncertainty and suggests next best actions.

Q: How can small teams add accountability quickly?
A: Start with a Model Card, add decision logging, ship user-facing reasons with counterfactuals, create a simple appeal workflow with SLAs, and monitor calibration and subgroup performance. These steps are high-impact and feasible without a large governance apparatus.

Q: What does the EU AI Act require about high-risk AI?
A: The Act (as approved by the European Parliament) emphasizes risk management, documentation, transparency, human oversight, and post-market monitoring for high-risk systems. See the overview: EU AI Act press release.

Q: How do we improve fraud detection without adding bias?
A: Use cost-sensitive learning and calibration, measure subgroup performance, constrain models (e.g., monotonicity) where appropriate, and include diverse, up-to-date data. Pair model improvements with human review and appeal processes to catch edge cases and systemic drift.

Q: Where can I learn more about fairness and accountability research?
A: Explore the FAccT community and proceedings: ACM FAccT. For governance frameworks, review NIST AI RMF and the OECD AI Principles.

—

Key takeaway: The question is no longer “Can AI do it?” but “Can AI show its work—and can we hold it to account?” If your system can change someone’s life, explanations and accountability aren’t extras. They’re the product.

Discover more at InnoVirtuoso.com

I would love some feedback on my writing so if you have any, please don’t hesitate to leave a comment around here or in any platforms that is convenient for you.

For more on tech and other topics, explore InnoVirtuoso.com anytime. Subscribe to my newsletter and join our growing community—we’ll create something magical together. I promise, it’ll never be boring!

Stay updated with the latest news—subscribe to our newsletter today!

Thank you all—wishing you an amazing day ahead!

Are We Trusting AI Too Much? New Research Calls for Real Accountability and Transparent Decisions

The University of Surrey’s Warning: Blind Trust in Opaque AI Is a High-Stakes Gamble