When AI stumbles: Navigating the risks of over-reliance

In today’s workplace, artificial intelligence (AI) is ubiquitous, streamlining workflows, optimising supply chains, personalising customer experiences, and even assisting in critical decision-making. The once-busy hum of human effort has given way to the seamless precision of automation. But while machines have transformed the modern office and factory floor, they bring their own set of vulnerabilities. What happens when the technology falters?

AI is not infallible, and the consequences of over-reliance are already evident. Mishaps abound, from software glitches disrupting operations to systems making erroneous decisions.

Ravi Mishra, head HR, BITS Pilani, recalls the Y2K crisis of the late 1990s, when fears of global IT breakdowns due to date-storage errors spurred a flurry of pre-emptive action. “Failures push us to think more holistically, making systems more robust,” Mishra explains.

“Failures push us to think more holistically, making systems more robust.”

Ravi Mishra, head HR, BITS Pilani

But today’s challenges are more nuanced and widespread. For example, a recent Microsoft software failure disrupted airline operations and hotel bookings globally, leaving passengers stranded for hours. While engineers resolved the glitch by midnight, the event highlighted a troubling reality: organisations that rely exclusively on AI risk paralysis when systems fail.

The fragility of automation

Consider the production floor, where precision is paramount. Automation has replaced manual checks with machine-led processes, ensuring that inputs are accurate and consistent. Yet, this efficiency can breed complacency. Operators who once verified inputs manually often abandon these practices altogether, leaving operations at the mercy of machines.

“When the system fails or makes an error, and operators have forgotten how to manually verify inputs, the entire production cycle grinds to a halt,” warns Manish Majumdar, head-HR, EMS, Centum Electronics,.

This reliance on automation becomes even more precarious when experienced personnel exit the workforce. Their departure takes away institutional knowledge—the “muscle memory” of manual operations—which is challenging and expensive to rebuild.

“When the system fails or makes an error, and operators have forgotten how to manually verify inputs, the entire production cycle grinds to a halt.”

Manish Majumdar, head-HR, EMS, Centum Electronics

In HR, similar risks emerge. Even routine tasks such as payroll processing are not immune to the pitfalls of automation. Previously, teams cross-checked benefits, deductions, and attendance using manual methods. With the advent of AI systems, these processes became automatic—and opaque. When errors occur, many employees no longer know how to identify or correct them.

Majumdar recounts: “Over time, my recruitment team forgot how to screen candidates manually. They didn’t know what to look for in a CV or how to ask the right questions during interviews.” This over-reliance on AI diluted the team’s ability to provide human insights, such as understanding industry trends or gauging cultural fit, which are critical for effective hiring.

Learning from AI failures

AI’s potential is undeniable, but its shortcomings are inevitable. The solution lies in preparation. Organisations must plan for disruptions proactively, ensuring that automation complements human expertise rather than replacing it. Adopting a hybrid approach—where automation is combined with manual oversight—can ensure continuity during AI failures. Equipping employees with the skills to operate both manual and automated systems is equally critical, as it reduces dependency on technology. Furthermore, conducting a thorough risk analysis to identify vulnerabilities in AI systems before deployment can help address potential issues proactively. Finally, establishing robust monitoring frameworks with designated personnel for early detection of failures can safeguard operations from significant disruptions.

Mishra adds that maintaining diversity in AI tools is equally important. Using multiple providers, such as Google and Microsoft, ensures fallback options during disruptions. He also advocates for continuous improvement, likening AI’s development to a child learning through feedback and testing. “Failures,” he notes, “are opportunities to make systems more resilient and effective.”

The high stakes of detachment

The risks of over-reliance on AI extend beyond operational disruptions to cultural and strategic challenges. Blind trust in algorithms can erode employee engagement, as individuals feel disconnected from the processes they oversee.

This detachment is particularly dangerous in customer-facing roles, where human intuition and adaptability are irreplaceable. For instance, automated chatbots may handle queries efficiently, but they lack the empathy and judgment needed to manage complex or emotionally charged interactions. When these systems fail or produce unsatisfactory outcomes, employees must step in—but without a deep understanding of the underlying processes, their interventions may be ineffective.

In manufacturing, finance, and HR alike, organisations must guard against allowing technology to obscure the human element. The most successful companies are those that strike a balance, leveraging AI for efficiency while retaining the adaptability and creativity of their workforce.

Reclaiming the human edge

In a world increasingly dominated by AI, the human edge lies in adaptability, critical thinking, and process expertise. These qualities enable organisations to respond effectively to disruptions, turning setbacks into opportunities for growth.

By fostering a culture of engagement, continuous learning, and process ownership, businesses can ensure that their teams remain prepared for the unexpected. Mishra’s reflections on AI resonate as both a caution and a call to action: “AI evolves like a child—through feedback, testing, and updates. Failures are inevitable, but they’re also opportunities to strengthen systems and build resilience.”

As organisations navigate the complexities of AI adoption, one thing is clear: machines can augment human capabilities, but they cannot replace them. The key to thriving in an AI-driven world is not to relinquish control but to stay connected to the processes that underpin success.