Companies across the globe are now formalising artificial intelligence systems as legitimate workplace personnel, even granting them positions on organisational charts and team rosters. This emerging trend caught the attention of Emma Wiles, a Boston University researcher specialising in workplace automation, when she encountered human resources executives at a conference last October touting AI employment as a productivity accelerator and competitive differentiator. Yet as Wiles and collaborators from Boston Consulting Group investigated this phenomenon more closely, they uncovered a troubling pattern that raises serious questions about how organisations are managing this technological transition.
The core concern centres on accountability and oversight. In experiments spanning dozens of companies, Wiles's team discovered that managers substantially relax their quality controls when they believe an AI system produced work, compared to when they think a human colleague generated it. Presented with identical documents containing deliberate errors, managers caught mistakes more readily when those papers were attributed to human staff members. When the same errors appeared in AI-generated material, many slipped past managerial scrutiny. The psychological mechanism underlying this lapse appears rooted in diffused responsibility; managers seem to unconsciously transfer accountability to technology teams or senior executives who championed AI adoption, rather than maintaining personal ownership of oversight duties.
This phenomenon represents just one dimension of a larger awareness gap afflicting corporate AI deployment. Over recent months, industry understanding of artificial intelligence's limitations has grown incrementally. Organisations now widely acknowledge that AI systems harbour biases against certain demographic groups, that language models generate convincingly presented but factually incorrect information, and that these systems occasionally expose confidential data. However, researchers are identifying increasingly subtle defects that remain largely invisible to corporate decision-makers. The gap between known problems and actual corporate awareness widens daily as deployment accelerates, creating a situation where productivity gains and cost savings promised by AI implementation face erosion from unforeseen complications.
Wiles herself emphasises the scale of this knowledge deficit, noting that even researchers studying artificial intelligence likely understand only a fraction of the difficulties the technology introduces. What academics call "unknown unknowns"—problems no one has yet identified or studied—almost certainly outnumber documented challenges. This epistemic boundary becomes particularly troubling when considering that most commercial users appear largely unaware even of recognised pitfalls. The speed of deployment vastly outpaces the pace of risk assessment, creating a gap that could undermine AI's fundamental business case.
One documented but widely overlooked vulnerability involves AI systems exhibiting preference bias toward machine-generated content. A 2025 research paper identified that AI evaluation models, particularly those used in recruitment, systematically favour resumes composed with artificial intelligence assistance over those written entirely by humans. When scholars spelled out this finding in subsequent publications, some corporate recruiters took notice and sought guidance on process improvement. Jane Yi Jiang, an operations professor at Ohio State University and author of the research, reported that firms inquired earnestly about correcting their recruitment algorithms. Yet she and her colleagues remain convinced that resume bias represents merely one manifestation of a broader pattern of unexamined vulnerabilities that organisations are inadvertently creating through hasty AI integration without sufficient deliberation regarding implications and ingrained biases.
Beyond hiring applications, companies increasingly employ AI systems for high-stakes strategic decisions such as pricing strategy and site selection for new facilities. This application category carries particular risks because AI language models process information through a fundamentally different decision-making framework than humans typically employ. Where human negotiators and business leaders frequently seek cooperative outcomes and mutually beneficial arrangements, AI systems tend to adopt the coldly rational calculus embedded in game theory. This theoretical orientation can produce aggressive strategies that maximise individual advantage while ignoring collective welfare; for instance, an AI system might recommend undercutting competitor pricing to the point where it triggers destructive price wars benefiting no participant. Research suggests that contemporary language models systematically overestimate human rationality, leading them toward outcomes that rational actors would collectively avoid, creating strategic miscalculations with real financial consequences.
The organisational psychology dimension compounds these technical vulnerabilities. In Wiles's broader survey of over one thousand corporate managers, approximately one-third reported that their organisations formally designated AI systems as teammates or employees, while nearly one-quarter stated their companies included these systems on organisational charts with equivalent peer status. One respondent even provided a specific example, noting that their company called an AI agent "Scout" and treated it as a technical equal on team rosters. This semantic shift—relabeling AI systems from "tools" to "employees"—appears to trigger profound psychological changes in how managers approach accountability. The research revealed that managers who view AI as inanimate tools maintain a sense of personal responsibility for outputs, much as they would for any equipment under their supervision. However, managers at organisations that anthropomorphise AI into employee status experience a marked diminishment in this accountability impulse.
The experimental design that revealed this dynamic was elegantly straightforward. Wiles and colleagues presented managers with five documents containing deliberate errors and gave them twenty minutes to identify flaws. Some managers were told the work came from an AI employee, others that an AI tool produced it, and still others that a human colleague created it. The attribution generally made minimal difference in detection rates—except for one critical exception. Managers employed at organisations with formalised AI employee structures caught substantially fewer errors when attributing work to AI employees specifically. This finding suggests that the act of organisational formalisation itself—the chart placement, the job title, the team membership—fundamentally alters human psychology in ways that undermine quality assurance.
The mechanisms driving this psychological shift reflect deeply ingrained management principles. Across centuries of institutional development, scholars and business leaders have cultivated reliable practices for managing human subordinates. When someone on a manager's team makes an error, accountability flows naturally upward; most managers viscerally understand that "if someone on my team makes a mistake, that's on me." This responsibility creates powerful incentives for close supervision and quality control. Managers similarly maintain oversight of inanimate tools and systems, viewing them as resources they must deploy appropriately. Yet AI employees occupy a conceptual grey zone that neither traditional human management nor tool supervision adequately addresses. Organisations have no established frameworks, no cultural intuitions, no developed practices for managing anthropomorphised artificial intelligence. As Wiles observes, this represents uncharted territory: organisations are venturing forward without reliable maps, adapting frameworks developed for entities fundamentally different from AI systems.
The implications extend beyond individual organisations to encompass broader economic and social dynamics. If widespread AI adoption within corporate structures is proceeding without proper oversight mechanisms, quality control frameworks, or understood accountability structures, then the promised productivity gains face significant erosion. Simultaneously, the risks—from biased hiring decisions to strategically damaging pricing choices to information security breaches—could accumulate across economic systems faster than detection and correction mechanisms can function. Malaysia and other Southeast Asian economies increasingly integrating AI systems into corporate operations face particular challenges, as many local firms may lack the research infrastructure and analytical capacity to independently validate AI risks before deployment.
The research community's challenge now involves accelerating both the identification of emerging vulnerabilities and the dissemination of protective practices before corporate adoption becomes too entrenched to modify easily. Jiang's observation that organisations are "moving so fast to use large language models without thinking too much about the implications" captures the fundamental problem: the velocity of technological deployment has vastly outpaced the velocity of risk assessment and framework development. Companies racing to capture AI productivity benefits risk creating embedded vulnerabilities that could prove expensive and difficult to remedy once organisational practices and cultural assumptions solidify around AI employee structures that lack proven governance mechanisms.
