As the whirlwind of change around artificial intelligence gathers force, people and businesses can feel adrift in the face of such powerful technology. While there is often reference to the need to keep the 'human in the loop', there is little clarity as to what this means in practice. Where in the loop should we sit? What skills will we need to be effective?

Creative destruction

Great strides are being made to harness the power of AI and reap its benefits, ensuring they outweigh the upheaval it may cause. This is nothing new in modern societies.

As far back as 1942, the Austrian economist Joseph Schumpeter described an economic process where innovations, new technologies and business models completely replace older, outdated ones, driving long-term economic growth and higher productivity, but causing short-term job losses and impacting traditional industries. He gave it a term: creative destruction.

Across the insurance market this is precisely what firms are now experiencing. The challenge is to manage the destructive part of the process as far as possible, while injecting creativity using vision, courage and dynamic risk management.

The insurance market is now embracing a vast array of challenges, from reimagining underwriting, claims and the customer interface, to putting in place robust governance and ethical standards. Crucially, this means reassessing where in the loop the human sits and how to exercise influence, oversight and, where possible, control over AI-driven processes.

The human in the lead

While there is some ready willingness to concede roles and regulation to the technology, Giles Tagg, a Partner in DAC Beachcroft’s Bristol office specialising in professional indemnity claims, counsels against this:

"The rush to embrace AI, especially among the professions, has sometimes been accompanied by a failure to pinpoint where human oversight and intervention is important. The occurrence of one or two unfortunate headline-grabbing events has led to too heavy a focus on its potential negative impacts and scaremongering about it taking professional jobs away.

"Looking at it in a more positive light, AI has the potential to create new jobs and skill sets. At the front end, these will focus on how to use AI as a tool: how to ask AI the right questions; how to manage the data sets; and how to control the different search and programming functions of AI.

“And then just as important is overseeing the back end. It is absolutely vital that a human has a way of verifying that AI has done what it has been asked to do, and not hallucinated or made errors.”

To be effective the human must be more than just in the loop, they must become the 'human in the lead'. Reshaping the human involvement in this way will put a sharper focus on the skills that are needed to be effective.

Many of the new skills needed will obviously be very technical and require expertise in data management, model development, and a deep knowledge of the subject areas AI is being asked to work in. However, there is a danger of overlooking some rather more obvious requirements, warns Ajay Gupta, an independent board adviser with a long experience in the London Market.

He puts language skills high on his list of priorities because the starting point of engaging with AI is interrogating it, carefully framing the questions and prompts so that it delivers to the human dictated brief, not one of its own invention.

“One challenge is that our education system does not necessarily develop these key skills because youngsters have grown up with mobile phones and social media, and everything is short. They read two words, three words. They do not read full sentences and that is a serious weakness. AI is based on sophisticated language models and that is why being able to have strong language skills, in my view, is so important: if you do not give AI tools the right instruction and right framework, they can go off on a tangent.”

He also says firms must recognise the limitations, as well as the strengths, of AI:

“AI excels at repetitive, precise, well-defined tasks, such as manufacturing automation, and data processing where accuracy approaches 99.9%, but it struggles with nuanced judgement and interpersonal negotiation.

“A finance director purchasing a multimillion-dollar insurance policy wants to speak to a credible underwriter who understands their risk. People buy from people. The intersection of talent and technology is where success lies. Both are needed; neither alone is sufficient.”

Super intelligence and product safety

Human oversight and intervention will be crucial as AI develops greater autonomy, an inevitable stage in its development which we have already entered. Warning signs abound with examples of AI systems escaping user control, exacerbating the already widespread fears that it could soon move into a phase, sometimes described as 'super intelligence', where it entirely escapes from the human world that created it.

Insurers must be on their guard for the implications of this, cautions DAC Beachcroft’s Legal Director Stephen Turner, who specialises in product safety, liability and recall:

“A critical emerging risk is the scenario where AI autonomously detects, diagnoses, and fixes a problem with no human visibility or intervention. If the fix works then the issue is resolved silently, with no record for insurers, regulators, or the business itself.

“However, if the fix is wrong and no-one picks it up because the right monitoring and reporting systems are not embedded, the AI's autonomous correction could create a new or more serious problem, such as altering ingredient ratios in food manufacturing, potentially making a product hazardous.

“Regardless of whether oversight is provided by a human or a separate supervisory AI layer, some form of independent monitoring, data capture, and reporting on AI decision-making is essential, particularly for safety-critical applications.

“This has direct implications for the underwriting process as insurers should be asking detailed questions about an insured's AI oversight and governance protocols.”

Policy drafting and silent AI

This is just one of an ever-lengthening list of new risks that AI is presenting to the insurance market across a wide range of policies. The market remains alert to the possibility of another wave of litigation around wordings, a legacy of the COVID-19 pandemic, says Jonathan Hopkins, a Senior Associate at DAC Beachcroft specialising in policy wordings:

“The market may wish to avoid uncertainty by either affirming AI is covered or clearly excluding it. Originally, we saw requests for assistance on tweaking policies to affirm AI. Now, the market seems to be engaging more comprehensively and trying to provide real comfort to policyholders on exactly which AI perils are covered. Conversely, we are also seeing interest in AI exclusions.”

Wordings are only one part of the equation as difficult decisions then have to be made about pricing, and that requires an understanding of potential exposures, an almost impossible task given the rapid extension of AI’s capabilities.

Supply chain vulnerability

The extent of the new risks presented by AI does not stop with a single insured but requires an overview of complex supply chains, says Jade Kowalski, a London-based DAC Beachcroft Partner specialising in data protection.

“Most AI systems are dependent on a complex supply chain of providers. If one of the pervasive providers were to suffer a system outage or be infiltrated by a bad actor, that could have a significant impact across an entire sector. With lots of different coverage terms within the market, there is the potential for confusion and a serious reputational risk for the market if it is not clear on coverage.”

The added complexity of the presence of AI at various points in a complex supply chain is something insurers need to address, agrees Turner.

“The law is relatively clear that consumers may pursue a manufacturer who brings everything together into a final product, or who puts their brand to it or the person who distributes it if they cannot identify any of the other actors. That is relatively simple, and we are used to it. But what does that mean for an AI-enabled or AI-dependent product that brings in technology companies and developers whose presence is probably much less obvious to consumers and may therefore be more of a supply chain issue? The challenge is very much there for insurers, particularly if they are themselves pursuing a subrogated recovery."

Cyber cover and agentic AI

Kowalski also highlights the challenges AI poses to the scope of cyber cover, with data poisoning or prompt injection standing out as a distinct threat for underwriters to consider.

“A threat actor could infiltrate an insurer's systems and silently alter the prompts configured for an AI system, causing it to produce incorrect or fraudulent outputs. For instance, an automated travel claims system instructed to pay verified delayed-flight claims within 24 hours could be manipulated to pay all travel claims indiscriminately.

“It is easy to say that the ‘human in the loop’ concept is the primary safeguard, but its effectiveness depends on whether staff have the skill sets to identify prompt manipulation.

“Even if there is human oversight that might not be enough as agentic AI poses an amplified risk as by design it operates without a human in the loop, executing actions autonomously, making it a potential minefield for insurers if compromised.”

AI capabilities go beyond the ability to act autonomously, within carefully crafted boundaries. AI also has the power to enhance its own capabilities. This recursive self-improvement lies at the heart of the fears of many experts that AI could run out of control.

The difficulty of placing a ‘human in the lead’ to meet these challenges should not be under-estimated but must be part of the conversation underwriters and brokers have with their clients.

AI-driven complaints volumes

Another more prosaic challenge that insurers are already facing is the use of AI by customers and complainants to generate complaints. This is an immediate and serious issue which is already manifesting itself. AI is enabling people to submit long, complex complaints and data subject access requests (known as DSARs). According to Kowalski:

“Our clients are seeing an exponential increase in the volume and complexity of complaints and DSARs. In many cases, the use of AI creates a request which is so complex that it masks the true intent and concern of the individual, meaning that the insurer stands little chance of providing a response which will resolve the issue.”

"The best approach is to put the human firmly back into the loop and break the cycle by reverting to direct human communication. This is where insurers need to apply additional care when considering where to deploy AI themselves. While there is merit in the use of AI to handle complaints and DSARs, there is also a risk of creating a cycle of AI to AI interaction, with both customers and insurers using AI to draft responses, resulting in lengthy correspondence chains. A 20-minute phone call will be far more efficient than months of protracted AI-generated email exchanges and allow the customer to articulate their genuine grievance."

Data centres

Also sitting at the heart of the new risks AI is presenting to the market are the data centres that drive it. The scale and operational complexity of the data centres has propelled them into wider public debates about energy security, the stress placed on water supplies (especially when faced with drought conditions), and potential pollution from leakages and noise. The UK government, now with a Minister for AI – Kanishka Narayan – sitting at the Cabinet table, has designated them as 'critical infrastructure', giving them priority in many areas when crises threaten.

Insurers have not been reticent in offering data centres cover but are treading carefully, says DAC Beachcroft Partner, Eleanor Whittaker, who specialises in property and liability insurance:

“There is strong insurer appetite to underwrite data centre risks, driven by the enormous scale and value of these projects. However, the accumulation and aggregation of risks across large, syndicated portfolios is a significant and potentially underappreciated concern.”

There are the obvious primary physical damage exposures such as fire, caused by electrical faults or overheating, often triggered by cooling system failures, and flood also resulting from cooling system failures. To these must be added business interruption.

“BI losses are likely to exceed the value of physical property damage in most scenarios and insurers need to be clear on what scenarios BI cover should extend to. An obvious question is whether cover should apply where there is no underlying physical damage, such as a shut-down caused by an external power supply failure.

"Another scenario that needs to be addressed is the potential for regulatory or governmental intervention restricting power or water access: could this mean prioritising agricultural use over data centres during drought conditions, which would typically not then trigger standard BI cover because of the absence of contingent physical damage?”

This carries echoes of the disputes over BI cover for loss of business during the COVID-19 pandemic when businesses were forced to shut by government order.

Whittaker adds to the list of challenges insurers need to be aware of the high global demand for new data centres which has stretched the supply chain for specialist equipment and skilled engineers. Competition for these resources is forcing up costs, adding to the complexity of accurately pricing data centre risks.

Giles Tagg also points out that AI is not just the reason why the data centres are being built, but it is also integral to their operation, with functions such as predictive maintenance, cooling control, workload automation, and dynamic power distribution all AI-dependent.

“This creates a new layer of risk: software errors, algorithmic failures, and the emerging threat of AI systems acting autonomously or being targeted by adversarial AI.”

War risks

There is another, chilling addition to the list of risks. According to DAC Beachcroft Partner Paul Baker, who specialises in political risk, trade credit and political violence:

“The nature of warfare is changing and data centres, much like energy assets in the Middle East in the Iran-US/Israel war, are high-value targets in modern asymmetric warfare. Future data centres may need to be physically hardened against drone attacks. This is a largely unexplored but significant risk.

"Similarly, the impact that Ukraine is having with its carefully targeted drone attacks on the Russian online retailer Wildberries shows how vulnerable large industrial units are in the era of drone warfare. With drones capable of being operated remotely, and flying huge distances, identifying the protagonist with certainty will often be laced with significant difficulty. This could pave the way to coverage disputes."

AI regulation

While insurers grapple with these emerging risks, legislators and regulators are also trying to tread a careful path.

Leading the way has been the European Union which, in addition to a specific AI Act, has updated a range of product liability and product safety directives and regulations. These are addressing some of the key issues AI has brought to the fore, says Turner:

“When assessing defectiveness, the EU Product Liability Directive (PLD) takes into account the possibility that there may be autonomous developments during the product's lifecycle and it is very much written with this sort of technology in mind.

“This is in contrast with the UK where our primary source of consumer product liability law is the Consumer Protection Act 1987, which still fuels debate about whether software is a product. It is a very long way adrift.

"The Law Commission is in the middle of consulting and is asking whether we should follow the EU. It seems clear there is a real political will to have greater alignment. I would be surprised if the UK didn't end up doing in large part what the EU has done with the new PLD,” says Turner.

The EU’s AI Act is the most detailed AI-specific legislation attempted globally and is in the process of being implemented, although it faces some delays in implementation of certain provisions, notably regarding high risk AI systems.

To date, the UK has chosen a 'pro-innovation' approach. All eyes will be on the new UK Minister for AI to see if he will continue or change this approach.

Financial advice and chatbots

Nearer to home for the financial services sector is the growing debate around the incursion of AI into key customer interfaces, including offering financial advice at the sales and claims stages. The European response has some influential supporters, including the British Insurance Brokers’ Association.

“My view is that if AI platforms and Large Language Models are offering advice then they should come into the FCA perimeter”, says Alastair Blundell, Head of Insurance at BIBA.

“In Europe, where the EU already has an AI Act, the principle is that any entity that deploys a chatbot is responsible for ensuring compliance with the Insurance Distribution Directive. It would seem sensible to replicate that approach here. The aim of this is to protect both customers and our members.

“Furthermore, if a broker in the EU uses AI to help formulate advice it gives to a customer, the broker remains legally responsible for it. The guiding principle is that automation does not lessen the duty of care owed by the broker to the customer and the advice must be clear and suitable. Again, this seems sensible and we are considering guidance to help BIBA members deploy AI safely within their operations.”

This is on the radar screen of UK regulators, as the Governor of the Bank of England, Andrew Bailey, outlined in a speech at the Mansion House earlier this summer, acknowledging that chatbots and other AI-enabled technologies are already giving advice:

“This creates a clear asymmetry. Regulated firms face strict obligations and liability, while unregulated AI providers can scale rapidly without equivalent safeguards, undermining consumer protection … This dynamic is already beginning to materialise and, without intervention, risks shifting trust, innovation and consumer engagement away from regulated institutions into unregulated channels”.

These comments came hard on the heels of the publication by the Financial Conduct Authority of the Mills Review into how AI could reshape financial services by 2030 and beyond. It rules out the immediate imposition of new regulations for AI but highlights some key areas it will be monitoring, including how AI may impact insurance, for example through embedded insurance, automated quote comparison, and claims triage and guidance. The review also highlights concerns about the potential erosion of traditional risk pooling.

For Mathew Rutter, a Partner at DAC Beachcroft specialising in financial services regulation, the challenge for regulators both in the UK and globally is to strike the right balance between threat and opportunity.

“The Mills Review highlights the potential for AI to support customers to make better financial decisions and to reach investors who are currently underserved. However, AI could also challenge the regulatory perimeter, offering unregulated advice and leaving clients unprotected."

A key challenge for the FCA as it digests the responses to the Mills Review will be to reshape regulation so that the ultimate responsibility for financial advice remains firmly in human control. This could emerge as a key battleground as the AI platforms gear up to resist what they will claim is overly restrictive regulation.

Looking forward: investing in people

With the valuations put on the imminent floatation of Anthropic – the developer of Claude – already exceeding US$2trn, this super-charged technological change will clearly not stall for lack of capital investment, says Ajay Gupta:

“AI is attracting an unprecedented level of capital - not millions or billions, but trillions of dollars. No prior technology wave, including the dotcom era, has attracted comparable investment. Of course, these huge sums are driving the expansion of computing power housed in enormous data centres. But it is important that money is invested in people too, so that they can acquire the skills that will enable them to prosper."

The message is simple: we need to invest to ensure that the human stays in the lead.