The dark side of digital finance: On financial machines, financial robots & financial AI

---

Note: I published a shorter version of this in Nesta's magazine The Long+Short as You Are the Robots. This is a modified and extended version, published under Creative Commons

---

A banker in 1716 had two main tools: a ledger book and a quill pen. A customer – perhaps a prominent carpenter – would enter a branch, request a withdrawal or make a deposit, and the banker would make a careful note of it within the ledger, editing the customer's previous entry to keep authoritative score of exactly what the bank promised to them.

ACCOUNT LEDGER BOOK OF 17th CENTURY BANKER EDWARD BACKWELL

Fast-forward to 2016 and we’ve entered into a world no longer dominated by tools, but by machines. The crucial difference between a tool and a machine is that the former relies on human energy, while and the latter relies on non-human energy channelled via a system that replicates - and accentuates - the action of a human using a tool. The carpenter is now a furniture corporation using computer-programmed CNC cutters. Likewise, the bank that keeps score of that company’s money runs humming datacentres with vast account databases. These are digital equivalents of the old ledger books, drawing upon fossil-fuel generated electricity to write and hold information as magnetised atoms on hard-drives.

THE 21st CENTURY BANKER'S LEDGER BOOK

We call the process of moving from manual tools to machines automation, and it appears in various forms within everyday financial life. The ATM, for instance, is an automated version of the bank teller of old who would have to exert energy to check your account, hand you cash, and alter your accounts. I use an interface to interact with this ATM, which gives me some form of control, but only within the inflexible rules of whatever it will allow me to do. This actually requires energy on my part, so while the machine seems to ‘do things for me’, the process also seems to be ‘self-service’.

Automation is creeping into more and more of personal finance. The glossy adverts of the financial marketing industry put an appealing spin on the future world of contactless payment, branchless banking and cashless society. They focus the mind on problems that are apparently being solved through new technology, but they simultaneously divert attention from the dark side of the automated financial regimes that are emerging around us. To get to grips with these processes of automation - and the sub-field of 'digitisation' - we first need to establish some definitions of machines, robots, and algorithms.

Financial machines vs. financial robots

I COME IN PEACE TO HELP YOU

Machines tend to require us to manually activate them towards a singular repeated action that they do no matter what, like the way a kettle always boils water if I manually push the ‘on’ button. The ATM is a multi-function machine that can do different things if I push different buttons on the interface, like ‘give me £30’ or ‘show me my balance’. It doesn’t, however, seem to ‘make decisions’ or have any ability to autonomously react. To make it feel like a robot, it must show some nominal agency to make decisions based on external information.

To understand what a financial robot looks like, we need to sketch some general characteristics of robots more generally. We might think of a traditional robot as a system comprised of four parts:

1. Body: An assemblage of mechanical parts
2. Mind: An algorithmic ‘mind’ that can compute or analyse information
3. Senses: Sensors that can detect external data
4. Energy source: For example, electricity

The traditional robot might take in data from sensors and compute it through an algorithmic mind that can activate the mechanical body, provided there is electricity. For example, a robot could be a vacuum cleaner (mechanical body) that receives data from photocell sensors (senses) to be processed through an algorithm (mind) to calculate its position, which in turn sends orders for the body to move around the room, thereby 'autonomously' vacuuming your lounge by 'making decisions'.  

Importantly, though, it may not be necessary to include the mechanical ‘body’ part at all. A robot might simply be a software-based algorithmic ‘mind’, taking in data and sending orders to other entities to act out its ‘will’. We might call this an algo-robot.

Let’s consider an Excel spreadsheet model that is used to estimate the fair price of a financial instrument like a share. A person armed with a pen and pad might take hours or even days to go through the relevant data and do the calculation manually. The spreadsheet model on the other hand, directs the electricity coursing through the hardware of a computer to do the same calculation in a fraction of the time. This is a financial machine, automating manual human calculation processes.

To make this into a robotic system, though, we must allow it to receive perceivable external data – such as a price feed from the London Stock Exchange – and allow it to process the data through its ‘mind’ of algorithmic formulas, and then give it the ability to make executive decisions based on its calculations (like the ability to send buy or sell orders back to the stock-market). And, voila, this is precisely what algorithmic automated trading is. The spreadsheet model has turned into a trader algo-robot. From this point the algorithmic coding can be developed into more ‘human’ forms, for example by equipping the robot with machine learning capacities and ‘evolutionary algorithms’ that can adapt to changing circumstances.

The algo-robotic managers of digital finance

ROBOTS: NO LONGER WORKERS and NO LONGER BODIES

‘Algo-robotic’ systems are particularly adept at accumulating power. Unlike the simple machine that offers static options via an interface, an algo-robot - or a series of linked algo-robots - have a greater ability to react in multiple ways in response to multiple data streams, and therefore to organise and co-ordinate. This trait makes senior corporate management warm to them, because, after all, reacting and co-ordinating are core elements of what a manager does.

The old hierarchy within a corporation was one where owners used managers to co-ordinate workers and machines. This gave rise to the traditional battles between owners and managers, managers and workers, and workers and machines. The emergent hierarchy is subtly different. The owners – often a disparate collection of distant shareholders – grant power to high-level management, who increasingly use algorithmic systems as ‘middle management’ to organise their workers and more basic machines.

And this is where we see the changing conception of the robotic system’s ‘body’. Rather than being a mechanical assemblage with an algorithmic ‘mind’, the robot could be an algorithmic mind co-ordinating a ‘body’ constituted out of ordinary employees, who increasingly act like machine parts. Think about the Amazon deliveryman driving the van to act out an order sent to him by an algorithm. This ‘body’ doesn’t even have to be constituted by the company’s own employees, as in the case of self-employed Uber drivers co-ordinated by the Uber algorithms.

These arrangements are often difficult to perceive, but algo-robotic systems have been embedding themselves into everyday forms of finance for decades, not necessarily 'taking over control' but often creating a hybrid structure in which manual human actions interact with automated machine-robot actions. For example, the investment bank trader might negotiate a derivatives deal over the phone and then book it into a partly automated back-office system.

"GOOD MORNING, HOW CAN MY MACHINE HELP YOU?"

The quintessential example, however, is the retail bank branch. You can talk with employees behind the Barclays counters, but often they are just there to enter data into a centralised system that tells them how to deal with you. To some degree these employees have agency – the ability to make quasi-autonomous decisions – but the dominant trend is for them to become subservient to the machinic system they work with, unable to operate outside the bounds set by their computer. Indeed, many bank employees cannot explain why the computers have made the decisions they have, and thus they appear as the human face put there to break the news of whatever the algorithm has decided. We might even say they are a human interface to an otherwise algo-robotic system that is accountable only to the senior corporate management, who you will never deal with.

---

NOTE: If you have enjoyed this so far, you might like my book

---

From hybrid systems to self-service digital purity

CHARLIE WAS LAID OFF FOR BEING TOO HUMAN

But, 'human interfaces' like that are actually quite costly to maintain. People are alive, and thus need food, sick leave, maternity leave and education. They also have a troublesome awareness of exploitation and an unpredictable ability to disobey, defraud, make mistakes or go rogue. Thus, over the years corporate managers have tried to push the power balance in this hybrid model towards the machine side. In their ideal world, bank executives would get rid of as many manual human elements as possible and replace them with software systems moving binary code around on hard drives, a process they refer to as 'digitisation'. Corporate management is fond of digitisation – and other forms of automation – because it is a force for scale, standardisation and efficiency – and in turn lowers costs, leading to enhanced profits. 

The process is perhaps most advanced in the realm of electronic payments, where money is shifted with very little human action at all. Despite recent talk of the rise of digital currencies, most money in advanced economies is digital already, and tapping your contactless payment card sets in motion an elaborate automated system of hard-drive editing that 'moves' your money from one bank data-centre to another. This technology underpins talk of a future 'cashless society'. Bouncy startups like Venmo and iZettle have got into the payments game, adding friendly new layers to an underlying digital payments infrastructure that is nonetheless still dominated by the banking industry and credit card networks.

HORRIFIC DARK AGE INEFFICIENCY

In the case of retail banking, an ideal situation for banks might be to get rid of the branches altogether, and to push for a world of ‘branchless digital banking’. This generally means slowly dismantling, delegitimising and denaturalising branches in the public imagination, while simultaneously getting people accustomed to 'self-service'. Indeed, many banks are cutting branches, and many new forms of financial services are found only online, like digital banks Fidor and Atom. Digital banking startup Kreditech claims that bank branches won't exist 10 years hence, "and neither will cost-intensive, manual banking processes". "We believe algorithms and automated processes are the way to customer-friendly banking," the startup declares confidently. 

Such digital banking is but one strand in the digital trajectory. Digitisation is starting to be applied to more specialist areas of finance, too, such as wealth management. Wealthfront, for example, now offers automated investment advice for wealthy individuals. In their investment white paper they state that sophisticated algorithms can "do a better job of evaluating risk than the average traditional advisor".

Digital systems like Wealthfront are often promoted as cutting out the middleman – assumed to be human, slow, incompetent and corrupt – and therefore as cutting costs in both money and time. Some startups use this to build a narrative of the 'democratisation of finance'. Quantopian, a system for building your own trading algorithms, comes with the tagline: "Levelling Wall Street's playing field". Robinhood draws on the name of the folk hero to pitch their low-fee mobile stock-trading system. 

It seems uncontroversial that these systems may individually lower costs to users in a short-term sense. Nevertheless, while startup culture is fixated upon using digital technology to narrowly improve short-term efficiency in many different business settings, it is woefully inept at analysing what problems this process may accumulate in the long term. Payments startups, for example, see themselves as incrementally working towards a 'cashless society', a futurist buzzword laden with positive connotations of hypermodern efficiency. It describes the downfall of something 'old' and archaic – cash – but doesn't actually describe what rises up in its place. If you like, 'cashless society' could be reframed as 'a society in which every transaction you make will have to be approved by a private intermediary who can watch your actions and exclude you.' It doesn’t roll off the tongue very well, and alarms the critical impulses, but nevertheless, that’s what cashless society would bring.

Forcing the 'inevitable progress' of digital finance

WHEN DOES 'PROGRESS' FINISH?

Part of the reason for the pervasive acceptance of these developments is the deeper ideological narrative underpinning them, one which is found within the tech industry more generally. It is the idea, firstly, that the automation of everything is inevitable; and that, secondly, this is 'progress': a step up from the inefficient, dirty services we have now. In this context, questioning the broader problems that might emerge from narrowly useful automation processes is ridiculed as Luddite, anti-progress or futile.

Of course, ‘progress’ is a contested term. If you’re cynical, you may see it as shorthand for ‘the situation an organised set of commercial interests view as desirable in the short-term’. It doesn’t necessarily mean ‘the thing that would be good for the broader public in the long term’.

Indeed, it is apparent that many people don't respond to 'progress' in the way they're supposed to. We still find people insisting on queueing to use the human cashiers at big supermarkets like Tesco, rather than diligently queueing up for the automated checkout. Likewise, we still find people stubbornly visiting the bank branches, making manual payment requests; even sending cheques.

Perhaps this is because there is something deeply deadening about interacting as a warm-blooded individual with a soulless automaton trying to sound like a human. The hollow fakeness of the cold clinical checkout voice makes you feel more alone than anything else, patronised by a machine clearly put there to cut costs as part of a faceless corporate revenue circuit.

The ongoing challenge for corporate management, therefore, is how to push automation while keeping it palatable. One key technique is to try to build more 'human-like' interfaces, and thus in London we find a hotbed of user-experience (UX) design firms. They are natural partners to the digitisation process, combining everything from ethnographic research to behavioural psychology to try to create banking interfaces that seem warm and inviting.

JESSICA SPENDS A SMALL FRACTION OF HER CORPORATE ENDORSEMENT EARNINGS ON COFFEE

Another key technique is marketing, because people often have to be 'taught' that they want something. In the case of contactless payment on the London Underground, the Mayor of London, Barclaycard, Visa and the Evening Standard have formed an unholy alliance to promote Penny for London, a thinly veiled front-group to encourage people to use the Barclaycard-run contactless payments system rather than those ancient Oyster cards. Sports stars like Jessica Ennis-Hill and Dan Carter have been co-opted into becoming the champions of automated finance. Signs have been popping up proclaiming 'contactless is here', as if it were something that people were supposed to be waiting for. These subtle hegemonic messages permeate every financial billboard in the city.

The dark side of digital finance

One key to developing a critical consciousness about technology is to realise that for each new innovation a new trade-off is simultaneously created. Think about the wonderful world of digital banking. A low-level bank branch manager might be subservient to the centralised system they work for, but can also deviate subtly from its rules; and can experience empathy that might override strict economic 'rationality'. Imagine you replace such an individual with an online query form. Its dropdown menu is the digital equivalent of George Orwell's Newspeak, forcing your nuanced, specific requests into blunt, standardised and limited options. If your problem is D, a system that only offers you solutions to A, B, or C is fundamentally callous. A carefully constructed user complaints system can build an illusion of accountability, while being coded firmly to bias the interests of the company, not the user.

Indeed, if you ever watch people around automated self-service systems, they often adopt a stance of submissive rule-abiding. The system might appear to be 'helpful', and yet it clearly only allows behaviour that agrees to its own terms. If you fail to interact exactly correctly, you will not make it through the digital gatekeeper, which – unlike the human gatekeeper – has no ability or desire to empathise or make a plan. It just says 'ERROR'.

This turns out to be the perfect accountability and cost cushion for senior corporate management. The responsibility and energy required for dealing with problems gets outsourced to the users themselves. And lost revenue from unhappy customers is more than compensated by cost savings from automation. This is the world of algorithmic regulation, the subtle unaccountable violence of systems that feel no solidarity with the people who have to use it, the foundation for the perfect scaled bureaucracy.

So, in some future world of purely digital banking we find the seeds of a worrying lack of accountability and an enormous amount of user alienation. The loan you applied for online gets rejected, but nobody is there to explain what hidden calculations were done to reach that decision. To the bank management, you are nothing more than an abstract entity represented by machine-readable binary code.

THAT'S YOU, AS MAGNETIC ATOMS

So where is the financial AI?

NARCISSUS TRIES HSBC's PLATFORM: "THEY UNDERSTAND ME SO WELL"

Of course, the banks don't want you to feel like that. In the absence of employees, they will have to use your data to create the illusion of some type of personally tailored service. Your historical interactions with the system will be sold back to you as a ghostly caricature of yourself, fed through the user-experience filters. And it is here that we find the emergence of new forms of financial artificial intelligence.

Let’s return to the earlier – somewhat blurry – distinction between machines and robots: robots are essentially machines that take in data from sensors and process it through an algorithmic 'mind' in order to react or 'make decisions'. Likewise, there is a blurry line between robots and artificial intelligence. At its most unambitious, AI is just a term for any form of calculation done by robots. It really comes into its own, however, when referring to robots that have adaptation and learning capabilities which allow them to show creativity and unexpected behaviour. Rather than merely responding to your actions or to external stimuli, the system begins to predict things, offer things, make suggestions, and do things without explicitly being asked to do them.

Imagine, for example, an ATM booth that uses facial recognition technology to identify you as you approach and make suggestions to you. Notice how the power dynamic changes? With a normal ATM I am still an active body, choosing to trigger the machine via the interface. In this new scenario, though, I’m a passive body who triggers the machine without any explicit conscious action on my part. It seems to 'take the initiative' and to direct me. It's only when we start to feel this as a power dynamic that we start to get closer to the feel of AI. The more you move towards AI, the more you feel increasingly passive relative to the robot (a passivity that is beautifully captured in this video).

CAN YOU PLEASE LET ME FINISH BEFORE INTERRUPTING?

Consider the customised ads Google feeds to us. We don't actively try to make them appear, yet it's still our actions that trigger the system to target us with specific information. That’s more like AI. There are many scenarios where this process could creep into finance, from machine-learning trading algorithms to creepy health insurance contracts that shift their prices according to your mobile payment data. "I see you paid for two chocolates today Brett. I will raise your premium."

WANNA PLAY?

But this can go beyond a single machine. Just like a robotic system may actually be constituted by an algorithmic 'mind' that coordinates a 'body' of people – like Uber drivers acting out the will of their invisible algo-boss – so the body of an AI may be fragmented, decentralised and hard to perceive. It could be a network of interacting algo-robotic systems that direct the actions of people who are unaware they are triggering the system. No individual node may be in control, but people may collectively become locked into reliance upon the system, pulled around by forces not immediately apparent to them, being manipulated by their own data. The AI could be a ghost in the collective machine, the manipulative 'invisible hand' in a technologically mediated market.

Don't panic, but don't not panic either

"DON'T WORRY, YOU CAN STILL PLAY FRISBEE IN THE MATRIX"

When thinking about the future of digital finance, the issue is not necessarily whether these services are narrowly useful to an individual. Sure, maybe the contactless card is cool if I'm in a hurry and maybe I can get a decent deal from the AI insurance contract. Rather, the issue is whether they collectively imprison people in digital infrastructures that increasingly undermine personal agency and replace it with coded, inflexible bureaucracy; or whether they truly offer forms of 'democratisation'.

It is easy to overhype these scenarios, though, because while it is true that payments, trading and retail banking are increasingly subject to automation, finance as a whole may not be especially amenable to it. Large loan financing decisions, complex multistage project-financing deals, exotic derivatives and other illiquid financial products cannot easily be standardised. They require teams of lawyers and dealmakers hashing out terms, conditions, and contingencies. Finance is an ancient politicised art of using contracts about the future to mobilise current action, and the dealmakers cannot easily be replaced with algos.

Furthermore, attempts to create more advanced and intuitive automated systems frequently fail. Semantic analysis algorithms – designed to read text – are terrible at understanding irony, sarcasm and contextual ambiguity within language. They may create feedbacks that thwart their own purposes, as in when people learn to game a credit-rating algorithm. High frequency trading falls apart under its own excesses and becomes less profitable. And there are customer backlashes: Metro Bank, the first new high-street bank in Britain for 150 years when it launched in 2010, has grown precisely because of its explicit focus on human-centred branch banking.

Nevertheless, it would be unwise to ignore the fact that the corporate trajectory is very much towards trying to automate as much as possible, and people need to come to terms with both the implications of this, and the vested interests behind it. It is not a neutral, 'inevitable' process. There are particular parties who seek it out. Take a moment to investigate who is on the board of Penny for London, that altruistic charity that insists contactless payment is a great way to help those in need. It includes hedge fund mogul Stanley Fink, and previously included the ex-CEO of Barclays, Bob Diamond.

So how should one respond? One approach is to ride with the technology, rather than to resist it. In intellectual leftwing circles the accelerationist sect advocates an embrace of automation, standing against sentimental calls for more human, local systems. It's an abstract position, founded on beliefs that automation will create conditions ideal for the downfall of capitalism. At some point it intersects with the cult of the Singularity, popular among evangelical tech entrepreneurs and transhumanists.

ARE WE THERE YET? ACCELERATE!

The ideological ambiguity is perhaps most acute in the emergent field of blockchain technology. Such systems potentially offer a way for strangers to freely interact with each other without central human intermediaries getting involved in the process. They may use blockchain systems to issue shares, enter into insurance contracts and form digital co-operatives, but the systems are underpinned by an extreme version of automation, one that is essentially autonomous. Indeed, the deep-level mission of projects such as Ethereum, a decentralised platform for 'trustless' transactions, is the replacement of human systems of institutional trust – like the legal and political systems that normally underpin all contracts and markets – with automated ones apparently detached from the human ambitions of those who historically have run such systems ('the politicians', 'the regulators', 'the bankers'). Libertarians long for an automated 'Techno-Leviathan' to replace the human sovereigns we have now, but it is a big question as to whether such automated systems truly provide a more 'democratic' infrastructure for interaction.

More down-to-earth are those who want to allow more creative interaction with the existing digital infrastructure. Take the Open Bank Project, for example, which wants to facilitate third-party customisation of digitised banking processes by opening up bank APIs, in the same way that independent developers might build third-party Twitter apps that draw data from Twitter's API.

And, finally, we have those who authentically seek to harness digital technology to bypass and challenge the standard economic rationality of large scale, short-term profit-seeking financial beasts, taking advantage of the lower startup costs of a digital setting to promote peer-to-peer finance, alternative currencies, crowdfunding platforms and non-monetary sharing platforms.

So, the scene is set. One thing is for sure: if the future of banking is going to be digital, we want it to be populated with those who value the deeper tenets of open source philosophy. Otherwise we could be left with increasingly alienating, exclusive and unaccountable financial surveillance states, presiding over increasingly passive and patronised users.

---

If you'd like to support my ongoing Creative Commons writing, please consider buying me a virtual beer

---