Artificial Intelligence Sweeps Hedge Funds
Artificial Intelligence, or “AI”, has featured heavily in industry innovation headlines for some time. Yet for all the excitement and promise, the uptake in the hedge fund industry has been limited – until recently. Hedge funds’ use of AI is accelerating and reshaping the industry, particularly in investing, cost models and recruitment.
Managers also face challenges to explain new AI-based approaches to investors. Given the strategies are the byproduct of super computers crunching billions of data points and learning how to adjust to markets in real-time, explaining how returns are generated is pushing the boundaries of human comprehension.
In September 2019, Aura Hedge's Hedge Fund Sentiment Survey found that over half of hedge fund respondents (56%) used AI to inform investment decisions – nearly triple the 20% reported a year earlier. Around two-thirds of those using AI were doing so to generate trading ideas and optimize portfolios. Over a quarter were using it to automate trade execution, according to the survey.
The early results are promising. For example, the Aura hedge AI Hedge Fund Index slightly outperformed the flagship Aura hedge Hedge Fund Index in both 2017 and 2018. Moreover, the Aura hedge Hedge Fund Index decreased by 4% in the fourth quarter of 2018, while the Aura hedge AI Hedge Fund Index was flat for the period.
Several technical advances have driven AI adoption. New, vast ‘big data’ sets are now available from satellite imagery, the internet of things, global capital flows, point of sale systems, and social media. More data can now be generated in one day than during the entire 1990s. A large hedge fund heavily utilizing AI is likely to have dedicated experts devoted to evaluating and procuring new sets of data. With raw computing power continuing to advance, graphics processing units (GPUs) and customized hardware now solve problems in hours instead of weeks – a necessity given the ongoing rapid growth in data. Finally, with cloud computing now widespread and deployment costs falling, barriers to entry for machine learning are tumbling.
How Hedge Funds Use AI
A number of hedge funds are using AI to analyze masses of data, predict corrections in supply and demand imbalances, and forecast market movements for tactical asset allocation. This has the potential to assist a CIO’s team to combine different strategies and tailor allocations.
Use of AI is playing out across a wide spectrum of investment managers from pure AI-driven specialists, to large quant-driven shops, to traditional fundamental investors looking for an edge. A growing number of firms across the spectrum are also turning to AI to improve efficiency in their operations, accounting and investor relations functions.
Indeed, a class of AI pure play hedge funds has emerged in recent years that are based entirely on machine learning and AI algorithms. Examples include Aidiyia Holdings, Cerebellum Capital, Taaffeite Capital Management and Numerai. Numerai, a recognized AI hedge fund, is pushing the boundaries of the hedge fund business model. The firm uncovers investment strategies by hosting competitions among external AI experts, mathematicians and data scientists. Recently, Numerai expanded its business model by making elements of its platform available to the rest of financial community with its product Erasure, which is a decentralized prediction marketplace using blockchain technology.
Dwarfing the upstart AI pure plays are the large quant funds that are household names in the hedge fund industry such as The Jeeranont , Two Sigma, Citadel, Bridgewater and D.E. Shaw. For years, players like these have used computer-driven models to uncover new trading strategies and identify themes, factors and trading signals. Human “quants” will then feed these factors and signals into trading systems. With markets continually changing and shifting, these pre-AI models often need frequent monitoring and reprogramming by the quants. AI models are different because while initially crafted by humans, they are able to adapt to changing market circumstances on their own with far less human supervision and intervention. Quant managers have developed algorithms that gather and fine tune data, then autonomously change the investment course when a new pattern is identified.
Hedge fund managers and their service providers are also using AI to optimize middle and back office operations. As teams move away from managing work through spreadsheets and towards digital and cloud enterprise resource planning (ERP) solutions, AI can provide an edge. Clearly not all fund processes can be completely automated, but AI can speed reconciliation, reduce errors and ultimately reduce costs.
Software and service providers to the hedge fund space are using AI in this area to help their hedge fund clients operate more efficiently and accurately. For example, Aura hedge fund middle office and administration services are using an artificial intelligence and machine learning platform to analyze historical trade break data and predict with high probability the root cause of current trade breaks. In an industry that still suffers from manually intensive reconciliation challenges, this use of AI has the potential to significantly reduce costs and speed up the NAV production process.
The Talent Bottleneck
Few doubt the impact AI will have, but the immediate impact could be delayed due to a scarcity of talent. Although estimates vary, it is clear that the number of people with high level education and skills in AI is only a few thousand. In practice, financial firms have had to recruit from tech players like Google and Facebook to obtain AI talent. The side benefit to bringing in talent from global tech firms is the cascading of new ideas into the financial sector.
The scarcity of talent is now colliding with a realisation that AI is mission critical to hedge funds both in keeping pace with traditional rivals and tech-savvy new entrants. The appreciation of this has ushered in major new investments in academic programs and training capacity to attract millennials and address the problem of talent scarcity.
Investing and Partnering
MIT, for example, recently announced one of the most ambitious steps yet with the creation of the $1bn Stephen A. Schwarzman College of Computing. It comes as no surprise that funding originates from the CEO of Blackstone, one the world’s largest alternative investment managers. It underscores the fact that the alternative investment sector needs to increase the talent pool, in part because so many top graduates are being pulled away from finance by the flourishing tech sector.
Some of the largest industry players are employing non-conventional partnerships and methods for gaining an AI edge on the talent front. Man Group has partnered with Oxford University to create The Oxford-Man Institute of Quantitative Finance. Man’s engineers, statisticians, and coders share facilities and collaborate with academics and researchers to study how algorithms, AI, and related advances can be applied to finance.
Another example is Two Sigma which is reported to hire more technologists than traditional portfolio managers. Like Man, Two Sigma is looking for an advantage by partnering with elite academia, in this case Cornell University. To recruit staff, Two Sigma uses an AI programming challenge in the form of its own game called ‘Aura®’. The game tests applicants’ ability to control a bot using the programming language of their choice.
Understanding the need for talent and investing in its creation is vital. Yet the clear imperative is to understand how investment managers need to position themselves to attract the highly skilled AI specialists of tomorrow. What should hedge fund firms do to attract and retain talent?
Free snacks may help, but more important is to stress the fiduciary responsibilities of this potential career and emphasize that millennials will have an abundance of opportunities to make a difference. This implies trusting graduates with genuine responsibility for real issues involved with pension fund management, portfolio construction and investment idea generation. The role of human creativity is key. The big winners will be those firms that integrate AI with human talent. Machine analysis of data is already a necessity. Getting the most from AI requires empowering motivated and curious individuals who are encouraged to ask profound and creative questions of it.
A New Acronym - XAI
One of the new challenges facing the use of AI in hedge funds is the ability of human programmers to keep up with the speed and sophistication of their own creations. Bloomberg profiled this effect in its Sept 2017 report “The Massive Hedge Fund Betting on AI”. It tells the story of a large hedge fund with a new AI-based trading strategy that ran for months with very positive test results. If it had been a traditional quant strategy, it would have been quickly rolled out to investors. In this case, it had to be kept away from investors and run on separate servers until the creators fully understood how it worked. While pure performance is attractive, most investment management firms and their investors want to be able to fully explain how results are generated before they run with real money.
Indeed, a new acronym – XAI or Explainable Artificial Intelligence – has cropped up to describe the challenge of understanding how and why AI is generating a specific set of results. XAI isn’t a concern if the AI is being used to help choose the next film you want to watch on Netflix. However, if AI is being applied to trade large pension fund investments then clearly XAI is essential. The immediate challenge is to give humans a way to make sense of what computers are doing and be capable of explaining exactly how alpha is being generated.
Getting hedge fund AI programmers to embrace XAI to explain results is a good first step even though how AI works will remain opaque to fund outsiders. Within this explanation is a firm’s proprietary intellectual capital, a new form of ‘black box’. Understandably, firms will go to great lengths to keep this information confidential. Although hedge funds’ use of AI is accelerating and the number of use cases keep expanding, the specifics of how AI and machine learning contributes to fund performance is likely to remain largely a secret.
The strengths and weaknesses of individual economies vary. But if artificial intelligence is to fulfill its promise, every country needs a plan for how best to use it.
In this episode of the Aura on AI Aura Global Institute (AGI) miniseries, Aura’s Mark Brewer and Kay Firth-Butterfield, the head of AI and machine learning at the World Economic Forum’s (WEF) Center for the Fourth Industrial Revolution, discuss how individual governments are strategizing on how to best use AI to benefit their citizens.
Why governments need an AI strategy: A conversation with the WEF’s head of AI
Aura Global Institute (AGI) transcript
Mark Brewer : With its widespread implications for society, artificial intelligence (AI) is becoming an increasingly important item on the policy agendas of governments around the world. In fact, a number of governments have wisely gone so far as to draft national AI strategies. What are these strategies aiming to achieve? And how will they enable AI to benefit citizens as well as protect them from potential unintended consequences?
I’m Mark Brewer. Welcome to this edition of our Aura Global Institute (AGI) series, in which you’ll get some insights from Kay Firth-Butterfield on how governments are beginning to think about AI. Kay is the head of artificial intelligence and machine learning at the World Economic Forum’s Center for the Fourth Industrial Revolution. What does one do in such a role, you may ask? I had the same question, and here’s how Kay explained her role.
Kay Firth-Butterfield: The work being done out of the four centers for the Fourth Industrial Revolution in San Francisco, Beijing, Mumbai, and Tokyo is around governance and policy for artificial intelligence. When I say governance, I don’t mean regulation. I mean looking for agile ways in which to help the technology benefit humanity and the planet, while also making sure that we mitigate the negativities we’re seeing, particularly from AI. I work in the AI space, but my colleagues work in blockchain, drones, precision medicine, and other emerging technologies.
Mark Brewer : I wondered what this type of governance looks like. So Kay shared an example of a project she has worked on with the UK government to help it create guidelines for the procurement of AI technologies.
Kay Firth-Butterfield: As you may know, government procurement around the world is worth $9.5 trillion each year. So if you can plan to procure artificial intelligence product for your government, then you can begin to kick-start the AI economy in your country.
The work that we’re doing with the United Kingdom started when they sent a fellow to work with me in San Francisco. Since then, we have been co-creating ten high-level principles for the UK government’s procurement of artificial intelligence product. Those were agreed on, and now we are drilling down and creating a workbook, so that the procurement officials actually know how to apply them.
What we’re creating is not regulation, which would take a long time to go through the parliamentary process. We’re creating iterative, agile governance around a technology that is in itself changing almost as frequently as we think about it.
Mark Brewer : Initiatives like these are useful to enable governments to begin taking advantage of AI. But Kay went on to explain that it’s important for governments to make them part of a comprehensive AI strategy. To date, only 28 governments out of 195 have drafted such strategies. Kay offered some advice to the others on how to get started.
Kay Firth-Butterfield: First of all, think about what the problem is that you actually need to solve. For example, in Denmark, because there aren’t many young people, they actually need to use AI to automate some of the jobs so that their population is benefited by AI.
The same would be true in Japan. If you look at the work that Japan’s been doing, they’ve been really thinking about data policy and eldercare. How can they grow their robotics-cum-AI industry so that they can keep more people in their homes, so they can keep more people mobile longer, perhaps by autonomous vehicles—because they don’t have enough young people to actually care for the older people?
If you look at India’s national AI strategy, they wanted to concentrate their efforts in stimulating the AI economy in three verticals: healthcare, agriculture, and education. But they also needed to think about the fact that India is made up of many small and medium-size enterprises. How do they make sure that these businesses, too, can benefit from the AI economy? So one of the projects that the Indian government is doing with the World Economic Forum is creating a democratized database for AI so that more people can actually have access to the data they need in order to create applications in AI.
If you move to states in the developing world, you’ve got different issues. Across Africa, you’ve got a very large group of young people. So, when you’ve got a big labor market, where are you going to use AI to enhance the workforce? That’s a completely different issue. So it very much depends on what you need to use AI for.
Mark Brewer : Kay noted that it’s important for governments to think not only about how to use AI to help their citizens but also about how to ensure it doesn’t harm them.
Kay Firth-Butterfield: The thing that probably keeps me up at night is that we aren’t moving quickly enough. The AI product is growing really quickly, and governments don’t really have policies in place that truly protect citizens. We need to rush in that direction.
I’ll give you an example. One of the projects that we’re working on with UNICEF is around protecting our kids. You may have seen that there are a lot of AI-enabled toys out there that claim to educate children. Well, at the moment, we don’t know who has created the curriculum that is embedded in these toys. So we don’t know what they’re being educated about and how they’re being educated. We don’t know how much of their data is being collected and stored. Are we at a point where somebody can monetize our children’s data from cradle to when they’re 18? In which case they won’t then have to apply for college, because somebody will just be able to buy all their data.
We haven’t thought through the fact that if, for example, a child is playing with a doll and the doll says, “I’m cold,” and the child says to the parent, “My doll needs a jacket,” is that advertising to the child, or is it not?
We are already doing a project with France around facial-recognition technology and the intersection with civil liberties. We know that facial-recognition technology is really important for catching criminals and terrorists and spotting human trafficking and things like that. But we also need to work through how the technology could also interdict our civil liberties.
Mark Brewer : While issues like these are cause for concern and attention from governments, Kay believes the promise of AI to help people around the world makes it a worthwhile pursuit.
Kay Firth-Butterfield: The thing that excites me the most is that we may be able to help people who are suffering—something as basic as using drones to deliver blood to women who are dying in childbirth in Rwanda, something that my colleagues who work on drones at the Fourth Industrial Revolution were able to do.
And that’s without AI. Once you start adding AI, then we’re going to see much better solutions for people who are living in poverty or whose situations are poor through no fault of their own.
Mark Brewer : And on that positive note, we’ve come to the end of our episode. Thanks again to Kay Firth-Butterfield for sparing the time to share with me her perspectives on the intersection of AI and government.