Market Indicators — ML project with Julia language.
Can we create AI models to predict macroeconomic trends?
Not too long ago, the prediction of the weather was a hit or a miss. The joke was that the TV weather forecaster was carrying an umbrella on what was to be supposedly a nice day. Today, we can model the weather patterns very accurately for several days into the future.
Similarly, given enough relevant data, including market indicators, human behavioral patterns, current events, and the recent “black swan” events, we should be able to model the macroeconomics.
In the image above, I am showing a tiny fraction of the market indicator data that I have used and which is free and up-to-date available for download in the tabular format, mostly as Comma-separated Values (CSV, or spreadsheets).
“You should take the approach that you’re wrong. Your goal is to be less wrong.” — Elon Musk
Being consistently better than an average, even by a few percent in predicting the future, would have a big benefit.
For years, I have been discussing with my friends the concept of AI modeling based on thousands of macroeconomic data streams, and the opportunity it could give to the architect of such a system. Now, I feel that the data, the software and the hardware is within the reach of that goal.
It is very common to hear the business or the political pundits speak of what amounts to a “mono-thematic bias” based on the very limited ability of a human mind to grasp only a few dimensions of a given topic.
Here, I am trying to have AI combine thousands of complex data streams (think of each as a different spreadsheet metric) into a consistent and verifiable “oracle”.
At the start, I employed easily accessible market indicators and data that influence the immediate macroeconomic changes such as manufacturing, unemployment, new housing permits, inventories, etc.
I made heavy use of averages as I realize that certain events build up to a change over periods of time, sometimes days, weeks, or months.
Of course, I realize that historical data is not enough.
I am interested in human behavior as well. Pundit and influencer opinions, even if wrong, definitely affect human behavior.
We know for a fact that markets, or rather large groups of people, behave illogically.
However, there are patterns in there in how an average Joe listens to the presidential speech, watches a few pundits argue, reads an article or two, talks to friends at work, and watches a few videos titled “Imminent crash in [insert any year]”, hears that someone got laid off, and finally starts selling their stocks a few days later, making a colossal investment mistake.
Such modeling is difficult, but it is achievable. We are already regularly deducting human sentiment from Google searches, YouTube videos on the newest Apple or Tesla innovations, Twitter blasts, and blog posts. It is a known and proven technology.
This project is not meant to be a quick win, I do realize that every hedge fund is pouring millions into similar research. Rather, I want it to be a gradual learning process, a labor of love, and a means of recurrent income for my retirement. I still have about 20 years to solve this complex problem.
The plan is that every few days I add new data sources and continue building the models until I get good at predicting something. I have used the word “something” on purpose. I am not sure if I will be able to predict the particular price of the S&P 500 company, but I am sure I will find patterns in some niche, and having a wizard-level insight might open doors, a path less taken.
This article, and the accompanying open-source code, are not meant to teach you about the market, nor to teach you how to code, but rather to share my progress and to receive positive feedback and cooperation.
Project Objectives
- to gather and analyze as many market indicators as possible
- to learn patterns and interactions in the data
- to predict market trends for a couple of days ahead
- to predict a particular stock price
- to produce a reusable code and document the process
What are Market Indicators?
Market Indicators are collections of data points (think spreadsheets) reflecting the historical performance of a particular area of interest, for example:
- “S&P 500” index shows how major 500 stocks as a whole are performing
- “ISM Manufacturing” index shows how well the manufacturing industry is doing
- “GDP” (Gross Domestic Product) index shows how the country is doing
- etc., etc., etc.
There are hundreds, if not thousands, of such indicators. Each country, state, county, and community have sets of data that reflect some trend.
Why not use spreadsheets?
The spreadsheets and their graphing capabilities are the bread and butter of an individual person’s market analysis.
However, spreadsheets are good when comparing only a few indicators, when the data starts multiplying the inputs are overwhelming. Please remember that an average person can remember about 7 numbers, now try to imagine daily data for the last few decades coming from a few hundred indications, multiplied by a couple of hundred or regions, and add to it sentiments from hundreds of sources. We can easily talk about millions, if not trillions of inputs.
Today’s computers can process many “tera” operations per second.
Tera is 1,000,000,000,000. I think the point is clear.
Machine Learning approach
The overwhelming advantage of machine learning, or Artificial Intelligence (AI), is apparent when we try to find subtle patterns in thousands of data flows.
The human brain (using spreadsheets) fails to grasp the wealth of the information presented. Machine learning, on the other hand, can easily detect the patterns in massive datasets and derive a conclusion.
Why am I using Julia?
If I had chosen Python I would probably make my life easier.
However, I was thinking about a long, multi-year, term investment, and wanted to select absolutely the best language.
Here are a few of the characteristics of the Julia language:
- Julia is modern, fast, elegant, multitasking, and does the math extremely well, of course, we are talking about Julia programming language
- unlike C/C++, it is a pleasure to read and write Julia
- it is designed at MIT specifically for scientific computing and machine learning
- similarly to C, it is extremely fast
- similar to Python, it is very easy to learn
- it is designed for parallelism
- It is designed for distributed computing, as one day I might decide to run hundreds of predictive models in parallel and not in sequence as I do now.
May the source be with you, Luke
I am making changes to the project’s code on a daily basis.
The logical approach and the code, will surely diverge from any examples I would be able to provide here, hence the pun on Obi-Wan Kenobi’s “Remember, Luke, use the source.”, in other words, study the actual code.
See the actual predictions
In my GitHub, I have provided a page where I actually post the results of my predictions, I hope you will find them interesting.
The rest of the project is documented at:
Please subscribe and share
If you enjoy my writing, please like, subscribe, and share this article.
Based on the statistics, I either get motivated to write more or get in touch with my innate introvert.
Contact me via LinkedIn.
Citation
Please copy and paste below if you are referring to this article:
Uki D. Lucas. 2020. Market Indicators — ML project with Julia language. https://docs.google.com/document/d/18HEN_r0rp-JqMBXsVRQSqnDV0vqh85en_-FHEbNUP54/edit?usp=sharing
References
- Like it on Medium:
https://medium.datadriveninvestor.com/market-indicators-a-machine-learning-project-with-julia-language-be1a452213f8 - This article on Google Docs (probably most up-to-date):
https://docs.google.com/document/d/18HEN_r0rp-JqMBXsVRQSqnDV0vqh85en_-FHEbNUP54/edit?usp=sharing
