In order to create and run a trading system, the first thing you need is some historical data to work with. Fortunately, there are several excellent free sources available on the internet. Probably the most popular is from the Yahoo Finance site.
Normally, in order to access the data, you would enter in a stock symbol and pull up the overview page for a company. For example, if we pull up the page for Apple Inc., you will see a link on the left side for Historical Prices. Clicking on that link will bring up a page with a nicely formatted table of the most recent prices for Apple. At the bottom of that page, there is a link to download all the prices as a comma separated file that is suitable for importing into a spreadsheet. The downloaded file is formatted like this:
The data is very useful, but it is not very easy to download and manage lots of different symbols for different date ranges.
Suppose we wanted to automate this process? If we could figure out the url format, it would allow us to automatically generate new url’s and download the data via a Python script.
Let’s start by examining the url that generates the download data. In this example, it is from September 7th, 1984 to December 3rd, 2011:
It looks pretty easy to decipher. There is the main part of the url, followed by a number of query options. They are:
s=AAPL, d=11, e=3, f=2011, g=d, a=8, b=7, c=1984, ignore=.csv
- s is obviously the symbol
- d appears to be the end month minus one
- e is the end day
- f is the end year
- g Not sure. Perhaps d is for daily?
- a appears to be the start month minus one
- b is the start day
- c is the start year
- ignore=.csv appears to specify csv as the output format
If we play around and build some new urls by hand by replacing the query values with new data and test them out, you find that we can leave the g and ignore options out and the downloaded files are identical.
So, now we know how to build a url to download historical data for any stock, and that’s all we need are the stock symbol and the start/end dates.
In the next post, we will create a set of Python classes to easily download and store the data in a consistent way that will be useful for our trading system.