Stock Spider

Motivation

This was intended to be the first step in a data pipeline to be used for future projects. I wanted to create an automated trading program, but in order to do this, I needed reliable interday (and ideally intraday) data. Although there are several free data providers, such as Yahoo and Google, accessing their historical data - especially intraday - is not freely available. Therefore, I set out to write my own web scraper to pull this data and store it in a database.

Current Thoughts & Learnings

This was a great introductory project to Python (before this, I worked pretty much exclusively in C++) and in particular was a joy to work with when it came to text processing and web scraping. I certainly would not choose to use C++ for this kind of task, as so much functionality comes for free from Python, and the performance benefits from running in C++ are actually undesirable, as you're more likely to get rate limited by the data sources.