Saturday, June 23, 2018

The dramas of a daily web scraper

I have a set of web scraping programs I run every day to update the data for my election betting market series.

These programs are very short, typically 20 to 30 lines of Python code. But they are highly idiosyncratic. They break when a bookmaker updates the way in which it serves up web pages. My usual approach when this happens is to do a quick fix as soon as I see one of my scrapers has failed.

Sometime between Tuesday morning and Wednesday morning this week, Sportsbet updated the way in which it serves up web pages. It was a dramatic change. Sportsbet want from using bog standard HTML to a JavaScript page that dynamically creates the HTML on the fly.

This necessitated a different approach to web scraping. And I only found the time today (Saturday afternoon) to update the Sportsbet web scraper.

In the mean time, it looks like the odds have moved at Sportsbet.



Sportsbet now has the Coalition in the best position it has been in since I started my daily monitoring of odds for the winner of the next Australian federal election. In case you are wondering, the next federal election is most likely to be held in the first half of 2019, but it could happen earlier.


No comments:

Post a Comment