I have all the game data in an excel file already. I just need to add some information to the dataset.

I drafted a simple scraper for your purpose here: http://pastebin.com/Hb105n7p

It reads and writes CSV files (easily imported and exported with Excel, Calc, ...). I tested it wth a few random games. Failures are handled gracefully (e.g. for canceled games).

Just fine-tune "FIELDNAMES" and "delimiter" according to your preferences.

Run the script with the input and output file as parameters, e.g.: python scraper.py input.csv output.csv

The code is suitable for Python 2. For Python 3 you will need to adjust the "print" and "exception" statements.

I hope this helps!

let me know what data you need, it'll probably be a lot faster if I just pull it for you. Just let me know what you need it for.

Hey thanks plong!

Ah, I took a look and I see I wasn't being clear. I want to know what year IRL the game ended (2013, 2014) and your script is pulling the game year (1910,etc...i think?). How easy would that be to switch around?

Impossible.

For this type of thing I recommend just asking me, it doesn't take long to write a query to pull data.

And my way you're not hitting the site with a ton of requests that Kestas might have written protection against :)

The timestamp (since epoch) of the "next" turn is part of each game overview page. For finished games this timestamp seems to indicate the end of the game (or maybe "end of game + phase lenght"). Obviously this is a fragile assumption that may break easily in the future.

I updated the snippet with regard to that timestamp: http://pastebin.com/sSrjX19d

Following jmo's hint regarding acceptable server traffic: maybe you want to add something like "time.sleep(5)" to the request loop.

@jmo: I often noticed that people do more creative stuff with data if they do not need to wait for / discuss with gate keepers. Thus personally I would not advise interested people to refrain from gathering the information on their own. But this is just my personal experience which may not work for webdiplomacy.net.

@plong, fair point, and nothing against your method at all, python is what came to mind when I saw this thread as well. Really the easiest web scraping language around. I do mention people can ask because we can typically pull more data then might be available readily on the site. Example being turn numbers, which while they can be calculated via subtracting the turn year from the starting year can get complicated since each variant has a different start year.

Just for people to keep in mind if looking for more data then readily shown on the game page.

Just for the sake of completeness (quitely ignoring the required time and effort to be put into the implementation): of course a machine-readable output of statistics for each game (e.g. in json format) would simplify external data collecting.

Yeah we're working on having that back up. It used to be there and was used for the ghostratings but got messed up during the last server upgrade. So A_Tin_Can and I remade it, but are awaiting to hear back from the person who ran it to ensure the data is how he needs it. Once so we'll readd it to the server so anyone can download it for their own statistical use.

That sounds good!