Python for Power Systems

A blog for power systems engineers to learn Python.

All You Need to Analyse the Electricity Market Pt 2

If you are an electrical engineer, and want to know how to use Python to get data from the internet and display it, this post is for you.

(This is the second part of a series. By the end we’ll have written a Python script to display a chart of electricity market prices.

Australian electricity prices are high – let’s analyse

Previously I mentioned that the electricity prices have gone through the roof (more than doubled) post introduction of the carbon tax in Australia.

Very high electricity prices [Larger Size]

We’re using these high prices as an excuse to learn how to download and display price data from the internet. This code will work wherever you are in the world.

Using Python to extract a zipped file

Last time, we downloaded a zipped file filled with electricity prices from the energy market operator’s website (read about it here if you want to catch up). Here is the final code from that post:

downloadzip.py
1
2
3
4
5
6
7
8
9
10

from __future__ import with_statement
from urllib2 import urlopen

PRICE_REPORTS_URL = 'http://www.nemweb.com.au/Reports/CURRENT/Public_Prices'
ZIP_URL = '/PUBLIC_PRICES_201207040000_20120705040607.ZIP'

zippedfile = urlopen(PRICE_REPORTS_URL + ZIP_URL)

with open('PUBLIC_PRICES.ZIP', 'wb') as pricesfile:
  pricesfile.write(zippedfile.read())

In this post we are going to open a zip file containing CSV files and read the data contained within. To begin, we open the zipped file and get a list of the file names inside it. We’ve used Python’s zipfile module and its ZipFile class to turn the zip file we downloaded into something that we can use.

A listing of all the filenames in the zip file can be produced using the namelist() method.

extractzip.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

from __future__ import with_statement
from urllib2 import urlopen
from StringIO import StringIO
from zipfile import ZipFile

PRICE_REPORTS_URL = 'http://www.nemweb.com.au/Reports/CURRENT/Public_Prices'
ZIP_URL = '/PUBLIC_PRICES_201207040000_20120705040607.ZIP'

# zippedfile is now one long string.
zippedfile = urlopen(PRICE_REPORTS_URL + ZIP_URL).read()

# StringIO turns the string into a real file-like object.
opened_zipfile = ZipFile(StringIO(zippedfile))
filenames = opened_zipfile.namelist()

print filenames
output of ‘print filenames’
1
2

# just one CSV file inside this zipped file.
['PUBLIC_PRICES_201207040000_20120705040607.CSV']

There was just one CSV file inside the zipped file we just opened. We will now add some code to open it and convert it to Python lists using the csv module included with Python. Here is a quick refresh on the CSV module.

extractziptocsv.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

from __future__ import with_statement
import csv
from urllib2 import urlopen
from StringIO import StringIO
from zipfile import ZipFile

PRICE_REPORTS_URL = 'http://www.nemweb.com.au/Reports/CURRENT/Public_Prices'
ZIP_URL = '/PUBLIC_PRICES_201207040000_20120705040607.ZIP'

# zippedfile is now one long string.
zippedfile = urlopen(PRICE_REPORTS_URL + ZIP_URL).read()

# StringIO turns the string into a real file-like object.
opened_zipfile = ZipFile(StringIO(zippedfile))

# assuming there is only one CSV in the zipped file.
csv_filename = opened_zipfile.namelist()[0]

prices_csv_file = opened_zipfile.open(csv_filename)

prices_csv_reader = csv.reader(prices_csv_file)

csv.reader creates an object that has special functions to effortlessly return the CSV price data as Python lists. Printing the object, print prices_csv_reader, shows how the csv module has read in the file:

output of print ‘prices_csv_reader’
1
2
3
4

[['C', 'NEMP.WORLD', 'PRICES', 'AEMO',  ...]
 ['I', 'DREGION', '', '2', 'SETTLEMENTDATE', 'RUNNO', 'REGIONID', ...]
  ...
]

Look carefully at that output, notice how csv.reader has detected all the commas and correctly separated all of the columns and rows. All the data is stored in a list of rows. Now that we have the data in this form, extracting useful data from this structure can be achieved using the usual Python techniques.

In the first two posts of this series, we’ve:

  1. downloaded a zipped file using urllib2;
  2. unzipped the file to get the CSV using zipfile;
  3. got ready to read the csv file contents using csv.

In the next blog post we’ll take a closer look at the CSV file using Python. In particular, we will manipulate the CSV file to extract only the data that is of interest to us.

If you want to be emailed when the next blog is ready, just enter your email up the top →. We’ll only email you when a new blog is ready and with tips on becoming an advanced Python using power systems engineer.