Find out how the new performance payments system can be gamed for profit and CAISO’s response to these claims.

Peak solar output increases over a two year period [Larger Size]

We found in a recent survey that peak solar power production in Amprion’s network has more than doubled since 2010.

An attractive investment

Sizeable increases have been attributed to the attractive feed-in tariff scheme set up by the German government.

The German solar industry is supported by feed in tariffs paid to solar energy producers. Solar farm owners of less than 10MW in size are paid handsomely for every kWh of energy they produce.

Amprion’s transmission region [Larger Size]

According to http://www.germanenergyblog.de/?p=9756 owners are paid between 13.1 and 19.92 European cents per kWh produced.

These payments are guaranteed for the next 20 years. The investment situation is very stable and banks love it.

Abandoning Nuclear power

The German government plans to abandon nuclear power by 2022. Nuclear plants produced about a third of Germany’s power, this leaves a large gap that renewable energy can fill. We think that the solar power production in Amprion’s transmission region will rise still higher in the future.

Do you think solar power output will have doubled again in two years time?

How did you get the solar production data?

The main focus of this blog is to make you a better Python programmer in the energy industry. Do you want to know where we found that solar power data?

Sign up for updates to our blog, we’ll cover it in the next post! Don’t miss it.

[..] I have a fancy now: to have applications like this one […] available for members interested

the 44th CIGRE Paris has 444 paper presentations

At the 44th CIGRE Paris this year, there are over 6000 engineers attending to see 444 paper presentations held in 4 rooms concurrently.

But CIGRE isn’t just about the presentations, its a time to duck out mid- session and take your loved one for a coffee, a time to meet with someone just like you that works in the same job in another country.

How do you fit it all in, and not miss those 10-30 “must see” papers?

Now there is the CIGRE paper scheduler A place for you to search and find the papers that interest you, and see when and where you need to be to watch the presentations. It has a calendar that updates live as you select papers, showing you when your free times are so you can schedule your own meeting on the Seine.

See you in Paris!

Used the CIGRE scheduler? We’re still improving it. What is one thing that you would change?

If you are an electrical engineer, and want to know how to use Python to get data from the Internet and display it, this post is for you.

This is the final part of a four part series. By the end we’ll have written a Python script to display a chart of electricity market prices.

• All you need to analyse the electricity market pt 1
• All you need to analyse the electricity market pt 2
• All you need to analyse the electricity market pt 3
• All you need to analyse the electricity market final

Australian electricity prices are high – let’s analyse

Very high electricity prices [Larger Size]

Previously I mentioned that the Australian electricity prices have gone through the roof (more than doubling) since the introduction of the carbon tax.

This series of posts is exploring how to analyse market data accessible from the internet. The methods described can be adapted to your country’s data or any sort of data available on the internet.

We began the series with a post detailing how to obtain a CSV file that contains the latest electricity market prices.

Then we unzipped the price data CSV file that was downloaded in Part 1 and had a brief look at its contents.

Then we pulled that CSV file apart using Python.

Now you’ll plot the prices and system demand using matplotlib.

The code we have developed over this series so far:

1. Downloads a zipped file;
2. Unzips it;
3. Reads the file contents as a CSV; and
4. Extracts the half hourly demand and price values.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

from __future__ import with_statement
import csv
import datetime
from urllib2 import urlopen
from StringIO import StringIO
from zipfile import ZipFile

PRICE_REPORTS_URL = 'http://www.nemweb.com.au/Reports/CURRENT/Public_Prices'
ZIP_URL = '/PUBLIC_PRICES_201207040000_20120705040607.ZIP'

# zippedfile is now one long string.
zippedfile = urlopen(PRICE_REPORTS_URL + ZIP_URL).read()

# StringIO turns the string into a real file-like object.
opened_zipfile = ZipFile(StringIO(zippedfile))

# assuming there is only one CSV in the zipped file.
csv_filename = opened_zipfile.namelist()[0]

prices_csv_file = opened_zipfile.open(csv_filename)

prices_csv_reader = csv.reader(prices_csv_file)

def is_halfhourly_data(row):
    """Returns True if the given row starts with 'D', 'TREGION', '', '1'"""
    return row[:4] == ["D", "TREGION", "", "1"]

halfhourly_data = filter(is_halfhourly_data, prices_csv_reader)

def get_date_region_and_rrp(row):
    """
    Returns the SETTLEMENTDATE, REGION and RRP from the given
    PUBLIC_PRICES CSV data row.

    SETTLEMENTDATE is converted to a Python date (the time is discarded);
    REGION is left as a string; and
    RRP is converted to a floating point.
    """
    return (datetime.datetime.strptime(row[4], '%Y/%m/%d %H:%M:%S').date(),
            row[6],
            float(row[7]))

date_region_price = map(get_date_region_and_rrp, halfhourly_data)

The completed example

Here is the complete code to download and plot the electricity prices with Python. We’ll step through the most important parts and show you two of Python’s advanced features defaultdict and yield.

If you aren’t interested in the advanced Python code, you can skip to the end. The matplotlib code that creates the chart is very short and easy to follow.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103

from __future__ import with_statement
from collections import defaultdict
import csv
import datetime
from urllib2 import urlopen
from StringIO import StringIO
from zipfile import ZipFile

import matplotlib.pyplot as plt

PRICE_REPORTS_URL = 'http://www.nemweb.com.au/Reports/CURRENT/Public_Prices'
ZIP_URL = '/PUBLIC_PRICES_201207040000_20120705040607.ZIP'
REGIONS = ("QLD1", "NSW1", "VIC1", "SA1", "TAS1")

# zippedfile is now one long string.
try:
    zippedfile = open(ZIP_URL.replace('/', '')).read()
except IOError:
    zippedfile = urlopen(PRICE_REPORTS_URL + ZIP_URL).read()
    f = open(ZIP_URL.replace('/', ''), 'wb')
    f.write(zippedfile)

# StringIO turns the string into a real file-like object.
opened_zipfile = ZipFile(StringIO(zippedfile))

# assuming there is only one CSV in the zipped file.
csv_filename = opened_zipfile.namelist()[0]

prices_csv_file = opened_zipfile.open(csv_filename)

prices_csv_reader = csv.reader(prices_csv_file)

def is_halfhourly_data(row):
    """Returns True if the given row starts with 'D', 'TREGION', '', '1'"""
    return row[:4] == ["D", "TREGION", "", "1"]

halfhourly_data = filter(is_halfhourly_data, prices_csv_reader)

def get_date_region_and_rrp(row):
    """
    Returns the SETTLEMENTDATE, REGION and RRP from the given
    PUBLIC_PRICES CSV data row.

    SETTLEMENTDATE is converted to a Python date (the time is discarded);
    REGION is left as a string; and
    RRP is converted to a floating point.
    """
    return (datetime.datetime.strptime(row[4], '%Y/%m/%d %H:%M:%S'),
            row[6],
            float(row[7]))

prices = map(get_date_region_and_rrp, halfhourly_data)

def get_region_price(date_region_prices, regions):
    """
    returns the dates and prices in two columns grouped by region,
    suitable for plotting with matplotlib.

    the order of returned prices is the same order as the `regions`
    argument.

    Args:
      date_region_prices: a list of (date, region, price) tuples
       [(datetime(2012, 08, 09), "QLD1", 45.6),
        (datetime(2012, 08, 09, 1), "NSW1", 46.0)
         ...
       ]
      regions: A list of the regions to return.
    >>> get_region_price([(datetime(2012, 09, 09), "QLD1", 43.2),
                          (datetime(2012, 09, 09), "NSW1", 45.5),
                          (datetime(2012, 09, 10), "NSW1", 44.2),
                          ...],
                          ("NSW1", "QLD1"))

    [(datetime(2012, 09, 09), datetime(2012, 09, 10))
     (45.5, 44.2)],

    [(datetime(2012, 09, 09),)
     (43.2,)]

    ...

    """
    region_prices = defaultdict(list)
    for date, region, price in date_region_prices:
        region_prices[region  + 'd'].append(date)
        region_prices[region  + 'p'].append(price)

    for region in regions:
        yield region_prices[region+'d'], region_prices[region+'p']

figure = plt.figure()

for dates, prices in get_region_price(prices, REGIONS):
    plt.plot(dates, prices, '-')

plt.legend(REGIONS)
plt.grid()
plt.xlabel("Time of day")
plt.ylabel("Electricity Price A$/MWh")
figure.autofmt_xdate()

plt.show()

Grouping the regions together

I want to plot each of the five Australian regions’s prices as a separate series. But I don’t have the data organised into separate x axis and y axis values. Instead there is one long Python list that has all regions.

1
2
3
4
5
6
7
8
9

[
('2012-01-01 00:00', 'NSW1', 34.5),
('2012-01-01 00:00', 'VIC1', 33.2),
('2012-01-01 00:00', 'SA1',  36.1),
('2012-01-01 00:00', 'TAS1', 38.2),
('2012-01-01 00:00', 'QLD1', 37.4),
('2012-01-01 00:30', 'NSW1', 34.6),
...
]

Here is a way to use defaultdict to collect the date and price per region. For example the 'NSW1' and 'VIC1' regions. The defaultdict (official docs) is just like a normal dictionary, except that it has one additional powerful feature:

`defaultdict` will auto-initialise a new value if you attempt to access a `key` that is missing.

Confused? Here is a concrete example. Grouping power station names by generator category (Nuclear, Wind,..) using a normal Python dict:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

gens = {}
gens['NUCLEAR'] = ['NUCLEAR-1', 'HILLVIEW-2', 'NUCLEAR-2']

# we can add a new nuclear generator name.
gens['NUCLEAR'].append('HILLVIEW-1')

# we can't add a new wind farm name - yet.
gens['WINDFARM'].append('WINDY-HILL-1')
# KeyError 'WINDFARM'!

# first must make a new empty wind farm list
gens['WINDFARM'] = []
# now this works.
gens['WINDFARM'].append('WINDY-HILL-1')

A KeyError exception will be raised on line 8, because 'WINDFARM' is a key that doesn’t exist in the gens dictionary yet. It isn’t until line 12 that the 'WINDFARM' key is entered into the dictionary and the first wind farm can be appended.

Here is the same code using defaultdict to initialise an empty list when there is a missing key. Notice that there is no need to create a key with an empty list before appending.

1
2
3
4
5
6
7
8
9
10

from collections import defaultdict

gens = defaultdict(list)

# empty list created for 'NUCLEAR' and straight away we extend it.
gens['NUCLEAR'].extend(['NUCLEAR-1', 'HILLVIEW-2', 'NUCLEAR-2'])
gens['NUCLEAR'].append('HILLVIEW-1')

# empty list created for 'WINDFARM' and straight away we append.
gens['WINDFARM'].append('WINDY-HILL-1')

Having seen defaultdict take another look at this code section from final.py:

1
2
3
4

region_prices = defaultdict(list)
for date, region, price in date_region_prices:
    region_prices[region  + 'd'].append(date)
    region_prices[region  + 'p'].append(price)

It makes two lists for every region. The key to the first list region + 'd' would look like NSW1d or VIC1d. The key to the second list is region + 'p' and looks like NSW1p or VIC1p.

The d stands for date, our x axis and the p stands for price, our y axis. Its time to plot those x and y values.

Making your own iterator

Use the yield keyword in a function to turn that function into something that can be used in a forloop (an iterable).

1
2

    for region in regions:
        yield region_prices[region+'d'], region_prices[region+'p']

I used the yield keyword in the get_region_price function to return the date and price (x and y axis) pairs that were grouped using defaultdict. They are returned one region at a time in the forloop.

1
2

for dates, prices in get_region_price(prices, REGIONS):
    plt.plot(dates, prices, '-')

yield will take some getting used to if you’ve never seen it before. Try working with this script on your computer so you can see what is happening:

1
2
3
4
5
6
7
8
9
10
11
12
13

names_age = [("NUCLEAR1", 10), ("NUCLEAR2", 11), ("WINDYPEAK", 2)]

def generator_names(names_age):

  print '[generator_names] started the generator_names generator'

  for name, age in names_age:
    print '[generator_names] about to yield', name
    yield name

for name in generator_names(names_age):
  print 'I got name: ', name

1
2
3
4
5
6
7

[generator_names] started the generator_names generator
[generator_names] about to yield NUCLEAR1
I got name:  NUCLEAR1
[generator_names] about to yield NUCLEAR2
I got name:  NUCLEAR2
[generator_names] about to yield WINDYPEAK
I got name:  WINDYPEAK

Plotting professionally in only 8 lines of code

Plotting the dates and prices is very easy once you have them in two lists, x axis and y axis.

The plot commands are similar to Matlab plotting routines:

1
2
3
4
5
6
7
8
9
10
11
12

figure = plt.figure()

for dates, prices in get_region_price(prices, REGIONS):
    plt.plot(dates, prices, '-')

plt.legend(REGIONS)
plt.grid()
plt.xlabel("Time of day")
plt.ylabel("Electricity Price A$/MWh")
figure.autofmt_xdate()

plt.show()

Here is the list of steps in the code above.

1. Create an empty figure;
2. Plot each region’s prices;
3. Display a legend;
4. Enable grid lines;
5. Set the x-axis label;
6. Set the y-axis label;
7. Auto-rotate the x axis date labels; and
8. Show the plot.

Conclusion

The final program is quite short, just 100 lines of code. But it covers such a wide range of tasks:

• Downloading files from the internet;
• Unzipping files;
• Reading CSV files;
• Sorting, transposing and filtering data; and
• Displaying data on a chart.

The post may not be clear in certain areas, or you may want us to write about something in more detail, so tell us using the form below.

If you are an electrical engineer, and want to know how to use Python to get data from the Internet and display it, this post is for you.

(This is the third part of a four part series. By the end we’ll have written a Python script to display a chart of electricity market prices. Enter your email → on the right so that you don’t miss the final post.)

• All you need to analyse the electricity market pt 1
• All you need to analyse the electricity market pt 2
• All you need to analyse the electricity market pt 3
• All you need to analyse the electricity market final

Australian electricity prices are high – let’s analyse

Very high electricity prices [Larger Size]

Previously I mentioned that the Australian electricity prices have gone through the roof (more than doubling) since the introduction of the carbon tax.

This series of posts is exploring how to analyse market data accessible from the internet. The methods described can be adapted to your country’s data or any sort of data available on the internet.

We began the series with a post detailing how to obtain a CSV file that contains the latest electricity market prices.

Then we unzipped the price data CSV file that was downloaded in Part 1 and had a brief look at its contents.

Now we will teach you how to pull that CSV file apart using Python. You will master the ability to highlight the columns and rows that you are interested in. Just like top Japanese chefs are qualified to cut the good meat from the fugu fish, you too will learn to slice the good data from the CSV file.

The code we have developed over this series so far:

1. downloads a zipped file;
2. unzips it; and
3. reads the file contents as a CSV.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

from __future__ import with_statement
import csv
from urllib2 import urlopen
from StringIO import StringIO
from zipfile import ZipFile

PRICE_REPORTS_URL = 'http://www.nemweb.com.au/Reports/CURRENT/Public_Prices'
ZIP_URL = '/PUBLIC_PRICES_201207040000_20120705040607.ZIP'

# zippedfile is now one long string.
zippedfile = urlopen(PRICE_REPORTS_URL + ZIP_URL).read()

# StringIO turns the string into a real file-like object.
opened_zipfile = ZipFile(StringIO(zippedfile))

# assuming there is only one CSV in the zipped file.
csv_filename = opened_zipfile.namelist()[0]

prices_csv_file = opened_zipfile.open(csv_filename)

prices_csv_reader = csv.reader(prices_csv_file)

The example CSV file downloaded earlier had over 30 columns of information and many thousands of rows. We’ll use Python to get exactly the columns and rows that we want.

Which rows are important? Knowing what is in the CSV file is paramount at this stage. To this end, the site that provides the data may also provide a specification of the file structure. Failing that, you may have to get intimate with the data and spend a bit of time working out the format for yourself.

For the data provided by the Australian electricity market operator, the first CSV column is a label. Each label describes the purpose of that row. There are three values, C, I or D. Shown below is an example of the data stored in the first column,

1
2
3
4
5
6
7
8
9
10
11

C,
I,
D,
D,
D,
D,
I,
D,
D,
I,
...

Rows marked with a C are comment rows, they give further information about the file itself but aren’t necessary for us to worry about.

Rows marked with an I are header rows. The header row is just like a header row you use in a normal Microsoft Excel spreadsheet, it indicates what data is stored in that column. For our goal of finding the price of electricity in different regions of Australia over time, the columns that we are looking for are SETTLEMENTDATE (date and time), REGIONID (price region) and RRP (electricity price $/MWh).

Rows marked with a D are the data rows. We’ll take these rows for the SETTLEMENTDATE, REGIONID and RRP then plot them on a chart.

Multiple header rows in a CSV file?

Immediately though, we run into a problem. Notice in the CSV file structure figure shown above that there are multiple I header rows? There are no less than four in the CSV file we downloaded. You can think of it as four CSV files crammed into a single file.

We are only interested in one of these four sections, the section with the columns we mentioned before, SETTLEMENTDATE, REGIONID and RRP. Through analysis of the file structure, we know that all the data rows we are interested in all begin with:

1

D,TREGION,,1

Master technique one: filter

Here is how to update the program to print the rows beginning with D,TREGION,,1:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

from __future__ import with_statement
import csv
from urllib2 import urlopen
from StringIO import StringIO
from zipfile import ZipFile

PRICE_REPORTS_URL = 'http://www.nemweb.com.au/Reports/CURRENT/Public_Prices'
ZIP_URL = '/PUBLIC_PRICES_201207040000_20120705040607.ZIP'

# zippedfile is now one long string.
zippedfile = urlopen(PRICE_REPORTS_URL + ZIP_URL).read()

# StringIO turns the string into a real file-like object.
opened_zipfile = ZipFile(StringIO(zippedfile))

# assuming there is only one CSV in the zipped file.
csv_filename = opened_zipfile.namelist()[0]

prices_csv_file = opened_zipfile.open(csv_filename)

prices_csv_reader = csv.reader(prices_csv_file)

def is_halfhourly_data(row):
    """Returns True if the given row starts with 'D', 'TREGION', '', '1'"""
    return row[:4] == ["D", "TREGION", "", "1"]

print filter(is_halfhourly_data, prices_csv_reader)

The filter function will only return the rows for which the function is_halfhourly_data returns True.

Master technique 2: map

Having correctly isolated the rows that interest us, slice up those columns and get the SETTLEMENTDATE, REGIONID and RRP columns (column numbers 4, 6 and 7).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

from __future__ import with_statement
import csv
from urllib2 import urlopen
from StringIO import StringIO
from zipfile import ZipFile

PRICE_REPORTS_URL = 'http://www.nemweb.com.au/Reports/CURRENT/Public_Prices'
ZIP_URL = '/PUBLIC_PRICES_201207040000_20120705040607.ZIP'

# zippedfile is now one long string.
zippedfile = urlopen(PRICE_REPORTS_URL + ZIP_URL).read()

# StringIO turns the string into a real file-like object.
opened_zipfile = ZipFile(StringIO(zippedfile))

# assuming there is only one CSV in the zipped file.
csv_filename = opened_zipfile.namelist()[0]

prices_csv_file = opened_zipfile.open(csv_filename)

prices_csv_reader = csv.reader(prices_csv_file)

def is_halfhourly_data(row):
    """Returns True if the given row starts with 'D', 'TREGION', '', '1'"""
    return row[:4] == ["D", "TREGION", "", "1"]

halfhourly_data = filter(is_halfhourly_data, prices_csv_reader)

def get_date_region_and_rrp(row):
    return (row[4], row[6], row[7])

date_region_price = map(get_date_region_and_rrp, halfhourly_data)

The map function will return a list of the result of get_date_region_and_rrp called with each row in halfhourly_data as an argument.

1
2
3
4

def get_date_region_and_rrp(row):
    return (row[4], row[6], row[7])

date_region_price = map(get_date_region_and_rrp, halfhourly_data)

The above section of code using map is the equivalent of this code using for loop.

1
2
3
4
5
6
7
8

def get_date_region_and_rrp(row):
    return (row[4], row[6], row[7])

date_region_price = []

for row in halfhourly_data:
    new_row = get_date_region_and_rrp(row)
    date_region_price.append(new_row)

map is an extremely versatile function. Those four lines of for loop code are replaced with one map line of code.

And now only the good data remains

The date_region_price variable will have these contents:

1
2
3
4
5

[["2012/04/02 10:30:00", "NSW1", "34.40"],
 ["2012/04/02 10:30:00", "QLD1", "34.67"],
 ["2012/04/02 10:30:00", "VIC1", "35.83"],
 ...
]

Only the columns that we are interested in. Nice work!

However there is still a small problem. All of the data values are still text. Update the get_date_region_and_rrp to convert the first column to a date, keep the second as a string and the third to a floating point value.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

from __future__ import with_statement
import csv
import datetime
from urllib2 import urlopen
from StringIO import StringIO
from zipfile import ZipFile

PRICE_REPORTS_URL = 'http://www.nemweb.com.au/Reports/CURRENT/Public_Prices'
ZIP_URL = '/PUBLIC_PRICES_201207040000_20120705040607.ZIP'

# zippedfile is now one long string.
zippedfile = urlopen(PRICE_REPORTS_URL + ZIP_URL).read()

# StringIO turns the string into a real file-like object.
opened_zipfile = ZipFile(StringIO(zippedfile))

# assuming there is only one CSV in the zipped file.
csv_filename = opened_zipfile.namelist()[0]

prices_csv_file = opened_zipfile.open(csv_filename)

prices_csv_reader = csv.reader(prices_csv_file)

def is_halfhourly_data(row):
    """Returns True if the given row starts with 'D', 'TREGION', '', '1'"""
    return row[:4] == ["D", "TREGION", "", "1"]

halfhourly_data = filter(is_halfhourly_data, prices_csv_reader)

def get_date_region_and_rrp(row):
    """
    Returns the SETTLEMENTDATE, REGION and RRP from the given
    PUBLIC_PRICES CSV data row.

    SETTLEMENTDATE is converted to a Python date (the time is discarded);
    REGION is left as a string; and
    RRP is converted to a floating point.
    """
    return (datetime.datetime.strptime(row[4], '%Y/%m/%d %H:%M:%S').date(),
            row[6],
            float(row[7]))

date_region_price = map(get_date_region_and_rrp, halfhourly_data)

Great, now our data is in a format that Python can understand and plot. This is what is contained in the date_region_price value now:

1
2
3
4
5

[[datetime.date(2012, 4, 2), "NSW1", 34.40],
 [datetime.date(2012, 4, 2), "QLD1", 34.67],
 [datetime.date(2012, 4, 2), "VIC1", 35.83],
 ...
]

Perfect. We now have all the data formatted exactly the way we want it.

Conclusion

We’ve used filter and map to quickly and efficiently sort and slice the CSV data. Just like a master Japanese chef, I’m sure that you will not poison your patrons with bad slices of data. filter and map are advanced level functions that are often used to replace for loops. Please practice using map and filter. Experiment with them so that you understand how they work!

In the next, and final, blog post of this series we’ll show you how to plot the results using matplotlib.

Just what went wrong yesterday in Alberta to cause rolling blackouts and electricity prices that were 50 times higher than normal?

If you are an electrical engineer, and want to know how to use Python to get data from the internet and display it, this post is for you.

(This is the second part of a series. By the end we’ll have written a Python script to display a chart of electricity market prices.

• All you need to analyse the electricity market pt 1
• All you need to analyse the electricity market pt 2
• All you need to analyse the electricity market pt 3
• All you need to analyse the electricity market final

Australian electricity prices are high – let’s analyse

Previously I mentioned that the electricity prices have gone through the roof (more than doubled) post introduction of the carbon tax in Australia.

Very high electricity prices [Larger Size]

We’re using these high prices as an excuse to learn how to download and display price data from the internet. This code will work wherever you are in the world.

Using Python to extract a zipped file

Last time, we downloaded a zipped file filled with electricity prices from the energy market operator’s website (read about it here if you want to catch up). Here is the final code from that post:

If you are an electrical engineer, and want to know how to use Python to get data from the internet and display it, this post is for you.

(This is the first part of a series. By the end we’ll have written a Python script to display a chart of electricity market prices.

• All you need to analyse the electricity market pt 1
• All you need to analyse the electricity market pt 2
• All you need to analyse the electricity market pt 3
• All you need to analyse the electricity market final

Electricity prices in Australia have gone bananas

Since the carbon tax was introduced in Australia, the spot price for electricity has been very high. From $22 / MWh last month to over $50 / MWh so far (see the chart below). Whether this is a long term change, or just a temporary reaction, we don’t know. Instead of considering the complexities of market forces, we’re going to show you step by step exactly how to build your own Python program to display electricity prices, using Australia as an example.

Very high electricity prices [Larger Size]

Feel free to take the code, and configure it for your own country’s system.

Downloading a zip file.

Australia’s electricity market operator (AEMO) keeps all of the market data online at http://www.nemweb.com.au/Reports/. The webpages are written in simple to understand HTML and link to zipped CSV files.

We’ll use Python to write a simple script that downloads a zipped file.

1
2
3
4
5
6
7
8
9
10

from __future__ import with_statement
from urllib2 import urlopen

PRICE_REPORTS_URL = 'http://www.nemweb.com.au/Reports/CURRENT/Public_Prices'
ZIP_URL = '/PUBLIC_PRICES_201207040000_20120705040607.ZIP'

zippedfile = urlopen(PRICE_REPORTS_URL + ZIP_URL)

with open('PUBLIC_PRICES.ZIP', 'wb') as pricesfile:
  pricesfile.write(zippedfile.read())

You may find that the link is expired, so you’ll get an error like this:

1
2
3
4
5
6
7
8
9
10
11
12
13

  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found

Got the error above? Change ZIP_URL to match the url of a zip you can find on this http://www.nemweb.com.au/Reports/CURRENT/Public_Prices page.

Unzip the downloaded file (the file is named PUBLIC_PRICES.ZIP) and look at the CSV file inside. You will find something like this:

1
2
3
4
5
6
7

C,NEMP.WORLD,PRICES,AEMO,PUBLIC,2012/04/02,04:06:14,0000000235749444,,0000000235749438
I,DREGION,,2,SETTLEMENTDATE,RUNNO,REGIONID,INTERVENTION,RRP,EEP,ROP,APCFLAG,MARKETSUSPENDEDFLAG,TOTALDEMAND,DEMANDFORECAST,DISPATCHABLEGENERATION,DISPATCHABLELOAD,NETINTERCHANGE,EXCESSGENERATION,LOWER5MINDISPATCH,LOWER5MINIMPORT,LOWER5MINLOCALDISPATCH,LOWER5MINLOCALPRICE,LOWER5MINLOCALREQ,LOWER5MINPRICE,LOWER5MINREQ,LOWER5MINSUPPLYPRICE,LOWER60SECDISPATCH,LOWER60SECIMPORT,LOWER60SECLOCALDISPATCH,LOWER60SECLOCALPRICE,LOWER60SECLOCALREQ,LOWER60SECPRICE,LOWER60SECREQ,LOWER60SECSUPPLYPRICE,LOWER6SECDISPATCH,LOWER6SECIMPORT,LOWER6SECLOCALDISPATCH,LOWER6SECLOCALPRICE,LOWER6SECLOCALREQ,LOWER6SECPRICE,LOWER6SECREQ,LOWER6SECSUPPLYPRICE,RAISE5MINDISPATCH,RAISE5MINIMPORT,RAISE5MINLOCALDISPATCH,RAISE5MINLOCALPRICE,RAISE5MINLOCALREQ,RAISE5MINPRICE,RAISE5MINREQ,RAISE5MINSUPPLYPRICE,RAISE60SECDISPATCH,RAISE60SECIMPORT,RAISE60SECLOCALDISPATCH,RAISE60SECLOCALPRICE,RAISE60SECLOCALREQ,RAISE60SECPRICE,RAISE60SECREQ,RAISE60SECSUPPLYPRICE,RAISE6SECDISPATCH,RAISE6SECIMPORT,RAISE6SECLOCALDISPATCH,RAISE6SECLOCALPRICE,RAISE6SECLOCALREQ,RAISE6SECPRICE,RAISE6SECREQ,RAISE6SECSUPPLYPRICE,AGGREGATEDISPATCHERROR,AVAILABLEGENERATION,AVAILABLELOAD,INITIALSUPPLY,CLEAREDSUPPLY,LOWERREGIMPORT,LOWERREGLOCALDISPATCH,LOWERREGLOCALREQ,LOWERREGREQ,RAISEREGIMPORT,RAISEREGLOCALDISPATCH,RAISEREGLOCALREQ,RAISEREGREQ,RAISE5MINLOCALVIOLATION,RAISEREGLOCALVIOLATION,RAISE60SECLOCALVIOLATION,RAISE6SECLOCALVIOLATION,LOWER5MINLOCALVIOLATION,LOWERREGLOCALVIOLATION,LOWER60SECLOCALVIOLATION,LOWER6SECLOCALVIOLATION,RAISE5MINVIOLATION,RAISEREGVIOLATION,RAISE60SECVIOLATION,RAISE6SECVIOLATION,LOWER5MINVIOLATION,LOWERREGVIOLATION,LOWER60SECVIOLATION,LOWER6SECVIOLATION,RAISE6SECRRP,RAISE6SECROP,RAISE6SECAPCFLAG,RAISE60SECRRP,RAISE60SECROP,RAISE60SECAPCFLAG,RAISE5MINRRP,RAISE5MINROP,RAISE5MINAPCFLAG,RAISEREGRRP,RAISEREGROP,RAISEREGAPCFLAG,LOWER6SECRRP,LOWER6SECROP,LOWER6SECAPCFLAG,LOWER60SECRRP,LOWER60SECROP,LOWER60SECAPCFLAG,LOWER5MINRRP,LOWER5MINROP,LOWER5MINAPCFLAG,LOWERREGRRP,LOWERREGROP,LOWERREGAPCFLAG
D,DREGION,,2,"2012/04/01 04:05:00",1,NSW1,0,24.01213,0,24.01213,0,,5880.16,1.20264,4358.97,0,-1521.19,0,,,100.55,,,,,,,,109.7,,,,,,,,34.35,,,,,,,,206,,,,,,,,94.27,,,,,,,,116.69,,,,,,0,10832.745,0,5907.18945,5912.38,,50,,,,38,,,,,,,,,,,,,,,,,,,0.96,0.96,0,0.39,0.39,0,0.85,0.85,0,1.35,1.35,0,0.5,0.5,0,1.5,1.5,0,3.85809,3.85809,0,6.222,6.222,0
D,DREGION,,2,"2012/04/01 04:05:00",1,QLD1,0,21.55498,0,21.55498,0,,4423.12,-0.58301,5327.25,0,904.13,0,,,13,,,,,,,,42.65,,,,,,,,40,,,,,,,,83,,,,,,,,82,,,,,,,,62,,,,,,0,10258,0,4442.88184,4441.41,,34,,,,76.44,,,,,,,,,,,,,,,,,,,0.96,0.96,0,0.39,0.39,0,0.85,0.85,0,1.35,1.35,0,0.5,0.5,0,1.5,1.5,0,3.85809,3.85809,0,6.222,6.222,0
D,DREGION,,2,"2012/04/01 04:05:00",1,SA1,0,22.2124,0,22.2124,0,,1144.89,-1.98004,1085.43,0,-59.46,0,,,0,,,,,,,,2,,,,,,,,2,,,,,,,,0,,,,,,,,30,,,,,,,,40.4,,,,,,4.93256,2768.82735,0,1141.90503,1144.93,,16,,,,0,,,,,,,,,,,,,,,,,,,0.96,0.96,0,0.39,0.39,0,0.85,0.85,0,1.35,1.35,0,0.5,0.5,0,1.5,1.5,0,3.85809,3.85809,0,6.222,6.222,0
D,DREGION,,2,"2012/04/01 04:05:00",1,TAS1,0,30.8637,0,30.8637,0,,896.9,3.52316,469.4,0,-427.5,0,,,47.74,,,,,,,,0,,,,,,,,13.59,,,,,,,,80.29,,,,,,,,73.59,,,,,,,,48.77,,,,,,-0.61674,2000,0,893.98877,896.9,,0,,,,6.53,,,,,,,,,,,,,,,,,,,1.1,1.1,0,0.9,0.9,0,2.3,2.3,0,2.8,2.8,0,0.1,0.1,0,0,0,0,0.1,0.1,0,2.46391,2.46391,0
D,DREGION,,2,"2012/04/01 04:05:00",1,VIC1,0,21.7,0,21.7,0,,4094.7,-22.9043,5268.16,0,1173.46,0,,,46,,,,,,,,40,,,,,,,,40,,,,,,,,11,,,,,,,,33,,,,,,,,45,,,,,,0,8780,0,4134.81494,4113.61,,20,,,,9.03,,,,,,,,,,,,,,,,,,,0.96,0.96,0,0.39,0.39,0,0.85,0.85,0,1.35,1.35,0,0.5,0.5,0,1.5,1.5,0,3.85809,3.85809,0,6.222,6.222,0

The file is very long. In fact there are four separate CSV files crammed into this one long CSV file. In the next blog post we’ll examine exactly how to unzip the file you just downloaded with Python and then use one of these CSV files.

If you want to be emailed when the next blog is ready, just enter your email up the top →. We’ll only email you when a new blog is ready and with tips on becoming an advanced Python using power systems engineer.

Graphics, when used correctly, can be the difference between an outstanding report and simply another piece of paper going around the office. The only thing more professional than a nice graphic, is a graphic that is automatically generated.

Exporting plots from the PSSE graphical interface can be a laborious task, with little flexibility in the final result. If you are running PSSE from Python, you have the option of looking beyond PSSE for tools to render your data.

matplotlib is a Python plotting tool which, given a little bit of effort, can make your life easier by automating figure generation; spend the time to get it right once, then generate the monthly figures with the press of a button.

matplotlib’s syntax is closely aligned with Matlab’s, which is fortunate if you are familiar with Matlab. If you are not familiar with them, don’t worry, the matplotlib community has provide an excellent tutorial and gallery displaying the immense possibilities of matplotlib (all examples come with code included for you to use as a starting point).

I’ll use matplotlib to create this chart:

It is the QV curve at bus 20001 (my favourite bus as it happens) and was plotted from data kept in a CSV file (qv.csv).

Here is the code we used to create that chart.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

from __future__ import with_statement
import csv

from pylab import plot, title, show, grid, xlabel, ylabel

# CSV file in format: V, Q
QV_CSVFILENAME = 'qv.csv'

def main():
  with open(QV_CSVFILENAME) as qvcsvfile:
      reader = csv.reader(qvcsvfile)
      # skip header row.
      header = reader.next()
      voltage_and_reactive = list(reader)

  # now convert the voltages and power values to floats.
  voltages, reactive = [], []
  for voltage, q in voltage_and_reactive:
      voltages.append(float(voltage))
      reactive.append(float(q))

  # now make the chart.
  plot(voltages, reactive, '-o')
  title("QV curve for bus number 20001")
  ylabel("Reactive Power [MVAr]")
  xlabel("Voltage [pu]")
  grid()
  show()

if __name__ == '__main__':
  main()

I’m using the pylab module of matplotlib. The pylab module most closely resembles Matlab code so its a really familiar place to start learning how to use matplotlib.

The first two sections open up the CSV file and convert it into a list of voltage and reactive floating point values.

Now, I’ll pull a section of that code out so that we can look closely:

1
2
3
4
5
6
7

   # now make the chart.
  plot(voltages, reactive, '-o')
  title("QV curve for bus number 20001")
  ylabel("Reactive Power [MVAr]")
  xlabel("Voltage [pu]")
  grid()
  show()

Thats the code that actually turns the lists of voltages and reactive values into a chart. The functions describe themselves pretty well. At the end of the routine show will pop up an interactive graph.

With the interactive graph you can:

• zoom in and out;
• pan around; and
• export the chart to an image file.

If you haven’t ever used matplotlib go ahead and try this example out. Before you do run the Python code here, just make sure you have both matplotlib and numpy installed. Install numpy first because matplotlib definitely needs it and will not install without it.

You’ve seen how easy it can be to start making your own charts with matplotlib. The quality of the resulting images is really exceptional. We’ve used them throughout many industry published documents. If you have any questions about making your own charts, jump onto our forum for power systems engineers we’ll help you out.

If you made it this far, you must be pretty keen on Python. We are too! Let us update you when we have something new to say about Python and Power Systems (semi regularly). Get Python Tips (subscribe on the right) →

Sri Lanka’s demand for electricity is growing quickly

Sri Lanka’s peak electricity demand of 2000 MW is growing at a rapid 5%-8% every year.

To keep pace with the increase, the Ceylon Electricity Board (CEB) have developed a clever plan. Over the next 10 years, over 400 MW of mini hydro and wind farms will be installed. It is anticipated the additional generation will meet the increased demand levels and reduce Sri Lanka’s dependance on foreign fossil fuel sources, such as coal, natural gas and petroleum.

Sri Lanka currently has over 200 MW of mini hydro power stations installed, which are not dispatched centrally, but shown as a reduction in demand. Additionally, there is 30 MW of wind generation installed and another 60 MW soon to be connected.

We met with Mr. Shihan Diddeniya, CEO of renewable energy at CEB, to learn more about Sri Lanka’s plan to increase the capacity of intermittent renewable generation.

Rapid growth in demand

Mr. Diddeniya explained that Sri Lanka currently has about 88% electrification. That is, 88% of households in Sri Lanka have direct access to the electricity grid. The proportion of electrified homes is growing quickly with complete electrification likely to occur over the next few years. The demand growth rate is a staggering 8% and once total electrification is complete, growth will remain at a high 5%.

A surplus of generation at low demand periods

Shihan told of how the challenge for Sri Lanka is not with fault level contributions from embedded mini hydros or building lines to remote wind farms. He emphasized Sri Lanka’s challenge is with excess generation capacity during low demand periods where electricity usage drops dramatically below the 2000 MW peak to 860 MW.

Mini hydro and wind power plant output cannot be reduced because their power is given first preference. As an example, if run of the river mini hydro and wind power plants were to be producing 400 MW, then only 460 MW remains to be supplied by thermal coal stations. The surplus generation could be resolved by switching off coal fired stations. But coal power stations are limited in their ability to switch on and off quickly, it may take days for a coal power station to run at full capacity from a cold start.

Sri Lanka will soon have 100% electrification

A new link to India

A transmission interconnect between Sri Lanka and India, which has been contemplated since the 70’s, is expected to be started after 2013. The 285km, high voltage DC link will have a total capacity of 1000MW, to be installed in two 500MW stages.

The extra capacity made available by this link will go a long way to relieve the immediate pressures the grid is currently experiencing.

Mr. Diddeniya will raise funds to complete the new renewable energy projects from local and international banks and investors during the next few years.