Python, the programming language, and netCDF, the file format used by the Bureau of Meteorology to publish weather forecast data, are an ideal combination. We recently were asked to develop a process that would extract the data from netCDF files and write it to a MySQL database.
Australian Digital Forecast Database
The Australian Digital Forecast Database (ADFD) contains official weather forecast elements produced by the Bureau of Meteorology. The weather elements include temperature, rainfall and weather types, presented in a gridded format. There are a set of weather element grids for different regions in Australia as well as a grid for the entire Australian continent. More information about the database can be found in the user guide.
The ADFD files are updated routinely twice per day at around 6am and 6pm local time. They are made available via an ftp service. A subscription is required to access the service. Two formats are made available but netCDF (Network Common Data Form) was chosen because of its proven integration with python.
netCDF files store weather forecast values in three dimensional arrays. There is a forecast value for a given point in Australia at a given time in the future. The value is therefore obtained using 3 ordinate values for row, column and tile, these correspond to latitude, longitude and time respectively. The rows have a constant latitude value, the columns have a constant longitude and the tiles each have a constant time value. Each are stored in one dimensional arrays in the file and are accessed when the files are opened. The weather element can be directly accessed from the 3 dimensional array and the latitude, longitude and time values can be directly accessed from their individual arrays.
Australian Digital Forecast Database – Bureau of Meteorology
There is a file for each weather element and there are over 25 weather element files including temperature, apparent temperature, dewpoint temperature, relative humidity, chance of rain and wind speed.
netCDF4 Python Package
The netCDF4 python package was used to read the netCDF files. There is also a Scientific.IO.NetCDF package for python but this was not used because it has been deprecated. NetCDF4 also integrates with the standard python numerical package (numpy) so that values can be accessed directly via arrays. NetCDF4 has prerequisite packages including HDF5, six and numpy.
Python code for reading netCDF
The following code is all that is required to read a value from a netCDF file at the origin (position and time). Note that this will work for a T_SFC file published by the Bureau of Meteorology. Other files may have different variable names and these can also be extracted using the attributes of the netCDF4 data set object.
from netCDF4 import Dataset
import numpy
ncdf = Dataset(‘c:\\data\\IDZ71000_AUS_T_SFC.nc’, mode=’r’)
lons = ncdf.variables[‘longitude’][:]
lats = ncdf.variables[‘latitude’][:]
weather_elements = ncdf.variables[‘T_SFC’][:] #T_SFC is the name of the temperature weather element in IDZ71000_AUS_T_SFC.nc
time_values = ncdf.variables[‘time’][:]
#lons, lats, weather_elements and time_values are numpy arrays
#print the longitude, latitude, time and temperature (T_SFC) at the first data point at the initial time
print lons[0], lats[0], time_values[0], weather_elements[0,0,0]
ncdf.close()
Using the above basics it is relatively easy to construct 3 nested FOR loops to extract all the values in the file. The size of the weather_elements array can be found using the shape attribute. For a 3 dimensional array the shape attribute returns a tuple of array dimensions. The individual dimensions can be obtained as follows:
count_time = weather_elements.shape[0]
count_lat = weather_elements.shape[1]
count_lng = weather_elements.shape[2]
The time values are in Coordinated Universal Time (UTC) and are represented as seconds since 1/1/1970. So typically these would need to be converted to a valid DateTime object.
Reading the netCDF files in this way using Python turned out to be the most efficient way of directly accessing data in these files. Note that ArcMap can directly read and display netCDF files however in this case the client did not have access to an ArcGIS licence.
Image courtesy of www.surf-forecast.com