Development, installation and testing of new models in the New Zealand Earthquake Forecast Testing Centre

Summary

The New Zealand Earthquake Forecast Testing Centre is one of four existing regional testing centres so far established around the globe to undertake rigorous tests of proposed earthquake forecasting models. The primary purpose of the testing centres is to carry out prospective testing but, because implementation of models for the testing centre is technically demanding, it is also useful to undertake retrospective testing to determine whether models have been properly implemented and installed.

In this study we report on the development and/or implementation for the New Zealand testing centre of six new models, including one new long-term model (a 2010 major revision of the New Zealand national seismic hazard model), three new medium-term models (which are updated at 3-month intervals) and two new short-term models (which are updated daily). Except for one of the new short-term models, these models have all been installed in the testing centre. The intention is to test all classes of model prospectively over an extended period of five or more years.

The new models and ten previously-installed models have been tested retrospectively using two simple statistical tests. The first test compares the total number of earthquakes expected under each model with the number observed over the test period. The second test estimates the overall information gain of one model over another. We use new versions of these tests which are easier to interpret than comparable tests which are part of the standard testing centre software. The retrospective testing has allowed for problems with the implementation of several of the models to be identified now rather than after years of computer-intensive testing.

Fifteen models were retrospectively tested. For seven models there is no indication of incorrect implementation. However, for five models there is a weak indication of incorrect implementation, and for three models, including two of the new models, there is a strong indication of incorrect implementation. The results will be referred back to the modellers, so that the implementation can be corrected if necessary.

The results of retrospective testing have thus been highly instructive in revealing teething problems with implementation of both new and existing models. The incidence of model implementation errors revealed in this study is similar to that found in retrospective testing of models submitted to other regional testing centres.

---

Technical Summary

The New Zealand Earthquake Forecast Testing Centre is one of four existing regional testing centres around the globe that are under the umbrella of the Collaboratory for the Study of Earthquake Predictability (CSEP). These centres were established to undertake verifiable and transparent tests of proposed earthquake forecasting models, so that the whole science community can have confidence in the results. The Centre is designed to test regional earthquake likelihood models for shallow earthquakes (h < 40 km) occurring within the New Zealand test region primarily on three timescales (5-years, 3-months and 1-day). In order to reliably measure forecast performance, an extended prospective testing period of five or more years is required for all classes of models. The tests and software used are compatible with other CSEP regional testing centres.

In this study, six new models have been developed, implemented and/or installed for testing in the centre, including two 1-day models for clustering of aftershocks, three new 3-month models – two being elaborations of the Every Earthquake a Precursor According to Scale (EEPAS) model and the other a double-branching process model proposed by Italian researchers – and one new 5-year model – the 2010 revision (NZNSH2010) of the New Zealand national seismic hazard model.

The new and previously existing models have been tested retrospectively using a new presentation of the N-test comparing the expected and actual number of earthquakes, and the recently proposed T-test for estimating the information gain of one model over another. These efficient and straightforward tests give a much clearer indication of possible implementation errors than the standard CSEP test outputs have provided in previous studies. This has allowed for problems with the implementation of several of the models to be identified now rather than after years of computer-intensive testing. Tests of the 5-year models over 25 years show that the new NZNSH2010 model is significantly more informative than the old NZNSHM model over this period, although it slightly over-estimates the number of earthquakes and appears not to have been implemented over all bins in the test region.

Tests of the 3-month models over a two-year period have shown that one of the new EEPAS models is not correctly implemented in the testing centre, but the other models are performing approximately as expected. However the precision of the estimates of the expected number of earthquakes needs to be checked for three of the models previously installed.

Tests of the short-term (1-day) models over six months have revealed problems with a new implementation of the Short-Term Earthquake Probability (STEP) model in the JAVA programming language. It underestimates the number of earthquakes in the test region. The tests show that neither the original STEP model nor the new JAVA version of this model are properly implemented for the testing centre.

The results of retrospective testing have thus been highly instructive in revealing teething problems with implementation of both new and existing models. The incidence of model implementation errors revealed in this study is similar to that found in retrospective testing of models submitted to other regional testing centres.