Release Changelog

v1.5.0

Built 6 July 2023

Breaking

  • RiskScape now requires a Java Runtime Environment of at least version 17. If RiskScape fails to run with an error message like:

    Exception in thread "main" java.lang.UnsupportedClassVersionError: nz/org/riskscape/engine/cli/Main has been compiled by a more recent version of the Java Runtime

    Then you will need to upgrade your Java Runtime. Refer to the Java documentation for more details on how to upgrade Java. (GL634)

Enhancements

  • The union step can now retain the CRS information associated with the input geometry, providing that all the inputs to the union step are in the same CRS. This means that more geometry-based operations, such as to_coverage() can now be performed on the output of the union step. (GL824)

  • A new aal_trapz() aggregation function has been added to the Beta plugin. This new function simplifies the process of calculating the Average Annual Loss (AAL) for Hazard-based probabilistic models. (GL993)

  • Added a new file output: QGIS project file. Model runs now produce a QGIS project file named project.qgs which can be opened in QGIS and contains all output data. This feature currently supports file system output files as GeoJSON, KML, shapefile, and GeoPackage. The new project.qgs will automatically be generated whenever the Beta plugin is enabled, although it can be disabled with the output.qgis-project configuration option if required.

Fixes

  • Various improvements to error handling, including:

    • An untidy ResultOrProblems.logIfWarningsIgnored() stack-trace was sometimes displayed when saving results, particularly for GeoPackage format files.

    • An UnsupportedOperationException when saving SQL Date attributes to a GeoPackage file has now been fixed.

    • More contextual information is now displayed if an error occurs writing to a GeoJSON file.

    • Previously, RiskScape could exit unexpectedly if a group pipeline step tried to use non-unique attribute names. Now, the error message is correctly displayed.

    • Occasionally, the user-friendly error message would not be displayed when an exception occurred, and instead a message like Problem(ERROR, STOPPED) would be shown. The full error message is now correctly displayed. (GL964)

  • An issue was resolved when models use raster data as relational input data for a model, for example, using a population grid as the exposure-layer. Some raster files may have a large extent and a small grid resolution (i.e. a large number of total pixels), even if the size of the raster file itself is not particularly large. If the total number of pixels was over 2.1 billion, RiskScape would incorrectly return no relational input data for the file. Now, the data will be read, however, the performance will still be very slow due to the sheer number of pixels. (GL964)

  • Previously, RiskScape would write an empty shapefile in a manner that would cause an exception when RiskScape tried to read the shapefile again later. This has been resolved so that RiskScape writes empty shapefiles in a format that it can successfully read again. (GL990)

v1.4.0

Built 14 April 2023

Breaking

  • Support for the legacy template = table models has now been dropped. Table models were useful if the exposure and hazard data had already been combined into a single data source. However, you can achieve the same functionality through a pipeline model - follow the How to use table models guide for a worked example of how to do this. (GL813)

  • The next RiskScape release (1.5.0) will require Java 17 to run. Users are advised to upgrade their Java environment now, if necessary, to ensure the next RiskScape version will run. A warning has been added to RiskScape to warn users if they are using a Java version that will not run RiskScape 1.5.0. (GL903)

  • The if_null(test, else) function will now report an error if the else argument is a nullable type. The previous behaviour when the else argument was null was that the value returned would always be null (i.e. Nothing or None), even if the test value was non-null. (GL940)

  • The bucket_range() function now names the lowest possible range as range_<_0, whereas previously it would have been range_-_0. Some models may now produce slightly different bucket_range outputs, as the ranges will now automatically include the highest (i.e. range_X_+) and lowest (i.e. range_<_Y) values possible, by default. (GL942)

  • Structs could be defined without a comma separating the struct attributes. E.g { 'foo' 'bar' } and { 'foo' as foo 'bar' as bar } should have a comma separating the foo and bar attributes but RiskScape was treating these expressions as valid.

    Now RiskScape will treat these expressions as invalid and produce an error that the comma is missing. (GL925)

Enhancements

  • A new, more efficient implementation is now used when cutting geometry into smaller pieces. Models that involve a lot of cutting operations, including the segment(), segment_by_grid(), and combine_coverages() functions, should now run faster. Existing models may emit minutely different geometry using this new method, but only at the smallest of resolutions (coordinates seem consistent within 1 x 10^-15 of precision when cutting WGS-84 polygons by a 1000² metric grid). (GL433)

  • The GeoTools and JTS libraries used by RiskScape for geometry processing have been updated. GeoTools has been updated from v26.0 to v28.2, and JTS has been updated from v1.18.2 to v1.19.0.

    The GeoTools update includes security fixes for OGC Filter SQL Injection Vulnerabilities which could have affected postgis bookmarks in RiskScape. Refer to this community post for more detail about how it affects RiskScape. (GL937)

  • When sharing models, you can now constrain what values users can enter for pipeline model parameters. For example, you can require that the user always enters a numeric value within a certain range. For more details, refer to Parameter properties. (GL441)

  • The Beta plugin is no longer required in order to use the union pipeline step. For more on the union step, refer to Combining exposure-layers.

  • New string utility functions ends_with and str_find have been added. These can be useful for filtering bookmark input data. (GL941)

  • The bucket_range() function will now automatically add the lower-most and upper-most bounds to the given range argument automatically. For example, if the given range argument were [ 1, 1000, 100000 ], it will now behave as if [ minint(), 1, 1000, 100000, maxint() ] had been specified. This ensures that any values that might fall outside the specified range will not be ignored by default. You can avoid this new behaviour and revert to the old behaviour by specifying a new options argument, e.g. bucket_range(pick: loss, select: { count(*) }, range: [ 1, 1000, 100000 ], options: { add_bounds: false }). (GL942)

  • A new tile-cache-percent configuration setting has been added to the [global] section of the Settings file. This lets you configure the size of the tile cache used when sampling raster data, such as a GeoTIFF, as a percentage of the total Java memory. Using a larger cache can improve the performance of riskscape model run, as it means the raster pixels need to be swapped in and out of memory less frequently. The default tile cache size is now 5% of the total Java memory. (GL949)

Fixes

  • When a raster bookmark used map-value or the map_coverage() function was used in a pipeline, sampling the coverage could sometimes produce inconsistent results. This issue has now been resolved. If you have used map-value in a bookmark or map_coverage() in a pipeline, we recommend upgrading RiskScape and re-running your model(s). (GL958)

  • Previously, running RiskScape on certain systems may have resulted in an exception RuntimeException: Waiting thread received a null tile. This issue should now be resolved. (GL830)

  • The riskscape model list command now displays any warnings that a model might have. Previously warnings would only be displayed when the model was actually run. (GL910)

  • The CLI wizard now lets you select any attribute for the group-by parameter. This includes geometry and struct attributes. For example, you can now aggregate by the entire area-layer struct, which will report the results broken down by region as a geospatial output (previously, only aggregated CSV outputs were supported in the CLI wizard). (GL884)

  • The Hazard-based probabilistic models documentation has been updated to exclude duplicated zero loss values from the AAL calculation. When the continuous fitter is used in the fit_curve function, it will now result in a model run error if the input data contains duplicate values. (GL911)

  • When RiskScape wrote GeoJSON output files it would save the first geometry as the GeoJSON geometry, as well as a text version of the same geometry. This would increase the size of the file needlessly. The text version of the geometry is no longer included. (GL948)

  • RiskScape would throw a null pointer exception for expressions that did an equality operation with a integer to float conversion on a null input. E.g null_of('floating') = 4 would fail. This has been resolved now. (GL943)

v1.3.0

Built 7 December 2022

Breaking

  • If you have manually specified a crs-name for a ESRI Grid (.asc file) bookmark, then the default crs-longitude-first setting has changed. Previously, the default was false, which meant the axis order defined in the EPSG CRS was used (typically lat,long). The default value is now true, which means RiskScape will read the data as long,lat coordinates. This only applies when a crs-name is set explicitly for a .asc bookmark. In general, you should not need to set the crs-name for a .asc bookmark unless the .prj file is missing or unreadable. (GL807)

  • The --parameters option has been removed from the riskscape pipeline evaluate command. Previously this allowed you to override low-level parameters for a given pipeline step. If this change affects you, we now recommend that you use Customizing your own model parameters instead. The riskscape pipeline evaluate and riskscape wizard commands now both save a pipeline.txt file in the output directory. (GL821)

Enhancements

  • The to_coverage() function now accepts raster data as an argument-type. This means that advanced users can now write a single pipeline that samples geospatial data from either a vector source (i.e. shapefile) or a raster source (i.e. a GeoTIFF). Previously, this would require writing two separate pipelines, one for each source. (GL809).

  • The performance of ‘all-intersections’ and ‘closest’ spatial sampling has been improved when dealing with large or complex polygons in the exposure-layer and a GeoTIFF or .asc hazard-layer. Previously, the entire polygon would be cut by the grid resolution of the hazard-layer, which could be very computationally expensive. Now, cutting will be skipped for parts of the polygon that overlap a ‘no-data’ cell in the hazard-layer. (GL820)

  • A new union step has been added to the beta plugin for advanced pipeline models. This means you can now combine multiple different exposure-layers (e.g. water pipes and pump stations) into a single model. For more details, see Combining exposure-layers. (GL436)

  • When the data types in your exposure-layer do not match the types that your Python function expects, the error message will now provide more detail on exactly which attributes are missing or are the wrong data type. (GL823)

  • Support has been added to the beta plugin to dynamically load relational data from multiple sources (GL837). This allows a pipeline to load CSV input that contains the locations of other CSV files to read. Advanced users can now do this in a pipeline using the following approach:

    input('list-of-csv-files.csv', name: 'csv')
-> select({*, to_list(bookmark('template-bookmark,
                               {location: csv.filename},
                               type: 'relation(struct(foo: floating))
                      ) as csv_data)
-> unnest(csv_data)

  • A new losses_by_period() function has been added to the beta plugin. This function can be used in a probabilistic pipeline to calculate loss metrics such as Probable Maximum Loss, Occurrence Exceedance Probability, and Aggregated Exceedance Probability. (GL864)

  • Support for reading and writing GeoPackage files is now enabled by default. Previously it required that the beta plugin was enabled. (GL853)

Fixes

  • Previously, using the exposed_ratio attribute or the scale-losses option in wizard models could occasionally calculate the incorrect ratio when sampling a polygon exposure-layer against a GeoTIFF or .asc hazard file. This meant that scaling the loss could calculate the incorrect result. If the input geometry aligned exactly with the hazard grid, the sample() operation could potentially return a line-string rather than a polygon geometry match. The exposed_ratio calculation would then measure() the sampled geometry, which returns the length of the line-string, rather than the area, as expected. This issue has now been fixed. Note that a work-around on previous versions of RiskScape would be to take the pipeline code and change the measure() function calls to measure_area() instead. This problem could also manifest itself as a ClassCastException when the analysis.save-raw-resultsparameter was set for the wizard model. The measure() function will also now return an error if given a collection of different geometry types (i.e. polygons and line-strings). (GL817)

  • Previously the crs-name was silently ignored for GeoTIFF bookmarks. Now the setting will take effect. Note that you should not normally need to specify crs-name explicitly for a GeoTIFF bookmark. (GL807)

  • Previously, if warnings were present for a model parameter, these warnings would either be silently ignored by the riskscape model run command, or a stack-trace would be logged along with the message Non-error problems are present and have not been consumed.... Now, RiskScape will display any warnings associated with the model when it is run. (GL763)

  • Performance and memory usage have been improved for aggregation pipeline steps. In particular, you may notice an improvement if your model pipeline contains many group steps (e.g. probabilistic models), or the model aggregation produces a large number of rows (e.g. aggregating by the original exposure). (GL808)

  • Previously, RiskScape could sometimes exit unexpectedly whilst running a model. When this problem occurred, the message ‘Deadlock detected!’ would be logged. This issue has now been fixed. (GL816)

  • Support has been improved when a file is specified for the --progress-indicator option for the riskscape model run command. Previously, if an intermittent file error occurred, for example the progress file was temporarily locked, then RiskScape would stop writing statistics completely for the duration of the model run. Now, RiskScape will continue to retry writing the progress statistics to file, even if file errors are encountered. (GL806)

  • Some Workarounds for ArcGIS users having shapefile projection issues have now been documented. (GL855)

v1.2.0

Built 29 July 2022

Breaking

  • Duplicate bookmark settings now show an error, rather than being silently ignored. Previously, the last defined setting was used, and any earlier values were ignored. This change brings bookmarks in to line line with other project configuration, such as model definitions where duplicates are not expected. You can fix this error by simply removing any surplus settings and leaving only the one your project needs (usually this is the last one). (GL769)

  • Previously, when RiskScape read an HDF5 dataset that contained a simple type (e.g. integer or floating values), it produced a relation containing a single attributed called value. Now, the name of the attribute will match the name of the dataset instead. For example, reading the /gmf_data/gmv_0 dataset will now produce a relation containing a gmv_0 attribute, instead of a value attribute. (GL718)

  • The --no-progress-indicator CLI option has been removed for the riskscape model run, riskscape model batch, and riskscape pipeline eval commands. Instead, this has been replaced with a new --progress-indicator=none option.

Enhancements

  • Loading plugins in the settings file has been simplified. E.g you can now load the beta plugin with a settings.ini containing: load-plugin = beta. (GL757)

  • The RiskScape expression language now supports floating point numbers in scientific notation, e.g 5.27e-10. (GL754)

  • RiskScape now supports GeoPackage as an output (e.g --output results.gpkg). This will save all result layers to the same output GeoPackage file, rather than separate files when using the --format geopackage option. (GL758)

  • Manifest information is now saved when writing output to GeoPackage and PostGIS outputs. (GL759)

  • RiskScape has been updated to support the v1.12.2 HDF5 library. Linux users may need to change their settings.ini‘s [hdf5] lib_path to point to /opt/hdf5-1.12.2/lib/libhdf5_java.so (GL760)

  • HDF5 bookmarks can now read multiple datasets at once and combine them into a single relation. This is only supported when the datasets all contain the exact same number of elements. (GL718)

  • Coverage bookmarks now accept a map-value parameter that lets you manipulate the value that gets returned when the coverage is sampled. For example, you could convert a hazard intensity measure from centimetres to metres by using map-value = value / 100. A new map_coverage() function has also been added, which lets you do the same thing directly from pipeline code. (GL782)

  • The round() function now lets you specify the number of decimal places to round to. (GL452)

  • New norm_ppf and lognorm_ppf functions have been added to return the Percent Point Function (inverse Cumulative Distribution Function) from a normal or log-normal curve. (GL744)

  • For USGS ShakeMaps, you can now specify a specific dataset to read. Previously RiskScape would only read the PGA data. Refer to USGS ShakeMaps for more details. (GL744)

  • Running a model with unknown parameters will now produce a warning. Unknown parameters are likely to be caused by mistyping the parameter name. (GL768)

  • The progress statistics, displayed on the terminal when a model runs, have been improved. They now include a percentage complete metric for some pipeline steps. For long-running models, the percentage complete for the exposure-layer (e.g. exposures_input.complete) can often be used as a guide to the overall progress of the model. Refer to Progress statistics for more details. (GL792)

  • Commands that support progress statistics now support a --progress-indicator CLI option. Instead of displaying the progress statistics on the CLI, you can specify a file that progress updates will be written to, e.g. --progress-indicator=output/progress.txt. This may be useful when running RiskScape as part of a script. (GL792)

  • More improvements have been made to the RiskScape documentation. Of note are:

Fixes

  • When RiskScape accessed files over HTTP, some websites could mistake RiskScape for a bot and reject the HTTP request with a 403 error. RiskScape now sets the ‘User-Agent’ in HTTP requests to avoid this problem (GL751).

  • Various issues with using WFS input data have now been resolved, including:

    • Connection errors now show the cause of the connection error. Previously these were obscured by an unhelpful Failed to create data store error. (GL461)

    • Previously RiskScape was only loading the first 100,000 features/records for a WFS layer and was ignoring any subsequent data. (GL753)

    • If the WFS server had a configuration setting (e.g. maxRecordCount) that limited how many features are returned in each WFS request, then RiskScape would only load that many features. The number of features per WFS request can now also be lowered using the wfs-page-size bookmark parameter. (GL761)

    • Trying to use a WFS layer that contained a large number of records (e.g. 500,000) could result in RiskScape exiting with an error, generally due to a connection timeout. (GL573)

    • Depending on the WFS server, RiskScape could exit with an exception trying to read the WFS data. (GL573) When the --show-stack-trace CLI option was used, the following error would be displayed in this case:

      java.lang.ClassCastException: java.util.HashMap cannot be cast to org.opengis.feature.simple.SimpleFeature

    • Some examples on how to use WFS input data have now been added to the RiskScape documentation. Refer to the new WFS data page.

  • Previously RiskScape could exit with an exception when the --show-project-errors CLI option was used, and the project contained a bookmark with an error. This has now been fixed. (GL764)

  • When input data was read in GeoJSON format, floating-point values could sometimes be truncated and turned into whole numbers. For example, the value 20.5 would be turned into 20.0. This would only occur if the same attribute in the input data contained a mixture of integer (i.e. whole number) and floating-point values. When this mixture of integer and floating-point values were present in the input data, RiskScape could also exit with an exception:

    IllegalStateException: Found conflicting types Double and Long for property XYZ

    These issues have now been fixed. (GL788)

  • For advanced users that write their own pipelines, RiskScape could sometimes complain about a pipeline cycle error when there was no cycle. The pipeline cycle checking has now been improved. (GL772)

  • Various minor improvements have been made to error handling, for example how errors and exceptions are displayed, particularly when the --show-stacktrace CLI option is used. (GL803)

v1.1.0

Built 5 May 2022

Enhancements

  • An optional Beta plugin has been added to make new/experimental features available. The beta plugin is not enabled by default, and any beta plugin features may still change significantly in future releases of RiskScape. (GL716)

  • Preliminary support has been added to the beta plugin for:

    • Saving results to a PostGIS database. (GL705)

    • Reading and writing vector data in the GeoPackage (.gpkg) file format (GL706, GL728)

  • New functions have been added to the beta plugin, including:

    • list_to_tuple, which makes it easier to save the contents of a list to CSV or shapefile. Each item in the list will be written as a separate column or attribute. (GL719)

    • switch, which is useful when you have a range of possible input values that you want to map to a smaller subset of values. E.g. you want to map any of the value codes 0, 12, 44, 22 to simply 'Timber'.

  • Using raster data as your exposure-layer now results in polygon geometries that cover the extent of each raster point. Previously the geometry produced was a point rather than a polygon. (GL620)

  • Type expressions can now use hyphens in struct attribute names as well as quoted identifiers, e.g struct(value_0-100: integer, "Total Cost": floating) (GL686)

  • More tutorials have been added to the RiskScape documentation, including an ‘Intermediate Concepts’ section that covers:

    • How the various RiskScape spatial sampling and consequence analysis techniques work. (GL649)

    • Improved CPython setup instructions and an example damage state function.

    • How the riskscape model batch lets you run a model repeatedly by varying the hazard-layer. (GL168)

    • How the various RiskScape geoprocessing features operate. (GL722)

  • A new riskscape bookmark evaluate command has been added. This lets you visualize the input data, typically in shapefile form, after it has been manipulated by any RiskScape bookmark settings. This is useful to check the CRS axis-order is correct when you need to construct geometry manually, such as from CSV input data. (GL608)

Fixes

  • Table rendering on Windows has been improved for info commands. Previously, commands such as riskscape format info shapefile, could result in poor rendering if the content being displayed contained line-breaks. (GL703)

  • The default validate-geometry setting is error, which means RiskScape detects and automatically fixes any invalid geometry in the input data. However, the default validate-geometry project setting was not being applied correctly when geometry was reprojected. This meant that geometry could become invalid as a result of reprojection and RiskScape would not detect or fix it (unless validate-geometry = error was explicitly set in the project.ini file). This issue has now been fixed. (GL724)

  • When setting up CPython support in RiskScape, you no longer need double backslashes in the python3-bin file path in settings.ini. For example, you can now use C:\Users\Ronnie\Anaconda3\python.exe instead of C:\\Users\\Ronnie\\Anaconda3\\python.exe. Existing settings.ini files that contain double backslashes for the Python path should still continue to work fine. Previously, exceptions or other errors could occur when single backslashes were used, but this has now been fixed. (GL-735)

  • Setting the output-base-location in the project would result in an extra output directory appearing in the output path. This has been fixed now. (GL743)

  • Running RiskScape without a project.ini or home directory would result in unusual errors if a relative file was accessed, e.g riskscape bookmark info FILE.csv. This has now been fixed. (GL710)

  • Spatial sampling has been improved for the sample_one() function, which is used by CLOSEST sampling in the wizard. For complex line and polygon geometry, such as shapes that look like a ‘C’ or multi-polygons, the geometric centroid of the shape can fall outside the shape itself. Previously, RiskScape would always attempt to use the geometric centroid first when sampling the hazard value, even if this centroid fell outside the shape itself. Now sample_one will always ensure that the sampled value falls inside the input geometry. (GL737)

  • When CPython functions were enabled on your system, sometimes the riskscape model run command could exit with a java.lang.NullPointerException error. This only occurred if the shapefile input data for the exposure-layer contained NULL values. This issue has now been fixed. (GL-739)

  • When all-intersections sampling is used, the wizard will now produce a single value for the exposed_ratio attribute. Previously exposed_ratio was a List type, which made it harder to use. (GL741)

  • Model run --parameter values may now contain white space around the equals sign. E.g --parameter "input-exposures.layer = Roads_SE_Upolu". (GL741)

v1.0.0

Built 4 February 2022

Version 1.0.0 is the first official public release of RiskScape.

If you have been using older, evaluation versions of RiskScape, you can view a record of the changes in functionality between releases here.