.. _changelog: # Release Changelog ## v1.5.0 *Built 6 July 2023* ### Breaking - RiskScape now requires a Java Runtime Environment of at least version 17. If RiskScape fails to run with an error message like: `Exception in thread "main" java.lang.UnsupportedClassVersionError: nz/org/riskscape/engine/cli/Main has been compiled by a more recent version of the Java Runtime` Then you will need to upgrade your Java Runtime. Refer to the [Java documentation](https://www.oracle.com/java/technologies/downloads/) for more details on how to upgrade Java. (GL634) ### Enhancements - The `union` step can now retain the CRS information associated with the input geometry, providing that all the inputs to the `union` step are in the same CRS. This means that more geometry-based operations, such as `to_coverage()` can now be performed on the output of the `union` step. (GL824) - A new `aal_trapz()` aggregation function has been added to the :ref:`beta-plugin`. This new function simplifies the process of calculating the Average Annual Loss (AAL) for :ref:`hazard_based_probabilistic`. (GL993) - Added a new file output: QGIS project file. Model runs now produce a QGIS project file named `project.qgs` which can be opened in QGIS and contains all output data. This feature currently supports file system output files as GeoJSON, KML, shapefile, and GeoPackage. The new `project.qgs` will automatically be generated whenever the :ref:`beta-plugin` is enabled, although it can be disabled with the `output.qgis-project` configuration option if required. ### Fixes - Various improvements to error handling, including: - An untidy `ResultOrProblems.logIfWarningsIgnored()` stack-trace was sometimes displayed when saving results, particularly for GeoPackage format files. - An `UnsupportedOperationException` when saving SQL Date attributes to a GeoPackage file has now been fixed. - More contextual information is now displayed if an error occurs writing to a GeoJSON file. - Previously, RiskScape could exit unexpectedly if a `group` pipeline step tried to use non-unique attribute names. Now, the error message is correctly displayed. - Occasionally, the user-friendly error message would not be displayed when an exception occurred, and instead a message like `Problem(ERROR, STOPPED)` would be shown. The full error message is now correctly displayed. (GL964) - An issue was resolved when models use raster data as relational input data for a model, for example, using a population grid as the exposure-layer. Some raster files may have a large extent and a small grid resolution (i.e. a large number of total pixels), even if the size of the raster file itself is not particularly large. If the total number of pixels was over 2.1 billion, RiskScape would incorrectly return no relational input data for the file. Now, the data will be read, however, the performance will still be very slow due to the sheer number of pixels. (GL964) - Previously, RiskScape would write an empty shapefile in a manner that would cause an exception when RiskScape tried to read the shapefile again later. This has been resolved so that RiskScape writes empty shapefiles in a format that it can successfully read again. (GL990) ## v1.4.0 *Built 14 April 2023* ### Breaking - Support for the legacy `template = table` models has now been dropped. Table models were useful if the exposure and hazard data had already been combined into a *single* data source. However, you can achieve the same functionality through a pipeline model - follow the :ref:`table_pipelines` guide for a worked example of how to do this. (GL813) - The next RiskScape release (1.5.0) will require Java 17 to run. Users are advised to upgrade their Java environment now, if necessary, to ensure the next RiskScape version will run. A warning has been added to RiskScape to warn users if they are using a Java version that will not run RiskScape 1.5.0. (GL903) - The `if_null(test, else)` function will now report an error if the `else` argument is a nullable type. The previous behaviour when the `else` argument was null was that the value returned would *always* be `null` (i.e. `Nothing` or `None`), even if the `test` value was non-null. (GL940) - The `bucket_range()` function now names the lowest possible range as `range_<_0`, whereas previously it would have been `range_-_0`. Some models may now produce slightly different `bucket_range` outputs, as the ranges will now automatically include the highest (i.e. `range_X_+`) and lowest (i.e. `range_<_Y`) values possible, by default. (GL942) - Structs could be defined without a comma separating the struct attributes. E.g `{ 'foo' 'bar' }` and `{ 'foo' as foo 'bar' as bar }` should have a comma separating the `foo` and `bar` attributes but RiskScape was treating these expressions as valid. Now RiskScape will treat these expressions as invalid and produce an error that the comma is missing. (GL925) ### Enhancements - A new, more efficient implementation is now used when cutting geometry into smaller pieces. Models that involve a lot of cutting operations, including the `segment()`, `segment_by_grid()`, and `combine_coverages()` functions, should now run faster. Existing models may emit minutely different geometry using this new method, but only at the smallest of resolutions (coordinates seem consistent within 1 x 10^-15 of precision when cutting WGS-84 polygons by a 1000² metric grid). (GL433) - The [GeoTools](https://www.geotools.org/) and [JTS](https://www.geotools.org/) libraries used by RiskScape for geometry processing have been updated. GeoTools has been updated from v26.0 to v28.2, and JTS has been updated from v1.18.2 to v1.19.0. The GeoTools update includes security fixes for [OGC Filter SQL Injection Vulnerabilities](https://github.com/geotools/geotools/security/advisories/GHSA-99c3-qc2q-p94m) which could have affected `postgis` bookmarks in RiskScape. Refer to this [community post](https://community.riskscape.org.nz/t/geotools-vulnerability-may-affect-riskscape-users-who-use-postgis-bookmarks/118) for more detail about how it affects RiskScape. (GL937) - When sharing models, you can now constrain what values users can enter for pipeline model parameters. For example, you can require that the user always enters a numeric value within a certain range. For more details, refer to :ref:`parameter_properties`. (GL441) - The :ref:`beta-plugin` is no longer required in order to use the `union` pipeline step. For more on the `union` step, refer to :ref:`union_step`. - New string utility functions `ends_with` and `str_find` have been added. These can be useful for filtering bookmark input data. (GL941) - The `bucket_range()` function will now automatically add the lower-most and upper-most bounds to the given `range` argument automatically. For example, if the given `range` argument were `[ 1, 1000, 100000 ]`, it will now behave as if `[ minint(), 1, 1000, 100000, maxint() ]` had been specified. This ensures that any values that might fall outside the specified range will *not* be ignored by default. You can avoid this new behaviour and revert to the old behaviour by specifying a new `options` argument, e.g. `bucket_range(pick: loss, select: { count(*) }, range: [ 1, 1000, 100000 ], options: { add_bounds: false })`. (GL942) - A new `tile-cache-percent` configuration setting has been added to the `[global]` section of the :ref:`settings_ini`. This lets you configure the size of the tile cache used when sampling raster data, such as a GeoTIFF, as a percentage of the total Java memory. Using a larger cache can improve the performance of `riskscape model run`, as it means the raster pixels need to be swapped in and out of memory less frequently. The default tile cache size is now 5% of the total Java memory. (GL949) ### Fixes - When a raster bookmark used `map-value` or the `map_coverage()` function was used in a pipeline, sampling the coverage could sometimes produce inconsistent results. This issue has now been resolved. If you have used `map-value` in a bookmark or `map_coverage()` in a pipeline, we recommend upgrading RiskScape and re-running your model(s). (GL958) - Previously, running RiskScape on certain systems may have resulted in an exception `RuntimeException: Waiting thread received a null tile.` This issue should now be resolved. (GL830) - The `riskscape model list` command now displays any warnings that a model might have. Previously warnings would only be displayed when the model was actually run. (GL910) - The CLI wizard now lets you select *any* attribute for the `group-by` parameter. This includes geometry and struct attributes. For example, you can now aggregate by the entire area-layer struct, which will report the results broken down by region as a geospatial output (previously, only aggregated CSV outputs were supported in the CLI wizard). (GL884) - The :ref:`hazard_based_probabilistic` documentation has been updated to exclude duplicated zero loss values from the AAL calculation. When the `continuous` fitter is used in the `fit_curve` function, it will now result in a model run error if the input data contains duplicate values. (GL911) - When RiskScape wrote GeoJSON output files it would save the first geometry as the GeoJSON geometry, as well as a text version of the same geometry. This would increase the size of the file needlessly. The text version of the geometry is no longer included. (GL948) - RiskScape would throw a null pointer exception for expressions that did an equality operation with a integer to float conversion on a null input. E.g `null_of('floating') = 4` would fail. This has been resolved now. (GL943) ## v1.3.0 *Built 7 December 2022* ### Breaking - If you have manually specified a `crs-name` for a ESRI Grid (`.asc` file) bookmark, then the default `crs-longitude-first` setting has changed. Previously, the default was `false`, which meant the axis order defined in the EPSG CRS was used (typically `lat,long`). The default value is now `true`, which means RiskScape will read the data as `long,lat` coordinates. This *only* applies when a `crs-name` is set explicitly for a `.asc` bookmark. In general, you should not need to set the `crs-name` for a `.asc` bookmark unless the `.prj` file is missing or unreadable. (GL807) - The `--parameters` option has been removed from the `riskscape pipeline evaluate` command. Previously this allowed you to override low-level parameters for a given pipeline step. If this change affects you, we now recommend that you use :ref:`parameterized_pipelines` instead. The `riskscape pipeline evaluate` and `riskscape wizard` commands now both save a `pipeline.txt` file in the output directory. (GL821) ### Enhancements - The `to_coverage()` function now accepts raster data as an argument-type. This means that advanced users can now write a single pipeline that samples geospatial data from *either* a vector source (i.e. shapefile) or a raster source (i.e. a GeoTIFF). Previously, this would require writing two separate pipelines, one for each source. (GL809). - The performance of 'all-intersections' and 'closest' spatial sampling has been improved when dealing with large or complex polygons in the exposure-layer and a GeoTIFF or `.asc` hazard-layer. Previously, the entire polygon would be cut by the grid resolution of the hazard-layer, which could be very computationally expensive. Now, cutting will be skipped for parts of the polygon that overlap a 'no-data' cell in the hazard-layer. (GL820) - A new `union` step has been added to the beta plugin for advanced pipeline models. This means you can now combine multiple different exposure-layers (e.g. water pipes *and* pump stations) into a single model. For more details, see :ref:`union_step`. (GL436) - When the data types in your exposure-layer do not match the types that your Python function expects, the error message will now provide more detail on exactly which attributes are missing or are the wrong data type. (GL823) - Support has been added to the beta plugin to dynamically load relational data from multiple sources (GL837). This allows a pipeline to load CSV input that contains the locations of other CSV files to read. Advanced users can now do this in a pipeline using the following approach: ``` input('list-of-csv-files.csv', name: 'csv') -> select({*, to_list(bookmark('template-bookmark, {location: csv.filename}, type: 'relation(struct(foo: floating)) ) as csv_data) -> unnest(csv_data) ``` - A new `losses_by_period()` function has been added to the beta plugin. This function can be used in a probabilistic pipeline to calculate loss metrics such as Probable Maximum Loss, Occurrence Exceedance Probability, and Aggregated Exceedance Probability. (GL864) - Support for reading and writing GeoPackage files is now enabled by default. Previously it required that the beta plugin was enabled. (GL853) ### Fixes - Previously, using the `exposed_ratio` attribute or the `scale-losses` option in wizard models could occasionally calculate the incorrect ratio when sampling a polygon exposure-layer against a GeoTIFF or `.asc` hazard file. This meant that scaling the loss could calculate the incorrect result. If the input geometry aligned exactly with the hazard grid, the `sample()` operation could potentially return a *line-string* rather than a polygon geometry match. The `exposed_ratio` calculation would then `measure()` the sampled geometry, which returns the *length* of the line-string, rather than the area, as expected. This issue has now been fixed. Note that a work-around on previous versions of RiskScape would be to take the pipeline code and change the `measure()` function calls to `measure_area()` instead. This problem could also manifest itself as a `ClassCastException` when the `analysis.save-raw-results`\ parameter was set for the wizard model. The `measure()` function will also now return an error if given a collection of different geometry types (i.e. polygons and line-strings). (GL817) - Previously the `crs-name` was silently ignored for GeoTIFF bookmarks. Now the setting will take effect. Note that you should not normally need to specify `crs-name` explicitly for a GeoTIFF bookmark. (GL807) - Previously, if warnings were present for a model parameter, these warnings would either be silently ignored by the `riskscape model run` command, or a stack-trace would be logged along with the message `Non-error problems are present and have not been consumed...`. Now, RiskScape will display any warnings associated with the model when it is run. (GL763) - Performance and memory usage have been improved for aggregation pipeline steps. In particular, you may notice an improvement if your model pipeline contains many `group` steps (e.g. probabilistic models), or the model aggregation produces a large number of rows (e.g. aggregating by the original exposure). (GL808) - Previously, RiskScape could sometimes exit unexpectedly whilst running a model. When this problem occurred, the message 'Deadlock detected!' would be logged. This issue has now been fixed. (GL816) - Support has been improved when a file is specified for the `--progress-indicator` option for the `riskscape model run` command. Previously, if an intermittent file error occurred, for example the progress file was temporarily locked, then RiskScape would stop writing statistics completely for the duration of the model run. Now, RiskScape will continue to retry writing the progress statistics to file, even if file errors are encountered. (GL806) - Some :ref:`arcgis_workarounds` users having shapefile projection issues have now been documented. (GL855) ## v1.2.0 *Built 29 July 2022* ### Breaking - Duplicate bookmark settings now show an error, rather than being silently ignored. Previously, the last defined setting was used, and any earlier values were ignored. This change brings bookmarks in to line line with other project configuration, such as model definitions where duplicates are not expected. You can fix this error by simply removing any surplus settings and leaving only the one your project needs (usually this is the last one). (GL769) - Previously, when RiskScape read an HDF5 dataset that contained a simple type (e.g. integer or floating values), it produced a relation containing a single attributed called `value`. Now, the name of the attribute will match the name of the dataset instead. For example, reading the `/gmf_data/gmv_0` dataset will now produce a relation containing a `gmv_0` attribute, instead of a `value` attribute. (GL718) - The `--no-progress-indicator` CLI option has been removed for the `riskscape model run`, `riskscape model batch`, and `riskscape pipeline eval` commands. Instead, this has been replaced with a new `--progress-indicator=none` option. ### Enhancements - Loading plugins in the settings file has been simplified. E.g you can now load the beta plugin with a `settings.ini` containing: `load-plugin = beta`. (GL757) - The RiskScape expression language now supports floating point numbers in scientific notation, e.g `5.27e-10`. (GL754) - RiskScape now supports GeoPackage as an output (e.g `--output results.gpkg`). This will save all result layers to the same output GeoPackage file, rather than separate files when using the `--format geopackage` option. (GL758) - Manifest information is now saved when writing output to GeoPackage and PostGIS outputs. (GL759) - RiskScape has been updated to support the v1.12.2 [HDF5 library](https://www.hdfgroup.org/solutions/hdf5/). Linux users may need to change their `settings.ini`'s `[hdf5] lib_path` to point to `/opt/hdf5-1.12.2/lib/libhdf5_java.so` (GL760) - HDF5 bookmarks can now read multiple datasets at once and combine them into a single relation. This is only supported when the datasets all contain the exact same number of elements. (GL718) - Coverage bookmarks now accept a `map-value` parameter that lets you manipulate the *value* that gets returned when the coverage is sampled. For example, you could convert a hazard intensity measure from centimetres to metres by using `map-value = value / 100`. A new `map_coverage()` function has also been added, which lets you do the same thing directly from pipeline code. (GL782) - The `round()` function now lets you specify the number of decimal places to round to. (GL452) - New `norm_ppf` and `lognorm_ppf` functions have been added to return the Percent Point Function (inverse Cumulative Distribution Function) from a normal or log-normal curve. (GL744) - For USGS ShakeMaps, you can now specify a specific dataset to read. Previously RiskScape would only read the PGA data. Refer to :ref:`shakemaps` for more details. (GL744) - Running a model with unknown parameters will now produce a warning. Unknown parameters are likely to be caused by mistyping the parameter name. (GL768) - The progress statistics, displayed on the terminal when a model runs, have been improved. They now include a percentage complete metric for some pipeline steps. For long-running models, the percentage complete for the exposure-layer (e.g. `exposures_input.complete`) can often be used as a guide to the overall progress of the model. Refer to :ref:`progress_monitor` for more details. (GL792) - Commands that support progress statistics now support a `--progress-indicator` CLI option. Instead of displaying the progress statistics on the CLI, you can specify a file that progress updates will be written to, e.g. `--progress-indicator=output/progress.txt`. This may be useful when running RiskScape as part of a script. (GL792) - More improvements have been made to the RiskScape documentation. Of note are: - :ref:`bookmark_examples` for many different bookmark data sources. - Some :ref:`performance tips` for models that run slow. ### Fixes - When RiskScape accessed files over HTTP, some websites could mistake RiskScape for a bot and reject the HTTP request with a 403 error. RiskScape now sets the 'User-Agent' in HTTP requests to avoid this problem (GL751). - Various issues with using WFS input data have now been resolved, including: - Connection errors now show the cause of the connection error. Previously these were obscured by an unhelpful `Failed to create data store` error. (GL461) - Previously RiskScape was only loading the first 100,000 features/records for a WFS layer and was ignoring any subsequent data. (GL753) - If the WFS server had a configuration setting (e.g. `maxRecordCount`) that limited how many features are returned in each WFS request, then RiskScape would only load that many features. The number of features per WFS request can now also be lowered using the `wfs-page-size` bookmark parameter. (GL761) - Trying to use a WFS layer that contained a large number of records (e.g. 500,000) could result in RiskScape exiting with an error, generally due to a connection timeout. (GL573) - Depending on the WFS server, RiskScape could exit with an exception trying to read the WFS data. (GL573) When the `--show-stack-trace` CLI option was used, the following error would be displayed in this case: ``` java.lang.ClassCastException: java.util.HashMap cannot be cast to org.opengis.feature.simple.SimpleFeature ``` - Some examples on how to use WFS input data have now been added to the RiskScape documentation. Refer to the new :ref:`wfs` page. - Previously RiskScape could exit with an exception when the `--show-project-errors` CLI option was used, and the project contained a bookmark with an error. This has now been fixed. (GL764) - When input data was read in GeoJSON format, floating-point values could sometimes be truncated and turned into whole numbers. For example, the value `20.5` would be turned into `20.0`. This would *only* occur if the same attribute in the input data contained a mixture of integer (i.e. whole number) and floating-point values. When this mixture of integer and floating-point values were present in the input data, RiskScape could also exit with an exception: ```none IllegalStateException: Found conflicting types Double and Long for property XYZ ``` These issues have now been fixed. (GL788) - For advanced users that write their own pipelines, RiskScape could sometimes complain about a pipeline cycle error when there was no cycle. The pipeline cycle checking has now been improved. (GL772) - Various minor improvements have been made to error handling, for example how errors and exceptions are displayed, particularly when the `--show-stacktrace` CLI option is used. (GL803) ## v1.1.0 *Built 5 May 2022* ### Enhancements - An optional :ref:`beta-plugin` has been added to make new/experimental features available. The beta plugin is not enabled by default, and any beta plugin features may still change significantly in future releases of RiskScape. (GL716) - Preliminary support has been added to the beta plugin for: - Saving results to a :ref:`PostGIS ` database. (GL705) - Reading and writing vector data in the GeoPackage (`.gpkg`) file format (GL706, GL728) - New functions have been added to the beta plugin, including: - `list_to_tuple`, which makes it easier to save the contents of a list to CSV or shapefile. Each item in the list will be written as a separate column or attribute. (GL719) - `switch`, which is useful when you have a range of possible input values that you want to map to a smaller subset of values. E.g. you want to map any of the value codes `0, 12, 44, 22` to simply `'Timber'`. - Using raster data as your exposure-layer now results in polygon geometries that cover the extent of each raster point. Previously the geometry produced was a point rather than a polygon. (GL620) - Type expressions can now use hyphens in struct attribute names as well as quoted identifiers, e.g `struct(value_0-100: integer, "Total Cost": floating)` (GL686) - More tutorials have been added to the RiskScape documentation, including an 'Intermediate Concepts' section that covers: - How the various RiskScape spatial sampling and consequence analysis techniques work. (GL649) - Improved CPython setup instructions and an example damage state function. - How the `riskscape model batch` lets you run a model repeatedly by varying the hazard-layer. (GL168) - How the various RiskScape geoprocessing features operate. (GL722) - A new `riskscape bookmark evaluate` command has been added. This lets you visualize the input data, typically in shapefile form, *after* it has been manipulated by any RiskScape bookmark settings. This is useful to check the CRS axis-order is correct when you need to construct geometry manually, such as from CSV input data. (GL608) ### Fixes - Table rendering on Windows has been improved for `info` commands. Previously, commands such as `riskscape format info shapefile`, could result in poor rendering if the content being displayed contained line-breaks. (GL703) - The default `validate-geometry` setting is `error`, which means RiskScape detects and automatically fixes any invalid geometry in the input data. However, the default `validate-geometry` project setting was not being applied correctly when geometry was reprojected. This meant that geometry could become invalid as a result of reprojection and RiskScape would not detect or fix it (unless `validate-geometry = error` was explicitly set in the `project.ini` file). This issue has now been fixed. (GL724) - When setting up CPython support in RiskScape, you no longer need double backslashes in the `python3-bin` file path in `settings.ini`. For example, you can now use `C:\Users\Ronnie\Anaconda3\python.exe` instead of `C:\\Users\\Ronnie\\Anaconda3\\python.exe`. Existing `settings.ini` files that contain double backslashes for the Python path should still continue to work fine. Previously, exceptions or other errors could occur when single backslashes were used, but this has now been fixed. (GL-735) - Setting the `output-base-location` in the project would result in an extra `output` directory appearing in the output path. This has been fixed now. (GL743) - Running RiskScape without a `project.ini` or home directory would result in unusual errors if a relative file was accessed, e.g `riskscape bookmark info FILE.csv`. This has now been fixed. (GL710) - Spatial sampling has been improved for the `sample_one()` function, which is used by `CLOSEST` sampling in the wizard. For complex line and polygon geometry, such as shapes that look like a 'C' or multi-polygons, the geometric centroid of the shape can fall outside the shape itself. Previously, RiskScape would always attempt to use the geometric centroid first when sampling the hazard value, even if this centroid fell outside the shape itself. Now `sample_one` will always ensure that the sampled value falls inside the input geometry. (GL737) - When CPython functions were enabled on your system, sometimes the `riskscape model run` command could exit with a `java.lang.NullPointerException` error. This only occurred if the shapefile input data for the exposure-layer contained `NULL` values. This issue has now been fixed. (GL-739) - When all-intersections sampling is used, the wizard will now produce a single value for the `exposed_ratio` attribute. Previously `exposed_ratio` was a `List` type, which made it harder to use. (GL741) - Model run `--parameter` values may now contain white space around the equals sign. E.g `--parameter "input-exposures.layer = Roads_SE_Upolu"`. (GL741) ## v1.0.0 *Built 4 February 2022* Version 1.0.0 is the first official public release of RiskScape. If you have been using older, evaluation versions of RiskScape, you can view a record of the changes in functionality between releases :ref:`here `.