Commit 70f93e62 authored by Graeme Weatherill's avatar Graeme Weatherill
Browse files

Updates to installation and documentation, Fix of Vs30 bug in Point Workflow

parent 038c91d2
...@@ -143,3 +143,5 @@ dmypy.json ...@@ -143,3 +143,5 @@ dmypy.json
# OSX # OSX
.DS_Store .DS_Store
# Don't track site data files
site_data/
No preview for this file type
...@@ -12,25 +12,32 @@ License (document): Creative Commons CC-By-4.0 ...@@ -12,25 +12,32 @@ License (document): Creative Commons CC-By-4.0
--- ---
``` ```
The exposure to site tool is a short(ish) Python 3 module for constructing the SERA site amplification files for a given exposure model configuration. It supports several different workflows, depending on whether the user wishes to define the resulting site model from a regularly spaced grid, an existing exposure model or simply a shapefile of administative regions. The exposure to site tool is a short(ish) Python 3 module for constructing the ESRM20 site amplification files for a given exposure model configuration. It supports several different workflows, depending on whether the user wishes to define the resulting site model from a regularly spaced grid, an existing exposure model or simply a shapefile of administative regions.
### Before you start
The tools require additional data files not included with the main code. These files contain the regional scale European site input data. More details can be found in the description below. The data files are distributed with a Createive Commons CC-By-4.0 license and are available for download from:
<https://nextcloud.gfz-potsdam.de/s/93ZR4ky8D4mDXb9>
### Installation ### Installation
The tool requires as standard set of Python libraries that can be installed via most common Python package managers. A `setup.py` file is included to facilitate installation. We recommend using a Python virtual environment for installing and running the tools (<https://docs.python.org/3.6/tutorial/venv.html>). The tool requires as standard set of Python libraries that can be installed via most common Python package managers. A `setup.py` file is included to facilitate installation. We recommend using a Python virtual environment for installing and running the tools (<https://docs.python.org/3.8/tutorial/venv.html>).
The required dependencies are: The required dependencies are:
(Required) (Required)
* Python >=3. * Python >=3.7
* h5py >=2.10, <2.11 * h5py >=2.10
* Numpy >=1.18, <1.20 * Numpy >=1.18
* Scipy >=1.3, <1.7 * Scipy >=1.3
* Pandas >=0.25 <1.3 * Pandas >=0.25
* Shapely >=1.7, < 1.8 * Shapely >=1.7, < 2.0
* Pyproj >=1.9 * Pyproj >= 1.9
* matplotlib >=3.0 * matplotlib >= 3.0
* geopandas >=0.9.0 * geopandas >= 0.9.0
(Additional) (Additional)
...@@ -68,9 +75,9 @@ pip install --update pip ...@@ -68,9 +75,9 @@ pip install --update pip
``` ```
Then run: Then run:
```bash ```bash
cd esrm20_sitemodel/exposure_site_tool/ pip install -e esrm20_site_model/
pip install -e .
``` ```
This should install the dependencies and the `exposure2site` package. Additional packages to help run the code can be installed via This should install the dependencies and the `exposure2site` package. Additional packages to help run the code can be installed via
...@@ -80,6 +87,12 @@ pip install ipython ...@@ -80,6 +87,12 @@ pip install ipython
pip install jupyter pip install jupyter
``` ```
<i><b>Note</b>: some installations may also require `pygeos`. If an error is raised indicating that it is not availablr then it can be installed via:</i>
```bash
pip install pygeos
```
##### Windows ##### Windows
...@@ -90,7 +103,7 @@ For other users we suggest installing and running the toolkit using Anaconda. De ...@@ -90,7 +103,7 @@ For other users we suggest installing and running the toolkit using Anaconda. De
To create the virtual environment via `conda` run: To create the virtual environment via `conda` run:
```bash ```bash
conda create --name exp2site python=3.7 conda create --name exp2site python=3.8
conda activate exp2site conda activate exp2site
``` ```
...@@ -109,10 +122,10 @@ pip install --update pip ...@@ -109,10 +122,10 @@ pip install --update pip
Navigate to the `exposure_site_tool` and run the installation Navigate to the `exposure_site_tool` and run the installation
```bash ```bash
cd esrm20_sitemodel\exposure_site_tool\ pip install -e esrm20_site_model/
pip install -e .
``` ```
This should install the dependencies and the `exposure2site` package. Additional packages to help run the code can be installed via This should install the dependencies and the `exposure2site` package. Additional packages to help run the code can be installed via
```bash ```bash
...@@ -125,7 +138,11 @@ pip install jupyter ...@@ -125,7 +138,11 @@ pip install jupyter
## Required Datasets ## Required Datasets
In addition to the tools themselves, you will need several files containing pre-processed site data, which can be made available on request from the author (<graeme.weatherill@gfz-potsdam.de>). These should be stored in a subdirectory named `data`, and placed in the folder where the `exposure_to_site_tools.py` code is found. In addition to the tools themselves, you will need several files containing pre-processed site data.
The zipped file containing all of the necessary data is available for download from here: <https://nextcloud.gfz-potsdam.de/s/93ZR4ky8D4mDXb9>
The downloaded data should be unzipped and stored in a directory named `site_data`, this directory should then be placed into the main code within the folder `exposure2site`.
The required datasets comprise 5 files: The required datasets comprise 5 files:
...@@ -173,10 +190,10 @@ Longitudes and latitudes in all 30 arc-second data sets refer to the locations o ...@@ -173,10 +190,10 @@ Longitudes and latitudes in all 30 arc-second data sets refer to the locations o
# General Usage # General Usage
The Ex2Site tool is mostly intended for command line usage (although Jupyter Notebooks illustrating how to call the functions directly are also provided). With the `exposure_to_site_tools.py` file in the current working directory, and the data files in the subdirectory `./data` the tool is run by: The Ex2Site tool is mostly intended for command line usage (although Jupyter Notebooks illustrating how to call the functions directly are also provided). With the `exposure_to_site_tools.py` file in the current working directory, and the data files in the subdirectory `./site_data` the tool is run by:
```bash ```bash
python exposure_to_site_tools.py --run=choice_of_workflow --output-file=path/to/output_site_model.csv exposure2site --run=choice_of_workflow --output-file=path/to/output_site_model.csv
``` ```
...@@ -269,7 +286,7 @@ The weighting approach described previously may be applicable for the case that ...@@ -269,7 +286,7 @@ The weighting approach described previously may be applicable for the case that
The following command would produce a 120 arc-second grid between 22˚E and 30˚E and 45˚N to 51.5˚N, clipping only to onshore sites, applying building density weighing and taking the resulting property as the maximum numerical value and modal geological category. The result will be exported to `example_site_model.csv` The following command would produce a 120 arc-second grid between 22˚E and 30˚E and 45˚N to 51.5˚N, clipping only to onshore sites, applying building density weighing and taking the resulting property as the maximum numerical value and modal geological category. The result will be exported to `example_site_model.csv`
```bash ```bash
python exposure_to_site_tools.py --run grid --output-file example_site_model.csv --bbox 22/45/30/51.5 --weighting True --averaging max --geological-weighting mode --onshore-only True exposure2site --run grid --output-file example_site_model.csv --bbox 22/45/30/51.5 --weighting True --averaging max --geological-weighting mode --onshore-only True
``` ```
*** ***
...@@ -297,7 +314,7 @@ All of the optional arguments specified for the `grid` option can be used here, ...@@ -297,7 +314,7 @@ All of the optional arguments specified for the `grid` option can be used here,
The following command will read the exposure model in the file `Exposure_Res_Dummy.csv`, whose geometry can be found in the file `Adm2_Dummy.shp` in the directory `Country_Shapefiles`. Weighting based on built density will be applied, this time with the value of the site property at the maximum weighted location. Geology will be determined from the modal geological category within the polygon. The result will be written to the output xml file `Dummy_Site_Model.xml` The following command will read the exposure model in the file `Exposure_Res_Dummy.csv`, whose geometry can be found in the file `Adm2_Dummy.shp` in the directory `Country_Shapefiles`. Weighting based on built density will be applied, this time with the value of the site property at the maximum weighted location. Geology will be determined from the modal geological category within the polygon. The result will be written to the output xml file `Dummy_Site_Model.xml`
```bash ```bash
python exposure_to_site_tools.py --run exposure --output-file Dummy_Site_Model.xml --input-exposure ./Exposure_Res_Dummy.csv --shapefile-dir ./Country_Shapefiles --weighting True --averaging max --geological-weighting mode exposure2site --run exposure --output-file Dummy_Site_Model.xml --input-exposure ./Exposure_Res_Dummy.csv --shapefile-dir ./Country_Shapefiles --weighting True --averaging max --geological-weighting mode
``` ```
*** ***
...@@ -357,7 +374,7 @@ The required input file can be any comma-separated text file, but the target sit ...@@ -357,7 +374,7 @@ The required input file can be any comma-separated text file, but the target sit
The following command will define the site model from a simple csv file of longitudes and latitudes (`./site_locations.csv`) and export the resulting model to `./example_point_site_model.csv`, clipping only to onshore sites The following command will define the site model from a simple csv file of longitudes and latitudes (`./site_locations.csv`) and export the resulting model to `./example_point_site_model.csv`, clipping only to onshore sites
```bash ```bash
python exposure_to_site_tools.py --run point --output-file ./example_point_site_model.csv --input-file ./site_locations.csv --onshore-only True exposure2site --run point --output-file ./example_point_site_model.csv --input-file ./site_locations.csv --onshore-only True
``` ```
## `join` ## `join`
...@@ -377,7 +394,7 @@ One of either: ...@@ -377,7 +394,7 @@ One of either:
The following command joins together the site model in the file `site_model_countryA.csv` and the site model in the file `site_model_countryB.xml`, exporting them to `site_model_countriesAB.csv`: The following command joins together the site model in the file `site_model_countryA.csv` and the site model in the file `site_model_countryB.xml`, exporting them to `site_model_countriesAB.csv`:
```bash ```bash
python exposure_to_site_tools.py --run join --output-file site_model_countriesAB.csv --files site_model_countryA.csv+site_model_countryB.xml exposure2site --run join --output-file site_model_countriesAB.csv --files site_model_countryA.csv+site_model_countryB.xml
``` ```
......
from exposure2site.exposure_to_site_tools import SiteManager from exposure2site.exposure_to_site_tools import (
SiteManager, get_site_set_from_exposure, get_maximum_admin_level)
from exposure2site.node_handler import Node, nrml_read, nrml_write from exposure2site.node_handler import Node, nrml_read, nrml_write
__all__ = [ __all__ = [
"SiteManager", "SiteManager",
"get_site_set_from_exposure",
"get_maximum_admin_level",
"Node", "Node",
"nrml_read", "nrml_read",
"nrml_write", "nrml_write",
......
...@@ -51,7 +51,7 @@ GEOL_DICT_NAME_TO_KEY = {"UNKNOWN": 0, ...@@ -51,7 +51,7 @@ GEOL_DICT_NAME_TO_KEY = {"UNKNOWN": 0,
# Data files # Data files
DATA_PATH = os.path.join(os.path.dirname(__file__), "data") DATA_PATH = os.path.join(os.path.dirname(__file__), "site_data")
# Geology shapefiles # Geology shapefiles
GEOLOGY_FILE = os.path.join(DATA_PATH, "GEOL_V8_ERA2.shp") GEOLOGY_FILE = os.path.join(DATA_PATH, "GEOL_V8_ERA2.shp")
# File containing 30 arc second grid of slope, geology, elevation and # File containing 30 arc second grid of slope, geology, elevation and
...@@ -849,7 +849,7 @@ def get_slope_geology_vs30_at_location(site_lons, site_lats, spc=(1. / 120.), ...@@ -849,7 +849,7 @@ def get_slope_geology_vs30_at_location(site_lons, site_lats, spc=(1. / 120.),
print("---- Retrieving data within bounding box") print("---- Retrieving data within bounding box")
lons, lats, slope = slice_wgs84_datafile(SITE_HDF5_FILE, bbox, "slope") lons, lats, slope = slice_wgs84_datafile(SITE_HDF5_FILE, bbox, "slope")
geology = slice_wgs84_datafile(SITE_HDF5_FILE, bbox, "geology")[2] geology = slice_wgs84_datafile(SITE_HDF5_FILE, bbox, "geology")[2]
vs30 = slice_wgs84_datafile(VS30_HDF5_FILE, bbox, "vs30")[2] lons_vs, lats_vs, vs30 = slice_wgs84_datafile(VS30_HDF5_FILE, bbox, "vs30")
if onshore_only: if onshore_only:
elevation = slice_wgs84_datafile(SITE_HDF5_FILE, bbox, "elevation")[2] elevation = slice_wgs84_datafile(SITE_HDF5_FILE, bbox, "elevation")[2]
else: else:
...@@ -858,6 +858,8 @@ def get_slope_geology_vs30_at_location(site_lons, site_lats, spc=(1. / 120.), ...@@ -858,6 +858,8 @@ def get_slope_geology_vs30_at_location(site_lons, site_lats, spc=(1. / 120.),
# Integer number of multiples of spc indicates the x and y locations # Integer number of multiples of spc indicates the x and y locations
dx = ((site_lons - (lons[0] - spc / 2.)) / spc).astype(int) dx = ((site_lons - (lons[0] - spc / 2.)) / spc).astype(int)
dy = (np.fabs((site_lats - (lats[0] + spc / 2.)) / spc)).astype(int) dy = (np.fabs((site_lats - (lats[0] + spc / 2.)) / spc)).astype(int)
dx_vs = ((site_lons - (lons_vs[0] - spc / 2.)) / spc).astype(int)
dy_vs = (np.fabs((site_lats - (lats_vs[0] + spc / 2.)) / spc)).astype(int)
# Get backarc distance # Get backarc distance
print("---- Getting volcanic distance") print("---- Getting volcanic distance")
xvf = interpolate_xvf_grid(site_lons, site_lats) xvf = interpolate_xvf_grid(site_lons, site_lats)
...@@ -867,7 +869,7 @@ def get_slope_geology_vs30_at_location(site_lons, site_lats, spc=(1. / 120.), ...@@ -867,7 +869,7 @@ def get_slope_geology_vs30_at_location(site_lons, site_lats, spc=(1. / 120.),
print("---- Building dataframe") print("---- Building dataframe")
elevation = elevation[dy, dx] elevation = elevation[dy, dx]
idx = elevation >= -5.0 idx = elevation >= -5.0
print(site_lons, site_lats) #print(site_lons, site_lats)
return gpd.GeoDataFrame({ return gpd.GeoDataFrame({
"geometry": gpd.GeoSeries([ "geometry": gpd.GeoSeries([
Point(lon, lat) Point(lon, lat)
...@@ -875,7 +877,7 @@ def get_slope_geology_vs30_at_location(site_lons, site_lats, spc=(1. / 120.), ...@@ -875,7 +877,7 @@ def get_slope_geology_vs30_at_location(site_lons, site_lats, spc=(1. / 120.),
"lon": site_lons[idx], "lon": site_lons[idx],
"lat": site_lats[idx], "lat": site_lats[idx],
"slope": slope[dy, dx][idx], "slope": slope[dy, dx][idx],
"vs30": vs30[dy, dx][idx], "vs30": vs30[dy_vs, dx_vs][idx],
"geology": [GEOL_DICT_KEY_TO_NAME[geol] "geology": [GEOL_DICT_KEY_TO_NAME[geol]
for geol in geology[dy, dx][idx]], for geol in geology[dy, dx][idx]],
"xvf": xvf[idx]}) "xvf": xvf[idx]})
...@@ -1707,7 +1709,8 @@ def set_up_arg_parser(): ...@@ -1707,7 +1709,8 @@ def set_up_arg_parser():
return parser return parser
if __name__ == "__main__": #if __name__ == "__main__":
def main():
parser = set_up_arg_parser() parser = set_up_arg_parser()
# Argparser - arguments # Argparser - arguments
args = parser.parse_args() args = parser.parse_args()
...@@ -1866,3 +1869,7 @@ if __name__ == "__main__": ...@@ -1866,3 +1869,7 @@ if __name__ == "__main__":
else: else:
# Export to csv # Export to csv
site_manager.to_csv(args.output_file) site_manager.to_csv(args.output_file)
if __name__ == "__main__":
main()
...@@ -8,18 +8,19 @@ setup( ...@@ -8,18 +8,19 @@ setup(
version="1.0", version="1.0",
description="Tool for building and calibrating OpenQuake site files from exposure models", description="Tool for building and calibrating OpenQuake site files from exposure models",
license="GPLv3", license="GPLv3",
extras_require = {"OQ": ['openquake.engine',]}, extras_require = {"OQ": ['openquake.engine>=3.13',]},
install_requires= [ install_requires= [
'h5py >=2.10, <2.11', 'h5py >=2.10, <2.11',
'numpy >=1.18, <1.20', 'numpy >=1.18',
'scipy >=1.3, <1.7', 'scipy >=1.3',
'pandas >=0.25, <1.3', 'pandas >=0.25',
'shapely >=1.7, <1.8', 'shapely >=1.7, <2.0',
'pyproj >=1.9', 'pyproj >=1.9',
'matplotlib', 'matplotlib',
'geopandas', 'geopandas',
'ipython'], 'ipython'],
packages=find_packages(), packages=find_packages(),
python_requires=">=3.6", python_requires=">=3.8",
entry_points={"console_scripts": ["exposure2site = exposure2site.exposure_to_site_tools:main"]}
) )
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment