The NextGen Hydrofabic Data Model
Mike Johnson
Lynker, NOAA-AffiliateSource:
vignettes/04-data-model-deep-dive.Rmd
04-data-model-deep-dive.RmdData Model
A rigorous data model has been developed to provide the minimal, yet efficient, types of data needed for modeling, hydrofabric subsetting, cross walking, and indexing.
Much of this design stems from the OGC report here with modifications made for the NextGen modelling application.
The current data model contains 7 data layers with an anticipated 8th for water bodies.
5 of these are spatial and 2 are a-spatial highlighting the value of the geopackage model:
1. Divides
Divides represent the incremental area draining to a flowpath.
They are a one-dimensional, hydrology-specific realization of the holistic catchment.
Geometrically, each divide is a an edge bounded by inflow and outflow nodes forming a POLYGON
Divide Type
Divides provide a continuous coverage of the modeling domain. While the majority of the domain is dendritic, there are areas where there is a divide with no flow, or, flow with no divide.
For example:
a coastal divide has an area, but no flowpath.
an internally draining region (sink) has an area, but no flow path
An artificial path (canal) has a flowpath, but no associated drainage area
When both a divide and flowpath representation exisit, there is a
1:1relationship meaning there is oneflowpathrepresentation for eachdivide. In this case the numeric ID of each is the same
In the future, we will support the ability to have
1:many divide:*flowline*
representations
divides = read_sf(poudre, "divides")
mapview(divides[5,])2. Flowpaths
- The flowpath references the path of a moving particle of water
- The flowpath can be represented as an edge bounded by inflow and outflow nodes, and associated with left-bank and right-bank sub-catchment faces.
- Geometrically, each flowpath is a LINESTRING connecting the inflow to the outflow with some level of sinuosity
- The representation of a flowpath can change per modeling application from straight line to “infinity” sinuous
# Read in all flowpaths
flowpaths = read_sf(poudre, "flowpaths")
# Define outflow
outflow = slice_max(flowpaths, hydroseq)
mapview(outflow) +
filter(divides, divide_id == outflow$divide_id)Durable Integration
- Many fields in the hydrofabric are designed to remain resilient through the refactor/aggregation process.
- Many of these are related to network indexing and integration
- A primary one of these is the mainstem identifier that is intended to remain persistent across network manipulations
- This data model is described in detail here and a video is here
# Mainstem Persistent Identifier
(pid = glue("https://geoconnex.us/ref/mainstems/{outflow$mainstem}"))## https://geoconnex.us/ref/mainstems/352913
# Geonconnex PID mainstem read for URL
geoconnex_mainstem = read_sf(pid)
# Mainstem defined by membership of the outlets
hf_mainstem = filter(flowpaths, mainstem == outflow$mainstem)
# View
mapview(geoconnex_mainstem) +
mapview(hf_mainstem, color = "red")- Anything that is mapped to a shared mainstem ID (observations, models, etc) can be shared across realizations and become more “durable”
3. Nexus
A nexus provides a conceptual outlet for water contained by a catchment.
They are the locations where NextGen will exchange information!
some are defined by POIs
Others are defined by locations in a network where a 1:1 inflow and outflow occur
Every catchment flows to a hydro nexus, conversely every location in a hydrologic system can be thought of as a hydro nexus that drains some catchment(s).
Consequense of Network Aggregation
More then one divide/flowpath (e.g. catchment) can contribute to a single nexus:
## [1] "nex-278677"
(sub_divides = filter(divides, toid == example))## Simple feature collection with 3 features and 9 fields
## Geometry type: MULTIPOLYGON
## Dimension: XY
## Bounding box: xmin: -822945 ymin: 1992135 xmax: -814845 ymax: 1998225
## Projected CRS: NAD83 / Conus Albers
## # A tibble: 3 × 10
## divide_id toid type ds_id areasqkm id lengthkm tot_drainage_areasqkm
## * <chr> <chr> <chr> <dbl> <dbl> <chr> <dbl> <dbl>
## 1 cat-278676 nex-2786… netw… NA 8.76 wb-2… 5.79 212.
## 2 cat-302482 nex-2786… netw… NA 10.8 wb-3… 5.93 10.8
## 3 cat-307410 nex-2786… netw… NA 4.60 wb-3… 3.97 4.60
## # ℹ 2 more variables: has_flowline <int>, geom <MULTIPOLYGON [m]>4. Hydrolocations
A hydrolocation can define any location of hydrologic significance located “on” a hydrologic network
In a given dataset, hydrolocations may or may not have an associated hydrologic nexus and catchment feature.
In such cases, hydro locations would typically be linearly-referenced to a defined set of flowpaths.
Topologically, a hydro-location can be understood as an inlet or outlet node located at the end of a flowpath edge.
In NextGen design, a subset of the community POIs, are treated as hydrolocations in which the network CANNOT be refactored or aggregated “over”.
## Simple feature collection with 1 feature and 6 fields
## Geometry type: POINT
## Dimension: XY
## Bounding box: xmin: -760037.4 ymin: 1989138 xmax: -760037.4 ymax: 1989138
## Projected CRS: NAD83 / Conus Albers
## # A tibble: 1 × 7
## hl_id hl_reference hl_link hl_uri hl_position id geometry
## * <chr> <chr> <chr> <chr> <chr> <chr> <POINT [m]>
## 1 8118 Gages 06752260 Gages… outflow nex-… (-760037.4 1989138)
glimpse(hl)## Rows: 1
## Columns: 7
## $ hl_id <chr> "8118"
## $ hl_reference <chr> "Gages"
## $ hl_link <chr> "06752260"
## $ hl_uri <chr> "Gages-06752260"
## $ hl_position <chr> "outflow"
## $ id <chr> "nex-278699"
## $ geometry <POINT [m]> POINT (-760037.4 1989138)
(pid = glue('https://reference.geoconnex.us/collections/{tolower(hl$hl_reference)}/items?provider_id={hl$hl_link}'))## https://reference.geoconnex.us/collections/gages/items?provider_id=06752260
pid = read_sf(pid)
glimpse(pid)## Rows: 1
## Columns: 13
## $ id <chr> "1015611"
## $ fid <int> 70633
## $ name <chr> "CACHE LA POUDRE RIVER AT FORT COLLINS, CO"
## $ subjectof <chr> "https://waterdata.usgs.gov/monitoring-location/0…
## $ provider_id <chr> "06752260"
## $ nhdpv2_reach_measure <dbl> 46.32248
## $ mainstem_uri <chr> "https://geoconnex.us/ref/mainstems/352913"
## $ description <chr> "USGS NWIS Stream/River/Lake Site 06752260: CACHE…
## $ uri <chr> "https://geoconnex.us/ref/gages/1015611"
## $ provider <chr> "https://waterdata.usgs.gov"
## $ nhdpv2_reachcode <chr> "10190007000017"
## $ nhdpv2_comid <dbl> 2900003
## $ geometry <POINT [°]> POINT (-105.0692 40.58808)Collectively can help find, and link data!
#Use the hl_link to extract this weeks rain event
instFlow <- dataRetrieval::readNWISdata(
sites = hl$hl_link, service = "iv",
parameterCd = "00060",
startDate = "2023-05-10"
)
# Use PID COMID to extract the 42 year NWM flow record
nwm21 = nwmTools::readNWMdata(comid = pid$nhdpv2_comid)
Multiplicity
The data design of the hydrolocation location layer intends to represent each realization of a POI independently.
This means we can have a shared hl_id and
id (nexus location) but distinct hl_reference,
and hl_link
(poi = filter(hydrolocations, hl_id == 7253))## Simple feature collection with 3 features and 6 fields
## Geometry type: POINT
## Dimension: XY
## Bounding box: xmin: -768368.2 ymin: 1991210 xmax: -768368.2 ymax: 1991210
## Projected CRS: NAD83 / Conus Albers
## # A tibble: 3 × 7
## hl_id hl_reference hl_link hl_uri hl_position id geometry
## * <chr> <chr> <chr> <chr> <chr> <chr> <POINT [m]>
## 1 7253 HUC12 1019000… HUC12… outflow nex-… (-768368.2 1991210)
## 2 7253 NID CO01659… NID-C… outflow nex-… (-768368.2 1991210)
## 3 7253 WBOut 2897349 WBOut… outflow nex-… (-768368.2 1991210)Rigorous community POI mapping
The geoconnex PID can provide the location of the source data.
Here we can see the community POI was moved slightly to integrate with the reference network:
geoconnex = read_sf('https://reference.geoconnex.us/collections/dams/items?provider_id=CO01659')
mapview(geoconnex, color = "red") + poi5. Network
The network layer is tabular and has no spatial information
It provides the ultimate cross walk between the NextGen fabric, the source hydrofabric, and all mapped hydrolocations.
This network is key to hydroindexing, data query, and network subsetting.
Example:
What is the NextGen ID near me?
Lets say we lived at this location in Laporte, CO
pt = st_sfc(st_point(c(-105.14044, 40.62949)), crs = 4326)
mapview(pt)We can use the NLDI to map the reference fabric to this point, and then search for how the reference fabric hf_id merged into the current network:
(x = findNLDI(location = pt))## $origin
## Simple feature collection with 1 feature and 3 fields
## Geometry type: LINESTRING
## Dimension: XY
## Bounding box: xmin: -105.1512 ymin: 40.61313 xmax: -105.1206 ymax: 40.62606
## Geodetic CRS: WGS 84
## # A tibble: 1 × 4
## sourceName identifier comid geometry
## <chr> <chr> <chr> <LINESTRING [°]>
## 1 NHDPlus comid 2899849 2899849 (-105.1512 40.62592, -105.1489 40.62606, -10…
(n = filter(read_sf(poudre, "network"), hf_id == x$origin$comid))## # A tibble: 1 × 15
## id toid divide_id ds_id mainstem hl_id hydroseq hl_uri hf_source hf_id
## <chr> <chr> <chr> <dbl> <int> <chr> <int> <chr> <chr> <dbl>
## 1 wb-2786… nex-… cat-2786… NA 352913 NA 11625 NA NHDPlusV2 2.90e6
## # ℹ 5 more variables: lengthkm <dbl>, areasqkm <dbl>,
## # tot_drainage_areasqkm <dbl>, type <chr>, vpu <chr>6. Flowpath Attributes
Flowpath attributes are extracted from the NWM Routelink file.
The network layer is used to “length weight average” these attributes to the new network.
7. Lakes
Lake attributes are extracted from the NWM Lake file file.
The network layer is used to “length weight average” these attributes to the new network.
lakes = read_sf(poudre, "lakes") %>%
filter(hl_id == 7253) %>%
glimpse()## Rows: 1
## Columns: 18
## $ id <chr> "nex-287191"
## $ toid <chr> "wb-287191"
## $ hl_id <chr> "7253"
## $ hl_reference <chr> "WBOut"
## $ hl_link <dbl> 2897349
## $ hl_uri <chr> "WBOut-2897349"
## $ Dam_Length <dbl> 10
## $ ifd <dbl> 0.9
## $ LkArea <dbl> 6.87471
## $ LkMxE <dbl> 1678.71
## $ OrificeA <dbl> 1
## $ OrificeC <dbl> 0.1
## $ OrificeE <dbl> 1625.35
## $ time <dbl> 0
## $ WeirC <dbl> 0.4
## $ WeirE <dbl> 1670.706
## $ WeirL <dbl> 10
## $ geom <POINT> POINT (-768368.2 1991210)8. Layer Style
When you open a hydrofabric gpkg created with many of the
hydrofabtools, you will find they load with a consistent symbology.These symbologies are saved with the Geopackage using the
hydrofab::append_style()Built on reference QGIS QML files.
read_sf(poudre, "layer_styles") %>%
glimpse()## Rows: 5
## Columns: 12
## $ f_table_catalog <chr> "", "", "", "", ""
## $ f_table_schema <chr> "", "", "", "", ""
## $ f_table_name <chr> "divides", "nexus", "flowpaths", "hydrolocations", "…
## $ f_geometry_column <chr> "geom", "geom", "geom", "geom", "geom"
## $ styleName <chr> "divides__hydrofabric_style", "nexus__hydrofabric_st…
## $ styleQML <chr> "<!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'S…
## $ styleSLD <chr> "", "", "", "", ""
## $ useAsDefault <lgl> TRUE, TRUE, TRUE, TRUE, TRUE
## $ description <chr> "Generated for hydrofabric", "Generated for hydrofab…
## $ owner <chr> "", "", "", "", ""
## $ ui <lgl> NA, NA, NA, NA, NA
## $ update_time <dttm> 2023-10-12 12:53:35, 2023-10-12 12:53:35, 2023-10-12…
a = system.file("qml", "divides.qml", package = "hydrofab")
readLines(a)[1:20]## [1] "<!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>"
## [2] "<qgis version=\"3.22.6-Białowieża\" styleCategories=\"Symbology\">"
## [3] " <renderer-v2 enableorderby=\"0\" attr=\"type\" forceraster=\"0\" type=\"categorizedSymbol\" symbollevels=\"0\" referencescale=\"-1\">"
## [4] " <categories>"
## [5] " <category symbol=\"0\" value=\"coastal\" label=\"coastal\" render=\"true\"/>"
## [6] " <category symbol=\"1\" value=\"network\" label=\"network\" render=\"true\"/>"
## [7] " <category symbol=\"2\" value=\"internal\" label=\"internal\" render=\"true\"/>"
## [8] " <category symbol=\"4\" value=\"terminal\" label=\"terminal\" render=\"true\"/>"
## [9] " </categories>"
## [10] " <symbols>"
## [11] " <symbol name=\"0\" force_rhr=\"0\" clip_to_extent=\"1\" alpha=\"1\" type=\"fill\">"
## [12] " <data_defined_properties>"
## [13] " <Option type=\"Map\">"
## [14] " <Option name=\"name\" value=\"\" type=\"QString\"/>"
## [15] " <Option name=\"properties\"/>"
## [16] " <Option name=\"type\" value=\"collection\" type=\"QString\"/>"
## [17] " </Option>"
## [18] " </data_defined_properties>"
## [19] " <layer locked=\"0\" class=\"SimpleFill\" enabled=\"1\" pass=\"0\">"
## [20] " <Option type=\"Map\">"