rewrite arc_open to support item id's or portal urls #275

JosiahParry · 2025-07-29T17:49:29Z

Checklist

update NEWS.md
documentation updated with devtools::document()
devtools::check() passes locally

Changes

This PR refactors arc_open() to support item IDs as well as portal URLs. This builds off of @elipousson's work at R-ArcGIS/arcgisutils#62

Issues that this closes

Please link any issues that are closed by this PR

Follow up tasks

add user and group classes to arc_user() and arc_group() functions in arcgisutils

JosiahParry · 2025-07-29T17:55:00Z

Here is most use cases:

library(arcgis)
#> Warning: package 'arcgis' was built under R version 4.4.2
#> Attaching core arcgis packages:
#> → arcgisutils v0.3.3.9000
#> → arcgislayers v0.4.0.9000
#> → arcgisgeocode v0.2.3
#> → arcgisplaces v0.1.1

test_cases <- c(
  map_server = "https://image.discomap.eea.europa.eu/arcgis/rest/services/Corine/CLC2000_WM/MapServer",
  feature_layer = "https://image.discomap.eea.europa.eu/arcgis/rest/services/Corine/CLC2000_WM/MapServer/0",
  feature_server = "https://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/USA_Major_Cities_/FeatureServer",
  tile_imagery = "https://image.arcgisonline.nl/arcgis/rest/services/KEA/Maximale_overstromingsdiepte/ImageServer",
  elevation = "https://tiles.arcgis.com/tiles/qHLhLQrcvEnxjtPr/arcgis/rest/services/British_National_Grid_Terrain_3D/ImageServer",
  webmap_app = "https://esri2.maps.arcgis.com/apps/instant/media/index.html?appid=80eb92ffc89b4086abe8cedd58ab160c",
  storymap = "https://storymaps.arcgis.com/stories/ad791fda858c46fdbe79636aa5f35dd8",
  instant_app = "https://actgov.maps.arcgis.com/apps/instant/interactivelegend/index.html?appid=f2dfd67d29ed4cabbb91e742e0297955",
  dashboard = "https://www.arcgis.com/apps/dashboards/84ba9c03786e462d960e3172bc1b2204",
  experience = "https://experience.arcgis.com/experience/6e360741bfd84db79d5db774a1147815",
  webapp = "https://governmentofbc.maps.arcgis.com/apps/webappviewer/index.html?id=950b4eec577a4dc5b298a61adab41c06",
  notebook_item = "https://geosaurus.maps.arcgis.com/home/item.html?id=9a9fca3f09bb41dd856c9cd4239b8519",
  notebook = "https://geosaurus.maps.arcgis.com/home/notebook/notebook.html?id=9a9fca3f09bb41dd856c9cd4239b8519",
  webscene = "https://analysis-1.maps.arcgis.com/home/webscene/viewer.html?webscene=7b506043536246faa4194d4c3d4c921b",
  item_db = "https://analysis-1.maps.arcgis.com/home/item.html?id=84ba9c03786e462d960e3172bc1b2204",
  item_mapserver = "https://analysis-1.maps.arcgis.com/home/item.html?id=1d150c40d9f642cb8bd691017bf22cee",
  feature_collection = "https://analysis-1.maps.arcgis.com/home/item.html?id=24aa36ce1d7747c2b5a6aa57711d03fb",
  geocode = "https://geocode.arcgis.com/arcgis/rest/services/World/GeocodeServer",
  table = "https://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/USA_Wetlands/FeatureServer/1",
  fserv_id = "3c164274a80748dda926a046525da610"
)

lapply(test_cases, arc_open)
#> $map_server
#> <MapServer <2 layers, 0 tables>>
#> CRS: 3857
#> Capabilities: Map,Query,Data
#>   0: Corine Land Cover 2000 vector (esriGeometryPolygon)
#>   1: Corine Land Cover 2000 raster (NA)
#> 
#> $feature_layer
#> <FeatureLayer>
#> Name: Corine Land Cover 2000 vector
#> Geometry Type: esriGeometryPolygon
#> CRS: 3857
#> Capabilities: Map,Query,Data
#> 
#> $feature_server
#> <FeatureServer <1 layer, 0 tables>>
#> CRS: 4326
#> Capabilities: Query,Extract
#>   0: USA Major Cities (esriGeometryPoint)
#> 
#> $tile_imagery
#> <ImageServer <1 bands, 0 fields>>
#> <list <1 bands, 0 fields>>
#> Name: KEA/Maximale_overstromingsdiepte
#> Description: 
#> Extent: 11825 284850 306750 619825 (xmin, xmax, ymin, ymax)
#> Resolution: 25 x 25
#> CRS: 28992
#> Capabilities: Image,Metadata,Mensuration
#> 
#> $elevation
#> <ImageServer <1 bands, 0 fields>>
#> <list <1 bands, 0 fields>>
#> Name: British_National_Grid_Terrain_3D
#> Description: <font color='#000000' size='4'><b>Overview</b></font><div><p style
#> Extent: 0.48 660002.07 -0.96 1230001.14 (xmin, xmax, ymin, ymax)
#> Resolution: 0 x 0
#> CRS: 27700
#> Capabilities: Image,TilesOnly,Tilemap
#> 
#> $webmap_app
#> <PortalItem<Web Mapping Application>>
#> id: 80eb92ffc89b4086abe8cedd58ab160c
#> title: Esri Community Maps Contributors of Road Closures
#> owner: dkensok_esri2
#> 
#> $storymap
#> <PortalItem<StoryMap>>
#> id: ad791fda858c46fdbe79636aa5f35dd8
#> title: Smart Mapping - Counts and Amounts (color)
#> owner: esri_policy_maps
#> 
#> $instant_app
#> <PortalItem<Web Mapping Application>>
#> id: f2dfd67d29ed4cabbb91e742e0297955
#> title: ACTGOV ACT Canopy Cover Map 2020
#> owner: environment_ACTGOV
#> 
#> $dashboard
#> <PortalItem<Dashboard>>
#> id: 84ba9c03786e462d960e3172bc1b2204
#> title: Coral Bleaching Locations
#> owner: kvangraafeiland_oceans
#> 
#> $experience
#> <PortalItem<Web Experience>>
#> id: 6e360741bfd84db79d5db774a1147815
#> title: Abortion Access Dashboard
#> owner: anieto_ANGP
#> 
#> $webapp
#> <PortalItem<Web Mapping Application>>
#> id: 950b4eec577a4dc5b298a61adab41c06
#> title: EMCR: EmergencyMapBC
#> owner: EM.EMBC
#> 
#> $notebook_item
#> <PortalItem<Notebook>>
#> id: 9a9fca3f09bb41dd856c9cd4239b8519
#> title: MSGIC Training Notebook
#> owner: NGiner_geosaurus
#> 
#> $notebook
#> <PortalItem<Notebook>>
#> id: 9a9fca3f09bb41dd856c9cd4239b8519
#> title: MSGIC Training Notebook
#> owner: NGiner_geosaurus
#> 
#> $webscene
#> <PortalItem<Web Scene>>
#> id: 7b506043536246faa4194d4c3d4c921b
#> title: 3D model of Zürich, CH
#> owner: demo_3dgis
#> 
#> $item_db
#> <PortalItem<Dashboard>>
#> id: 84ba9c03786e462d960e3172bc1b2204
#> title: Coral Bleaching Locations
#> owner: kvangraafeiland_oceans
#> 
#> $item_mapserver
#> <MapServer <2 layers, 0 tables>>
#> CRS: 3857
#> Capabilities: Map,Query,Data
#>   0: Corine Land Cover 2000 vector (esriGeometryPolygon)
#>   1: Corine Land Cover 2000 raster (NA)
#> 
#> $feature_collection
#> <PortalItem<Feature Collection>>
#> id: 24aa36ce1d7747c2b5a6aa57711d03fb
#> title: California Cities Open Platform
#> owner: jparry_ANGP
#> 
#> $geocode
#> <GeocodeServer>
#> Description: World Geocoder
#> Version: 11.4
#> CRS: 4326
#> 
#> $table
#> <Table>
#> Name: Pop_Up_Table
#> Capabilities: Query,Extract,Sync
#> 
#> $fserv_id
#> <FeatureServer <1 layer, 0 tables>>
#> CRS: 4326
#> Capabilities: Query,Extract
#>   0: USA Counties - Generalized (esriGeometryPolygon)

^{Created on 2025-07-29 with reprex v2.1.1}

JosiahParry · 2025-07-29T17:56:47Z

@elipousson this is super exciting! thank you for making it possible.

Would you be able to take a look at it sometime this week?

elipousson · 2025-07-29T22:21:23Z

@JosiahParry Will do! Super exciting to see this update.

elipousson · 2025-07-30T01:59:52Z

I went looking for edge cases and found some opportunities for improved validation (or additional support). Reprex below. A few more thoughts although I may give this another look tomorrow:

The @inheritParams arc_item in the arc_open docs should be @inheritParams arcgisutils::arc_item.

Among the examples listed below, I think adding support for base server URLs and folder URLs could be really helpful. That would require pattern matching on the "rest/services" path in the URL for arcgislayers:: arc_url_type() to return a new type value.

I can't recall if you planned on supporting profile and group URLs with this refactor or not. If they work as is, I think adding a print method would make more sense than forcing an error – but it is weird that they work but lack the print method of portal items and layers/servers.

Lastly, I think parsing the date-time values returned by items would be very helpful. A less-experienced user may not know what to do with epoch time values.

library(arcgislayers)

# Error because internal experience builder URL returns NULL value for `info$type` then passed to switch
experience_builder_url <- "https://experience.arcgis.com/builder/?id=cdacbe8122924255b21542cdf16b0ee4&views=page"

arc_open(experience_builder_url)
#> Error in switch(info$type, FeatureServer = {: EXPR must be a length 1 vector

# Same error for ArcGIS Hub item URL
hub_item_url <- "https://data.baltimorecity.gov/datasets/189e6d1c65df4e13b38c0027cee574f6_3/explore"

arc_open(hub_item_url)
#> Error in switch(info$type, FeatureServer = {: EXPR must be a length 1 vector

# Same error for mapviewer URL
mapviewer_url <- "https://www.arcgis.com/apps/mapviewer/index.html?panel=gallery&layers=189e6d1c65df4e13b38c0027cee574f6"

arc_open(mapviewer_url)
#> Error in switch(info$type, FeatureServer = {: EXPR must be a length 1 vector

# Same error for sharing URL (item metadata link from bottom of layer page)
sharing_url <- "https://www.arcgis.com/sharing/rest/content/items/189e6d1c65df4e13b38c0027cee574f6/info/metadata/metadata.xml?format=default&output=html"

arc_open(sharing_url)
#> Error in switch(info$type, FeatureServer = {: EXPR must be a length 1 vector

# Same error for server base URL
server_url <- "https://egisdata.baltimorecity.gov/egis/rest/services"

arc_open(server_url)
#> Error in switch(info$type, FeatureServer = {: EXPR must be a length 1 vector

# Same error for folder URL
folder_url <- "https://egisdata.baltimorecity.gov/egis/rest/services/BaseMaps"

arc_open(folder_url)
#> Error in switch(info$type, FeatureServer = {: EXPR must be a length 1 vector

# Error for cli message with search URL
search_url <- "https://www.arcgis.com/home/search.html?restrict=false&sortField=relevance&sortOrder=desc&searchTerm=pdf#content"

arc_open(search_url)
#> Error in "fun(..., .envir = .envir)": ! Could not evaluate cli `{}` expression: `layer_class`.
#> Caused by error in `eval(expr, envir = envir)`:
#> ! object 'layer_class' not found

# Works but has no print method for profile URL
profile_url <- "https://www.arcgis.com/home/user.html?user=Cook_County_GIS"

arc_open(profile_url)
#> $username
#> [1] "Cook_County_GIS"
#> 
#> $udn
#> NULL
#> 
#> $id
#> [1] "9aef18fe02cd29ef46fc231aa7c4ec6e"
#> 
#> $fullName
#> [1] "Cook County GIS"
#> 
#> $firstName
#> [1] "Cook County"
#> 
#> $lastName
#> [1] "GIS"
#> 
#> $description
#> [1] "This is the official Cook County, Illinois ArcGIS Online account.  The authoritative GIS data published by this user is maintained by Cook County employees and published by the Cook County GIS Department.  The department can be reached by email at gis@cookcountyil.gov or by phone at (312) 603-1369."
#> 
#> $tags
#> NULL
#> 
#> $culture
#> [1] "en"
#> 
#> $cultureFormat
#> [1] "us"
#> 
#> $region
#> [1] "US"
#> 
#> $units
#> [1] "english"
#> 
#> $thumbnail
#> [1] "blob.png"
#> 
#> $access
#> [1] "public"
#> 
#> $created
#> [1] 1.413905e+12
#> 
#> $modified
#> [1] 1.740155e+12
#> 
#> $provider
#> [1] "arcgis"

# Works the same for group URL
group_url <- "https://www.arcgis.com/home/group.html?id=2e93ce445a314d0f897c3b6ed53b601d#overview"

arc_open(group_url)
#> $id
#> [1] "2e93ce445a314d0f897c3b6ed53b601d"
#> 
#> $title
#> [1] "Forest Preserve District (open data)"
#> 
#> $isInvitationOnly
#> [1] TRUE
#> 
#> $owner
#> [1] "Cook_County_GIS"
#> 
#> $description
#> [1] ""
#> 
#> $snippet
#> [1] "Forest Preserve District group that contains data/services that will be available as open data."
#> 
#> $tags
#> [1] "Forest Preserve" "Parks"           "Cook County"     "Recreation"     
#> 
#> $typeKeywords
#> NULL
#> 
#> $phone
#> [1] ""
#> 
#> $sortField
#> [1] "title"
#> 
#> $sortOrder
#> [1] "asc"
#> 
#> $isViewOnly
#> [1] TRUE
#> 
#> $isOpenData
#> [1] TRUE
#> 
#> $featuredItemsId
#> NULL
#> 
#> $thumbnail
#> [1] "FPDCC2.jpg"
#> 
#> $created
#> [1] 1.473342e+12
#> 
#> $modified
#> [1] 1.47336e+12
#> 
#> $access
#> [1] "public"
#> 
#> $capabilities
#> NULL
#> 
#> $isFav
#> [1] FALSE
#> 
#> $isReadOnly
#> [1] FALSE
#> 
#> $protected
#> [1] FALSE
#> 
#> $autoJoin
#> [1] FALSE
#> 
#> $notificationsEnabled
#> [1] FALSE
#> 
#> $provider
#> NULL
#> 
#> $providerGroupName
#> [1] ""
#> 
#> $leavingDisallowed
#> [1] FALSE
#> 
#> $hiddenMembers
#> [1] FALSE
#> 
#> $membershipAccess
#> [1] "org"
#> 
#> $displaySettings
#> $displaySettings$itemTypes
#> [1] ""
#> 
#> 
#> $orgId
#> [1] "I5Or36sMcO7Y9vQ3"
#> 
#> $properties
#> NULL
#> 
#> $collaborationInfo
#> NULL

# Items include epoch-time formatted dates in ouput
item_url <- "https://www.arcgis.com/home/item.html?id=22787a4b4d1b42bfa65de8435fa09686"

item <- arc_open(item_url)

item[c("created", "modified", "lastViewed")]
#> $created
#> [1] 1.643826e+12
#> 
#> $modified
#> [1] 1.670616e+12
#> 
#> $lastViewed
#> [1] 1.753834e+12

item[c("created", "modified", "lastViewed")] <- lapply(
  item[c("created", "modified", "lastViewed")],
  function(x) {
    if (is.null(x)) {
      return(NULL)
    }

    # Format epoch date
    as.POSIXct(
      as.numeric(x) / 1000,
      origin = "1970-01-01",
      tz = ""
    )
  }
)

item[c("created", "modified", "lastViewed")]
#> $created
#> [1] "2022-02-02 13:26:28 EST"
#> 
#> $modified
#> [1] "2022-12-09 15:01:29 EST"
#> 
#> $lastViewed
#> [1] "2025-07-29 20:00:00 EDT"

^{Created on 2025-07-29 with reprex v2.1.1}

JosiahParry · 2025-07-30T15:21:03Z

This is super helpful! Thanks @elipousson I'll give this a review later today.

And I agree about the print method as well—I'll do that in arcgisutils. It is handy to be able to get user metadata for other services to get things such as available geocoders or available service urls.

Do you have an idea for detecting which fields should be parsed as dates? My gut instinct is to grep for "date" and then use from_esri_date() on those fields

elipousson · 2025-07-30T16:30:28Z

I'm not sure about the dates. I wish there was a single giant YAML file with the specs for all of these objects in a single place. It seems a bit weird to be doing this reverse engineering process when you actually work for ESRI!

JosiahParry · 2025-07-30T17:03:11Z

@elipousson trust me i know! 🙃 not to mention there are changes to each of the services every few months adding and deprecating things.

Though the reference documentation typically has really thorough JSON response types—e.g. so that can be used too https://developers.arcgis.com/rest/services-reference/enterprise/feature-service/#json-response-syntax

JosiahParry · 2025-08-01T17:32:21Z

Thank you so much for these! I'll make sure that these edge cases are handled with at least an informative error and warning as well as handling the service folders.

However mapview is a wholeeeee can of worms that I don't think I want to address with this PR. I'll put some examples below.

Service folder: "https://egisdata.baltimorecity.gov/egis/rest/services/BaseMaps"
- we should include /rest/services in the lookup. We should probably select the first detected type. If no service or other types are detected, this will be found. then we can use fetch_item_metadata() on it as it is a folder type
Search—first we need to support search prototype at feat: support search arcgisutils#65
- this should probably be its own S7 class with handling for pagination—i don't think we should automatically paginate everything because that could be....massive. there could be a helper to get_all_search_pages() or something though
Print method issue: enh: add print method for group and users arcgisutils#67
Date handling issue: enh: parse dates for arc_item() arcgisutils#66

# This is a "builder" url which is different than the experience itself 
# this is the view that we expect from the creator's perspective
url <- "https://experience.arcgis.com/builder/?id=cdacbe8122924255b21542cdf16b0ee4&views=page"

# Feature Service
# NOTE the underscore which specifies the layer id
hub_dataset <- "https://hub.arcgis.com/datasets/84076e6188cb441b9a5d6ee93d30eee8_2/explore"
# feature layer (see the same underscore)
hub_dataset <- "https://hub.arcgis.com/datasets/cd9dec307bd24d1bb5841fcd32b48e1f_1/explore?location=51.339690%2C-54.178767%2C5.69"
# csv dataset
hub_dataset <- "https://hub.arcgis.com/datasets/65119972a2d34f78884b30724f63f602/about"
# csv document
hub_document <- "https://hub.arcgis.com/documents/4b5524149d6e4086b94e02399a930856/about" 

# pdf document
hub_document <- "https://hub.arcgis.com/documents/a7e4a2085cc14b8ba11bf5462b6da03d/explore" 

# mapviewer urls
# refers to a webmap ID
"https://analysis-1.maps.arcgis.com/apps/mapviewer/index.html?webmap=a72330776413434b8439c762a7507388"

# refers to a url of an item
"https://www.arcgis.com/apps/mapviewer/index.html?url=https://services1.arcgis.com/WnzC35krSYGuYov4/ArcGIS/rest/services/Public_Art/FeatureServer/0&source=sd"

# specifies a single layer
"https://www.arcgis.com/apps/mapviewer/index.html?panel=gallery&layers=189e6d1c65df4e13b38c0027cee574f6"

# specifies nultiple layers which would need to be split
info <- httr2::ur_parse("https://www.arcgis.com/apps/mapviewer/index.html?panel=gallery&layers=189e6d1c65df4e13b38c0027cee574f6,310f72684ff947358a5d9dbe87a79bb7")

# We'd need to strsplit here:
info$query$layers |> strsplit(",")

# metadata page
# we need to check for `items/{id}/metadata/` for the path in arc_url_type
metadata <- "https://www.arcgis.com/sharing/rest/content/items/189e6d1c65df4e13b38c0027cee574f6/info/metadata/metadata.xml?format=default&output=html"
metadata <- "https://analysis-1.maps.arcgis.com/sharing/rest/content/items/310f72684ff947358a5d9dbe87a79bb7/info/metadata/metadata.xml?format=default&output=html"

JosiahParry · 2025-08-01T17:47:29Z

Follow up for more url type parsing & handling based on @elipousson feedback

R-ArcGIS/arcgisutils#68

rewrite arc_open to support item id's or portal urls

81ad329

update print method for image server

ea2a284

This was referenced Aug 1, 2025

feat: support search R-ArcGIS/arcgisutils#65

Open

enh: parse dates for arc_item() R-ArcGIS/arcgisutils#66

Closed

update docs, NEWS.md and handle edge cases from review

24bf44a

JosiahParry merged commit a538316 into main Aug 1, 2025
0 of 6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

rewrite arc_open to support item id's or portal urls #275

rewrite arc_open to support item id's or portal urls #275

Uh oh!

JosiahParry commented Jul 29, 2025 •

edited

Loading

Uh oh!

JosiahParry commented Jul 29, 2025

Uh oh!

JosiahParry commented Jul 29, 2025

Uh oh!

elipousson commented Jul 29, 2025

Uh oh!

elipousson commented Jul 30, 2025 •

edited

Loading

Uh oh!

JosiahParry commented Jul 30, 2025 •

edited

Loading

Uh oh!

elipousson commented Jul 30, 2025

Uh oh!

JosiahParry commented Jul 30, 2025

Uh oh!

JosiahParry commented Aug 1, 2025 •

edited

Loading

Uh oh!

Uh oh!

JosiahParry commented Aug 1, 2025

Uh oh!

Uh oh!

rewrite arc_open to support item id's or portal urls #275

rewrite arc_open to support item id's or portal urls #275

Uh oh!

Conversation

JosiahParry commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Changes

Follow up tasks

Uh oh!

JosiahParry commented Jul 29, 2025

Uh oh!

JosiahParry commented Jul 29, 2025

Uh oh!

elipousson commented Jul 29, 2025

Uh oh!

elipousson commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JosiahParry commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elipousson commented Jul 30, 2025

Uh oh!

JosiahParry commented Jul 30, 2025

Uh oh!

JosiahParry commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

JosiahParry commented Aug 1, 2025

Uh oh!

Uh oh!

JosiahParry commented Jul 29, 2025 •

edited

Loading

elipousson commented Jul 30, 2025 •

edited

Loading

JosiahParry commented Jul 30, 2025 •

edited

Loading

JosiahParry commented Aug 1, 2025 •

edited

Loading