Add URL conversion functions

[elipousson]

This is a draft PR building on the code provided in this comment on arcgislayers: R-ArcGIS/arcgislayers#204 (comment)

These functions handle a fairly wide variety of URLs with decent validation. If the general approach seems reasonable, I can add tests and improve the documentation to describe what types of conversions are and are not allowed. Marking this as a draft but I'm still hoping for feedback.

[elipousson]

Any feedback on this @JosiahParry?

[JosiahParry]

Hey! None yet. You made the PR as I was beginning travel for our annual conference in san diego last week—I just got back and am finishing up a move across town. I'll be looking at this this week! Please ping me on thursday if there isn't a review yet

[JosiahParry]

I'm reviewing this PR right now and my immediate thoughts are that there is too much indirection and assumptions that we cannot make—for example assuming that layer = 0 exists (just last week was using a feature service that has only one layer with the id = 12) and that the host is always arcgis.com. We have to be able to support arcgis online, location platform, and enterprise.

I think this PR is helping me rethink just about everything! Most importantly, I think we should try and focus everything around item IDs, user IDs, and group IDs.

The objective, I think, is to standardize how each item is accessed and used.

arc_item_info <- function(item_id, token = arc_token(), host = arc_host()) {
  check_string(item_id, allow_empty = FALSE)
  res <- arc_base_req(
    host,
    path = paste0("sharing/rest/content/items/", item_id),
    token,
    query = c("f" = "json")
  ) |> 
    httr2::req_perform() |>
    httr2::resp_body_string() |> 
    RcppSimdJson::fparse()

  structure(res, class = c("PortalItem", "list"))
}


# experience builder
url <- "https://experience.arcgis.com/experience/6e360741bfd84db79d5db774a1147815"

if (arc_url_type(url) == "experience") {
  exp_id <- strsplit(arc_url_parse(url)$path, "/")[[1]][3]
  arc_item_info(exp_id)
}

# mapserver
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=1d150c40d9f642cb8bd691017bf22cee"

# feature service
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=9df5e769bfe8412b8de36a2e618c7672"

# imagery layer
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=27947c9d5d5f417b8b46a9d75a084549"

# scene layer
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=95b02a7d189d4a9f81616cd9ca5bdd15"

# tile imagery layer (considered Image Service)
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=1ed4cc16610e460ba86604c467f72f0e"

# FIXME: group layer does not have a URL
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=7bba544e0a6040ac9062480b22307b2c"
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=f1bdbdcac3c54ad8b5b21f5b17d91e8d"
 
# elevation layer (considered Image Service)
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=351b97ba2e2f45f49286e5a08df63d9f"

# table (considered feature service)
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=8786714600724851827992b3081c8a38"


# raster function template
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=cb9d7bc241c74d5db507e4953986b93d"

# webmap app
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=8d47b1f2ccf141bbab8b73f5f8acc979"

# dashboard
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=84ba9c03786e462d960e3172bc1b2204"

# feature collection
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=24aa36ce1d7747c2b5a6aa57711d03fb"

if (arc_url_type(url) == "item") {
  item_info <- arc_item_info(arc_url_parse(url)$query$id)
  
  switch(
    item_info$type,
    "Map Service" = arcgislayers::arc_open(item_info$url),
    "Feature Service" = arcgislayers::arc_open(item_info$url),
    "Image Service" = arcgislayers::arc_open(item_info$url),
    # FIXME: this is assumed as a feature server it is not
    "Scene Service" = arcgislayers::arc_open(item_info$url),
    # no url to open
    "Group Layer" = item_info,
    # no url to open
    "Raster function template" = item_info,
    "Web Mapping Application" = item_info,
    "Dashboard" = item_info,
    "Feature Collection" = item_info,
  )

}

# define basic printing method
print.PortalItem <- function(x, ...) {
  cat(sprintf("<PortalItem<%s>>\n", x$type))
  for (field in c("id", "title", "owner")) {
    cat(field, ": ", x[[field]], "\n", sep = "")
  }
  invisible(x)
}

I think maybe there should be a generic PortalItem class which would be the output of arc_item_info()

arc_item_info("84ba9c03786e462d960e3172bc1b2204")
#> <PortalItem<Dashboard>>
#> id: 84ba9c03786e462d960e3172bc1b2204
#> title: Coral Bleaching Locations
#> owner: kvangraafeiland_oceans

fserv_item <- arc_item_info("9df5e769bfe8412b8de36a2e618c7672") 
#> <PortalItem<Feature Service>>
#> id: 9df5e769bfe8412b8de36a2e618c7672
#> title: USA Major Cities
#> owner: esri_dm

# this doesnt work but would be a great idea
arc_open(fserv_item)

i think the arc_url_parse() will help us get there! i need to spend more time with this though

[JosiahParry]

Here are some test cases.

i think we need to ensure that we can parse these urls and be able to extract the item IDs and optionally the layer url for a few of them.

# mapserver
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=1d150c40d9f642cb8bd691017bf22cee"
url <- "https://image.discomap.eea.europa.eu/arcgis/rest/services/Corine/CLC2000_WM/MapServer"

# feature service
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=9df5e769bfe8412b8de36a2e618c7672"
url <- "https://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/USA_Major_Cities_/FeatureServer"

# imagery layer
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=27947c9d5d5f417b8b46a9d75a084549"
url <- "https://gis.earthdata.nasa.gov/image/rest/services/C2930761273-LARC_CLOUD/TEMPO_HCHO_L3_V03_HOURLY_VERTICAL_COLUMN/ImageServer"

# scene layer
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=95b02a7d189d4a9f81616cd9ca5bdd15"
url <- "https://tiles.arcgis.com/tiles/oPre3pOfRfefL8y0/arcgis/rest/services/3D_Buildings_Switzerland_wgs84/SceneServer"

# tile imagery layer (considered Image Service)
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=1ed4cc16610e460ba86604c467f72f0e"
url <- "https://image.arcgisonline.nl/arcgis/rest/services/KEA/Maximale_overstromingsdiepte/ImageServer"
 
# elevation layer (considered Image Service)
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=351b97ba2e2f45f49286e5a08df63d9f"
url <- "https://tiles.arcgis.com/tiles/qHLhLQrcvEnxjtPr/arcgis/rest/services/British_National_Grid_Terrain_3D/ImageServera"

# table (considered feature service)
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=8786714600724851827992b3081c8a38"

# webmap app
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=80eb92ffc89b4086abe8cedd58ab160c"
url <- "https://esri2.maps.arcgis.com/apps/instant/media/index.html?appid=80eb92ffc89b4086abe8cedd58ab160c"

# story map
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=ad791fda858c46fdbe79636aa5f35dd8"
url <- "https://storymaps.arcgis.com/stories/ad791fda858c46fdbe79636aa5f35dd8"

# instant app
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=f2dfd67d29ed4cabbb91e742e0297955"
url <- "https://actgov.maps.arcgis.com/apps/instant/interactivelegend/index.html?appid=f2dfd67d29ed4cabbb91e742e0297955"

# dashboard
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=84ba9c03786e462d960e3172bc1b2204"
url <- "https://www.arcgis.com/apps/dashboards/84ba9c03786e462d960e3172bc1b2204"

# data pipelines
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=10db3e15b22948faabf8ecdf9af30065"
url <- "https://analysis-1.maps.arcgis.com/apps/datapipelines/editor?item=10db3e15b22948faabf8ecdf9af30065"

# experience builder
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=6e360741bfd84db79d5db774a1147815"
url <- "https://experience.arcgis.com/experience/6e360741bfd84db79d5db774a1147815"

# WebApp builder
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=950b4eec577a4dc5b298a61adab41c06"
url <- "https://governmentofbc.maps.arcgis.com/apps/webappviewer/index.html?id=950b4eec577a4dc5b298a61adab41c06"

# feature collection
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=24aa36ce1d7747c2b5a6aa57711d03fb"

# deep learning package
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=97ff14a6cd3149a4ba4ae2a383db727e"

# raster function template
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=cb9d7bc241c74d5db507e4953986b93d"

# FIXME: group layer does not have a URL
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=7bba544e0a6040ac9062480b22307b2c"
url <- "https://analysis-1.maps.arcgis.com/home/item.html?id=f1bdbdcac3c54ad8b5b21f5b17d91e8d"

[elipousson]

This is great. I wasn’t entirely satisfied with this solution either so I’m happy to hear it is a generative start even if we haven’t hit on the right technical solution yet. I’ll take a closer look at these URLs and play around when I get a chance.

One consideration: with both esri2sf and arcgislayers I’ve seen a number of users who just get confused between item URLs and FeatureLayer URLs. If arc_open doesn’t handle item URLs, there would still be a lot of benefit if it could recognize them for what they are and point users towards another option like your proposed arc_item_info function.

[JosiahParry]

Yeah my thinking is that arc_open() can be a high level function that will work with an id, a portal url, or the correct feature server url.

We can make a url parser that identifies the resource type and then does the right thing.

[JosiahParry]

@elipousson i made some changes to your branch!

Notably, look at dev/url-mapping.R, I think this can be the approach we start to take for a new arc_open() experience.

[JosiahParry]

@elipousson i'm going to take this one over and merge in some changes here and make a follow up PR to arcgislayers—thank you for getting this moving

[elipousson]

Looks good to me! Thanks for picking up the ball and running with it.