minor cleanup to remove file list and finish last sentence

_freeze/posts/2024-09-21-aws-cors-cog-leaflet/2024-09-21-aws-cors-cog-leaflet/execute-results/html.json

@@ -1,8 +1,8 @@
 {
-  "hash": "91676bf0ad8ea5cc25cbc69ce8b0484d",
+  "hash": "6ff6e84d40c75c15b05eb90e09144ece",
   "result": {
     "engine": "knitr",
-    "markdown": "---\ntitle: \"CORS for serving of COGs of UAV imagery on AWS with R\"\nauthor: \"al\"\ndate: \"2024-09-21\"\ndate-modified: \"2024-10-21\"\ncategories: [aws, s3, r, paws, s3sf, leaflet, leafem, COG, CORS]\nimage: \"image.jpg\"\nparams:\n  repo_owner: \"NewGraphEnvironment\"\n  repo_name: \"new_graphiti\"\n  post_dir_name: \"aws-storage-permissions\"\nformat: \n  html:\n    code-fold: true\n---\n\n\nWhoa Billy.  Time to host our UAV imagery on AWS and serve it out through leaflet and do dank moves like put before\nafter images on slippy maps next to each other for world peace. Ha. Well that is a bit dramatic but hey. Still pretty\ncool.  \n\nFirst thing is to convert the image to a cog and sync it up to a bucket. Not doing that here.  Will do soon though. What we do here is - after we are sure there are public permissions allowed to the bucket but we also need to deal with big bad CORS.  We can set a viewing [Cross-origin resource sharing (CORS)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html?icmpid=docs_amazons3_console). \"CORS defines a way for client web applications that are loaded in one domain to interact with resources in a different domain\". This is done with the following JSON.\n\n```\n[\n    {\n        \"AllowedHeaders\": [\n            \"*\"\n        ],\n        \"AllowedMethods\": [\n            \"GET\"\n        ],\n        \"AllowedOrigins\": [\n            \"*\"\n        ],\n        \"ExposeHeaders\": [\n            \"x-amz-server-side-encryption\",\n            \"x-amz-request-id\",\n            \"x-amz-id-2\"\n        ],\n        \"MaxAgeSeconds\": 3000\n    }\n]\n```\n\nWe went the daft way here and just copied and pasted this into the `CORS` section of our bucket in the console but we\nshould be able to use the `paws` package and the `s3_put_bucket_cors` function (I would think).  We are going to leave \nthat for another day.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(paws)\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\nWarning: package 'paws' was built under R version 4.4.1\n```\n\n\n:::\n\n```{.r .cell-code}\nlibrary(s3fs)\nlibrary(leaflet)\nlibrary(leafem)\n```\n:::\n\n\nList your buckets kid.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ns3 <- paws::s3()\nbuckets <- s3$list_buckets()\npurrr::map_chr(buckets$Buckets, \"Name\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] \"23cog\"        \"new-graphiti\"\n```\n\n\n:::\n:::\n\n\n\nNow list em with s3fs and show the file structure kid.\n\n::: {.cell}\n\n```{.r .cell-code}\ns3fs::s3_dir_ls(refresh = TRUE) \n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] \"s3://23cog\"        \"s3://new-graphiti\"\n```\n\n\n:::\n\n```{.r .cell-code}\nbucket_path <- s3fs::s3_path(\"23cog\")\ns3fs::s3_dir_tree(bucket_path)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\ns3://23cog\n├── .DS_Store\n├── 20210906lampreymoricetribv220230317-DEM.tif\n├── 20210906lampreymoricetribv220230317.las\n├── 20210906lampreymoricetribv220230317.tif\n├── FHAP_template.csv\n├── bc_093l026_xli1m_utm09_2018.tif\n├── bc_093l048_xli1m_utm09_2019.tif\n├── bc_093l056_xli1m_utm09_2019_cog.tif\n├── bc_093l058_xli1m_utm09_2019.tif\n├── bc_093m032_xli1m_utm09_2019_cog2.tif\n├── ept.json\n├── glen_valle_lidar_2019_cog.tif\n├── xli1m_utm09_2019_093L04811.tif\n├── xli1m_utm09_2019_093L04812.tif\n├── xli1m_utm09_2019_093L04813.tif\n├── xli1m_utm09_2019_093L04814.tif\n├── xli1m_utm09_2019_093L04821.tif\n├── xli1m_utm09_2019_093L04822.tif\n├── xli1m_utm09_2019_093L04823.tif\n├── xli1m_utm09_2019_093L04824.tif\n├── xli1m_utm09_2019_093L04831.tif\n├── xli1m_utm09_2019_093L04832.tif\n├── xli1m_utm09_2019_093L04833.tif\n├── xli1m_utm09_2019_093L04834.tif\n├── xli1m_utm09_2019_093L04841.tif\n├── xli1m_utm09_2019_093L04842.tif\n├── xli1m_utm09_2019_093L04843.tif\n├── xli1m_utm09_2019_093L04844.tif\n└── private\n    └── FHAP_template.xlsx\n```\n\n\n:::\n:::\n\n\nList all the darned full paths please\n\n::: {.cell}\n\n```{.r .cell-code}\ns3_dir_ls(bucket_path)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n [1] \"s3://23cog/.DS_Store\"                                  \n [2] \"s3://23cog/20210906lampreymoricetribv220230317-DEM.tif\"\n [3] \"s3://23cog/20210906lampreymoricetribv220230317.las\"    \n [4] \"s3://23cog/20210906lampreymoricetribv220230317.tif\"    \n [5] \"s3://23cog/FHAP_template.csv\"                          \n [6] \"s3://23cog/bc_093l026_xli1m_utm09_2018.tif\"            \n [7] \"s3://23cog/bc_093l048_xli1m_utm09_2019.tif\"            \n [8] \"s3://23cog/bc_093l056_xli1m_utm09_2019_cog.tif\"        \n [9] \"s3://23cog/bc_093l058_xli1m_utm09_2019.tif\"            \n[10] \"s3://23cog/bc_093m032_xli1m_utm09_2019_cog2.tif\"       \n[11] \"s3://23cog/ept.json\"                                   \n[12] \"s3://23cog/glen_valle_lidar_2019_cog.tif\"              \n[13] \"s3://23cog/xli1m_utm09_2019_093L04811.tif\"             \n[14] \"s3://23cog/xli1m_utm09_2019_093L04812.tif\"             \n[15] \"s3://23cog/xli1m_utm09_2019_093L04813.tif\"             \n[16] \"s3://23cog/xli1m_utm09_2019_093L04814.tif\"             \n[17] \"s3://23cog/xli1m_utm09_2019_093L04821.tif\"             \n[18] \"s3://23cog/xli1m_utm09_2019_093L04822.tif\"             \n[19] \"s3://23cog/xli1m_utm09_2019_093L04823.tif\"             \n[20] \"s3://23cog/xli1m_utm09_2019_093L04824.tif\"             \n[21] \"s3://23cog/xli1m_utm09_2019_093L04831.tif\"             \n[22] \"s3://23cog/xli1m_utm09_2019_093L04832.tif\"             \n[23] \"s3://23cog/xli1m_utm09_2019_093L04833.tif\"             \n[24] \"s3://23cog/xli1m_utm09_2019_093L04834.tif\"             \n[25] \"s3://23cog/xli1m_utm09_2019_093L04841.tif\"             \n[26] \"s3://23cog/xli1m_utm09_2019_093L04842.tif\"             \n[27] \"s3://23cog/xli1m_utm09_2019_093L04843.tif\"             \n[28] \"s3://23cog/xli1m_utm09_2019_093L04844.tif\"             \n[29] \"s3://23cog/private\"                                    \n```\n\n\n:::\n:::\n\n\nBiuld a functshi to give us the actual https url. Sure there is a function somewhere already to do this but couldn't find it.\n\n::: {.cell}\n\n```{.r .cell-code}\n# Define your S3 path\ns3_path <-  \"s3://23cog/20210906lampreymoricetribv220230317.tif\"\n\ns3_path_to_https <- function(s3_path) {\n  # Remove the 's3://' prefix\n  path_without_prefix <- sub(\"^s3://\", \"\", s3_path)\n  \n  # Split the path into bucket and key\n  parts <- strsplit(path_without_prefix, \"/\", fixed = TRUE)[[1]]\n  bucket_name <- parts[1]\n  object_key <- paste(parts[-1], collapse = \"/\")\n  \n  # Construct the HTTPS URL\n  https_url <- sprintf(\"https://%s.s3.amazonaws.com/%s\", bucket_name, object_key)\n  return(https_url)\n}\n\nurl <- s3_path_to_https(s3_path)\nprint(url)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] \"https://23cog.s3.amazonaws.com/20210906lampreymoricetribv220230317.tif\"\n```\n\n\n:::\n:::\n\n\n\nSince we already have some valid COGs up on AWS we will link to one to be sure it works. \n\n\n::: {.cell}\n\n```{.r .cell-code}\n  leaflet::leaflet() |>\n    leaflet::addTiles() |>\n    leafem:::addCOG(\n      url = url\n      , group = \"COG\"\n      , resolution = 512\n      , autozoom = TRUE\n    )\n```\n\n::: {.cell-output-display}\n\n```{=html}\n<div class=\"leaflet html-widget html-fill-item\" id=\"htmlwidget-918eae40fe22d1ce18cb\" style=\"width:100%;height:464px;\"></div>\n<script type=\"application/json\" data-for=\"htmlwidget-918eae40fe22d1ce18cb\">{\"x\":{\"options\":{\"crs\":{\"crsClass\":\"L.CRS.EPSG3857\",\"code\":null,\"proj4def\":null,\"projectedBounds\":null,\"options\":{}}},\"calls\":[{\"method\":\"addTiles\",\"args\":[\"https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png\",null,null,{\"minZoom\":0,\"maxZoom\":18,\"tileSize\":256,\"subdomains\":\"abc\",\"errorTileUrl\":\"\",\"tms\":false,\"noWrap\":false,\"zoomOffset\":0,\"zoomReverse\":false,\"opacity\":1,\"zIndex\":1,\"detectRetina\":false,\"attribution\":\"&copy; <a href=\\\"https://openstreetmap.org/copyright/\\\">OpenStreetMap<\\/a>,  <a href=\\\"https://opendatacommons.org/licenses/odbl/\\\">ODbL<\\/a>\"}]},{\"method\":\"addCOG\",\"args\":[\"https://23cog.s3.amazonaws.com/20210906lampreymoricetribv220230317.tif\",\"COG\",null,512,0.8,{\"minZoom\":0,\"maxZoom\":18,\"tileSize\":256,\"subdomains\":\"abc\",\"errorTileUrl\":\"\",\"tms\":false,\"noWrap\":false,\"zoomOffset\":0,\"zoomReverse\":false,\"opacity\":1,\"zIndex\":1,\"detectRetina\":false},null,null,true,false]}]},\"evals\":[],\"jsHooks\":[]}</script>\n```\n\n:::\n:::\n\n\nDope.\n\n\nWhen we work with the `paws` package - when we want to get help we use `?s3` and navigate in from there. This next section doesn't work yet so we turn eval = F and get on with our lives.  Would like to activate this policy from\n\n::: {.cell}\n\n```{.r .cell-code}\nmy_bucket_name = \"23cog\" \n\n# Define the CORS policy as a list\nmy_policy <- list(\n    list(\n        AllowedHeaders = list(\"*\"),  # Must be a list\n        AllowedMethods = list(\"GET\"),  # Must be a list\n        AllowedOrigins = list(\"*\"),  # Must be a list\n        ExposeHeaders = list(\n            \"x-amz-server-side-encryption\",\n            \"x-amz-request-id\",\n            \"x-amz-id-2\"\n        ),  # Must be a list\n        MaxAgeSeconds = 3000  # Must be a number\n    )\n)\n\n\n# Convert the policy list to a pretty JSON format\nmy_policy_json <- jsonlite::toJSON(my_policy, pretty = TRUE, auto_unbox = TRUE)\n\n\n# Set the CORS configuration directly using the list\npaws::s3()$put_bucket_cors(\n    Bucket = my_bucket_name,\n    CORSConfiguration = my_policy_json  # Pass the list of rules directly\n)\n```\n:::\n",
+    "markdown": "---\ntitle: \"CORS for serving of COGs of UAV imagery on AWS with R\"\nauthor: \"al\"\ndate: \"2024-09-21\"\ndate-modified: \"2024-09-21\"\ncategories: [aws, s3, r, paws, s3sf, leaflet, leafem, COG, CORS]\nimage: \"image.jpg\"\nparams:\n  repo_owner: \"NewGraphEnvironment\"\n  repo_name: \"new_graphiti\"\n  post_dir_name: \"aws-storage-permissions\"\nformat: \n  html:\n    code-fold: true\n---\n\n\nWhoa Billy.  Time to host our UAV imagery on AWS and serve it out through leaflet and do dank moves like put before\nafter images on slippy maps next to each other for world peace. Ha. Well that is a bit dramatic but hey. Still pretty\ncool.  \n\nFirst thing is to convert the image to a cog and sync it up to a bucket. Not doing that here.  Will do soon though. What we do here is - after we are sure there are public permissions allowed to the bucket but we also need to deal with big bad CORS.  We can set a viewing [Cross-origin resource sharing (CORS)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html?icmpid=docs_amazons3_console). \"CORS defines a way for client web applications that are loaded in one domain to interact with resources in a different domain\". This is done with the following JSON.\n\n```\n[\n    {\n        \"AllowedHeaders\": [\n            \"*\"\n        ],\n        \"AllowedMethods\": [\n            \"GET\"\n        ],\n        \"AllowedOrigins\": [\n            \"*\"\n        ],\n        \"ExposeHeaders\": [\n            \"x-amz-server-side-encryption\",\n            \"x-amz-request-id\",\n            \"x-amz-id-2\"\n        ],\n        \"MaxAgeSeconds\": 3000\n    }\n]\n```\n\nWe went the daft way here and just copied and pasted this into the `CORS` section of our bucket in the console but we\nshould be able to use the `paws` package and the `s3_put_bucket_cors` function (I would think).  We are going to leave \nthat for another day.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(paws)\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\nWarning: package 'paws' was built under R version 4.4.1\n```\n\n\n:::\n\n```{.r .cell-code}\nlibrary(s3fs)\nlibrary(leaflet)\nlibrary(leafem)\n```\n:::\n\n\nList your buckets kid.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ns3 <- paws::s3()\nbuckets <- s3$list_buckets()\npurrr::map_chr(buckets$Buckets, \"Name\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] \"23cog\"        \"new-graphiti\"\n```\n\n\n:::\n:::\n\n::: {.cell}\n\n```{.r .cell-code}\ns3fs::s3_dir_ls(refresh = TRUE) \n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] \"s3://23cog\"        \"s3://new-graphiti\"\n```\n\n\n:::\n\n```{.r .cell-code}\nbucket_path <- s3fs::s3_path(\"23cog\")\n```\n:::\n\n::: {.cell}\n\n```{.r .cell-code}\n# too much info\ns3fs::s3_dir_tree(bucket_path)\n```\n:::\n\n\nNow list em with full paths\n\n::: {.cell}\n\n```{.r .cell-code}\n# full paths\ns3_dir_ls(bucket_path)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n [1] \"s3://23cog/.DS_Store\"                                  \n [2] \"s3://23cog/20210906lampreymoricetribv220230317-DEM.tif\"\n [3] \"s3://23cog/20210906lampreymoricetribv220230317.las\"    \n [4] \"s3://23cog/20210906lampreymoricetribv220230317.tif\"    \n [5] \"s3://23cog/FHAP_template.csv\"                          \n [6] \"s3://23cog/bc_093l026_xli1m_utm09_2018.tif\"            \n [7] \"s3://23cog/bc_093l048_xli1m_utm09_2019.tif\"            \n [8] \"s3://23cog/bc_093l056_xli1m_utm09_2019_cog.tif\"        \n [9] \"s3://23cog/bc_093l058_xli1m_utm09_2019.tif\"            \n[10] \"s3://23cog/bc_093m032_xli1m_utm09_2019_cog2.tif\"       \n[11] \"s3://23cog/ept.json\"                                   \n[12] \"s3://23cog/glen_valle_lidar_2019_cog.tif\"              \n[13] \"s3://23cog/xli1m_utm09_2019_093L04811.tif\"             \n[14] \"s3://23cog/xli1m_utm09_2019_093L04812.tif\"             \n[15] \"s3://23cog/xli1m_utm09_2019_093L04813.tif\"             \n[16] \"s3://23cog/xli1m_utm09_2019_093L04814.tif\"             \n[17] \"s3://23cog/xli1m_utm09_2019_093L04821.tif\"             \n[18] \"s3://23cog/xli1m_utm09_2019_093L04822.tif\"             \n[19] \"s3://23cog/xli1m_utm09_2019_093L04823.tif\"             \n[20] \"s3://23cog/xli1m_utm09_2019_093L04824.tif\"             \n[21] \"s3://23cog/xli1m_utm09_2019_093L04831.tif\"             \n[22] \"s3://23cog/xli1m_utm09_2019_093L04832.tif\"             \n[23] \"s3://23cog/xli1m_utm09_2019_093L04833.tif\"             \n[24] \"s3://23cog/xli1m_utm09_2019_093L04834.tif\"             \n[25] \"s3://23cog/xli1m_utm09_2019_093L04841.tif\"             \n[26] \"s3://23cog/xli1m_utm09_2019_093L04842.tif\"             \n[27] \"s3://23cog/xli1m_utm09_2019_093L04843.tif\"             \n[28] \"s3://23cog/xli1m_utm09_2019_093L04844.tif\"             \n[29] \"s3://23cog/private\"                                    \n```\n\n\n:::\n:::\n\n\nBiuld a functshi to give us the actual https url. Sure there is a function somewhere already to do this but couldn't find it.\n\n::: {.cell}\n\n```{.r .cell-code}\n# Define your S3 path\ns3_path <-  \"s3://23cog/20210906lampreymoricetribv220230317.tif\"\n\ns3_path_to_https <- function(s3_path) {\n  # Remove the 's3://' prefix\n  path_without_prefix <- sub(\"^s3://\", \"\", s3_path)\n  \n  # Split the path into bucket and key\n  parts <- strsplit(path_without_prefix, \"/\", fixed = TRUE)[[1]]\n  bucket_name <- parts[1]\n  object_key <- paste(parts[-1], collapse = \"/\")\n  \n  # Construct the HTTPS URL\n  https_url <- sprintf(\"https://%s.s3.amazonaws.com/%s\", bucket_name, object_key)\n  return(https_url)\n}\n\nurl <- s3_path_to_https(s3_path)\nprint(url)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] \"https://23cog.s3.amazonaws.com/20210906lampreymoricetribv220230317.tif\"\n```\n\n\n:::\n:::\n\n\n\nSince we already have some valid COGs up on AWS we will link to one to be sure it works. \n\n\n::: {.cell}\n\n```{.r .cell-code}\n  leaflet::leaflet() |>\n    leaflet::addTiles() |>\n    leafem:::addCOG(\n      url = url\n      , group = \"COG\"\n      , resolution = 512\n      , autozoom = TRUE\n    )\n```\n\n::: {.cell-output-display}\n\n```{=html}\n<div class=\"leaflet html-widget html-fill-item\" id=\"htmlwidget-50c660fc0a1752b47581\" style=\"width:100%;height:464px;\"></div>\n<script type=\"application/json\" data-for=\"htmlwidget-50c660fc0a1752b47581\">{\"x\":{\"options\":{\"crs\":{\"crsClass\":\"L.CRS.EPSG3857\",\"code\":null,\"proj4def\":null,\"projectedBounds\":null,\"options\":{}}},\"calls\":[{\"method\":\"addTiles\",\"args\":[\"https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png\",null,null,{\"minZoom\":0,\"maxZoom\":18,\"tileSize\":256,\"subdomains\":\"abc\",\"errorTileUrl\":\"\",\"tms\":false,\"noWrap\":false,\"zoomOffset\":0,\"zoomReverse\":false,\"opacity\":1,\"zIndex\":1,\"detectRetina\":false,\"attribution\":\"&copy; <a href=\\\"https://openstreetmap.org/copyright/\\\">OpenStreetMap<\\/a>,  <a href=\\\"https://opendatacommons.org/licenses/odbl/\\\">ODbL<\\/a>\"}]},{\"method\":\"addCOG\",\"args\":[\"https://23cog.s3.amazonaws.com/20210906lampreymoricetribv220230317.tif\",\"COG\",null,512,0.8,{\"minZoom\":0,\"maxZoom\":18,\"tileSize\":256,\"subdomains\":\"abc\",\"errorTileUrl\":\"\",\"tms\":false,\"noWrap\":false,\"zoomOffset\":0,\"zoomReverse\":false,\"opacity\":1,\"zIndex\":1,\"detectRetina\":false},null,null,true,false]}]},\"evals\":[],\"jsHooks\":[]}</script>\n```\n\n:::\n:::\n\n\nDope.\n\n\nWhen we work with the `paws` package - when we want to get help we use `?s3` and navigate in from there. This next section doesn't work yet so we turn eval = F and get on with our lives.  Would like to activate this policy from R but can't seem to pull it of yet. To be continued - maybe.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmy_bucket_name = \"23cog\" \n\n# Define the CORS policy as a list\nmy_policy <- list(\n    list(\n        AllowedHeaders = list(\"*\"),  # Must be a list\n        AllowedMethods = list(\"GET\"),  # Must be a list\n        AllowedOrigins = list(\"*\"),  # Must be a list\n        ExposeHeaders = list(\n            \"x-amz-server-side-encryption\",\n            \"x-amz-request-id\",\n            \"x-amz-id-2\"\n        ),  # Must be a list\n        MaxAgeSeconds = 3000  # Must be a number\n    )\n)\n\n\n# Convert the policy list to a pretty JSON format\nmy_policy_json <- jsonlite::toJSON(my_policy, pretty = TRUE, auto_unbox = TRUE)\n\n\n# Set the CORS configuration directly using the list\npaws::s3()$put_bucket_cors(\n    Bucket = my_bucket_name,\n    CORSConfiguration = my_policy_json  # Pass the list of rules directly\n)\n```\n:::\n",
     "supporting": [],
     "filters": [
       "rmarkdown/pagebreak.lua"

posts/2024-09-21-aws-cors-cog-leaflet/2024-09-21-aws-cors-cog-leaflet.qmd

@@ -63,15 +63,20 @@ purrr::map_chr(buckets$Buckets, "Name")
 ```
 
 
-Now list em with s3fs and show the file structure kid.
 ```{r}
 s3fs::s3_dir_ls(refresh = TRUE) 
 bucket_path <- s3fs::s3_path("23cog")
+
+```
+
+```{r eval= F}
+# too much info
 s3fs::s3_dir_tree(bucket_path)
 ```
 
-List all the darned full paths please
+Now list em with full paths
 ```{r}
+# full paths
 s3_dir_ls(bucket_path)
 ```