Target audience
@@ -478,19 +476,20 @@Typical workflow
R withmodisfast involves the following steps :
- Defining the parameters of interest as natural R objects, -
- Login to NASA EOSDIS EarthData with the function +
- Login to NASA EOSDIS EarthData, with the function
mf_login(),
- - Building the URL(s) of the dataset(s) of interest with the function +
- Building the URL(s) of the dataset(s) of interest, with the function
mf_get_url(),
- - Downloading the dataset(s) with the function +
- Downloading the dataset(s), with the function
mf_download_data()(the preferred and default output format is the widely used NetCDF format, although other formats can be defined),
- - Importing the dataset(s) in R as a SpatRast object of
-
terralibrary (Hijmans 2024) -with the functionmf_import_data().
+ - Importing the dataset(s) in R as a
SpatRastobject of +theterralibrary (Hijmans +2024), with the functionmf_import_data().
This workflow is graphically summarized in figure .
+This workflow is graphically summarized in figure along with a toy +example.
Workflow for MODIS, VIIRS or GPM data download
@@ -518,7 +517,7 @@ Ability to import the downloaded data as a Virtual
Raster Dataset (VRT) (useful for high-volume data, e.g. country
or continental scale data at high spatial resolution) (see documentation
-of function
+of the function
Special features
mf_import_data()).mf_import_data()).
@@ -548,26 +547,27 @@ Foundational framework of
that develops them. The OPeNDAP is designed to simplify access to
structured and high-volume data, such as satellite products, over the
Web. It is a collaborative effort involving multiple institutions and
-companies, with open-source code, free software, and adherence to OGC
-standards. It is widely used by NASA, which partly finances it. A key
-feature of OPeNDAP is its capability to apply filters at the data
-download process, ensuring that only the necessary data is retrieved.
-These filters, specified within a URL, can be spatial, temporal, or
-dimensional. Nevertheless, OPeNDAP URLs are not trivial to build.
-
Foundational framework of modisfast
that develops them. The OPeNDAP is designed to simplify access to
structured and high-volume data, such as satellite products, over the
Web. It is a collaborative effort involving multiple institutions and
-companies, with open-source code, free software, and adherence to OGC
-standards. It is widely used by NASA, which partly finances it. A key
-feature of OPeNDAP is its capability to apply filters at the data
-download process, ensuring that only the necessary data is retrieved.
-These filters, specified within a URL, can be spatial, temporal, or
-dimensional. Nevertheless, OPeNDAP URLs are not trivial to build.
-modisfast constructs these URLs based on the filters that
-are specified by the user through standard R objects.
-This robust, sustainable, and cost-free foundational framework, both
-for the data provider (NASA) and the software (R, OPeNDAP, the
-tidyverse (Wickham et al.
-2019) and GDAL (GDAL/OGR
-contributors 2024) suite of packages), guarantees the long-term
-stability and reliability of the modisfast package.
A key feature of OPeNDAP is its capability to apply filters at the
+data download process, ensuring that only the necessary data is
+retrieved. These filters, specified within a URL, can be spatial,
+temporal, or dimensional. Although powerful, OPeNDAP URLs are not
+trivial to build. The main objective of modisfast is to
+construct these URLs based on the filters that are specified by the user
+through standard R objects.
Importantly, the robust, sustainable, and cost-free foundational
+framework of modisfast, both for the data provider (NASA)
+and the software (R, OPeNDAP, the tidyverse and
+GDAL suite of packages (Wickham et
+al. 2019; GDAL/OGR contributors 2024)), guarantees the long-term
+stability and reliability of the package.
Comparison with similar packages
-In addition to modisfast, there are several open-source
-R packages available for accessing MODIS or VIIRS data :
Besides modisfast, there are several open-source R
+packages available for accessing MODIS or VIIRS data :
The MODIS
package offers access to some MODIS data through global online data
archives, but it lacks comprehensive documentation and was removed from
@@ -585,9 +585,9 @@
Comparison with similar packages
programmatic interface to the ‘MODIS Land Products Subsets’ web service, providing access to 46 MODIS and VIIRS collections. This package, available on CRAN, extracts data at -specific points or buffer zones around coordinates, outputting in R -data.frame or .csv format, which is not a standard
-geospatial format. This makes it suitable for point-based data
+specific points or buffer zones around coordinates, outputting a R
+data.frame or a file in .csv format, which is not a
+standard geospatial format. This makes it suitable for point-based data
extraction but less effective for area-based queries.
The appeears
package (Hufkens and Campitelli 2023) acts
@@ -598,11 +598,11 @@
Comparison with similar packages
allows accessing data from various NASA federal archives, including MODIS and VIIRS, and enables users to subset geospatial datasets using spatial, temporal, and band/layer parameters. Indeed, as for -modisfast, the main sources of data are NASA OPeNDAP
-servers. While similar to modisfast, appeears
-offers a broader range of data sources but has a latency period (ranging
-from minutes to hours) for query processing due to server-side
-post-processing (mosaicking, reprojection, etc.).
+modisfast, AppEEARS uses OPeNDAP. While similar to
+modisfast, appeears offers a broader range of
+data sources but has a latency period (ranging from minutes to hours)
+for query processing due to server-side post-processing (mosaicking,
+reprojection, etc.).
Finally, some R packages, such as rgee (Aybar et al. 2020), rely on proprietary
software or data access protocols and are not discussed here for that
reason.
Future work
Future development of the package may include access to additional data collections from other OPeNDAP servers, and support for a variety of data formats as they become available from data providers through -their OPeNDAP servers. Furthermore, the creation of an RShiny -application (Chang et al. 2024) on top of -the package is being considered, as a means of further simplifying data -access for users with limited coding skills.
+their OPeNDAP servers. The addition of a meta-function that facilitates +executing the entire process at once (login, URL building, data +download, and data import into R) is also envisaged. Furthermore, the +creation of an RShiny application (Chang et al. +2024) on top of the package is being considered, as a means of +further simplifying data access for users with limited coding +skills.Acknowledgements
We thank NASA and its partners for making all their Earth science
-data freely available, and implementing open data access protocols such
-as OPeNDAP. modisfast heavily builds on top of the OPeNDAP,
-so we thank the non-profit OPeNDAP, Inc. for developing
+data freely available, and financing and implementing open data access
+protocols such as OPeNDAP. modisfast heavily builds on top
+of the OPeNDAP, so we thank the non-profit OPeNDAP, Inc. for developing
the eponym tool in an open and collaborative way.
We also thank the contributors that have tested the package, reviewed the documentation and brought valuable feedbacks to improve the package -: Florian de Boissieu, Julien Taconet, Annelise Tran.
+: Florian de Boissieu, Julien Taconet, Annelise Tran, Alexandre +Cebeillac.This work has been developed over the course of several research -projects :
+projects conducted by the French National Research Institute for +Sustainable Development (IRD) and its partners :-
-
- the REACT 1 project with funding from the French National Research -Institute for Sustainable Development (IRD), -
- the REACT 2 project funded by the l’Initiative - Expertise -France, +
- the REACT 1 and REACT 2 projects funded by the l’Initiative - +Expertise France,
- the ANORHYTHM project (ANR-16-CE35-008) funded by the French National Research Agency (ANR),
- the DIV-YOO project (ANR-23-CE35-0005) funded by the French National diff --git a/joss-paper/paper.md b/joss-paper/paper.md index 1c88457..714695a 100644 --- a/joss-paper/paper.md +++ b/joss-paper/paper.md @@ -28,15 +28,15 @@ affiliations: # Summary -`modisfast` is an R package that allows for easy and fast downloads of various Earth Observation (EO) data, including the Moderate Resolution Imaging Spectroradiometer (MODIS) Land products, the Visible Infrared Imaging Radiometer Suite (VIIRS) Land products, and the Global Precipitation Measurement mission (GPM) products. Based on the Open-source Project for a Network Data Access Protocol (OPeNDAP) standard framework, it enables users to apply filters (spatial, temporal, and dimensional) directly at the downloading phase, supports parallelized downloads, and streamlines data import into R. Therefore, `modisfast` offers R users a cost-effective, time-efficient, and energy-saving approach to accessing a set of key EO datasets. +`modisfast` is an R package designed for easy and fast downloads of various Earth Observation (EO) data, including the Moderate Resolution Imaging Spectroradiometer (MODIS) Land products, the Visible Infrared Imaging Radiometer Suite (VIIRS) Land products, and the Global Precipitation Measurement mission (GPM) products. It enables users to subset the data directly during the download phase using spatial, temporal, and dimensional filters and supports parallelized downloads. It also streamlines the process of importing the downloaded data into R. Overall, `modisfast` offers R users a cost-effective, time-efficient, and energy-saving approach to accessing a set of key EO datasets with R. # Statement of need -Data from Earth Observation satellites are a crucial and increasingly valuable resource for monitoring and understanding our planet, especially in the context of global change. EO data from the U.S. federal government National Aeronautics and Space Administration (NASA) are among the richest and longest-standing in the field. Iconic and widely used NASA EO data collections include the MODIS Land products [@JUSTICE20023], the VIIRS products - which continue the legacy of MODIS [@ROMAN2024113963], and the GPM mission products [@SkofronickJackson2017]. Collectively, these products have provided essential data for over 20 years, enabling the study of Earth's dynamics, including (but not limited to) global land cover, vegetation health, land surface temperature, rainfall patterns, burned areas. They support research in climate change, disaster response, biodiversity, ecosystem monitoring, ecology, epidemiology, and more [@modis-applications]. +Data from Earth Observation satellites are a crucial and increasingly valuable resource for monitoring and understanding our planet, especially in the context of global change. EO data from the U.S. federal government National Aeronautics and Space Administration (NASA) are among the richest and longest-standing in the field. Iconic, widely used and free NASA EO data collections include the MODIS Land products [@JUSTICE20023], the VIIRS products - which continue the legacy of MODIS [@ROMAN2024113963], and the GPM mission products [@SkofronickJackson2017]. Collectively, these products have provided essential data for over 20 years, enabling the study of Earth's dynamics, including global land cover, vegetation health, burned areas, land surface temperature, rainfall patterns. They support research in climate change, disaster response, biodiversity, ecosystem monitoring, ecology, public health, and more [@modis-applications]. -Despite the increasing availability and utility of EO data, accessing and using them presents several challenges. Researchers often encounter issues such as multiple data sources, data complexity, large file sizes, and the need for advanced technical skills to process the information [@AGNOLI2023122357]. These data are typically distributed as multidimensional layers over extensive areas, making accessibility and processing difficult, especially when large time series are required. This problem is particularly acute in developing regions where internet infrastructure can be limited. The complexity of accessing EO data often leads to siloed data processing workflows, separating data extraction from pre-processing and analysis, thereby hindering transparent and reproducible open science practices. While some powerful tools, such as [Google Earth Engine](https://earthengine.google.com/) [@GORELICK201718] offer some solutions to these problems, they also have important limitations, such as proprietary software. Altogether, these barriers hinder the full potential of Earth observation data to support global research and decision-making. Efforts to simplify and streamline access to these data, while maintaining an open-source and open-science framework, are essential to overcoming these obstacles and maximizing the benefits of satellite data for all. +Despite the increasing availability and utility of these EO data, accessing and using them presents several challenges. Researchers often encounter issues such as multiple data sources, data complexity, or large file sizes [@AGNOLI2023122357]. These data are typically distributed as multidimensional layers over extensive areas, making accessibility and processing difficult, especially when large time series are required. This problem is particularly acute in developing regions where internet infrastructure, needed to access the data, can be limited. The complexity of accessing EO data can result in obstacles such as underutilization or the creation of siloed data processing workflows, which separate data extraction from pre-processing and analysis, thereby hindering transparent and reproducible open science practices. While some powerful tools, such as [Google Earth Engine](https://earthengine.google.com/) [@GORELICK201718] offer some solutions to these problems, they also have important limitations, such as proprietary and energy-intensive software. Altogether, these barriers hinder the full potential of Earth observation data to support global research and decision-making. Efforts to simplify and streamline access to these data, while maintaining an open-source and open-science framework, are essential to overcoming these obstacles and maximizing the benefits of satellite data for all. -Here, we introduce `modisfast`, an open-source R package [@R] designed to simplify, streamline, and accelerate the download and import of MODIS, VIIRS, and GPM time series for R users. This package expands and the existing ecosystem of R tools for accessing MODIS data, enhancing it by introducing new features (see section \ref(comp-other-soft). `modisfast` allows users to subset these datasets using spatial, temporal, and band/layer directly at the downloading phase, optimizing data download and processing while promoting digital sobriety. Additionally, downloads can be parallelized for increased efficiency. `modisfast` thus facilitates access to EO data for R users, particularly in regions with limited internet infrastructure, and enables embedding data extraction within complex and holistic data workflows in R - fostering transparency and reproducibility in the context of Open Science. Importantly, the foundational framework of `modisfast` (see section \ref(foundational-fmwrk)) guarantees the package's long-term sustainability, open-source nature, and cost-free availability. +Here, we introduce `modisfast`, an open-source R package [@R] designed to simplify, streamline, and accelerate the download and import of MODIS, VIIRS, and GPM time series for R users. This package expands and the existing ecosystem of R tools for accessing MODIS data, enhancing it by introducing new features and data sources (see section \autoref(comp-other-soft)). `modisfast` allows users to subset these datasets using spatial, temporal, and band/layer directly at the downloading phase, optimizing data download and processing. Additionally, downloads can be parallelized for increased efficiency. `modisfast` thus facilitates access to EO data for R users, particularly in regions with limited internet infrastructure, and enables embedding data extraction within complex and holistic data workflows in R - fostering transparency and reproducibility in the context of Open Science. Importantly, the foundational framework of `modisfast` (see section \autoref(foundational-fmwrk)) guarantees the package's long-term sustainability, open-source nature, and cost-free availability. # Target audience @@ -57,12 +57,12 @@ Here, we introduce `modisfast`, an open-source R package [@R] designed to simpli The typical workflow to access and import MODIS, VIIRS or GPM data in R with `modisfast` involves the following steps : 1. Defining the parameters of interest as natural R objects, -2. Login to NASA EOSDIS EarthData with the function `mf_login()`, -3. Building the URL(s) of the dataset(s) of interest with the function `mf_get_url()`, -4. Downloading the dataset(s) with the function `mf_download_data()` (the preferred and default output format is the widely used [NetCDF format](https://www.unidata.ucar.edu/software/netcdf/), although other formats can be defined), -5. Importing the dataset(s) in R as a SpatRast object of `terra` library [@terra] with the function `mf_import_data()`. +2. Login to NASA EOSDIS EarthData, with the function `mf_login()`, +3. Building the URL(s) of the dataset(s) of interest, with the function `mf_get_url()`, +4. Downloading the dataset(s), with the function `mf_download_data()` (the preferred and default output format is the widely used [NetCDF format](https://www.unidata.ucar.edu/software/netcdf/), although other formats can be defined), +5. Importing the dataset(s) in R as a `SpatRast` object of the `terra` library [@terra], with the function `mf_import_data()`. -This workflow is graphically summarized in figure \autoref{fig:wf_modisfast}. +This workflow is graphically summarized in figure \autoref{fig:wf_modisfast} along with a toy example. {width="90%"} @@ -77,7 +77,7 @@ To further simplify and fasten data download and import in R, `modisfast` offers - Ability to download data for multiple regions of interest simultaneously (see [dedicated vignette](https://ptaconet.github.io/modisfast/articles/modisfast2.html)); - Ability to download data for multiple time periods of interest simultaneously (see [dedicated vignette](https://ptaconet.github.io/modisfast/articles/modisfast2.html)) ; -- Ability to import the downloaded data as a [*Virtual Raster Dataset* (VRT)](https://search.r-project.org/CRAN/refmans/terra/html/vrt.html) (useful for high-volume data, e.g. country or continental scale data at high spatial resolution) (see documentation of function `mf_import_data()`). +- Ability to import the downloaded data as a [*Virtual Raster Dataset* (VRT)](https://search.r-project.org/CRAN/refmans/terra/html/vrt.html) (useful for high-volume data, e.g. country or continental scale data at high spatial resolution) (see documentation of the function `mf_import_data()`). ## Available data collections @@ -91,21 +91,23 @@ The full list of available data collections can be accessed with the function `m ## Foundational framework of `modisfast` {#foundational-fmwrk} -Technically, `modisfast` is a programmatic interface to several NASA OPeNDAP (https://www.opendap.org/) servers. OPeNDAP is the acronym for "Open-source Project for a Network Data Access Protocol" and designates both the software, the access protocol, and the corporation that develops them. The OPeNDAP is designed to simplify access to structured and high-volume data, such as satellite products, over the Web. It is a collaborative effort involving multiple institutions and companies, with open-source code, free software, and adherence to OGC standards. It is widely used by NASA, which partly finances it. A key feature of OPeNDAP is its capability to apply filters at the data download process, ensuring that only the necessary data is retrieved. These filters, specified within a URL, can be spatial, temporal, or dimensional. Nevertheless, OPeNDAP URLs are not trivial to build. `modisfast` constructs these URLs based on the filters that are specified by the user through standard R objects. +Technically, `modisfast` is a programmatic interface to several NASA OPeNDAP (https://www.opendap.org/) servers. OPeNDAP is the acronym for "Open-source Project for a Network Data Access Protocol" and designates both the software, the access protocol, and the corporation that develops them. The OPeNDAP is designed to simplify access to structured and high-volume data, such as satellite products, over the Web. It is a collaborative effort involving multiple institutions and companies, with open-source code, free software, and adherence to the [Open Geospatial Consortium](https://www.ogc.org/) (OGC) standards. It is widely used by NASA, which partly finances it. -This robust, sustainable, and cost-free foundational framework, both for the data provider (NASA) and the software (R, OPeNDAP, the `tidyverse` [@tidyverse] and `GDAL` [@gdal] suite of packages), guarantees the long-term stability and reliability of the `modisfast` package. +A key feature of OPeNDAP is its capability to apply filters at the data download process, ensuring that only the necessary data is retrieved. These filters, specified within a URL, can be spatial, temporal, or dimensional. Although powerful, OPeNDAP URLs are not trivial to build. The main objective of `modisfast` is to construct these URLs based on the filters that are specified by the user through standard R objects. + +Importantly, the robust, sustainable, and cost-free foundational framework of `modisfast`, both for the data provider (NASA) and the software (R, OPeNDAP, the `tidyverse` and `GDAL` suite of packages [@tidyverse; @gdal]), guarantees the long-term stability and reliability of the package. # Comparison with similar packages {#comp-other-soft} -In addition to `modisfast`, there are several open-source R packages available for accessing MODIS or VIIRS data : +Besides `modisfast`, there are several open-source R packages available for accessing MODIS or VIIRS data : The [`MODIS`](https://github.com/fdetsch/MODIS) package offers access to some MODIS data through global online data archives, but it lacks comprehensive documentation and was removed from the CRAN repository in 2023 for policy violations. Furthermore, some of its dependencies are not available anymore on CRAN. The [`MODIStsp`](https://github.com/ropensci/MODIStsp) package [@MODIStsp] provides both a command-line and a user-friendly graphical interface to extract MODIS data in standard TIFF or original HDF formats. However, it does not allow to extract data at a sub-tile level (i.e. spatial subsetting capabilities are limited), and it was also removed from the CRAN repository in 2023 at the maintainer's request. -The [`MODIStools`](https://github.com/ropensci/MODISTools) package [@modistools] serves as a programmatic interface to the ['MODIS Land Products Subsets' web service](https://modis.ornl.gov/data/modis_webservice.html), providing access to 46 MODIS and VIIRS collections. This package, available on CRAN, extracts data at specific points or buffer zones around coordinates, outputting in R `data.frame` or .csv format, which is not a standard geospatial format. This makes it suitable for point-based data extraction but less effective for area-based queries. +The [`MODIStools`](https://github.com/ropensci/MODISTools) package [@modistools] serves as a programmatic interface to the ['MODIS Land Products Subsets' web service](https://modis.ornl.gov/data/modis_webservice.html), providing access to 46 MODIS and VIIRS collections. This package, available on CRAN, extracts data at specific points or buffer zones around coordinates, outputting a R `data.frame` or a file in .csv format, which is not a standard geospatial format. This makes it suitable for point-based data extraction but less effective for area-based queries. -The [`appeears`](https://github.com/bluegreen-labs/appeears) package [@appeears] acts as a programmatic interface to the [NASA AppEEARS API](https://appeears.earthdatacloud.nasa.gov/) services. AppEEARS is a NASA-built application that offers a simple and efficient way to access and transform geospatial data from a variety of federal data archives (including MODIS and VIIRS, but not GPM). AppEEARS allows accessing data from various NASA federal archives, including MODIS and VIIRS, and enables users to subset geospatial datasets using spatial, temporal, and band/layer parameters. Indeed, as for `modisfast`, the main sources of data are NASA OPeNDAP servers. While similar to `modisfast`, `appeears` offers a broader range of data sources but has a latency period (ranging from minutes to hours) for query processing due to server-side post-processing (mosaicking, reprojection, etc.). +The [`appeears`](https://github.com/bluegreen-labs/appeears) package [@appeears] acts as a programmatic interface to the [NASA AppEEARS API](https://appeears.earthdatacloud.nasa.gov/) services. AppEEARS is a NASA-built application that offers a simple and efficient way to access and transform geospatial data from a variety of federal data archives (including MODIS and VIIRS, but not GPM). AppEEARS allows accessing data from various NASA federal archives, including MODIS and VIIRS, and enables users to subset geospatial datasets using spatial, temporal, and band/layer parameters. Indeed, as for `modisfast`, AppEEARS uses OPeNDAP. While similar to `modisfast`, `appeears` offers a broader range of data sources but has a latency period (ranging from minutes to hours) for query processing due to server-side post-processing (mosaicking, reprojection, etc.). Finally, some R packages, such as [`rgee`](https://github.com/r-spatial/rgee) [@rgee], rely on proprietary software or data access protocols and are not discussed here for that reason. @@ -196,19 +198,18 @@ plot(r, col = rev(terrain.colors(20))) # Future work -Future development of the package may include access to additional data collections from other OPeNDAP servers, and support for a variety of data formats as they become available from data providers through their OPeNDAP servers. Furthermore, the creation of an RShiny application [@shiny] on top of the package is being considered, as a means of further simplifying data access for users with limited coding skills. +Future development of the package may include access to additional data collections from other OPeNDAP servers, and support for a variety of data formats as they become available from data providers through their OPeNDAP servers. The addition of a meta-function that facilitates executing the entire process at once (login, URL building, data download, and data import into R) is also envisaged. Furthermore, the creation of an RShiny application [@shiny] on top of the package is being considered, as a means of further simplifying data access for users with limited coding skills. # Acknowledgements -We thank NASA and its partners for making all their Earth science data freely available, and implementing open data access protocols such as OPeNDAP. `modisfast` heavily builds on top of the OPeNDAP, so we thank the non-profit [OPeNDAP, Inc.](https://www.opendap.org/about/) for developing the eponym tool in an open and collaborative way. +We thank NASA and its partners for making all their Earth science data freely available, and financing and implementing open data access protocols such as OPeNDAP. `modisfast` heavily builds on top of the OPeNDAP, so we thank the non-profit [OPeNDAP, Inc.](https://www.opendap.org/about/) for developing the eponym tool in an open and collaborative way. -We also thank the contributors that have tested the package, reviewed the documentation and brought valuable feedbacks to improve the package : Florian de Boissieu, Julien Taconet, Annelise Tran. +We also thank the contributors that have tested the package, reviewed the documentation and brought valuable feedbacks to improve the package : Florian de Boissieu, Julien Taconet, Annelise Tran, Alexandre Cebeillac. -This work has been developed over the course of several research projects : +This work has been developed over the course of several research projects conducted by the French National Research Institute for Sustainable Development (IRD) and its partners : -- the REACT 1 project with funding from the French National Research Institute for Sustainable Development (IRD), -- the REACT 2 project funded by the *l'Initiative - Expertise France*, -- the ANORHYTHM project (ANR-16-CE35-008) funded by the French National Research Agency (ANR), +- the REACT 1 and REACT 2 projects funded by the *l'Initiative - Expertise France*. +- the ANORHYTHM project (ANR-16-CE35-008) funded by the French National Research Agency (ANR). - the DIV-YOO project (ANR-23-CE35-0005) funded by the French National Research Agency (ANR). # References