Easier Way To Download!? #3
Replies: 5 comments 18 replies
-
|
The way I did it is this. note I have no AWS account. Download the Amazon AWS command line tool, then run It will download the datasets with no further prompting. The entire dataset is 215.5 GB as of this writing. |
Beta Was this translation helpful? Give feedback.
-
|
What is the type of downloaded file? How can I see it on the map in the easiest way? |
Beta Was this translation helpful? Give feedback.
-
|
The data is currently only available as parquet. You can follow this blog on how to convert the parquet data to a format like geojson or geopackage using DuckDB. https://til.simonwillison.net/overture-maps/overture-maps-parquet |
Beta Was this translation helpful? Give feedback.
-
|
If you are just downloading the |
Beta Was this translation helpful? Give feedback.
-
|
This can be easily done by the GeoParquet version of OvertureMap data produced by Apache Sedona. Please refer to my blog post: https://medium.com/@dr.jiayu/harnessing-overture-maps-data-apache-sedonas-journey-from-parquet-to-geoparquet-d99f7767a499 Wherobots uses Sedona to generate these data and provide them for free.Buildings: Method 1: use Sedona to download data according to the exact shape of a regionSedona supports geospatial filter pushdown on geoparquet so you can easily download the data of a country using its real shape. Example spatial SQL queries: Try it now. This query on the 120GB Building dataset will take about 3 min to finish on the docker image. The Jupyter notebook will be available at https://localhost:8888/ Method 2: Use AWS CLI to download a specific region according to GeoHash ID:All GeoParquet data provided by Wherobots are partitioned by GeoHash ID with 2 character precision. See below: You can download a specific region as follows You can find the geohash id of your target geospatial region using this online tool: https://www.movable-type.co.uk/scripts/geohash.html |
Beta Was this translation helpful? Give feedback.
-
|
@justinelliotmeyers I am not exactly sure why. Can you try apache/sedona:1.5.1? I tried both arm64 and amd64 version on my Mac Apple Silicon and they all work fine. |
Beta Was this translation helpful? Give feedback.
-
|
@Jigeeshu can you paste the full traceback? Are you navigating to the localhost url provided by the docker image after it is running? On WSL or Powershell, after running the container, I navigate to |
Beta Was this translation helpful? Give feedback.
-
|
@rbavery , Thank you for your reply. I am having issues running the docker container. When I try to run the docker container I get server error (HTTP code 500) saying the port 8080 is not available. Even if I kill the process running on 8080 it still shows already in use. It is running some java command is all I can make of it. |
Beta Was this translation helpful? Give feedback.
-
|
can you give more details about your system (OS, version)? How are you running the container? In a VM, WSL, Powershell, Linux terminal? Pasting the terminal output would also help. It could be your system has something running on port 8080 already that i conflicting with the Sedona image. |
Beta Was this translation helpful? Give feedback.
-
|
@justinelliotmeyers When run the Sedona docker image, the only really needed port of the host is 8888. Other ports are optional. Minimal command: Full command |
Beta Was this translation helpful? Give feedback.
-
|
@jiayuasu I am trying to run the docker image: but when I go to https://localhost:8888/ it says Token authentication is enabled but I don't see any token. |
Beta Was this translation helpful? Give feedback.
-
|
@d3netxer This is strange. I know a few folks tested it several times and it worked fine. Can you try (1) open it in a incognito window (2) when you start it in the terminal, can you check which address the jupyter lab is binded to, then try those addresses? In my terminal, it is as follows:
In addition, we provide images for both amd64 and arm64. I believe this serves 90% of use cases such as Intel 64-bit chips and Apple Silicon. But please double check. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot for your help @jiayuasu, I figured out that I had an unrelated jupyter notebook already running on port 8888. I stopped the other notebook and it is working fine now. |
Beta Was this translation helpful? Give feedback.
-
|
@d3netxer , @jiayuasu I'm working on simplifying the data download process in a fork of the @wherobots repository, which you can find here: https://github.com/Youssef-Harby/OvertureMaps. Once I've completed the changes, I plan to submit a pull request to the original repository at https://github.com/wherobots/OvertureMaps. To test it out, you can use the Docker Compose file included in my fork. After cloning the repo and cd to it docker compose up -d after you run the part of getting the df you can save the Sedona DataFrame as a example of save the df of building to offline parquet (it will go directly to your current project folder because it's mounted volume to the host directory): df_building.write.format("geoparquet").save("/opt/workspace/building.parquet")to convert it you can use for example : ogr2ogr -f GPKG output.gpkg building.parque |
Beta Was this translation helpful? Give feedback.
-
|
unfortunately it is not working for me @Youssef-Harby. I am getting an error when tying the Docker compose command: |
Beta Was this translation helpful? Give feedback.
-
|
@d3netxer, I am attempting to build the latest version of GDAL/OGR with the GeoParquet Driver enabled for conversion to and from GeoParquet and other extensions. I am extending the official Sedona Docker image with the following Dockerfile: FROM apache/sedona
# Update and install build tools and Python
RUN apt-get update && \
apt-get install -y build-essential cmake libboost-all-dev
# Install wget separately for debugging
RUN apt-get install -y wget
# Define the GDAL version here
ARG GDAL_VERSION=3.7.1
# Install Apache Arrow
RUN apt-get install -y -V ca-certificates lsb-release wget
RUN wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
RUN apt-get install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
RUN apt-get update
RUN apt-get install -y -V libarrow-dev libparquet-dev
WORKDIR /home
# Download GDAL
RUN wget http://download.osgeo.org/gdal/${GDAL_VERSION}/gdal-${GDAL_VERSION}.tar.gz
# Unzip and build GDAL with specific drivers enabled, then remove the tar file
RUN tar -xzvf gdal-${GDAL_VERSION}.tar.gz && \
cd gdal-${GDAL_VERSION} && \
mkdir build && \
cd build && \
cmake .. -DGDAL_USE_ARROW=ON -DGDAL_USE_PARQUET=ON && \
cmake --build . && \
cmake --build . --target install && \
cd /home && \
rm gdal-${GDAL_VERSION}.tar.gz
# Update environment variables
ENV PATH="/home/gdal-${GDAL_VERSION}/build:${PATH}"
ENV LD_LIBRARY_PATH="/home/gdal-${GDAL_VERSION}/build:${LD_LIBRARY_PATH}"
ENV GDAL_DATA="/home/gdal-${GDAL_VERSION}/data:${GDAL_DATA}"
# Update shared library cache
RUN ldconfig
# Install Python GDAL package
RUN pip3 install GDAL==3.7.1.1The official Sedona Docker image is using GDAL version 3.4.1, which does not support Arrow or Parquet/GeoParquet. I am currently encountering issues related to the Parquet build. I can confirm that both Parquet and Arrow have been built successfully. When I run the command Unable to open datasource 'building.parquet' with the following drivers.
...
-> `Parquet'
-> `Arrow'
...It could potentially be an issue related to Docker file permissions or the GDAL/OGR build process; however, I am not certain about the exact cause at this point. |
Beta Was this translation helpful? Give feedback.
-
|
@Youssef-Harby I've been working moving to parquet to postgis over the last few weeks using ogr2ogr and the notes here. The fact you can see parquet in the list of formats makes me think it's a permissions issue. Have you tried running with sudo if possible. Are you in the correct directory to access buildings.parquet? Another thing to try is running with better logging like so |
Beta Was this translation helpful? Give feedback.
-
|
@mtravis I did consider permissions issues and tried running the command with elevated permissions (i.e., sudo), but the issue persisted. I also double-checked the directory paths to make sure the building.parquet file is accessible. I'll try running the command with debug logging enabled, as you suggested, to get more information on what might be causing the problem. since you mentioned you've been working on a similar task, it would be great to collaborate |
Beta Was this translation helpful? Give feedback.
-
|
@Youssef-Harby Would you please push your update to the official Sedona dockerfile, if you eventually figure out the solution? Our Sedona community will really appreciate your contribution. The official Sedona docker image will also bring more visibility to your contribution. |
Beta Was this translation helpful? Give feedback.
-
|
@jiayuasu Once I've resolved the issue, I'll be more than happy to push the update to the official Sedona Dockerfile. I believe that having a robust and reliable Docker image will indeed benefit the community and streamline the pipeline and automation processes. I'm also excited about the possibility of collaborating on similar tasks. Your expertise and the collective knowledge of the Sedona community would undoubtedly make the process more efficient and the solution more robust. I'll keep you updated on my progress and will reach out for a review once the update is ready for the official Docker image. |
Beta Was this translation helpful? Give feedback.
-
Is there an easier way to download these? I understand the data is massive but the few ways that are setup are going to be very difficult for the masses to access.
Appreciate any other options your teams can put out for the average user
Beta Was this translation helpful? Give feedback.
All reactions