Changing map routing server will affect the reward

[xhluca]

As discussed in #3, the tile server from CMU is down (it appears that the routing is still functioning, not sure about geocoding).

Apart from self-hosting (challenging and takes a lot of space), one approach is to use the official server. See this for more information on how to do that: web-arena-x/webarena#190

If you switch the routing URL to the official routing server yield different results. For example,

Official routing (reward=0)	Old routing (reward=1)

Note the difference: in the official routing (routing.openstreetmap.de), the route takes 10min, 5.2km and 10 steps. In the old routing (from cmu.edu's hosted routing server), it takes 9min, 5.1km, and 8 steps.

Here's the official answer:

  {
    "sites": [
      "map"
    ],
    "task_id": 155,
    "require_login": true,
    "storage_state": null,
    "start_url": "__MAP__",
    "geolocation": null,
    "intent_template": "What is the minimum travel time by car from {{location1}} to {{location2}}?",
    "instantiation_dict": {
      "location1": "Animal Rescue League of Pittsburgh",
      "location2": "Schenley park"
    },
    "intent": "What is the minimum travel time by car from Animal Rescue League of Pittsburgh to Schenley park?",
    "require_reset": false,
    "eval": {
      "eval_types": [
        "string_match"
      ],
      "reference_answers": {
        "fuzzy_match": [
          "9min"
        ]
      },
      "reference_url": "",
      "program_html": [],
      "string_note": "",
      "reference_answer_raw_annotation": "9min"
    },
    "intent_template_id": 36
  },

So it's something to consider before switching map server

[xhluca]

I would suggest changing the following:

webarena-setup/webarena/03_docker_create_containers.sh

Lines 16 to 51 in 55c9356

	# openstreetmap docker set up
	cd openstreetmap-website/
	# tile server URL (use default openstreetmap server)
	OSM_TILE_SERVER_URL="https://tile.openstreetmap.org/{z}/{x}/{y}.png"
	# geocoding server URL (use default openstreetmap server)
	OSM_GEOCODING_SERVER_URL="https://nominatim.openstreetmap.org/"
	# routing server URLs (use default openstreetmap server)
	OSM_ROUTING_SERVER_URL="https://routing.openstreetmap.de"
	OSM_CAR_SUFFIX="/routed-car"
	OSM_BIKE_SUFFIX="/routed-bike"
	OSM_FOOT_SUFFIX="/routed-foot"
	# original WebArena config (CMU server with different ports for each vehicule type)
	# OSM_ROUTING_SERVER_URL="http://metis.lti.cs.cmu.edu"
	# OSM_CAR_SUFFIX=":5000"
	# OSM_BIKE_SUFFIX=":5001"
	# OSM_FOOT_SUFFIX=":5002"
	# copy template files to be set up
	cp ../openstreetmap-templates/docker-compose.yml ./docker-compose.yml
	cp ../openstreetmap-templates/leaflet.osm.js ./vendor/assets/leaflet/leaflet.osm.js
	cp ../openstreetmap-templates/fossgis_osrm.js ./app/assets/javascripts/index/directions/fossgis_osrm.js
	# set up web server port
	sed -i "s|MAP_PORT|${MAP_PORT}|g" docker-compose.yml
	# set up tile server URL
	sed -i "s|url: 'https://tile.openstreetmap.org/{z}/{x}/{y}.png'|url: '${OSM_TILE_SERVER_URL}'|g" ./vendor/assets/leaflet/leaflet.osm.js
	# set up geocoding server URL
	sed -i "s|nominatim_url:.*|nominatim_url: \"$OSM_GEOCODING_SERVER_URL\"|g" ./config/settings.yml
	# set up routing server URLs
	sed -i "s|fossgis_osrm_url:.*|fossgis_osrm_url: \"$OSM_ROUTING_SERVER_URL\"|g" ./config/settings.yml
	sed -i "s|__OSMCarSuffix__|${OSM_CAR_SUFFIX}|g" ./app/assets/javascripts/index/directions/fossgis_osrm.js
	sed -i "s|__OSMBikeSuffix__|${OSM_BIKE_SUFFIX}|g" ./app/assets/javascripts/index/directions/fossgis_osrm.js
	sed -i "s|__OSMFootSuffix__|${OSM_FOOT_SUFFIX}|g" ./app/assets/javascripts/index/directions/fossgis_osrm.js
	docker compose create

Into this:

# MAP CONTAINER SETUP STARTS HERE

cd openstreetmap-website/
cp ../openstreetmap_docker-compose.template.yml docker-compose.yml

# replace the ports in the docker-compose file using sed
sed -i "s|MAP_PORT|${MAP_PORT}|g" docker-compose.yml
sed -i "s|MAP_DB_PORT|${MAP_DB_PORT}|g" docker-compose.yml
# replace the name "openstreetmap_web" with the container name in $OPENSTREETMAP_WEB_CONTAINER_NAME
sed -i "s|openstreetmap_web|${OPENSTREETMAP_WEB_CONTAINER_NAME}|g" docker-compose.yml
# replace the name "openstreetmap_db" with the container name in $OPENSTREETMAP_DB_CONTAINER_NAME
sed -i "s|openstreetmap_db|${OPENSTREETMAP_DB_CONTAINER_NAME}|g" docker-compose.yml

# The following are the default URLs
OSM_TILE_SERVER_URL="https://tile.openstreetmap.org/{z}/{x}/{y}.png"
OSM_ROUTING_SERVER_URL="https://routing.openstreetmap.de/"  # trailing / allows for routed-<name> to be appended
OSM_GEOCODING_SERVER_URL="https://nominatim.openstreetmap.org/"

# The following are the original URLs released by CMU.
WA_TILE_SERVER_URL="http://ogma.lti.cs.cmu.edu:8080/tile/{z}/{x}/{y}.png"
WA_ROUTING_SERVER_URL="http://metis.lti.cs.cmu.edu:"  # trailing : allows for port to be appended
WA_GEOCODING_SERVER_URL="http://metis.lti.cs.cmu.edu:8085/"

# We use the OSM tile server since the webarena server is not available. However, we use the
# routing and geocoding servers from webarena, since the OSM version was updated since webarena's release
# which means it will have different paths and time estimate, thus skewing the results.
# See: https://github.com/gasse/webarena-setup/issues/4
TILE_SERVER_URL=$OSM_TILE_SERVER_URL
ROUTING_SERVER_URL=$WA_ROUTING_SERVER_URL
GEOCODING_SERVER_URL=$WA_GEOCODING_SERVER_URL

cp ../leaflet.osm.template.js ./vendor/assets/leaflet/leaflet.osm.js
sed -i "s|url: 'https://tile.openstreetmap.org/{z}/{x}/{y}.png'|url: '${TILE_SERVER_URL}'|g" ./vendor/assets/leaflet/leaflet.osm.js
sed -i "s|.*fossgis_osrm_url:.*|fossgis_osrm_url: \"$ROUTING_SERVER_URL\"|g" ./config/settings.yml
sed -i "s|.*nominatim_url:.*|nominatim_url: \"$GEOCODING_SERVER_URL\"|g" ./config/settings.yml

# use the following text for the routing servers, e.g. cloudflare or official
OSM_CAR_SUFFIX="routed-car"
OSM_BIKE_SUFFIX="routed-bike"
OSM_FOOT_SUFFIX="routed-foot"

# alternatively, use ports, e.g. for webarena's setup
WA_CAR_SUFFIX=5000
WA_BIKE_SUFFIX=5001
WA_FOOT_SUFFIX=5002

# We use the webarena suffixes since we are using the webarena routing server
MAP_CAR_SUFFIX=$WA_CAR_SUFFIX
MAP_BIKE_SUFFIX=$WA_BIKE_SUFFIX
MAP_FOOT_SUFFIX=$WA_FOOT_SUFFIX

# now copy the fossgis_osrm.template.js to fossgis_osrm.js and replace __OSMCarSuffix__ with $OSM_CAR_SUFFIX
cp ../fossgis_osrm.template.js ./app/assets/javascripts/index/directions/fossgis_osrm.js

# replace the suffixes in the fossgis_osrm.js file
sed -i "s|__OSMCarSuffix__|${MAP_CAR_SUFFIX}|g" ./app/assets/javascripts/index/directions/fossgis_osrm.js
sed -i "s|__OSMBikeSuffix__|${MAP_BIKE_SUFFIX}|g" ./app/assets/javascripts/index/directions/fossgis_osrm.js
sed -i "s|__OSMFootSuffix__|${MAP_FOOT_SUFFIX}|g" ./app/assets/javascripts/index/directions/fossgis_osrm.js

sudo docker compose create

[xhluca]

For cloudflare, one needs to do a redirect. here's some instructions:

source 00_vars.sh

WA_ROUTING_CF_NAME_CAR="cmu-redirect-routing-car"
WA_ROUTING_CF_NAME_BIKE="cmu-redirect-routing-bike"
WA_ROUTING_CF_NAME_FOOT="cmu-redirect-routing-foot"

WA_ROUTING_URL_CAR="http://metis.lti.cs.cmu.edu:5000"
WA_ROUTING_URL_BIKE="http://metis.lti.cs.cmu.edu:5001"
WA_ROUTING_URL_FOOT="http://metis.lti.cs.cmu.edu:5002"

screen -S $WA_ROUTING_CF_NAME_CAR -dm bash create_and_run_redirect.sh $WA_ROUTING_CF_NAME_CAR $WA_ROUTING_URL_CAR
screen -S $WA_ROUTING_CF_NAME_BIKE -dm bash create_and_run_redirect.sh $WA_ROUTING_CF_NAME_BIKE $WA_ROUTING_URL_BIKE
screen -S $WA_ROUTING_CF_NAME_FOOT -dm bash create_and_run_redirect.sh $WA_ROUTING_CF_NAME_FOOT $WA_ROUTING_URL_FOOT

Content of create_and_run_redirect.sh:

CF_DOMAIN="your-cloudflare-domain.com"

create_route_and_run_redirect() {
    CF_TUNNEL_NAME=$1
    CF_TUNNEL_TARGET_URL=$2

    # delete the tunnel if it exists, and recreate it
    echo "Cleaning up tunnel: $CF_TUNNEL_NAME"
    cloudflared tunnel cleanup $CF_TUNNEL_NAME
    echo "Deleting tunnel: $CF_TUNNEL_NAME"
    cloudflared tunnel delete $CF_TUNNEL_NAME
    echo "Creating tunnel: $CF_TUNNEL_NAME"
    cloudflared tunnel create $CF_TUNNEL_NAME

    # route and start the tunnel
    echo "Routing tunnel: $CF_TUNNEL_NAME"
    cloudflared tunnel route dns --overwrite-dns $CF_TUNNEL_NAME "${CF_TUNNEL_NAME}.${CF_DOMAIN}"
    echo "Running tunnel ${CF_TUNNEL_NAME} at URL: https://${CF_TUNNEL_NAME}.${CF_DOMAIN}"
    echo "Target URL: $CF_TUNNEL_TARGET_URL"
    cloudflared tunnel run --url $CF_TUNNEL_TARGET_URL $CF_TUNNEL_NAME
}

create_route_and_run_redirect $1 $2