Celery Tasks
The name of the task (registered) is marked as italic. e.g. smbackend_turku.tasks.import_mds_data is the register task name of the task that import MDS data.
To create a new task go to the admin page and create a new Periodic Task. It is possible to edit, disable, enable or delete the task afterwards.
The results of the task can be monitored under the results.
Email notifications are sent to users that are in group 'CeleryAdmin'. The name of the group can be configured in settings.py with the variable 'CELERY_ADMIN_GROUP'. Note, the group must be created in the django admin and added to the users.
- Municipalities
- Index search columns
- Import MDS Data
- Addresses
- Divisions
- Counter data
- Environment data
- Exceptional situations
- Unit maintenance(ski trail and ice track maintenance history)
Note it is possible to import all mobility data sources with one task, see.
- Index mobility data search columns
- Culture routes
- Bicycle stands
- Payment zones
- Gas filling stations
- Charging stations
- Scooter restrictions
- Speed limit zones
- Accessories (public toilets, tables, benches, furniture groups)
- Bike service stations
- Parking places for car sharing cars
- Bicycle networks
- Marinas
- Disabled and no staff parkings
- Loading and unloading places
- Lounaistieto shapefiles
- Street maintenance history
- Paavonpolkus
- Outdoor trails
- Traffic signs
- Permits
- Outdoor gym devices
- Föli stops
- Föli parkandride stops
- Outdoor places
- Parking machines
- Playgrounds
- Over and underpasses
- School and kindergarten accessibility areas
- Street area information
- Parking garages
- IoT-data
Note If the database is empty, all the tasks should be created and then run manually in the listed order to populate the database. The tasks are then continuously run according to how they are scheduled.
- Name, descriptive name of the task
- Task (registered), this is the name of task that is registered to Celery.
- Crontab or Interval Schedule, Note Avoid scheduling tasks concurrently.
- Start Datetime
Optionally:
- Arguments
- Setting the task to One-off task, e.g. will only be run once.
- Select the task
- Choose Run selected tasks
- Press the button tun run the task After the Task has finished, the result can be found under task results.
Configure the task as one-off task and run it manually when needed. This task indexes the search columns and must be run after the first time the registered task smbackend_turku.tasks.import_mds_data is run. It can also be used if the search columns needs to be indexed for some reason. The search uses the search columns when searching data, if they are empty or outdated the search results will be faulty, most of the importers updates the search columns automatically. To hyphenate all addresses add the positional argument "--hyphenate_all_addresses".
It is possible to separately import MDS entities, the entities names are: services, accessibility, units. To separately import a entity create a task with the registered name smbackend_turku.tasks.turku_services_import and pass the entity name as argument. e.g. to only import the units.
Create the task with the registered task smbackend_turku.tasks.delete_external_source and specify the name of the sources to be deleted in the arguments. e.g. this deletes gas_filling_stations.
Configure the task and run it manually before setting up other tasks, as they depend on the data imported. This task should be configured as a periodic task to be run once a year. e.g. At 01:00 on day-of-month 2 in January.
It imports the Services, ServiceNodes, Units and Accessability data.
MDS data should be imported once a day, add the following crontab schedule.
If MDS_data is imported the first time into a empty database the smbackend_turku.tasks.import_index_search_columns task must be run afterwards.
Imports:
- Major districts
- Districts
- Sub districts
- Voting districts
- Zip code areas
- Finnish school districts
- Swedish lower comprehensive school districts
- Swedish upper comprehensive school districts
Addresses should be imported ~3-4 times in a year. e.g. At 22:00 on day-of-month 4 in January, May, and September.
This imports all the addresses in Southwest Finland and indexes search columns, note the import might take 8-10hours.
There are three different registered tasks for address importing and the they should be imported in the following order.
- smbackend_turku.tasks.import_addresses, this imports addresses from the WFS server hosted by Turku. The data contains addresses of Kaarina and Turku.
- smbackend_turku.tasks.import_enriched_addresses, this enriches the addresses imported from the WFS server, i.e. Kaarina and Turku. This can be set to run 1h after the first address importer.
- smbackend_turku.tasks.import_geo_search_addresses, this imports the addresses of Southwest Finland, except for Turku and Kaarina, note this task might run for 7-9h.
- Finally manually run the task with registered task smbackend_turku.tasks.import_index_search_columns to update the search columns
Telraam data is imported from a csv file that is populated with the eco_counter.tasks.import_telraam_to_csv importer. The importer should be run every hour. Note, the importer runs hourly and populates daily CSV files. If the importer fails for a day zero(0) data will be populated. It is possible to start the importer with a from date argument, e.g., ["--from-date", "2023-06-05"]. The date is in YYYY-MM-DD format and the date cannot be more than three months in the past, which is the maximum length of history the Telraam API supports.
Before starting hourly imports, this one-off task must be run, this can be done manually. The task can also be used afterwards, if the hourly importer for some reason gets stuck etc. The task should be configured as a One-off Task. The task requires the counter(s) to be initialized as arguments. The counters are EC(Eco Counter), TC(Traffic Counter) and LC(Lam Counter). e.g., of arguments ["EC","TC"] would initialize and import the data for the Eco Counter and the Lam Counter.
Same as the task above, except for that it discards the check if data is available.
The Interval Schedule should be set every hour for Eco-Counter(EC), Traffic-Counter(TC), Telraam-Counter(TR) and daily for LAM-Counter(LC) as the data gets updated on a daily basis.
The counter(s) to be imported must be set as arguments, e.g., ["TC","EC", "LC", "TR"] would import all counters, i.e., Eco Counter, Traffic Counter,
Lam counter and Telraam counter.
Task can be used to delete specific counter datas. The counter is set as argument, e.g.,["LC"] would delete Lam Counter datas.
Imports AQ (Air Quality) and WO (Weather Observation) history. Import initial data and station with the task: environment_data.tasks.initial_import_with_stations. To import initial data without deleting/importing the stations run task: environment_data.tasks.initial_import. After the initial import the data should be imported every hour with the incremental task: environment_data.tasks.incremental_import. The importer task takes the data type as positional argument, e.g., to import Air Quality data the positional argument is ["AQ"] and for Weather Observations the positional argument is ["WO"]. The incremental tasks should be run every hour. To delete all the data (also stations) run task: environment_data.tasks.delete_all_data.
Imports road works and traffic announcements in Southwest Finland from digitraffic.fi, the importer should be run once a hour.
Imports excavation permits in Turku area. The importer task should be run once a day.
The task can be used to delete inactive situations, deletes also the related announcements and locations. Todo, should this task be scheduled in the future?
Configure the task as one-off task and run it manually when needed. This task indexes the search columns for mobility data tables and must be run after the mobility data search is taken into use. It can also be used if the search columns needs to be indexed for some reason.The search uses the search columns when searching data, if they are empty or outdated the search results will be faulty, most of the importers updates the search columns automatically.
Before importing ski trail maintenance history the ski trails must be imported. The ski trails must also be imported if there are changes in the source data. The task can be set as a one-off task.
Imports the maintenance history of ski trails, the importer should be set to run every 15 minutes.
Imports the maintenance history of ice tracks, the import should be set to run every 15 minutes.
Imports all external sources, i.e., mobility data sources to the services list.
Imports all mobility data source to the mobility view.
Schedule the culture routes to be imported once a month. Optionally add argument "--delete" to delete all imported culture routes before importing new ones. e.g. At 02:00 on day-of-month 3
Schedule to be imported once a month.
Schedule to be imported once a year.
Gas Filling Stations, smbackend_turku.tasks.import_gas_filling_stations, source name gas_filling_stations
Schedule to be imported ~3 times a year.
The data should be imported when the data is changed.
Schedule to be imported once a month.
Schedule to be imported twice a year.
Imports benches, public toilets, tables and furniture groups. Schedule to be imported 4 times a year.
The task should be run every time the static data is added or updated. This can also be done from the mobility data DataSource admin view when adding/updating the data.
The task should be run every time the static data is added or updated. This can also be done from the mobility data DataSource admin view when adding/updating the data.
Imports brush salted and brush sanded bicycle networks. The task should be scheduled to be run 3 times a year.
Imports marinas, guest marina and boat parking. The task should be run once a year(before the boating season) or if the data changes.
Imports parking places that are for disabled and not for the staff, as the data is static the importer should be run when the data changes.
Imports loading and unloading places. The data is static and therefore the importer should be run when the data changes.
The importer should be set to run once a month. Currently is set to import:
- Marinas in Southwest Finland
- Slipways in Southwest Finland
- Ferry routes in Southwest Finland
- Fishing spots in Southwest Finland
- Common ferry routes(yhteysalusreitti) in Southwest Finland
- Ferry docks in Southwest Finland
- Bus stop Southwest Finland
All street maintenance history is imported via the same importer. The importer should be run every 15 minutes. There are four providers:
- Destia
- Infraroad
- Kuntec
- YIT
The arguments are sent to the importer via kwargs as a dictionary(key value pair). The parameters(keys) are:
- providers(mandatory), list of providers to import, e.g., {"providers": ["kuntec"]} or {"providers": ["infraroad", "destia"]}
- history-size, number of days to fetch, e.g., {"providers": ["kuntec"], "history-size": 1}. Note the value is an integer and the max value is 31.
- fetch-size, (only available for Destia and Infraroad), The number of works(point data with a timestamp and a events) to be fetched for every unit. e.g., {"providers": ["destia"], "fetch-size": 1000}. The default value is 10_000 and the value is of type int.
Example of kwargs, imports providers destia and infraroad with history-size 1:
Example 2 of kwargs, imports only provider kuntec.:
Note, only the MaintenanceWorks and MaintenanceUnits for the given provider from the latest import are stored and the rest are deleted. The GeometryHistory is generated only if more than one MaintenanceWork is created.
This task can be used to delete street maintenance history for a provider. The name of the provider must be given as argument, e.g, ["DESTIA"].
The data should be imported if changed, otherwise once a year.
Imports paddling, hiking, nature and fitness trails. The importer should be run once a month.
Importer should be run once a month. Imports following signs:
- Crosswalk
- Disabled parking
- Route for disabled
- Parking terminal
- Ticket machine
- Local traffic bus stop
- Long-distance bus stop
- Parking forbidden
- Obligation to use parking disc
- Paid parking
- Parking lot
- Railway level crossing without booms
- Railway level crossing with booms
- Single track railway level crossing with booms
- taxi stand
The importer should be run once a week. Imports following permits:
- ExcavationPermit
- FencePermit
- PlacementPermit
The data is "imported" from the services.units model. The ID of the Unit is stored to the mobile_unit row.
Imports the Föli stops.
Import the park and ride stops for bikes and cars.
Imports:
- Barbecue places
- Lean-tos The task can be set to be run once a month.
The importer imports from a static GEJSON file, so the importer should be run every time the data changes. The data can be updated by modifying the DataSource table from the admin.
Imports the playgrounds in the city of Turku, the importer can be run once a month.
Imports both under and overpasses from Väylävirasto. The importer could be run every month.
School and kindergarten accessibility areas, mobility_data.tasks.import_school_and_kindergarten_accessibility_areas
Imports school and kindergarten accessibility areas. The importer should be run every time the data has changed(yearly?).
Imports information (owner, street maintenance class etc..). The importer should be run once a month.
As the source data is a static CSV file, the importer should be run when the source data is updated.
Create a task for every IoT data source. The scheduling depends on how often the source data is updated and what is reasonable to keep the data up to date. Every task needs the three letter name of the IoT data source to be added to the arguments. See, creating IoT data sources.
e.g., to add a IoT data source with the three letter name R24. Note the quotes surrounding the name(R24).
Note, the type_name, e.g., R24 must not change, as they are hard coded in the front end.
- R24 (24Rent shared use cars). This is updated every hour.
- CBI (information about city bike stations). This is updated every 30 minutes.
- CBS (statistics of every city bike station). This is updated every 30 minutes.
- FAA (flights that will arrive to Turku Airport). This is updated every hour.
- FAD (flights that will depart from Turku Airport). This is updated every hour.
- SDR (scooters by Ryde). This is updated every 30 minutes.
- SDV (scooters by Voi).
- SDT (scooters by Tier).
Whit this task it is possible to delete mobility units for given content types. Give content type name as argument e.g., ["PaavonPolku"], would delete all the mobile units that are 'Paavonpolku' content type. It is also possible to give many content types as argument e.g., ["PaavonPolku", "FitnessTrail"].