OSM-realtime-update

:clock2: OpenStreetMap Data Extracts - updated on the fly!

View the Project on GitHub GIScience/OSM-realtime-update

Real-time OSM

A service providing real-time OSM data.

Three components:

  1. The API adds, deletes and requests information about tasks that provide region-bound real-time OSM data in PBF file format via a permanent URL

  2. The backend runs, manages and gathers statistics for tasks

  3. The frontend allows manipulating tasks and serves the OSM data produced by the tasks.

Installation

Manual

  1. git clone https://github.com/GIScience/OSM-realtime-update

  2. cd OSM-realtime-update/server

  3. npm install

  4. npm start

Docker (caution: not fully tested)

Use the Dockerfile provided.

  1. docker build -t realtimeosm .

  2. docker run -v /path/to/data:/OSM-realtime-update/server/data -p 1234:1234 -d realtimeosm (NOTE: data folder must not be existent, will be created by container)

Config file

You will find a config.js file in the server directory controlling server and api global settings. For instance this will let you determine the number of workers to be run in parallel, specific storage paths, ports and update settings.

Usage

Real-time OSM provides a web-frontend, that is accessible via port 1234.

Point your browser to http://localhost:1234/.

API Specification

Base Url: /api

Main collection: /api/tasks

Individual resource: /api/tasks/:id

Add task

curl --data "name=test_region&coverage=Polygon ((-0.61 53.05, -0.22 51.92, -3.04 52.20, -0.61 53.05))&expirationDate=2020-05-01" http://localhost:1234/api/tasks

curl --data 'name=test_region&coverage={"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[-0.61,53.05],[-3.04,52.20],[-0.22,51.92],[-0.61,53.05]]]}, "properties": {"name": "GB"}}&expirationDate=2020-05-01' http://localhost:1234/api/tasks

curl --data 'name=Togo&coverage=togo&updateInterval=10' http://localhost:1234/api/tasks

Delete task

 

Get information about tasks

 

Backend Specification

Data storage

The OSM data files are stored in a subfolder that is served by the web app. File names are generated using name and id.

A main sqlite3 database keeps track of the application state, it contains two tables:

  1. tasks
    • stores all information about tasks
    • header: id, name, url, coverage, lastUpdated, averageRuntime, addedDate, expirationDate
  2. taskstats
    • stores all benchmarks
    • header: timestamp, taskID, timing

Format:

Update strategy

For each task, initialise data by downloading a suitable .pbf from Geofabrik, clip it to the task’s boundary and update it using osmupdate.

Algorithm:

  1. Is an update process running for this task?
    • Yes: Abort
    • No: go to step 2 and start update
  2. Is there an initial .pbf-file?
    • Yes: Update the file using osmupdate (+timing)
    • No: Generate initial .pbf-file:
      • download smallest Geofabrik extract that covers the task (sample code)
      • clip extract using the task polygon/bbox
      • go to step 2

 

Task management

Worker objects care for a specific task including scheduled updates. A master object manages all workers and checks regularly whether a) workers became obsolete because tasks have been removed or b) a new worker should be spawned because a new task has been added.