Clipping Extracts Without Breaking Boundary Ways

Clipping a continental file down to the region you actually serve is the obvious first step of any OSM pipeline, and the obvious way to do it produces a graph with a ring of damage around its edge. A simple bounding-box extract keeps nodes inside the box and drops those outside, which severs every way crossing the boundary — the motorway that leaves the region loses its far endpoints, its segments lose their geometry, and the junction where it met the local network becomes a dead end. Routing then works perfectly in the middle and fails in a band around the outside, which is the hardest kind of failure to attribute because the graph is not obviously broken.

Prerequisites & Versions

Extraction is done by the osmium toolchain before Python sees the data.

Requirement Minimum version Install
Python 3.11 —
osmium-tool 1.16 apt install osmium-tool
PyOsmium 3.6 pip install "osmium>=3.6"
neo4j (async driver) 5.20 pip install "neo4j>=5.20"

Implementation

import subprocess
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class ExtractSpec:
    source: Path
    output: Path
    poly: Path              # a polygon file, not a bounding box
    buffer_km: float = 25.0 # routing room outside the served area


class Extractor:
    """Wraps `osmium extract` with the strategy that keeps ways whole.

    `complete_ways` keeps every node referenced by any way that has at least one
    node inside the region — so a road crossing the boundary retains its full
    geometry rather than being truncated at the edge. The file is larger than a
    simple clip and the difference is exactly the data that makes the boundary
    routable.
    """

    STRATEGIES = {
        # Keeps only nodes inside the region. Smallest output, severed ways.
        "simple": [],
        # Keeps all nodes of any way with a node inside. What routing needs.
        "complete_ways": ["--strategy=complete_ways"],
        # Also keeps relations' members. Needed for turn restrictions that
        # reference a way just outside the boundary.
        "smart": ["--strategy=smart", "--option=types=multipolygon,restriction"],
    }

    def run(self, spec: ExtractSpec, strategy: str = "smart") -> Path:
        if strategy not in self.STRATEGIES:
            raise ValueError(f"unknown strategy: {strategy}")
        cmd = [
            "osmium", "extract",
            "--polygon", str(spec.poly),
            *self.STRATEGIES[strategy],
            "--overwrite",
            "-o", str(spec.output),
            str(spec.source),
        ]
        subprocess.run(cmd, check=True)
        return spec.output


# After import, the damage a bad extract leaves is detectable in the graph.
BOUNDARY_DAMAGE = """
// Dead ends within the buffer are expected at the very edge and suspicious
// anywhere inside it. A cluster of them along a straight line is a clipped
// boundary rather than a real set of cul-de-sacs.
MATCH (n:Junction)
WHERE size([(n)-[:SEGMENT]-() | 1]) = 1
  AND n.location.latitude  >= $min_lat AND n.location.latitude  <= $max_lat
  AND n.location.longitude >= $min_lon AND n.location.longitude <= $max_lon
RETURN count(n) AS dead_ends
"""

SEVERED_TRUNK = """
// A motorway or trunk road ending in a dead end is almost never real.
MATCH (n:Junction)-[s:SEGMENT]-()
WHERE s.highway IN ['motorway', 'trunk', 'primary']
WITH n, count(s) AS degree
WHERE degree = 1
RETURN n.id AS junction, n.location AS location
ORDER BY junction
"""

How It Works

The buffer and the strategy solve different halves of the problem. The buffer gives the router room to leave the served area and come back — a route between two points near the edge frequently goes outside it, and without the surrounding roads that route does not exist. The strategy decides whether the ways that do cross the buffer’s own boundary survive intact. Both are needed: a generous buffer with a simple clip just moves the ring of damage further out.

complete_ways keeps every node of a partially-included way. That is the property that matters, because a way’s nodes carry its geometry. Truncating a way at the boundary leaves segments whose length and bearing are computed from a coordinate that was never loaded, and the pipeline either drops them or writes something wrong — the PyOsmium guard that skips ways with unresolved node locations exists precisely for what a simple clip produces.

smart additionally keeps relation members. Turn restrictions are relations referencing two ways and a junction node, and a restriction whose via way sits just outside the boundary is dropped by complete_ways — leaving the junction with no restriction, which is a silently permitted illegal turn. On a graph that models turn restrictions as edges, that is a fabricated movement of exactly the kind that arrangement was chosen to make impossible.

The same region under three extract strategies A road crossing the extract boundary, under three strategies. A simple clip keeps only the nodes inside the region, so the way is truncated at the edge and its final segment has one endpoint whose coordinate was never loaded — the junction becomes a dead end. The complete_ways strategy keeps every node of any way with a node inside, so the road retains its full geometry and leaves the region cleanly. The smart strategy additionally keeps relation members, so a turn restriction whose via way lies just outside the boundary survives rather than silently permitting an illegal movement. One road crossing the boundary, three strategies simple region dead end way truncated, geometry incomplete complete_ways region way whole; a restriction outside is still lost smart region restriction kept ways and relations both survive All three imports succeed and report sensible counts. The first produces a ring of dead ends around the region, the second a ring of permitted illegal turns, and only the third leaves the boundary behaving like the middle. The cost is file size, which is the cheapest resource in the pipeline.

Common Failure Patterns

1. Clipping to the served area rather than to a buffered one. A delivery region and a routing region are different things: a route between two addresses in the same district can legitimately leave it, and if the surrounding roads are absent the router either fails or returns an absurd detour along the boundary. Twenty-five kilometres is a reasonable default for road routing and should be larger where the network is sparse.

2. Using a bounding box where a polygon is meant. A box around an irregular region includes large areas nobody serves — which costs import time and store — while still cutting through the parts of the region that stick out. A polygon file costs nothing extra and both fits better and clips less.

3. Trusting the import’s own counts. A severed extract imports cleanly, reports plausible node and relationship totals, and passes every schema check. The damage is in the topology, not in the counts, and the queries above are what expose it — particularly the trunk-road check, since a motorway ending in a dead end is essentially never a real feature.

// Dead ends per distance band from the boundary. A real network has them
// scattered; a severed extract has them concentrated in the outermost band.
MATCH (n:Junction)
WHERE size([(n)-[:SEGMENT]-() | 1]) = 1
RETURN n.boundary_distance_km / 5 AS band, count(*) AS dead_ends
ORDER BY band;

Performance Notes

The strategies differ in both extraction cost and output size, and the ordering is the same for both:

$$|F_{\text{simple}}| < |F_{\text{complete}}| < |F_{\text{smart}}|$$

On a metropolitan extract from a continental source, complete_ways typically produces a file some 5–15 per cent larger than a simple clip and takes roughly twice as long to produce, because it needs a second pass to collect the referenced nodes. smart adds a further pass for relations and a few per cent more. Against an import that takes tens of minutes and a store measured in gigabytes, those are rounding errors — and they buy the removal of an entire class of boundary defect.

The buffer is the more consequential size decision. Extract area grows with the square of the buffer, so doubling it from 25 to 50 km on a compact region can double the imported graph. The right way to choose it is from routing behaviour rather than from instinct: sample real origin-destination pairs, route them against an unclipped graph, and measure how far outside the served region the routes actually stray. On dense urban networks that is usually under ten kilometres; on rural ones with few through routes it can be far more, because the only sensible path between two nearby villages may loop a long way around.

One further habit is worth adopting: keep the polygon and the strategy in the pipeline’s configuration rather than in a shell script someone ran once. The extract is the step every downstream defect is eventually traced back to, and being able to say exactly which polygon and which strategy produced the graph currently in production is the difference between a five-minute diagnosis and an afternoon.

How far real routes stray outside the served region The distribution of maximum excursion outside the served region, measured over ten thousand real origin-destination pairs routed against an unclipped graph. On a dense urban network the great majority stay within five kilometres and almost all within twelve, so a fifteen-kilometre buffer covers essentially every route. On a rural network the distribution has a long tail — the only sensible path between two nearby villages can loop thirty kilometres around — so the same buffer would break a meaningful share of routes. Measuring the distribution is what turns the buffer from a guess into a decision. Maximum excursion outside the served region, 10,000 routes 0 km10203550 kilometres outside the region dense urban rural — long tail a 15 km buffer covers urban and misses the rural tail One buffer figure cannot serve both networks. The measurement takes an afternoon and settles it for the life of the pipeline.

Two operational habits follow from all of this. The first is to treat the extract as a versioned artefact rather than a step: record the source file’s timestamp, the polygon, the strategy and the buffer alongside the resulting graph, so the question “which extract is production running on” has an answer. Every boundary defect is eventually traced back to this step, and the trace is only possible if the inputs were recorded at the time.

The second is to run the damage queries as an acceptance gate rather than as a diagnostic. Dead-end counts and severed trunk roads are cheap to compute immediately after import, they are stable between imports of the same region, and a sudden jump in either is a far better signal than anything a routing test will produce — because a routing test only fails if a sampled route happens to touch the damaged band, while the counts see the whole boundary at once. Gating the promotion of a freshly imported graph on those numbers turns a silent regression into a failed import, which is where you want it.

This guide is part of OSM Data Ingestion Pipelines, within Spatial Graph Construction & OSM Ingestion.

Siblings