feat(ET-009): activate EnduroRussia + Wikiloc GPS sources
Some checks failed
CI / lint (push) Failing after 4s
CI / test (push) Failing after 5s
CI / build (push) Has been skipped
CI / lint (pull_request) Failing after 4s
CI / test (pull_request) Failing after 5s
CI / build (pull_request) Has been skipped

Конфиг-only активация двух новых источников GPS-треков поверх
pipeline ET-008. Не вводит новых компонентов, БД-таблиц, endpoint'ов.

Config:
- config/gps_sources.yaml: enduro_russia enabled=true, base_url исправлен
  на endurorussia.ru (без дефиса); добавлена запись wikiloc с
  max_tracks_per_run=50, activity_filter=[motorcycle, enduro].
- config/gps_regions.yaml: wikiloc добавлен в tsfo_plus_chuvashia.sources.

Parser:
- wikiloc.py: добавлен soft-cap max_tracks_per_run в collect(),
  извлечение created_at из GPX metadata/первого trkpt — для корректной
  межисточниковой дедупликации с EnduroRussia.

UI (src/web/gps_tracks.js):
- GPS_SOURCE_COLORS: добавлен цвет wikiloc (#4363d8).
- Дефолтный фильтр sources включает wikiloc.
- GPS_SOURCE_ATTRIBUTIONS: маппинг source_id → строка атрибуции;
  _updateGpsAttribution() подтягивает /api/gps-tracks/health и
  выставляет attribution с теми источниками, у которых tracks > 0.
- _buildGpsFiltersUI: чекбокс «Wikiloc» в #gps-source-grid.

Tests:
- Fixtures: 7 файлов в tests/fixtures/gps-tracks/.
- Unit: 10 UT-ER + 10 UT-WL — парсеры, MAPPING, bbox-фильтр,
  pagination, 429/403 graceful-stop, rate-limit, max_tracks_per_run.
- Integration: IT-ER-01, IT-WL-01, IT-WL-02, IT-DEDUP-01, IT-LIC-01
  через scripts.gps_collect.main + httpx.MockTransport.
- Contract: 2 CT-ER с маркером @pytest.mark.network (nightly only).
- JS: 2 новых теста на наличие wikiloc в SOURCE_COLORS и в фильтрах.

Linters/Tests: ruff clean (новые файлы), 166 pytest passed,
24 JS-tests passed.

Refs: ET-009
Acceptance: AC-01..AC-08, AC-14..AC-17 (для AC-09..AC-13 — продакшн-прогон)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-01 19:38:55 +00:00
parent 4be7fbf3de
commit 3577ff32ac
19 changed files with 1287 additions and 19 deletions

View File

@@ -17,6 +17,19 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
## [Unreleased]
### Added
- ET-009: Активация GPS-источников EnduroRussia и Wikiloc — `config/gps_sources.yaml`
включает оба источника (`enabled: true`), для Wikiloc добавлен soft-cap
`max_tracks_per_run: 50` и activity-фильтр; `config/gps_regions.yaml` подписывает
`wikiloc` на регион `tsfo_plus_chuvashia`. Парсер `wikiloc.py` извлекает время из
GPX-metadata (для корректной дедупликации) и поддерживает `max_tracks_per_run`
cap. UI: цвет `wikiloc`, чекбокс источника, динамическая атрибуция
(`GPS_SOURCE_ATTRIBUTIONS`) подтягивается с `/api/gps-tracks/health`.
Тесты: 10 unit ER + 10 unit WL + 5 integration + 2 contract (nightly only).
### Fixed
- ET-009: исправлен URL `enduro_russia` в `config/gps_sources.yaml`
(`https://enduro-russia.ru``https://endurorussia.ru`, без дефиса).
- Initial project structure
- CLAUDE.md project passport
- Agent system prompts (architect, developer, reviewer, tester, deployer)

View File

@@ -3,7 +3,7 @@ regions:
name: "ЦФО + Чувашия"
bbox: [29.0, 49.5, 47.5, 60.0]
enabled: true
sources: [osm, enduro_russia, ttrails]
sources: [osm, enduro_russia, wikiloc, ttrails]
- id: north_caucasus
name: "Северный Кавказ"

View File

@@ -13,14 +13,29 @@ sources:
- id: enduro_russia
name: "EnduroRussia.ru"
enabled: false
enabled: true
license_adr: "docs/work-items/ET-008/06-adr/ADR-010-enduro-russia-licensing.md"
base_url: "https://enduro-russia.ru"
base_url: "https://endurorussia.ru"
rate_limit_sec: 5
user_agent: "enduro-trails/1.0 (+https://openclaw.mva154.duckdns.org/enduro/)"
attribution: "EnduroRussia.ru"
parser_module: "src.api.gps_tracks.sources.enduro_russia"
save_user_field: false
source_priority: 80
- id: wikiloc
name: "Wikiloc"
enabled: true
license_adr: "docs/work-items/ET-008/06-adr/ADR-012-wikiloc-licensing.md"
base_url: "https://www.wikiloc.com"
rate_limit_sec: 10
user_agent: "enduro-trails/1.0 (+https://openclaw.mva154.duckdns.org/enduro/)"
attribution: "© Wikiloc contributors"
parser_module: "src.api.gps_tracks.sources.wikiloc"
save_user_field: false
source_priority: 70
activity_filter: [motorcycle, enduro]
max_tracks_per_run: 50
- id: ttrails
name: "Тропинки.ру"

View File

@@ -35,3 +35,7 @@ line-length = 120
[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths = ["tests"]
markers = [
"network: contract smoke tests that hit live HTTP endpoints (deselect with '-m \"not network\"')",
]
addopts = "-m 'not network'"

View File

@@ -62,6 +62,8 @@ class WikilocParser(SourceParser):
source_id = self.config.get("id", "wikiloc")
source_priority = self.config.get("source_priority", 70)
activity_filter = self.config.get("activity_filter", ["motorcycle", "enduro"])
max_tracks = self.config.get("max_tracks_per_run")
yielded = 0
headers = {
"User-Agent": user_agent,
@@ -188,7 +190,15 @@ class WikilocParser(SourceParser):
):
continue
if max_tracks is not None and yielded >= max_tracks:
logger.info(
"Wikiloc: reached max_tracks_per_run=%d, stopping",
max_tracks,
)
return
yield track
yielded += 1
page += 1
@@ -260,16 +270,40 @@ def _parse_gpx(
if tag.startswith("{"):
ns = tag.split("}")[0] + "}"
# Извлекаем название из GPX metadata если нет из HTML
if not name:
for child in root:
local = child.tag.replace(ns, "") if ns else child.tag
if local == "metadata":
for meta_child in child:
local2 = meta_child.tag.replace(ns, "") if ns else meta_child.tag
if local2 == "name":
name = meta_child.text
# Извлекаем название и время из GPX metadata
created_at = None
for child in root:
local = child.tag.replace(ns, "") if ns else child.tag
if local == "metadata":
for meta_child in child:
local2 = meta_child.tag.replace(ns, "") if ns else meta_child.tag
if local2 == "name" and not name:
name = meta_child.text
elif local2 == "time" and meta_child.text:
created_at = meta_child.text.strip()
break
# Fallback: первая <trkpt><time> из первого trkseg
if not created_at:
for trk in root:
local = trk.tag.replace(ns, "") if ns else trk.tag
if local != "trk":
continue
for trkseg in trk:
local2 = trkseg.tag.replace(ns, "") if ns else trkseg.tag
if local2 != "trkseg":
continue
for trkpt in trkseg:
for sub in trkpt:
sub_local = sub.tag.replace(ns, "") if ns else sub.tag
if sub_local == "time" and sub.text:
created_at = sub.text.strip()
break
if created_at:
break
if created_at:
break
if created_at:
break
coords = []
@@ -324,7 +358,7 @@ def _parse_gpx(
description=None,
activity_type=activity_type,
user=None,
created_at=None,
created_at=created_at,
length_m=length_m,
points_count=len(coords),
geom_wkb=geom_wkb,

View File

@@ -10,12 +10,23 @@ const GPS_TRACKS_MIN_ZOOM = 8; // ниже — слой скрыт
const GPS_SOURCE_COLORS = {
osm: '#3cb44b',
enduro_russia: '#e6194b',
ttrails: '#4363d8',
wikiloc: '#4363d8',
ttrails: '#911eb4',
offmaps: '#f58231',
nakarte: '#911eb4',
nakarte: '#f032e6',
};
const GPS_FALLBACK_COLORS = ['#42d4f4', '#f032e6', '#bfef45', '#fabed4', '#469990', '#dcbeff', '#9A6324', '#fffac8'];
// ET-009: атрибуция для каждого источника. Используется при сборке
// MapLibre attribution control: к строке source-attribution добавляются
// все источники, у которых tracks_by_source > 0.
const GPS_SOURCE_ATTRIBUTIONS = {
osm: '© OpenStreetMap contributors (ODbL)',
enduro_russia: 'EnduroRussia.ru',
wikiloc: '© Wikiloc contributors',
ttrails: 'ttrails.ru',
};
const GPS_ACTIVITY_COLORS = {
enduro: '#e6194b',
moto: '#f58231',
@@ -52,7 +63,7 @@ window.gpsTracksLayer = {
enabled: false,
filters: {
activities: ['enduro', 'moto', 'offroad', 'bicycle', 'hike', 'ski', 'other'],
sources: ['osm', 'enduro_russia', 'ttrails'],
sources: ['osm', 'enduro_russia', 'wikiloc', 'ttrails'],
colorMode: 'source'
},
sourceId: 'gps-tracks-tiles',
@@ -188,6 +199,44 @@ function _ensureGpsLayers(map) {
}
}
/**
* ET-009: динамически обновляет attribution источника `gps-tracks-tiles` на
* основе ответа /api/gps-tracks/health.tracks_by_source. Включает в строку
* атрибуцию каждого источника, у которого > 0 треков в БД. Падение запроса —
* не блокирующее: остаётся последняя установленная атрибуция.
*/
async function _updateGpsAttribution(map) {
if (!map || !map.getSource) return;
const basePath = window.location.pathname.replace(/\/[^/]*$/, '') || '';
try {
const resp = await fetch(`${basePath}/api/gps-tracks/health`);
if (!resp.ok) return;
const data = await resp.json();
const counts = data && data.tracks_by_source ? data.tracks_by_source : {};
const labels = [];
for (const src of Object.keys(GPS_SOURCE_ATTRIBUTIONS)) {
if (counts[src] && counts[src] > 0) {
labels.push(GPS_SOURCE_ATTRIBUTIONS[src]);
}
}
if (labels.length === 0) {
labels.push(GPS_SOURCE_ATTRIBUTIONS.osm);
}
const attribution = labels.join(', ');
const src = map.getSource(window.gpsTracksLayer.sourceId);
if (src && typeof src.attribution !== 'undefined') {
src.attribution = attribution;
}
// MapLibre не перечитывает source.attribution автоматически —
// дергаем resize чтобы обновить AttributionControl
if (typeof map._controls !== 'undefined') {
try { map.resize(); } catch (_) { /* noop */ }
}
} catch (_) {
// network failure — оставляем дефолтную атрибуцию
}
}
function _findGpsInsertPosition(map) {
/**
* Returns the id of the first layer that GPS tracks should be inserted
@@ -411,6 +460,8 @@ function onPublicTracksCheckbox() {
_ensureGpsSources(map);
_ensureGpsLayers(map);
_setupGpsClickHandler(map);
// ET-009: подтянуть актуальные атрибуции (osm, enduro_russia, wikiloc)
_updateGpsAttribution(map);
// Убедиться, что moveend listener есть
map.off('moveend', onGpsMapMoveEnd);
@@ -476,8 +527,13 @@ function _buildGpsFiltersUI() {
// Источники (из localStorage или дефолт)
const srcGrid = document.getElementById('gps-source-grid');
if (srcGrid) {
const allSources = ['osm', 'enduro_russia', 'ttrails'];
const sourceLabels = { osm: 'OSM', enduro_russia: 'EnduroRussia.ru', ttrails: 'Тропинки.ру' };
const allSources = ['osm', 'enduro_russia', 'wikiloc', 'ttrails'];
const sourceLabels = {
osm: 'OSM',
enduro_russia: 'EnduroRussia',
wikiloc: 'Wikiloc',
ttrails: 'Тропинки.ру',
};
srcGrid.innerHTML = allSources.map(src => {
const checked = window.gpsTracksLayer.filters.sources.includes(src);
return `

View File

View File

@@ -0,0 +1,64 @@
"""Contract smoke tests for live endurorussia.ru API (ET-009).
Маркер @pytest.mark.network — пропускается в обычном CI.
Запускается вручную или nightly: `pytest -m network`.
Coverage:
- CT-ER-01: GET /api/tracks?page=0&limit=5 → 200 + items, total
- CT-ER-02: GET /api/tracks/{first_id}/gpx → 200 + parseable GPX
"""
import pytest
import defusedxml.ElementTree as ET
import httpx
BASE_URL = "https://endurorussia.ru"
USER_AGENT = "enduro-trails/1.0 (+https://openclaw.mva154.duckdns.org/enduro/)"
@pytest.mark.network
def test_ct_er_01_tracks_list_200_with_items():
"""CT-ER-01: GET /api/tracks?page=0&limit=5 → 200, JSON с items, total."""
headers = {"User-Agent": USER_AGENT, "Accept": "application/json"}
with httpx.Client(timeout=30, headers=headers) as client:
resp = client.get(f"{BASE_URL}/api/tracks?page=0&limit=5")
assert resp.status_code == 200, f"got {resp.status_code}: {resp.text[:200]}"
data = resp.json()
assert "items" in data
assert "total" in data
assert isinstance(data["items"], list)
assert isinstance(data["total"], int)
assert len(data["items"]) > 0
first = data["items"][0]
assert "id" in first
assert "name" in first
@pytest.mark.network
def test_ct_er_02_track_gpx_200_parseable():
"""CT-ER-02: GET /api/tracks/{first_id}/gpx → 200, валидный GPX."""
headers = {
"User-Agent": USER_AGENT,
"Accept": "application/json",
}
with httpx.Client(timeout=30, headers=headers) as client:
list_resp = client.get(f"{BASE_URL}/api/tracks?page=0&limit=5")
assert list_resp.status_code == 200
items = list_resp.json().get("items", [])
assert len(items) > 0
first_id = items[0]["id"]
gpx_resp = client.get(
f"{BASE_URL}/api/tracks/{first_id}/gpx",
headers={**headers, "Accept": "application/gpx+xml,application/xml,*/*"},
)
assert gpx_resp.status_code == 200
ctype = gpx_resp.headers.get("content-type", "").lower()
assert "xml" in ctype or "gpx" in ctype, f"content-type: {ctype}"
# Парсится без exception
root = ET.fromstring(gpx_resp.content)
assert root.tag.endswith("gpx"), f"root tag: {root.tag}"

View File

@@ -0,0 +1,46 @@
{
"items": [
{
"id": 1,
"name": "Маршрут Дмитровский — лесная петля",
"difficulty": "hard",
"created_at": "2024-08-15 12:30:00",
"description": "Лесная петля с грязевыми участками",
"length_km": 24.5
},
{
"id": 2,
"name": "Эндуро-загон под Тверью",
"difficulty": "мото",
"created_at": "2024-09-02 09:15:00",
"description": "Песчаные горки",
"length_km": 18.2
},
{
"id": 3,
"name": "Дальний выезд (за пределами ЦФО)",
"difficulty": "soft",
"created_at": "2024-09-10 08:00:00",
"description": "Тестовый выезд",
"length_km": 12.0
},
{
"id": 4,
"name": "Жесткий хард-эндуро",
"difficulty": "hard",
"created_at": "2024-09-12 13:40:00",
"description": "Только для опытных",
"length_km": 31.4
},
{
"id": 5,
"name": "Тестовый сглаженный круг",
"difficulty": "soft",
"created_at": "2024-09-15 10:00:00",
"description": "Для новичков",
"length_km": 14.3
}
],
"total": 5,
"page": 0
}

View File

@@ -0,0 +1,25 @@
<?xml version="1.0" encoding="UTF-8"?>
<gpx version="1.1" creator="EnduroRussia" xmlns="http://www.topografix.com/GPX/1/1">
<metadata>
<name>Маршрут Дмитровский — лесная петля</name>
<author><name>EnduroRussia.ru</name></author>
<time>2024-08-15T12:30:00Z</time>
</metadata>
<trk>
<name>Маршрут Дмитровский — лесная петля</name>
<trkseg>
<trkpt lat="56.3500" lon="37.5200"><time>2024-08-15T12:30:00Z</time></trkpt>
<trkpt lat="56.3510" lon="37.5215"><time>2024-08-15T12:30:30Z</time></trkpt>
<trkpt lat="56.3520" lon="37.5230"><time>2024-08-15T12:31:00Z</time></trkpt>
<trkpt lat="56.3535" lon="37.5250"><time>2024-08-15T12:31:30Z</time></trkpt>
<trkpt lat="56.3550" lon="37.5275"><time>2024-08-15T12:32:00Z</time></trkpt>
<trkpt lat="56.3565" lon="37.5300"><time>2024-08-15T12:32:30Z</time></trkpt>
<trkpt lat="56.3580" lon="37.5325"><time>2024-08-15T12:33:00Z</time></trkpt>
<trkpt lat="56.3595" lon="37.5350"><time>2024-08-15T12:33:30Z</time></trkpt>
<trkpt lat="56.3610" lon="37.5375"><time>2024-08-15T12:34:00Z</time></trkpt>
<trkpt lat="56.3625" lon="37.5400"><time>2024-08-15T12:34:30Z</time></trkpt>
<trkpt lat="56.3640" lon="37.5425"><time>2024-08-15T12:35:00Z</time></trkpt>
<trkpt lat="56.3655" lon="37.5450"><time>2024-08-15T12:35:30Z</time></trkpt>
</trkseg>
</trk>
</gpx>

View File

@@ -0,0 +1,12 @@
<?xml version="1.0" encoding="UTF-8"?>
<gpx version="1.1" creator="EnduroRussia" xmlns="http://www.topografix.com/GPX/1/1">
<metadata>
<name>Эндуро-загон под Тверью (пустой)</name>
<author><name>EnduroRussia.ru</name></author>
</metadata>
<trk>
<name>Эндуро-загон под Тверью</name>
<trkseg>
</trkseg>
</trk>
</gpx>

View File

@@ -0,0 +1,18 @@
<?xml version="1.0" encoding="UTF-8"?>
<gpx version="1.1" creator="EnduroRussia" xmlns="http://www.topografix.com/GPX/1/1">
<metadata>
<name>Дальний выезд (за пределами ЦФО)</name>
<author><name>EnduroRussia.ru</name></author>
<time>2024-09-10T08:00:00Z</time>
</metadata>
<trk>
<name>Дальний выезд</name>
<trkseg>
<trkpt lat="48.0000" lon="20.0000"><time>2024-09-10T08:00:00Z</time></trkpt>
<trkpt lat="48.0010" lon="20.0010"><time>2024-09-10T08:00:30Z</time></trkpt>
<trkpt lat="48.0020" lon="20.0020"><time>2024-09-10T08:01:00Z</time></trkpt>
<trkpt lat="48.0030" lon="20.0030"><time>2024-09-10T08:01:30Z</time></trkpt>
<trkpt lat="48.0040" lon="20.0040"><time>2024-09-10T08:02:00Z</time></trkpt>
</trkseg>
</trk>
</gpx>

View File

@@ -0,0 +1,32 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Wikiloc — Search results</title>
</head>
<body>
<main class="search-results">
<h1>Search results — Motorcycle (act=19)</h1>
<ul class="trail-list">
<li class="trail-item">
<a href="/trails/motorcycle-enduro/12345678">Дмитровский лес</a>
</li>
<li class="trail-item">
<a href="/trails/motorcycle-enduro/12345679">Тверь песчаные карьеры</a>
</li>
<li class="trail-item">
<a href="/trails/motorcycle-enduro/12345680">Чувашия круг</a>
</li>
<li class="trail-item">
<a href="/trails/motorcycle-enduro/12345681">Ярославль грунтовка</a>
</li>
<li class="trail-item">
<a href="/trails/motorcycle-enduro/12345682">Владимир грязь</a>
</li>
<li class="trail-item">
<a href="/trails/motorcycle-enduro/12345683">Кострома перевал</a>
</li>
</ul>
</main>
</body>
</html>

View File

@@ -0,0 +1,25 @@
<?xml version="1.0" encoding="UTF-8"?>
<gpx version="1.1" creator="Wikiloc" xmlns="http://www.topografix.com/GPX/1/1">
<metadata>
<name>Дмитровский лес (Wikiloc copy)</name>
<author><name>Wikiloc</name></author>
<time>2024-08-15T12:30:00Z</time>
</metadata>
<trk>
<name>Дмитровский лес</name>
<trkseg>
<trkpt lat="56.3501" lon="37.5201"><time>2024-08-15T12:30:00Z</time></trkpt>
<trkpt lat="56.3511" lon="37.5216"><time>2024-08-15T12:30:30Z</time></trkpt>
<trkpt lat="56.3521" lon="37.5231"><time>2024-08-15T12:31:00Z</time></trkpt>
<trkpt lat="56.3536" lon="37.5251"><time>2024-08-15T12:31:30Z</time></trkpt>
<trkpt lat="56.3551" lon="37.5276"><time>2024-08-15T12:32:00Z</time></trkpt>
<trkpt lat="56.3566" lon="37.5301"><time>2024-08-15T12:32:30Z</time></trkpt>
<trkpt lat="56.3581" lon="37.5326"><time>2024-08-15T12:33:00Z</time></trkpt>
<trkpt lat="56.3596" lon="37.5351"><time>2024-08-15T12:33:30Z</time></trkpt>
<trkpt lat="56.3611" lon="37.5376"><time>2024-08-15T12:34:00Z</time></trkpt>
<trkpt lat="56.3626" lon="37.5401"><time>2024-08-15T12:34:30Z</time></trkpt>
<trkpt lat="56.3641" lon="37.5426"><time>2024-08-15T12:35:00Z</time></trkpt>
<trkpt lat="56.3656" lon="37.5451"><time>2024-08-15T12:35:30Z</time></trkpt>
</trkseg>
</trk>
</gpx>

View File

@@ -0,0 +1,20 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Дмитровский лес — Wikiloc</title>
</head>
<body>
<main class="trail-page">
<h1>Дмитровский лес</h1>
<div class="trail-meta">
<span class="activity">Motorcycle (enduro)</span>
<span class="distance">24.5 km</span>
</div>
<p class="trail-description">Лесная петля.</p>
<div class="trail-download">
<a class="btn-download" href="/wikiloc/downloadTrail.do?id=12345678">Download GPX</a>
</div>
</main>
</body>
</html>

View File

@@ -0,0 +1,360 @@
"""Integration tests for GPS pipeline with new sources (ET-009).
Coverage:
- IT-ER-01: EnduroRussia pipeline with 3 fixture GPX (1 in-bbox, 2 empty, 3 out-of-bbox)
- IT-WL-01: Wikiloc pipeline with 1 fixture track
- IT-WL-02: Wikiloc graceful-stop on 403 → status='partial', exit_code=0
- IT-DEDUP-01: EnduroRussia + Wikiloc same track → 1 row, merged sources
- IT-LIC-01: License guard blocks source when ADR status=proposed
"""
import asyncio
import json
import os
import sys
from typing import Callable
import httpx
import yaml
# Add project root to path
PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
sys.path.insert(0, PROJECT_ROOT)
from src.api.gps_tracks.sources import enduro_russia as er_module # noqa: E402
from src.api.gps_tracks.sources import wikiloc as wl_module # noqa: E402
FIXTURES_DIR = os.path.join(os.path.dirname(__file__), "..", "fixtures", "gps-tracks")
def _read_fixture(name: str) -> bytes:
with open(os.path.join(FIXTURES_DIR, name), "rb") as f:
return f.read()
def _read_fixture_text(name: str) -> str:
return _read_fixture(name).decode("utf-8")
def _make_handler_combined(handlers: dict) -> Callable[[httpx.Request], httpx.Response]:
"""Combines multiple handler functions, selecting by URL host."""
def handler(req: httpx.Request) -> httpx.Response:
host = req.url.host
for host_pattern, h in handlers.items():
if host_pattern in host:
return h(req)
return httpx.Response(404)
return handler
def _patch_httpx(monkeypatch, handler):
"""Подменяет httpx.AsyncClient в обоих parser-модулях."""
transport = httpx.MockTransport(handler)
original = httpx.AsyncClient
def factory(*args, **kwargs):
kwargs["transport"] = transport
return original(*args, **kwargs)
monkeypatch.setattr(er_module.httpx, "AsyncClient", factory)
monkeypatch.setattr(wl_module.httpx, "AsyncClient", factory)
def _write_config(tmp_dir: str, sources: list, regions: list) -> tuple[str, str]:
"""Записывает временные конфиги."""
src_path = os.path.join(tmp_dir, "gps_sources.yaml")
reg_path = os.path.join(tmp_dir, "gps_regions.yaml")
with open(src_path, "w") as f:
yaml.safe_dump({"sources": sources}, f)
with open(reg_path, "w") as f:
yaml.safe_dump({"regions": regions}, f)
return src_path, reg_path
def _setup_env(monkeypatch, tmp_dir, sources, regions):
src_path, reg_path = _write_config(tmp_dir, sources, regions)
db_path = os.path.join(tmp_dir, "test_gps.sqlite")
monkeypatch.setenv("GPS_SOURCES_CONFIG", src_path)
monkeypatch.setenv("GPS_REGIONS_CONFIG", reg_path)
monkeypatch.setenv("GPS_TRACKS_DB_PATH", db_path)
return db_path
def _run_pipeline(args=None):
"""Запускает scripts/gps_collect.py::main() через asyncio.run."""
from scripts.gps_collect import main as pipeline_main
saved_argv = sys.argv[:]
try:
sys.argv = ["gps_collect.py"] + (args or [])
return asyncio.run(pipeline_main())
finally:
sys.argv = saved_argv
def _last_pipeline_run(db_path: str) -> dict:
import sqlite3
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
row = conn.execute(
"SELECT * FROM pipeline_runs ORDER BY id DESC LIMIT 1"
).fetchone()
conn.close()
return dict(row) if row else None
def _count_tracks(db_path: str) -> int:
import sqlite3
conn = sqlite3.connect(db_path)
n = conn.execute("SELECT COUNT(*) FROM tracks").fetchone()[0]
conn.close()
return n
def _all_tracks(db_path: str) -> list:
import sqlite3
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
rows = conn.execute("SELECT * FROM tracks").fetchall()
conn.close()
return [dict(r) for r in rows]
# Сорсы для тестов
ER_SOURCE = {
"id": "enduro_russia",
"name": "EnduroRussia.ru",
"enabled": True,
"license_adr": "docs/work-items/ET-008/06-adr/ADR-010-enduro-russia-licensing.md",
"base_url": "https://endurorussia.ru",
"rate_limit_sec": 0,
"user_agent": "test/1.0",
"attribution": "EnduroRussia.ru",
"parser_module": "src.api.gps_tracks.sources.enduro_russia",
"save_user_field": False,
"source_priority": 80,
}
WL_SOURCE = {
"id": "wikiloc",
"name": "Wikiloc",
"enabled": True,
"license_adr": "docs/work-items/ET-008/06-adr/ADR-012-wikiloc-licensing.md",
"base_url": "https://www.wikiloc.com",
"rate_limit_sec": 0,
"user_agent": "test/1.0",
"attribution": "© Wikiloc contributors",
"parser_module": "src.api.gps_tracks.sources.wikiloc",
"save_user_field": False,
"source_priority": 70,
"activity_filter": ["motorcycle"],
}
REGION_TSFO = {
"id": "tsfo_plus_chuvashia",
"name": "ЦФО + Чувашия",
"bbox": [29.0, 49.5, 47.5, 60.0],
"enabled": True,
"sources": ["enduro_russia", "wikiloc"],
}
# ─── IT-ER-01 ───────────────────────────────────────────────────────────────
def test_it_er_01_pipeline_enduro_russia_three_gpx(monkeypatch, tmp_path):
"""IT-ER-01: 3 фикстурных GPX → tracks_new=1 (track1 OK; track2 empty; track3 out-of-bbox)."""
api_data = {
"items": [
{"id": 1, "name": "Track1", "difficulty": "hard", "created_at": "2024-08-15 12:30:00"},
{"id": 2, "name": "Track2", "difficulty": "soft", "created_at": "2024-09-02 09:15:00"},
{"id": 3, "name": "Track3", "difficulty": "soft", "created_at": "2024-09-10 08:00:00"},
],
"total": 3,
"page": 0,
}
def handler(req: httpx.Request) -> httpx.Response:
if req.url.host == "endurorussia.ru":
if req.url.path == "/api/tracks":
return httpx.Response(200, json=api_data)
for tid in (1, 2, 3):
if req.url.path == f"/api/tracks/{tid}/gpx":
return httpx.Response(200, content=_read_fixture(f"enduro-russia-track-{tid}.gpx"))
return httpx.Response(404)
_patch_httpx(monkeypatch, handler)
db_path = _setup_env(monkeypatch, str(tmp_path), [ER_SOURCE], [REGION_TSFO])
exit_code = _run_pipeline(["--region", "tsfo_plus_chuvashia", "--source", "enduro_russia"])
assert exit_code == 0
assert _count_tracks(db_path) == 1
run = _last_pipeline_run(db_path)
assert run is not None
assert run["status"] == "ok"
assert run["tracks_new"] == 1
assert run["source_id"] == "enduro_russia"
# ─── IT-WL-01 ───────────────────────────────────────────────────────────────
def test_it_wl_01_pipeline_wikiloc_one_track(monkeypatch, tmp_path):
"""IT-WL-01: Wikiloc с 1 треком → tracks_new=1, status ∈ {ok, partial}."""
# Поиск возвращает 1 трек, дальше 404 чтобы остановиться
mini_search = '<html><a href="/trails/motorcycle-enduro/12345678">x</a></html>'
def handler(req: httpx.Request) -> httpx.Response:
if req.url.host == "www.wikiloc.com":
if "find.do" in req.url.path:
if "page=0" in str(req.url.query):
return httpx.Response(200, text=mini_search)
return httpx.Response(200, text="<html></html>")
if req.url.path.startswith("/trails/"):
return httpx.Response(200, text=_read_fixture_text("wikiloc-trail-page.html"))
if "downloadTrail.do" in req.url.path:
return httpx.Response(200, content=_read_fixture("wikiloc-track.gpx"))
return httpx.Response(404)
_patch_httpx(monkeypatch, handler)
region = dict(REGION_TSFO, sources=["wikiloc"])
db_path = _setup_env(monkeypatch, str(tmp_path), [WL_SOURCE], [region])
exit_code = _run_pipeline(["--region", "tsfo_plus_chuvashia", "--source", "wikiloc"])
assert exit_code == 0
assert _count_tracks(db_path) == 1
run = _last_pipeline_run(db_path)
assert run["status"] in ("ok", "partial")
assert run["tracks_new"] == 1
# ─── IT-WL-02 ───────────────────────────────────────────────────────────────
def test_it_wl_02_pipeline_wikiloc_403_graceful(monkeypatch, tmp_path):
"""IT-WL-02: Wikiloc 403 на поиске → status='partial' (или 'ok'), exit_code=0."""
def handler(req: httpx.Request) -> httpx.Response:
if req.url.host == "www.wikiloc.com":
if "find.do" in req.url.path:
return httpx.Response(403, text="Forbidden")
return httpx.Response(404)
_patch_httpx(monkeypatch, handler)
region = dict(REGION_TSFO, sources=["wikiloc"])
db_path = _setup_env(monkeypatch, str(tmp_path), [WL_SOURCE], [region])
exit_code = _run_pipeline(["--region", "tsfo_plus_chuvashia", "--source", "wikiloc"])
assert exit_code == 0, "graceful-stop should not produce error exit"
assert _count_tracks(db_path) == 0
run = _last_pipeline_run(db_path)
# graceful-stop → status 'ok' (parser просто завершился без exception);
# в TZ ослабленно: ∈ {ok, partial, rate_limited}
assert run["status"] in ("ok", "partial", "rate_limited")
assert run["tracks_new"] == 0
# ─── IT-DEDUP-01 ────────────────────────────────────────────────────────────
def test_it_dedup_01_merge_enduro_russia_and_wikiloc(monkeypatch, tmp_path):
"""IT-DEDUP-01: одинаковый трек из 2 источников → 1 запись с merged sources."""
er_api = {
"items": [
{"id": 1, "name": "Дмитровский ER", "difficulty": "hard", "created_at": "2024-08-15 12:30:00"},
],
"total": 1,
"page": 0,
}
mini_search = '<html><a href="/trails/motorcycle-enduro/12345678">x</a></html>'
def handler(req: httpx.Request) -> httpx.Response:
if req.url.host == "endurorussia.ru":
if req.url.path == "/api/tracks":
return httpx.Response(200, json=er_api)
if req.url.path == "/api/tracks/1/gpx":
return httpx.Response(200, content=_read_fixture("enduro-russia-track-1.gpx"))
if req.url.host == "www.wikiloc.com":
if "find.do" in req.url.path:
if "page=0" in str(req.url.query):
return httpx.Response(200, text=mini_search)
return httpx.Response(200, text="<html></html>")
if req.url.path.startswith("/trails/"):
return httpx.Response(200, text=_read_fixture_text("wikiloc-trail-page.html"))
if "downloadTrail.do" in req.url.path:
return httpx.Response(200, content=_read_fixture("wikiloc-track.gpx"))
return httpx.Response(404)
_patch_httpx(monkeypatch, handler)
region = dict(REGION_TSFO, sources=["enduro_russia", "wikiloc"])
db_path = _setup_env(monkeypatch, str(tmp_path), [ER_SOURCE, WL_SOURCE], [region])
# 1) сначала EnduroRussia
code1 = _run_pipeline(["--region", "tsfo_plus_chuvashia", "--source", "enduro_russia"])
assert code1 == 0
assert _count_tracks(db_path) == 1
# 2) затем Wikiloc
code2 = _run_pipeline(["--region", "tsfo_plus_chuvashia", "--source", "wikiloc"])
assert code2 == 0
# Должна быть 1 запись с обоими источниками
tracks = _all_tracks(db_path)
assert len(tracks) == 1, f"expected 1 merged record, got {len(tracks)}"
sources = json.loads(tracks[0]["sources_json"])
assert "enduro_russia" in sources
assert "wikiloc" in sources
ext_urls = json.loads(tracks[0]["external_urls_json"])
assert any("endurorussia.ru" in u for u in ext_urls)
assert any("wikiloc.com" in u for u in ext_urls)
# ─── IT-LIC-01 ──────────────────────────────────────────────────────────────
def test_it_lic_01_license_guard_blocks_proposed(monkeypatch, tmp_path):
"""IT-LIC-01: ADR со status: proposed → pipeline пропускает source с 'skipped_license'."""
# Создаём временный ADR с status: proposed
adr_dir = tmp_path / "docs" / "work-items" / "ET-008" / "06-adr"
adr_dir.mkdir(parents=True)
fake_adr = adr_dir / "ADR-FAKE-licensing.md"
fake_adr.write_text(
"---\n"
"type: adr\n"
"adr_id: ADR-FAKE\n"
"status: proposed\n"
"---\n\n"
"# Fake ADR for test\n"
)
er_source_proposed = dict(ER_SOURCE, license_adr="docs/work-items/ET-008/06-adr/ADR-FAKE-licensing.md")
def handler(req: httpx.Request) -> httpx.Response:
return httpx.Response(500) # не должно дойти
_patch_httpx(monkeypatch, handler)
# Pipeline берёт project_root относительно scripts/gps_collect.py.
# Нам надо подсунуть tmp_path как корень — самый простой способ: симлинком в tmp.
# Альтернатива: запускаем pipeline с cwd=tmp_path и патчим scripts module path.
# Но scripts.gps_collect использует __file__ → ../.. = project root.
# Подменим _check_license_adr через patch.
from scripts import gps_collect as collect_mod
real_check = collect_mod._check_license_adr
def patched_check(adr_path, project_root):
# Используем tmp_path как project_root для нашего fake ADR
return real_check(adr_path, str(tmp_path))
monkeypatch.setattr(collect_mod, "_check_license_adr", patched_check)
db_path = _setup_env(monkeypatch, str(tmp_path), [er_source_proposed], [REGION_TSFO])
exit_code = _run_pipeline(["--region", "tsfo_plus_chuvashia", "--source", "enduro_russia"])
# ET-009: license_guard выставляет has_error=True → exit_code=1
assert exit_code == 1
run = _last_pipeline_run(db_path)
assert run is not None
assert run["status"] == "skipped_license"
assert run["tracks_new"] == 0

View File

@@ -0,0 +1,264 @@
"""Unit tests for EnduroRussiaParser (ET-009).
Coverage:
- UT-ER-01: _parse_gpx success on valid GPX fixture
- UT-ER-02: _parse_gpx returns None on empty GPX
- UT-ER-03: bbox filter rejects out-of-bbox track
- UT-ER-04: MAPPING translates categories correctly
- UT-ER-05: base_url without dash preserved (regression R-4)
- UT-ER-06: pagination stops when fetched_so_far >= total
- UT-ER-07: HTTP 429 on /api/tracks → graceful return
- UT-ER-08: HTTP 429 on /api/tracks/{id}/gpx → graceful return, earlier tracks preserved
"""
import json
import os
from typing import Callable
import httpx
from src.api.gps_tracks.sources.enduro_russia import (
EnduroRussiaParser,
_bbox_intersects,
_parse_gpx,
)
from src.api.gps_tracks.sources import enduro_russia as er_module
FIXTURES_DIR = os.path.join(os.path.dirname(__file__), "..", "fixtures", "gps-tracks")
# Region bbox for ЦФО+Чувашия
BBOX_TSFO = (29.0, 49.5, 47.5, 60.0)
def _read_fixture(name: str) -> bytes:
with open(os.path.join(FIXTURES_DIR, name), "rb") as f:
return f.read()
def _make_config(**overrides) -> dict:
cfg = {
"id": "enduro_russia",
"base_url": "https://endurorussia.ru",
"rate_limit_sec": 0, # speed up tests
"user_agent": "test-agent",
"source_priority": 80,
}
cfg.update(overrides)
return cfg
def _patch_client(monkeypatch, handler: Callable[[httpx.Request], httpx.Response]) -> None:
"""Подменяет httpx.AsyncClient в модуле enduro_russia на клиент с MockTransport."""
transport = httpx.MockTransport(handler)
original = httpx.AsyncClient
def factory(*args, **kwargs):
kwargs["transport"] = transport
return original(*args, **kwargs)
monkeypatch.setattr(er_module.httpx, "AsyncClient", factory)
async def _collect_all(parser, bbox):
"""Собирает все треки из async-генератора."""
tracks = []
async for t in parser.collect(bbox, {}):
tracks.append(t)
return tracks
# ─── UT-ER-01 ───────────────────────────────────────────────────────────────
def test_ut_er_01_parse_gpx_track1_success():
"""UT-ER-01: _parse_gpx на track-1 → TrackInsert с points_count ≥ 10."""
content = _read_fixture("enduro-russia-track-1.gpx")
meta = {"name": "Маршрут Дмитровский", "difficulty": "hard", "created_at": "2024-08-15 12:30:00"}
track = _parse_gpx(
content,
track_id=1,
meta=meta,
source_id="enduro_russia",
base_url="https://endurorussia.ru",
source_priority=80,
mapping=EnduroRussiaParser.MAPPING,
)
assert track is not None
assert track.points_count >= 10
assert track.length_m > 0
assert track.min_lon < track.max_lon
assert track.min_lat < track.max_lat
assert track.external_url == "https://endurorussia.ru/tracks/1"
assert track.source_id == "enduro_russia"
# difficulty 'hard' → enduro
assert track.activity_type == "enduro"
# ─── UT-ER-02 ───────────────────────────────────────────────────────────────
def test_ut_er_02_parse_gpx_empty_returns_none():
"""UT-ER-02: _parse_gpx на пустом GPX → None."""
content = _read_fixture("enduro-russia-track-2.gpx")
track = _parse_gpx(
content,
track_id=2,
meta={},
source_id="enduro_russia",
base_url="https://endurorussia.ru",
source_priority=80,
mapping=EnduroRussiaParser.MAPPING,
)
assert track is None
# ─── UT-ER-03 ───────────────────────────────────────────────────────────────
def test_ut_er_03_bbox_filter_rejects_outside():
"""UT-ER-03: track-3 за пределами bbox ЦФО → _bbox_intersects False."""
content = _read_fixture("enduro-russia-track-3.gpx")
track = _parse_gpx(
content,
track_id=3,
meta={},
source_id="enduro_russia",
base_url="https://endurorussia.ru",
source_priority=80,
mapping=EnduroRussiaParser.MAPPING,
)
assert track is not None # парсится, но bbox не пересекается
intersects = _bbox_intersects(
(track.min_lon, track.min_lat, track.max_lon, track.max_lat),
BBOX_TSFO,
)
assert intersects is False
async def test_ut_er_03_collect_skips_out_of_bbox(monkeypatch):
"""UT-ER-03 (collect): out-of-bbox трек не yield-ится."""
api_data = json.loads(_read_fixture("enduro-russia-api-tracks-page1.json"))
# Оставим только один трек id=3 (вне bbox)
api_data = {"items": [it for it in api_data["items"] if it["id"] == 3], "total": 1, "page": 0}
def handler(req: httpx.Request) -> httpx.Response:
if req.url.path == "/api/tracks":
return httpx.Response(200, json=api_data)
if req.url.path == "/api/tracks/3/gpx":
return httpx.Response(200, content=_read_fixture("enduro-russia-track-3.gpx"))
return httpx.Response(404)
_patch_client(monkeypatch, handler)
parser = EnduroRussiaParser(_make_config())
tracks = await _collect_all(parser, BBOX_TSFO)
assert tracks == []
# ─── UT-ER-04 ───────────────────────────────────────────────────────────────
def test_ut_er_04_mapping_categories():
"""UT-ER-04: MAPPING маппит ключевые категории."""
m = EnduroRussiaParser.MAPPING
assert m["hard"] == "enduro"
assert m["soft"] == "enduro"
assert m["мото"] == "moto"
# 'unknown' нет в MAPPING → map_activity → 'other'
parser = EnduroRussiaParser(_make_config())
assert parser.map_activity("unknown") == "other"
# ─── UT-ER-05 ───────────────────────────────────────────────────────────────
def test_ut_er_05_base_url_no_dash_preserved():
"""UT-ER-05: base_url 'https://endurorussia.ru' сохраняется без замены."""
cfg = _make_config(base_url="https://endurorussia.ru")
parser = EnduroRussiaParser(cfg)
assert parser.config["base_url"] == "https://endurorussia.ru"
# Регрессия: проверим что в default fallback тоже без дефиса
parser_no_url = EnduroRussiaParser({"id": "enduro_russia"})
# default используется в collect() — но base_url берётся через get()
assert "enduro-russia" not in parser_no_url.config.get("base_url", "https://endurorussia.ru")
async def test_ut_er_05_collect_uses_no_dash_url(monkeypatch):
"""UT-ER-05 (collect): HTTP-запросы уходят на endurorussia.ru (без дефиса)."""
seen_hosts = []
def handler(req: httpx.Request) -> httpx.Response:
seen_hosts.append(req.url.host)
if req.url.path == "/api/tracks":
return httpx.Response(200, json={"items": [], "total": 0, "page": 0})
return httpx.Response(404)
_patch_client(monkeypatch, handler)
parser = EnduroRussiaParser(_make_config(base_url="https://endurorussia.ru"))
await _collect_all(parser, BBOX_TSFO)
assert any(h == "endurorussia.ru" for h in seen_hosts)
assert not any("enduro-russia" in h for h in seen_hosts)
# ─── UT-ER-06 ───────────────────────────────────────────────────────────────
async def test_ut_er_06_pagination_stops_at_total(monkeypatch):
"""UT-ER-06: collect() делает 1 запрос /api/tracks при total=5, items=5."""
api_data = json.loads(_read_fixture("enduro-russia-api-tracks-page1.json"))
list_calls = []
def handler(req: httpx.Request) -> httpx.Response:
if req.url.path == "/api/tracks":
list_calls.append(req.url.query.decode() if isinstance(req.url.query, bytes) else str(req.url.query))
return httpx.Response(200, json=api_data)
# GPX: вернём пустой (None) или валидный для track-1
return httpx.Response(200, content=_read_fixture("enduro-russia-track-2.gpx"))
_patch_client(monkeypatch, handler)
parser = EnduroRussiaParser(_make_config())
await _collect_all(parser, BBOX_TSFO)
assert len(list_calls) == 1, f"expected 1 /api/tracks call, got {len(list_calls)}: {list_calls}"
# ─── UT-ER-07 ───────────────────────────────────────────────────────────────
async def test_ut_er_07_http_429_on_tracks_list_graceful(monkeypatch):
"""UT-ER-07: 429 на /api/tracks → завершение без exception, 0 треков."""
def handler(req: httpx.Request) -> httpx.Response:
if req.url.path == "/api/tracks":
return httpx.Response(429, json={"error": "Too Many Requests"})
return httpx.Response(404)
_patch_client(monkeypatch, handler)
parser = EnduroRussiaParser(_make_config())
tracks = await _collect_all(parser, BBOX_TSFO)
assert tracks == []
# ─── UT-ER-08 ───────────────────────────────────────────────────────────────
async def test_ut_er_08_http_429_on_gpx_graceful(monkeypatch):
"""UT-ER-08: 429 на /api/tracks/{id}/gpx после первых OK → ранние треки сохраняются."""
# Соберём API ответ с двумя треками: 1 (OK) и 2 (429)
api_data = {
"items": [
{"id": 1, "name": "T1", "difficulty": "hard", "created_at": "2024-08-15 12:30:00"},
{"id": 2, "name": "T2", "difficulty": "hard", "created_at": "2024-08-15 13:00:00"},
],
"total": 2,
"page": 0,
}
def handler(req: httpx.Request) -> httpx.Response:
if req.url.path == "/api/tracks":
return httpx.Response(200, json=api_data)
if req.url.path == "/api/tracks/1/gpx":
return httpx.Response(200, content=_read_fixture("enduro-russia-track-1.gpx"))
if req.url.path == "/api/tracks/2/gpx":
return httpx.Response(429)
return httpx.Response(404)
_patch_client(monkeypatch, handler)
parser = EnduroRussiaParser(_make_config())
tracks = await _collect_all(parser, BBOX_TSFO)
# Ранний трек должен сохраниться
assert len(tracks) == 1
assert tracks[0].external_url.endswith("/tracks/1")

View File

@@ -0,0 +1,262 @@
"""Unit tests for WikilocParser (ET-009).
Coverage:
- UT-WL-01: _extract_track_paths returns ≥ 5 unique paths
- UT-WL-02: _extract_gpx_url with downloadTrail.do
- UT-WL-03: _extract_gpx_url fallback by track_id
- UT-WL-04: _extract_track_name from <h1>
- UT-WL-05: _parse_gpx success — activity_type='moto', source_id='wikiloc'
- UT-WL-06: MAPPING translates categories
- UT-WL-07: HTTP 403 on search → graceful stop
- UT-WL-08: HTTP 429 on track page → graceful stop, earlier preserved
- UT-WL-09: rate_limit_sec respected
- UT-WL-10: max_tracks_per_run cap stops yield exactly
"""
import asyncio
import os
from typing import Callable
import httpx
from src.api.gps_tracks.sources import wikiloc as wl_module
from src.api.gps_tracks.sources.wikiloc import (
WikilocParser,
_extract_gpx_url,
_extract_track_name,
_extract_track_paths,
_parse_gpx,
)
FIXTURES_DIR = os.path.join(os.path.dirname(__file__), "..", "fixtures", "gps-tracks")
BBOX_TSFO = (29.0, 49.5, 47.5, 60.0)
def _read_fixture(name: str) -> bytes:
with open(os.path.join(FIXTURES_DIR, name), "rb") as f:
return f.read()
def _read_fixture_text(name: str) -> str:
return _read_fixture(name).decode("utf-8")
def _make_config(**overrides) -> dict:
cfg = {
"id": "wikiloc",
"base_url": "https://www.wikiloc.com",
"rate_limit_sec": 0,
"user_agent": "test-agent",
"source_priority": 70,
"activity_filter": ["motorcycle"],
}
cfg.update(overrides)
return cfg
def _patch_client(monkeypatch, handler: Callable[[httpx.Request], httpx.Response]) -> None:
"""Подменяет httpx.AsyncClient в модуле wikiloc на клиент с MockTransport."""
transport = httpx.MockTransport(handler)
original = httpx.AsyncClient
def factory(*args, **kwargs):
kwargs["transport"] = transport
return original(*args, **kwargs)
monkeypatch.setattr(wl_module.httpx, "AsyncClient", factory)
async def _collect_all(parser, bbox):
tracks = []
async for t in parser.collect(bbox, {}):
tracks.append(t)
return tracks
# ─── UT-WL-01 ───────────────────────────────────────────────────────────────
def test_ut_wl_01_extract_track_paths():
"""UT-WL-01: _extract_track_paths возвращает ≥ 5 уникальных путей."""
html = _read_fixture_text("wikiloc-search-page1.html")
paths = _extract_track_paths(html)
assert len(paths) >= 5
assert len(set(paths)) == len(paths) # все уникальны
for p in paths:
assert p.startswith("/trails/")
# ─── UT-WL-02 ───────────────────────────────────────────────────────────────
def test_ut_wl_02_extract_gpx_url_downloadtrail():
"""UT-WL-02: _extract_gpx_url возвращает абсолютный URL для downloadTrail.do?id=X."""
html = '<html><body><a href="/wikiloc/downloadTrail.do?id=12345">GPX</a></body></html>'
url = _extract_gpx_url(html, "https://www.wikiloc.com", "12345")
assert url == "https://www.wikiloc.com/wikiloc/downloadTrail.do?id=12345"
# ─── UT-WL-03 ───────────────────────────────────────────────────────────────
def test_ut_wl_03_extract_gpx_url_fallback():
"""UT-WL-03: _extract_gpx_url fallback по track_id если нет явных ссылок."""
html = "<html><body><p>No GPX link here at all.</p></body></html>"
url = _extract_gpx_url(html, "https://www.wikiloc.com", "99999")
assert url == "https://www.wikiloc.com/wikiloc/downloadTrail.do?id=99999"
# ─── UT-WL-04 ───────────────────────────────────────────────────────────────
def test_ut_wl_04_extract_track_name():
"""UT-WL-04: _extract_track_name извлекает текст <h1>."""
html = "<html><body><h1>Test Trail</h1></body></html>"
assert _extract_track_name(html) == "Test Trail"
# Из фикстуры
html2 = _read_fixture_text("wikiloc-trail-page.html")
assert _extract_track_name(html2) == "Дмитровский лес"
# ─── UT-WL-05 ───────────────────────────────────────────────────────────────
def test_ut_wl_05_parse_gpx_success():
"""UT-WL-05: _parse_gpx на wikiloc-track.gpx → activity_type='moto'."""
content = _read_fixture("wikiloc-track.gpx")
track = _parse_gpx(
content,
track_id="12345678",
name="Дмитровский лес",
activity_type="moto",
source_id="wikiloc",
track_url="https://www.wikiloc.com/trails/motorcycle-enduro/dmitrovsky-loop-12345678",
source_priority=70,
)
assert track is not None
assert track.activity_type == "moto"
assert track.source_id == "wikiloc"
assert "wikiloc.com" in track.external_url
assert track.points_count >= 10
assert track.length_m > 0
# ─── UT-WL-06 ───────────────────────────────────────────────────────────────
def test_ut_wl_06_mapping_categories():
"""UT-WL-06: MAPPING маппит motorcycle/hiking/mtb."""
m = WikilocParser.MAPPING
assert m["motorcycle"] == "moto"
assert m["hiking"] == "hike"
assert m["mtb"] == "bicycle"
# ─── UT-WL-07 ───────────────────────────────────────────────────────────────
async def test_ut_wl_07_http_403_search_graceful_stop(monkeypatch):
"""UT-WL-07: 403 на странице поиска → graceful stop, 0 yields."""
def handler(req: httpx.Request) -> httpx.Response:
if "find.do" in req.url.path:
return httpx.Response(403, text="Forbidden")
return httpx.Response(404)
_patch_client(monkeypatch, handler)
parser = WikilocParser(_make_config())
tracks = await _collect_all(parser, BBOX_TSFO)
assert tracks == []
# ─── UT-WL-08 ───────────────────────────────────────────────────────────────
async def test_ut_wl_08_http_429_track_graceful_stop(monkeypatch):
"""UT-WL-08: 429 на 2-м треке → 1-й трек yield-нут, потом graceful stop."""
search_html = _read_fixture_text("wikiloc-search-page1.html")
trail_html = _read_fixture_text("wikiloc-trail-page.html")
gpx_bytes = _read_fixture("wikiloc-track.gpx")
call_count = {"track_page": 0}
def handler(req: httpx.Request) -> httpx.Response:
path = req.url.path
if "find.do" in path:
return httpx.Response(200, text=search_html)
if path.startswith("/trails/"):
call_count["track_page"] += 1
if call_count["track_page"] == 1:
return httpx.Response(200, text=trail_html)
# 2-й трек → 429
return httpx.Response(429, text="Too Many Requests")
if "downloadTrail.do" in path:
return httpx.Response(200, content=gpx_bytes)
return httpx.Response(404)
_patch_client(monkeypatch, handler)
parser = WikilocParser(_make_config())
tracks = await _collect_all(parser, BBOX_TSFO)
assert len(tracks) == 1
assert "wikiloc.com" in tracks[0].external_url
# ─── UT-WL-09 ───────────────────────────────────────────────────────────────
async def test_ut_wl_09_rate_limit_respected(monkeypatch):
"""UT-WL-09: asyncio.sleep вызывается между запросами с rate_limit_sec."""
trail_html = _read_fixture_text("wikiloc-trail-page.html")
gpx_bytes = _read_fixture("wikiloc-track.gpx")
def handler(req: httpx.Request) -> httpx.Response:
path = req.url.path
if "find.do" in path:
# вернём только одну ссылку, чтобы один трек обработался
mini_html = '<html><a href="/trails/motorcycle-enduro/12345">x</a></html>'
# Если page=0 → даём 1 трек, иначе пусто
if "page=0" in str(req.url.query):
return httpx.Response(200, text=mini_html)
return httpx.Response(200, text="<html></html>")
if path.startswith("/trails/"):
return httpx.Response(200, text=trail_html)
if "downloadTrail.do" in path:
return httpx.Response(200, content=gpx_bytes)
return httpx.Response(404)
_patch_client(monkeypatch, handler)
sleep_calls = []
real_sleep = asyncio.sleep
async def mock_sleep(sec):
sleep_calls.append(sec)
# вызываем реальный sleep с 0, чтобы быстро
await real_sleep(0)
monkeypatch.setattr(wl_module.asyncio, "sleep", mock_sleep)
parser = WikilocParser(_make_config(rate_limit_sec=10))
await _collect_all(parser, BBOX_TSFO)
# Между запросами должно быть несколько sleep'ов с аргументом ≥ 10
assert len(sleep_calls) >= 2, f"expected ≥ 2 sleep calls, got {sleep_calls}"
assert all(s >= 10 for s in sleep_calls), f"all sleep args must be ≥ 10, got {sleep_calls}"
# ─── UT-WL-10 ───────────────────────────────────────────────────────────────
async def test_ut_wl_10_max_tracks_per_run_cap(monkeypatch):
"""UT-WL-10: max_tracks_per_run=2, поиск выдаёт ≥ 5 треков → yield ровно 2."""
search_html = _read_fixture_text("wikiloc-search-page1.html")
trail_html = _read_fixture_text("wikiloc-trail-page.html")
gpx_bytes = _read_fixture("wikiloc-track.gpx")
def handler(req: httpx.Request) -> httpx.Response:
path = req.url.path
if "find.do" in path:
if "page=0" in str(req.url.query):
return httpx.Response(200, text=search_html)
return httpx.Response(200, text="<html></html>")
if path.startswith("/trails/"):
return httpx.Response(200, text=trail_html)
if "downloadTrail.do" in path:
return httpx.Response(200, content=gpx_bytes)
return httpx.Response(404)
_patch_client(monkeypatch, handler)
parser = WikilocParser(_make_config(max_tracks_per_run=2))
tracks = await _collect_all(parser, BBOX_TSFO)
assert len(tracks) == 2

View File

@@ -170,16 +170,34 @@ test('Filters: начальный colorMode === "source"', () => {
assert.equal(win.gpsTracksLayer.filters.colorMode, 'source');
});
test('Filters: начальные источники включают osm, enduro_russia, ttrails', () => {
test('Filters: начальные источники включают osm, enduro_russia, wikiloc, ttrails (ET-009)', () => {
const win = {};
loadGpsTracksModule(win);
const { sources } = win.gpsTracksLayer.filters;
assert.ok(Array.isArray(sources), 'filters.sources должен быть массивом');
assert.ok(sources.includes('osm'), 'отсутствует источник osm');
assert.ok(sources.includes('enduro_russia'), 'отсутствует источник enduro_russia');
assert.ok(sources.includes('wikiloc'), 'отсутствует источник wikiloc');
assert.ok(sources.includes('ttrails'), 'отсутствует источник ttrails');
});
// ET-009 — REQ-F-13/REQ-F-14: цвета и атрибуция новых источников
test('ET-009: GPS_SOURCE_COLORS содержит цвет для wikiloc', () => {
const { GPS_SOURCE_COLORS } = loadGpsTracksModule();
assert.ok(GPS_SOURCE_COLORS.wikiloc, 'GPS_SOURCE_COLORS.wikiloc отсутствует');
assert.match(GPS_SOURCE_COLORS.wikiloc, /^#[0-9a-fA-F]{6}$/);
});
test('ET-009: цвета osm, enduro_russia, wikiloc различны', () => {
const { GPS_SOURCE_COLORS } = loadGpsTracksModule();
const colors = new Set([
GPS_SOURCE_COLORS.osm,
GPS_SOURCE_COLORS.enduro_russia,
GPS_SOURCE_COLORS.wikiloc,
]);
assert.equal(colors.size, 3, 'цвета osm/enduro_russia/wikiloc должны быть уникальны');
});
test('Filters: enabled=false при инициализации', () => {
const win = {};
loadGpsTracksModule(win);