RSSproObce: who reads municipal websites and why
Purpose
The RSSproObce program reads the public official notice boards (úřední desky) and news pages (aktuality) of Czech municipalities and turns them into feeds (RSS, Atom), so that people learn in time about new documents from their municipality. It is developed in the project „Vývoj softwarové platformy pro automatizované generování a inteligentní (AI) agregaci informačních kanálů (RSS) z webů českých obcí“ (development of a software platform for automated generation and AI aggregation of feeds from the websites of Czech municipalities).
Who is behind it
The project is carried out by MASTER IT Technologies, a.s. (MITT), the beneficiary of innovation voucher CZ.01.01.01/05/24_043/0010262. The platform is developed and validated for MITT by VŠB – Technical University of Ostrava (VŠB-TUO).
What is read
- A municipality's official notice board and news pages on its public website, and the documents they link.
- The municipality's own RSS feeds and open data (OFN), where it publishes them.
- A notice board that a municipality with extended powers (ORP) publishes for it.
Only pages available without login are read. The program never logs in, fills in forms or accepts terms of use. Personal data of private individuals are not taken into the feeds, and an item leaves the feeds at the latest when its document is taken down from the notice board.
How fast it reads
- One connection at most to one website and to one server IP address at a time.
- At least 5 s between two requests to the same website or the same address, longer where the website asks for it.
- robots.txt is read and recorded, and its Crawl-delay is honoured.
- A 503 answer with a Retry-After header is waited out.
- Conditional requests (ETag, Last-Modified) are used, so an unchanged page is not transferred again.
- Each website has a daily limit on the number of requests.
Any refusal (HTTP 401, 403, 407, 429 or 451, a captcha or challenge page, a redirect to a login page, or a blocked address) stops reading that website. A person then reviews the case, and reading is never resumed under another identity or from another address.
How the program identifies itself
Every request carries the User-Agent header RSSproObce/1.0 (+https://crawler.rss.databridge.city/) and a From header with the address rss-crawler@hub440.cz. The program uses no other identity.
Where it reads from
All requests leave from one fixed address, 3.78.33.17. The address is not changed and no proxy is used.
Objections and contact
Send questions and objections to rss-crawler@hub440.cz.
- A municipality or a website operator can ask for reading to be slowed down or stopped. The rate is reduced or reading paused on the working day it is received; the request is then reviewed.
- Anyone who objects to the processing of data that concern them can raise an objection. It is dealt with within one working day of receipt.