Rubber Wheels and Cron Jobs: Putting Your Council's Bin Data on Autopilot

Bin day is one of those small, recurring facts of British life that still, somehow, runs on human memory. The lorry arrives at 6:42, the recycling is still in the kitchen, and you spend the rest of the week explaining yourself to a neighbour. It's 2026. This is a solved problem — it just happens to be solved with a scraper, a SQLite file, and a systemd timer on a Raspberry Pi that's already doing three other jobs.

The data is a mess, and that's fine

Council waste data in the UK comes in roughly four flavours. There's the PDF calendar, one page per street, frequently a scan of something photocopied in 2011. There's the web form that takes a postcode and returns a list of addresses — behind it is usually a UPRN lookup against the council's LLPG. There's the Bartec Collective-style API, which quite a few authorities dress up in their own branding and which returns clean JSON if you ask it nicely. And then there's the "we'll tell you on the day" brigade, whose entire digital strategy is a Facebook page.

Before you write a line of Python, open the council's bin page, hit F12, and watch the Network tab while you type your postcode. Nine times out of ten the page is a thin skin over a JSON endpoint. Note the request shape: usually a POST with a postcode, then a GET with a UPRN. That UPRN — Unique Property Reference Number — is the closest thing the UK has to a universal address key, and it's the one thing worth caching.

Be a good citizen about it. One lookup a day per property is plenty. Set a real User-Agent with a contact address, respect robots.txt where it applies, and don't retry in a tight loop when the council's servers fall over at 3am during their patching window.

Normalise early, or suffer later

The temptation is to keep the raw payload and parse it at notification time. Don't. Councils redesign these pages about as often as they change their recycling rules, and when they do you want the breakage to happen in one place, at fetch time, where you can see it — not silently, three days later, when nobody gets told about the garden waste.

Flatten everything into a single table:

CREATE TABLE collections (
  uprn        TEXT NOT NULL,
  service     TEXT NOT NULL,      -- refuse, recycling, garden, food
  due_date    TEXT NOT NULL,      -- ISO 8601, YYYY-MM-DD
  fetched_at  TEXT NOT NULL,
  source_url  TEXT,
  PRIMARY KEY (uprn, service, due_date)
);

Plus a fetch_log table with a row per run — timestamp, HTTP status, number of collections parsed. If that count drops to zero, that's your canary. A scraper that returns nothing looks identical to a scraper that's working perfectly, right up until the morning you miss the lorry.

Wrapping the mess in one function

Keep the council-specific code behind a single interface: get_collections(uprn) -> list[Collection]. Everything downstream — storage, scheduling, notifications — talks to that and nothing else. When your council moves from HTML to a shiny new portal, you rewrite thirty lines and the rest of the stack doesn't notice.

Where the site is JavaScript-heavy, resist reaching for a headless browser. Try the underlying API first; if you genuinely can't, Playwright in a container is acceptable but it's a fat dependency on a Pi with 2GB of RAM. Cache aggressively, and pin your parsing with a couple of saved fixture files so you can test after a redesign without hitting the live site.

Two timers, not one

Use systemd timers rather than cron, if only because systemctl list-timers tells you exactly what's happening without you having to reverse-engineer a crontab someone else wrote.

The fetch timer runs once a day, early, with a bit of jitter:

[Timer]
OnCalendar=*-*-* 05:30:00 Europe/London
RandomizedDelaySec=20m
Persistent=true

Persistent=true matters more than people think — it catches up on a missed run if the box was off. And naming the timezone explicitly means you don't have to think about the clocks going back in October, which is more than can be said for most of us.

The notify timer runs in the evening, checks the database for anything due tomorrow, and sends a message:

[Timer]
OnCalendar=*-*-* 19:00:00 Europe/London

Two separate units means a failed scrape doesn't stop the reminder, and a failed reminder doesn't stop the scrape. Add OnFailure= to both so you get told when the thing that tells you things has stopped telling you things.

Picking a channel

ntfy is the easy answer if you want push without a cloud account: run it yourself in Docker, publish to a topic, and curl -d "Recycling tomorrow, green bin" ntfy.sh/your-topic from the Pi. It costs nothing and it works on a locked phone screen, which is the whole point.

Telegram is the pragmatic option if you already live in it — a bot token, a chat ID, one POST. Email via msmtp is the boring, robust one; it'll still be working in a decade, and it composes nicely with a Sunday evening weekly digest.

The edge cases that will catch you

Bank holidays shift collections, and councils are inconsistent about whether they tell you. Christmas is a fortnight of chaos. Garden waste subscriptions lapse in winter and resume in spring, and food waste might be weekly in one borough and mixed with refuse in the next. Some authorities publish "we'll come back the next working day" as prose rather than data.

Log a reported flag for missed collections, keep a small table of known bank-holiday shifts for your nation, and — most importantly — make the bot say something when it has nothing to say. A weekly "nothing unusual this week" beats a suspicious silence every time.

It's a daft little project. But there's real satisfaction in a notification at 7pm that means you never again sprint down the drive in a dressing gown at dawn.