SwissDelphiCenter.ch tips scraper
=================================

The SwissDelphiCenter.ch site went offline in 2026. These two Python scripts read the
Delphi tips it used to host back out of the Wayback Machine and turn them into a
collection that the GExperts Code Librarian can import.

Do with these scripts whatever you like. They come without warranty of any kind.

if you only want the data, see here:
https://blog.dummzeuch.de/2026/07/17/swissdelphicenter-delphi-tips/

scrape_tips.py
--------------
Downloads the tip pages. Before you run it, open it and put your email address into the
CONTACT constant near the top: it goes into the User-Agent of every request so that the
Internet Archive knows who is calling. The script refuses to start without it.

    python3 scrape_tips.py              # everything, about 4.5 hours for 1587 tips
    python3 scrape_tips.py --index-only # only build the index of tips
    python3 scrape_tips.py --limit 5    # download 5 tips, to try it out
    python3 scrape_tips.py --repair     # refetch pages that show no code
    python3 scrape_tips.py --delay 20   # be even friendlier (default: 10 seconds)

It waits 10 seconds between requests, and it is resumable: pages already downloaded are
skipped, so you can stop it with Ctrl+C and start it again later.

Rather than follow the links on the site, it asks the Wayback CDX index which pages exist,
because the category listings paginate and their later pages are hardly archived at all.
It also has to sort out the snapshots that returned HTTP 200 but contain the site's
farewell page, or a page whose content did not render.

Output, all below Tips/:
    tips/<id>.html      one file per tip
    topics/*.html       the category listings, used for the categories and titles
    tips_index.json     id, title, category, date, hits and snapshots per tip

convert_tips.py
---------------
Turns what the scraper downloaded into the collection:

    python3 convert_tips.py

It writes Collection/ and packs it into swissdelphicenter-tips.zip, in the format of the
DelphiDabbler Code Snippets Collection (categories.ini, one ini per category and one .dat
per tip), which is what the Code Librarian's "Import Collection" reads.

Requirements
------------
Python 3 and the requests library (python3 -m pip install requests). Only the scraper
needs it, the converter works offline.
