1. Forum
  2. FidoNet
  3. uk.railway

  • Re: Usenet-Rewind: search engine for Usenet messages back to 1981

    From Chris J Dixon@3:633/10 to All on Tuesday, September 08, 2026 13:47:03
    Over in alt.folklore.computers

    Craig Stadler wrote:

    I spent the last year gathering and normalizing as many resources for
    Usenet messages and built a search engine over public Usenet
    discussions going back to 1981. Posting here because this seemed like
    the one group that would actually care.

    https://www.usenet-rewind.com/

    Since Google Groups stopped indexing new Usenet content a
    while back, and search over its historical archive has always been
    rough- I wanted something that treats this as a real research
    archive -- full-text search, date-range/filtering,etc and most
    importantly as much Usenet data as possible going back as far as I could >find.

    The index currently holds (to date) roughly 980 million messages and is
    still growing. Sources are archival backups, data donations, and ongoing >crawls of several thousand still-active public NNTP servers. Binary and >yEnc-encoded content is stripped where possible to keep the index
    focused on text.

    The index was built using Apache Solr 10, MariaDB 12 (rocksdb), Ubuntu >22.04.5 LTS & custom Python scripts, all on nvme disk.

    This is public archival content, same legal basis as DejaNews and
    Google Groups before it. There's a removal process for anyone who
    wants their own posts taken down, and author contact info is masked
    by default.

    I thought that some might find it interesting.

    Chris
    --
    Chris J Dixon Nottingham UK
    chris@cdixon.me.uk @ChrisJDixon1

    Plant amazing Acers.

    --- PyGate Linux v1.5.19
    * Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)
  • From Anna Noyd-Dryver@3:633/10 to All on Tuesday, September 08, 2026 16:13:46
    Chris J Dixon <chris@cdixon.me.uk> wrote:
    Over in alt.folklore.computers

    Craig Stadler wrote:

    I spent the last year gathering and normalizing as many resources for
    Usenet messages and built a search engine over public Usenet
    discussions going back to 1981. Posting here because this seemed like
    the one group that would actually care.

    https://www.usenet-rewind.com/

    Since Google Groups stopped indexing new Usenet content a
    while back, and search over its historical archive has always been
    rough- I wanted something that treats this as a real research
    archive -- full-text search, date-range/filtering,etc and most
    importantly as much Usenet data as possible going back as far as I could
    find.

    The index currently holds (to date) roughly 980 million messages and is
    still growing. Sources are archival backups, data donations, and ongoing
    crawls of several thousand still-active public NNTP servers. Binary and
    yEnc-encoded content is stripped where possible to keep the index
    focused on text.

    The index was built using Apache Solr 10, MariaDB 12 (rocksdb), Ubuntu
    22.04.5 LTS & custom Python scripts, all on nvme disk.

    This is public archival content, same legal basis as DejaNews and
    Google Groups before it. There's a removal process for anyone who
    wants their own posts taken down, and author contact info is masked
    by default.

    I thought that some might find it interesting.


    Sounds fantastic - hopefully a worthy replacement for Deja News which has subsequently been wrecked by Google.

    I remember the original project to collate old usenet posts from many
    different private collections and sources, for the Deja News project.

    --- PyGate Linux v1.5.19
    * Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)