How to Use Usenet for Research and OSINT Investigations

  • 14/11/2024
  • 12-minute read

OSINT, especially the SOCMINT (Social Media Intelligence) specialty, is often focused on widely used social media platforms like Twitter/X, Facebook, etc. However, there are lesser-known alternatives, such as Usenet, that can be a goldmine for OSINT investigations.

This article is not only for OSINT practitioners but also for researchers and internet historians who want to understand the specific ins and outs of Usenet research.

What is Usenet?

Before diving into how to use Usenet for OSINT, let’s briefly cover what it is. Usenet, also previously known as “NetNews,” is a decentralized network of servers that provide access to the Usenet network. Usenet is not a product; much like Email or IRC, it is an open internet standard (NNTP) that anyone can use. In many ways, you can think of it as the message board cousin to Email.

Usenet has been operating non-stop since 1979, before the Internet and before the institutions that started it (UNC and Duke University) even had access to the ARPAnet. Since Usenet articles (Usenet refers to messages or posts as “articles”) are plain text, archives from that time until today still exist and are available. From the 1980s until the early 2000s, Usenet was the online message forum for the Internet.

How Does It Work and How Is It Organized?

Each server sends and receives articles in a standard plain text format not unlike Email. Each message is then categorized into one of hundreds of hierarchies, which are then broken down into individual discussion groups called newsgroups. Currently, there are over 40,000 newsgroups on Usenet. Within these newsgroups, you’ll find thousands of topics, each focused on a particular subject. Users post articles, questions, or files to these groups, and anyone with access to a Usenet server can read or respond.

Many hierarchies are location-based, such as tor.* (Toronto) or uk.* (United Kingdom), while others are more topical, such as comp.* (computing), sci.* (science), rec.* (recreation), and alt.* (alternative topics). Within these categories, you’ll find thousands of newsgroups, each focused on particular subjects. Users post articles, questions, or files to these groups, and anyone with access to a Usenet server can read or respond.

Unlike federated services like Mastodon or Lemmy, every Usenet server receives every message unless they specifically request not to receive articles for some newsgroups (more about that later). This means that if you join or create a news server, you will have access to all incoming articles from every newsgroup on Usenet that can be shared with you.

Anatomy of a Usenet Article

From -5182564358058015669
Path: gmdzi!unido!fauern!ira.uka.de!sol.ctr.columbia.edu!zaphod.mps.ohio-state.edu!wupost!uunet!mcsun!news.funet.fi!hydra!klaava!torvalds
From: torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds)
Newsgroups: comp.os.minix
Subject: What would you like to see most in minix?
Summary: small poll for my new operating system
Keywords: 386, preferences
Message-ID: <1991Aug25.205708.9541@klaava.Helsinki.FI>
Date: 25 Aug 91 20:57:08 GMT
Organization: University of Helsinki
Lines: 20


Hello everybody out there using minix -

I'm doing a (free) operating system (just a hobby, won't be big and
professional like gnu) for 386(486) AT clones.  This has been brewing
since april, and is starting to get ready.  I'd like any feedback on
things people like/dislike in minix, as my OS resembles it somewhat
(same physical layout of the file-system (due to practical reasons)
among other things). 

I've currently ported bash(1.08) and gcc(1.40), and things seem to work. 
This implies that I'll get something practical within a few months, and
I'd like to know what features most people would want.  Any suggestions
are welcome, but I won't promise I'll implement them :-)

		Linus (torvalds@kruuna.helsinki.fi)

PS.  Yes - it's free of any minix code, and it has a multi-threaded fs. 
It is NOT protable (uses 386 task switching etc), and it probably never
will support anything other than AT-harddisks, as that's all I have :-(.

This is a rather famous example of a Usenet article sent by Linus Torvalds, the creator of Linux, discussing his new operating system as a university student in 1991. Usenet hasn’t changed greatly in how it functions since the early days, so we’ll use this article as an example of what to look for.

The Header

From -5182564358058015669
Path: gmdzi!unido!fauern!ira.uka.de!sol.ctr.columbia.edu!zaphod.mps.ohio-state.edu!wupost!uunet!mcsun!news.funet.fi!hydra!klaava!torvalds
From: torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds)
Newsgroups: comp.os.minix
Subject: What would you like to see most in minix?
Summary: small poll for my new operating system
Keywords: 386, preferences
Message-ID: <1991Aug25.205708.9541@klaava.Helsinki.FI>
Date: 25 Aug 91 20:57:08 GMT
Organization: University of Helsinki
Lines: 20
  • The Path tells you which route the article took from the Usenet server at the University of Helsinki to the server where this article was originally received. This may not be very useful for most people today, but if you were trying to generate a list of Usenet servers that were available at one time, you could use this information to start generating a map.
  • From is the email address and name of the sender. Traditionally, there has never been anything to force this information to be legitimate, with a few exceptions.
  • Newsgroups is a list of groups that the article was sent to. Cross-posting articles to multiple newsgroups used to be considered bad “netiquette” and could land you in hot water with your server administrator if someone complained. Today, such violations are rarely enforced.
  • Message-ID is the article’s fingerprint. With this, you could theoretically find this article on any Usenet server that still had that article in storage. Because this is a rather famous article, if you Google, 1991Aug25.205708.9541@klaava.Helsinki.FI you will find it republished on many different websites.
  • Summary, Subject, Keywords, and Date are all pretty self-explanatory.

The Body

Hello everybody out there using minix -

I'm doing a (free) operating system (just a hobby, won't be big and
professional like gnu) for 386(486) AT clones.  This has been brewing
since april, and is starting to get ready.  I'd like any feedback on
things people like/dislike in minix, as my OS resembles it somewhat
(same physical layout of the file-system (due to practical reasons)
among other things).
...

The body of the article is typically plain text, and prior to 2000, they were mostly just ASCII with only a maximum of 80 characters wide per line. Occasionally, if you are looking for non-English articles, you will find other typesetting formats, but be prepared for them to look odd if your computer doesn’t know how to interpret non-ASCII characters and diacritical marks.

Binary Articles?

There is a system called yEnc, which converts binary data (movies, images, PDFs, etc.) into a format that can be split between multiple Usenet articles as the article’s body. Many of the newsgroups that carry these articles begin with alt.binaries.*, but not all of them.

The truth is that today, most traffic on Usenet is illicit file-sharing. There are specialty software applications that are used to download movies, etc. The articles that contain these files are not independently archived. In the past, this has been used as a way to also share CSAM, which is one of the many reasons why Usenet is not used much anymore. At one time, Usenet access was a standard part of an ISP’s service package.

How Do I Get Access to Usenet Records?

  • Google Groups: In 2001, Google acquired DejaNews, the largest Usenet provider and archiver at that time. In 2024, Google stopped providing access to incoming Usenet articles and removed the ability to post directly to Usenet.

    The historical records are still available, but the search function is poor, and headers are missing except for the date, subject, and the sender’s name.

  • Internet Archive: The Internet Archive holds nearly a terabyte (or more) of Usenet archives. These archives are usually organized by hierarchy and then by newsgroup. Each newsgroup is in .mbox format. .mbox is a text-only format that can be read with a text editor, searched using grep, or opened with email clients like Mozilla Thunderbird, with the aid of a plugin.

    The Internet Archive’s collection typically covers content up to early 2013 and has limited coverage after that.

  • The UTZOO Tapes: These are archives of the earliest Usenet articles from 1981 to 1991. They were once available on the Internet Archive but were removed at the request of a user. However, they can still be found on other websites with some searching.

  • Paid Usenet Providers: Paid providers often claim to have 10 years or more of article retention. However, this is usually not entirely accurate. It is an open secret that these providers primarily offer access to illicit materials in binary groups. Their retention of non-binary articles is limited, but some groups may still contain useful information.

  • Free Usenet Providers: Free services like eternal-september.org typically have archives spanning a few years. However, they do not carry binary newsgroups but are a great free source for monitoring existing newsgroups.

Why Usenet is Useful for research and OSINT?

While it might seem outdated compared to modern social media platforms, Usenet has several features that make it a valuable resource for research:

  1. Historical Data: Usenet has been around for decades, meaning it contains a wealth of historical discussions, opinions, and data. This is especially useful for investigations that require a long-term perspective or historical context.

  2. Niche Communities: Many Usenet newsgroups are home to niche communities that discuss specialized topics in depth. These can include everything from software development to obscure hobbies, and even controversial subjects that might not be as openly discussed on mainstream platforms.

  3. Anonymity: While not entirely anonymous, Usenet users often post under pseudonyms, making it a less censored space where people might share information more freely. This can be useful for gathering unfiltered opinions or uncovering discussions that might not happen on more public-facing platforms. Many Usenet providers, both paid and free, do not have the infrastructures in place to keep out users using Tor or VPNs. While Google required an actual Google account to post to Usenet, paid and free services do no require it. This make Usenet ideal for anonymous discussions.

  4. File Sharing: Today, Usenet is mostly known for ilicit file-sharing, with users posting everything from software to documents. This can be a treasure trove for finding leaked data, rare documents, or software not easily accessible elsewhere.

How to Access Usenet

To use Usenet, you’ll need access to a Usenet server and a newsreader, a program that allows you to browse and download content from newsgroups. Here’s a quick overview of the steps:

  1. Choose a Usenet Provider: Usenet access typically requires an account with a service that provides access to Usenet servers. Both paid and free providers are available. Paid providers usually sell access either monthly or by blocks of bandwidth. This is because they primarily sell access to pirated material rather than access to newsgroups for discussion.

  2. Install a Newsreader: A newsreader is the software you’ll use to browse Usenet. The most common option, available almost everywhere, is Mozilla Thunderbird. On Linux, Pan is fantastic. For those who prefer the command line, SLRN is another option.

  3. Subscribe to Newsgroups: Once you’re set up, you can start searching for relevant newsgroups. Most newsreaders allow you to search by keyword, making it easier to find discussions related to your investigation.

Types of Research Where Usenet Excels

With an understanding of how to access Usenet, it’s essential to explore specific research scenarios where Usenet can be particularly advantageous:

  1. Historical Research: Usenet’s vast archives are a treasure trove for historical research. Whether you’re delving into the early development of a particular technology, tracing the progression of a political movement, or investigating the roots of a conspiracy theory, Usenet offers unique historical perspectives that are often difficult to find elsewhere.

  2. Technology and Software Development: Many Usenet newsgroups are rich in technical discussions, ranging from software development to cybersecurity and hacking. For researchers focused on cyber threats, software vulnerabilities, or the evolution of specific tools, Usenet provides in-depth discussions and access to original source code that may not be available on mainstream platforms.

  3. Counter-Culture and Subversive Movements: Due to its decentralized nature and anonymity, Usenet has long been a gathering place for counter-culture and subversive movements. If your research involves exploring fringe communities, underground movements, or controversial topics, Usenet can reveal uncensored opinions and discussions that are often absent from more regulated platforms.

  4. Intellectual Property and Copyright Infringement: Usenet hosts a variety of groups dedicated to sharing pirated software, music, movies, and other media. Monitoring these groups can provide crucial insights into distribution networks and the extent of intellectual property infringement, making Usenet a useful tool for those investigating these issues.

Final Notes

Things to be aware of:

Usenet, at its core, is largely unmoderated. While moderated newsgroups do exist, it has long been assumed that users would manage “trolls” by employing software tools known as “killfiles.”

Usenet is also riddled with spam. When Google took over DejaNews in 2001, their only attempt at controlling spam was requiring users to have a Google account. They did not implement Captchas or other mechanisms to block bot accounts. This lack of spam prevention contributed heavily to Usenet’s decline in the mainstream.

When researching Usenet, be prepared to encounter a high ratio of spam to legitimate messages, especially in articles posted after 2001. Spam did exist before this period, but not at the scale that followed.

Since Google no longer allows access to post to the Usenet, the amount of spam has decreased dramatically, with the past six months being almost entirely spam-free.

However, there has been a mild resurgence in Classic (non file-sharing) Usenet activity over the past few years. There has been renewed interest in many older technologies from retro video games to retro computrers, leading many to rediscover Usenet. Real conversations are happening in newsgroups that have sat dorment for years.

For the Researcher and OSINT analyst:

While Usenet may seem like a relic of the early internet, it remains a valuable resource for researchers and OSINT analysts. Its combination of historical depth, niche communities, and uncensored discussions makes it particularly useful for investigations that require more than just surface-level data. By understanding how to navigate Usenet effectively, OSINT analysts can unlock a wealth of information that might otherwise remain hidden.

Whether you’re conducting historical research, exploring subversive movements, or analyzing technical discussions, Usenet has the potential to provide unique insights that can significantly enhance your intelligence efforts.