{"$schema":"https://ejfox.com/schema/page-twin@1.json","kind":"blog_post","url":"https://ejfox.com/blog/2025/paramilitary-telegram-leaks-001","json_url":"https://ejfox.com/blog/2025/paramilitary-telegram-leaks-001.json","generator":"ejfox.com/json-twin@1","data":{"cacheVersion":"2026-05-13-gear-cards","html":"

Paramilitary Telegram Leaks: From Dump To Data

\n

The other day I was perusing Mastodon, as one does, and noticed a post from journalist and author Micah Lee saying he was exploring a 200GB dump from American paramilitary groups.

\n

This leak was made possible through the incredible bravery of John Williams who infiltrated these groups and exfiltrated all this data. He is currently crowdfunding $10k in part, to \"move safe houses and to change his identity\" and he also has a Patreon where you can support him

\n
\n\"Terminal\n
Terminal showing system processes and a shell script that fetches tasks, recent repos, Mastodon posts, and calculates sunset time for New York City.
\n
\n

I hadn't heard of such a dump before, and immediately sent Micah an email offering to help with any dataviz.

\n
\n\"Email\n
Email from journalist EJ Fox proposing interactive graphics collaboration to visualize a large paramilitary dataset using network analysis techniques.
\n
\n

From SQLite to Parquet: Processing Telegram Leaks for Fast Web Visualization

\n

The Challenge

\n
    \n
  1. Extract the relevant data
    2. \n
  2. Convert it to a web-friendly format
    3. \n
  3. Make it quickly accessible for visualization
    4. \n
  4. Cache it for fast loading across browsers
    5. \n
\n
\n\"Torrent\n
Torrent client downloading conspiracy-related files including election terrorism claims, state leaders chats, and constitutional IRS documents.
\n
\n

The Workflow

\n

We start with the repo Micah started that takes the raw dump of .html exports from Telegram and transforms them into a structured sqlite db we can work with.

\n

Once you get that repo setup you can just do

\n
poetry run tasks build-db ~/RAW-LEAK-LOCATION`
\n

… and you get a sqlite database in your output folder.

\n

Step 1: Export from SQLite to CSV

\n

First, I extract the messages table to a .csv using SQLite's command-line tool:

\n
sqlite3 -header -csv output/data.db 'select * from messages;' > output/telegram_chats.csv
\n

This gives us a clean CSV file with all our message data, ready for the next transformation- we could also play around with something like csvkit or visitdata

\n

Step 2: Convert CSV to Parquet

\n

Parquet is a columnar storage format that's incredibly efficient for analytical queries and works brilliantly in browser environments. I use DuckDB (an in-process analytical database) to handle the conversion:

\n
duckdb -c \"INSTALL parquet; LOAD parquet; CREATE TABLE temp AS SELECT * FROM read_csv('output/telegram_chats.csv'); COPY temp TO 'output/telegram_chats.r5.parquet' (FORMAT PARQUET, COMPRESSION 'SNAPPY');\"
\n

The compression with Snappy ensures it's both compact and fast to decompress.

\n

Step 3: Host on R2 with Cloudflare Caching

\n

To make the data available globally with minimal latency (and cost to me) we throw it on Cloudflare R2 for hosting

\n
# R2 Configuration\nR2_PARQUET_URL=https://r2.ejfox.com/para-leaks/telegram_chats.r5.parquet\n
\n

The Frontend

\n

I've developed a Nuxt / regl-scatterplot frontend application to interact with this data, available on GitHub:

\n

https://github.com/ejfox/paramilitary-leaks-frontend

\n
\n \n
\n

You can view it the front-end https://para-leaks.ejfox.com

\n

Data Exploration Examples

\n

Now that we've chopped and screwed our data between sqlite, parquet, and .csv – depending on what application we want to use it for, we can start asking interesting questions from the sqlite database with datasette.

\n

For example, to find the top senders of media you could run:

\n
select id, timestamp, sender, text, media_note, media_filename, filename, group_chat_id \nfrom messages \nwhere \"media_filename\" is not null \norder by media_filename desc\nlimit 101
\n
\n\"Datasette\n
Datasette query for top media senders
\n
\n

Because so much information is in these media files, I took a stab at having Gemini write data transcriptions of each photo so they could be indexed for text search later.

\n
\n\"Man\n
Man in Santa Claus costume making peace sign in a lobby, wearing traditional red suit with white fur trim and black belt.
\n
\n
\n\"CSV\n
CSV spreadsheet cataloging visual media assets with detailed descriptions of photographs depicting outdoor scenes, people, animals, and landscapes.
\n
\n

This experiment went pretty well, but I shelved it to come back to, there's enough to analyze here without involving LLMs and prompt engineering.

\n

Building Custom Visualizations

\n

The real power of this approach is that we can build all sorts of different front-ends or visualizations on top of this data. The Parquet format is super-fast in the browser, making it ideal for interactive visualizations. We've taken a 200GB leak and turned it into something we can explore in the browser.

\n

Some ideas I'm exploring:

\n\n\n \n\n

For more information on this topic, I recommend reading Micah Lee's excellent article:

\n

Exploring the Paramilitary Leaks

\n

ProPublica: The Militia and the Mole (Jan. 4, 2025)

","title":"Processing Telegram Leaks for Fast Web Visualization","metadata":{"date":"2025-03-09T17:49:01.000Z","modified":"2025-03-16T01:25:05.000Z","title":"Processing Telegram Leaks for Fast Web Visualization","dek":"In which I describe my workflow for transforming a Telegram database dump into a web-friendly format for analysis and visualization","inprogress":true,"tags":["data","dataviz","journalism","politics"],"words":782,"images":6,"imageDetails":{"total":6,"cloudinary":6,"withDimensions":0},"links":21,"codeBlocks":5,"headers":{"h2":2},"toc":[{"text":"Paramilitary Telegram Leaks: From Dump To Data","slug":"paramilitary-telegram-leaks-from-dump-to-data","level":"h2","children":[]},{"text":"From SQLite to Parquet: Processing Telegram Leaks for Fast Web Visualization","slug":"from-sqlite-to-parquet-processing-telegram-leaks-for-fast-web-visualization","level":"h2","children":[{"text":"The Challenge","slug":"the-challenge","level":"h3","children":[]},{"text":"The Workflow","slug":"the-workflow","level":"h3","children":[]},{"text":"The Frontend","slug":"the-frontend","level":"h3","children":[]},{"text":"Data Exploration Examples","slug":"data-exploration-examples","level":"h3","children":[]},{"text":"Building Custom Visualizations","slug":"building-custom-visualizations","level":"h3","children":[]}]}],"type":"post"}},"_links":{"self":"https://ejfox.com/blog/2025/paramilitary-telegram-leaks-001.json","html":"https://ejfox.com/blog/2025/paramilitary-telegram-leaks-001","index":"/blog.json"}}