Initial commit: Thread Ripper containerized application
This commit is contained in:
airent106 committed
commit
a648a59c0f
16 files changed
+2213
No files matched your search
@@ -0,0 +1,12 @@
|
||||
__pycache__
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
.Python
|
||||
env/
|
||||
venv/
|
||||
.venv/
|
||||
data/
|
||||
.git
|
||||
.gitignore
|
||||
.env
|
||||
+40
@@ -0,0 +1,40 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
env/
|
||||
venv/
|
||||
.venv/
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
|
||||
# Thread Ripper runtime data
|
||||
data/threads/
|
||||
data/*.md
|
||||
data/*.txt
|
||||
data/*.html
|
||||
data/*.json
|
||||
|
||||
# IDE / Editor
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# Environment
|
||||
.env
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
FROM mcr.microsoft.com/playwright/python:v1.49.0-noble
|
||||
|
||||
# Set environment variables
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
PORT=5000 \
|
||||
DATA_DIR=/app/data
|
||||
|
||||
# Working directory inside the container
|
||||
WORKDIR /app
|
||||
|
||||
# Install Python requirements
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir --upgrade pip && \
|
||||
pip install --no-cache-dir -r requirements.txt && \
|
||||
playwright install chromium
|
||||
|
||||
# Copy application source code
|
||||
COPY thread_ripper/ ./thread_ripper/
|
||||
|
||||
# Create persistent data directory
|
||||
RUN mkdir -p /app/data/threads
|
||||
|
||||
# Expose server port
|
||||
EXPOSE 5000
|
||||
|
||||
# Volume for persisting saved threads
|
||||
VOLUME ["/app/data"]
|
||||
|
||||
# Run Thread Ripper server with uvicorn
|
||||
CMD ["uvicorn", "thread_ripper.server:app", "--host", "0.0.0.0", "--port", "5000"]
|
||||
@@ -0,0 +1,157 @@
|
||||
# 🧵 Thread Ripper
|
||||
|
||||
**Thread Ripper** is a containerized application designed to unroll, extract, and stitch multi-part posts on Meta's Threads back into a single cohesive document.
|
||||
|
||||
Paste any Threads link (including share links like `https://www.threads.com/share/BASy90mpUT/` or direct post URLs), and Thread Ripper automatically detects the author's continuation posts, merges them in chronological order, and lets you view, copy, or export them as **Markdown**, **Plain Text**, **Standalone HTML**, or **JSON**.
|
||||
|
||||
---
|
||||
|
||||
## ✨ Features
|
||||
|
||||
- **Any Link Accepted:** Handles share links (`threads.com/share/...`), shortlinks, and direct URLs (`threads.net/@user/post/...`), resolving redirects automatically.
|
||||
- **Author Thread Stitching:** Detects the author's consecutive continuation replies and stitches them into a continuous long-form read, filtering out unrelated noise, likes, and reply counts.
|
||||
- **Media Preservation:** Retains attached images and photos embedded within thread posts.
|
||||
- **Multi-Format Export:**
|
||||
- 📝 **Markdown (`.md`)** – with formatted headers, quotes, lists, and images.
|
||||
- 📄 **Plain Text (`.txt`)** – clean, distraction-free text.
|
||||
- 🌐 **Standalone HTML (`.html`)** – styled, responsive reader document ready to share or archive.
|
||||
- 📦 **JSON (`.json`)** – structured data with individual segments, timestamps, author metadata, and URLs.
|
||||
- 🖨️ **Print / PDF** – print-ready reader view.
|
||||
- **Modern Web Dashboard:** Clean, responsive UI with one-click clipboard copy, reader preview, segment breakdown, and saved threads library.
|
||||
- **REST API & CLI:** Includes an automated REST API (with Swagger docs at `/docs`) and a CLI command for batch scripting.
|
||||
- **Docker-First Architecture:** Built on Microsoft's official Playwright container with Chromium, optimized to run seamlessly on a server (e.g. `vloser`).
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start with Docker (Deploying to `vloser`)
|
||||
|
||||
### 1. Transfer Project to `vloser`
|
||||
Copy this project folder to your server:
|
||||
```bash
|
||||
# Example using scp or rsync:
|
||||
scp -r thread-ripper user@vloser:/opt/thread-ripper
|
||||
# Or rsync:
|
||||
rsync -avz thread-ripper/ user@vloser:/opt/thread-ripper/
|
||||
```
|
||||
|
||||
### 2. Start with Docker Compose
|
||||
On `vloser`:
|
||||
```bash
|
||||
cd /opt/thread-ripper
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
### 3. Open Web UI
|
||||
Navigate in your browser to:
|
||||
```
|
||||
http://vloser:5000
|
||||
# or
|
||||
http://<vloser-ip>:5000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ Web UI Walkthrough
|
||||
|
||||
1. **Paste Link:** Paste any Threads post or share URL (e.g. `https://www.threads.com/share/BASy90mpUT/`).
|
||||
2. **Click "Rip Thread":** The container resolves the URL in a headless browser, tracks author continuation replies, and stitches the pieces together.
|
||||
3. **View & Export:**
|
||||
- **Reader View:** Read the full thread formatted as a clean article.
|
||||
- **Markdown View:** View or copy the unrolled Markdown.
|
||||
- **Segments:** Inspect each individual post segment.
|
||||
- **Download:** Download `.md`, `.txt`, `.html`, or `.json`.
|
||||
- **Library:** Access previously saved threads anytime from the persistent sidebar.
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ REST API Endpoints
|
||||
|
||||
Thread Ripper includes a complete REST API:
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|---|---|---|
|
||||
| `/api/rip` | `POST` | Rip a thread by URL (`{"url": "...", "mode": "author_only"}`) |
|
||||
| `/api/history` | `GET` | List all saved threads |
|
||||
| `/api/history/{id}` | `GET` | Get full details of a saved thread |
|
||||
| `/api/history/{id}` | `DELETE` | Delete a saved thread |
|
||||
| `/api/download/{id}/{format}` | `GET` | Download in format (`md`, `txt`, `html`, `json`) |
|
||||
| `/api/health` | `GET` | Health check endpoint |
|
||||
| `/docs` | `GET` | Interactive Swagger API documentation |
|
||||
|
||||
### Example cURL Request:
|
||||
```bash
|
||||
curl -X POST http://vloser:5000/api/rip \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"url": "https://www.threads.com/share/BASy90mpUT/"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 💻 CLI Usage
|
||||
|
||||
You can also use Thread Ripper directly from the command line:
|
||||
|
||||
```bash
|
||||
# Run CLI and print output
|
||||
python -m thread_ripper.cli "https://www.threads.com/share/BASy90mpUT/"
|
||||
|
||||
# Export directly to Markdown file
|
||||
python -m thread_ripper.cli "https://www.threads.com/share/BASy90mpUT/" -o unrolled.md
|
||||
|
||||
# Export as Standalone HTML
|
||||
python -m thread_ripper.cli "https://www.threads.com/share/BASy90mpUT/" -f html -o unrolled.html
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📁 Project Structure
|
||||
|
||||
```
|
||||
thread-ripper/
|
||||
├── Dockerfile # Container definition (Playwright + Python)
|
||||
├── docker-compose.yml # Compose service configuration
|
||||
├── requirements.txt # Python dependencies
|
||||
├── .dockerignore # Docker build exclusions
|
||||
├── README.md # Documentation
|
||||
├── data/ # Persistent volume for saved threads
|
||||
│ └── threads/ # Stored .json, .md, .html, and .txt files
|
||||
└── thread_ripper/
|
||||
├── __init__.py
|
||||
├── models.py # Pydantic data schemas
|
||||
├── scraper.py # Playwright browser engine & extractor
|
||||
├── stitcher.py # Formatter (Markdown, Text, HTML)
|
||||
├── storage.py # File-based persistence layer
|
||||
├── server.py # FastAPI application
|
||||
├── cli.py # Command-line interface
|
||||
└── static/ # Modern web frontend
|
||||
├── index.html
|
||||
├── style.css
|
||||
└── app.js
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 Tested Example
|
||||
|
||||
Target URL: `https://www.threads.com/share/BASy90mpUT/`
|
||||
Author: **Beth Parent McMillian (@bethmcmilliandesigns)**
|
||||
|
||||
### Reconstructed Output:
|
||||
|
||||
> # A GenX retirement community in a renovated shopping mall.
|
||||
> **Author:** Beth Parent McMillian (@bethmcmilliandesigns)
|
||||
> **Original Post:** https://www.threads.com/@bethmcmilliandesigns/post/Dd717khDOk_
|
||||
> **Thread Segments:** 2
|
||||
>
|
||||
> A GenX retirement community in a renovated shopping mall. Half the building is apartments. The other half *looks* like an 80s/90s mall, but:
|
||||
> - the Waldenbooks is a library
|
||||
> - all the food places are part of your meal plan, so you can get an Orange Julius, eat anywhere at the food court, or have a nice sit-down meal at Shoney’s
|
||||
> - there’s a movie theater and a free arcade with old-school game cabinets
|
||||
>
|
||||
> - you can just sit in Sam Goody and listen to Bowie or The Ramones or whatever (they also DJ monthly dance parties)
|
||||
> - you can reserve a massage chair in the Sharper Image for an hour at a time
|
||||
> - the in-house pharmacy is an Eckerds
|
||||
> - you can rent movies and video games from Blockbuster
|
||||
> - there’s an aerobics studio where we do GenXercise, Tae-Bo, and MTV’s The Grind workouts
|
||||
> - there are fountains and a lazy river (because why not?)
|
||||
> - we all go mall walking daily
|
||||
@@ -0,0 +1,20 @@
|
||||
services:
|
||||
thread-ripper:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
container_name: thread-ripper
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "5000:5000"
|
||||
environment:
|
||||
- PORT=5000
|
||||
- DATA_DIR=/app/data
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "python -c 'import urllib.request; urllib.request.urlopen(\"http://localhost:5000/api/health\")'"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
@@ -0,0 +1,8 @@
|
||||
fastapi>=0.110.0
|
||||
uvicorn[standard]>=0.28.0
|
||||
playwright>=1.42.0
|
||||
pydantic>=2.6.0
|
||||
jinja2>=3.1.3
|
||||
python-multipart>=0.0.9
|
||||
markdown>=3.5.2
|
||||
httpx>=0.27.0
|
||||
@@ -0,0 +1,2 @@
|
||||
"""Thread Ripper package."""
|
||||
__version__ = "1.0.0"
|
||||
@@ -0,0 +1,69 @@
|
||||
import argparse
|
||||
import asyncio
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from .scraper import ThreadScraper
|
||||
from .storage import ThreadStorage
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="Thread Ripper - Unroll and stitch Threads posts into a single post.")
|
||||
parser.add_argument("url", help="Threads URL (e.g., https://www.threads.com/share/BASy90mpUT/ or https://www.threads.net/@user/post/...)")
|
||||
parser.add_argument("-o", "--output", help="Output file path (e.g. output.md, output.txt, output.html)")
|
||||
parser.add_argument("-f", "--format", choices=["markdown", "txt", "html", "json"], default="markdown", help="Output format (default: markdown)")
|
||||
parser.add_argument("--mode", choices=["author_only", "all_replies"], default="author_only", help="Extraction mode (default: author_only)")
|
||||
parser.add_argument("--scrolls", type=int, default=5, help="Number of lazy-load scrolls (default: 5)")
|
||||
return parser.parse_args()
|
||||
|
||||
async def main_async():
|
||||
args = parse_args()
|
||||
print(f"[*] Thread Ripper v1.0.0")
|
||||
print(f"[*] Target URL: {args.url}")
|
||||
print(f"[*] Mode: {args.mode}")
|
||||
|
||||
scraper = ThreadScraper(headless=True)
|
||||
try:
|
||||
print("[*] Fetching and parsing thread...")
|
||||
result = await scraper.rip(url=args.url, mode=args.mode, max_scrolls=args.scrolls)
|
||||
|
||||
# Save to persistent storage as well
|
||||
storage = ThreadStorage("data")
|
||||
storage.save(result)
|
||||
|
||||
print(f"\n[+] Successfully ripped thread!")
|
||||
print(f" Author: {result.author_name} (@{result.author_handle})")
|
||||
print(f" Segments stitched: {result.segment_count}")
|
||||
print(f" Total words: {result.total_words}")
|
||||
print(f" Thread ID: {result.id}")
|
||||
|
||||
content = ""
|
||||
if args.format == "markdown":
|
||||
content = result.stitched_markdown
|
||||
elif args.format == "txt":
|
||||
content = result.stitched_text
|
||||
elif args.format == "html":
|
||||
content = result.stitched_html
|
||||
elif args.format == "json":
|
||||
content = result.model_dump_json(indent=2)
|
||||
|
||||
if args.output:
|
||||
out_path = Path(args.output)
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out_path, "w", encoding="utf-8") as f:
|
||||
f.write(content)
|
||||
print(f"[+] Output saved to: {out_path.resolve()}")
|
||||
else:
|
||||
print("\n" + "=" * 40 + " RIPPED POST PREVIEW " + "=" * 40)
|
||||
print(content[:1500] + ("\n... [truncated]" if len(content) > 1500 else ""))
|
||||
print("=" * 102)
|
||||
|
||||
except Exception as e:
|
||||
print(f"[-] Error: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
finally:
|
||||
await scraper.close()
|
||||
|
||||
def main():
|
||||
asyncio.run(main_async())
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,52 @@
|
||||
from typing import List, Optional, Dict, Any
|
||||
from pydantic import BaseModel, Field
|
||||
from datetime import datetime
|
||||
|
||||
class MediaItem(BaseModel):
|
||||
type: str = "image"
|
||||
url: str
|
||||
alt_text: Optional[str] = None
|
||||
|
||||
class PostSegment(BaseModel):
|
||||
index: int
|
||||
author_handle: str
|
||||
author_name: Optional[str] = None
|
||||
author_avatar: Optional[str] = None
|
||||
timestamp: Optional[str] = None
|
||||
url: Optional[str] = None
|
||||
text: str
|
||||
is_author: bool = True
|
||||
images: List[MediaItem] = Field(default_factory=list)
|
||||
stats: Dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
class RippedThread(BaseModel):
|
||||
id: str
|
||||
original_url: str
|
||||
canonical_url: str
|
||||
author_handle: str
|
||||
author_name: str
|
||||
author_avatar: Optional[str] = None
|
||||
created_at: str
|
||||
title: str
|
||||
segments: List[PostSegment]
|
||||
stitched_markdown: str
|
||||
stitched_text: str
|
||||
stitched_html: str
|
||||
segment_count: int
|
||||
total_words: int
|
||||
saved_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
|
||||
class RipRequest(BaseModel):
|
||||
url: str
|
||||
mode: str = "author_only" # 'author_only' or 'all_replies'
|
||||
max_scrolls: int = 6
|
||||
|
||||
class HistorySummary(BaseModel):
|
||||
id: str
|
||||
title: str
|
||||
author_handle: str
|
||||
author_name: str
|
||||
segment_count: int
|
||||
total_words: int
|
||||
saved_at: str
|
||||
original_url: str
|
||||
@@ -0,0 +1,228 @@
|
||||
import asyncio
|
||||
import re
|
||||
import hashlib
|
||||
from typing import Tuple, List, Optional
|
||||
from playwright.async_api import async_playwright, Browser, BrowserContext, Page
|
||||
from .models import PostSegment, MediaItem, RippedThread
|
||||
from .stitcher import build_markdown, build_plain_text, build_standalone_html
|
||||
|
||||
class ThreadScraper:
|
||||
def __init__(self, headless: bool = True):
|
||||
self.headless = headless
|
||||
self._browser: Optional[Browser] = None
|
||||
self._playwright = None
|
||||
|
||||
async def get_browser(self) -> Browser:
|
||||
if self._browser is None or not self._browser.is_connected():
|
||||
if self._playwright is None:
|
||||
self._playwright = await async_playwright().start()
|
||||
self._browser = await self._playwright.chromium.launch(
|
||||
headless=self.headless,
|
||||
args=[
|
||||
"--no-sandbox",
|
||||
"--disable-setuid-sandbox",
|
||||
"--disable-dev-shm-usage",
|
||||
"--disable-gpu",
|
||||
]
|
||||
)
|
||||
return self._browser
|
||||
|
||||
async def close(self):
|
||||
if self._browser:
|
||||
await self._browser.close()
|
||||
self._browser = None
|
||||
if self._playwright:
|
||||
await self._playwright.stop()
|
||||
self._playwright = None
|
||||
|
||||
async def rip(self, url: str, mode: str = "author_only", max_scrolls: int = 5) -> RippedThread:
|
||||
"""Scrapes and rips a Threads post and its chained continuation posts."""
|
||||
browser = await self.get_browser()
|
||||
context: BrowserContext = await browser.new_context(
|
||||
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||||
viewport={"width": 1280, "height": 900}
|
||||
)
|
||||
page: Page = await context.new_page()
|
||||
|
||||
try:
|
||||
# Navigate and follow redirects
|
||||
await page.goto(url, wait_until="networkidle", timeout=30000)
|
||||
canonical_url = page.url
|
||||
|
||||
# Small delay for hydration
|
||||
await page.wait_for_timeout(2500)
|
||||
|
||||
# Extract Open Graph metadata
|
||||
og_title = await page.evaluate("() => document.querySelector('meta[property=\"og:title\"]')?.content || ''")
|
||||
og_image = await page.evaluate("() => document.querySelector('meta[property=\"og:image\"]')?.content || ''")
|
||||
og_desc = await page.evaluate("() => document.querySelector('meta[property=\"og:description\"]')?.content || ''")
|
||||
|
||||
# Determine original author handle from URL or title
|
||||
author_handle = ""
|
||||
author_name = ""
|
||||
|
||||
# Check canonical URL for /@username
|
||||
match_handle = re.search(r"/@([a-zA-Z0-9_\.]+)", canonical_url)
|
||||
if match_handle:
|
||||
author_handle = match_handle.group(1)
|
||||
|
||||
# Check og:title e.g. "Beth Parent McMillian (@bethmcmilliandesigns) on Threads"
|
||||
if og_title:
|
||||
m_title = re.search(r"^(.*?)\s*\(@?([a-zA-Z0-9_\.]+)\)", og_title)
|
||||
if m_title:
|
||||
author_name = m_title.group(1).strip()
|
||||
if not author_handle:
|
||||
author_handle = m_title.group(2).strip()
|
||||
|
||||
if not author_name:
|
||||
author_name = author_handle or "Threads Author"
|
||||
|
||||
# Author profile avatar
|
||||
author_avatar = og_image if og_image else None
|
||||
|
||||
# Progressively scroll to load dynamic replies in the thread
|
||||
for _ in range(max_scrolls):
|
||||
await page.evaluate("window.scrollBy(0, 1200)")
|
||||
await page.wait_for_timeout(1000)
|
||||
|
||||
# Query all pressable post containers
|
||||
containers = await page.query_selector_all("div[data-pressable-container='true']")
|
||||
|
||||
raw_segments: List[PostSegment] = []
|
||||
seen_texts = set()
|
||||
|
||||
for idx, c in enumerate(containers):
|
||||
# 1. Identify post author handle
|
||||
user_links = await c.query_selector_all("a[href*='/@']")
|
||||
post_handle = None
|
||||
post_url = None
|
||||
|
||||
for ul in user_links:
|
||||
href = await ul.get_attribute("href")
|
||||
if href:
|
||||
if "/@" in href and not post_handle:
|
||||
parts = href.split("/@")
|
||||
if len(parts) > 1:
|
||||
post_handle = parts[1].split("/")[0].split("?")[0]
|
||||
if "/post/" in href and not post_url:
|
||||
post_url = href.split("?")[0]
|
||||
|
||||
# Check for "Author" badge
|
||||
is_author_badge = False
|
||||
badge = await c.query_selector("xpath=.//*[text()='Author']")
|
||||
if badge:
|
||||
is_author_badge = True
|
||||
|
||||
# Determine if this segment belongs to the original author
|
||||
is_author_post = False
|
||||
if idx == 0:
|
||||
is_author_post = True
|
||||
if post_handle and not author_handle:
|
||||
author_handle = post_handle
|
||||
elif is_author_badge:
|
||||
is_author_post = True
|
||||
elif post_handle and author_handle and (post_handle.lower() == author_handle.lower()):
|
||||
is_author_post = True
|
||||
|
||||
# If author_only mode is on and this is not by the author, skip
|
||||
if mode == "author_only" and not is_author_post:
|
||||
continue
|
||||
|
||||
# 2. Extract text with dir='auto'
|
||||
dir_els = await c.query_selector_all("[dir='auto']")
|
||||
segment_text_parts = []
|
||||
for del_el in dir_els:
|
||||
txt = (await del_el.inner_text()).strip()
|
||||
if not txt:
|
||||
continue
|
||||
# Ignore handle, author badge, bullet dots, stats
|
||||
if txt == post_handle or txt == "Author" or txt == "·" or txt == author_handle:
|
||||
continue
|
||||
# Ignore pure stat numbers (e.g., '16.2K', '5d', '2K', '711')
|
||||
if re.match(r"^(\d+(\.\d+)?[KMkmdh]?|\d+)$", txt):
|
||||
continue
|
||||
if txt not in segment_text_parts:
|
||||
segment_text_parts.append(txt)
|
||||
|
||||
combined_text = "\n\n".join(segment_text_parts).strip()
|
||||
if not combined_text and not is_author_post:
|
||||
continue
|
||||
|
||||
# Deduplicate identical consecutive segments
|
||||
text_hash = hashlib.md5(combined_text.encode("utf-8")).hexdigest()
|
||||
if text_hash in seen_texts and combined_text:
|
||||
continue
|
||||
seen_texts.add(text_hash)
|
||||
|
||||
# 3. Extract media images inside post (exclude profile pictures)
|
||||
imgs = await c.query_selector_all("img")
|
||||
post_images: List[MediaItem] = []
|
||||
for img in imgs:
|
||||
src = await img.get_attribute("src")
|
||||
alt = await img.get_attribute("alt") or ""
|
||||
if src:
|
||||
# Exclude small avatars or generic SVG icons
|
||||
if "profile_pic" in src or "scontent" in src and "206x206" in src:
|
||||
continue
|
||||
post_images.append(MediaItem(type="image", url=src, alt_text=alt))
|
||||
|
||||
raw_segments.append(
|
||||
PostSegment(
|
||||
index=len(raw_segments),
|
||||
author_handle=post_handle or author_handle,
|
||||
author_name=author_name if is_author_post else (post_handle or "User"),
|
||||
author_avatar=author_avatar if is_author_post else None,
|
||||
url=post_url,
|
||||
text=combined_text,
|
||||
is_author=is_author_post,
|
||||
images=post_images
|
||||
)
|
||||
)
|
||||
|
||||
# If no segments could be extracted from containers, fallback to OG description
|
||||
if not raw_segments and og_desc:
|
||||
raw_segments.append(
|
||||
PostSegment(
|
||||
index=0,
|
||||
author_handle=author_handle,
|
||||
author_name=author_name,
|
||||
author_avatar=author_avatar,
|
||||
url=canonical_url,
|
||||
text=og_desc,
|
||||
is_author=True,
|
||||
images=[]
|
||||
)
|
||||
)
|
||||
|
||||
# Construct stitched outputs
|
||||
markdown_out = build_markdown(author_name, author_handle, canonical_url, raw_segments)
|
||||
plain_text_out = build_plain_text(author_name, author_handle, canonical_url, raw_segments)
|
||||
html_out = build_standalone_html(author_name, author_handle, author_avatar, canonical_url, raw_segments)
|
||||
|
||||
# Generate unique ID for this ripped thread
|
||||
thread_id = hashlib.md5(canonical_url.split("?")[0].encode("utf-8")).hexdigest()[:12]
|
||||
title = raw_segments[0].text.split("\n")[0] if raw_segments and raw_segments[0].text else f"Thread by @{author_handle}"
|
||||
if len(title) > 90:
|
||||
title = title[:87] + "..."
|
||||
|
||||
total_words = sum(len(s.text.split()) for s in raw_segments)
|
||||
|
||||
return RippedThread(
|
||||
id=thread_id,
|
||||
original_url=url,
|
||||
canonical_url=canonical_url,
|
||||
author_handle=author_handle,
|
||||
author_name=author_name,
|
||||
author_avatar=author_avatar,
|
||||
created_at="Recent",
|
||||
title=title,
|
||||
segments=raw_segments,
|
||||
stitched_markdown=markdown_out,
|
||||
stitched_text=plain_text_out,
|
||||
stitched_html=html_out,
|
||||
segment_count=len(raw_segments),
|
||||
total_words=total_words
|
||||
)
|
||||
|
||||
finally:
|
||||
await context.close()
|
||||
@@ -0,0 +1,113 @@
|
||||
import os
|
||||
from contextlib import asynccontextmanager
|
||||
from fastapi import FastAPI, HTTPException, Response
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import FileResponse, PlainTextResponse, HTMLResponse, JSONResponse
|
||||
from .models import RipRequest, RippedThread
|
||||
from .scraper import ThreadScraper
|
||||
from .storage import ThreadStorage
|
||||
|
||||
DATA_DIR = os.environ.get("DATA_DIR", "data")
|
||||
storage = ThreadStorage(DATA_DIR)
|
||||
scraper = ThreadScraper(headless=True)
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
yield
|
||||
await scraper.close()
|
||||
|
||||
app = FastAPI(
|
||||
title="Thread Ripper",
|
||||
description="Containerized Thread Ripper for Meta Threads. Takes Threads links and stitches threads into single posts.",
|
||||
version="1.0.0",
|
||||
lifespan=lifespan
|
||||
)
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
@app.post("/api/rip", response_model=RippedThread)
|
||||
async def rip_endpoint(req: RipRequest):
|
||||
"""Rip a Threads post and its chained continuation posts."""
|
||||
if not req.url or "threads." not in req.url:
|
||||
raise HTTPException(status_code=400, detail="Invalid Threads URL. Please provide a valid threads.net or threads.com link.")
|
||||
|
||||
try:
|
||||
ripped = await scraper.rip(url=req.url, mode=req.mode, max_scrolls=req.max_scrolls)
|
||||
storage.save(ripped)
|
||||
return ripped
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=f"Failed to rip thread: {str(e)}")
|
||||
|
||||
@app.get("/api/history")
|
||||
def get_history():
|
||||
"""Retrieve history of saved ripped threads."""
|
||||
return storage.list_history()
|
||||
|
||||
@app.get("/api/history/{thread_id}", response_model=RippedThread)
|
||||
def get_thread_by_id(thread_id: str):
|
||||
"""Get complete ripped thread by ID."""
|
||||
thread = storage.get(thread_id)
|
||||
if not thread:
|
||||
raise HTTPException(status_code=404, detail="Thread not found")
|
||||
return thread
|
||||
|
||||
@app.delete("/api/history/{thread_id}")
|
||||
def delete_thread_by_id(thread_id: str):
|
||||
"""Delete thread from history."""
|
||||
deleted = storage.delete(thread_id)
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail="Thread not found")
|
||||
return {"status": "success", "deleted_id": thread_id}
|
||||
|
||||
@app.get("/api/download/{thread_id}/{format}")
|
||||
def download_thread(thread_id: str, format: str):
|
||||
"""Download thread in chosen format (.md, .txt, .html, .json)."""
|
||||
thread = storage.get(thread_id)
|
||||
if not thread:
|
||||
raise HTTPException(status_code=404, detail="Thread not found")
|
||||
|
||||
safe_title = "".join(c for c in thread.author_handle if c.isalnum() or c in "-_")
|
||||
filename_base = f"thread_{safe_title}_{thread_id}"
|
||||
|
||||
if format == "md" or format == "markdown":
|
||||
return Response(
|
||||
content=thread.stitched_markdown,
|
||||
media_type="text/markdown; charset=utf-8",
|
||||
headers={"Content-Disposition": f'attachment; filename="{filename_base}.md"'}
|
||||
)
|
||||
elif format == "txt" or format == "text":
|
||||
return Response(
|
||||
content=thread.stitched_text,
|
||||
media_type="text/plain; charset=utf-8",
|
||||
headers={"Content-Disposition": f'attachment; filename="{filename_base}.txt"'}
|
||||
)
|
||||
elif format == "html":
|
||||
return Response(
|
||||
content=thread.stitched_html,
|
||||
media_type="text/html; charset=utf-8",
|
||||
headers={"Content-Disposition": f'attachment; filename="{filename_base}.html"'}
|
||||
)
|
||||
elif format == "json":
|
||||
return Response(
|
||||
content=thread.model_dump_json(indent=2),
|
||||
media_type="application/json; charset=utf-8",
|
||||
headers={"Content-Disposition": f'attachment; filename="{filename_base}.json"'}
|
||||
)
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail="Supported formats: md, txt, html, json")
|
||||
|
||||
@app.get("/api/health")
|
||||
def health_check():
|
||||
return {"status": "ok", "app": "Thread Ripper", "version": "1.0.0"}
|
||||
|
||||
# Mount static frontend directory
|
||||
static_dir = os.path.join(os.path.dirname(__file__), "static")
|
||||
if os.path.exists(static_dir):
|
||||
app.mount("/", StaticFiles(directory=static_dir, html=True), name="static")
|
||||
@@ -0,0 +1,329 @@
|
||||
document.addEventListener("DOMContentLoaded", () => {
|
||||
const threadUrlInput = document.getElementById("thread-url");
|
||||
const pasteBtn = document.getElementById("paste-btn");
|
||||
const ripBtn = document.getElementById("rip-btn");
|
||||
const sampleMallBtn = document.getElementById("sample-mall-btn");
|
||||
const statusBanner = document.getElementById("status-banner");
|
||||
const statusTitle = document.getElementById("status-title");
|
||||
const statusDesc = document.getElementById("status-desc");
|
||||
const resultSection = document.getElementById("result-section");
|
||||
|
||||
// Results elements
|
||||
const resAvatar = document.getElementById("res-avatar");
|
||||
const resAuthorName = document.getElementById("res-author-name");
|
||||
const resAuthorLink = document.getElementById("res-author-link");
|
||||
const resSegmentCount = document.getElementById("res-segment-count");
|
||||
const resWordCount = document.getElementById("res-word-count");
|
||||
const tabSegmentCount = document.getElementById("tab-segment-count");
|
||||
const readerContent = document.getElementById("reader-content");
|
||||
const markdownContent = document.getElementById("markdown-content");
|
||||
const segmentsContainer = document.getElementById("segments-container");
|
||||
|
||||
// Export buttons
|
||||
const copyMdBtn = document.getElementById("copy-md-btn");
|
||||
const copyTxtBtn = document.getElementById("copy-txt-btn");
|
||||
const downloadDropdownBtn = document.getElementById("download-dropdown-btn");
|
||||
const downloadMenu = document.getElementById("download-menu");
|
||||
const dlMd = document.getElementById("dl-md");
|
||||
const dlTxt = document.getElementById("dl-txt");
|
||||
const dlHtml = document.getElementById("dl-html");
|
||||
const dlJson = document.getElementById("dl-json");
|
||||
const printBtn = document.getElementById("print-btn");
|
||||
|
||||
// Library / History elements
|
||||
const toggleHistoryBtn = document.getElementById("toggle-history-btn");
|
||||
const librarySidebar = document.getElementById("library-sidebar");
|
||||
const closeLibraryBtn = document.getElementById("close-library-btn");
|
||||
const libraryList = document.getElementById("library-list");
|
||||
const libraryCount = document.getElementById("history-count");
|
||||
const librarySearch = document.getElementById("library-search");
|
||||
|
||||
// Toast
|
||||
const toast = document.getElementById("toast");
|
||||
|
||||
let currentThread = null;
|
||||
let allHistory = [];
|
||||
|
||||
// Initialize history
|
||||
fetchHistory();
|
||||
|
||||
// Paste button
|
||||
pasteBtn.addEventListener("click", async () => {
|
||||
try {
|
||||
const text = await navigator.clipboard.readText();
|
||||
if (text) {
|
||||
threadUrlInput.value = text.trim();
|
||||
}
|
||||
} catch (e) {
|
||||
showToast("Clipboard access denied or unavailable", true);
|
||||
}
|
||||
});
|
||||
|
||||
// Example button
|
||||
sampleMallBtn.addEventListener("click", () => {
|
||||
threadUrlInput.value = sampleMallBtn.getAttribute("data-url");
|
||||
triggerRip();
|
||||
});
|
||||
|
||||
// Rip button
|
||||
ripBtn.addEventListener("click", () => {
|
||||
triggerRip();
|
||||
});
|
||||
|
||||
threadUrlInput.addEventListener("keydown", (e) => {
|
||||
if (e.key === "Enter") {
|
||||
triggerRip();
|
||||
}
|
||||
});
|
||||
|
||||
async function triggerRip() {
|
||||
const url = threadUrlInput.value.trim();
|
||||
if (!url) {
|
||||
showToast("Please paste a Threads URL", true);
|
||||
return;
|
||||
}
|
||||
|
||||
const mode = document.querySelector('input[name="rip-mode"]:checked').value;
|
||||
|
||||
setLoading(true);
|
||||
statusTitle.textContent = "Connecting to Threads...";
|
||||
statusDesc.textContent = "Launching headless browser and resolving URL...";
|
||||
|
||||
try {
|
||||
const response = await fetch("/api/rip", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ url, mode, max_scrolls: 5 })
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const err = await response.json();
|
||||
throw new Error(err.detail || "Failed to rip thread");
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
currentThread = data;
|
||||
renderThread(data);
|
||||
fetchHistory();
|
||||
showToast("Thread successfully ripped & saved!");
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
showToast(`Error: ${err.message}`, true);
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
}
|
||||
|
||||
function setLoading(isLoading) {
|
||||
const btnText = ripBtn.querySelector(".btn-text");
|
||||
const spinner = ripBtn.querySelector(".btn-spinner");
|
||||
if (isLoading) {
|
||||
ripBtn.disabled = true;
|
||||
btnText.textContent = "Ripping...";
|
||||
spinner.classList.remove("hidden");
|
||||
statusBanner.classList.remove("hidden");
|
||||
resultSection.classList.add("hidden");
|
||||
} else {
|
||||
ripBtn.disabled = false;
|
||||
btnText.textContent = "Rip Thread";
|
||||
spinner.classList.add("hidden");
|
||||
statusBanner.classList.add("hidden");
|
||||
}
|
||||
}
|
||||
|
||||
function renderThread(thread) {
|
||||
resultSection.classList.remove("hidden");
|
||||
|
||||
// Header info
|
||||
resAuthorName.textContent = thread.author_name;
|
||||
resAuthorLink.textContent = `@${thread.author_handle}`;
|
||||
resAuthorLink.href = thread.canonical_url;
|
||||
if (thread.author_avatar) {
|
||||
resAvatar.src = thread.author_avatar;
|
||||
resAvatar.classList.remove("hidden");
|
||||
} else {
|
||||
resAvatar.classList.add("hidden");
|
||||
}
|
||||
|
||||
resSegmentCount.textContent = thread.segment_count;
|
||||
resWordCount.textContent = thread.total_words;
|
||||
tabSegmentCount.textContent = thread.segment_count;
|
||||
|
||||
// Render Markdown view
|
||||
markdownContent.textContent = thread.stitched_markdown;
|
||||
|
||||
// Render Reader view (using marked.js parser)
|
||||
if (window.marked) {
|
||||
readerContent.innerHTML = marked.parse(thread.stitched_markdown);
|
||||
} else {
|
||||
readerContent.innerHTML = thread.stitched_html;
|
||||
}
|
||||
|
||||
// Render Segments breakdown view
|
||||
segmentsContainer.innerHTML = "";
|
||||
thread.segments.forEach((seg, idx) => {
|
||||
const segEl = document.createElement("div");
|
||||
segEl.className = "segment-item";
|
||||
segEl.innerHTML = `
|
||||
<div class="segment-header">
|
||||
<span class="segment-num">Segment #${idx + 1} ${seg.is_author ? "(Author)" : "(Reply)"}</span>
|
||||
<span>${seg.author_handle ? "@" + seg.author_handle : ""}</span>
|
||||
</div>
|
||||
<div class="segment-text">${escapeHtml(seg.text)}</div>
|
||||
`;
|
||||
segmentsContainer.appendChild(segEl);
|
||||
});
|
||||
|
||||
// Setup download links
|
||||
dlMd.href = `/api/download/${thread.id}/md`;
|
||||
dlTxt.href = `/api/download/${thread.id}/txt`;
|
||||
dlHtml.href = `/api/download/${thread.id}/html`;
|
||||
dlJson.href = `/api/download/${thread.id}/json`;
|
||||
|
||||
// Switch to reader view
|
||||
switchTab("reader");
|
||||
}
|
||||
|
||||
// Tabs switching
|
||||
document.querySelectorAll(".tab-btn").forEach((btn) => {
|
||||
btn.addEventListener("click", () => {
|
||||
const tabName = btn.getAttribute("data-tab");
|
||||
switchTab(tabName);
|
||||
});
|
||||
});
|
||||
|
||||
function switchTab(tabName) {
|
||||
document.querySelectorAll(".tab-btn").forEach((b) => b.classList.remove("active"));
|
||||
document.querySelector(`.tab-btn[data-tab="${tabName}"]`)?.classList.add("active");
|
||||
|
||||
document.querySelectorAll(".tab-view").forEach((v) => {
|
||||
v.classList.add("hidden");
|
||||
v.classList.remove("active");
|
||||
});
|
||||
|
||||
const targetView = document.getElementById(`view-${tabName}`);
|
||||
if (targetView) {
|
||||
targetView.classList.remove("hidden");
|
||||
targetView.classList.add("active");
|
||||
}
|
||||
}
|
||||
|
||||
// Copy buttons
|
||||
copyMdBtn.addEventListener("click", () => {
|
||||
if (!currentThread) return;
|
||||
navigator.clipboard.writeText(currentThread.stitched_markdown);
|
||||
showToast("Copied Markdown to clipboard!");
|
||||
});
|
||||
|
||||
copyTxtBtn.addEventListener("click", () => {
|
||||
if (!currentThread) return;
|
||||
navigator.clipboard.writeText(currentThread.stitched_text);
|
||||
showToast("Copied Plain Text to clipboard!");
|
||||
});
|
||||
|
||||
// Download dropdown toggle
|
||||
downloadDropdownBtn.addEventListener("click", (e) => {
|
||||
e.stopPropagation();
|
||||
downloadMenu.classList.toggle("show");
|
||||
});
|
||||
|
||||
document.addEventListener("click", () => {
|
||||
downloadMenu.classList.remove("show");
|
||||
});
|
||||
|
||||
// Print button
|
||||
printBtn.addEventListener("click", () => {
|
||||
if (!currentThread) return;
|
||||
const printWindow = window.open("", "_blank");
|
||||
printWindow.document.write(currentThread.stitched_html);
|
||||
printWindow.document.close();
|
||||
printWindow.focus();
|
||||
setTimeout(() => {
|
||||
printWindow.print();
|
||||
}, 500);
|
||||
});
|
||||
|
||||
// Library / History
|
||||
toggleHistoryBtn.addEventListener("click", () => {
|
||||
librarySidebar.classList.toggle("open");
|
||||
});
|
||||
|
||||
closeLibraryBtn.addEventListener("click", () => {
|
||||
librarySidebar.classList.remove("open");
|
||||
});
|
||||
|
||||
async function fetchHistory() {
|
||||
try {
|
||||
const res = await fetch("/api/history");
|
||||
if (res.ok) {
|
||||
allHistory = await res.json();
|
||||
libraryCount.textContent = allHistory.length;
|
||||
renderHistoryList(allHistory);
|
||||
}
|
||||
} catch (e) {
|
||||
console.warn("Could not load history", e);
|
||||
}
|
||||
}
|
||||
|
||||
function renderHistoryList(items) {
|
||||
if (!items || items.length === 0) {
|
||||
libraryList.innerHTML = '<p class="empty-state">No threads ripped yet.</p>';
|
||||
return;
|
||||
}
|
||||
|
||||
libraryList.innerHTML = "";
|
||||
items.forEach((item) => {
|
||||
const div = document.createElement("div");
|
||||
div.className = "history-item";
|
||||
div.innerHTML = `
|
||||
<div class="history-item-title">${escapeHtml(item.title)}</div>
|
||||
<div class="history-item-meta">
|
||||
<span>@${escapeHtml(item.author_handle)} (${item.segment_count} posts)</span>
|
||||
<span>${item.saved_at.split("T")[0] || ""}</span>
|
||||
</div>
|
||||
`;
|
||||
div.addEventListener("click", async () => {
|
||||
try {
|
||||
const res = await fetch(`/api/history/${item.id}`);
|
||||
if (res.ok) {
|
||||
currentThread = await res.json();
|
||||
renderThread(currentThread);
|
||||
librarySidebar.classList.remove("open");
|
||||
}
|
||||
} catch (e) {
|
||||
showToast("Failed to load thread", true);
|
||||
}
|
||||
});
|
||||
libraryList.appendChild(div);
|
||||
});
|
||||
}
|
||||
|
||||
librarySearch.addEventListener("input", (e) => {
|
||||
const q = e.target.value.toLowerCase();
|
||||
const filtered = allHistory.filter(
|
||||
(item) =>
|
||||
item.title.toLowerCase().includes(q) ||
|
||||
item.author_handle.toLowerCase().includes(q)
|
||||
);
|
||||
renderHistoryList(filtered);
|
||||
});
|
||||
|
||||
function showToast(msg, isError = false) {
|
||||
toast.textContent = msg;
|
||||
toast.style.background = isError ? "#da3633" : "#238636";
|
||||
toast.classList.remove("hidden");
|
||||
setTimeout(() => {
|
||||
toast.classList.add("hidden");
|
||||
}, 3500);
|
||||
}
|
||||
|
||||
function escapeHtml(str) {
|
||||
if (!str) return "";
|
||||
return str
|
||||
.replace(/&/g, "&")
|
||||
.replace(/</g, "<")
|
||||
.replace(/>/g, ">")
|
||||
.replace(/"/g, """)
|
||||
.replace(/'/g, "'");
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,199 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Thread Ripper - Stitch Threads into Single Posts</title>
|
||||
<link rel="stylesheet" href="style.css">
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet">
|
||||
<!-- Markdown parser for preview -->
|
||||
<script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>
|
||||
</head>
|
||||
<body>
|
||||
<div class="app-layout">
|
||||
<!-- Main Content Area -->
|
||||
<main class="main-content">
|
||||
<header class="app-header">
|
||||
<div class="logo-group">
|
||||
<div class="logo-icon">
|
||||
<svg width="28" height="28" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round">
|
||||
<path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/>
|
||||
<path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/>
|
||||
</svg>
|
||||
</div>
|
||||
<div>
|
||||
<h1>Thread Ripper</h1>
|
||||
<p class="subtitle">Extract and stitch multi-part Threads into a single cohesive post</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="header-badges">
|
||||
<span class="badge docker-badge">
|
||||
<svg width="14" height="14" viewBox="0 0 24 24" fill="currentColor">
|
||||
<path d="M4.5 10.5h2v2h-2zm3 0h2v2h-2zm3 0h2v2h-2zm3 0h2v2h-2zm-6-3h2v2h-2zm3 0h2v2h-2zm3 0h2v2h-2zm-3-3h2v2h-2zm12 5.5c-.3-.2-1.5-.7-2.3-.2-.2-.8-.7-1.4-1.4-1.8l-.5-.3-.4.4c-.6.7-.8 1.6-.7 2.4-.9.2-1.8.8-2.2 1.6H2.5c-.8 0-1.5.7-1.5 1.5 0 3.3 2.1 6.3 5.4 7.4 3.7 1.3 7.8.4 10.7-2.2 2.3-2.1 3.5-5.2 3.4-8.3-.1-.3-.3-.5-.5-.5z"/>
|
||||
</svg>
|
||||
Docker Container
|
||||
</span>
|
||||
<button id="toggle-history-btn" class="btn btn-ghost" title="Toggle Saved Library">
|
||||
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2">
|
||||
<path d="M12 8v4l3 3"/>
|
||||
<circle cx="12" cy="12" r="9"/>
|
||||
</svg>
|
||||
Library (<span id="history-count">0</span>)
|
||||
</button>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<!-- Input Form Card -->
|
||||
<section class="card input-card">
|
||||
<div class="url-input-container">
|
||||
<div class="input-prefix">
|
||||
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2">
|
||||
<path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/>
|
||||
<path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/>
|
||||
</svg>
|
||||
</div>
|
||||
<input
|
||||
type="url"
|
||||
id="thread-url"
|
||||
placeholder="Paste any Threads link (e.g. https://www.threads.com/share/BASy90mpUT/ or https://www.threads.net/@user/post/...)"
|
||||
autocomplete="off"
|
||||
spellcheck="false"
|
||||
/>
|
||||
<button id="paste-btn" class="btn btn-secondary btn-sm" title="Paste from clipboard">Paste</button>
|
||||
<button id="rip-btn" class="btn btn-primary">
|
||||
<span class="btn-text">Rip Thread</span>
|
||||
<span class="btn-spinner hidden"></span>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div class="input-controls">
|
||||
<div class="sample-links">
|
||||
<span class="sample-label">Try example:</span>
|
||||
<button class="pill-btn" id="sample-mall-btn" data-url="https://www.threads.com/share/BASy90mpUT/">
|
||||
GenX Mall Retirement Community (@bethmcmilliandesigns)
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div class="mode-selector">
|
||||
<label class="radio-label">
|
||||
<input type="radio" name="rip-mode" value="author_only" checked />
|
||||
<span>Author Thread Only</span>
|
||||
</label>
|
||||
<label class="radio-label">
|
||||
<input type="radio" name="rip-mode" value="all_replies" />
|
||||
<span>Include Community Replies</span>
|
||||
</label>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- Status Banner -->
|
||||
<div id="status-banner" class="status-banner hidden">
|
||||
<div class="status-spinner"></div>
|
||||
<div class="status-content">
|
||||
<p class="status-title" id="status-title">Ripping thread...</p>
|
||||
<p class="status-desc" id="status-desc">Launching headless browser, following redirect, and collecting chained posts...</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Result Card -->
|
||||
<section id="result-section" class="result-section hidden">
|
||||
<!-- Thread Info Header -->
|
||||
<div class="card meta-card">
|
||||
<div class="author-profile">
|
||||
<img id="res-avatar" class="author-avatar" src="" alt="Avatar" />
|
||||
<div class="author-meta">
|
||||
<h2 id="res-author-name">Author Name</h2>
|
||||
<a id="res-author-link" href="#" target="_blank" class="author-handle">@username</a>
|
||||
</div>
|
||||
</div>
|
||||
<div class="thread-stats">
|
||||
<div class="stat-box">
|
||||
<span class="stat-number" id="res-segment-count">0</span>
|
||||
<span class="stat-label">Segments Merged</span>
|
||||
</div>
|
||||
<div class="stat-box">
|
||||
<span class="stat-number" id="res-word-count">0</span>
|
||||
<span class="stat-label">Total Words</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Action Bar -->
|
||||
<div class="action-bar">
|
||||
<div class="tab-group">
|
||||
<button class="tab-btn active" data-tab="reader">Reader View</button>
|
||||
<button class="tab-btn" data-tab="markdown">Markdown</button>
|
||||
<button class="tab-btn" data-tab="segments">Segments (<span id="tab-segment-count">0</span>)</button>
|
||||
</div>
|
||||
|
||||
<div class="export-group">
|
||||
<button id="copy-md-btn" class="btn btn-secondary btn-sm" title="Copy as Markdown">
|
||||
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"><rect x="9" y="9" width="13" height="13" rx="2" ry="2"/><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"/></svg>
|
||||
Copy MD
|
||||
</button>
|
||||
<button id="copy-txt-btn" class="btn btn-secondary btn-sm" title="Copy Plain Text">
|
||||
Copy Text
|
||||
</button>
|
||||
<div class="dropdown">
|
||||
<button id="download-dropdown-btn" class="btn btn-primary btn-sm">
|
||||
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"><path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4"/><polyline points="7 10 12 15 17 10"/><line x1="12" y1="15" x2="12" y2="3"/></svg>
|
||||
Download ▼
|
||||
</button>
|
||||
<div class="dropdown-menu" id="download-menu">
|
||||
<a href="#" id="dl-md" class="dropdown-item">Download .md (Markdown)</a>
|
||||
<a href="#" id="dl-txt" class="dropdown-item">Download .txt (Plain Text)</a>
|
||||
<a href="#" id="dl-html" class="dropdown-item">Download .html (Standalone Page)</a>
|
||||
<a href="#" id="dl-json" class="dropdown-item">Download .json (Raw Data)</a>
|
||||
</div>
|
||||
</div>
|
||||
<button id="print-btn" class="btn btn-ghost btn-sm" title="Print or save as PDF">
|
||||
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"><polyline points="6 9 6 2 18 2 18 9"/><path d="M6 18H4a2 2 0 0 1-2-2v-5a2 2 0 0 1 2-2h16a2 2 0 0 1 2 2v5a2 2 0 0 1-2 2h-2"/><rect x="6" y="14" width="12" height="8"/></svg>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Tab Views -->
|
||||
<div class="card view-card">
|
||||
<!-- Reader View -->
|
||||
<div id="view-reader" class="tab-view active">
|
||||
<div id="reader-content" class="reader-typography"></div>
|
||||
</div>
|
||||
|
||||
<!-- Markdown View -->
|
||||
<div id="view-markdown" class="tab-view hidden">
|
||||
<pre><code id="markdown-content"></code></pre>
|
||||
</div>
|
||||
|
||||
<!-- Segments View -->
|
||||
<div id="view-segments" class="tab-view hidden">
|
||||
<div id="segments-container" class="segments-list"></div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
<!-- Slide-out Library Sidebar -->
|
||||
<aside id="library-sidebar" class="library-sidebar">
|
||||
<div class="sidebar-header">
|
||||
<h3>Saved Threads</h3>
|
||||
<button id="close-library-btn" class="btn btn-ghost btn-icon">✕</button>
|
||||
</div>
|
||||
<div class="sidebar-search">
|
||||
<input type="text" id="library-search" placeholder="Search saved threads..." />
|
||||
</div>
|
||||
<div class="sidebar-list" id="library-list">
|
||||
<!-- Rendered dynamically -->
|
||||
<p class="empty-state">No threads ripped yet.</p>
|
||||
</div>
|
||||
</aside>
|
||||
</div>
|
||||
|
||||
<!-- Toast Notification -->
|
||||
<div id="toast" class="toast hidden"></div>
|
||||
|
||||
<script src="app.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,643 @@
|
||||
:root {
|
||||
--bg-main: #0d1117;
|
||||
--bg-card: #161b22;
|
||||
--bg-input: #21262d;
|
||||
--bg-hover: #30363d;
|
||||
--border: #30363d;
|
||||
--border-focus: #58a6ff;
|
||||
--text-main: #f0f6fc;
|
||||
--text-muted: #8b949e;
|
||||
--accent: #2f81f7;
|
||||
--accent-hover: #388bfd;
|
||||
--accent-light: rgba(56, 139, 253, 0.15);
|
||||
--success: #238636;
|
||||
--danger: #da3633;
|
||||
--radius: 12px;
|
||||
--shadow: 0 8px 24px rgba(0, 0, 0, 0.25);
|
||||
--font: "Inter", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
||||
--font-mono: "JetBrains Mono", Consolas, Monaco, monospace;
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: var(--font);
|
||||
background-color: var(--bg-main);
|
||||
color: var(--text-main);
|
||||
min-height: 100vh;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
.app-layout {
|
||||
display: flex;
|
||||
min-height: 100vh;
|
||||
position: relative;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
.main-content {
|
||||
flex: 1;
|
||||
max-width: 900px;
|
||||
margin: 0 auto;
|
||||
padding: 40px 20px;
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
/* Header */
|
||||
.app-header {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
margin-bottom: 32px;
|
||||
}
|
||||
|
||||
.logo-group {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 16px;
|
||||
}
|
||||
|
||||
.logo-icon {
|
||||
width: 48px;
|
||||
height: 48px;
|
||||
border-radius: 14px;
|
||||
background: linear-gradient(135deg, #1f6feb 0%, #8957e5 100%);
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
color: #fff;
|
||||
box-shadow: 0 4px 16px rgba(31, 111, 235, 0.35);
|
||||
}
|
||||
|
||||
.app-header h1 {
|
||||
font-size: 1.6rem;
|
||||
font-weight: 700;
|
||||
letter-spacing: -0.02em;
|
||||
}
|
||||
|
||||
.subtitle {
|
||||
font-size: 0.9rem;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.header-badges {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.badge {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
padding: 6px 12px;
|
||||
border-radius: 20px;
|
||||
font-size: 0.8rem;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.docker-badge {
|
||||
background: rgba(36, 150, 237, 0.15);
|
||||
color: #58a6ff;
|
||||
border: 1px solid rgba(56, 139, 253, 0.3);
|
||||
}
|
||||
|
||||
/* Cards */
|
||||
.card {
|
||||
background-color: var(--bg-card);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
padding: 24px;
|
||||
box-shadow: var(--shadow);
|
||||
}
|
||||
|
||||
/* Input Card */
|
||||
.input-card {
|
||||
margin-bottom: 24px;
|
||||
}
|
||||
|
||||
.url-input-container {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
background-color: var(--bg-input);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 10px;
|
||||
padding: 6px 10px;
|
||||
gap: 10px;
|
||||
transition: border-color 0.2s, box-shadow 0.2s;
|
||||
}
|
||||
|
||||
.url-input-container:focus-within {
|
||||
border-color: var(--border-focus);
|
||||
box-shadow: 0 0 0 3px rgba(88, 166, 255, 0.2);
|
||||
}
|
||||
|
||||
.input-prefix {
|
||||
color: var(--text-muted);
|
||||
display: flex;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.url-input-container input {
|
||||
flex: 1;
|
||||
background: transparent;
|
||||
border: none;
|
||||
outline: none;
|
||||
color: var(--text-main);
|
||||
font-size: 0.95rem;
|
||||
font-family: var(--font);
|
||||
}
|
||||
|
||||
.input-controls {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
margin-top: 16px;
|
||||
flex-wrap: wrap;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.sample-links {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
|
||||
.sample-label {
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.pill-btn {
|
||||
background: var(--bg-input);
|
||||
border: 1px solid var(--border);
|
||||
color: var(--text-main);
|
||||
padding: 4px 10px;
|
||||
border-radius: 14px;
|
||||
font-size: 0.8rem;
|
||||
cursor: pointer;
|
||||
transition: all 0.2s;
|
||||
}
|
||||
|
||||
.pill-btn:hover {
|
||||
background: var(--bg-hover);
|
||||
border-color: var(--border-focus);
|
||||
}
|
||||
|
||||
.mode-selector {
|
||||
display: flex;
|
||||
gap: 16px;
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
|
||||
.radio-label {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
cursor: pointer;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.radio-label input:checked + span {
|
||||
color: var(--text-main);
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
/* Buttons */
|
||||
.btn {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
gap: 8px;
|
||||
padding: 8px 16px;
|
||||
font-size: 0.9rem;
|
||||
font-weight: 500;
|
||||
border-radius: 8px;
|
||||
cursor: pointer;
|
||||
border: 1px solid transparent;
|
||||
transition: all 0.2s;
|
||||
font-family: var(--font);
|
||||
}
|
||||
|
||||
.btn-primary {
|
||||
background-color: var(--accent);
|
||||
color: #fff;
|
||||
}
|
||||
|
||||
.btn-primary:hover {
|
||||
background-color: var(--accent-hover);
|
||||
}
|
||||
|
||||
.btn-secondary {
|
||||
background-color: var(--bg-input);
|
||||
border-color: var(--border);
|
||||
color: var(--text-main);
|
||||
}
|
||||
|
||||
.btn-secondary:hover {
|
||||
background-color: var(--bg-hover);
|
||||
}
|
||||
|
||||
.btn-ghost {
|
||||
background: transparent;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.btn-ghost:hover {
|
||||
color: var(--text-main);
|
||||
background: var(--bg-hover);
|
||||
}
|
||||
|
||||
.btn-sm {
|
||||
padding: 6px 12px;
|
||||
font-size: 0.8rem;
|
||||
}
|
||||
|
||||
.btn-icon {
|
||||
padding: 6px;
|
||||
border-radius: 6px;
|
||||
}
|
||||
|
||||
/* Status Banner */
|
||||
.status-banner {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 16px;
|
||||
background: var(--accent-light);
|
||||
border: 1px solid var(--accent);
|
||||
border-radius: var(--radius);
|
||||
padding: 16px 20px;
|
||||
margin-bottom: 24px;
|
||||
}
|
||||
|
||||
.status-spinner {
|
||||
width: 24px;
|
||||
height: 24px;
|
||||
border: 3px solid rgba(88, 166, 255, 0.3);
|
||||
border-top-color: var(--accent);
|
||||
border-radius: 50%;
|
||||
animation: spin 1s infinite linear;
|
||||
}
|
||||
|
||||
.btn-spinner {
|
||||
width: 14px;
|
||||
height: 14px;
|
||||
border: 2px solid rgba(255, 255, 255, 0.3);
|
||||
border-top-color: #fff;
|
||||
border-radius: 50%;
|
||||
animation: spin 1s infinite linear;
|
||||
}
|
||||
|
||||
@keyframes spin {
|
||||
to { transform: rotate(360deg); }
|
||||
}
|
||||
|
||||
.status-title {
|
||||
font-weight: 600;
|
||||
font-size: 0.95rem;
|
||||
}
|
||||
|
||||
.status-desc {
|
||||
font-size: 0.85rem;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
/* Meta Card */
|
||||
.meta-card {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
padding: 16px 24px;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
|
||||
.author-profile {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 14px;
|
||||
}
|
||||
|
||||
.author-avatar {
|
||||
width: 48px;
|
||||
height: 48px;
|
||||
border-radius: 50%;
|
||||
object-fit: cover;
|
||||
border: 2px solid var(--border);
|
||||
}
|
||||
|
||||
.author-meta h2 {
|
||||
font-size: 1.1rem;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.author-handle {
|
||||
color: var(--accent);
|
||||
text-decoration: none;
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
|
||||
.author-handle:hover {
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
.thread-stats {
|
||||
display: flex;
|
||||
gap: 20px;
|
||||
}
|
||||
|
||||
.stat-box {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: flex-end;
|
||||
}
|
||||
|
||||
.stat-number {
|
||||
font-size: 1.2rem;
|
||||
font-weight: 700;
|
||||
color: var(--text-main);
|
||||
}
|
||||
|
||||
.stat-label {
|
||||
font-size: 0.75rem;
|
||||
color: var(--text-muted);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.05em;
|
||||
}
|
||||
|
||||
/* Action Bar */
|
||||
.action-bar {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
margin-bottom: 16px;
|
||||
flex-wrap: wrap;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.tab-group {
|
||||
display: flex;
|
||||
gap: 4px;
|
||||
background: var(--bg-card);
|
||||
border: 1px solid var(--border);
|
||||
padding: 4px;
|
||||
border-radius: 8px;
|
||||
}
|
||||
|
||||
.tab-btn {
|
||||
background: transparent;
|
||||
border: none;
|
||||
color: var(--text-muted);
|
||||
padding: 6px 14px;
|
||||
font-size: 0.85rem;
|
||||
font-weight: 500;
|
||||
border-radius: 6px;
|
||||
cursor: pointer;
|
||||
transition: all 0.2s;
|
||||
font-family: var(--font);
|
||||
}
|
||||
|
||||
.tab-btn.active {
|
||||
background: var(--bg-hover);
|
||||
color: var(--text-main);
|
||||
}
|
||||
|
||||
.export-group {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
position: relative;
|
||||
}
|
||||
|
||||
/* Dropdown */
|
||||
.dropdown {
|
||||
position: relative;
|
||||
}
|
||||
|
||||
.dropdown-menu {
|
||||
position: absolute;
|
||||
top: 100%;
|
||||
right: 0;
|
||||
margin-top: 6px;
|
||||
background: var(--bg-card);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
box-shadow: var(--shadow);
|
||||
display: none;
|
||||
flex-direction: column;
|
||||
min-width: 220px;
|
||||
z-index: 100;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.dropdown-menu.show {
|
||||
display: flex;
|
||||
}
|
||||
|
||||
.dropdown-item {
|
||||
padding: 10px 14px;
|
||||
color: var(--text-main);
|
||||
text-decoration: none;
|
||||
font-size: 0.85rem;
|
||||
transition: background 0.2s;
|
||||
}
|
||||
|
||||
.dropdown-item:hover {
|
||||
background: var(--bg-hover);
|
||||
}
|
||||
|
||||
/* Views */
|
||||
.view-card {
|
||||
min-height: 400px;
|
||||
}
|
||||
|
||||
.reader-typography {
|
||||
font-size: 1.1rem;
|
||||
line-height: 1.7;
|
||||
color: #e6edf3;
|
||||
}
|
||||
|
||||
.reader-typography h1,
|
||||
.reader-typography h2 {
|
||||
margin: 1.5rem 0 1rem;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.reader-typography p {
|
||||
margin-bottom: 1.25rem;
|
||||
}
|
||||
|
||||
.reader-typography ul {
|
||||
margin-bottom: 1.25rem;
|
||||
padding-left: 1.5rem;
|
||||
}
|
||||
|
||||
.reader-typography li {
|
||||
margin-bottom: 0.5rem;
|
||||
}
|
||||
|
||||
.reader-typography hr {
|
||||
border: 0;
|
||||
border-top: 1px solid var(--border);
|
||||
margin: 2rem 0;
|
||||
}
|
||||
|
||||
.reader-typography img {
|
||||
max-width: 100%;
|
||||
border-radius: 10px;
|
||||
margin: 1rem 0;
|
||||
}
|
||||
|
||||
/* Markdown view */
|
||||
#view-markdown pre {
|
||||
background: var(--bg-input);
|
||||
padding: 18px;
|
||||
border-radius: 8px;
|
||||
overflow-x: auto;
|
||||
font-family: var(--font-mono);
|
||||
font-size: 0.88rem;
|
||||
line-height: 1.5;
|
||||
color: #c9d1d9;
|
||||
}
|
||||
|
||||
/* Segments breakdown */
|
||||
.segment-item {
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
padding: 16px;
|
||||
margin-bottom: 14px;
|
||||
background: var(--bg-input);
|
||||
}
|
||||
|
||||
.segment-header {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
margin-bottom: 8px;
|
||||
font-size: 0.8rem;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.segment-num {
|
||||
font-weight: 600;
|
||||
color: var(--accent);
|
||||
}
|
||||
|
||||
.segment-text {
|
||||
font-size: 0.95rem;
|
||||
line-height: 1.5;
|
||||
white-space: pre-wrap;
|
||||
}
|
||||
|
||||
/* Library Sidebar */
|
||||
.library-sidebar {
|
||||
position: fixed;
|
||||
top: 0;
|
||||
right: -360px;
|
||||
width: 340px;
|
||||
height: 100vh;
|
||||
background: var(--bg-card);
|
||||
border-left: 1px solid var(--border);
|
||||
box-shadow: -4px 0 24px rgba(0,0,0,0.4);
|
||||
transition: right 0.3s ease;
|
||||
z-index: 200;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
.library-sidebar.open {
|
||||
right: 0;
|
||||
}
|
||||
|
||||
.sidebar-header {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
padding: 20px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.sidebar-search {
|
||||
padding: 12px 20px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.sidebar-search input {
|
||||
width: 100%;
|
||||
background: var(--bg-input);
|
||||
border: 1px solid var(--border);
|
||||
color: var(--text-main);
|
||||
padding: 8px 12px;
|
||||
border-radius: 6px;
|
||||
font-size: 0.85rem;
|
||||
outline: none;
|
||||
}
|
||||
|
||||
.sidebar-list {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 16px;
|
||||
}
|
||||
|
||||
.history-item {
|
||||
background: var(--bg-input);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
padding: 12px;
|
||||
margin-bottom: 10px;
|
||||
cursor: pointer;
|
||||
transition: border-color 0.2s, background 0.2s;
|
||||
}
|
||||
|
||||
.history-item:hover {
|
||||
border-color: var(--border-focus);
|
||||
background: var(--bg-hover);
|
||||
}
|
||||
|
||||
.history-item-title {
|
||||
font-size: 0.9rem;
|
||||
font-weight: 600;
|
||||
margin-bottom: 4px;
|
||||
display: -webkit-box;
|
||||
-webkit-line-clamp: 2;
|
||||
-webkit-box-orient: vertical;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.history-item-meta {
|
||||
font-size: 0.75rem;
|
||||
color: var(--text-muted);
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
}
|
||||
|
||||
.empty-state {
|
||||
text-align: center;
|
||||
color: var(--text-muted);
|
||||
font-size: 0.85rem;
|
||||
padding: 30px 0;
|
||||
}
|
||||
|
||||
/* Toast */
|
||||
.toast {
|
||||
position: fixed;
|
||||
bottom: 24px;
|
||||
left: 50%;
|
||||
transform: translateX(-50%);
|
||||
background: #238636;
|
||||
color: #fff;
|
||||
padding: 10px 20px;
|
||||
border-radius: 8px;
|
||||
font-size: 0.9rem;
|
||||
font-weight: 500;
|
||||
box-shadow: 0 6px 20px rgba(0,0,0,0.3);
|
||||
z-index: 999;
|
||||
transition: opacity 0.3s;
|
||||
}
|
||||
|
||||
.hidden {
|
||||
display: none !important;
|
||||
}
|
||||
@@ -0,0 +1,233 @@
|
||||
import re
|
||||
from typing import List
|
||||
from .models import PostSegment
|
||||
|
||||
def clean_segment_text(text: str) -> str:
|
||||
"""Clean up whitespace and trailing artifacts from text."""
|
||||
lines = [l.strip() for l in text.split("\n")]
|
||||
# Remove empty lines while preserving deliberate paragraph breaks
|
||||
cleaned_lines = []
|
||||
prev_blank = False
|
||||
for line in lines:
|
||||
if line:
|
||||
cleaned_lines.append(line)
|
||||
prev_blank = False
|
||||
elif not prev_blank:
|
||||
cleaned_lines.append("")
|
||||
prev_blank = True
|
||||
return "\n".join(cleaned_lines).strip()
|
||||
|
||||
def build_markdown(author_name: str, author_handle: str, canonical_url: str, segments: List[PostSegment]) -> str:
|
||||
"""Stitches segments into a single cohesive Markdown document."""
|
||||
lines = []
|
||||
title = segments[0].text.split("\n")[0] if segments and segments[0].text else "Threads Post"
|
||||
if len(title) > 80:
|
||||
title = title[:77] + "..."
|
||||
|
||||
lines.append(f"# {title}")
|
||||
lines.append("")
|
||||
lines.append(f"**Author:** [{author_name} (@{author_handle})](https://www.threads.net/@{author_handle}) ")
|
||||
lines.append(f"**Original Post:** [{canonical_url}]({canonical_url}) ")
|
||||
lines.append(f"**Thread Segments:** {len(segments)}")
|
||||
lines.append("")
|
||||
lines.append("---")
|
||||
lines.append("")
|
||||
|
||||
for idx, seg in enumerate(segments):
|
||||
cleaned_text = clean_segment_text(seg.text)
|
||||
if not cleaned_text and not seg.images:
|
||||
continue
|
||||
|
||||
if cleaned_text:
|
||||
lines.append(cleaned_text)
|
||||
lines.append("")
|
||||
|
||||
# Include media images if any
|
||||
for img in seg.images:
|
||||
alt = img.alt_text or f"Thread Image {idx + 1}"
|
||||
lines.append(f"")
|
||||
lines.append("")
|
||||
|
||||
lines.append("---")
|
||||
lines.append(f"*Ripped with Thread Ripper from [Threads]({canonical_url})*")
|
||||
return "\n".join(lines).strip()
|
||||
|
||||
def build_plain_text(author_name: str, author_handle: str, canonical_url: str, segments: List[PostSegment]) -> str:
|
||||
"""Stitches segments into clean plain text."""
|
||||
output = []
|
||||
output.append(f"{author_name} (@{author_handle})")
|
||||
output.append(f"Source: {canonical_url}")
|
||||
output.append("=" * 60)
|
||||
output.append("")
|
||||
|
||||
for seg in segments:
|
||||
cleaned_text = clean_segment_text(seg.text)
|
||||
if cleaned_text:
|
||||
output.append(cleaned_text)
|
||||
output.append("")
|
||||
|
||||
output.append("=" * 60)
|
||||
return "\n".join(output).strip()
|
||||
|
||||
def build_standalone_html(author_name: str, author_handle: str, author_avatar: str, canonical_url: str, segments: List[PostSegment]) -> str:
|
||||
"""Generates an elegant, responsive standalone HTML file."""
|
||||
title = segments[0].text.split("\n")[0] if segments and segments[0].text else f"Thread by @{author_handle}"
|
||||
if len(title) > 80:
|
||||
title = title[:77] + "..."
|
||||
|
||||
content_sections = []
|
||||
for idx, seg in enumerate(segments):
|
||||
cleaned_text = clean_segment_text(seg.text)
|
||||
paragraphs = cleaned_text.split("\n")
|
||||
section_html = "<div class='thread-segment'>"
|
||||
|
||||
# Render bullet points or paragraphs cleanly
|
||||
in_list = False
|
||||
for p in paragraphs:
|
||||
p = p.strip()
|
||||
if not p:
|
||||
continue
|
||||
if p.startswith("- ") or p.startswith("• "):
|
||||
if not in_list:
|
||||
section_html += "<ul>"
|
||||
in_list = True
|
||||
section_html += f"<li>{p[2:]}</li>"
|
||||
else:
|
||||
if in_list:
|
||||
section_html += "</ul>"
|
||||
in_list = False
|
||||
section_html += f"<p>{p}</p>"
|
||||
if in_list:
|
||||
section_html += "</ul>"
|
||||
|
||||
# Attached images
|
||||
for img in seg.images:
|
||||
section_html += f"<div class='segment-image'><img src='{img.url}' alt='{img.alt_text or ''}' loading='lazy' /></div>"
|
||||
|
||||
section_html += "</div>"
|
||||
content_sections.append(section_html)
|
||||
|
||||
body_content = "\n".join(content_sections)
|
||||
avatar_tag = f"<img class='avatar' src='{author_avatar}' alt='{author_handle}' />" if author_avatar else ""
|
||||
|
||||
html = f"""<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>{title} - Thread Ripper</title>
|
||||
<style>
|
||||
:root {{
|
||||
--bg: #0e1117;
|
||||
--card: #161b22;
|
||||
--text: #e6edf3;
|
||||
--text-muted: #8b949e;
|
||||
--border: #30363d;
|
||||
--accent: #58a6ff;
|
||||
}}
|
||||
@media (prefers-color-scheme: light) {{
|
||||
:root {{
|
||||
--bg: #f6f8fa;
|
||||
--card: #ffffff;
|
||||
--text: #1f2328;
|
||||
--text-muted: #656d76;
|
||||
--border: #d0d7de;
|
||||
--accent: #0969da;
|
||||
}}
|
||||
}}
|
||||
body {{
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
line-height: 1.65;
|
||||
margin: 0;
|
||||
padding: 32px 16px;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
}}
|
||||
.article {{
|
||||
max-width: 680px;
|
||||
width: 100%;
|
||||
background: var(--card);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 16px;
|
||||
padding: 40px;
|
||||
box-shadow: 0 8px 30px rgba(0,0,0,0.12);
|
||||
}}
|
||||
.header {{
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 16px;
|
||||
margin-bottom: 28px;
|
||||
padding-bottom: 20px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}}
|
||||
.avatar {{
|
||||
width: 54px;
|
||||
height: 54px;
|
||||
border-radius: 50%;
|
||||
object-fit: cover;
|
||||
}}
|
||||
.author-info h2 {{
|
||||
margin: 0;
|
||||
font-size: 1.25rem;
|
||||
}}
|
||||
.author-info a {{
|
||||
color: var(--accent);
|
||||
text-decoration: none;
|
||||
font-size: 0.95rem;
|
||||
}}
|
||||
.thread-segment {{
|
||||
margin-bottom: 24px;
|
||||
font-size: 1.1rem;
|
||||
}}
|
||||
.thread-segment p {{
|
||||
margin: 0 0 16px 0;
|
||||
}}
|
||||
.thread-segment ul {{
|
||||
margin: 0 0 16px 0;
|
||||
padding-left: 24px;
|
||||
}}
|
||||
.thread-segment li {{
|
||||
margin-bottom: 8px;
|
||||
}}
|
||||
.segment-image img {{
|
||||
max-width: 100%;
|
||||
border-radius: 12px;
|
||||
margin-top: 12px;
|
||||
}}
|
||||
.footer {{
|
||||
margin-top: 36px;
|
||||
padding-top: 20px;
|
||||
border-top: 1px solid var(--border);
|
||||
font-size: 0.85rem;
|
||||
color: var(--text-muted);
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
}}
|
||||
.footer a {{
|
||||
color: var(--accent);
|
||||
text-decoration: none;
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<article class="article">
|
||||
<div class="header">
|
||||
{avatar_tag}
|
||||
<div class="author-info">
|
||||
<h2>{author_name}</h2>
|
||||
<a href="{canonical_url}" target="_blank">@{author_handle} on Threads</a>
|
||||
</div>
|
||||
</div>
|
||||
<div class="thread-body">
|
||||
{body_content}
|
||||
</div>
|
||||
<div class="footer">
|
||||
<span>Reconstructed as 1 single post ({len(segments)} segments)</span>
|
||||
<a href="{canonical_url}" target="_blank">View Original Post ↗</a>
|
||||
</div>
|
||||
</article>
|
||||
</body>
|
||||
</html>"""
|
||||
return html
|
||||
@@ -0,0 +1,76 @@
|
||||
import os
|
||||
import json
|
||||
from typing import List, Optional
|
||||
from pathlib import Path
|
||||
from .models import RippedThread, HistorySummary
|
||||
|
||||
class ThreadStorage:
|
||||
def __init__(self, data_dir: str = "data"):
|
||||
self.data_dir = Path(data_dir)
|
||||
self.threads_dir = self.data_dir / "threads"
|
||||
self.threads_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def save(self, thread: RippedThread) -> str:
|
||||
"""Saves a ripped thread to disk in JSON, MD, and HTML formats."""
|
||||
file_path = self.threads_dir / f"{thread.id}.json"
|
||||
with open(file_path, "w", encoding="utf-8") as f:
|
||||
f.write(thread.model_dump_json(indent=2))
|
||||
|
||||
# Also save standalone .md and .html files for direct browsing
|
||||
md_path = self.threads_dir / f"{thread.id}.md"
|
||||
with open(md_path, "w", encoding="utf-8") as f:
|
||||
f.write(thread.stitched_markdown)
|
||||
|
||||
html_path = self.threads_dir / f"{thread.id}.html"
|
||||
with open(html_path, "w", encoding="utf-8") as f:
|
||||
f.write(thread.stitched_html)
|
||||
|
||||
txt_path = self.threads_dir / f"{thread.id}.txt"
|
||||
with open(txt_path, "w", encoding="utf-8") as f:
|
||||
f.write(thread.stitched_text)
|
||||
|
||||
return thread.id
|
||||
|
||||
def get(self, thread_id: str) -> Optional[RippedThread]:
|
||||
"""Loads a saved ripped thread by ID."""
|
||||
file_path = self.threads_dir / f"{thread_id}.json"
|
||||
if not file_path.exists():
|
||||
return None
|
||||
with open(file_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
return RippedThread(**data)
|
||||
|
||||
def list_history(self) -> List[HistorySummary]:
|
||||
"""Lists summaries of all previously ripped threads, newest first."""
|
||||
history = []
|
||||
for file in self.threads_dir.glob("*.json"):
|
||||
try:
|
||||
with open(file, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
history.append(
|
||||
HistorySummary(
|
||||
id=data["id"],
|
||||
title=data.get("title", "Untitled Thread"),
|
||||
author_handle=data.get("author_handle", "unknown"),
|
||||
author_name=data.get("author_name", "Unknown"),
|
||||
segment_count=data.get("segment_count", len(data.get("segments", []))),
|
||||
total_words=data.get("total_words", 0),
|
||||
saved_at=str(data.get("saved_at", ""))[:19],
|
||||
original_url=data.get("original_url", "")
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
continue
|
||||
# Sort by saved_at descending
|
||||
history.sort(key=lambda x: x.saved_at, reverse=True)
|
||||
return history
|
||||
|
||||
def delete(self, thread_id: str) -> bool:
|
||||
"""Deletes a saved thread and all associated format files."""
|
||||
deleted = False
|
||||
for ext in [".json", ".md", ".html", ".txt"]:
|
||||
f = self.threads_dir / f"{thread_id}{ext}"
|
||||
if f.exists():
|
||||
f.unlink()
|
||||
deleted = True
|
||||
return deleted
|
||||
Reference in new issue
Block a user