Docurip Documentation

General Information

Docurip is a Tauri v2 desktop application for crawling documentation websites and converting them into offline Markdown archives. It combines a Rust backend (high-performance crawling, security filtering, and disk I/O) with a React 19 / TypeScript / Tailwind CSS frontend.

• Current version: v0.5.2
• Platforms: Windows (primary target), built with Tauri v2
• Architecture: Rust backend + React 19 frontend
• Default output directory: ~/.docurip (falls back to ./output if the home directory cannot be resolved)
• Data storage: All job metadata and application settings are stored locally on your machine. No cloud service is used.

Detailed Features

Crawling Engine

Docurip’s crawler is built around a parallel, queue-based fetch loop:

Crawl Profiles

Since v0.5.0, Docurip includes pre-configured crawl profiles that set sensible defaults for common documentation types. Select a profile in the New Crawl view to auto-fill maxDepth, pageLimit, respectRobotsTxt, content selectors, and exclude patterns.

Selecting a profile sets the initial values; you can still adjust any field before starting the crawl.

Crawl Configuration

Each crawl job has its own configuration. Defaults are taken from Settings unless overridden when starting a crawl.

Application Settings

Settings are persisted across sessions in %APPDATA%/com.docurip.app/settings.json via tauri-plugin-store.

Security & Safety

Docurip includes several protections so it can be run safely on public or internal networks:

• SSRF protection – Prevents the crawler from requesting loopback, RFC 1918, link-local, IPv6 ULA, localhost, *.local, and hostnames resolving to private IPs. Applied to the start URL at validation time and to every discovered link during the crawl.
• robots.txt enforcement – Respects the target site’s crawler rules before following links.
• Domain restriction – stayWithinDomain limits crawling to the start URL’s host.
• Asset MIME-type allow-list – Downloaded assets are checked against an allow-list (image/*, font/*, text/css, text/javascript, application/javascript, application/json, application/pdf, application/octet-stream, audio/*, video/*, etc.). text/html and application/xhtml+xml are rejected so login pages or error pages are not saved as broken assets.
• Asset size cap – Assets larger than 50 MB are rejected before the response body is consumed.
• Path sanitization – Filenames and paths are sanitized to block . and .. segments, preventing directory traversal. Query strings and fragments are stripped from page filenames.

Output Organization

Since v0.3.2, every crawl creates a subfolder named after the target domain:

{outputDir}/
  {domain}/
    main/      # Crawled Markdown files
    zip/       # ZIP exports
    formats/   # Markdown files, merged Markdown, PDF files, merged PDF, HTML files, merged HTML, JSON files, merged JSON

For example, crawling https://docs.example.com with the default output directory produces ~/.docurip/docs.example.com/main/ containing the Markdown pages.

Exports

Completed jobs can be exported from the History view or the Result Browser.

The app detects headless support at runtime and disables PDF options when the feature is not compiled in.

Markdown Cleaning

Since v0.5.0, the HTML-to-Markdown pipeline includes automatic cleaning steps to produce cleaner output:

• Script/style removal – <script> and <style> tags with their content are stripped before Markdown conversion.
• Empty link removal – <a href="#"> tags without meaningful content are removed before conversion.
• Boilerplate filtering – After conversion, common UI noise is removed: "Copy page" / "Edit page" links, cookie banners, newsletter signups, and "Copy code" buttons.
• Post-processing – Excessive blank lines are collapsed, empty link URLs ([text]()) are unwrapped to plain text, and broken images (![alt]()) are removed.
• Trailing heading stubs – Orphaned heading lines at the end of a document (often from navigation fragments) are stripped.

PDF/EPUB Importer

Starting with v0.5.0, Docurip features a local importer for PDF and EPUB documents, allowing you to convert local manuals, books, or guides to clean Markdown offline.

• Drag & Drop: Drop local files directly into the Import view. Uses Tauri's native file drop event handlers for reliable file handling.
• Automatic Image Extraction: Automatic extraction of images from PDF and EPUB files, saved in local subfolders and referenced via relative Markdown image links.
• Text Layout Cleaner: Configurable pipeline that detects and strips headers, footers, page numbers, footnotes, and boilerplate content using zone-restricted pattern matching and cross-page frequency analysis. Toggleable in the Import UI (enabled by default).
• Rust Engine: Powered by the high-performance pdf_extract and epub crates.

User Interface

Job Persistence

• Job metadata is saved as JSON files in %APPDATA%/com.docurip.app/jobs/{jobId}.json.
• Jobs are persisted after each page is processed and on pause / stop / completion.
• Restarting the app reloads all persisted jobs into the History view.
• Deleting a job removes its JSON file and output is left untouched.

Live Events & Stats

• Crawl progress, page completion, logs, errors, and status changes are streamed to the UI in real time via Tauri events.
• Dashboard stats are polled every 3 seconds while a crawl is active and roughly every 12 seconds otherwise to reduce idle load.
• Animated stat counters show updated values smoothly.

Updates

Docurip checks for application updates on startup. When an update is available, a banner appears with the new version and release notes. If the install fails, the banner displays the error and switches the action to Retry.

Best-Practice Guides

Crawling Large Documentation Sites (500+ Pages)

1. Scope the crawl first with a low pageLimit (e.g., 50) and shallow maxDepth (2) to understand the site structure before running a full crawl.
2. Use stayWithinDomain to avoid accidentally following external links.
3. Add excludePatterns to skip noise such as /blog/, /changelog/, version-switcher pages, search result pages, or tags/categories. Example patterns:
   • .*/tag/.*
   • .*/search.*
   • .*/changelog/.*
4. Increase pageLimit to the expected number of pages (up to 10,000). The crawler will stop cleanly once the limit is reached.
5. Set a polite requestDelay (1,000–2,000 ms) and keep concurrency moderate (3–5) to avoid overwhelming the documentation server.
6. Use headless auto so JavaScript-rendered pages fall back to Chrome automatically without slowing down simple static pages.
7. Disable downloadAssets unless you need offline images or stylesheets; this significantly reduces disk usage and crawl time.
8. Respect robots.txt unless the site explicitly blocks useful documentation paths. If it does, consider contacting the site owner rather than disabling the check.
9. Split huge sites into multiple jobs by starting from different sections (e.g., /docs/api/, /docs/guides/) instead of one enormous crawl.
10. Monitor the live console for repeated errors or SSRF blocks, and adjust exclude patterns accordingly.

Preparing Docs for LLM / RAG Use

Docurip’s Markdown output is ideal for feeding into retrieval-augmented generation (RAG) pipelines, embedding models, or large language model (LLM) context windows.

1. Use content selectors to strip navigation, headers, footers, and sidebars. Good selectors:
   • main
   • article
   • .content
   • .markdown-body
   • article div.prose
2. Disable asset downloads unless the model needs images or the layout must be preserved. Most text-only pipelines do not need CSS, images, or scripts.
3. Export as Merged Markdown for a single file that is easy to split into chunks. The --- separators between pages make natural chunk boundaries.
4. Exclude noisy pages such as:
   • Changelogs and release notes with many version-specific tables
   • Search result pages
   • Tag/category indexes
   • Login / account pages
5. Keep respectRobotsTxt enabled to stay a good citizen while building training or retrieval corpora.
6. For very large corpora, export individual Markdown files and process them with your own chunking script. File paths preserve the original URL structure, making citations easier.
7. Use a sensible crawl velocity (concurrency 3, requestDelay 1000–2000 ms) so the source site is not burdened while you build your dataset.
8. Sanitize before ingestion – Markdown preview in Docurip is already DOMPurify-sanitized, but you should run your own cleanup if you convert Markdown back to HTML for model consumption.

Troubleshooting

Data & Privacy

• Docurip does not send documentation content, job metadata, or usage statistics to any remote service.
• Update checks contact the Tauri updater endpoint configured for the app; no crawl data is included.
• All crawling, conversion, and storage happens locally on your computer.

Version Notes

This documentation reflects Docurip v0.5.2.

v0.5.x Highlights

• Crawl Profiles — Pre-configured presets for API docs, wikis, blogs, and documentation sites.
• HTML Export — New export formats: individual HTML files and merged HTML.
• Markdown Cleaning — Automatic removal of script/style tags, UI boilerplate, empty links, broken images, and excessive blank lines.
• Virtualized Result Tree — The Result Browser file tree uses windowed rendering for large crawl results.
• Lazy-loaded Markdown Preview — The preview component is code-split and loaded on demand.
• Debounced Search — Search input in the Result Browser is debounced (300 ms) to reduce redundant filtering.

v0.4.x Highlights

• Disk-error auto-pause — Crawls pause automatically on permission denied, read-only filesystem, or no space left.
• Typed error classification — All crawl errors are classified by kind (Network, Http, Disk, Parse, Config, Cancelled, Unknown).
• Queue backpressure — Internal queue depth is bounded to prevent unbounded memory growth on large crawls.
• Throttled job persistence — Active job state is persisted at most every 5 seconds to reduce disk I/O.
• Active crawl navigation — Navigate directly to a running crawl from the Dashboard.

v0.3.x Highlights

• MIME-type validation for downloaded assets
• SSRF checks on start URLs
• Throttled dashboard polling
• Headless Chrome compatibility fixes for headless_chrome 1.x