Defuddle — Get the main content of any page as Markdown

Nota de ingest: esta nota fue generada con la propia skill Defuddle (defuddle parse https://github.com/kepano/defuddle --md --frontmatter) como ejemplo práctico de su uso. El frontmatter original extraído (site, domain, language, word_count) se normalizó al formato de esta carpeta.

de·fud·dle /diˈfʌdl/ transitive verb
to remove unnecessary elements from a web page, and make it easily readable.

Beware! Defuddle is very much a work in progress!

Defuddle extracts the main content from web pages. It cleans up web pages by removing clutter like comments, sidebars, headers, footers, and other non-essential elements, leaving only the primary content.

Overview

Defuddle takes a URL or HTML, finds the main content, and returns cleaned HTML or Markdown. Defuddle was created for the browser extension Obsidian Web Clipper, but it is designed to run in any environment.

Defuddle can be used as a replacement for Mozilla Readability with a few differences:

• More forgiving, removes fewer uncertain elements.
• Provides a consistent output for footnotes, math, code blocks, etc.
• Uses a page’s mobile styles to guess at unnecessary elements.
• Extracts more metadata from the page, including schema.org data.

Usage

Browser

import Defuddle from 'defuddle';
 
// Parse the current document
const defuddle = new Defuddle(document);
const result = defuddle.parse();
 
// Access the content and metadata
console.log(result.content);
console.log(result.title);

Node.js

defuddle/node accepts a DOM Document from any implementation (JSDOM, linkedom, happy-dom, etc.).

import { parseHTML } from 'linkedom';
import { Defuddle } from 'defuddle/node';
 
const { document } = parseHTML(html);
const result = await Defuddle(document, 'https://example.com/article', {
  markdown: true
});

Note: for defuddle/node to import properly, the module format in your package.json has to be set to { "type": "module" }

CLI

Defuddle includes a command-line interface for parsing web pages directly from the terminal. You can run it with npx or install it globally. The CLI accepts a file path, a URL, or HTML piped over stdin.

# Parse a local HTML file
npx defuddle parse page.html
 
# Parse a URL
npx defuddle parse https://example.com/article
 
# Parse HTML from stdin
cat page.html | npx defuddle parse
 
# Output as markdown
npx defuddle parse page.html --markdown
 
# Output as JSON with metadata
npx defuddle parse page.html --json
 
# Extract a specific property
npx defuddle parse page.html --property title
 
# Save output to a file
npx defuddle parse page.html --output result.html
 
# Use a custom User-Agent (helps with sites that return 403 to the default UA)
npx defuddle parse https://example.com/article --user-agent "Mozilla/5.0 ..."

CLI Options

Option	Alias	Description
--output <file>	-o	Write output to a file instead of stdout
--markdown	-m	Convert content to markdown format
--md		Alias for --markdown
--json	-j	Output as JSON with metadata and content
--frontmatter	-f	Prepend YAML frontmatter (title, author, source, etc.) to the output
--property <name>	-p	Extract a specific property (e.g., title, description, domain)
--debug		Enable debug mode
--lang <code>	-l	Preferred language (BCP 47, e.g. en, fr, ja)
--user-agent <string>	-u	Custom User-Agent header for HTTP requests (helps with 403/FORBIDDEN responses)

When no <source> argument is provided, defuddle parse reads HTML from stdin. You can also pass - explicitly to force stdin input.

Installation

npm install defuddle

For Node.js usage, install a DOM implementation (linkedom or jsdom).

CLI installation

npm install -g defuddle

Or use npx to run the CLI without installing globally.

Response

Defuddle returns an object with the following properties:

Property	Type	Description
author	string	Author of the article
content	string	Cleaned up string of the extracted content
description	string	Description or summary of the article
domain	string	Domain name of the website
favicon	string	URL of the website’s favicon
image	string	URL of the article’s main image
language	string	Language of the page in BCP 47 format (e.g. en, en-US)
metaTags	object	Meta tags
parseTime	number	Time taken to parse the page in milliseconds
published	string	Publication date of the article
site	string	Name of the website
schemaOrgData	object	Raw schema.org data extracted from the page
title	string	Title of the article
wordCount	number	Total number of words in the extracted content
debug	object	Debug info including content selector, removals, and caught errors (when debug: true)

Bundles

Defuddle is available in three different bundles:

1. Core bundle (defuddle): The main bundle for browser usage. No dependencies.
2. Full bundle (defuddle/full): Includes additional features for math equation parsing and Markdown conversion.
3. Node.js bundle (defuddle/node): For Node.js environments. Accepts any DOM Document (e.g. from linkedom, JSDOM, or happy-dom). Includes full capabilities for math and Markdown conversion.

HTML standardization

Defuddle attempts to standardize HTML elements to provide a consistent input for subsequent manipulation such as conversion to Markdown.

• Headings: The first H1 or H2 heading is removed if it matches the title. H1s are converted to H2s. Anchor links in headings are removed.
• Code blocks: Line numbers and syntax highlighting are removed, but the language is retained as a data attribute and class.
• Footnotes: Inline references and footnotes are converted to a standard format.
• Math: Math elements, including MathJax and KaTeX, are converted to standard MathML.
• Callouts: Callout and alert elements from various sources (GitHub markdown alerts, Obsidian callouts, Bootstrap alerts) are standardized to blockquotes with a data-callout attribute. In Markdown these become Obsidian-style callouts:

> [!info] Info
> This is an informational callout.

Third-party services

When using parseAsync(), if no content can be extracted from the local HTML, Defuddle may fetch content from third-party APIs as a fallback (e.g. FxTwitter API for X/Twitter content). This only happens when the page HTML contains no usable content (e.g. client-side rendered SPAs). You can disable this with useAsync: false.

Debugging

Enable debug mode to get a debug field with contentSelector, removals (list of removed elements with reasons), and errors. You can disable individual pipeline steps (removeLowScoring, removeHiddenElements, removeSmallImages) or bypass auto-detection with a custom contentSelector.