A document ingestion pipeline has four discrete stages: load, parse, clean, and metadata extraction. Most developers collapse these into one blob of code. That works until a PDF comes in with a scanned cover page, or an HTML page arrives with a React SPA that serves content via JavaScript rather than static markup. Keeping stages separate means you can swap out a parser, add a cleaning step, or backfill metadata without rewriting everything.
Loading is where you decide whether you're reading from the local filesystem, an S3 bucket, a URL, or a database blob column. For a RAG system ingesting from multiple sources, normalize here: every loader returns a raw bytes object plus a source identifier. This means the parser stage never has to care about where the file came from. For URLs, use httpx with a timeout and retry policy rather than requests, especially if you're running ingestion async. For S3, use boto3 with IAM role auth, not hardcoded credentials.
Parsing is where format-specific complexity lives. PDFs are the hardest. A PDF is not a text document -- it's a set of drawing instructions for a renderer. Text extraction is inherently an approximation. pypdf handles most digitally created PDFs well and is fast. pdfminer.six gives you finer layout analysis and is better at recovering reading order in multi-column documents. pdfplumber wraps pdfminer.six with a cleaner API and has table extraction built in. For scanned PDFs (images of pages), none of these work at all -- you need OCR, typically via pytesseract wrapping Tesseract, or a cloud vision API like Google Document AI or AWS Textract. The rule of thumb: if pypdf returns an empty string for a page, that page is an image and needs OCR. For HTML, BeautifulSoup with the lxml parser is the standard, but you must explicitly remove nav, footer, header, script, and style tags before extracting text. Leaving them in pollutes your chunks with "Home | Products | Contact" and minified JavaScript. For JavaScript-rendered pages, BeautifulSoup sees nothing useful -- you need a headless browser like Playwright to render the page first. For Markdown, markdown-it-py or mistune can render to HTML, which you then clean with BeautifulSoup. This round-trip sounds inefficient but preserves more semantic structure than regex-stripping backticks.
Cleaning is often underestimated. After extraction you typically have excessive whitespace, Unicode control characters, broken hyphenation from PDF line wrapping, ligature characters that tokenize oddly (the fi ligature in many PDFs is a single Unicode code point), and encoding artifacts. A basic cleaning pass: normalize Unicode with unicodedata.normalize("NFKC", text), collapse whitespace runs to single spaces, rejoin hyphenated line breaks (re.sub(r"-\n(\w)", r"\1", text)), and strip null bytes. For pages with very low character-to-page-area ratios (a common sign of failed extraction), flag or drop them rather than embedding empty noise. A validation step that checks len(text.strip()) > 50 before passing to the chunker saves a lot of grief.
Metadata extraction should happen at parse time, not after. For PDFs, pypdf exposes document metadata (/Title, /Author, /CreationDate) and you get page number for free. For HTML, scrape the <title> tag, meta description, Open Graph og:url, and <h1> as section title. For Markdown, parse frontmatter (the YAML block between --- delimiters at the top of many Markdown files) with python-frontmatter to get title, date, author, and tags. The metadata dict that travels with each chunk should include at minimum: source (file path or URL), page (integer for PDFs, null otherwise), section_title (nearest heading), document_title, ingested_at (ISO timestamp), and content_hash (MD5 or SHA256 of the raw text, used for deduplication and change detection). That content hash is especially important for incremental ingestion -- if the hash matches what's already in your vector store, skip re-embedding and save the API cost.
At ten documents a day, you can run ingestion synchronously in a script. At ten thousand documents, you need a queue. The pattern that scales well: a producer process walks the source (S3 bucket, database table, crawl results) and enqueues document references to a queue like Redis Streams or SQS. Worker processes consume from the queue, each handling load-parse-clean-metadata for one document and writing the resulting chunks to a staging table. A separate embedding worker batch-embeds from the staging table. This decouples the CPU-heavy parsing from the network-heavy embedding API calls, and you can scale each independently. At ten million documents, you want a dedicated document processing service like Apache Tika or Unstructured.io's hosted API, which handles format detection and extraction as a service so you don't ship Tesseract and pdfminer into every worker container.
Cost and latency implications: parsing is cheap (CPU-bound, no API calls). OCR via Tesseract is slow -- a 200-page scanned PDF takes 60-90 seconds on a standard machine. Cloud OCR (Google Document AI) is faster and more accurate but adds per-page cost. Embedding is where most RAG ingestion cost lives; the more aggressive your cleaning and deduplication before embedding, the less you spend. Incremental ingestion with content hashing can eliminate 40-80% of re-embedding work in a corpus that changes slowly.
Key Takeaways
- Parse each format with a format-aware library; generic text extraction destroys structure and metadata.
- Attach source, page, section, and timestamp metadata at parse time — retrofitting it later is painful.
- Treat ingestion as a pipeline with discrete, testable stages, not a single monolithic script.
- Validate extracted text quality before embedding; bad text in means irretrievable chunks out.
Pro tips
- Use
pypdffor a fast first pass on PDFs, then fall back topdfminer.sixonly whenpypdfreturns less than 100 characters per page. Runningpdfminer.sixon every page adds 3-5x latency for no gain on standard PDFs. - Store the raw extracted text (before chunking) in a separate column or S3 object alongside your vector store entries. When you change your chunking strategy -- and you will -- you can re-chunk from the stored raw text without re-running the expensive OCR or parsing step.
- Content-hash deduplication pays double dividends: it prevents duplicate vectors that skew retrieval scores, and it makes incremental re-ingestion cheap. Compute the hash before cleaning so truly identical source documents collapse even if minor whitespace differs.
- HTML pages rarely have clean linear structure. Splitting by
<h2>or<section>boundaries at ingest time produces semantically coherent chunks naturally, which means the chunking stage has less work to do and you get better retrieval precision on hierarchical documentation sites.
Common pitfalls
- Mistake: Extracting text from every PDF page with
pypdfwithout checking for empty results, silently indexing blank chunks. Fix: Checklen(text.strip()) < 50per page and log a warning; flag those pages for OCR review. - Mistake: Calling
soup.get_text()on raw HTML without removing nav, footer, and script tags first, polluting every chunk with site chrome. Fix: Calltag.decompose()on non-content tags before any text extraction. - Mistake: Generating embeddings immediately inside the ingestion loop, so a single API timeout fails the entire batch. Fix: Separate ingestion (write cleaned chunks to a staging table) from embedding (a separate worker reads from staging).
- Mistake: Discarding metadata like page number and section title at parse time to save schema complexity, making retrieved chunks untraceable. Fix: Attach source, page, section_title, and ingested_at at extraction time; these fields are cheap to store.
Which PDF extraction library to use
| Option | Use when | Avoid when |
|---|---|---|
| pypdf | Digitally created PDFs with selectable text; you need fast batch ingestion at low overhead. | Scanned PDFs, complex multi-column layouts, or PDFs with embedded tables you need to preserve structurally. |
| pdfminer.six / pdfplumber | Multi-column academic papers, PDFs with tables you want to extract as structured data, or when reading order matters. | Simple single-column PDFs at scale; the added latency (3-5x vs pypdf) is not worth it. |
| Tesseract via pytesseract | Scanned PDFs or image-only pages where digital text extraction returns empty strings. | Digitally created PDFs; OCR on text-layer PDFs adds latency and often produces worse output than direct extraction. |
| Cloud OCR (Google Document AI, AWS Textract) | High-volume scanned document ingestion where accuracy and speed matter more than per-page cost. | Low volume or cost-sensitive pipelines; per-page pricing adds up quickly above ~10k pages/month. |
| Unstructured.io (hosted API or library) | Mixed format corpora where you don't want to maintain per-format parsing logic; handles PDF, HTML, DOCX, PPTX. | You need full control over extraction logic, have tight latency budgets, or are processing only one or two formats. |
Code Example
# Uses: pypdf==4.2.0, beautifulsoup4==4.12.3, markdown-it-py==3.0.0
import pathlib
from pypdf import PdfReader
from bs4 import BeautifulSoup
from markdown_it import MarkdownIt
def ingest_pdf(path: str) -> list[dict]:
reader = PdfReader(path)
pages = []
for i, page in enumerate(reader.pages):
text = page.extract_text() or ""
pages.append({"text": text.strip(), "source": path, "page": i + 1})
return pages
def ingest_html(path: str) -> list[dict]:
html = pathlib.Path(path).read_text(encoding="utf-8")
soup = BeautifulSoup(html, "lxml")
for tag in soup(["script", "style", "nav", "footer", "header"]):
tag.decompose()
text = soup.get_text(separator="\n", strip=True)
return [{"text": text, "source": path, "page": None}]
def ingest_markdown(path: str) -> list[dict]:
raw = pathlib.Path(path).read_text(encoding="utf-8")
md = MarkdownIt()
# Strip markup to plain text via rendered HTML then BS4
html = md.render(raw)
text = BeautifulSoup(html, "lxml").get_text(separator="\n", strip=True)
return [{"text": text, "source": path, "page": None}]How this code works
This code defines three distinct functions designed to extract clean, plain text content from different document types: PDF, HTML, and Markdown. This is a crucial first step in a document ingestion pipeline for RAG, converting complex file formats into a uniform text format suitable for further processing. Each function, ingest_pdf, ingest_html, and ingest_markdown, is specialized to handle its respective file type efficiently, utilizing dedicated Python libraries for parsing and text extraction.
The ingest_pdf function uses pypdf's PdfReader to read each page, extracting text. A subtle but important detail is text = page.extract_text() or "", which ensures that even if a page has no text, an empty string is used instead of None, preventing potential errors later. For HTML files, ingest_html employs BeautifulSoup to parse the document, intelligently removing non-content tags like script and style with tag.decompose() before extracting the main text using soup.get_text(). Finally, ingest_markdown processes Markdown files by first converting them to HTML using markdown-it-py's md.render(), then reusing BeautifulSoup to strip the HTML markup, yielding pure text content.
Production-grade example
Adds per-format error handling, OCR-skip logging, retry-with-backoff for URLs, NFKC cleaning, section-level HTML splitting, and content hashing for dedup.
# Uses: pypdf==4.2.0, beautifulsoup4==4.12.3, httpx==0.27.0, tenacity==8.3.0
import hashlib
import logging
import os
import re
import time
import unicodedata
from dataclasses import dataclass, field
from typing import Iterator
import httpx
from bs4 import BeautifulSoup
from pypdf import PdfReader
from pypdf.errors import PdfReadError
from tenacity import retry, stop_after_attempt, wait_exponential
logger = logging.getLogger(__name__)
@dataclass
class Document:
text: str
source: str
page: int | None
section_title: str
document_title: str
ingested_at: str
content_hash: str
extra: dict = field(default_factory=dict)
def _clean_text(raw: str) -> str:
text = unicodedata.normalize("NFKC", raw)
text = re.sub(r"-\n(\w)", r"\1", text) # rejoin PDF hyphenation
text = re.sub(r"[\x00-\x08\x0b\x0c\x0e-\x1f]", "", text) # control chars
text = re.sub(r" {2,}", " ", text)
return text.strip()
def _make_doc(text: str, source: str, page: int | None,
section_title: str = "", document_title: str = "",
extra: dict | None = None) -> Document | None:
cleaned = _clean_text(text)
if len(cleaned) < 50:
logger.warning("Skipping near-empty extraction", extra={"source": source, "page": page})
return None
return Document(
text=cleaned,
source=source,
page=page,
section_title=section_title,
document_title=document_title,
ingested_at=time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
content_hash=hashlib.sha256(cleaned.encode()).hexdigest(),
extra=extra or {},
)
def ingest_pdf(path: str) -> Iterator[Document]:
try:
reader = PdfReader(path)
except PdfReadError as exc:
logger.error("Failed to open PDF", extra={"source": path, "error": str(exc)})
return
meta = reader.metadata or {}
doc_title = str(meta.get("/Title", "")).strip() or os.path.basename(path)
for i, page in enumerate(reader.pages, start=1):
try:
raw = page.extract_text() or ""
except Exception as exc:
logger.warning("Page extraction failed", extra={"source": path, "page": i, "error": str(exc)})
continue
doc = _make_doc(raw, source=path, page=i, document_title=doc_title)
if doc:
yield doc
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def _fetch_url(url: str, timeout: float = 15.0) -> str:
token = os.environ.get("SCRAPER_TOKEN", "")
headers = {"Authorization": f"Bearer {token}"} if token else {}
with httpx.Client(timeout=timeout) as client:
resp = client.get(url, headers=headers, follow_redirects=True)
resp.raise_for_status()
return resp.text
def ingest_html(source: str, *, is_url: bool = False) -> Iterator[Document]:
try:
raw_html = _fetch_url(source) if is_url else open(source, encoding="utf-8").read()
except (httpx.HTTPError, OSError) as exc:
logger.error("Failed to load HTML", extra={"source": source, "error": str(exc)})
return
soup = BeautifulSoup(raw_html, "lxml")
for tag in soup(["script", "style", "nav", "footer", "header", "aside"]):
tag.decompose()
doc_title = soup.title.string.strip() if soup.title and soup.title.string else ""
# Extract per-section documents using h2 boundaries
current_heading = ""
current_parts: list[str] = []
for element in soup.find_all(["h2", "p", "li", "pre", "blockquote"]):
if element.name == "h2":
if current_parts:
doc = _make_doc("\n".join(current_parts), source, None, current_heading, doc_title)
if doc:
yield doc
current_heading = element.get_text(strip=True)
current_parts = []
else:
current_parts.append(element.get_text(strip=True))
if current_parts:
doc = _make_doc("\n".join(current_parts), source, None, current_heading, doc_title)
if doc:
yield docHow this code works
This code defines a robust pipeline for ingesting content from PDF and HTML sources, preparing it for AI applications like RAG. It produces a stream of Document objects, each containing cleaned text, its source (file path or URL), page number (for PDFs), section_title, document_title, a unique content_hash, and an ingested_at timestamp. The core _clean_text function handles common text issues like Unicode normalization, recombining hyphenated words across lines (a PDF commonality), and removing invisible control characters or excess whitespace. The _make_doc helper ensures only substantial text segments (at least 50 characters) become Documents, preventing noisy, near-empty extractions.
For PDFs, the ingest_pdf function iterates through pages using pypdf, extracting text and metadata like the document's title. HTML ingestion is more complex: ingest_html first loads content (from a local file or _fetch_url, which robustly retries failed network requests using httpx and the @retry decorator). It then uses BeautifulSoup to parse the HTML, removing non-content tags like script or style. A subtle but crucial detail is how it then intelligently breaks the HTML into multiple Documents. It segments text primarily around h2 headings, capturing distinct sections of a webpage as separate, meaningful units, each with its own section_title, which greatly improves retrieval relevance compared to processing the entire page as one block.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Build a function ingest_directory(dirpath: str) -> list[dict] that walks a directory containing .pdf, .html, and .md files, parses each using an appropriate library, cleans the text, and returns a flat list of document dicts. Each dict must include: text, source, format, page, and content_hash. Skip any file whose extracted text is under 50 characters.
# Uses: pypdf==4.2.0, beautifulsoup4==4.12.3, markdown-it-py==3.0.0
import hashlib
import pathlib
def clean_text(raw: str) -> str:
# TODO: normalize unicode, collapse whitespace, strip control chars
return raw
def parse_pdf(path: pathlib.Path) -> list[dict]:
# TODO: use PdfReader, extract text per page, return list of dicts
pass
def parse_html(path: pathlib.Path) -> list[dict]:
# TODO: use BeautifulSoup, strip nav/footer/script, extract text
pass
def parse_markdown(path: pathlib.Path) -> list[dict]:
# TODO: render to HTML with MarkdownIt, clean with BeautifulSoup
pass
def ingest_directory(dirpath: str) -> list[dict]:
results = []
for path in pathlib.Path(dirpath).rglob("*"):
# TODO: route to correct parser by suffix, skip short extractions,
# attach content_hash, collect all dicts into results
pass
return resultsQuick check
You run
pypdfon a PDF and get an empty string for every page. What is the most likely cause?Why should you strip
<nav>,<footer>, and<script>tags from HTML before callingget_text()?What is the primary benefit of computing a content hash for each extracted document at ingest time?