Going from PDF to Chunks the smart way · AI Developer Accelerator

Going from PDF to Chunks the smart way

I got asked on yesterdays call about how to take a PDF into a more consistent way into chunks for RAG. The first challenge you have with converting any PDF file is dealing with the unique underlying way that the PDF document may be formated. Much of that formating has no impact on the printed output but does have an impact if you are using python to extract with Langchain making the output often inconsistent with sections often being wrongly aggregated for the chunking process.

A better approach that has worked consistantly for me is to first convert the PDF into Markdown then convert the Markdown into chunking see:

Step One:

import pymupdf4llm

import pathlib

# Convert PDF to markdown

md_text = pymupdf4llm.to_markdown("input.pdf")

# Save markdown to file (optional) perhaps just save as a string

pathlib.Path("output.md").write_bytes(md_text.encode())

Step Two:

from langchain_text_splitters

import MarkdownHeaderTextSplitter

# Define headers to split on