The- most- reliable- chunking- heuristic- is- the- one- nobody- talks- about:- test- your- chunk- sizes- against- real- queries- from- your- actual- users.- A- chunk- size- that- works- perfectly- for- a- benchmark- dataset- can- fail- silently- on- your- specific- corpus- because- your- documents- have- different- paragraph- density,- heading- structure,- or- average- query- length.- Run- retrieval- recall- tests- on- a- sample- of- real- queries- before- locking- in- a- strategy.

##- What- this- means

Chunking- decides- what- the- retriever- can- see- and- what- it- cannot.- The- embedding- model- turns- each- chunk- into- a- vector;- the- retriever- compares- query- vectors- against- those- chunk- vectors.- If- the- chunk- boundaries- cut- through- a- meaningful- passage- —- a- paragraph- that- explains- why- a- specific- method- works,- a- function- that- implements- a- full- algorithm- —- the- retriever- cannot- find- the- complete- answer.- It- can- only- find- fragments.

This- is- not- a- preprocessing- detail.- It- is- part- of- the- product- design.- A- team- that- treats- chunk- size- as- a- fixed- parameter- set- once- and- forgotten- will- make- retrieval- decisions- that- silently- degrade- as- the- corpus- grows,- as- query- patterns- shift,- and- as- embedding- models- change- their- context-window- behaviour.- Chunking- also- interacts- with- your- broader- retrieval- architecture- choices- —- see- when- semantic- search- is- enough- and- when- it- is- not- for- the- retrieval-strategy- context- that- shapes- what- your- chunker- needs- to- deliver.

##- Where- teams- get- it- wrong,- with- specific- consequences

###- Choosing- chunk- size- by- token- count- alone

A- team- sets- chunk- size- to- 512- tokens- across- their- entire- corpus- because- it- fits- the- embedding- model’s- context- window.- The- corpus- includes- warranty- documents- (long- prose- paragraphs),- API- reference- material- (short- code- blocks),- and- customer- support- transcripts- (conversational- turns).- The- fixed- token- window- chops- paragraphs- mid-sentence,- splits- code- blocks- at- arbitrary- lines,- and- concatenates- unrelated- chat- turns- into- single- chunks.

Consequence:- Semantic- search- queries- like- “what- happens- if- my- boiler- leaks- gas- during- installation?”- retrieve- chunks- that- start- mid-sentence- and- end- in- the- middle- of- a- warranty- exclusion- —- neither- the- answer- nor- the- legal- context- is- usable.- The- team- blames- the- embedding- model- when- the- real- problem- is- chunk- geometry.

Practical- fix:- Use- a- splitter- that- respects- document- structure- first,- token- budget- second.- LangChain- offers- splitters- for- character,- recursive- character,- code- language,- and- Markdown- —- choose- the- one- that- matches- your- document- type- rather- than- force-fitting- a- single- strategy- across- everything- [1].

###- Using- overlap- as- a- substitute- for- good- structure

A- team- knows- their- PDF- parser- loses- section- headings.- Rather- than- fixing- the- parser,- they- set- chunk- overlap- to- 50%- so- that- “most- chunks- contain- a- heading- somewhere.”- The- result- is- massive- chunk- duplication- —- the- corpus- stores- 1.5x- the- original- document- volume- —- and- retrieval- returns- near-identical- chunks- from- the- same- passage- because- the- same- sentence- appears- in- two- overlapping- chunks- with- slightly- different- context.

Consequence:- The- reranker- or- downstream- LLM- sees- redundant- content- that- inflates- context- windows- without- adding- information.- A- user- query- that- should- return- one- relevant- passage- returns- three- near-copies.- The- effective- context- window- shrinks- because- half- of- it- is- duplicate- text.

Practical- fix:- Overlap- should- compensate- for- natural- boundary- loss,- not- for- missing- structure.- If- your- source- documents- have- headings,- tables- of- contents,- or- section- markers,- expose- them- to- the- chunker- rather- than- hiding- them- behind- aggressive- overlap.- Use- metadata- to- carry- section- hierarchy- instead- of- relying- on- overlap- to- preserve- continuity- [3].

###- Dropping- metadata- that- later- matters- for- filters- or- citations

A- chunking- pipeline- strips- source- metadata- during- preprocessing- because- “we- only- need- the- text- for- embedding.”- Later,- the- team- wants- to- filter- retrieval- results- by- document- source,- date- range,- or- section- heading- —- but- there- is- no- way- to- do- it.- Every- chunk- is- a- plain- text- blob- with- no- provenance.

Consequence:- A- legal- team- using- RAG- to- search- contract- archives- cannot- restrict- retrieval- to- current-year- agreements.- A- customer- support- system- cannot- cite- which- document- version- a- chunk- came- from.- The- only- option- is- to- re-chunk- the- entire- corpus- with- metadata- tracking,- which- costs- time- and- compute.

Practical- fix:- Carry- at- minimum- a- minimal- metadata- schema- with- every- chunk:

• source_path- —- original- file- or- document- identifier
• heading_hierarchy- —- array- of- ancestor- headings- (e.g.,- ["Chapter- 3",- "Warranty- terms",- "Exclusions"])
• chunk_index- —- position- within- the- parent- document- (for- reassembly)
• page_number- or- equivalent- document-internal- locator

Pinecone,- Weaviate- and- Qdrant- all- support- metadata- filtering- natively- [3][4].- Using- it- costs- nothing- at- chunk- time- and- saves- a- full- corpus- reindex- later.