I have successfully converted PDF to markdown.
Then I used Jina segmenter to split it into chunks.
Each chunk is ~1000 characters long, but sometimes it is as short as just the title of the section.
I have then stored all of these chunks in a vector database and use cosine distance to sort chunks, pick the first 100, and include the associated chunks into LLM prompt that's used to answer user's question.
However...
I feel like I am missing a step.
The chunks that are returned by the query, while mostly relevant, they ...
* do not include the full recipe * include snippets of unrelated recipes
Is there a step am I missing?