Ask My Thesis · José Leitão

Purpose

Nobody reads a master's thesis cover to cover - not even the people who are asked to. So I built an assistant that does it for them: ask a question in Portuguese or English and get an answer in that same language, with the thesis sections it was drawn from cited underneath. If the thesis doesn't cover the question, the assistant says so rather than inventing an answer.

The project is built end to end: PDF preprocessing, chunking, embedding and indexing, retrieval, grounded generation, a full evaluation suite and a Streamlit chat UI. It was presented as the final project for the EDIT. - Deep Learning with TensorFlow bootcamp.

Highlights

Bilingual, grounded answers. European Portuguese and English, answered only from retrieved context, with section citations.
Refusal by design. Out-of-scope and unanswerable questions are declined in the question's language - 100% refusal on 50 negative test cases.
Fully local, evaluate what you run. Generation, embeddings and the evaluation judge all run locally (Ollama + bge-m3) - no API keys, no usage costs, and the app runs exactly the model that was evaluated.
Rigorously evaluated. Retrieval (recall@k, hit@k, MRR) and generation (RAGAS faithfulness, answer relevancy, semantic similarity, plus a refusal check) both scored against a hand-built golden set.

Architecture

The pipeline has two halves. Preprocessing and indexing run once, offline: the thesis PDF is split into sections, cleaned, chunked and embedded into a vector database. Retrieval and generation run per question: the query is embedded, the closest chunks are retrieved, and the LLM writes an answer grounded only in them.

More technical information

Preprocessing: pdfplumber extracts the text into structured sections with metadata, then a cleaning step dehyphenates, reflows paragraphs and normalizes the text.
Chunking: recursive, token-aware chunks of roughly 500 tokens, each with its section title prepended.
Embeddings and retrieval: bge-m3 (BAAI) in a persisted Chroma collection with cosine distance - strong multilingual embeddings for a PT/EN corpus. The app retrieves the top 4 chunks per question.
Generation: Qwen2.5-7B-Instruct, run locally as qwen2.5:7b on Ollama - the same model in the app and in the evaluation, with greedy decoding for reproducibility.
Safety nets: the index is stamped with its embedding model, and retrieval refuses to load an index built with a different one, so query and index can never silently diverge.
Streaming: answers appear in the chat word by word as the model writes them, instead of all at once after a spinner.
Traceable results: every evaluation report records the git commit and settings it was produced with, so any number can be traced back to the exact code behind it.

How grounding works

The system prompt binds the model to a few rules: answer only from the provided context, decline briefly when the context doesn't cover the question, and reply in the question's language.

That last rule turned out to be the tricky one. A Portuguese context pulls a small model toward answering in Portuguese, even when the question is in English. So the reply language is detected deterministically in code and forced via a directive placed right next to the question, instead of trusting the model to pick it.

Evaluation

Everything is scored against a hand-built golden set of 123 questions (84 PT / 39 EN):

73 answerable - questions the thesis answers, each with relevant section labels and a reference answer
27 out of scope - off-topic questions that should be declined
23 unanswerable on topic - on-topic questions the thesis never actually answers, the hard hallucination case

Retrieval

Retrieval is evaluated entirely locally, sweeping k and search type (similarity vs MMR). It saturates at k=4, with a hit rate of 0.99 and an MRR of 0.88 - a fifth chunk adds nothing - so both the app and the generation eval retrieve 4 chunks per question. Plain similarity search beat MMR at every k above 1 on this corpus.

Generation

Generation is judged by gemma3:12b running locally. On the answerable questions it scores 0.98 faithfulness, 0.85 answer relevancy and 0.82 semantic similarity. On the negatives, it correctly declined 27/27 out-of-scope and 23/23 unanswerable on-topic questions.

Why a judge from a different model family?

LLM judges tend to rate text from their own model family more favorably - a self-preference bias. Pombal et al. (2026) show it holds even for simple yes/no verdicts on objective criteria - a good and interesting read if you work with LLM judges.

Since the generator is a Qwen model, the judge is Google's gemma3:12b: free and deterministic when run locally, and from a different model family, so it has no reason to favor the answers it is scoring.

What I learned

Retrieval is the principal part of building a RAG system. If the right chunks aren't retrieved, no model can give a good answer - so I evaluated retrieval first and only moved on to generation once it was near-saturated.
Negative questions matter as much as answerable ones. A distance cutoff screens off-topic questions, but only an explicit unanswerable-on-topic set shows whether the model hallucinates.
Don't trust a small model with what code can do deterministically. Language detection now lives in code, not in the prompt.
Evaluate what you ship, and keep it stable. I first also hosted the model on Hugging Face, until its providers started dropping it - each drop would have meant serving a model I never evaluated. Running it locally keeps the evaluated model and the served one the same, for good.
Never let a model grade its own family. LLM judges favor answers from their own model family, so the judge should always come from a different family than the generator.