Can AI spot that two tasks are secretly the same, even after rewording and renaming, with no human checking?
Can semantic search find paraphrased and renamed tasks without human review?
This explores whether meaning-based (embedding) search can recognize that two tasks are the same problem when one has been reworded or had its names and labels swapped, for example to catch duplicates or benchmark leakage, with no person checking the matches.
This explores whether meaning-based search can spot a task that has been reworded or had its names swapped, with no human checking the matches. The corpus has no note that tests this directly, so the answer below is built from neighboring findings. They suggest: mostly yes for paraphrase, shakily for renaming, and unproven for dropping the human. Matching by meaning is the strong part. Long-context models match standard RAG on purely semantic retrieval without special training, so 'same idea, different words' is a job current systems do well Can long-context LLMs replace retrieval-augmented generation systems?.
Renaming is harder. GrepSeek found that dense embeddings conflate entities: they blur which specific thing a query is about, and searching the raw text with grep-style commands recovers that precision Can direct corpus search beat embedding-based retrieval?. For your problem this cuts both ways (my inference, not something the note tests). A task with swapped names will still land right next to its original, which is what you want. But unrelated tasks that share the same template with different names will land there too, which produces false alarms. Exact-text search has the opposite flaw: it finds nothing once the wording changes. The two errors are mirror images, so embeddings suit finding candidates and exact-match or entity checks suit confirming them.
A paraphrase also isn't neutral, even when it's found. Models do systematically better on the more common phrasing of an identical meaning, across math, translation, commonsense reasoning and tool calling Do language models really understand meaning or just surface frequency?. So 'this is a rewording of that task' and 'this is the same test' are different claims. The note is about how models perform, not how embedding models retrieve. Whether embeddings also stumble on rare phrasings isn't tested here, though unusual rewordings are the likeliest place to expect misses.
Similarity also doesn't have to be the only score. TempRALM added a time-relevance term alongside semantic similarity and gained up to 74% on time-sensitive questions, with no retraining or index changes Can retrieval systems ground answers in the right time?. The same move could add a term for shared entities or answer format. The limits are real, though. Long-context models fail on queries that need relational joins, which suggests that matching wording and meaning is different from matching underlying structure Can long-context LLMs replace retrieval-augmented generation systems?.
On 'without human review', the corpus's closest example of trusting a scalable signal is Chatbot Arena, whose crowd votes earned credibility by agreeing with expert raters Can crowdsourced votes reliably rank language models?. The practical lesson is to shrink human review rather than drop it: hand-label a small sample, measure how often the matcher misses reworded or renamed tasks, and only then decide how much checking it can go without.
Sources 5 notes
The LOFT benchmark shows LCLMs match RAG on semantic retrieval without explicit training, but cannot execute relational queries requiring joins across structured tables. Context length alone cannot bridge this gap.
GrepSeek trains agents to retrieve via executable shell commands over raw text, achieving better multi-hop performance on entity-constrained queries than dense embeddings. The approach scaffolds unstable search mechanics with supervised trajectories, then refines task-oriented behavior through reinforcement learning.
LLMs show consistent preference for higher-frequency surface forms over semantically equivalent rare paraphrases across math, machine translation, commonsense reasoning, and tool calling. This suggests models track statistical mass from pretraining rather than meaning-recognition as their primary mechanism.
TempRALM adds a temporal term to retrieval scoring alongside semantic similarity, achieving up to 74% improvement over baseline systems when documents have multiple time-stamped versions. The approach requires no model retraining or index changes.
Chatbot Arena's 240K+ crowdsourced preference votes produce credible model rankings because the underlying questions are diverse and discriminating, and crowd judgments correlate with expert raters—validating human preference as a scalable evaluation signal.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- A New Role for Relevance: Guiding Corpus Interaction in Agentic Search
- BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms
- Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
- FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions
- GrepSeek: Training Search Agents for Direct Corpus Interaction
- Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
- It's About Time: Incorporating Temporality in Retrieval Augmented Language Models
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference