When an AI helps write a research paper, which jobs can plain code handle so you don't have to trust its judgment?
What deterministic operations can replace model judgment in scientific writing?
This explores which steps in writing a research paper can be handed to code, fixed rules, or tools that give the same answer every time, so the model doesn't have to be trusted to judge them.
This is about which parts of paper-writing can be handed to code, pre-set rules, or tools instead of asking a language model to decide. The corpus has one note squarely on this, Can separating judgment from verification improve research paper reliability?. It gives a design principle more than a checklist. Split the work into model judgment (what the argument is, how to phrase it) and executable, verifiable operations, so the paper's reliability rests on the second half rather than on the model being right. It also requires writing down what evidence would count before any results are observed, so the claim can't be bent to fit the numbers afterward.
Two kinds of operation qualify. The first is procedural work a model does badly in plain text. Are reasoning model collapses really failures of reasoning? finds that models often know the right algorithm but can't carry out long multi-step procedures as text. When they're given tools, they solve problems past the supposed reasoning cliff. Do tools actually expand what language models can reason about? shows formally that tools open up strategies that are impossible or absurdly verbose in text alone. Applied to writing, this suggests recomputing figures and running cross-checks instead of asking the model to eyeball them. That application is my extrapolation, since neither note is about papers.
The second kind is rules fixed before you look at the output. Does iterative prompt engineering undermine scientific validity? argues that one researcher tweaking prompts until the output looks right builds in bias and self-fulfilling feedback. Its replacement is a validated pipeline with pre-specified criteria and inter-coder reliability. The 'deterministic' part here is procedural: the standard is decided in advance, and agreement is measured rather than assumed.
The word 'deterministic' has a trap. Does setting temperature to zero actually make LLM outputs reliable? shows that temperature zero and a fixed seed give you the same output every time, but that output is still one draw from the model's distribution. Testing across 100 repetitions shows consistency isn't reliability. Locking the model's dials doesn't replace judgment; the check has to sit outside the model. The notes also say where judgment is weakest. Does model confidence predict robustness to prompt changes? finds that low-confidence outputs swing hard with small rephrasings, so those are the steps to move onto a check. Asking the model to think harder doesn't fix it either. Why does more reasoning sometimes make models worse? shows longer reasoning can make a model overwrite an answer it already had right.
Judgment still has a job. Can specialized agents write better scientific papers than single models? reports specialized agents beating single-agent baselines by 50 to 68 percent on literature review. That splits judgment across agents rather than replacing it. And Can LLMs predict novel scientific results better than experts? shows LLMs predicting neuroscience results better than experts, so the pattern-integrating habit is a real strength when it's pointed the right way. The picture is that the model synthesizes and frames, while deterministic checks bound what it can get wrong. The corpus has no catalog of specific checks, such as citation verification or number reconciliation, so it supports the principle better than it supplies a toolkit.