INQUIRING LINE

If an AI turns code into reusable skills, what should travel with each one so you can check it against the original?

What metadata properties make code-derived skills auditable and comparable to their original source?

This explores what information should travel with a skill extracted from source code, so a reader can check it and compare it back to the code it came from. The corpus supplies ingredients for this rather than one agreed metadata schema.


This explores what should travel with a skill lifted from source code so you can audit it and compare it to the original. The corpus has no note that lists metadata fields, so what follows is assembled from several neighbouring ideas. The closest match is Code2Skill, which extracted over a million skills from GitHub repositories. Can source code replace experience as skill raw material? It calls them auditable for two reasons: the skills keep executable evidence, and each one passes source-blind reconstruction verification. In plain terms, the skill is checked by whether the original behaviour can be rebuilt from it without looking at the code. That gives you two properties. The evidence attached to a skill can be run rather than just read, and the record of the reconstruction test is a comparison against the source that anyone can repeat.

The next properties come from how the corpus treats skills as files. COLLEAGUE.SKILL keeps distilled expertise as versioned files you can inspect, correct and roll back, rather than hiding it in prompt state. It also keeps what someone knows separate from how they act, so each can be audited on its own. Can person-grounded skills remain auditable without hidden prompt state? That note is about people, not code. But it suggests a code-derived skill needs a version history so drift from the source is visible, and a clear split between what the code does and how the skill tells an agent to use it. Dr. Claw makes a similar point at the process level. Persistent state objects give a traceable, recoverable trail while the underlying agent stays unchanged. Can orchestration layers make coding agents more auditable? SkillOpt adds a recorded justification for each change. Edits to a skill document are accepted only if they beat a held-out validation check, so a skill's history can say why each edit was kept. Can skill documents be optimized like neural network weights?

Comparability also depends on what is pinned. BenchShield fixes the artifacts its audit agents see, limits their remit, and requires them to cite evidence. It treats infrastructure records as the part nobody can dispute, and the audit judgment as the part open to argument. Can scoped agents reliably judge semantic hacks in runtime analysis? Applied to skills, a comparison to the source only means something if the exact source revision is fixed. Any claim of fidelity should then rest on recorded evidence rather than a reviewer's impression.

Two warnings explain why the checks need to be mechanical. First, in DELEGATE-52 even the strongest models silently corrupted about 25% of document content over long workflows, and spot checks missed it. Do frontier LLMs silently corrupt documents in long workflows? A skill written from code by an LLM can drift the same way while still looking fine, so an eyeball comparison with the source is unreliable. Second, LLM judges score higher when a response has fake references or rich formatting. Can LLM judges be fooled by fake credentials and formatting? So citations and tidy structure in a skill are not evidence of provenance unless they can be verified. ColluSkill adds a third point: scanners that score skills one at a time were beaten 96% of the time by attackers who kept the chain-level intent intact. Can attackers evade skill scanners by refining individual skills? Provenance metadata therefore probably needs to record how skills link and compose, not just what each one says.

In short, the corpus points to five properties: runnable evidence tied to the source, a repeatable source-blind reconstruction result, versioned files with rollback, a pinned source revision with evidence-citing audits, and a recorded validation gate for every edit. It does not say what the actual field names or format should be.


Sources 8 notes

Can source code replace experience as skill raw material?

Code2Skill extracts over 1 million auditable skills from GitHub repositories through source-blind reconstruction verification, improving performance from 42.90 to 47.90 macro-average across eight benchmarks, outperforming trajectory and document-derived alternatives.

Can person-grounded skills remain auditable without hidden prompt state?

COLLEAGUE.SKILL treats distilled expertise as versioned files subject to inspection, correction, and rollback—not hidden prompt state. Separating capability tracks from behavior tracks enables independent audit of what someone knows versus how they act.

Can orchestration layers make coding agents more auditable?

Dr. Claw wraps existing coding agents in persistent state objects and skill libraries, reporting higher research completeness and a traceable, recoverable process trail while keeping the underlying executor unchanged.

Can skill documents be optimized like neural network weights?

SkillOpt treats skill documents as trainable external state of frozen agents, using a text-space optimizer with held-out validation gating to accept only edits that improve performance. Across 52 benchmark cells and seven models, the approach matches or exceeds baselines while adding zero inference cost and enabling transfer across models.

Can scoped agents reliably judge semantic hacks in runtime analysis?

BenchShield constrains audit agents by limiting their remit, fixing the artifacts they see, and requiring evidence citation. This positions infrastructure records as unchallengeable checks and audit judgments as the arguable step after them, though reported reliability remains unquantified.

Show all 8 sources
Do frontier LLMs silently corrupt documents in long workflows?

Even the strongest models (Gemini 3.1 Pro, Claude 4.6 Opus, GPT 5.4) degrade documents by ~25% over long relay workflows across 52 domains. Degradation decelerates but never plateaus, and errors compound silently, remaining undetected in spot-checked outputs.

Can LLM judges be fooled by fake credentials and formatting?

Research identified four evaluation biases in LLM judges, with authority and beauty biases being semantics-agnostic and trivially exploitable through fake references and formatting—zero-shot attacks requiring no model access or optimization.

Can attackers evade skill scanners by refining individual skills?

ColluSkill combines chain planning with scanner-feedback refinement to reach 96% average attack success. The approach works because scanners score skills individually, allowing feedback to reduce suspicion per skill while chain-level semantics remain intact.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.