SYNTHESIS NOTE
Topics›Agent Harness›this note

Can source code replace experience as skill raw material?

Existing skill synthesis relies on agent trajectories or documents, each with limitations. Could static code repositories serve as a more reliable, scalable foundation for deriving reusable procedural skills?

Synthesis note · 2026-09-25 · sourced from Agent Harness

The paper's claim is about where reusable skills should come from. It names two weaknesses in existing synthesis: trajectory-based skills are "inherently coupled to the environments" that produced them, and document-sourced skills "may lack evidence support and verifications." Source code, it argues, "requires no previous experience while retaining executable evidence for grounding later abstractions." Code2Skill is the pipeline built on that argument, and CodeSkillBank is its output: 1,006,822 accepted records drawn from 19,769 popular, actively maintained GitHub repositories.

The mechanism has two parts. First, selected code units are lifted into three kinds of implementation-anchored record: atomic-operation, composite-workflow, and recurring-pattern skills. Each accepted record carries workflow, boundary, provenance, and source-evidence metadata. Second, every record is verified by "source-body-blind reconstruction followed by source-aware comparison." A skill has to be specific enough that the function can be rebuilt without seeing its body, and the rebuild is then checked against the original. The evidence is a property of the record, which is why the authors call the result "auditable" as well as scalable.

The reported downstream result is that skills retrieved from CodeSkillBank lift the macro-average score from 42.90 to 47.90 across nine model settings and eight benchmarks. They win 57 of 72 protocol-matched evaluations, "especially" on coding tasks. Under a unified interface they also beat the compared trajectory-derived skill banks on all seven shared benchmarks.

This sits against the trajectory-driven skill notes in the library. Can a separate trained curator improve skill libraries better than frozen agents?, How can agent systems share learned skills across users? and Can agents learn better from their failures than successes? all treat lived experience as the raw material and put the effort into curating, aggregating or distilling it. Code2Skill removes the experience requirement: the raw material already exists as static artifacts, and its supply is bounded by the number of repositories rather than by agent runs. It also gives a different answer to the grounding problem. Where those notes lean on feedback and self-judgment, this pipeline leans on comparison against source. Can agents learn new skills without forgetting old ones? is the closest structural relative, since both hold executable skills in a retrievable library, but there the library grows through an agent's own exploration.

The excerpt leaves a lot unsaid. It does not define the baseline behind 42.90, name the eight benchmarks or the nine model settings, or say how skills are retrieved. It gives no rejection rate for the verification stage, so "accepted" carries no information about how selective it was. It does not say how a "code unit" is selected, or whether the composite and pattern records are built from atomic ones or lifted directly. The 15 of 72 evaluations without a gain are not described. The stress on coding tasks suggests domain proximity matters, but no per-domain breakdown appears. The claim that stands is narrower than "code beats experience": for the compared trajectory-derived banks, code-derived skills did better on the shared benchmarks. Whether code can supply skills for work that leaves no code behind remains open.

Inquiring lines that read this note 8

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

When should work require human-AI partnership versus full automation? How should agent systems validate and persist generated code artifacts? How do agent-learned skills transfer and improve across different tasks?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
14 direct connections · 91 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

skills lifted from source code need no prior experience and keep executable evidence for grounding, unlike trajectory-derived or document-sourced skills