If you flip a pattern and then swap its colors, does that still work when the shapes are arranged differently?
Can reflection and color swapping compose reliably across different motif layouts?
This explores whether two simple pattern operations, mirroring a motif and swapping its colors, can be chained and still work when the motif's arrangement changes.
This explores whether mirroring a motif and swapping its colors can be chained reliably when the motif's arrangement changes. The corpus has no study of this exact pairing, so what follows is inference from how models handle composition in general, not a direct answer.
The closest evidence leans toward "not reliably." Research on transformers finds they Do transformers actually learn systematic compositional reasoning? succeed on familiar tasks by memorizing computation paths seen in training, not by learning general rules. On novel combinations they fail sharply, and errors compound with each added step. A new layout is a new combination. A model that has seen "reflect, then recolor" on grid-like motifs may only have memorized that path for those layouts, and a ring or a scattered cluster would start from scratch.
There is a counterweight. Networks often do Do neural networks naturally learn modular compositional structure? split compositional tasks into separate subnetworks. Ablating one affects only its own function, and pretraining makes this split more consistent across architectures and domains. So a model may hold a clean "mirror" part and a clean "swap colors" part. If both findings hold, the weak point is the wiring between the parts, not the parts themselves. That reading is my inference, not something either paper tests.
The two operations also have a special property. Mirroring changes where things are, and color swapping changes what color they are, so they should give the same result in either order and on any layout. A system that truly understood both would show that invariance. A subgraph-matching system would treat each order and each layout as a separate case to memorize. Checking that order-swapped and layout-swapped versions agree would be a sharp test of which one you have.
The corpus also suggests some ways to make composition sturdier, though none were tested on this problem. How you split a task matters a lot: What blocks skill retrieval in task decomposition? found that standard decomposition recovered only 34% of the right steps, and that this gated everything after it. Separating a where-question from a what-question also helps in other settings, as when Why do vision-only GUI agents struggle with screen interpretation? pre-parsing a screen removed the burden of identifying and acting at once. And Can bounding boxes replace image encoders for document understanding? treats layout as its own signal, kept apart from content. Together these hint that giving layout its own representation, separate from color, might make the two operations less likely to interfere.
Sources 5 notes
Research shows transformers succeed on in-distribution tasks by memorizing computation subgraphs from training data, not by learning systematic rules. They fail drastically on novel compositions, with errors compounding across reasoning steps.
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
Standard LLM decomposition reaches only 34% step-level recall, gating retrieval success. Correcting step count recovers 75% of gains in iterative methods, shifting the bottleneck to representation-level reranking rather than vocabulary alignment.
OmniParser demonstrates that GPT-4V fails when forced to simultaneously identify icon meanings and predict actions from raw screenshots. Pre-parsing screenshots into structured semantic elements with descriptions lets the model focus solely on action prediction, removing the composite-task bottleneck.
DocLLM shows that bounding-box spatial information combined with decomposed transformer attention can capture text-spatial alignment in documents without pixel-based visual encoding. Pretraining on text-infilling objectives suited to irregular layouts achieves this at substantially lower computational cost than multimodal LLMs using image encoders.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Break It Down: Evidence for Structural Compositionality in Neural Networks
- Scaling can lead to compositional generalization
- Faith and Fate: Limits of Transformers on Compositionality
- How do Transformers Learn Implicit Reasoning?
- ScreenAI: A Vision-Language Model for UI and Infographics Understanding
- OmniParser for Pure Vision Based GUI Agent
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent
- From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks