refactor: improve verification workflow with visual comparison

Major changes:
- paper-image-extractor: Generate reference_plots.py for visual verification
- paper-director: Add image understanding checkpoint with side-by-side comparison
- paper-analyzer: Add data source labeling with reliability levels
- code-writer: Change from TDD to VDD (Verification-Driven Development)
- test-runner: Generate comparison reports with images and explanations
- verification skill: Add difference classification system
- code-generation skill: Emphasize result independence

Key principles:
- Code results are authoritative, paper values are references
- Differences are expected and documented, not bugs to fix
- Visual comparison prioritized over exact numerical match
- Tests verify sanity (shape, gradient, range), not exact values
This commit is contained in:
hc
2026-03-31 19:55:36 +08:00
parent db731f6745
commit 5d5aee1f83
7 changed files with 683 additions and 270 deletions
+84 -4
View File
@@ -7,10 +7,27 @@ description: Use when verifying replication results against paper's reported val
## Overview
Systematic approach to verifying that replicated code produces results matching the original paper.
Systematic approach to verifying that replicated code produces results comparable to the original paper. **Note**: Exact matches are rare; the goal is verifiable, explainable results.
**Announce at start:** "I'm using the verification skill to validate replication accuracy."
## Core Philosophy
1. **Code results are authoritative** - Our implementation's output is ground truth
2. **Paper values are references** - Used for comparison, not as test assertions
3. **Differences require explanations** - Not fixes (unless clearly buggy)
4. **Visual comparison over numerical** - Trends matter more than exact values
## Difference Classification System
| Status | Symbol | Criteria | Action |
|--------|--------|----------|--------|
| MATCH | ✅ | < 2% difference | Document, no action needed |
| ACCEPTABLE | ⚠️ | 2-10% difference | Document with brief explanation |
| EXPLAINABLE | 📝 | > 10%, cause identified | Document cause thoroughly |
| INVESTIGATE | 🔍 | > 10%, cause unknown | Review implementation |
| PAPER_ISSUE | 📄 | Our results more reasonable | Document evidence |
## Verification Levels
### Level 1: Code Correctness
@@ -176,15 +193,78 @@ def compare_with_variance(
```markdown
## Verification Result: {Metric Name}
**Paper Value**: {value} ± {std}
**Paper Value**: {value} ± {std} (Source: {figure/table/text})
**Our Value**: {value} ± {std}
**Difference**: {absolute} ({relative}%)
**Status**: MATCH | ACCEPTABLE | INVESTIGATE | MISMATCH
**Status**: MATCH | ACCEPTABLE | EXPLAINABLE | INVESTIGATE | PAPER_ISSUE
**Analysis**:
{explanation of difference}
{explanation of difference - required for all non-MATCH statuses}
**Confidence**: {HIGH | MEDIUM | LOW}
{reasoning for confidence level}
```
## Visual Comparison Guidelines
### Side-by-Side Figure Comparison
Always present figures in side-by-side format:
```markdown
| Paper Reference | Our Replication |
|-----------------|-----------------|
| ![](ref_fig.png) | ![](our_fig.png) |
```
### What to Compare
1. **Trends**: Does the curve go up/down at the same places?
2. **Shape**: Is the overall shape similar?
3. **Key points**: Do peaks/valleys occur at similar locations?
4. **Scale**: Are values in the same order of magnitude?
### Acceptable vs Unacceptable Differences
**Acceptable** (document and move on):
- Curve shifted slightly up/down (offset)
- Slightly faster/slower convergence
- Small noise differences
**Unacceptable** (investigate):
- Opposite trends (going up vs down)
- Completely different shapes
- Order of magnitude differences
- Missing features (e.g., expected oscillation absent)
## Common Difference Sources
### Expected Differences (ACCEPTABLE)
| Source | Typical Impact | Mitigation |
|--------|---------------|------------|
| Random seed | 1-3% | Run multiple seeds, report mean±std |
| Floating point | < 0.1% | Use float64 for verification |
| Framework differences | 1-5% | Document framework version |
| Hardware differences | 0.5-2% | Note in report |
| Batch size changes | 2-10% | Adjust LR proportionally |
### Concerning Differences (INVESTIGATE)
| Source | Typical Impact | Action |
|--------|---------------|--------|
| Wrong architecture | > 10% | Review code vs paper |
| Wrong hyperparameters | 5-20% | Verify all settings |
| Data preprocessing | Variable | Match paper exactly |
| Bug in implementation | Variable | Debug systematically |
### Paper Issues (PAPER_ISSUE)
Sometimes the paper contains errors. Signs include:
- Results that violate mathematical constraints
- Impossible performance claims
- Inconsistencies between text and figures
- Known errata
Document evidence thoroughly if claiming paper issue.