refactor: improve verification workflow with visual comparison
Major changes: - paper-image-extractor: Generate reference_plots.py for visual verification - paper-director: Add image understanding checkpoint with side-by-side comparison - paper-analyzer: Add data source labeling with reliability levels - code-writer: Change from TDD to VDD (Verification-Driven Development) - test-runner: Generate comparison reports with images and explanations - verification skill: Add difference classification system - code-generation skill: Emphasize result independence Key principles: - Code results are authoritative, paper values are references - Differences are expected and documented, not bugs to fix - Visual comparison prioritized over exact numerical match - Tests verify sanity (shape, gradient, range), not exact values
This commit is contained in:
@@ -7,10 +7,27 @@ description: Use when verifying replication results against paper's reported val
|
||||
|
||||
## Overview
|
||||
|
||||
Systematic approach to verifying that replicated code produces results matching the original paper.
|
||||
Systematic approach to verifying that replicated code produces results comparable to the original paper. **Note**: Exact matches are rare; the goal is verifiable, explainable results.
|
||||
|
||||
**Announce at start:** "I'm using the verification skill to validate replication accuracy."
|
||||
|
||||
## Core Philosophy
|
||||
|
||||
1. **Code results are authoritative** - Our implementation's output is ground truth
|
||||
2. **Paper values are references** - Used for comparison, not as test assertions
|
||||
3. **Differences require explanations** - Not fixes (unless clearly buggy)
|
||||
4. **Visual comparison over numerical** - Trends matter more than exact values
|
||||
|
||||
## Difference Classification System
|
||||
|
||||
| Status | Symbol | Criteria | Action |
|
||||
|--------|--------|----------|--------|
|
||||
| MATCH | ✅ | < 2% difference | Document, no action needed |
|
||||
| ACCEPTABLE | ⚠️ | 2-10% difference | Document with brief explanation |
|
||||
| EXPLAINABLE | 📝 | > 10%, cause identified | Document cause thoroughly |
|
||||
| INVESTIGATE | 🔍 | > 10%, cause unknown | Review implementation |
|
||||
| PAPER_ISSUE | 📄 | Our results more reasonable | Document evidence |
|
||||
|
||||
## Verification Levels
|
||||
|
||||
### Level 1: Code Correctness
|
||||
@@ -176,15 +193,78 @@ def compare_with_variance(
|
||||
```markdown
|
||||
## Verification Result: {Metric Name}
|
||||
|
||||
**Paper Value**: {value} ± {std}
|
||||
**Paper Value**: {value} ± {std} (Source: {figure/table/text})
|
||||
**Our Value**: {value} ± {std}
|
||||
**Difference**: {absolute} ({relative}%)
|
||||
|
||||
**Status**: MATCH | ACCEPTABLE | INVESTIGATE | MISMATCH
|
||||
**Status**: MATCH | ACCEPTABLE | EXPLAINABLE | INVESTIGATE | PAPER_ISSUE
|
||||
|
||||
**Analysis**:
|
||||
{explanation of difference}
|
||||
{explanation of difference - required for all non-MATCH statuses}
|
||||
|
||||
**Confidence**: {HIGH | MEDIUM | LOW}
|
||||
{reasoning for confidence level}
|
||||
```
|
||||
|
||||
## Visual Comparison Guidelines
|
||||
|
||||
### Side-by-Side Figure Comparison
|
||||
|
||||
Always present figures in side-by-side format:
|
||||
|
||||
```markdown
|
||||
| Paper Reference | Our Replication |
|
||||
|-----------------|-----------------|
|
||||
|  |  |
|
||||
```
|
||||
|
||||
### What to Compare
|
||||
|
||||
1. **Trends**: Does the curve go up/down at the same places?
|
||||
2. **Shape**: Is the overall shape similar?
|
||||
3. **Key points**: Do peaks/valleys occur at similar locations?
|
||||
4. **Scale**: Are values in the same order of magnitude?
|
||||
|
||||
### Acceptable vs Unacceptable Differences
|
||||
|
||||
**Acceptable** (document and move on):
|
||||
- Curve shifted slightly up/down (offset)
|
||||
- Slightly faster/slower convergence
|
||||
- Small noise differences
|
||||
|
||||
**Unacceptable** (investigate):
|
||||
- Opposite trends (going up vs down)
|
||||
- Completely different shapes
|
||||
- Order of magnitude differences
|
||||
- Missing features (e.g., expected oscillation absent)
|
||||
|
||||
## Common Difference Sources
|
||||
|
||||
### Expected Differences (ACCEPTABLE)
|
||||
|
||||
| Source | Typical Impact | Mitigation |
|
||||
|--------|---------------|------------|
|
||||
| Random seed | 1-3% | Run multiple seeds, report mean±std |
|
||||
| Floating point | < 0.1% | Use float64 for verification |
|
||||
| Framework differences | 1-5% | Document framework version |
|
||||
| Hardware differences | 0.5-2% | Note in report |
|
||||
| Batch size changes | 2-10% | Adjust LR proportionally |
|
||||
|
||||
### Concerning Differences (INVESTIGATE)
|
||||
|
||||
| Source | Typical Impact | Action |
|
||||
|--------|---------------|--------|
|
||||
| Wrong architecture | > 10% | Review code vs paper |
|
||||
| Wrong hyperparameters | 5-20% | Verify all settings |
|
||||
| Data preprocessing | Variable | Match paper exactly |
|
||||
| Bug in implementation | Variable | Debug systematically |
|
||||
|
||||
### Paper Issues (PAPER_ISSUE)
|
||||
|
||||
Sometimes the paper contains errors. Signs include:
|
||||
- Results that violate mathematical constraints
|
||||
- Impossible performance claims
|
||||
- Inconsistencies between text and figures
|
||||
- Known errata
|
||||
|
||||
Document evidence thoroughly if claiming paper issue.
|
||||
|
||||
Reference in New Issue
Block a user