refactor: improve verification workflow with visual comparison
Major changes: - paper-image-extractor: Generate reference_plots.py for visual verification - paper-director: Add image understanding checkpoint with side-by-side comparison - paper-analyzer: Add data source labeling with reliability levels - code-writer: Change from TDD to VDD (Verification-Driven Development) - test-runner: Generate comparison reports with images and explanations - verification skill: Add difference classification system - code-generation skill: Emphasize result independence Key principles: - Code results are authoritative, paper values are references - Differences are expected and documented, not bugs to fix - Visual comparison prioritized over exact numerical match - Tests verify sanity (shape, gradient, range), not exact values
This commit is contained in:
@@ -17,6 +17,36 @@ Guidelines for translating paper descriptions into working PyTorch code.
|
||||
2. **Testability**: Write code that can be unit tested
|
||||
3. **Readability**: Prefer clarity over cleverness
|
||||
4. **Modularity**: One component per file
|
||||
5. **Independence**: Code logic based on paper methodology, NOT reverse-engineered from expected outputs
|
||||
|
||||
## Critical: Result Independence
|
||||
|
||||
The code must implement the **paper's described method**, not be reverse-engineered to match reference values.
|
||||
|
||||
### DO NOT:
|
||||
```python
|
||||
# WRONG: Using values from reference_plots.py as targets
|
||||
expected_accuracy = 0.952 # Copied from paper figure
|
||||
assert abs(accuracy - expected_accuracy) < 0.01 # This defeats the purpose
|
||||
```
|
||||
|
||||
### DO:
|
||||
```python
|
||||
# CORRECT: Implement the method, let results be what they are
|
||||
# Paper Section 4.1: "We use Adam with lr=1e-4"
|
||||
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
|
||||
|
||||
# Run training, record actual results
|
||||
accuracy = evaluate(model, test_loader)
|
||||
# This accuracy is authoritative - compare with paper in report
|
||||
```
|
||||
|
||||
### Reference Values Are For Comparison Only
|
||||
|
||||
Values from `image_understanding.md` and `reference_plots.py` should:
|
||||
- Be used in the **final report** for comparison
|
||||
- **NOT** be used as assertion targets in tests
|
||||
- **NOT** influence implementation decisions
|
||||
|
||||
## Paper-to-Code Mapping
|
||||
|
||||
@@ -199,3 +229,5 @@ Before completing a module:
|
||||
- [ ] Example in docstring works
|
||||
- [ ] No hardcoded dimensions (use params)
|
||||
- [ ] Gradient flow verified (no in-place ops breaking autograd)
|
||||
- [ ] **No reference values hardcoded as expected outputs**
|
||||
- [ ] **Implementation based on paper method, not reverse-engineered from results**
|
||||
|
||||
Reference in New Issue
Block a user