code-review-orchestrator

A finding reproduced before integrating the change.

1. Regenerate without retraining

ShiftLab added --from-results to regenerate results.json, cobertura-error.html and informe.txt from a historical JSON file. It had to preserve the threshold and counts without retraining.

2. Review against explicit criteria

review_diff.py evaluated the change against explicit criteria. The reviewer flagged a specific failure: an invalid Unicode character could leave partial output. The finding was treated as a hypothesis to reproduce.

3. Reproduce and fix

The failure was reproduced. The fix prepared and encoded all three contents as UTF-8 before creating the output directory, preventing partial files from being left by that error. A regression test was added.

4. Verify and integrate

All 61 ShiftLab tests passed. The second review had no findings, and running with the historical JSON confirmed all three files, the same threshold and the same counts. The PR was merged as 5d3ad83.

Later: isolating evaluation

Dynamic test inventory and evaluation started from orchestrate_project.py execute project code: test discovery can import it too.

PR #27 isolated inventory_base() and evaluate_flow(), including configurable tasks and multi-file submissions, in rootless Podman, with networking disabled and read-only inputs. Direct calls to other runners and the agents remain outside that scope.

Per-evaluation quotas cap results at 32 MiB and 1024 inodes, and stdout and stderr at 4 MiB each. If isolation fails, the workflow stops without running the candidate on the host.

Local evidence for that change: 10 passing integrations with real Podman containers and 834 passing ordinary tests (10 other opt-in tests skipped). This evidence is separate from ShiftLab’s 61 tests; the linked CI also completed successfully.

The ShiftLab review did not use this isolated evaluator. Integrating changes remains a human decision.