Passing ordinary tests proves selected outputs, but it may miss unsafe behavior around one-shot iterables, object identity, input mutation, time snapshots, injected dependencies and stable ordering.
Shivanand Kumar← Back to public work
Independent Open-Source Project · Fully synthetic · CI passing
Independent Open-Source Project
Constraint-Aware Coding Agent Evaluation Lab
A reproducible evaluation-engineering project that tests returned product values and implementation behavior as separate axes, using only independently authored synthetic scenarios, candidates and evidence.
16project tests passed
45evaluator checks passed
66.7 ppcompliance separation
6/6 vs 2/6synthetic constraint results
Problem and ownership
A real engineering question, built to be inspected.
I designed and built the scenario format, standard-library evaluator, focused runtime probes, accepted and adversarial validation fixtures, dual-axis grading, reproducible reports, release scanner, manifest checks, tests and CI.
Verified output
Concrete behavior, not a concept slide.
Abbreviated verified result derived from the reproducible example report. Both authored candidates pass the same ordinary functional tests; the focused probes expose four behavior violations in the comparison candidate.
{
"synthetic_reference": {
"functional_tests": "PASS",
"constraint_compliance": "6/6"
},
"synthetic_comparison": {
"functional_tests": "PASS",
"constraint_compliance": "2/6"
},
"compliance_separation": "66.7 percentage points"
}What the repository proves
Designed, tested and documented end to end.
- Python 3.11+ standard-library evaluation runtime and command-line interface
- Six independently evaluated runtime behavior constraints
- Fourteen accepted and twelve isolated adversarial implementation fixtures
- Nineteen functional expectation checks and two decoupling fixtures
- Byte-reproducible JSON and Markdown comparison reports
- Public-release scanning and an 84-file SHA-256 manifest
Architecture flowDesign decisions ↗
- 01
Validate the fully synthetic scenario, paths and evidence map
- 02
Materialize each authored candidate in a temporary workspace
- 03
Run the same ordinary functional tests for both candidates
- 04
Observe six behavior constraints through focused runtime probes
- 05
Grade both axes independently and generate reproducible evidence
Limits and next validation
What this project does not prove—yet.
- This is a small deterministic teaching example, not a production benchmark or a claim about real-world model performance.
- The candidates were authored to exercise evaluation paths; results must not be generalized to external repositories, agents or models.
- Temporary workspaces provide file separation, not a security sandbox. Only trusted synthetic fixtures should be evaluated.
- The release scanner checks configured patterns in publishable text files; it does not replace credential rotation or repository-host history cleanup after an accidental disclosure.
Credibility boundary
Independent work, described honestly.
Independent Open-Source Project. It is fully synthetic and independently authored, with no employer or client work, confidential tasks, prompts, tests, trajectories or datasets. It is not a production deployment, a real benchmark submission, an accepted third-party contribution or evidence about any commercial model.