Software Engineering Evaluation Specialist (Remote)

Job summary

This role involves designing coding tasks for AI agents to evaluate their capabilities in fixing software issues. Each task is delivered as a complete package, including broken code, tests, and instructions. The position is project-based and fully remote.

Qualifications

• Minimum of 3 years of experience in production software development using one backend stack (Python, Go, Node.js, Java, or Rust).
• Proficiency in Python and pytest is essential, regardless of the primary programming language.
• Experience in creating reproducible Docker environments with pinned dependencies.
• Comfortable with Linux and Bash for debugging within containers.
• Familiarity with AI coding agents and their evaluation.
• English proficiency at B2 level or higher.

Responsibilities

Not in scope

Preferred qualifications

Process

Compensation

• Create realistic developer scenarios that reflect actual software issues.
• Develop reproducible Docker environments for coding tasks.
• Write pytest scripts to verify outcomes without leaking fixes.
• Draft clear instructions that resemble Jira tickets for developers.
• Provide reference solutions demonstrating task solvability.
• Adjust task difficulty based on performance metrics of AI agents.
• Review and provide feedback on tasks created by other authors.
• Data labeling, prompt engineering, or production code development.
• Simple coding puzzles; tasks must reflect real developer work.
• Expertise in Security, System Administration, Scientific Computing, DevOps, or Git internals.
• Familiarity with modern Python tooling and coverage tools.
• Experience with fuzzing or property-based testing.
• Contributions to agent-evaluation benchmarks or related frameworks.
• Application includes a qualification screen and a behavioral interview.
• Flexible time commitment with an estimated workload of 8–20 hours per week.
• Paid contributions with rates up to $35/hour, depending on expertise and project needs.