Freelance Agent Evaluation Engineer (Remote)

Job summary

This role involves evaluating AI coding agents by creating realistic developer environments and designing tasks that challenge these models. The position is project-based and fully remote.

Qualifications

• Minimum of 5 years of experience in software development.
• Proficient in English (B2+ level).

Responsibilities

• Develop realistic virtual company environments with codebases, infrastructure, and context.
• Create tasks from intermediate states of these environments, defining what constitutes a 'solved' task.
• Write tests to verify agent solutions, ensuring a balance between strictness and leniency.
• Iterate on tasks and tests based on quality assurance feedback, refining evaluations for fairness and robustness.

Skills

Compensation

• Proficient in Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
• Experience in writing functional and integration tests.
• Up to $50 per hour, depending on experience and task complexity. Tasks are estimated to take around 20 hours each, with flexible scheduling.