Freelance Agent Evaluation Engineer (Remote)

Job summary

This role involves evaluating AI coding agents by creating realistic developer environments and tasks to assess their performance on real-world developer tasks. The position is project-based and fully remote.

Qualifications

• Minimum of 5 years of experience in software development
• Proficiency in English at B2 level or higher

Responsibilities

• Develop realistic developer environments, including a virtual company with a codebase and context
• Design tasks based on intermediate states of these environments, defining success criteria for AI agents
• Write tests to verify the solutions provided by AI agents, ensuring a balance between strictness and leniency
• Iterate on tasks and tests based on quality assurance feedback, refining evaluations for fairness and robustness

Skills

Compensation

• Strong knowledge of Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis
• Experience in writing functional and integration tests
• Up to $50 per hour, depending on experience and task completion pace. Tasks are estimated to take around 20 hours each, with flexible scheduling.