Freelance Agent Evaluation Engineer (Remote)

Job summary

This role involves evaluating AI coding agents by creating realistic developer environments and designing tasks that challenge AI models. The position is project-based and fully remote.

Qualifications

• Minimum of 5 years in software development.
• Master’s Degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, or related fields; a Bachelor’s degree is acceptable with 5 years of experience.
• Minimum of 3 years of experience in QA-automation/testing or cybersecurity roles.
• English proficiency at B2 level or higher.

Responsibilities

• Build realistic developer environments that simulate a virtual company with a codebase and context.
• Design tasks from intermediate states of these environments, defining what constitutes a 'solved' task.
• Write tests to verify agent solutions, ensuring a balance between strictness and leniency.
• Iterate on tasks and tests based on quality assurance feedback to refine evaluations.

Skills

Compensation

• Core stack proficiency: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis.
• Experience in writing functional and integration tests.
• Payment is per accepted task, with rates up to the equivalent of $50/hr based on qualification tier and task efficiency.