Freelance Agent Evaluation Engineer (Remote)

Job summary

This role involves evaluating AI coding agents by creating realistic developer tasks and environments. The position is project-based and fully remote, focusing on testing and improving AI systems for leading tech companies.

Qualifications

• Minimum of 5 years in software development.
• Educational background: Master’s Degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, or related fields; a Bachelor’s degree is acceptable with 5 years of experience.
• At least 3 years of professional experience in QA-automation/testing or cybersecurity roles.
• English proficiency at B2 level or higher.

Responsibilities

• Develop realistic developer environments, including a virtual company setup with codebase and context.
• Design tasks based on intermediate states of these environments, defining success criteria for AI agents.
• Write tests to verify solutions provided by AI agents, ensuring a balance between acceptance of valid approaches and rejection of incorrect ones.
• Iterate on tasks and tests based on quality assurance feedback, refining evaluations to ensure fairness and robustness.

Skills

Compensation

• Proficiency in Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
• Experience in writing functional and integration tests.
• Payment is per accepted task, with rates up to the equivalent of $50/hr based on qualification tier and task efficiency. This freelance opportunity allows for flexible part-time work that can accommodate other professional or academic commitments.