Senior Python Engineer (Remote)

Job summary

This role involves evaluating AI coding agents by creating realistic developer tasks and environments. You will be responsible for designing tasks, writing tests, and iterating based on feedback to ensure fair evaluations of AI solutions.

Qualifications

• Over 8 years of experience in software development
• Proficient in Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis
• Experience in writing functional and integration tests
• English proficiency at B2 level or higher

Responsibilities

• Build realistic developer environments that simulate a virtual company with a codebase and context
• Design tasks from intermediate states of these environments, defining what constitutes a 'solved' task
• Write tests to verify agent solutions, ensuring a balance between accepting valid approaches and rejecting incorrect ones
• Iterate on tasks and tests based on quality assurance feedback to refine evaluations

Skills

Compensation

• Strong understanding of AI model limitations and capabilities in coding
• Ability to create challenging tasks that effectively evaluate AI performance
• Payment is based on accepted tasks, with rates up to $200/hr depending on qualifications and efficiency.