Senior Python Engineer (Remote)

Job summary

This role involves evaluating AI coding agents by creating realistic developer tasks and environments. The focus is on building datasets that assess how well AI models perform in real-world scenarios.

Qualifications

• Over 8 years of experience in software development
• Proficiency in Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis
• Experience in writing functional and integration tests
• English proficiency at B2 level or higher

Responsibilities

• Develop realistic developer environments that simulate a virtual company with a codebase and context
• Design tasks based on intermediate states of these environments, defining success criteria for AI agents
• Write tests to verify the solutions provided by AI agents, ensuring a balance between strictness and leniency
• Iterate on tasks and tests based on quality assurance feedback, refining them to ensure fair evaluations

Skills

Compensation

• Strong understanding of AI model capabilities and limitations in coding
• Ability to create challenging tasks that effectively differentiate between good and bad solutions
• Up to $200 per hour, based on experience and task complexity. Tasks are estimated to take around 30 hours each, with flexible scheduling.